As part of our launch of Nexus 2.0 and the Repository Health Check, we’re telling some stories about security and how security affects working developers. As developers we’re not always focused on security, but as attacks grow more complex, more aware of platforms like Java and .NET, and more capable of affecting custom application code, security is going to play an increasingly important role in development. In this post, I talk about a security incident I watched unfold last year and identify some of the lessons I walked away from the experience with.
But first, a message from our sponsor: Sonatype’s Nexus 2.0 offers a Repository Health Check. It’s not an answer to security by itself, but it can play a critical part in a larger, organization-wide approach to security. If you are developing applications with Tomcat, Spring, and other well-known components you’d be surprised at the kinds of vulnerabilities that are floating around in production. Once you have this capability to run a detailed RHC report in Nexus Professional, you can remove these components from applications, iterate, and get rid of known vulnerabilities. Click here to learn more about Nexus 2.0.
To most non-technical people, technology is magic...
I spent a good amount of time last year dealing with a complex security incident. First, let me be clear, this incident had nothing to do with Sonatype. It was a company that I happened to share office space with and to protect the innocent, I won’t name the company. Let’s just say this is a company that depended on the internet for its business to keep working. If the site went down, the business stopped until the site came back up. It’s as simple as that.
While the business relied on technology, none of the direct employees were very technical. They outsourced hosting and application development to contractors and they didn’t give technology another thought. To most non-technical people, technology is magic: websites just work, databases just sit there and “database” all day. If you start talking about different types of hosting servers and implementation languages you can’t expect most people to get into the technology.
Developers are flexible
That's why companies pay people like you and I. I’m a developer, I understand most things you’d expect a developer to know about a particular technology domain. For example, everyone who develops web applications should know at least the basic of TCP/IP and HTTP protocol. Basic stuff, things that you’d expect to be asked in an initial interview. Beyond that basic collection of facts we’re all expected to know, there are extras. For example, some developers know a little bit more about operations than others, some developers can double as DBAs, and other developers do a good job pretending to be managers.
Developers are good at adapting and of everyone in the IT department we’re the most likely to have multiple responsibilities or responsibilities that span job descriptions. As a good developer, you have to be able to understand abstractions and model process, you have to be able to adapt. But, despite the flexibility of our profession, I’ve noticed one glaring gap - security.
Yeah, I don’t know, we’ve got security people for that...
Back to this security incident. A web hosting company had deployed some seriously custom application logic that was full of problems. For starters, they created a website that didn’t accept last names with apostrophes. “Why?”, I asked. “Well, the web doesn’t work apostrophes.” Ignoring the fact that this was wrong, the SQL error produced by the last name “O’Brien” was also evidence of a SQL injection vulnerability. As a good neighbor, I notified the CEO of the company of the vulnerability and told him to ask his contractors to fix it.
Their response, “Yeah, our ISP has security people, I’m sure it’s secure.”
Weeks passed and the site was finally hacked. Someone had found a way to perform a cross-site scripting hack using a vulnerable plugin. Overnight, my colleague’s business went from making money to selling Viagra via a hacked Google Sitemap. Not just that, the machine was owned, being repeatedly hit from several IP addresses in Russia and the Philippines, and it was triggering some Level 1 SMTP blacklists.
We’re not security people, do you have any security people?
This is when it started to get fun. My neighbor, the CEO, initially got some feedback that everything was fixed and not to worry. A week passed and the attack expanded. Instead of just a single page selling ED medication, the entire site was compromised. The server was now being flagged as a Level 2 SMTP blacklist threat (which means that the entire ISP was being compromised). No customers were returning his calls and the business was dead in the water.
While the application developers had identified several modified files in the application and were confident that the problem had been isolated, my colleague had lost confidence in his developers. The fact that they had initially side-stepped responsibility for security was what bugged him the most.
If you don’t think you are responsible for security, think again.
I wasn’t working for either the company or contractor at the time, but I was there as a sort of second-opinion, a sounding board and the lesson I took away from the experience is that non-technical people don’t care about the nuanced differences between developers, sysadmins, DBA, and security people. The other lesson to take away from this is that we’re all security people. Unless you work behind seventeen national-security level firewalls, you’d better be focused on building secure systems.
You’d better not use plugins and software that have known vulnerabilities and the big secret, there’s no such thing as “A Security Person”. The big news flash here is that *you* are the security person and if you are writing any code whatsoever, you should be thinking about vulnerabilities and how to avoid them.
When you suffer through an attack (and it’s going to happen, eventually), there’s no excuse. There’s especially no excuse if you already knew about a vulnerability.
Today’s Attacks: New tricks up sleeves
Today’s attacks are complex. They are “multi-modal”. Instead of just compromising a single system-level component like BIND, you’ll find that today’s attacks are starting to involve custom application code as well as system-level hacks. That single web application that was compromised set the stage for an attack on other systems because someone’s SSH password was the same as a password stored, unencrypted in a MySQL database. This means that even if you were to clean up the affected application code, there’s no telling what else was compromised. In this way, it’s become next to impossible to clean up a compromised system. More often than not, the best answer when you suffer a complex attack is to start over (as crazy and impractical as that sounds).
That, I guess, was one of the biggest lessons I learned from watching this unfold. People are doing this these days and they are taking advantage of the same levels of automation that we are. Everything is scripted and automatic. Once they find one way in, they’ll compromise several vulnerabilities at once and leave so many back doors into your system, there’s no securing it once it’s broken.
Pay attention to security. Even if you don’t think it is your job, it is.
