Immutable Security

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

When I first heard about this, I thought to myself, “Say it isn’t so. Tell me this is just a big misunderstanding. Tell me that my favorite place to buy cables at great prices wasn’t breached.” Alas, it seems to be true. Monoprice.com had a breach.

I wasn’t too concerned since all of my credit card numbers are unique and automatically generated, and all of the e-mail addresses I use for businesses are also unique, but just the same, I checked my statement. So far, all is well.

As a security guy, I suppose I should forever relegate monoprice.com to the vendor blacklist. After all, they must have been doing something wrong to allow the bad guys to get in, right? That may indeed be the case. They may have had security so bad you could drive a truck though it. Then again, maybe it was very good.

Breaches happen. It’s very, very hard to cover every known threat, let alone the unknown. Security professionals have to anticipate and protect against all threats, while the criminal only has to find one vulnerability. Most of the time it’s not even under our direct control. We are charged with the responsibility of security and get the blame when a breach happens, but encounter fierce resistance when we tell management what really needs to be done to properly secure a site.

Unlike Gexa Energy, who took almost a year to notify affected customer of a breach, monoprice.com took the bold move of prominently and publicly placing a warning on the front page of their web site.  They further went on to stop accepting orders while the breach investigation was continuing. Right now they’re trying to get back on their feet.

Preventing a breach is difficult enough, but the truer measure of an effective security program is how you respond to a breach. Do you issue carefully-crafted letters from the PR department or do you level with your customers? Do you attempt to shun responsibility or do you recognize your mistakes and learn from them?

It seems that monoprice.com recognizes its responsibility to its customers and is doing the right thing. That, combined with my personal risk-mitigation strategies, probably means I will do business with them again.

I recently blogged about security practices in a hospital environment that I was witness to. It was interesting to see how security worked (or perhaps didn’t work), rather than post about another standard, tool or best-practice.

Today, I bring you the story of Grandpa. Grandpa had a little information security incident and, well, it didn’t go so well. In fact, it went something like this:

Scene opens with me exiting out of the bedroom after a good night’s sleep…

“Grandpa thinks he has a virus,” my wife dutifully said.

“Coffee,” was the best reply I could muster.

You see, I knew that viruses were really no big deal. I’ve battled many a virus and have always been victorious.  I certainly knew that it could wait until after a cup o’ joe, so I relegated to call him a bit later.

I forgot.

While I was forgetting, Grandpa was in a mini-panic. Malware was telling him he was infected with all kinds of other malware, using a fake scan window of course, and that the only resource was to pay $49.95 for the program.

He bought it. Of course, that didn’t fix his problem.

Your first reaction might be something along the lines of, “what a dummy.” Maybe you’re thinking, “doesn’t he know not to click on links?” Perhaps you think he got what he deserved. While all of those responses may be justified, and maybe even correct given a certain perspective, let’s take another look at this.

Grandpa can’t hear. He is almost completely deaf. Since he grew up hearing, he doesn’t know sign language. He finds family functions to be extremely saddening and frustrating. When the little ones try to tell him something, he doesn’t know what to say because he didn’t hear a word they said.

Being a retiree, he also doesn’t have much income. In fact, there’s really nothing extra. Spending that $50 meant that he had to take $50 from another, probably essential, area. It wasn’t planned; it was more along the lines of, “I need to do this now!”

So, why did he pay this money when I would have helped him for free? Why did he fall victim to the scam? Why did he make an impulsive decision and not think it through?

The answer has nothing to do with how he has or hasn’t been educated about not clicking on untrusted links or installing untrustworthy software. He became a victim because, in that moment, he felt as if he had no other choice. His fight-or-flight response kicked in. He emotionally made a decision because he felt as if he was losing control over something very important. If the computer were no longer his, how would he communicate? What would he do with this time? How would he take his mind off of the multiple health problems that fills his thoughts?

I was able to fix his computer and even helped to get the charge reversed, but the point of this story is that security problems usually aren’t technical in nature. Obviously, trying to teach people to become security experts just to be able to have some fun or do their job effectively isn’t working. The approaches have to change.

We need to understand people better. The age of the security geek is passing. If we are to truly make progress on issues like this, the solutions will be less technical and more people-focused. If Grandpa had felt secure in knowing that I would help him no matter what, perhaps he would have made a different choice. If he had known that his computer would not be relinquished to this scammer unless he paid money, he might have taken awhile to watch a movie until I had called him back.

Security is about people. Let’s make that a mantra for 2010.

On October 29, I gave a presentation at the Rochester Security Summit entitled “OSSEC in the Enterprise.” After a brief delay as a courtesy to the summit organizers, I’m pleased to be able to share the presentation with everyone else.

Daniel Cid over at ossec.net has graciously offered to provide hosting for the presentation. It can be accessed directly here, or for a list that includes other great OSSEC presentations, go here.

It’s a bit long at well over 100 slides, but it goes fast. I hope you enjoy it and would welcome any feedback you might have.

Tools like AIDE and the original Tripwire have long been used in the Unix world for so-called intrusion detection. They let you know when a file has changed by presenting you with an old and new checksum, but usually don’t provide much more info than that.

While knowing that a file changed is useful, how much inherent value is there in simple file hash reports and alerts? In my opinion, very little.

These old tools don’t provide any context about the change. They don’t tell you who changed it, what happened just before and after the change, and what in the file actually changed. At best, you’re left with a bewildered goose-chase trying to figure out what happened.

To make matters worse, a well administered systems is supposed to change. If critical system files aren’t changing, you’re probably not patching. And if you are patching, you’re getting these alerts all the time and will eventually ignore them. The more systems you have, the worse the problem is.

Does that mean there is no value at all in integrity checks? Of course not. The value lies in two things:

1. Getting alerts for rare but important events. Knowing about a file added to the init.d directory or the Windows run key in the registry might be valuable.
2. Having the checksums available on a secured system for forensics.

Security is always a balancing act. Getting flooded with information about low value events is just as bad, if not worse, than not getting any alerts. Alerts should be meaningful and should, at the very least, cause you to stand up and take notice of something that isn’t normal.

Addendum: Complete IDS systems are also sometimes lacking in context. It probably took three or four tries just to post this due to content in the post that the IDS thought was a web attack. :)

I have spent the last several days responding to a 0-day worm outbreak. We didn’t have signatures when the you-know-what hit the fan. Fortunately, some tooling we already had in place allowed us to contain the initial spread while we waited for emergency signatures from our AV vendor.

The last stage of any incident response is the lessons learned phase. This is where you examine the response and analyze all the things that went well and all of the things you could have done better.  There are always ways to improve the response.

I got to thinking that simulating a worm outbreak would be an excellent way to test a response, along with the detection tools you currently have in place. Although it wouldn’t cover all scenarios, and maybe not even most of them, creating an EICAR worm seems like an excellent way to set this up.

For the uninitiated, an EICAR signature is a string identified by AV vendors as a “test virus.” It is meant to show that AV is working.

So why not take this one step further and create a custom worm from the EICAR string? It would be simple: simply create a batch file with the EICAR string and attempt to loop through a list of network shares using a privileged account.

There are risks, of course. First and foremost, there would have to be absolutely no way for this to “infect” a system not under your management. That would simply be inexcusable. Second, there may need to be a way to throttle and control the spread. Some kind of “dead man’s switch.” Finally, there needs to be a way to pull-the-plug should things go really awry.

While this may sound like a crazy idea at first, which is better: to have finely tuned responses based on well-practiced and controlled scenarios, or to struggle through the next worm incident?