Immutable Security

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

Letting people go is rarely a feel-good experience. Even if everything goes smoothly, there are always human emotions involved, and those emotions make some people unpredictable.  It’s important, therefore, to make sure extra attention and detail is paid to certain areas before and during a layoff. Here are five things to watch out for in your logs and monitoring tools.

1. Outbound connections to a single IP. If you have the benefit of knowing what normal network traffic patterns look like, watch for variations and patterns that may indicate a back-door connection being initiated from within the network. Does outbound traffic on one port spike at around 5:00 PM and stay that way for an extended period of time? Does it look encrypted? Is it using a common port like 53, normally reserved for DNS, but does not seem to be DNS traffic? These could all be indications of a back-door or unauthorized remote access channel.
2. New and changed scheduled tasks/cron jobs. Pay special attention to scheduled tasks and cron jobs and the scripts they run. Monitor changes to those scripts. Look for new and changed tasks and cron jobs. For Windows tasks, refer to IDs 602, 4698, 4699, 4700, 4701 and 4702. For ‘nix, look for changes in the cron log. Scheduled tasks and cron jobs are often used to plant logic bombs, which can execute even months after an employee has left.
3. E-mail to competitors and media. E-mail going to domains which are not considered “friendly” to the organization may indicate that someone has a hatchet to bury. In particular, if there are several of these organizations listed as recipients in the same e-mail, that could be someone trying to sling mud or disseminate confidential information.
4. Data leakage. Let’s get one thing straight. If someone wants to steal data, chances are they will be able to do so undetected if they are smart enough. Even so, people don’t always take the time to cover their tracks. Some things to check: proxy and mail gateway logs for interesting content types (i.e. attachments), USB drives being plugged in, and the good ol’ physical equipment walking out the door.
5. New and Changed Accounts. Some people aren’t too good at letting go. They may try to maintain access into the environment. This may involve either a password change or a new account. People know their account may be disabled, so they may try to create a new, innocuous looking account to maintain access. Perhaps they will try to use a service account with a well-known password. Whatever the case, pay special attention to accounts.

Perhaps the best thing to do is to level with your staff as much as is possible. Treating people with respect and dignity could go much further than any technical measure you can dream up.

When I first heard about this, I thought to myself, “Say it isn’t so. Tell me this is just a big misunderstanding. Tell me that my favorite place to buy cables at great prices wasn’t breached.” Alas, it seems to be true. Monoprice.com had a breach.

I wasn’t too concerned since all of my credit card numbers are unique and automatically generated, and all of the e-mail addresses I use for businesses are also unique, but just the same, I checked my statement. So far, all is well.

As a security guy, I suppose I should forever relegate monoprice.com to the vendor blacklist. After all, they must have been doing something wrong to allow the bad guys to get in, right? That may indeed be the case. They may have had security so bad you could drive a truck though it. Then again, maybe it was very good.

Breaches happen. It’s very, very hard to cover every known threat, let alone the unknown. Security professionals have to anticipate and protect against all threats, while the criminal only has to find one vulnerability. Most of the time it’s not even under our direct control. We are charged with the responsibility of security and get the blame when a breach happens, but encounter fierce resistance when we tell management what really needs to be done to properly secure a site.

Unlike Gexa Energy, who took almost a year to notify affected customer of a breach, monoprice.com took the bold move of prominently and publicly placing a warning on the front page of their web site.  They further went on to stop accepting orders while the breach investigation was continuing. Right now they’re trying to get back on their feet.

Preventing a breach is difficult enough, but the truer measure of an effective security program is how you respond to a breach. Do you issue carefully-crafted letters from the PR department or do you level with your customers? Do you attempt to shun responsibility or do you recognize your mistakes and learn from them?

It seems that monoprice.com recognizes its responsibility to its customers and is doing the right thing. That, combined with my personal risk-mitigation strategies, probably means I will do business with them again.

I recently blogged about security practices in a hospital environment that I was witness to. It was interesting to see how security worked (or perhaps didn’t work), rather than post about another standard, tool or best-practice.

Today, I bring you the story of Grandpa. Grandpa had a little information security incident and, well, it didn’t go so well. In fact, it went something like this:

Scene opens with me exiting out of the bedroom after a good night’s sleep…

“Grandpa thinks he has a virus,” my wife dutifully said.

“Coffee,” was the best reply I could muster.

You see, I knew that viruses were really no big deal. I’ve battled many a virus and have always been victorious.  I certainly knew that it could wait until after a cup o’ joe, so I relegated to call him a bit later.

I forgot.

While I was forgetting, Grandpa was in a mini-panic. Malware was telling him he was infected with all kinds of other malware, using a fake scan window of course, and that the only resource was to pay $49.95 for the program.

He bought it. Of course, that didn’t fix his problem.

Your first reaction might be something along the lines of, “what a dummy.” Maybe you’re thinking, “doesn’t he know not to click on links?” Perhaps you think he got what he deserved. While all of those responses may be justified, and maybe even correct given a certain perspective, let’s take another look at this.

Grandpa can’t hear. He is almost completely deaf. Since he grew up hearing, he doesn’t know sign language. He finds family functions to be extremely saddening and frustrating. When the little ones try to tell him something, he doesn’t know what to say because he didn’t hear a word they said.

Being a retiree, he also doesn’t have much income. In fact, there’s really nothing extra. Spending that $50 meant that he had to take $50 from another, probably essential, area. It wasn’t planned; it was more along the lines of, “I need to do this now!”

So, why did he pay this money when I would have helped him for free? Why did he fall victim to the scam? Why did he make an impulsive decision and not think it through?

The answer has nothing to do with how he has or hasn’t been educated about not clicking on untrusted links or installing untrustworthy software. He became a victim because, in that moment, he felt as if he had no other choice. His fight-or-flight response kicked in. He emotionally made a decision because he felt as if he was losing control over something very important. If the computer were no longer his, how would he communicate? What would he do with this time? How would he take his mind off of the multiple health problems that fills his thoughts?

I was able to fix his computer and even helped to get the charge reversed, but the point of this story is that security problems usually aren’t technical in nature. Obviously, trying to teach people to become security experts just to be able to have some fun or do their job effectively isn’t working. The approaches have to change.

We need to understand people better. The age of the security geek is passing. If we are to truly make progress on issues like this, the solutions will be less technical and more people-focused. If Grandpa had felt secure in knowing that I would help him no matter what, perhaps he would have made a different choice. If he had known that his computer would not be relinquished to this scammer unless he paid money, he might have taken awhile to watch a movie until I had called him back.

Security is about people. Let’s make that a mantra for 2010.

I recently started getting spammed by Incredimail.  I’m not going to give them the courtesy of a link here, but Incredimail is one of those useless (to me) applications that is supposed to make e-mail “fun” by junking it up with a bunch of graphics. I prefer my e-mail plain-text, thanks.

The interesting part is that they are spamming an e-mail address I have only used to provide my parents help with their computer. When I setup Remote Assistance for them, I used an address very distinct and which I have never used anywhere else.

It seems that they harvested the “to” address and decided I needed some of their useless junk, despite not so much as an unsolicited “invitation” (which would still be spam) to receive their newsletters.

Spam is spam, whether it comes from some guy in Elbonia hawking Viagara or from some supposed legitimate company. Don’t do business with spammers. Don’t use Incredimail.

Awhile back, I wrote an article for the “Security Catalyst” blog about the economics of data breaches. In the article, I wondered if companies should compensate customers for poor handling of their information, even before a breach. Since companies often roll the dice with poor security, they have little financial incentive to change their behavior unless a breach happens and it hits the bottom line.

Now, Hannaford, who helped to enable the exploitation of millions of credit card numbers by their poor handling of the information, may be forced to compensate the victims.  At issue is whether one’s time and effort to prevent further damage to their identity, is worth compensation.

Previously, a judge ruled that since the banks offered the customer zero-liability protection, the customer wasn’t owed anything by Hannaford. That gave Hannaford some level of protection against costly lawsuits.

However, as the courts are now considering, one’s time may have value. If one is spending time trying to clean up Hannaford’s mess and prevent further harm, then maybe they should be compensated.

I agree that we need strong financial penalties to send the right message to the custodians of our information. Money seems to be the only thing most large corporations understand, so we need to make it more financially preferable to protect the information properly in the first place.

My concern is that we won’t really see improved security because of this. I envision the companies merely passing the breach cost off to consumers as they would an increase in corporate taxes. I’m not confident that they would “get it” and really start to take the responsibility of data custodianship seriously. That takes a conscience and that’s something many large companies lack.

To be fair, I’m also concerned that data breaches in companies who practice security very well would be seen as equivalent to those who shirked their responsibility. I work in the security profession and I’ll be the first to admit that even those who seem to be doing everything right have breaches. All it takes is one missing patch.

In the end, perhaps the solution to work toward is something of a hybrid between financial penalties for poor handling of information–even before a breach–and criminal penalties for the executives who enabled a breach to happen under their leadership. Just as one who is present at a murder scene can be charged as an accessory to a crime, perhaps an executive is also liable for enabling a breach to happen.

Finally, let’s not forget that the criminal who exploited the vulnerability is most at fault. Although the data custodians deserve their fair share of the blame, it takes someone at the other end of the keyboard doing something they shouldn’t be doing for a compromise to happen.

Peter Handsteen, a.k.a “That Grumpy BSD Guy” recently observed a round of low and slow SSH password guessing attacks against his server. For those not familiar with the term “low and slow” this is an attack meant to be slow enough that IDS systems aren’t tripped but fast enough where the attacker has a decent chance of getting access. The interesting thing about this attack is that it is coming from several different sources and is likely a botnet design just for this purpose. There is about one attempt per minute against the root account. While this doesn’t sound like much, it’s enough for an attacker to cycle through 1440 common passwords in a single day. And since each few attempts come from a different source, it would be difficult to block at the network level. Admins who use passwords like root, test, admin and password are likely to fall victim.

OSSEC detects the invalid logins, but a cursory look at the length of time between each failed attempt reveals that it is not likely to trip the “SSHD brute force” rule (ID 5712). This means that while OSSEC will detect this, the threshold is low enough where you may not get an e-mail alert. This makes sense, since most people would not want an alert for just a couple of failed logins every minute or two.

So what can we do to defend against these types of attacks?

• Well, for starters, the basics of choosing good passwords apply. There is no vulnerability being exploited here other than poor password practices. This is in effect a people problem. Choose stong passwords and audit them regularly.
• It’s also a poor practice to login as root directly, especially thorugh SSH. Adding ‘PermitRootLogin no’ to sshd_config will prevent exploitation through SSH even with a poorly chosen password.
• Invalidate the password for root by locking the account, or just placing something like ‘*’ in the password field of the shadow file. Then give your non-privileged account access through sudoers. To attain root, simply type ’sudo su -’ and the password of your non-privileged account.
• “Tune down,” then “tune up.” Tune your IDS to the point at which the alerts you’re getting are meaningful and need to be seen in near real-time. After you have done that (which could take several months in a large environment), use the GUI interface of the tool to see what it thinks isn’t important enough to alert you on. The tool’s idea of importance may be different than your own.

Exploitation from attacks like this are entirely preventable. Using layers of security mean that even if one defense fails, another steps in to hopefully take its place. The key is to implement the layers before the system is placed into production, then manage it well throughout its life.

The Age reports that computer criminals from an underground hacker forum broke into Australian Federal Police computer systems after the police infiltrated their group. And, according to The Age, it was all because the cops forgot to set a MySQL database password.

We may need to pass around the clue stick to everyone here. If the compromise was real, the police should have known better than to leave a database exposed to the Internet and unprotected with a password. The alleged criminals need to understand that drawing more attention to yourself after you already know your under investigation is not the brightest thing to do.

Surely, the system could have been secured. It’s doubtful that it needed to be on the Internet in the first place. Or maybe, as some speculate in the article, it really was a honeypot designed to lure the not-so-bright (alleged) criminal into a trap. Maybe the police are a bit brighter than the bad guys give them credit for.

Let’s assume for the moment that this was an honest security blunder. It’s certainly the type of thing that happens every day. What’s the security lesson here and how could this have been prevented? In this particular case, two things come to mind:

1. MySQL could listen on localhost or use a socket, by default.
2. MySQL could require a decent password or heck, a password at all, to run. No password and the process aborts. For those that really want to live dangerously, they could pass a –stupid flag to run without a password.

Many security problems are preventable. Whether or not this was a honeypot, this can be used as a lesson for developers. Run secure by default and make the user choose to be insecure.