Immutable Security

Here are the five links that I found interesting for this week:

• The Shadowserver foundation is comprised of a group of volunteer security professionals who gather information about Internet-based crime. One of the more interesting projects is a compilation of how various antivirus vendors fare against 0-day threats. How does your vendor hold up?
• Logs are not much good if you can’t trust them. Maintaining log integrity is vital to a robust incident response process. Here is a great article on how to protect your logs from tampering. It’s not fool-proof, but it can go a long way.
• Information security is a profession that necessitates a solid ethical foundation. Security professionals are often trusted with the most sensitive of data. This presentation, from the Honeynet Project, tackles some of the more thorny situations about performing ethical research.
• Looking for a really awesome way to store and compare your Cisco configs? Rancid, or the Really Awesome New Cisco confIg Differ, may be just the tool for you. It stores Cisco configs in CVS and can let you know something changed. By the way, OSSEC is also capable of something very similar.
• Are you looking to use virtualization in your PCI program? It can be done, but like most technologies, has to be approached carefully. This guide will show you some of the things that need to be considered.

That’s it for today. Have a great weekend!

Here are the five links that I found interesting for this week:

• Mitigating the Apache Range Header Attack. This is a pretty good overview of several ways you can protect yourself for little to no cost. Also, see my post, Detecting the Apache Range Header DoS Attack with OSSEC.
• Automatically encrypt all inbound email part I and part II. Even if you have full-disk encryption, it does not protect you if someone can access your account. This method allows you to keep the private key off the server and does not rely on convincing other people to encrypt email to you. Very impressive.
• Process Monitor is a tool that helps you to see what it really happening under the Windows hood. It’s truly indispensable for Windows troubleshooting and incident response. These filters are specifically designed for malware analysis. I imagine they will be very useful on my next incident.
• Have you ever wanted to open a command prompt as SYSTEM? Most people think that having administrator rights is the same thing, but there can be subtle differences. This short little script allows you to become SYSTEM for those rare situations where you may need to be.
• Would you know if your web site was compromised? Here are eight tips for detecting a web site compromise.

That’s it for today. Have a great weekend!

One of the reasons I started this blog was to share things I had encountered in the security and privacy world. I have done quite a bit of editorializing, but not too many of the quick and useful posts. I thought it might be helpful to post about five of my favorite reads and links on Fridays–unless I get too busy. So let’s start off with a few interesting links:

• PacketFence is a free and open source NAC system. I haven’t used it so I can’t vouch for it either way, but it’s nice to see a NAC in the free software world. NACs are good at preventing things like man-in-the-middle attacks, help you with asset control and help to keep the worm-of-the-day off your network when a contractor plugs in his laptop. Free software can also be a good way to meet a requirement even with limited or no budgets.
• Need a forensics tool? Maltego may fit the bill. It’s also free to use, but not free software in the sense that it doesn’t seem to have an OSI compatible license. Like PacketFence, there are also commercial support and versions available.
• Jamie Riden wrote a very nice piece on his/her response to an SSH attack. There are some nice recovery and lessons-learned aspects to the article. Another possible countermeasure would be the use of OSSEC along with its active response capabilities. This might have been able to prevent the compromise entirely.
• Would you like to have a log of all commands entered on a Cisco router? This is something that can be very useful for audit and compliance, as well as change management needs. This is a great one for PCI environments.
• The ‘nix mtr tool can be useful for troubleshooting network problems. The WinMTR does pretty much the same thing from a Windows host. It’s also free software.

That’s it for today. Have a wonderful weekend!

*(&$#@!!

I stepped outside tonight to water the garden and what did I find? A fuzzy-tailed rabbit happily hanging out inside my garden–with the gate closed. My perimeter has been breached!

How did he get in? I am still doing an analysis, but I believe he squeezed in below the gate. He was a small bunny and this seems like the biggest vulnerability to exploit for a critter his size.

How can I close the hole? I am still pondering this, but I am thinking of something that works kind of like tire strips, which will hopefully dissuade him from crossing the perimeter. I might also post a picture of Chuck Norris for good measure.

I thought I should post this in the interest of full disclosure. It has been a long day.

Sometime when I wasn’t paying attention, a bunch of marketing folds must have gotten together to come up with a new, catchy acronym. I imagine the meeting must have gone something like this:

Joe: We’re not selling enough of our <insert product here>. We need a way to really connect with people.

Linda: The problem is branding. Cross-site Request Forgery doesn’t really roll off the tongue too well.

Bill: Hmm… Advanced.. Problem.. Fixer..

Tom: That’s a good start. How about Advanced Persistent.. Software..?

Joe: Wait for it… Advanced Persistent Threat!

All in unison: Awesome! Let’s go for drinks.

OK, so maybe it didn’t go down exactly that way, but it’s fun to imagine.

So, what exactly is an Advanced Persistent Threat, or APT? According to Wikipedia, APT attacks utilize traditional attack vectors such as malware and social engineering, but also extend to advanced attacks such as satellite imaging. It’s a low-and-slow attack, designed to go undetected.  Finally, there is a specific objective behind it, rather than the incoherent activity of some fifteen-year-old hacking away in a basement for brownie points with his buddies.

It’s not just vendors getting in on the game–companies are increasingly blaming their security failures on APT, as if it was far too sophisticated for them to possibly defend against.

There’s no doubt that such attacks exist. Corporate espionage and nation-state attacks are very real and, in some cases, extremely sophisticated. But these attacks are very rare. The truth is that the vast majority of attacks are not very advanced because they don’t need to be. It is extremely difficult to defend against all known attack vectors. The defenders have to get everything right, all of the time. The attackers only have to find one or a few small holes to work their way in. That’s just the current state of information security.

I think we should generally avoid using the term Advanced Persistent Threat. There are two main reasons I feel this way.

1. It’s highly likely that it’s not an APT at all. Even if you have a great security program with the smartest security people in the world, a company of any appreciable size is going to have hundreds of ways in. You can have everything patched and the front-desk lady will give up her password. You can require two-factor authentication for all remote access until the smart phones come along. You can have great perimeter security until your business partner gets compromised, and you realize that your perimeter really doesn’t begin and end at the firewall. Face it, your company is insecure on its best day.
2. Using terms such as APT, while sexy, encourage us to gloss over the actual facts. Just as intellectual property is a way of lumping together things like copyright and trademark, APT discussions keep us from focusing on which attacks were actually used and how our defenses failed. There is no APT attack. There are SQL injection attacks. There are social engineering attacks. There are buffer overflows in software. There are default passwords left on systems. There are insecure trust relationships. APT is a dangerous umbrella term.

Even those few who do face what people call APT attacks need to break them down into their core elements in order to understand and defend against them. For the rest of us, let’s go back to discussing how our security design failures could lead to a compromise. And if one should occur, let’s speak to the specifics of the attacks so we can learn our lesson, even if a little humility is in order.

Awhile back, I blogged about how not to handle notification of a possible breach. In that case, I began to receive spam to a very unique address only used at one place. When I attempted to report the potential breach, I was at first stonewalled, and then “cautioned” against publicly discussing the issue.

Unfortunately, the stakes have risen from spam to outright malicious e-mails and this time the suspected company is Payscale.com. When I received a malicious PDF to a very unique address only used for that site, I wanted to let them know about a possible breach. So I sent an e-mail to every one of the contacts I could find on their web site. I wanted to make sure someone knew about this.

After over a week with no response from anyone, I received another e-mail, this time a malicious link posing as an Adobe Flash Player update. Still no response.

I think the right thing to do is to publicly discuss the issue. I think that when a company doesn’t respond to concerns such as this, and the public entrusts their data to them, it is ethical and right to publicly discuss the issue so people can make an informed choice about doing business with that company.

I can’t say that Payscale.com has been breached because I don’t know. What I can say is that the source of these malicious e-mails seems to have a strong connection to this company, and that they did not respond to a possible breach notification. Consumer beware.

Update: I am happy to report that, as a result of this post, I was contacted by someone at Payscale.com. My contact does seem genuinely concerned with looking into the issue. Again, this may or may not be anything, but the point of the post seems to have succeeded–getting someone to acknowledge that a security issue may exist.

In the security research community, it is commonly held that the ethical thing to do when discovering a vulnerability is to contact the software developer. Only after a lack of response, after the vulnerability has been fixed, or after the vulnerability has not been fixed within a reasonable time, should the information be disclosed.

But what is the right thing to do when there is an indication that a company may have been compromised? Is it ethical to disclose at all? Is it ethical to not disclose, since others would presumably entrust their data to the company without having this important piece of information? This is the question I have been pondering for some time.

While I think there is still room for healthy debate, I have come to the tentative conclusion that the company should be notified first and given an opportunity to respond, and only if they either do not respond or respond in such a way as to indicate that they are not going to consider the issue, should your concerns be disclosed publicly. I think that companies who may have had a breach have the right to deal with it internally. However, if it affects others and the company does not give an indication that they have looked into the issue, the right thing to do is to publicly discuss the issue so as to warn others about the potential consequences of entrusting their data with that company.

Publicly discussing a potential breach could be just the catalyst a company needs to begin an investigation that helps them to limit their losses. And while this shouldn’t be necessary, the reality of the world we live in is that information protection is rarely practiced well, let alone incident response.

On the other hand, this may only work well (or not) in the consumer area. If there is anything that the Wikileaks debate has taught us, it is that–regardless of whether you think the release of this kind of information is a good thing–there are consequences. One has to carefully consider the potential ethical and legal consequences when choosing to make a public disclosure. What if it initially looked like a breach, but was simply a misunderstanding? What if your words lean towards libel and not opinion? What if disclosing the potential breach results in people losing their jobs?

While there are no right or wrong answers, it is an area that deserves attention and discussion in the security community. As protectors of information, we need to be diligent about the way we handle it, and that involves careful reflection about the ethics and consequences of our actions.

What do you think?

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

When I first heard about this, I thought to myself, “Say it isn’t so. Tell me this is just a big misunderstanding. Tell me that my favorite place to buy cables at great prices wasn’t breached.” Alas, it seems to be true. Monoprice.com had a breach.

I wasn’t too concerned since all of my credit card numbers are unique and automatically generated, and all of the e-mail addresses I use for businesses are also unique, but just the same, I checked my statement. So far, all is well.

As a security guy, I suppose I should forever relegate monoprice.com to the vendor blacklist. After all, they must have been doing something wrong to allow the bad guys to get in, right? That may indeed be the case. They may have had security so bad you could drive a truck though it. Then again, maybe it was very good.

Breaches happen. It’s very, very hard to cover every known threat, let alone the unknown. Security professionals have to anticipate and protect against all threats, while the criminal only has to find one vulnerability. Most of the time it’s not even under our direct control. We are charged with the responsibility of security and get the blame when a breach happens, but encounter fierce resistance when we tell management what really needs to be done to properly secure a site.

Unlike Gexa Energy, who took almost a year to notify affected customer of a breach, monoprice.com took the bold move of prominently and publicly placing a warning on the front page of their web site.  They further went on to stop accepting orders while the breach investigation was continuing. Right now they’re trying to get back on their feet.

Preventing a breach is difficult enough, but the truer measure of an effective security program is how you respond to a breach. Do you issue carefully-crafted letters from the PR department or do you level with your customers? Do you attempt to shun responsibility or do you recognize your mistakes and learn from them?

It seems that monoprice.com recognizes its responsibility to its customers and is doing the right thing. That, combined with my personal risk-mitigation strategies, probably means I will do business with them again.