Immutable Security

Here are the five or more links that I found interesting for this week:

• This is just all kinds of awesome. It’s not that I am with the bad guys, but when they get this creative you have to give them credit. A bunch of criminals used 3D printers to print out ATM Skimmers. This is just another way that the face of information security is changing.
• Also on the ATM theme, here’s a method to steal ATM PIN numbers using a thermal camera. I am not entirely sure, but given how the cameras are used in house energy audits, my guess is that this can be done from some distance.
• Just for fun, check out this security architecture fail. Can you spot the defect?
• RSA blogged about the recent breach that they experienced. Shortly after they announced the attack, I also blogged about the tendency to call attacks APTs. I fear that describing an attack as an APT is simply another way of failing to take responsibility. It is understandable that they had a breach, but the truth is that attachments exploiting 0-day holes in client software is not particularly advanced today. I have dealt with several 0-day pieces of malware. RSA had layers of security that failed. Again, it’s understandable that they failed–securing everything is hard–but use it an an opportunity to examine the individual layers that led to the breach. There was no magic here. This is  a standard attack method these days.
• Did you know that OSSEC can audit your system? It’s better to know you have vulnerabilities before they are exploited. Daniel Cid explains how to detect outdated web applications with OSSEC. Good stuff.

That’s it for today. Have a great weekend!

Here are the five links that I found interesting for this week:

• The Shadowserver foundation is comprised of a group of volunteer security professionals who gather information about Internet-based crime. One of the more interesting projects is a compilation of how various antivirus vendors fare against 0-day threats. How does your vendor hold up?
• Logs are not much good if you can’t trust them. Maintaining log integrity is vital to a robust incident response process. Here is a great article on how to protect your logs from tampering. It’s not fool-proof, but it can go a long way.
• Information security is a profession that necessitates a solid ethical foundation. Security professionals are often trusted with the most sensitive of data. This presentation, from the Honeynet Project, tackles some of the more thorny situations about performing ethical research.
• Looking for a really awesome way to store and compare your Cisco configs? Rancid, or the Really Awesome New Cisco confIg Differ, may be just the tool for you. It stores Cisco configs in CVS and can let you know something changed. By the way, OSSEC is also capable of something very similar.
• Are you looking to use virtualization in your PCI program? It can be done, but like most technologies, has to be approached carefully. This guide will show you some of the things that need to be considered.

That’s it for today. Have a great weekend!

Here are the five links that I found interesting for this week:

• Mitigating the Apache Range Header Attack. This is a pretty good overview of several ways you can protect yourself for little to no cost. Also, see my post, Detecting the Apache Range Header DoS Attack with OSSEC.
• Automatically encrypt all inbound email part I and part II. Even if you have full-disk encryption, it does not protect you if someone can access your account. This method allows you to keep the private key off the server and does not rely on convincing other people to encrypt email to you. Very impressive.
• Process Monitor is a tool that helps you to see what it really happening under the Windows hood. It’s truly indispensable for Windows troubleshooting and incident response. These filters are specifically designed for malware analysis. I imagine they will be very useful on my next incident.
• Have you ever wanted to open a command prompt as SYSTEM? Most people think that having administrator rights is the same thing, but there can be subtle differences. This short little script allows you to become SYSTEM for those rare situations where you may need to be.
• Would you know if your web site was compromised? Here are eight tips for detecting a web site compromise.

That’s it for today. Have a great weekend!

Virtualization is a quickly growing area in IT right now. The prospect of running dozens of virtual servers on one physical server is most appealing. As long as licensing costs don’t eat up too many of the savings, it really does have the potential to transform an infrastructure into something much more efficient.

Of course, with any new technologies there are risks. Sometimes the risks are completely new and sometimes they are the same risks we have been dealing with for years — just repackaged in a slightly different way.

When most people think of virtualization security, they think of things like breaking out of the guest machine onto the host. This would be a very serious attack if successful because it has the potential to leverage one attack into something that can compromise several hosts. Even malware is getting into the business. The Blue Pill proof-of-concept used a small hypervisor to intercept calls to and from the guest, thereby controlling it according to desires of the attacker.

These are real threats, but for most organizations they do not represent the most serious threats to virtualized hosts. I was reminded of this last week when I read about Jason Cornish, a former IT employee of drug maker Shionogi. Jason had a beef with his former employer and he took it out by logging in and deleting 88 virtualized hosts. That’s the functional equivalent of a tornado taking out a small data center. He was able to do this simply because of a password which was not changed upon his termination, thereby allowing him remote access.

What’s the lesson here? The point is that we cannot forget the fundamentals of information security when implementing new technologies. That means doing the non-sexy stuff like changing default passwords, revoking access immediately for terminated employees and sending logs to a centralized log server. In the virtual world, we might review the access levels of the virtual administrators to make sure they are appropriate, separate hosts of different sensitivities onto different hypervisors and get alerted when a virtual machine is added, deleted or moved.

We do need to keep an eye on evolving virtual security threats, but we also have to keep the other eye on the fundamentals. We need both.

If you run Apache, you may have heard about the DoS vulnerability last week. Apache suffers from a condition where an attacker can remotely cause the web server to consume huge amounts of memory. This causes the system to be unstable and eventually, maybe even crash.

The question was raised: “Can OSSEC detect this attack?” I got to thinking about this and the answer is “probably.” Since OSSEC is primarily a log-based HIDs, we first have to look at the logs to see if there is anything juicy in there that we can use. We also need an exploit and a vulnerable system so we can reproduce the conditions of the attack. Since my server wasn’t vulnerable, FrankS in the #OSSEC channel on IRC offered to lend a hand in the rule research efforts. The first thing we noticed is several logs like this:

172.16.0.1 – - [27/Aug/2011:21:42:53 +0000] “HEAD / HTTP/1.1″ 206 354 “-” “-”

There are two things about this log that don’t quite look right: one, there are multiple HEAD requests to the root of the domain (/) and two, there are several 206 HTTP status codes. Generally, you would see 206 status codes  in the context of requesting compressed content, and the request would probably be a GET. The other thing we noticed in the logs was a page allocation failure coming from Apache, like so:

Aug 27 21:59:43 hostname kernel: [ 1181.719148] apache2: page allocation failure. order:0, mode:0×20

This was promising. If we could simply look for multiple HEAD requests to / with 206 status codes in a very short amount of time, followed by a ‘page allocation failure,’ it’s probably an attack. OSSEC can do that.

There’s a certain amount of art and experience which goes into writing IDS rules. The goal is to make the rule as accurate as possible: if it does not detect the attack (false-negative), you will lose faith in the IDS; on the other hand, if it detects things which aren’t really an attack (false-positive), then you will also lose faith in the IDS and miss potential attacks. Finally, it is best to avoid making the rule exploit-specific. This can result in a situation where a small change in the exploit can avoid the rule being triggered.

Knowing what we do about the logs, how can we make a rule or set of rules that will trip when the host is attacked and be somewhat accurate? Multiple HEAD requests to / are certainly suspicious, but does the attack rely on HEAD to be successful? Section 10.2.7 of the the RFC for HTTP specifically refers to GET requests, and the attack does not seem to be specific to the HTTP method, so we can’t necessarily rely on the HEAD as an indicator. Next, we see the series of 206 codes. That is also not necessarily an indicator of the attack, but it likely is something that the webmaster may want to know about. If there are several of them in a small amount of time, we can alert the analyst to the condition for further inspection. Still, we aren’t really sure if it is the Range Header DoS. In this case, we’ll need two rules. The first rule detects the 206 status code but does not send an alert, while the subordinate rule looks for 10 of them in a 5 second interval coming from the same location (agent) and from the same source IP:

<rule id=”100002″ level=”5″>
<if_sid>31108</if_sid>
<id>^206$</id>
<description>Web Server 206 Error Code</description>
</rule>

<rule id=”100003″ level=”10″ frequency=”8″ timeframe=”5″>
<if_matched_sid>100002</if_matched_sid>
<same_location />
<same_source_ip />
<description>Multiple Web Server 206 Error Codes </description>
<description>from Same Source IP</description>
<group>web_scan,recon,</group>
</rule>

At this point, the analyst will get alerted to the DoS condition, but we are not necessarily confident that it is the Range Header attack. There’s one more rule we can create that looks for the ‘page allocation failure’ occurring within five minutes of rule 100003 firing. If we see all of this, we are reasonably confident in what is going on:

<rule id=”100004″ level=”12″ timeframe=”300″>
<if_matched_sid>100003</if_matched_sid>
<if_sid>1002</if_sid>
<program_name>kernel</program_name>
<match>page allocation failure</match>
<description>Apache Range Header DoS Attack</description>
<group>attack,</group>
<info type=”cve”>2011-3192</info>
</rule>

So, what can go wrong? Lots. The system may be so unstable that it cannot send logs to the manager. The attack might be successful with fewer than ten 206 requests within five seconds. In some cases, the ‘page allocation failure’ does not appear in the logs, although the attack still might trigger rule 10003. At least this would give the analyst a chance to visually inspect the sample of logs OSSEC sends in the alert.

Are there better ways to detect this? Certainly. Mod_Security can not only detect but also prevent the attack. Tools such as Snort are better positioned than OSSEC in situations like this. Of course, OSSEC can monitor the logs of those tools, and still alert you. The point here was to demonstrate that there is an OSSEC-only way to detect attacks like this.

Do you see a problem with the rules? Can they be subverted easily? Let me know in the comments.

One of the reasons I started this blog was to share things I had encountered in the security and privacy world. I have done quite a bit of editorializing, but not too many of the quick and useful posts. I thought it might be helpful to post about five of my favorite reads and links on Fridays–unless I get too busy. So let’s start off with a few interesting links:

• PacketFence is a free and open source NAC system. I haven’t used it so I can’t vouch for it either way, but it’s nice to see a NAC in the free software world. NACs are good at preventing things like man-in-the-middle attacks, help you with asset control and help to keep the worm-of-the-day off your network when a contractor plugs in his laptop. Free software can also be a good way to meet a requirement even with limited or no budgets.
• Need a forensics tool? Maltego may fit the bill. It’s also free to use, but not free software in the sense that it doesn’t seem to have an OSI compatible license. Like PacketFence, there are also commercial support and versions available.
• Jamie Riden wrote a very nice piece on his/her response to an SSH attack. There are some nice recovery and lessons-learned aspects to the article. Another possible countermeasure would be the use of OSSEC along with its active response capabilities. This might have been able to prevent the compromise entirely.
• Would you like to have a log of all commands entered on a Cisco router? This is something that can be very useful for audit and compliance, as well as change management needs. This is a great one for PCI environments.
• The ‘nix mtr tool can be useful for troubleshooting network problems. The WinMTR does pretty much the same thing from a Windows host. It’s also free software.

That’s it for today. Have a wonderful weekend!

Sometime when I wasn’t paying attention, a bunch of marketing folds must have gotten together to come up with a new, catchy acronym. I imagine the meeting must have gone something like this:

Joe: We’re not selling enough of our <insert product here>. We need a way to really connect with people.

Linda: The problem is branding. Cross-site Request Forgery doesn’t really roll off the tongue too well.

Bill: Hmm… Advanced.. Problem.. Fixer..

Tom: That’s a good start. How about Advanced Persistent.. Software..?

Joe: Wait for it… Advanced Persistent Threat!

All in unison: Awesome! Let’s go for drinks.

OK, so maybe it didn’t go down exactly that way, but it’s fun to imagine.

So, what exactly is an Advanced Persistent Threat, or APT? According to Wikipedia, APT attacks utilize traditional attack vectors such as malware and social engineering, but also extend to advanced attacks such as satellite imaging. It’s a low-and-slow attack, designed to go undetected.  Finally, there is a specific objective behind it, rather than the incoherent activity of some fifteen-year-old hacking away in a basement for brownie points with his buddies.

It’s not just vendors getting in on the game–companies are increasingly blaming their security failures on APT, as if it was far too sophisticated for them to possibly defend against.

There’s no doubt that such attacks exist. Corporate espionage and nation-state attacks are very real and, in some cases, extremely sophisticated. But these attacks are very rare. The truth is that the vast majority of attacks are not very advanced because they don’t need to be. It is extremely difficult to defend against all known attack vectors. The defenders have to get everything right, all of the time. The attackers only have to find one or a few small holes to work their way in. That’s just the current state of information security.

I think we should generally avoid using the term Advanced Persistent Threat. There are two main reasons I feel this way.

1. It’s highly likely that it’s not an APT at all. Even if you have a great security program with the smartest security people in the world, a company of any appreciable size is going to have hundreds of ways in. You can have everything patched and the front-desk lady will give up her password. You can require two-factor authentication for all remote access until the smart phones come along. You can have great perimeter security until your business partner gets compromised, and you realize that your perimeter really doesn’t begin and end at the firewall. Face it, your company is insecure on its best day.
2. Using terms such as APT, while sexy, encourage us to gloss over the actual facts. Just as intellectual property is a way of lumping together things like copyright and trademark, APT discussions keep us from focusing on which attacks were actually used and how our defenses failed. There is no APT attack. There are SQL injection attacks. There are social engineering attacks. There are buffer overflows in software. There are default passwords left on systems. There are insecure trust relationships. APT is a dangerous umbrella term.

Even those few who do face what people call APT attacks need to break them down into their core elements in order to understand and defend against them. For the rest of us, let’s go back to discussing how our security design failures could lead to a compromise. And if one should occur, let’s speak to the specifics of the attacks so we can learn our lesson, even if a little humility is in order.

Awhile back, I blogged about how not to handle notification of a possible breach. In that case, I began to receive spam to a very unique address only used at one place. When I attempted to report the potential breach, I was at first stonewalled, and then “cautioned” against publicly discussing the issue.

Unfortunately, the stakes have risen from spam to outright malicious e-mails and this time the suspected company is Payscale.com. When I received a malicious PDF to a very unique address only used for that site, I wanted to let them know about a possible breach. So I sent an e-mail to every one of the contacts I could find on their web site. I wanted to make sure someone knew about this.

After over a week with no response from anyone, I received another e-mail, this time a malicious link posing as an Adobe Flash Player update. Still no response.

I think the right thing to do is to publicly discuss the issue. I think that when a company doesn’t respond to concerns such as this, and the public entrusts their data to them, it is ethical and right to publicly discuss the issue so people can make an informed choice about doing business with that company.

I can’t say that Payscale.com has been breached because I don’t know. What I can say is that the source of these malicious e-mails seems to have a strong connection to this company, and that they did not respond to a possible breach notification. Consumer beware.

Update: I am happy to report that, as a result of this post, I was contacted by someone at Payscale.com. My contact does seem genuinely concerned with looking into the issue. Again, this may or may not be anything, but the point of the post seems to have succeeded–getting someone to acknowledge that a security issue may exist.

Yesterday, I blogged about how we could beat OSSEC up, or, to put it more accurately, the people and protocols behind it. Today, we’re going to discuss how we can fight back against the bullies. For this post to make sense, you’ll need to read the first one. Go ahead, I’ll wait…

Ok. Put your gloves on. Here are some countermeasures and considerations for all of the risks we discussed previously.

1. Alert manipulation. Recall that alert manipulation is the process by which alerts are used against us.
   1. Alert tickling. The idea with alert tickling is to send just enough normal-looking alerts to create a window for the attacker to go undetected. There are a couple of ways to defend against this. One is active response. By default, anything that is level 6 or higher can trigger an active response such as blocking the IP for 15 minutes. The cool thing about active response is that it will continue to work even if your alerts are delayed until the next hour. This means the attacker might still be blocked regardless of when you get the e-mail. Another countermeasure is to increase the default number of alerts per hour to something like 9999. Of course, this opens you up to the next risk of alert flooding. Perhaps the best countermeasure is to configure the do_not_delay option in ossec.conf. Adding something like this would save all e-mails over 12 until the next hour, except for those that appear to be an attack:

   <email_alerts>
   <email_to>secanalyst@example.com</email_to>
   <group>attacks</group>
   <do_not_delay />
   </email_alerts>

   2. Alert flooding. With alert flooding, the attacker would like to swamp you with so many alerts that you just can’t bear to read them all. Besides tuning, the best option to protect against alert flooding is to keep the email_maxperhour setting at a reasonable level. If you choose to go that route, be sure to note the risks that can introduce and plan accordingly.
2. Killing the OSSEC processes. I am a firm believer in the mantra that if someone has administrative access to your system, they can do whatever they want. Ultimately, there is nothing you can do to completely protect against this risk. On the other hand, not all administrative users are savvy enough to be able to subvert the ossec processes. A well-written SeLinux policy just might be enough to hold them back. Also, recall that I talked about the non-malicious attacker. Non-technical safeguards such as background checks, job rotation and common workstation areas might be enough of a deterrent for those situations.
3. Log injection. With log injection, the attacker wants to control the log output. There are two main types of log injection: remote and unauthenticated. They can also be combined for greater efficiency.
   1. Remote log injection. The basic defense against remote log injection is to first assume that all input is malicious. Nothing from the user can be trusted, even if your OSSEC installation is on a private network. The second line of defense is to write your OSSEC decoders in such a way as to restrict where in the message input can be coming from. Referring again to Daniel’s paper, Attacking Log Analysis Tools, we can see that adding a ^ (to denote the beginning of the line) or a $ (to denote the end of a line) is often enough.
   2. Unauthenticated log sources. The obvious countermeasure to unauthenticated log sources is to not use them. However, while that is the best defense, it is rarely practical in a modern network. Sometimes you are simply faced with the choice of getting logs through syslog, or not at all. Given the choice, I would choose to get the logs. One approach is to design an out-of-band network where your unauthenticated network devices can send their logs. That approach is rather expensive and not very pragmatic for most people. The more practical approach is to enable IP spoofing protection on your network devices, such as in this example. Be aware, though, that it may not protect against attacks within the same broadcast domain.
4. Traffic manipulation. This attack can be difficult to prevent. Not only does it involve the direct manipulation of traffic, as is the case with something like ARP poisoning, but also intentional or unintentional flooding. Common countermeasures include NAC protection, blocking DoS attacks upstream and not using unauthenticated log sources.
5. Active response manipulation. Active response can prevent an attack from becoming a breach. It can also enable an attacker to block legitimate traffic. OSSEC accounts for this by giving you the white_list option for use in ossec.conf. Any IP or range added to this option won’t be blocked by active response. Keep in mind that a trusted host may be compromised first, with the attacker looking to elevate the access. The safest option is to not whitelist anything. OSSEC also includes a timeout for responses, so if you are accidentally blocked, you can try again in about 15 minutes.

By now you may be starting to realize that the default configuration of OSSEC is very sane and has taken most of these risks into account. At the same time, it gives you enough flexibility to meet your needs and to make your own risk decision. The important thing is to consider the possible consequences of changing a default setting and be comfortable with the additional risks it may present.

No discussion about the effectiveness of a security monitoring tool would be complete without exploring ways to defeat that tool. While this may seem self-defeating, it is my belief that an honest perspective about strengths and weaknesses of the tools we use leads to better risk decisions and ultimately, better tools.

OSSEC can be abused in a number of ways. Many of those are not the fault of OSSEC, but rather they speak to weaknesses in basic protocols or the analyst using the tool. Here are some of the ways an attacker may try to abuse OSSEC:

1. Alert manipulation. Alerts can be manipulated in a few different ways. The basic idea is to attack the alerting mechanism in such a way as to throw the defender off the trail. Here are a few of those ways:
   1. Alert tickling. By default, OSSEC will send no more than 12 alerts per hour. This is to prevent an attacker from flooding you with a sea of alerts. This leaves the potential for the bad guy to intentionally engage in something, maybe from a different IP, that looks like noise. You’ll chalk it up as nothing to worry about and the attacker will have a window to proceed while being undetected.
   2. Alert flooding. In this attack, the point is to flood you with as many alerts as possible. The attacker may know that you have raised the default number of alerts per hour, and will count on you not reading them all. A savvy attacker will begin the attack with simple stuff that looks like noise, hoping you will read the first few alerts and delete the remaining 1,000.
2. Killing the OSSEC processes. If your attacker is an administrator, then it is easy to simply kill the processes. Anything logged while the OSSEC processes aren’t running will not be sent once OSSEC is restarted. It is valid to say that if one has administrative access, then all bets are off. Understand, however, that not all attackers are outright malicious or untrusted. Your attacker could simply be an administrator who doesn’t want to follow the change control process, and stops OSSEC before making the unauthorized change.
3. Log injection. When the attacker can manipulate the log file to their liking, we call it log injection.
   1. Remote log injection. While SQL injection is well-understood, it’s log-based sister, remote log injection, is not widely discussed. The basic premise is the same. Anything coming from a user should not be trusted, as it could be malicious. As Daniel demonstrates in his paper, Attacking Log Analysis Tools, when the regexes responsible for parsing the message are not well-written, it gives the attacker the possibility to manipulate the log monitoring program in his favor.
   2. Unauthenticated log sources. By default, OSSEC sends all logs encrypted with blowfish. In addition, a counter is added to each log to prevent log replaying (an attack in itself). However, OSSEC also supports syslog, because, well, let’s face it, that’s sometimes the only option. As you may know, syslog messages can be spoofed. Since the protocol is UDP, the attacker does not have to worry about completing the connection. So, if an attacker can spoof the source IP, he can cause you all sorts of grief. For a great example of how to spoof syslog, check out the paper, The Ins and Outs of System Logging Using Syslog.
4. Traffic manipulation. At attacker who can interfere with the traffic between the OSSEC agent and the manager might be able to keep the message from being received. If the message isn’t received, the manager can’t correlate and alert on the log.
5. Active response manipulation. Active response can be a double-edged sword. On one hand, it can prevent an attack from becoming a breach. On the other hand, when combined with unauthenticated log sources, the attacker might be able to spoof traffic and block legitimate hosts from accessing your host.

Understanding the risks leads to the inevitable question, “how do I protect myself?” In part II, we’ll discuss some countermeasures for these attacks, as well as the trade-offs for these countermeasures.