Immutable Security

If you run Apache, you may have heard about the DoS vulnerability last week. Apache suffers from a condition where an attacker can remotely cause the web server to consume huge amounts of memory. This causes the system to be unstable and eventually, maybe even crash.

The question was raised: “Can OSSEC detect this attack?” I got to thinking about this and the answer is “probably.” Since OSSEC is primarily a log-based HIDs, we first have to look at the logs to see if there is anything juicy in there that we can use. We also need an exploit and a vulnerable system so we can reproduce the conditions of the attack. Since my server wasn’t vulnerable, FrankS in the #OSSEC channel on IRC offered to lend a hand in the rule research efforts. The first thing we noticed is several logs like this:

172.16.0.1 – - [27/Aug/2011:21:42:53 +0000] “HEAD / HTTP/1.1″ 206 354 “-” “-”

There are two things about this log that don’t quite look right: one, there are multiple HEAD requests to the root of the domain (/) and two, there are several 206 HTTP status codes. Generally, you would see 206 status codes  in the context of requesting compressed content, and the request would probably be a GET. The other thing we noticed in the logs was a page allocation failure coming from Apache, like so:

Aug 27 21:59:43 hostname kernel: [ 1181.719148] apache2: page allocation failure. order:0, mode:0×20

This was promising. If we could simply look for multiple HEAD requests to / with 206 status codes in a very short amount of time, followed by a ‘page allocation failure,’ it’s probably an attack. OSSEC can do that.

There’s a certain amount of art and experience which goes into writing IDS rules. The goal is to make the rule as accurate as possible: if it does not detect the attack (false-negative), you will lose faith in the IDS; on the other hand, if it detects things which aren’t really an attack (false-positive), then you will also lose faith in the IDS and miss potential attacks. Finally, it is best to avoid making the rule exploit-specific. This can result in a situation where a small change in the exploit can avoid the rule being triggered.

Knowing what we do about the logs, how can we make a rule or set of rules that will trip when the host is attacked and be somewhat accurate? Multiple HEAD requests to / are certainly suspicious, but does the attack rely on HEAD to be successful? Section 10.2.7 of the the RFC for HTTP specifically refers to GET requests, and the attack does not seem to be specific to the HTTP method, so we can’t necessarily rely on the HEAD as an indicator. Next, we see the series of 206 codes. That is also not necessarily an indicator of the attack, but it likely is something that the webmaster may want to know about. If there are several of them in a small amount of time, we can alert the analyst to the condition for further inspection. Still, we aren’t really sure if it is the Range Header DoS. In this case, we’ll need two rules. The first rule detects the 206 status code but does not send an alert, while the subordinate rule looks for 10 of them in a 5 second interval coming from the same location (agent) and from the same source IP:

<rule id=”100002″ level=”5″>
<if_sid>31108</if_sid>
<id>^206$</id>
<description>Web Server 206 Error Code</description>
</rule>

<rule id=”100003″ level=”10″ frequency=”8″ timeframe=”5″>
<if_matched_sid>100002</if_matched_sid>
<same_location />
<same_source_ip />
<description>Multiple Web Server 206 Error Codes </description>
<description>from Same Source IP</description>
<group>web_scan,recon,</group>
</rule>

At this point, the analyst will get alerted to the DoS condition, but we are not necessarily confident that it is the Range Header attack. There’s one more rule we can create that looks for the ‘page allocation failure’ occurring within five minutes of rule 100003 firing. If we see all of this, we are reasonably confident in what is going on:

<rule id=”100004″ level=”12″ timeframe=”300″>
<if_matched_sid>100003</if_matched_sid>
<if_sid>1002</if_sid>
<program_name>kernel</program_name>
<match>page allocation failure</match>
<description>Apache Range Header DoS Attack</description>
<group>attack,</group>
<info type=”cve”>2011-3192</info>
</rule>

So, what can go wrong? Lots. The system may be so unstable that it cannot send logs to the manager. The attack might be successful with fewer than ten 206 requests within five seconds. In some cases, the ‘page allocation failure’ does not appear in the logs, although the attack still might trigger rule 10003. At least this would give the analyst a chance to visually inspect the sample of logs OSSEC sends in the alert.

Are there better ways to detect this? Certainly. Mod_Security can not only detect but also prevent the attack. Tools such as Snort are better positioned than OSSEC in situations like this. Of course, OSSEC can monitor the logs of those tools, and still alert you. The point here was to demonstrate that there is an OSSEC-only way to detect attacks like this.

Do you see a problem with the rules? Can they be subverted easily? Let me know in the comments.

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

I was recently troubleshooting a problem with WordPress and wanted to post the solution here for others to search on. If you’re experiencing the following symptoms:

• Clicking “Status: Draft Edit” does nothing
• Clicking “Visibility: Public Edit” moves the page down
• Clicking “Publish immediately Edit” does nothing
• You can’t add tags from the “Add New Post” screen

…the reason may be the Apache LimitRequestFieldSize directive. There are a few other similar symptoms, but they mostly boil down to, “I clicked something and nothing happened.”

Most posts I encountered said something along the lines of, “clear your cache, disable all your plugins and try again.” For some people this works, but it didn’t work for me.

I discovered that the problem was directly related to the value of the LimitRequestFieldSize directive. The default value of 8190 is probably OK for almost all situations, although there is a slight risk leaving the value that high. The CIS Apache Benchmark recommends a value of 100, which in my experience is not usable for most applications. I tried a value of 500, which didn’t make much of a difference. Inching the value up bit-by-bit eventually hit that magic number that made everything start working.

Depending on your environment and the size of the headers your browser is sending, you may need to adjust this value up or down, but try to stay on the conservative side to protect yourself from buffer overflows.

One final note, this is a server-wide directive and cannot be placed in a virtual host container.