Immutable Security

No discussion about the effectiveness of a security monitoring tool would be complete without exploring ways to defeat that tool. While this may seem self-defeating, it is my belief that an honest perspective about strengths and weaknesses of the tools we use leads to better risk decisions and ultimately, better tools.

OSSEC can be abused in a number of ways. Many of those are not the fault of OSSEC, but rather they speak to weaknesses in basic protocols or the analyst using the tool. Here are some of the ways an attacker may try to abuse OSSEC:

1. Alert manipulation. Alerts can be manipulated in a few different ways. The basic idea is to attack the alerting mechanism in such a way as to throw the defender off the trail. Here are a few of those ways:
   1. Alert tickling. By default, OSSEC will send no more than 12 alerts per hour. This is to prevent an attacker from flooding you with a sea of alerts. This leaves the potential for the bad guy to intentionally engage in something, maybe from a different IP, that looks like noise. You’ll chalk it up as nothing to worry about and the attacker will have a window to proceed while being undetected.
   2. Alert flooding. In this attack, the point is to flood you with as many alerts as possible. The attacker may know that you have raised the default number of alerts per hour, and will count on you not reading them all. A savvy attacker will begin the attack with simple stuff that looks like noise, hoping you will read the first few alerts and delete the remaining 1,000.
2. Killing the OSSEC processes. If your attacker is an administrator, then it is easy to simply kill the processes. Anything logged while the OSSEC processes aren’t running will not be sent once OSSEC is restarted. It is valid to say that if one has administrative access, then all bets are off. Understand, however, that not all attackers are outright malicious or untrusted. Your attacker could simply be an administrator who doesn’t want to follow the change control process, and stops OSSEC before making the unauthorized change.
3. Log injection. When the attacker can manipulate the log file to their liking, we call it log injection.
   1. Remote log injection. While SQL injection is well-understood, it’s log-based sister, remote log injection, is not widely discussed. The basic premise is the same. Anything coming from a user should not be trusted, as it could be malicious. As Daniel demonstrates in his paper, Attacking Log Analysis Tools, when the regexes responsible for parsing the message are not well-written, it gives the attacker the possibility to manipulate the log monitoring program in his favor.
   2. Unauthenticated log sources. By default, OSSEC sends all logs encrypted with blowfish. In addition, a counter is added to each log to prevent log replaying (an attack in itself). However, OSSEC also supports syslog, because, well, let’s face it, that’s sometimes the only option. As you may know, syslog messages can be spoofed. Since the protocol is UDP, the attacker does not have to worry about completing the connection. So, if an attacker can spoof the source IP, he can cause you all sorts of grief. For a great example of how to spoof syslog, check out the paper, The Ins and Outs of System Logging Using Syslog.
4. Traffic manipulation. At attacker who can interfere with the traffic between the OSSEC agent and the manager might be able to keep the message from being received. If the message isn’t received, the manager can’t correlate and alert on the log.
5. Active response manipulation. Active response can be a double-edged sword. On one hand, it can prevent an attack from becoming a breach. On the other hand, when combined with unauthenticated log sources, the attacker might be able to spoof traffic and block legitimate hosts from accessing your host.

Understanding the risks leads to the inevitable question, “how do I protect myself?” In part II, we’ll discuss some countermeasures for these attacks, as well as the trade-offs for these countermeasures.

Since version 1.2, OSSEC has generated daily, chained MD5/SHA1 checksums of the alerts log, and if logall is enabled, the archives.log. As Daniel notes in his blog post, this can provide you with a mechanism to prove that a particular log was not tampered with. If you compare a hash that was e-mailed to you with one on the manager, you can reasonably say the log has not been tampered with.

Or can you? What if the e-mail server was on the same network as the manager? What if they were running the same OS version and had the same vulnerability exposure? Can you still say the hash has not been tampered with?

One way to have a high degree of confidence in the integrity of the hash is to get it out to as many people as possible. The more remote, unrelated computers that have a copy of the hash, the higher degree of confidence and non-repudiation in that hash. Since the hash is a one-way function and does not contain the actual log data, the confidentiality of your logs remains intact.

Here are just a few ways this can be done.

• Twitter. Tweet your hashes, maybe even to an account set up especially for this. Make sure they are public. Here’s an example of what I mean.
• Facebook. Set up a facebook page for your hashes and make it public.
• Your web site. Set up a cron job to update a page daily that has all of your hashes. Make sure it gets indexed.

Again, the more unrelated sites that have these hashes, the higher degree of assurance you will have. Although I am not a lawyer, I can imagine a situation where one would make a very compelling argument that the logs were intact, because in order for the other side to repudiate that argument, they would have to prove that all of the external sources were also compromised and the hash modified.

What other ideas do you have for log integrity and non-repudiation? Speak up in the comments!

When many companies think about log management, they immediately jump to SIEMs, or Security Information and Event Managers. You’ll also find many in the infosec community who jump to the conclusion that a SIEM is the right solution for the problem, when they haven’t really even taken the time to set requirements, expectations and understand what it is they are trying to achieve.

I was reminded of this today when a client needed a refresh and storage update of their pre-existing SIEM solution. The quote was $77,000 (USD). They are collecting logs from about 300 servers and running daily reports of successful and failed logins. That’s it.

I couldn’t help to think, what is so special about this solution that they need to spend $77,000 for another log collector appliance (which is really just a Dell server with Windows installed on it) and about 2TB of storage? If they are paying this much for a simple upgrade, what was their initial investment? If it’s like most companies who buy SIEMs, they didn’t get out of the room for under six figures.

To be clear, a good SIEM does more than simply collect logs and allow one to run reports. It can correlate log and vulnerability information, elevate alerts based on asset value, identify suspicious patterns and integrate with ticketing and incident response systems.

The question is: who is sufficiently advanced in their security program to make use of such capabilities? Furthermore, what regulations, contracts or risk programs require this level of sophistication? The answer, based on my experience, is very, very few.

Does that mean having all of this wouldn’t be a good thing? Of course not. In a perfect world, security ninjas would be able to make these things sing and dance in all kinds of cool ways. But most of us don’t live in that world. We live in a world where management gives us conflicting priorities, doesn’t adequately staff the security team and turns down our requests to limit user access.

So I am here to tell you that you probably don’t need a SIEM. And here are the top five reasons why I think so:

1. They are ridiculously expensive. Sure, the cost of security should be commensurate with the risk. But when you’re talking about laying down six or seven figures for a tool, you better be damn sure it is going to pick up your toys and call you sweetheart.
2. The requirements often don’t necessitate it. But I have SoX, HIPAA and PCI requirements! Bah! I have been in all of those environments and, in most cases, a SIEM was nowhere to be found. Most of them had simple syslog servers with daily reports sent out by scripts. They passed the audits just fine.
3. It may never be fully deployed. SIEMs are complex beasts. They are also often buggy, frail and slow. To properly deploy, integrate and test a SIEM can take a team months or even years. The most often line I hear with SIEM administrators is, “We haven’t gotten to that part yet.”
4. They often provide little value. If the cost were high and the value was high, maybe the price wouldn’t be so much of an issue. But the bang for the buck you get is often very low beyond what you could do with a solution that costs one-tenth of the price, or less. I have seen many SIEMs that don’t even alert you to multiple failed logins to a single host. You have to configure these yourself. That means each analyst at each company becomes a research analyst, and has to do their own threat modeling. You should get the experience and expertise of the SIEM vendor from day one.
5. You’ll experience vendor lock-in. Now that you have all of this data in the SIEM, how do you get it out should you want to change course? or will you be forced to pay the piper when you need an upgrade?

Now, obviously, I have come down pretty hard on SIEM vendors here. I expect someone will come along and want to refute my assertions. I most definitely welcome that.

There’s a missing market in the log management world. It lies somewhere between solutions like OSSEC and syslog-ng, and Arcsight. It’s a market which would offer intelligent log collection, analysis, correlation and alerting, combined with valuable reports made available through a nice front-end. And there’s no reason a product like this should cost more than five-thousand dollars, at most.

Note to prospective SIEM employers if I am ever looking for work–just kidding! :)

Most people that use the OSSEC Windows agent download a pre-compiled copy from the OSSEC site. While that is a good option for many individual users, it may not suit those with more specific needs and/or those in enterprise environments. Users who fall into those categories could benefit from customizing the agent and maintaining internal builds in order to suit their individual needs.

There are already instructions on how to compile the Windows agent on Linux, but ironically the process doesn’t work so well on Windows. I had a need to make this work on Windows, so I thought I would share the process with you.

First, there are some prerequisites.  You’ll need:

• The Nullsoft Scriptable Install System (NSIS)
• The Minimalist GNU for Windows compiler (MinGW)
• My gen_win batch file.  Simply rename from gen_win.txt to gen_win.cmd.
• 7-Zip for Windows
• The public domain Unix2DOS utility
• The latest OSSEC for Unix/Linux (this contains the Windows source code)

Here are the steps:

1. Download and install the required programs. Be sure to pay special attention to the steps for properly installing and configuring MinGW, particularly the part about modifying the PATH environment variable.
2. Next, we’re going to extract OSSEC using 7-Zip. To do so, simply right-click on the file and select 7-Zip, extract to “folder name.tar,” where folder name is the name of the package. This decompresses the archive. Navigate within that folder and repeat this step to untar the archive. At this point, you should see all of the files in the package.
3. Place gen_win.txt in the src\win32 folder and rename the extension to .cmd.
4. Download Unix2DOS and place it in the src\win32 folder
5. Open a command prompt. Navigate to src\win32, make any desired customizations, and execute gen_win.cmd. This should gather all of the required files and place them in src\win-pkg.
6. Next, we compile the Windows agent by navigating to src\win-pkg and executing make.bat (I assume you have the chops to know how to change directories :) ).
7. Now we have all of the files we need but no way to effectively install it. To generate the installer, simply execute the NSIS compiler like so: “c:\Program Files\NSIS\makensis.exe” ossec-installer.nsi

If you see no errors and a binary named ossec-win32-agent.exe, everything was successful. Congratulations, you now have a custom-made version of OSSEC!

Logrotate is a fantastic little utility for, well, rotating logs. It has several options available. It can rotate by date, file size, set specific owners on rotated files, and it even gives you the capability to run pre and post-rotate scripts.

Getting it to work on a centralized log server, however, can be a bit tricky. The issue lies in the unique way these log servers are often configured. Rather than a static set of directories, log servers often dynamically create directories based on the hosts that are logging to them. It’s not uncommon to see a directory structure something along the lines of /mnt/logs/2010/May/18/host/messages. While logrotate can use wildcards in the directory structure definition, it gets a bit hairy when trying to define the rotated log directory.

There are two options for where to put the rotated logs. You can put them in another directory, or in the same directory as the rotated files. Another directory is usually the better option, as rotated logs in the same directory could be rotated again and again, unless you are careful with the way logs are created.

So what happens if the directory you specify for the rotated log files doesn’t exist (remember, the log server is creating them dynamically based on the hosts that report in)? In that case, logrotate will complain and refuse to rotate your logs.

By now you might be thinking, “Why, I’ll just use that handy pre-rotate option. Then I can have it run a script prior to the rotation.” Sorry, no dice. Logrotate will simply check the config prior to even starting and refuse to do anything useful.

The solution is to use a script, completely outside the scope of logrotate, to create these rotated logs directories just before logrotate runs. Let’s say all remote hosts log to /mnt/logs/yyyy/mmm/dd/<hostname>/<log-name>. Your logrotate.conf file might look like this:

/mnt/logs/*/*/*/* {
weekly
dateext
compress
maxage 365
olddir rotated

}

This would rotate all logs within that directory structure to the relative directory called “rotated.” But of course we need the rotated directory to exist. So let’s create a small script to do that:

#!/bin/bash

# Create “rotated” directory so logrotate has somewhere to put the rotated files.
# Logrotate will not do this by itself, even when putting the script in the “prerotate” section

for i in /mnt/logs/*/*/*; do
if ! [ -d $i/rotated ]; then
/bin/mkdir $i/rotated
/bin/chown root.log_analyst $i/rotated
/bin/chmod 550 $i/rotated
fi
done

In the above script, we’re checking to see if the directory exists, and if it doesn’t, we create it. Then we set proper permissions, owner and group in order to ensure integrity of the rotated log directory. I am assuming a group of log_analyst, so change as appropriate.

The final step it to ensure the script runs just before the logrotate happens. In RHEL, logrotate is called from within the cron.daily directory. So we’ll just name the script make_rotated.sh and link it in this directory in such a way that it runs before logrotate:

[root@hostname log]# ll /etc/cron.daily/ | grep logrotate
total 64
lrwxrwxrwx 1 root root 31 Apr 8 15:19 01_pre-logrotate -> /usr/local/sbin/make_rotated.sh
-rwx—— 1 root root 180 Feb 26 2009 logrotate

There are certainly other ways to solve this problem, but this seems to work well. Until next time…

Awhile back, I posted about how certain versions of Windows always have the capability to lose logs. I encourage you to read the full post to understand the issues involved, then come back here and continue reading.

The basic problem is that Windows loads the full log into a shared memory space, and if it’s too big, then logs will be silently dropped. That’s why it is very important to have a centralized logging solution with logs sent in real-time or pretty close to it.

So how does centralized logging affect the local retention settings? To know that, you have to make certain assumptions about log size, average number of events generated per day, and so-on. Let’s start with this quote from Microsoft’s Threats and Countermeasures Guide:

Although there is no simple equation to determine the best log size for a particular server, you can calculate a reasonable size. The average event takes up about 500 bytes within each log, and the log file sizes must be a multiple of 64 KB. If you can estimate the average number of events that are generated each day for each type of log in your organization, you can determine a good size for each type of log file.

For example, if your file server generates 5,000 events per day in its Security log and you want to ensure that you have at least 4 weeks of data available at all times, then you would want to configure the size of that log to about 70 MB. (500 bytes * 5000 events/day * 28 days = 70,000,000 bytes.)

70MB doesn’t sound so bad. It’s probably well within the architectural limits of this memory-mapped I/O thing, but even that may be too big.

A better way of calculating what the local log size should be is to consider the recovery time objective and recovery point objective of the centralized log server. That is, consider how long it might take you to get your log server back online in the event of a disaster, then base your local Windows logging settings on that.

For example, if your recovery point objective is three days, you want to make sure that the local Windows logs will collect for at least three days, but not much more than that. Ideally, you also want to make sure that the solution you’re using can forward the logs collected during the downtime, or have a high-availability solution as part of the plan.

Remember, the goal is to get the local log sizes as small as possible, not as large as possible. You want to make sure you’re not losing any logs, and the two best ways to achieve that are to use centralized logging and try to avoid maxing out the services.exe process. This is counter-intuitive, but when you think it through, you’ll realize that it makes sense.