Immutable Security

Special edition: Here are five log-a-licious links that I found interesting for this week:

• How can you know what amount of storage you will need when your centralized log server is fully operational? There are no hard-and-fast answers, but it is important to at least come up with an estimate. In his blog, Eric Romang shares a handy script to get you at least part of the way there.
• Eric does equally well in a follow-up post about calculating your events-per-second, or EPS.
• Have you ever had a need to write events to the Windows Application Log? Your scripts do output logs, don’t they? Here’s a short little VB script to help you along the way.
• What is the difference between a Windows “Account Logon” event and a “Logon/Logoff” event? I’m sure it made sense to someone at Microsoft at the time, but to many it is confusing. Here’s a good explanation.
• Are you collecting workstation logs? While I wouldn’t necessarily put it at the top of my list, it’s not a bad idea at least for select hosts. Think admins, kiosks, conference rooms and the like. More food for thought from Randy.

That’s it for today. Have a great weekend!

The OSSEC team is pleased to announce the general availability of v2.6. This version includes support for IPV6, a new tool for key management of ‘nix agents, an option to increase the block timeout for repeat offenders, and many other goodies.

Major kudos for this release go to Dan Parriott (ddpbsd). Dan is the most active person helping OSSEC users today on the mailing list and IRC. He also seems to find time to write documentation, which–let’s face it–no one really likes to do, and writes new rules and decoders. Thanks, Dan.

If you would like to see what I am up to in the OSSEC world, check out my repository on Bitbucket. My commits are generally tested and ready for integration into the next release, so try them out and let me know how they work for you. The tickets section is basically my task list of things I am already working on or plan to implement.

As always, thanks to everyone who contributes and supports our work. If you have some free time, stop by #ossec on freenode to say hi.

Open source log analysis types sometimes need to be crafty. We often don’t have relationships with the companies who’s products we support, but that doesn’t stop the users from wanting to use software like OSSEC to analyze logs from big, enterprise-type closed source stuff.

We scour the Internet for log samples and documentation–anything, really–that might help us out.  So today when I came across some documentation for IBM’s Compliance Insight Manager, I thought I would pass along what I found.

Under the Configuring auditing for specific platforms section, you’ll find instructions on how to configure auditing for dozens of enterprise systems. It even goes far beyond what to click or which file to edit. There is info on audit capabilities, monitoring recommendations, suggested audit settings and tuning.

This is a worthwhile resource to bookmark. Until next time..

One of the things us log analysis types love is good documentation. It’s rare to find well-formatted, well-documented logs, so when we do find good log info, it’s like being a kid in a candy store. So without further ado, I present all of the links you will need to describe every Windows Server Security Event log that exists:

Windows NT (please tell me you don’t really need this):

KB174074 – Security Event Descriptions

Windows 2000

KB299475 – Windows 2000 Security Event Descriptions (Part 1 of 2)

KB301677 – Windows 2000 Security Event Descriptions (Part 2 of 2)

Windows 2003

Windows 2003 Security Guide, Chapter 4, Audit Policy

Windows Vista to Present

KB947226 – Description of security events in Windows Vista and in Windows Server 2008

Security audit events for Microsoft Windows Server 2008 and Microsoft Windows Vista (This one is fantastic–every log in spreadsheet form. Export to csv for an easily-parsable and machine-readable format!)

Of course, just having the documentation doesn’t necessarily tell you the whole picture. Sometimes it’s nice to have a discussion about the log event. Two of my favorite sites for this are Ultimate Windows Security and EventID.net. In particular, the Security Log Encyclopedia is a great reference when you want to quickly know what a particular event ID means, but also get some contextual and practical information about the event.

Remember that documentation isn’t always correct. There may be subtle differences in what events are logged or how they are presented depending on the service pack level. What other resources have you found?

It was inevitable. Cloud services are popping up everywhere and it was only a matter of time before log-based services started to appear. But does that mean the cloud is the right place for your logs? What are the key indicators of success for such services? What can we learn from years of logging on private networks? If I were to design a cloud-based logging service, here are the top things I would consider:

• Avoid legacy syslog transport at all costs. Plain-text, UDP syslog is so last-century. The only reason we (security types) tolerate it is because we usually have no choice. Most of the network devices force our hand. To support syslog as a transport method over the Internet is asking for all sorts of trouble. There will be no way to completely eliminate log spoofing, log sniffing and lost logs. Customers serious about logging will simply not tolerate such chaos.
• Support syslog message format at all costs. Notice the difference here. I don’t want to see the syslog transport method being used over the Internet. Logs are not radio station streams. I do want to see the syslog message format supported. Any service who does not support the syslog message format will not be successful.
• Secure the transport. Again, we need to address the problem of log spoofing, log sniffing and lost logs. This necessitates TCP for trasmission over UDP, authentication of the log source and encryption of the data. There are a couple of ways this can be implemented: a VPN gateway can be established with the log provider or a log-collecting agent can be installed on one or more local machines. In both scenarios, the transport of the logs is addressed.
• Implement flexible delivery options. Not all logs can or should be sent in real time. Firewall and proxy logs, in particular, are very chatty, but can also be very important indicators of an attack. Some mainframe logs have to be submitted as batch FTP transfers. Some systems push their logs and some need to be polled. Some logs are only produced when a monthly or quarterly report runs. The point is that we need to consider multiple delivery options. A successful cloud offering will allow for log traffic shaping, queueing, real-time and batch transfers.
• Protect the application layer. Logs are pretty useless unless you can index, search and correlate them. This means you need a robust application to navigate the millions of messages you might be sending. It’s inevitable that people will attempt to attack the application layer. Not only does this mean securing the site against traditional web attacks, but also remote log injection. Logs are volatile and anything might show up. That gives the attacker a log of flexibility. Can the bad guy drop the customers table with a well-formed SQL query put into the username field of a local server? It’s certainly within the realm of possibility.
• Meet the traditional requirements. Log management programs are often driven by compliance needs. It is imperative that traditional requirements for log management programs can be met. This means we need controls around who accesses these logs, when they accessed them, from where, and why. This includes employees of the cloud service. How long will they be stored online and offline? What happens in the event of a subpeona or a breach? Customers may even demand that the cloud service providers, themselves, be audited under something like ISO 27001.
• Start planning for interoperability. I haven’t seen any efforts in the area of cloud log service interoperability, but I suspect this will become a pain point in the future. If there is anything we have learned from vendor lock-in in the past, it is that the customer’s needs are not best served by this approach. I, for one, would be extremely wary of any vendor that did not allow me to export all of my logs in a common industry-accepted format. What if the service were to go out of business? Would I lose years of valuable log data?

While this is not an exhaustive list, I do believe that now is the time to start a dialogue. Rather than try to stop new technology and services, let’s find ways to make them work effectively. Let’s talk about the risks and address them. Let’s learn from the mistakes of the past and prevent the same mistakes in the future.

There are already several good posts out there about decoders and rules, and how one uses both to add new application support in OSSEC. What I haven’t seen is the non-technical process behind adding new apps and making sure it is successful. In this final post for the Second Annual Week of OSSEC, I’d like to share my thought process on the correct way to add new applications.

The first thing to consider is whether or not this will provide enough security value to you for the effort. If these logs ordinarily don’t contain useful information such as authentication attempts, it may not be worth the effort.

Assuming it is worth it, the next thing you want to look for is an authoritative source for all possible logs that the app may generate. This usually means going to the vendor’s web site or asking them directly. Be prepared, however, for the equivalent of blank stares and in some cases, subtle hostility, as they think you must be a competitor trying to pry out some secret sauce.

If you’re lucky enough to find a good source for log entries, you’ll need to spend some time studying them to find the gems and to look for format anomalies. Is the username always in the same place? Are the delimiters consistent? Could a delimiter end up in the user name? More often than not, they won’t be well-formatted and there may even be variations between different versions of the software. As a Mark Minasi book I once read said, “be prepared to learn that documentation lies.”

When writing your decoders and rules, be as specific as possible. Imagine that someone will try to tamper with the logs and break your rules. Make liberal use of the ^ and $ characters, denoting the start and end of strings.  Be specific and avoid the use of regexes such as \. (which means anything) whenever possible.

When writing rules, try to pre-tune them in such a way as to alert the user only when it is important. Generally, this means you only want to alerts on things like policy changes, new privileged users, possible attacks, rare events, and so-on. Consider that two or more seemingly innocuous events combined together could be an attack, so think laterally and creatively. Try to use groups and the info tag in every rule.

Test, test, test. When you think it is perfect, test some more. Try to run them in a live environment for awhile to see how things go. I am certainly guilty of writing some bad decoders and rules, and have learned a lot from the experience. Sometimes the testing might even reveal a bug in OSSEC, or they might work when you test them, but stop working or work differently with another snapshot. So follow up with the anomalies.

Finally, if you think others might benefit, submit them to the project. You might even want to send them to the mailing list first for some peer review. If they’re useful, they may just get included in the next release, which means you will have the satisfaction of knowing you have helped many other people stay safe.

One of the hallmarks of good software is that you can cajole it into doing things it may not have been originally designed to do. Well designed software is small, modular, portable and secure. OSSEC falls into this category.

OSSEC can be run stand-alone or in a distributed architecture with one manager and many agents. But did you know that you could also run multiple, independent instances of OSSEC on one box? Think of them as little OSSEC minions designed to do your bidding.

Why might you want to do this? How about having separate development, test and production instances? How about testing new versions of OSSEC? How about separating different customers from one another, ensuring the log data does not intermix? Whatever the case, it is possible with OSSEC.

Here’s how to install one instance of OSSEC to protect the server on which it is installed, and another which will only monitor remote hosts. Additional remote instances can be installed based on your needs. These steps are designed to be followed on a host without OSSEC installed, but can be easily modified if you already have it installed. The steps are Red Hat/CentOS specific and again, are easily adaptable.

1. Begin by installing a local OSSEC instance (question one from the install script). This will be the installation designed to protect this host.
2. Once the installation has completed, start the installer again. This time, when it detects that you already have OSSEC installed and asks if you want to upgrade, tell it no. You will now be able to choose an alternate location to install your second instance. Install the second instance, but when it asks you if you want to run syscheck and rootcheck, tell it no. We do this so we don’t have competing rootcheck and syscheck scans running. Active response should also be disabled unless you take the extra precaution of ensuring your responses only execute on the agents.
3. Bind the instances to unique IPs. At this point, you have all you need for a local and a remote install. Each will work independently. However, if you want another instance, you’ll need to make another modification to ensure the remote instances each bind to only one IP address. If you already have multiple interfaces, this is relatively simple. If not, simply follow this excellent guide to create an alias off eth0.  Now, go into the <remote> section of ossec.conf in each remote instance and configure the <local_ip> option to point to the correct IP. Make sure each instance points to a unique IP.
4. Replace the default OSSEC init script. Finally, we need a custom init script to manage our minions. For this to work, we’ll take the concept that OSSEC already uses with the /etc/ossec-init.conf file and expand upon it.
   Create the directory /etc/ossec and assign it the proper permissions:

   #mkdir /etc/ossec && chown root.root /etc/ossec && chmod 700 /etc/ossec

   For each remote instance, copy the ossec-init.conf file from the /etc directory relative to the OSSEC instance (e.g. /var/ossec1/etc/ossec-init.conf, to the system /etc/ossec directory, while appending a random string to the end of the filename:

   #cp -a /var/ossec1/etc/ossec-init.conf /etc/ossec/ossec-init-$RANDOM.conf

   Back up and replace the OSSEC init script /etc/init.d/ossec with this one. Make it immutable so it doesn’t get overwritten on new installs.

   #chattr +i /etc/init.d/ossec

One final tip: I find it helpful to modify the maild source for each instance and preprend the subject line with something meaningful. Then, the alerts will come across as something like: Test Environment: 7 – Integrity checksum changed.

That’s it! You now have several OSSEC instances that can be managed individually with the ossec-control.sh script in each instance’s /bin directory, or globally with the system init script.

Just the other day, someone said to me, “How do I tame syscheck? I get all of these alerts right after I patch and it just drives me nuts!” Ok, that’s not really what they said. What they said was, “Who are you and what the heck is syscheck?” But I’m sure you get the idea.

I’m generally not a huge fan of file integrity alerts. In an environment which is patched often, you can be bombarded with them. Then, because of the human tendency to filter out noise, you’ll miss the important stuff. So I generally think you should only get these kinds of alerts when they really matter.

Before we discuss how to deal with patching, let’s do a quick review of how syscheck scans work in OSSEC.

1. OSSEC scans the system and sends the hashes to the manager.
2. The manager then inserts or updates the syscheck database, which is an ASCII file with a name like: queue/syscheck/(agent-name) 1.2.3.4->syscheck
3. You get an alert, unless one of the following is true:
   1. The file hasn’t changed.
   2. The file has changed more than 3 times and <auto_ignore> is set to yes in ossec.conf.
   3. You have ignored the file with the syscheck_control -f option.
   4. You have ignored the file/directory with <ignore> in ossec.conf.
   5. A rule matches on the file and is configured in such a way as to not send you an alert.

So what can you do to not be bombarded with alerts when you patch? There are a few options. These suggestions assume you have real-time syscheck enabled. Just before patching:

1. Clear the database. Execute syscheck_control -u <agent_id>. Only use this option if you are comfortable with not being able to report on file changes for this host using syscheck_control. You can still query the SQL database for file changes if you are using that option, or by using the reporting daemon.
2. Create a rule to ignore the changes. Of course, you probably don’t want to ignore all file changes all the time, so you might use a rule like this to coincide with your patch window:

<rule id=”100000″ level=”5″>
<if_group>syscheck</if_group>
<time>08:00-10:00</time>
<weekday>sunday</weekday>
<match>C:\WINDOWS</match>
<description>Ignore file changes during patch Sunday</description>
</rule>

You could take this one step further by creating a parent rule, which this one is dependent upon, which looks for the event indicating that a patch has been installed. In this case, it might be useful to use the frequency and timeframe options in the dependent rule instead of time and weekday, so as to match a smaller window of time.

3. Let OSSEC auto ignore files that change more than three times. I generally prefer more control and granularity over my environment, and don’t want to be surprised by too much, so I disable this.

These steps are Windows-centric, but can easily apply to other platforms. For Linux systems, you may want to have a yum event as the parent rule and look for changes to something like /bin.

There are some risks. It’s possible that an attacker can modify files in a system directory while you are patching. These rules don’t actually correlate the process which is patching with the process that changed the files, so you’ll want to use a pretty tight time frame.

Finally, I’ll wet your appetite for what OSSEC holds in the future. Using CDB lists, it will be possible to add your pre-approved hashes to a list and have a rule check to see if they are in the list before alerting you. With a central source for trusted hashes, this should be easy to automate.

Update: ddpbsd correctly pointed out that clearing the database will create a new baseline. This means that you won’t get alerted to the next change on existing files, even if they are not in a patched location–a potentially worrisome problem! You’re better off just using rules.

Are you an OSSEC ninja? Do you dress in orange and red and laugh maniacally at all of the frustrated attackers who have tried to take you down? Do you take medication for that condition? Ok, well, best of luck to you. Back to the topic at hand.

Here are some of my favorite tips and tricks. They promise to elevate you to the next level of OSSEC guru-ness, or maybe they’ll just help you solve a problem. Either way, in no particular order of importance:

1. Ensure a clean restart. After you change your local_rules.xml file, execute “ossec-logtest -t” before you restart OSSEC. If it doesn’t complain, you will have no problems restarting. If it complains, OSSEC would not have started successfully (and you would not be analyzing logs until the error was fixed).
2. Set up a simple test case. Not sure if OSSEC should be sending alerts? Set up a testing rule with something not likely to be found in a log:

<rule id=”100000″ level=”8″>
<match>oogabooga</match>
<description>Testing Rule</description>
</rule>

Now test the rule with the logger utility:

# logger oogabooga

3. Monitor thousands of agents. By default, OSSEC will handle up to 256 agents. But there are successful installations with thousands of agents. You simply need to increase one setting in OSSEC.

#cd src; make setmaxagents
#cd ..; ./install.sh

It also helps to increase a few kernel parameters for your system:
# ulimit -n 2048
# sysctl -w kern.maxfiles=2048
# sysctl -w net.core.rmem_default=5123840
# sysctl -w net.core.rmem_max = 5123840

I like to put OSSEC on its own partition and optimize it a bit with the noatime value. For example:

/dev/VolGroup00/ossec /log/ossec ext3 defaults,noatime 1 2

As an added benefit, if something goes crazy and starts sending out millions of logs, it won’t fill up the root partition and take down the entire server.

4. Fire and forget. Normally, the OSSEC manager and agents talk to one another. The manager keeps track of agent status and so-on. If you would prefer to simply have your agents fire off the message to the manager and not have it reply (such as when using a restrictive firewall), try the setoneway option:

#cd src; make setoneway
#cd ..; ./install.sh

Now you can put the manager behind an uber-restrictive firewall and feel like a true security ninja!

5. Create a nicer looking subject. The default OSSEC alert format is a bit verbose. It tells you it is OSSEC in the from field, as well as in the subject and the body. OK, I get it. You’re OSSEC. By doing this:

#cd src; make setfullsubject
#cd ..; ./install.sh

And this:

#cp -a /var/ossec/etc/internal_options.conf /var/ossec/etc/internal_options_orig.conf
#sed ‘s/maild.full_subject=1/maild.full_subject=0/g’ etc/internal_options.conf

The look of your subject line will change from this:

OSSEC Notification – (agent-name) 1.2.3.4 – Alert level 10

To this:

10 – Multiple web server 500 error code (Internal Error). – (agent-name) 1.2.3.4

In the second example, we have the most important information first (the level), followed by a meaningful alert name, then the agent name and IP. This allows you to quickly discern the importance of the alert, and is particularly useful when you have a large number of alerts to read. Just bear in mind that, unless you use the do_not_group option, this one alert subject could represent several different actual alerts in the body of the e-mail.

If you comb through the source code, you might also be able to find some other nuggets worth having. What have you discovered?

Yesterday, I blogged about how we could beat OSSEC up, or, to put it more accurately, the people and protocols behind it. Today, we’re going to discuss how we can fight back against the bullies. For this post to make sense, you’ll need to read the first one. Go ahead, I’ll wait…

Ok. Put your gloves on. Here are some countermeasures and considerations for all of the risks we discussed previously.

1. Alert manipulation. Recall that alert manipulation is the process by which alerts are used against us.
   1. Alert tickling. The idea with alert tickling is to send just enough normal-looking alerts to create a window for the attacker to go undetected. There are a couple of ways to defend against this. One is active response. By default, anything that is level 6 or higher can trigger an active response such as blocking the IP for 15 minutes. The cool thing about active response is that it will continue to work even if your alerts are delayed until the next hour. This means the attacker might still be blocked regardless of when you get the e-mail. Another countermeasure is to increase the default number of alerts per hour to something like 9999. Of course, this opens you up to the next risk of alert flooding. Perhaps the best countermeasure is to configure the do_not_delay option in ossec.conf. Adding something like this would save all e-mails over 12 until the next hour, except for those that appear to be an attack:

   <email_alerts>
   <email_to>secanalyst@example.com</email_to>
   <group>attacks</group>
   <do_not_delay />
   </email_alerts>

   2. Alert flooding. With alert flooding, the attacker would like to swamp you with so many alerts that you just can’t bear to read them all. Besides tuning, the best option to protect against alert flooding is to keep the email_maxperhour setting at a reasonable level. If you choose to go that route, be sure to note the risks that can introduce and plan accordingly.
2. Killing the OSSEC processes. I am a firm believer in the mantra that if someone has administrative access to your system, they can do whatever they want. Ultimately, there is nothing you can do to completely protect against this risk. On the other hand, not all administrative users are savvy enough to be able to subvert the ossec processes. A well-written SeLinux policy just might be enough to hold them back. Also, recall that I talked about the non-malicious attacker. Non-technical safeguards such as background checks, job rotation and common workstation areas might be enough of a deterrent for those situations.
3. Log injection. With log injection, the attacker wants to control the log output. There are two main types of log injection: remote and unauthenticated. They can also be combined for greater efficiency.
   1. Remote log injection. The basic defense against remote log injection is to first assume that all input is malicious. Nothing from the user can be trusted, even if your OSSEC installation is on a private network. The second line of defense is to write your OSSEC decoders in such a way as to restrict where in the message input can be coming from. Referring again to Daniel’s paper, Attacking Log Analysis Tools, we can see that adding a ^ (to denote the beginning of the line) or a $ (to denote the end of a line) is often enough.
   2. Unauthenticated log sources. The obvious countermeasure to unauthenticated log sources is to not use them. However, while that is the best defense, it is rarely practical in a modern network. Sometimes you are simply faced with the choice of getting logs through syslog, or not at all. Given the choice, I would choose to get the logs. One approach is to design an out-of-band network where your unauthenticated network devices can send their logs. That approach is rather expensive and not very pragmatic for most people. The more practical approach is to enable IP spoofing protection on your network devices, such as in this example. Be aware, though, that it may not protect against attacks within the same broadcast domain.
4. Traffic manipulation. This attack can be difficult to prevent. Not only does it involve the direct manipulation of traffic, as is the case with something like ARP poisoning, but also intentional or unintentional flooding. Common countermeasures include NAC protection, blocking DoS attacks upstream and not using unauthenticated log sources.
5. Active response manipulation. Active response can prevent an attack from becoming a breach. It can also enable an attacker to block legitimate traffic. OSSEC accounts for this by giving you the white_list option for use in ossec.conf. Any IP or range added to this option won’t be blocked by active response. Keep in mind that a trusted host may be compromised first, with the attacker looking to elevate the access. The safest option is to not whitelist anything. OSSEC also includes a timeout for responses, so if you are accidentally blocked, you can try again in about 15 minutes.

By now you may be starting to realize that the default configuration of OSSEC is very sane and has taken most of these risks into account. At the same time, it gives you enough flexibility to meet your needs and to make your own risk decision. The important thing is to consider the possible consequences of changing a default setting and be comfortable with the additional risks it may present.