Immutable Security

It has been awhile since the last release of OSSEC and some users wonder if the project is really still active. Well, I am here to tell you that not only is it active, but it has been the most active it has ever been!

So, what have we been up to? As we prepare for the next beta release, which will happen in September, there has been lots going on:

1. We have been actively searching for uncommitted patches they may have been overlooked. Some of these are over a year old and have been contributed by other users. They fix bugs which have been lingering for awhile.
2. We have been dusting off rules and decoders that some of us have forgotten to contribute. Many of these are designed to decode additional fields, which should make rules more accurate.
3. Documentation is being worked on. Dan Parriott has done a wonderful job of writing and maintaining most of the documentation. It gets better all the time. Of course, Dan appreciates tickets and contributions against the doumentation.
4. Of course, there are new features. I won’t let the cat out of the bag yet, but I think many of them are pretty cool.

The end result is that we hope this will be the most stable and usable version of OSSEC yet. And we hope you’ll try it out and report any issues.

As to the next release after that? Expect big changes that fundamentally change the philosophy of OSSEC. Expect it to have more insight and context about attacks, with dynamic updates designed to have more up-to-date information on a much more frequent basis.

Trend did a great job of outlining our plan for OSSEC in this post. They begin by describing the Symposium, just as I did in my previous post, then go on to lay out a detailed plan for the future. A key theme is making OSSEC easier to use and more relevant for everyone. How do we get to this future? Well, that depends a lot on the amount of help we can get from the very talented members of the community. Remember, OSSEC is a volunteer community project and we welcome the assistance of anyone who wants to pitch in.

They are also hosting my presentations if you care to have a look. You probably won’t get as much out of them as you would have at the Symposium, since my slides tend to be simple, but the general ideas are there.

Day 1 details my experiences applying OSSEC over the last several years and can be found here.

Day 2 outlines what I think OSSEC can be–an extension of the great community that already exists. Access it here.

Ready to pitch in? Let us know…

If you missed the first OSSEC Symposium, you missed a great opportunity to meet fellow OSSEC users and developers, partake in great food and drink and immerse yourself in a day-and-a-half of pure OSSEC geekiness!

I arrived a bit early to Trend Micro’s corporate headquarters on Thursday and was warmly greeted by the receptionist, Vic and J.B. The weather was a beautiful and sunny 72 degrees and the Trend office was framed by a lovely California vista of rolling hills. We chatted casually as we dined on delicious sandwiches and waited for the Symposium to start.

J.B. led introductions and introduced presenters. My first keynote presentation (to be posted shortly) on my experiences using OSSEC was warmly received and I even managed to get a few chuckles with my airport travel shtick. :) We discussed our pain points deploying, using and administering OSSEC, and of course we had many of the same things to say. The day ended with a wonderful Italian dinner with my new friends.

On day two we started the day with breakfast and proceeded to talk about some of the larger deployments OSSEC has seen (over 4000+ hosts reporting to a single manager!) I gave a brief demo of my integration work with ELSA (to be finished shortly) and then we had some yummy Japanese food. After lunch, I had my chance to present my second keynote–my vision for OSSEC. That is to say, what I would do with the project if I had unlimited time, cooperation and talent. A central theme to my second presentation was not just technology, but how valuable and crucial the friendly community aspect is. Combine these two and you start to develop a shared vision.

The day ended with discussions on the practical aspects of how to move forward: these included bug fixes, rule tuning, agent deployment improvements and roles.

Trend has shown their renewed commitment to the project and has publicly promised to keep it free. I can only hope that we join them as community members to make OSSEC better for everyone.

Please join me at the first OSSEC Symposium, sponsored by Trend Micro. This is a forum for the OSSEC community to come together and discuss all things OSSEC. We’ll not only talk about what makes OSSEC so effective, but what we can improve upon, with the specific goal of laying out a road map for the next year. The cost is free and Trend is even throwing in the food, so this is a great way to get some free CPEs for your certifications.

I have also been asked to present and I’ll be talking about my experience using and developing for OSSEC, along with my vision for taking it to the next level. I hope too see you there!

When I first read about ELSA, I knew it was going to be a game changer. From the very beginning, this log collection and analysis application had addressed many of the problems plaguing adoption of open source log front-ends in the enterprise. It had scalability (as in pretty much unlimited scalability), it integrated with Active Directory, it had reporting and it was designed to handle thousands of logs per second without breaking a sweat. In other words, it was what many expensive SIEMs promised to be but failed to be in execution.

With an upcoming log infrastructure expansion project, I decided to give it a trial run on an old laptop. If things went well there, I could consider piloting the application for production usage. So with a fair bit of trepidation, I followed the instructions in the README and with two simple commands later it was furiously downloading and installing all sorts of stuff. The installation took over an hour on this poor little laptop, since many elements of ELSA were being compiled. Honestly, I didn’t expect things to go well. With all of these Perl modules and Ubuntu packages being installed, something had to fail and stop the entire chain of events. I was a bit surprised to see that it all just worked. At the end of the installation I opened a browser and was greeted with this nice and clean screen.

Since I already had the firewall logging to this box, it was simply a matter of disabling that application so ELSA could grab the port and start receiving syslogs. A few minutes later, I was able to run a basic query by simply putting in the IP of the firewall.

I expected some sort of hourglass. After all, this was a five year old laptop, not a server. But the query results came back pretty much immediately (199 milliseconds to be exact).

Performing additional queries proved to be somewhat more difficult, simply because I had not yet wrapped my mind around the search syntax. I tried 192.168 thinking I would see everything on that segment, since the doc mentioned that wildcards were not an option, but I got no results. After a bit more reading and experimenting , I got the hang of it. Still, it has taken me out of my comfort zone of grep-style regular expression searching and that will take some getting used to.

By clicking on the ‘Report On’ button and then selecting ‘Hour’ I had an instant visual of the level of log activity from this one host. From there I could save or export the results in PDF, Excel, CSV or HTML. Doing a line count in a days worth of logs for one IP using grep would have taken several minutes at least, never mind plotting it. The chart is somewhat Splunk-like, but without five thousand dollars to get started and the threat of locking you out of your log history should you go over your limit.

Poking around a bit more, I begin to see where ELSA really shines. The use of plugins allows you to leverage community intelligence and to do things like see of the IP in the logs is known to be malicious. Can I get an AMEN!!!??

So, what would I change? I am starting to get some ideas, but really, they are just cosmetic things that really don’t seem all that important right now. One thing is clear: ELSA was designed by someone who understands the needs of log analysts. We need a no-nonsense, clean and stable interface, fast results and enough enterprise-level features to be able to justify it in the work place and ELSA fits the bill perfectly. Listen up log pros, this is the one to watch.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Yesterday, I blogged about some annoying malware. The point was to learn some of the techniques that this general class of malware uses, so we could write some OSSEC rules to detect it. If you haven’t already read that post, go back and read it so you’ll understand this one.

One indicator of malware is that it changes the hosts file and redirects sites to localhost. There are three different ways we can detect this with OSSEC:

1. A basic file integrity check, although this doesn’t tell us what changed.
2. A command output check which echoes the contents of the file
3. A compliance check

File integrity checks are pretty simple. All you have to do is tell OSSEC to monitor the file. Let’s focus on the command output and compliance checks.

1. Command output. Unfortunately, the report_changes option is not available for Windows right now. To get the content of the file into OSSEC, we’ll simply use the built-in Windows type command to achieve our objective. Put this in your ossec.conf on the agent (watch for wrapping):

<localfile>
<log_format>full_command</log_format>
<command>%COMSPEC% /C type %WINDIR%\system32\drivers\etc\hosts | %WINDIR%\system32\findstr.exe /BVC:”#”</command> <alias>Windows Hosts File</alias>
</localfile>

Now, just add a rule like this so you’ll get the diff:

<rule id=”100027″ level=”7″>
<if_sid>530</if_sid>
<match>ossec: output: ‘Windows Hosts File’</match>
<check_diff />
<description>Windows Hosts File Changed</description>
</rule>

At this point in time, there is a bug in OSSEC, which means that the diff of the file will not be in the email, but it should be in the original alert in alerts.log.

2. Compliance checks. Compliance checks are part of the rootcheck functionality of OSSEC and are run right after the syscheck (file integrity) scans. They allow you to look within files, registry keys and processes for certain values, and alert on them. For this example, we want to know if there are any entries other than the expected localhost entry. Simply add this to your etc/shared/win_malware_rcl.txt file (watch for wrapping):

[ Site Redirected to localhost] [any] []

f:%WINDIR%\System32\Drivers\etc\HOSTS -> r:^127.0.0.1 && !%WINDIR%\System32\Drivers\etc\HOSTS -> r:^127.0.0.1\s+localhost;

Referring back to the original post, it would not be difficult to take the other examples and make them into general purpose checks. One word of caution: there is currently no local_win_malware_rcl.txt capability, so be sure to back up your changes; otherwise, they will be overwritten when you next upgrade.

What other examples from malware can you think of that would make for good rules? I would be happy to hear about your experiences in the comments.

When most people receive an email with a malicious attachment, they do one of two things: either they delete it, knowing that it is malicious, or they get fooled into executing the attachment, which ruins their day. Then there is the third category of people. They are people like me. These people intentionally infect a system to see what kinds of interesting things the malware does. Some guys like football. I like to analyze malware. So when I got an unexpected email purporting to be from FedEx, I had to let the attachment loose on a virtual machine.  I’m going to skip the detailed analysis and post a few screenshots, because the purpose of this post will be to convert what we have learned into something that OSSEC can detect.

As you can see, this is a garden-variety fake HDD scanner. It was pretty annoying–hiding my drives, killing my analysis tools, redirecting search results and the other usual things.

There’s one thing that is almost universally true about malware like this; it leaves evidence on the system. Our job is to look for those changes and write rules to the general class of behavior. It wouldn’t make much sense to write a rule looking for the specific, random file names that this trojan dropped; instead, we want to understand what happens in general.

Take a look at what MalwareBytes found. It also added the following to the hosts file:

127.0.0.1 thepiratebay.org
127.0.0.1 www.thepiratebay.org
127.0.0.1 mininova.org
127.0.0.1 www.mininova.org
127.0.0.1 forum.mininova.org
127.0.0.1 blog.mininova.org
127.0.0.1 suprbay.org
127.0.0.1 www.suprbay.org

What can we learn from this? Consider the following:

1. Three files were added to run at Windows startup. All of them are from locations in the user’s profile. Two of them are particular suspicious, as the run line begins with rundll32.
2. Several policies were changed. The effect was hiding standard parts of the OS. This may be legitimate for something like a kiosk, but is not generally used.
3. Two executable files were written to the root of \application data. Normal apps should not do this and I see this particular behavior with malware a lot these days.
4. Several URLs were  redirected to localhost. Generally, this is only done legitimately to block ad-serving sites, and there are better ways to do that these days.

As you can see, these are behavioral observations. In part II, we’ll take these observations and make them actionable within OSSEC. Stay tuned.

 

Are you an OSSEC ninja? Do you dress in orange and red and laugh maniacally at all of the frustrated attackers who have tried to take you down? Do you take medication for that condition? Ok, well, best of luck to you. Back to the topic at hand.

Here are some of my favorite tips and tricks. They promise to elevate you to the next level of OSSEC guru-ness, or maybe they’ll just help you solve a problem. Either way, in no particular order of importance:

1. Log everything. If you have the space, I highly recommend that you add the <logall>yes</logall> element to ossec.conf. After restarting OSSEC, all events will be written to logs/archives/archives.log. By default, OSSEC will discard something that doesn’t match a rule, but if you are ever compromised you really should have all logs to examine.
2. No decoder necessary. Do you need to match on specific text in an unsupported application? While decoders definitely extend the functionality of OSSEC, they are not strictly necessary to get an alert. You can simply create a rule using a simple match. For example: <match>hax0r logged in</match>.
3. Find all alerts above a specific level. I’m stealing this one from a past ossec-users list post of mine. If you have ever wanted to get a list of alerts above a certain level from your alerts log, this will do it. In this example, we look for anything above level 12:

grep -E -B2 -A3 ‘^Rule: [0-9]{1,6} \(level 1[2345]\) -> ‘.*’$’ logs/alerts/alerts.log \

| sed ‘s/^–$/ /’

4. Use OSSEC to monitor OSSEC. Oddly, OSSEC does not monitor itself out of the box. It’s a good idea to monitor your OSSEC instance, if for no other reason than to be confident that you or a trusted colleague are the only ones making production changes. Add this to your ossec.conf (watch for wrapping):

<directories check_all=”yes” realtime=”yes” report_changes=”yes”>/var/ossec</directories>

Now, because certain directories change frequently, you’ll need some exclusions:

<ignore>/var/ossec/queue</ignore>
<ignore>/var/ossec/logs</ignore>
<ignore>/var/ossec/stats</ignore>
<ignore>/var/ossec/var</ignore>

Finally, you’ll want to be careful about the alerts that you receive. One file that is monitored by this is client.keys, which contains, well, your keys. It’s not really a good idea to have OSSEC email you the diff of this file, so I would recommend creating a rule at level 0 to filter it out.

5. Manage centrally. Having only a few agents, it’s not that big of a deal to configure settings locally. When you get into the hundreds or thousands, it becomes very onerous. OSSEC has the ability to centrally manage your agent settings and I highly recommend that you use it. With this feature, all you need in your agent ossec.conf is the destination IP or hostname of the manager. I also like to enable active response locally in the Windows ossec.onf file because it uses active response to restart the agent–which, in turn, can be used to read the new agent.conf.

If you comb through the source code, you might also be able to find some other nuggets worth having. What have you discovered?

The future of successful HIDS will not rely solely upon research from a small group of people. There are simply far too many things to look for to be an expert in all areas. Attacks involve changes to the file system, running processes, new listening ports, various kinds of injections, overflows, and so-on. On top of that, we have to consider the reputation of networks that connect to us, how threats morph over time, and many other non-technical but interesting data points.

OSSEC is flexible enough to allow us to leverage other sources of community intelligence. It’s not necessary to just use the built-in rules. We can use OSSEC to bring in data that the rules then analyze for possible alerting. Two of the features that make this possible are Process Monitoring and CDB Lists. With these two features, we can run a command to download some interesting data, compile it into a list and reference that list from a rule.

One implementation of this idea is to take the list of bad IPs from the Dragon Research Group (or anyone else) and compile it into a list. Here’s an example of how to do just that by adding this to your ossec.conf on the manager (this assumes you have already set up CDB lists):

<localfile>
<log_format>full_command</log_format>
<command>wget -qO- http://www.dragonresearchgroup.org/insight/sshpwauth.txt | grep -v ‘#’ | awk -F “|” ‘{print $3,”:”$5;}’ | tr -d ‘ \t’ > /var/ossec/rules/lists/dragon_ips && /var/ossec/bin/ossec-makelists > /dev/null 2>&1</command>
<frequency>43200</frequency>
<alias>Dragon IP List</alias>
</localfile>

What does this do? It downloads the list of IPs from the URL in the command, massages the format, outputs it to /var/ossec/rules/lists/dragon_ips and compiles it into a CDB list with the ossec-makelists command. The <frequency> option is very important. In this case, we’re telling it to download once every 12 hours at the most. It would be rude to have this hit the remote web site too often.

To make this useful, you’re going to want a rule to reference the list. Here’s an example rule:

<rule id=”100050″ level=”7″>
<if_group>syslog|firewall|web</if_group>
<list field=”srcip” lookup=”address_match_key”>rules/lists/dragon_ips</list>
<description>IP Found in Dragon List</description>
</rule>

This rule simply looks for the IP in the Dragon list for any incoming log decoded as syslog, firewall or web. This rule could detect not only those IPs connecting to your firewall, but also your SSH server, web server and possibly even outbound proxy requests.

Now I’m going to tell you why this is a bad idea. OSSEC runs this command as root, so you’re essentially trusting that the remote web site is going to be nice and not feed you data designed to exploit this privilege. A much better way to do this would be to run a small script which downloads the file as a non-privileged user, does some sanity checking on the file, then compiles it, again, as a non-privileged user.

This idea can be used in-house to do things like compiling a list of privileged users across a Windows domain and having special rules for those people. Use your imagination.

One final note: be sure to request permission from the remote site or at least ensure that it doesn’t violate their terms of service. Being a good Internet citizen is important to efforts like this if they are to succeed.