Immutable Security

Well, despite my best efforts, the day 7 post is going to be a bit delayed. But I think you’ll like it. So, stay tuned.

Yesterday, I blogged about some annoying malware. The point was to learn some of the techniques that this general class of malware uses, so we could write some OSSEC rules to detect it. If you haven’t already read that post, go back and read it so you’ll understand this one.

One indicator of malware is that it changes the hosts file and redirects sites to localhost. There are three different ways we can detect this with OSSEC:

1. A basic file integrity check, although this doesn’t tell us what changed.
2. A command output check which echoes the contents of the file
3. A compliance check

File integrity checks are pretty simple. All you have to do is tell OSSEC to monitor the file. Let’s focus on the command output and compliance checks.

1. Command output. Unfortunately, the report_changes option is not available for Windows right now. To get the content of the file into OSSEC, we’ll simply use the built-in Windows type command to achieve our objective. Put this in your ossec.conf on the agent (watch for wrapping):

<localfile>
<log_format>full_command</log_format>
<command>%COMSPEC% /C type %WINDIR%\system32\drivers\etc\hosts | %WINDIR%\system32\findstr.exe /BVC:”#”</command> <alias>Windows Hosts File</alias>
</localfile>

Now, just add a rule like this so you’ll get the diff:

<rule id=”100027″ level=”7″>
<if_sid>530</if_sid>
<match>ossec: output: ‘Windows Hosts File’</match>
<check_diff />
<description>Windows Hosts File Changed</description>
</rule>

At this point in time, there is a bug in OSSEC, which means that the diff of the file will not be in the email, but it should be in the original alert in alerts.log.

2. Compliance checks. Compliance checks are part of the rootcheck functionality of OSSEC and are run right after the syscheck (file integrity) scans. They allow you to look within files, registry keys and processes for certain values, and alert on them. For this example, we want to know if there are any entries other than the expected localhost entry. Simply add this to your etc/shared/win_malware_rcl.txt file (watch for wrapping):

[ Site Redirected to localhost] [any] []

f:%WINDIR%\System32\Drivers\etc\HOSTS -> r:^127.0.0.1 && !%WINDIR%\System32\Drivers\etc\HOSTS -> r:^127.0.0.1\s+localhost;

Referring back to the original post, it would not be difficult to take the other examples and make them into general purpose checks. One word of caution: there is currently no local_win_malware_rcl.txt capability, so be sure to back up your changes; otherwise, they will be overwritten when you next upgrade.

What other examples from malware can you think of that would make for good rules? I would be happy to hear about your experiences in the comments.

When most people receive an email with a malicious attachment, they do one of two things: either they delete it, knowing that it is malicious, or they get fooled into executing the attachment, which ruins their day. Then there is the third category of people. They are people like me. These people intentionally infect a system to see what kinds of interesting things the malware does. Some guys like football. I like to analyze malware. So when I got an unexpected email purporting to be from FedEx, I had to let the attachment loose on a virtual machine.  I’m going to skip the detailed analysis and post a few screenshots, because the purpose of this post will be to convert what we have learned into something that OSSEC can detect.

As you can see, this is a garden-variety fake HDD scanner. It was pretty annoying–hiding my drives, killing my analysis tools, redirecting search results and the other usual things.

There’s one thing that is almost universally true about malware like this; it leaves evidence on the system. Our job is to look for those changes and write rules to the general class of behavior. It wouldn’t make much sense to write a rule looking for the specific, random file names that this trojan dropped; instead, we want to understand what happens in general.

Take a look at what MalwareBytes found. It also added the following to the hosts file:

127.0.0.1 thepiratebay.org
127.0.0.1 www.thepiratebay.org
127.0.0.1 mininova.org
127.0.0.1 www.mininova.org
127.0.0.1 forum.mininova.org
127.0.0.1 blog.mininova.org
127.0.0.1 suprbay.org
127.0.0.1 www.suprbay.org

What can we learn from this? Consider the following:

1. Three files were added to run at Windows startup. All of them are from locations in the user’s profile. Two of them are particular suspicious, as the run line begins with rundll32.
2. Several policies were changed. The effect was hiding standard parts of the OS. This may be legitimate for something like a kiosk, but is not generally used.
3. Two executable files were written to the root of \application data. Normal apps should not do this and I see this particular behavior with malware a lot these days.
4. Several URLs were  redirected to localhost. Generally, this is only done legitimately to block ad-serving sites, and there are better ways to do that these days.

As you can see, these are behavioral observations. In part II, we’ll take these observations and make them actionable within OSSEC. Stay tuned.

Are you an OSSEC ninja? Do you dress in orange and red and laugh maniacally at all of the frustrated attackers who have tried to take you down? Do you take medication for that condition? Ok, well, best of luck to you. Back to the topic at hand.

Here are some of my favorite tips and tricks. They promise to elevate you to the next level of OSSEC guru-ness, or maybe they’ll just help you solve a problem. Either way, in no particular order of importance:

1. Log everything. If you have the space, I highly recommend that you add the <logall>yes</logall> element to ossec.conf. After restarting OSSEC, all events will be written to logs/archives/archives.log. By default, OSSEC will discard something that doesn’t match a rule, but if you are ever compromised you really should have all logs to examine.
2. No decoder necessary. Do you need to match on specific text in an unsupported application? While decoders definitely extend the functionality of OSSEC, they are not strictly necessary to get an alert. You can simply create a rule using a simple match. For example: <match>hax0r logged in</match>.
3. Find all alerts above a specific level. I’m stealing this one from a past ossec-users list post of mine. If you have ever wanted to get a list of alerts above a certain level from your alerts log, this will do it. In this example, we look for anything above level 12:

grep -E -B2 -A3 ‘^Rule: [0-9]{1,6} \(level 1[2345]\) -> ‘.*’$’ logs/alerts/alerts.log \

| sed ‘s/^–$/ /’

4. Use OSSEC to monitor OSSEC. Oddly, OSSEC does not monitor itself out of the box. It’s a good idea to monitor your OSSEC instance, if for no other reason than to be confident that you or a trusted colleague are the only ones making production changes. Add this to your ossec.conf (watch for wrapping):

<directories check_all=”yes” realtime=”yes” report_changes=”yes”>/var/ossec</directories>

Now, because certain directories change frequently, you’ll need some exclusions:

<ignore>/var/ossec/queue</ignore>
<ignore>/var/ossec/logs</ignore>
<ignore>/var/ossec/stats</ignore>
<ignore>/var/ossec/var</ignore>

Finally, you’ll want to be careful about the alerts that you receive. One file that is monitored by this is client.keys, which contains, well, your keys. It’s not really a good idea to have OSSEC email you the diff of this file, so I would recommend creating a rule at level 0 to filter it out.

5. Manage centrally. Having only a few agents, it’s not that big of a deal to configure settings locally. When you get into the hundreds or thousands, it becomes very onerous. OSSEC has the ability to centrally manage your agent settings and I highly recommend that you use it. With this feature, all you need in your agent ossec.conf is the destination IP or hostname of the manager. I also like to enable active response locally in the Windows ossec.onf file because it uses active response to restart the agent–which, in turn, can be used to read the new agent.conf.

If you comb through the source code, you might also be able to find some other nuggets worth having. What have you discovered?

The future of successful HIDS will not rely solely upon research from a small group of people. There are simply far too many things to look for to be an expert in all areas. Attacks involve changes to the file system, running processes, new listening ports, various kinds of injections, overflows, and so-on. On top of that, we have to consider the reputation of networks that connect to us, how threats morph over time, and many other non-technical but interesting data points.

OSSEC is flexible enough to allow us to leverage other sources of community intelligence. It’s not necessary to just use the built-in rules. We can use OSSEC to bring in data that the rules then analyze for possible alerting. Two of the features that make this possible are Process Monitoring and CDB Lists. With these two features, we can run a command to download some interesting data, compile it into a list and reference that list from a rule.

One implementation of this idea is to take the list of bad IPs from the Dragon Research Group (or anyone else) and compile it into a list. Here’s an example of how to do just that by adding this to your ossec.conf on the manager (this assumes you have already set up CDB lists):

<localfile>
<log_format>full_command</log_format>
<command>wget -qO- http://www.dragonresearchgroup.org/insight/sshpwauth.txt | grep -v ‘#’ | awk -F “|” ‘{print $3,”:”$5;}’ | tr -d ‘ \t’ > /var/ossec/rules/lists/dragon_ips && /var/ossec/bin/ossec-makelists > /dev/null 2>&1</command>
<frequency>43200</frequency>
<alias>Dragon IP List</alias>
</localfile>

What does this do? It downloads the list of IPs from the URL in the command, massages the format, outputs it to /var/ossec/rules/lists/dragon_ips and compiles it into a CDB list with the ossec-makelists command. The <frequency> option is very important. In this case, we’re telling it to download once every 12 hours at the most. It would be rude to have this hit the remote web site too often.

To make this useful, you’re going to want a rule to reference the list. Here’s an example rule:

<rule id=”100050″ level=”7″>
<if_group>syslog|firewall|web</if_group>
<list field=”srcip” lookup=”address_match_key”>rules/lists/dragon_ips</list>
<description>IP Found in Dragon List</description>
</rule>

This rule simply looks for the IP in the Dragon list for any incoming log decoded as syslog, firewall or web. This rule could detect not only those IPs connecting to your firewall, but also your SSH server, web server and possibly even outbound proxy requests.

Now I’m going to tell you why this is a bad idea. OSSEC runs this command as root, so you’re essentially trusting that the remote web site is going to be nice and not feed you data designed to exploit this privilege. A much better way to do this would be to run a small script which downloads the file as a non-privileged user, does some sanity checking on the file, then compiles it, again, as a non-privileged user.

This idea can be used in-house to do things like compiling a list of privileged users across a Windows domain and having special rules for those people. Use your imagination.

One final note: be sure to request permission from the remote site or at least ensure that it doesn’t violate their terms of service. Being a good Internet citizen is important to efforts like this if they are to succeed.

Do you know how many events your OSSEC server receives? This can be an important piece of information for scaling a roll-out, or just to brag. I was curious, so I whipped up a little script to tell me.

#Reset counters
COUNT=0
EPSSUM=0
EPSAVG=0
#Source OSSEC Dir
. /etc/ossec-init.conf

for i in $(grep 'Total events for day' \
${DIRECTORY}/stats/totals/*/*/ossec-totals-*.log | cut -d: -f3); do
  COUNT=$((COUNT+1))
  DAILYEVENTS=$i
  EPSSUM=$(($DAILYEVENTS+$EPSSUM))
done

EPSAVG=$(($EPSSUM/$COUNT/(86400)))

echo Your total lifetime number of events colected is: $EPSSUM
echo Your total daily number of events average is: $(($EPSSUM/$COUNT))
echo Your daily events per second average is: $EPSAVG

I hope you find this useful.

Today begins the Third Annual Week of OSSEC. I put together a list of discussion topics (one per day) for the ossec-users mailing list, which are designed to build upon one another. I would like to encourage everyone to post something about OSSEC and to visit the mailing list.

Here are the upcoming topics…

Day 1: Kickoff: the week ahead.
This post. If you think about it too long, you’ll realize it’s a recursive loop and go crazy.

Day 2: Tell your story. How has OSSEC helped you?
This is the day we get to recount our experiences of how OSSEC has saved the day, or just saved us some scratch. People coming by later on who read these will get a sense for OSSEC and if it can work in their environment.

Day 3: Time to share: rules, configs, tips and tricks.
Post your rules, best-practices and so on. This is a great option for those who don’t want to create something like a full blog post. Just reply with something quick and dirty.

Day 4: What bugs you: problems, challenges and room for improvement.
List the most annoying bugs. What makes OSSEC difficult to use? What is the biggest area for improvement? What are we missing?

Day 5: Shared intelligence: what does an attack look like?
Let’s think about the actual attack vectors and hallmarks of an attack. What happens when a host is attacked? What are the usual sequence of events that take place? How can OSSEC effectively detect these while keeping the noise down?

Day 6: Time to dream: what does the future of OSSEC look like?
This is the big perspective on the future of OSSEC. Think BIG! It doesn’t matter if no HIDs has ever done it before. It doesn’t even matter if you think it can’t be done. Let’s dream.

Day 7: Making it happen: who, what, when and how?
On this day, we’ll try to take some of what we have learned and develop a plan of action. We’ll take the combined community intelligence and see if we can make it real. Feel free to jump in with your talents to solve a problem!

Sorry, there will be no Fav Five this week. Instead, I am spending my time on writing a journal article and preparing for the Week of OSSEC. Have a great weekend!

#include<stdio.h>

main()

{

printf(“R.I.P., Dennis. Your contributions will not be forgotten.\n”);

}

Special edition: Here are five log-a-licious links that I found interesting for this week:

• How can you know what amount of storage you will need when your centralized log server is fully operational? There are no hard-and-fast answers, but it is important to at least come up with an estimate. In his blog, Eric Romang shares a handy script to get you at least part of the way there.
• Eric does equally well in a follow-up post about calculating your events-per-second, or EPS.
• Have you ever had a need to write events to the Windows Application Log? Your scripts do output logs, don’t they? Here’s a short little VB script to help you along the way.
• What is the difference between a Windows “Account Logon” event and a “Logon/Logoff” event? I’m sure it made sense to someone at Microsoft at the time, but to many it is confusing. Here’s a good explanation.
• Are you collecting workstation logs? While I wouldn’t necessarily put it at the top of my list, it’s not a bad idea at least for select hosts. Think admins, kiosks, conference rooms and the like. More food for thought from Randy.

That’s it for today. Have a great weekend!