Immutable Security

Most people that use the OSSEC Windows agent download a pre-compiled copy from the OSSEC site. While that is a good option for many individual users, it may not suit those with more specific needs and/or those in enterprise environments. Users who fall into those categories could benefit from customizing the agent and maintaining internal builds in order to suit their individual needs.

There are already instructions on how to compile the Windows agent on Linux, but ironically the process doesn’t work so well on Windows. I had a need to make this work on Windows, so I thought I would share the process with you.

First, there are some prerequisites.  You’ll need:

• The Nullsoft Scriptable Install System (NSIS)
• The Minimalist GNU for Windows compiler (MinGW)
• My gen_win batch file.  Simply rename from gen_win.txt to gen_win.cmd.
• 7-Zip for Windows
• The public domain Unix2DOS utility
• The latest OSSEC for Unix/Linux (this contains the Windows source code)

Here are the steps:

1. Download and install the required programs. Be sure to pay special attention to the steps for properly installing and configuring MinGW, particularly the part about modifying the PATH environment variable.
2. Next, we’re going to extract OSSEC using 7-Zip. To do so, simply right-click on the file and select 7-Zip, extract to “folder name.tar,” where folder name is the name of the package. This decompresses the archive. Navigate within that folder and repeat this step to untar the archive. At this point, you should see all of the files in the package.
3. Place gen_win.txt in the src\win32 folder and rename the extension to .cmd.
4. Download Unix2DOS and place it in the src\win32 folder
5. Open a command prompt. Navigate to src\win32, make any desired customizations, and execute gen_win.cmd. This should gather all of the required files and place them in src\win-pkg.
6. Next, we compile the Windows agent by navigating to src\win-pkg and executing make.bat (I assume you have the chops to know how to change directories :) ).
7. Now we have all of the files we need but no way to effectively install it. To generate the installer, simply execute the NSIS compiler like so: “c:\Program Files\NSIS\makensis.exe” ossec-installer.nsi

If you see no errors and a binary named ossec-win32-agent.exe, everything was successful. Congratulations, you now have a custom-made version of OSSEC!

For those that did not see me at the Rochester Security Summit last year, and who would like to see me present my OSSEC in the Enterprise presentation, I will be giving it again at the ISSA Ft. Worth chapter meeting, next Wednesday. Details are as follows:

Meeting Location: Devry University, 301 W. Commerce Street, Fort Worth, TX 76102

Phone: 817-810-9114

Parking is up to you, there is a parking garage, lots & meters.

FWT Consulting will be sponsoring the meeting & providing lunch, so please RSVP to steve.streiffert <a t> gmail.com by Monday, May 17, 2010.

Many years ago, after I had been using OSSEC in an enterprise setting for a few months, I noticed an interesting phenomenon. Administrators, many of whom I had forwarded “was this you?” alerts to, were now coming to me to rat on themselves.

I would be working away in my cubicle when someone would come up behind me. It went something like this:

“Hey, Mike. Just wanted to let you know what you’ll be seeing me <insert action here> in the logs. No cause for concern.”

“Thanks for the heads up,” I would reply.

The administrators knew they were being monitored, but didn’t exactly know the full details of the monitoring. They naturally assumed I would see what they were up to. In many cases I wouldn’t have known.

The vast majority of these folks were honest to begin with, but I can’t help think it assisted them in following process and being just a bit more transparent with what they were doing. Maybe it even dissuaded someone who was on the fence from doing something vindictive.

After seeing this in other environments, I think it deserves a term. At the risk of coining a term for something that has already been identified, I hereby declare this the “OSSEC Effect.” The definition (which could use some refinement) is as follows:

OSSEC Effect: The alteration of a computer user’s behavior when they know their actions are being monitored, but do not realize or understand the extent of the monitoring. Users will, without provocation, volunteer information they believe could be seen as questionable, whether the monitoring system would have known about it or not.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

Here’s a little secret that the sales guys of the million-dollar SIEMs are probably going to gloss over. Most of them do not offer a way to encrypt logs in transit end-to-end. Worse, many of them use downright silly methods to collect logs on common operating systems, such as installing an FTP client on every host and sending them along as a big batch every so often.

For some reason, this is seen as acceptable since it’s usually on the internal network. At the same time, we give mean looks to anyone who even mentions the use of a clear-text protocol like telnet.

Here’s another little secret: many of the big SIEM vendors use syslog-ng as the log daemon on their appliances. It’s reliable and secure and generally works very well.

What if there was a way to add encrypted log transit to the expensive SIEM, or to the other log management solution you already have in place? What if you can’t use or don’t need the full functionality of a log analysis solution like OSSEC, but would like the ability to use encrypted log transport? Read on, my friend, because I’m going to show you exactly how to do that, and it can be done in just a few minutes.

1. Install OSSEC in server mode on the existing syslog server from the vendor. I won’t go into detail here on how to install OSSEC, as it’s well documented elsewhere. These systems are generally just minimal Linux installations, so if you are lacking a compiler then you may need to install it on a like system and move OSSEC over. If your vendor won’t let you do this, ask them why you are paying them gobs of money without even the basic functionality of encrypted log transit!
2. Install OSSEC agents on the hosts you want to collect logs from. Configure the agents to collect the logs you are interested in and add them to the manager. Don’t forget to restart the OSSEC manager after adding the agents.
3. Add the following lines to /opt/syslog-ng/etc/syslog-ng.conf under the “source s_local” section:
   # Read OSSEC queue
   unix-dgram(”/var/ossec/queue/ossec/queue”);
4. Edit /var/ossec/etc/ossec-control and modify the two variables which are responsible for starting the OSSEC daemons. After modification, they should look like this:
   DAEMONS=”ossec-remoted”
   SDAEMONS=”ossec-remoted”
5. Finally, stop both syslog-ng and OSSEC. Start syslog-ng first, then start OSSEC.

The reason this works is due to the way OSSEC is designed. In fine Unix tradition, OSSEC separates the tasks into individual daemons that do only one job and do it well. In this case, ossec-remoted is writing to a Unix socket that syslog-ng is reading from in real-time. OSSEC is simply handing off the message to the local syslog daemon, at which point the message continues on its way as it normally would.

I used to use Snare, Zebedee and OSSEC for secure log delivery and file integrity. Now, it can all be handled with one open-source agent, already designed for secure and reliable log delivery.

There is a downside, which is primarily due to me not taking this to the next level and looking into the issue. In this configuration, you will not receive a notification if the OSSEC agent disconnects, since there is nothing to act on the absence of the heartbeat OSSEC sends. I’m pretty sure that can be solved by taking it up a notch and including one or two more daemons, combined with a stripped down decoder and rules file.

So the next time your expensive SIEM vendor tells you there’s no way to encrypt the logs in transit, tell them you can set it up in under an hour. Then send them a bill for your time.

OSSEC is one of those tools that continues to surprise me with its ability to perform low-level and important security tasks. In fine Unix tradition, individual parts of OSSEC can often be persuaded to do your bidding in ways not previously conceived. Put another way, it’s nice to hack. :)

Today, I’m going to show you how to use OSSEC to detect sensitive info within flat files. This example will demonstrate American social security numbers, but the same logic could be used to detect clear-text credit card numbers (a violation of the PCI DSS standard), or other personally identifiable information which should not be in clear-text.

We’re going to rely on the rootcheck functionality of OSSEC. Rootcheck does all kinds of nifty things, one of which is to scan within files for patterns. Normally, you would expect to look for things like insecure system configuration settings. We’re going to use it to find possible social security numbers.

First, create a test file containing a properly formatted SSN in a location that OSSEC is monitoring:

echo 123-45-6789 > /var/www/html/example.com/www/customer_data

Next, add the following to /var/ossec/etc/shared/system_audit_rcl.txt:

# Detect possible SSNs
[Possible Unencrypted Social Security Number Detected] [any] []
d:$web_dirs -> r:^\. -> r:\d\d\d-\d\d-\d\d\d\d;

Now, let’s create an alert so we get a heads up on what OSSEC finds (change the rule ID as needed):

<rule id=”100024″ level=”12″>
<if_sid>516</if_sid>
<match>Unencrypted Social Security Number</match>
<description>Possible Unencrypted Social Security Number Detected</description>
</rule>

Restart OSSEC:

/var/ossec/bin/ossec-control restart

Finally, we’ll initiate a scan on the local system:

/var/ossec/bin/agent_control -r -u 000

And here is what the resulting alert looks like:

OSSEC HIDS Notification.

2010 Jan 12 18:25:43
Received From: hostname->rootcheck
Rule: 100024 fired (level 12) -> “Possible Unencrypted Social Security Number Detected”
Portion of the log(s):

System Audit: Possible Unencrypted Social Security Number Detected. File: /var/www/html/example.com/www/customer_data.

–END OF NOTIFICATION

One of the nice things about the alert above is that it doesn’t actually include the contents of the file, so you don’t have to worry about clear-text data traversing your mail server.

There are also a few caveats to keep in mind.

First, it is slightly prone to false-positives. Since the OSSEC regex library is designed for speed, rather than full regex support, it’s not possible to use something like \b to define a word boundary. This means that while the regex will match SSNs like 123-45-6789, it would also match something like 00123-45-678911. It also wouldn’t match a delimiter other than the dash. You might be able to mitigate some of these risks if you’re sure how the data would be formatted (but then again, if you’re sure you have clear-text data on the server why do you need this :) ?)

Second, it won’t find data in databases, which is probably where it is likely to be stored. It’s really only useful against flat-files.

Finally, there’s currently no facility to save this change when upgrading, such as you can already do with local_rules.xml. Make sure you have a backup of system_audit_rcl.txt handy that you can restore.

Despite these shortcomings, if you are already using OSSEC then this can be an excellent way to quickly add some rudimentary data checking to the OSSEC scans. It shouldn’t be relied on exclusively and the absense of an alert doesn’t necessarily mean you’re safe, but for a 5-minute change and in the absence of a budget for a more robust tool, this might lead to quite a surprise!

Thanks to Daniel Cid for help in getting this working, and of course for the excellent tool that OSSEC is.

OSSEC 2.3 is out. It has some interesting new features such as the ability to run a command and capture the output like a log. This really opens up possibilities. I’d also like to get some time to explore how this can possibly be abused. We’ll see.

The Dovecot support I added awhile back is in this release, as well as a few miscellaneous rules and bug reports. All in all, things seems to be pretty stable. Get it here. and don’t forget to thank Daniel for all the hard work!

On October 29, I gave a presentation at the Rochester Security Summit entitled “OSSEC in the Enterprise.” After a brief delay as a courtesy to the summit organizers, I’m pleased to be able to share the presentation with everyone else.

Daniel Cid over at ossec.net has graciously offered to provide hosting for the presentation. It can be accessed directly here, or for a list that includes other great OSSEC presentations, go here.

It’s a bit long at well over 100 slides, but it goes fast. I hope you enjoy it and would welcome any feedback you might have.

Along with my recent OSSEC presentation at the Rochester Security Summit, I helped out the SPARSA guys with the Capture the Flag contest. To demonstrate the capabilities of OSSEC, I created a virtual machine running OSSEC and Snort. It had two virtual interfaces: one ran in promiscuous mode and the other was for management and OSSEC.  The virtual switch was configured to mirror the traffic to the interface I was monitoring with Snort and OSSEC monitored not only the agents but also the Snort logs. Everything was virtual. It worked very well.

One snafu we ran into was with the OSSEC keys. The SPARSA guys had already downloaded the Windows OSSEC agent to the Windows machines, but we saved the key stuff until just before it was to begin.

To keep everyone safe, the virtual network had no Internet access. The Windows machines had no way to communicate with the OSSEC server other than through SSH, and we didn’t have PuTTY available.

After trying in vain to manually enter a couple of keys, one of the SPARSA guys had a great idea: how about netcat?

We set up a netcat listener on the OSSEC box to serve the client.keys file over port 80, then connected with IE from the Windows boxes.  I believe the command was: nc -l -k 80 < etc/client.keys. After serving up the keys with IE, we created the client.keys file on the Windows boxes in the ossec-agent directory and were on our way!

Although not a secure way of doing things, it presented no problem on a closed network which was designed to be exploited, and solved a real problem.