Immutable Security

Most people that use the OSSEC Windows agent download a pre-compiled copy from the OSSEC site. While that is a good option for many individual users, it may not suit those with more specific needs and/or those in enterprise environments. Users who fall into those categories could benefit from customizing the agent and maintaining internal builds in order to suit their individual needs.

There are already instructions on how to compile the Windows agent on Linux, but ironically the process doesn’t work so well on Windows. I had a need to make this work on Windows, so I thought I would share the process with you.

First, there are some prerequisites.  You’ll need:

• The Nullsoft Scriptable Install System (NSIS)
• The Minimalist GNU for Windows compiler (MinGW)
• My gen_win batch file.  Simply rename from gen_win.txt to gen_win.cmd.
• 7-Zip for Windows
• The public domain Unix2DOS utility
• The latest OSSEC for Unix/Linux (this contains the Windows source code)

Here are the steps:

1. Download and install the required programs. Be sure to pay special attention to the steps for properly installing and configuring MinGW, particularly the part about modifying the PATH environment variable.
2. Next, we’re going to extract OSSEC using 7-Zip. To do so, simply right-click on the file and select 7-Zip, extract to “folder name.tar,” where folder name is the name of the package. This decompresses the archive. Navigate within that folder and repeat this step to untar the archive. At this point, you should see all of the files in the package.
3. Place gen_win.txt in the src\win32 folder and rename the extension to .cmd.
4. Download Unix2DOS and place it in the src\win32 folder
5. Open a command prompt. Navigate to src\win32, make any desired customizations, and execute gen_win.cmd. This should gather all of the required files and place them in src\win-pkg.
6. Next, we compile the Windows agent by navigating to src\win-pkg and executing make.bat (I assume you have the chops to know how to change directories :) ).
7. Now we have all of the files we need but no way to effectively install it. To generate the installer, simply execute the NSIS compiler like so: “c:\Program Files\NSIS\makensis.exe” ossec-installer.nsi

If you see no errors and a binary named ossec-win32-agent.exe, everything was successful. Congratulations, you now have a custom-made version of OSSEC!

For those that did not see me at the Rochester Security Summit last year, and who would like to see me present my OSSEC in the Enterprise presentation, I will be giving it again at the ISSA Ft. Worth chapter meeting, next Wednesday. Details are as follows:

Meeting Location: Devry University, 301 W. Commerce Street, Fort Worth, TX 76102

Phone: 817-810-9114

Parking is up to you, there is a parking garage, lots & meters.

FWT Consulting will be sponsoring the meeting & providing lunch, so please RSVP to steve.streiffert <a t> gmail.com by Monday, May 17, 2010.

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

Letting people go is rarely a feel-good experience. Even if everything goes smoothly, there are always human emotions involved, and those emotions make some people unpredictable.  It’s important, therefore, to make sure extra attention and detail is paid to certain areas before and during a layoff. Here are five things to watch out for in your logs and monitoring tools.

1. Outbound connections to a single IP. If you have the benefit of knowing what normal network traffic patterns look like, watch for variations and patterns that may indicate a back-door connection being initiated from within the network. Does outbound traffic on one port spike at around 5:00 PM and stay that way for an extended period of time? Does it look encrypted? Is it using a common port like 53, normally reserved for DNS, but does not seem to be DNS traffic? These could all be indications of a back-door or unauthorized remote access channel.
2. New and changed scheduled tasks/cron jobs. Pay special attention to scheduled tasks and cron jobs and the scripts they run. Monitor changes to those scripts. Look for new and changed tasks and cron jobs. For Windows tasks, refer to IDs 602, 4698, 4699, 4700, 4701 and 4702. For ‘nix, look for changes in the cron log. Scheduled tasks and cron jobs are often used to plant logic bombs, which can execute even months after an employee has left.
3. E-mail to competitors and media. E-mail going to domains which are not considered “friendly” to the organization may indicate that someone has a hatchet to bury. In particular, if there are several of these organizations listed as recipients in the same e-mail, that could be someone trying to sling mud or disseminate confidential information.
4. Data leakage. Let’s get one thing straight. If someone wants to steal data, chances are they will be able to do so undetected if they are smart enough. Even so, people don’t always take the time to cover their tracks. Some things to check: proxy and mail gateway logs for interesting content types (i.e. attachments), USB drives being plugged in, and the good ol’ physical equipment walking out the door.
5. New and Changed Accounts. Some people aren’t too good at letting go. They may try to maintain access into the environment. This may involve either a password change or a new account. People know their account may be disabled, so they may try to create a new, innocuous looking account to maintain access. Perhaps they will try to use a service account with a well-known password. Whatever the case, pay special attention to accounts.

Perhaps the best thing to do is to level with your staff as much as is possible. Treating people with respect and dignity could go much further than any technical measure you can dream up.

Many years ago, after I had been using OSSEC in an enterprise setting for a few months, I noticed an interesting phenomenon. Administrators, many of whom I had forwarded “was this you?” alerts to, were now coming to me to rat on themselves.

I would be working away in my cubicle when someone would come up behind me. It went something like this:

“Hey, Mike. Just wanted to let you know what you’ll be seeing me <insert action here> in the logs. No cause for concern.”

“Thanks for the heads up,” I would reply.

The administrators knew they were being monitored, but didn’t exactly know the full details of the monitoring. They naturally assumed I would see what they were up to. In many cases I wouldn’t have known.

The vast majority of these folks were honest to begin with, but I can’t help think it assisted them in following process and being just a bit more transparent with what they were doing. Maybe it even dissuaded someone who was on the fence from doing something vindictive.

After seeing this in other environments, I think it deserves a term. At the risk of coining a term for something that has already been identified, I hereby declare this the “OSSEC Effect.” The definition (which could use some refinement) is as follows:

OSSEC Effect: The alteration of a computer user’s behavior when they know their actions are being monitored, but do not realize or understand the extent of the monitoring. Users will, without provocation, volunteer information they believe could be seen as questionable, whether the monitoring system would have known about it or not.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

Awhile back I signed up for the CouponMom.com newsletter (hey, who doesn’t like to save a few bucks), using a very unique and distinctive e-mail address used only for this purpose.

Awhile later, I started to get garden variety spam to this e-mail address (Viagra, etc).

There are a few reasons this could happen:

1. I have been compromised and the spammers think it would be clever to use that address.
2. Spammers start spamming that address as a matter of chance or because they think, “hey, this guy likes Coupon Mom, maybe he’ll like some male enhancement!”
3. Coupon Mom is supplementing her income by spamming or selling the data, which makes its way into spammer’s hands.
4. Coupon Mom has been compromised.

Usually, when this happens, it’s number 4.

I got to thinking, “hey, they might want to know there might be a problem. I should tell them.”

I fill out their contact form and wait. More than a week goes by with no response.

I try to post a cautionary word to the forum. More than a week goes by and I don’t pass moderation.

I fill out the form again, indicating that it would be better for them to investigate this and notify their members of a breach, if one happened, than it would be for me to speculate about it.

Finally, I get a response. The response, in part, states:

You must have signed up for a Google advertiser link on the site, since the email signups for my site are not shared with any other party.

I am sorry you have had this experience, but caution you against publicly slandering The Coupon Mom program and our member database as the source of the unsolicited email.

Can I say that the Coupon Mom database has been breached? Categorically, no. But I can say that there are symptoms which, in my opinion, should cause a reasonable person to take a closer look.

What’s the lesson here? When someone tells you of a potential problem with your security, don’t just assume you are impenetrable. That person may serve as an early warning of a serious problem you would want to be on top of.

Here’s a little secret that the sales guys of the million-dollar SIEMs are probably going to gloss over. Most of them do not offer a way to encrypt logs in transit end-to-end. Worse, many of them use downright silly methods to collect logs on common operating systems, such as installing an FTP client on every host and sending them along as a big batch every so often.

For some reason, this is seen as acceptable since it’s usually on the internal network. At the same time, we give mean looks to anyone who even mentions the use of a clear-text protocol like telnet.

Here’s another little secret: many of the big SIEM vendors use syslog-ng as the log daemon on their appliances. It’s reliable and secure and generally works very well.

What if there was a way to add encrypted log transit to the expensive SIEM, or to the other log management solution you already have in place? What if you can’t use or don’t need the full functionality of a log analysis solution like OSSEC, but would like the ability to use encrypted log transport? Read on, my friend, because I’m going to show you exactly how to do that, and it can be done in just a few minutes.

1. Install OSSEC in server mode on the existing syslog server from the vendor. I won’t go into detail here on how to install OSSEC, as it’s well documented elsewhere. These systems are generally just minimal Linux installations, so if you are lacking a compiler then you may need to install it on a like system and move OSSEC over. If your vendor won’t let you do this, ask them why you are paying them gobs of money without even the basic functionality of encrypted log transit!
2. Install OSSEC agents on the hosts you want to collect logs from. Configure the agents to collect the logs you are interested in and add them to the manager. Don’t forget to restart the OSSEC manager after adding the agents.
3. Add the following lines to /opt/syslog-ng/etc/syslog-ng.conf under the “source s_local” section:
   # Read OSSEC queue
   unix-dgram(”/var/ossec/queue/ossec/queue”);
4. Edit /var/ossec/etc/ossec-control and modify the two variables which are responsible for starting the OSSEC daemons. After modification, they should look like this:
   DAEMONS=”ossec-remoted”
   SDAEMONS=”ossec-remoted”
5. Finally, stop both syslog-ng and OSSEC. Start syslog-ng first, then start OSSEC.

The reason this works is due to the way OSSEC is designed. In fine Unix tradition, OSSEC separates the tasks into individual daemons that do only one job and do it well. In this case, ossec-remoted is writing to a Unix socket that syslog-ng is reading from in real-time. OSSEC is simply handing off the message to the local syslog daemon, at which point the message continues on its way as it normally would.

I used to use Snare, Zebedee and OSSEC for secure log delivery and file integrity. Now, it can all be handled with one open-source agent, already designed for secure and reliable log delivery.

There is a downside, which is primarily due to me not taking this to the next level and looking into the issue. In this configuration, you will not receive a notification if the OSSEC agent disconnects, since there is nothing to act on the absence of the heartbeat OSSEC sends. I’m pretty sure that can be solved by taking it up a notch and including one or two more daemons, combined with a stripped down decoder and rules file.

So the next time your expensive SIEM vendor tells you there’s no way to encrypt the logs in transit, tell them you can set it up in under an hour. Then send them a bill for your time.

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!