Immutable Security

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!

Awhile back I bought a new Toshiba laptop. I got a great deal and it looked like it would serve my needs just fine. I knew exactly how I was going to use it.

When powering new computers on for the first time, most of them immediately lead you into the Windows installation routine. That makes sense because most people want Windows. But I didn’t. This laptop came with Vista, which from a previous experience playing with, I knew wasn’t for me. For kicks, I reviewed the license agreement. That was the nail in the coffin. I declined the EULA, rebooted to an Ubuntu installation CD and proceeded to send Vista to the bit-bucket.

I knew I couldn’t return Windows for a refund. Toshiba covered that nicely with a small notice on bright paper in the laptop box. But perhaps someone else can use the copy and I can get a few bucks back, I thought. Even thirty bucks would be enough for a casual dinner out.

I decided to look at the possibility of selling my copy on eBay. According to eBay’s policy on OEM software, I cannot sell it unless I also sell the original hardware the computer came with. Apparently, I can’t just ship the customer an old motherboard. Or if I do, it sounds like it would be a technical violation of the policy.

So, here I am. I have a copy of Vista I don’t want. I did not agree to the license terms and still, I can’t sell my copy of Windows (at least on eBay). Although I in no way agreed to Microsoft’s terms and conditions, including the condition that OEM software must be sold with hardware, I am still somehow bound by it.

If copyright law grants me the first-sale doctrine right and if I refuse to accept additional licensing terms from the copyright holder, how can the copyright holder assert any additional rights over what the law allows? Food for thought.