Immutable Security

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!

Pop quiz: how many places can an application install itself to ensure it will survive a Windows reboot? If you named the usual suspects like the run and autorun locations in the registry, you would be correct, but have mentioned only a small subsection of the many places. I’m going to resist the urge to rant about how insane it is to have so many possible places for an application (i.e. trojan) to ensure its survivial. Instead, I’ll just give you the most comprehensive list I have come across, courtesy of the Autoruns utility from Microsoft (formerly SysInternals):

C:\Documents and Settings\All Users\Start Menu\Programs\Startup

C:\WINDOWS\Tasks

HKLM\System\CurrentControlSet\Control\Terminal Server\Wds\rdpwd\StartupPrograms

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Userinit

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run

HKCU\Software\Microsoft\Windows\CurrentVersion\Run

HKLM\SOFTWARE\Classes\Protocols\Filter

HKLM\SOFTWARE\Classes\Protocols\Handler

HKCU\SOFTWARE\Microsoft\Internet Explorer\Desktop\Components

HKLM\SOFTWARE\Microsoft\Active Setup\Installed Components

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\SharedTaskScheduler

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\ShellServiceObjectDelayLoad

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellExecuteHooks

HKLM\Software\Classes\*\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\AllFileSystemObjects\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\Shellex\DragDropHandlers

HKLM\Software\Classes\Directory\Shellex\PropertySheetHandlers

HKLM\Software\Classes\Directory\Shellex\CopyHookHandlers

HKLM\Software\Classes\Folder\Shellex\ColumnHandlers

HKLM\Software\Classes\Folder\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\Background\ShellEx\ContextMenuHandlers

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers

HKLM\Software\Microsoft\Windows\CurrentVersion\Shell Extensions\Approved

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\Browser Helper Objects

HKCU\Software\Microsoft\Internet Explorer\UrlSearchHooks

HKLM\Software\Microsoft\Internet Explorer\Toolbar

HKLM\Software\Microsoft\Internet Explorer\Extensions

HKLM\System\CurrentControlSet\Services

HKLM\System\CurrentControlSet\Services

HKLM\System\CurrentControlSet\Control\Session Manager\BootExecute

HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options

HKLM\System\CurrentControlSet\Control\Session Manager\KnownDlls

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\UIHost

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Notify

HKCU\Control Panel\Desktop\Scrnsave.exe

HKLM\System\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9\Catalog_Entries

HKLM\System\CurrentControlSet\Services\WinSock2\Parameters\NameSpace_Catalog5\Catalog_Entries

HKLM\SYSTEM\CurrentControlSet\Control\Print\Monitors

HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SecurityProviders

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Authentication Packages

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Notification Packages

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Security Packages

HKLM\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order

Prior to Windows Vista, Microsoft’s default posture in Windows was to let the user have admin rights. That meant that not only did the user have admin rights, but so did the malware when it took advantage of one of the several juicy Windows vulnerabilities.

Doing your daily computing as an administrative user is dangerous, but running without admin can also be problematic, since many poorly written Windows applications assume the user has administrator rights. Using browsers, e-mail clients and instant messaging applications as admin is just asking for trouble.

Michael Howard from Microsoft saught to find a middle ground where one could stay logged in as admin but run certain applications in a non-admin context. The result is DropMyRights. Michael describes DropMyRights:

DropMyRights is a very simple application to help users who must run as an administrator run applications in a much-safer context—that of a non-administrator. It does this by taking the current user’s token, removing various privileges and SIDs from the token, and then using that token to start another process, such as Internet Explorer or Outlook. This tool works just as well with Mozilla’s Firefox, Eudora, or Lotus Notes e-mail.

If you’re still using a version of Windows prior to Vista, DropMyRights is a worthwhile tool for your desktop security.

And for those running Linux, none of this need apply. You’re already running as a non-privileged user.

Awhile back I bought a new Toshiba laptop. I got a great deal and it looked like it would serve my needs just fine. I knew exactly how I was going to use it.

When powering new computers on for the first time, most of them immediately lead you into the Windows installation routine. That makes sense because most people want Windows. But I didn’t. This laptop came with Vista, which from a previous experience playing with, I knew wasn’t for me. For kicks, I reviewed the license agreement. That was the nail in the coffin. I declined the EULA, rebooted to an Ubuntu installation CD and proceeded to send Vista to the bit-bucket.

I knew I couldn’t return Windows for a refund. Toshiba covered that nicely with a small notice on bright paper in the laptop box. But perhaps someone else can use the copy and I can get a few bucks back, I thought. Even thirty bucks would be enough for a casual dinner out.

I decided to look at the possibility of selling my copy on eBay. According to eBay’s policy on OEM software, I cannot sell it unless I also sell the original hardware the computer came with. Apparently, I can’t just ship the customer an old motherboard. Or if I do, it sounds like it would be a technical violation of the policy.

So, here I am. I have a copy of Vista I don’t want. I did not agree to the license terms and still, I can’t sell my copy of Windows (at least on eBay). Although I in no way agreed to Microsoft’s terms and conditions, including the condition that OEM software must be sold with hardware, I am still somehow bound by it.

If copyright law grants me the first-sale doctrine right and if I refuse to accept additional licensing terms from the copyright holder, how can the copyright holder assert any additional rights over what the law allows? Food for thought.