Immutable Security

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.

OSSEC is one of those tools that continues to surprise me with its ability to perform low-level and important security tasks. In fine Unix tradition, individual parts of OSSEC can often be persuaded to do your bidding in ways not previously conceived. Put another way, it’s nice to hack. :)

Today, I’m going to show you how to use OSSEC to detect sensitive info within flat files. This example will demonstrate American social security numbers, but the same logic could be used to detect clear-text credit card numbers (a violation of the PCI DSS standard), or other personally identifiable information which should not be in clear-text.

We’re going to rely on the rootcheck functionality of OSSEC. Rootcheck does all kinds of nifty things, one of which is to scan within files for patterns. Normally, you would expect to look for things like insecure system configuration settings. We’re going to use it to find possible social security numbers.

First, create a test file containing a properly formatted SSN in a location that OSSEC is monitoring:

echo 123-45-6789 > /var/www/html/example.com/www/customer_data

Next, add the following to /var/ossec/etc/shared/system_audit_rcl.txt:

# Detect possible SSNs
[Possible Unencrypted Social Security Number Detected] [any] []
d:$web_dirs -> r:^\. -> r:\d\d\d-\d\d-\d\d\d\d;

Now, let’s create an alert so we get a heads up on what OSSEC finds (change the rule ID as needed):

<rule id=”100024″ level=”12″>
<if_sid>516</if_sid>
<match>Unencrypted Social Security Number</match>
<description>Possible Unencrypted Social Security Number Detected</description>
</rule>

Restart OSSEC:

/var/ossec/bin/ossec-control restart

Finally, we’ll initiate a scan on the local system:

/var/ossec/bin/agent_control -r -u 000

And here is what the resulting alert looks like:

OSSEC HIDS Notification.

2010 Jan 12 18:25:43
Received From: hostname->rootcheck
Rule: 100024 fired (level 12) -> “Possible Unencrypted Social Security Number Detected”
Portion of the log(s):

System Audit: Possible Unencrypted Social Security Number Detected. File: /var/www/html/example.com/www/customer_data.

–END OF NOTIFICATION

One of the nice things about the alert above is that it doesn’t actually include the contents of the file, so you don’t have to worry about clear-text data traversing your mail server.

There are also a few caveats to keep in mind.

First, it is slightly prone to false-positives. Since the OSSEC regex library is designed for speed, rather than full regex support, it’s not possible to use something like \b to define a word boundary. This means that while the regex will match SSNs like 123-45-6789, it would also match something like 00123-45-678911. It also wouldn’t match a delimiter other than the dash. You might be able to mitigate some of these risks if you’re sure how the data would be formatted (but then again, if you’re sure you have clear-text data on the server why do you need this :) ?)

Second, it won’t find data in databases, which is probably where it is likely to be stored. It’s really only useful against flat-files.

Finally, there’s currently no facility to save this change when upgrading, such as you can already do with local_rules.xml. Make sure you have a backup of system_audit_rcl.txt handy that you can restore.

Despite these shortcomings, if you are already using OSSEC then this can be an excellent way to quickly add some rudimentary data checking to the OSSEC scans. It shouldn’t be relied on exclusively and the absense of an alert doesn’t necessarily mean you’re safe, but for a 5-minute change and in the absence of a budget for a more robust tool, this might lead to quite a surprise!

Thanks to Daniel Cid for help in getting this working, and of course for the excellent tool that OSSEC is.

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!

OSSEC 2.3 is out. It has some interesting new features such as the ability to run a command and capture the output like a log. This really opens up possibilities. I’d also like to get some time to explore how this can possibly be abused. We’ll see.

The Dovecot support I added awhile back is in this release, as well as a few miscellaneous rules and bug reports. All in all, things seems to be pretty stable. Get it here. and don’t forget to thank Daniel for all the hard work!

Pop quiz: how many places can an application install itself to ensure it will survive a Windows reboot? If you named the usual suspects like the run and autorun locations in the registry, you would be correct, but have mentioned only a small subsection of the many places. I’m going to resist the urge to rant about how insane it is to have so many possible places for an application (i.e. trojan) to ensure its survivial. Instead, I’ll just give you the most comprehensive list I have come across, courtesy of the Autoruns utility from Microsoft (formerly SysInternals):

C:\Documents and Settings\All Users\Start Menu\Programs\Startup

C:\WINDOWS\Tasks

HKLM\System\CurrentControlSet\Control\Terminal Server\Wds\rdpwd\StartupPrograms

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Userinit

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run

HKCU\Software\Microsoft\Windows\CurrentVersion\Run

HKLM\SOFTWARE\Classes\Protocols\Filter

HKLM\SOFTWARE\Classes\Protocols\Handler

HKCU\SOFTWARE\Microsoft\Internet Explorer\Desktop\Components

HKLM\SOFTWARE\Microsoft\Active Setup\Installed Components

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\SharedTaskScheduler

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\ShellServiceObjectDelayLoad

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellExecuteHooks

HKLM\Software\Classes\*\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\AllFileSystemObjects\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\Shellex\DragDropHandlers

HKLM\Software\Classes\Directory\Shellex\PropertySheetHandlers

HKLM\Software\Classes\Directory\Shellex\CopyHookHandlers

HKLM\Software\Classes\Folder\Shellex\ColumnHandlers

HKLM\Software\Classes\Folder\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\Background\ShellEx\ContextMenuHandlers

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers

HKLM\Software\Microsoft\Windows\CurrentVersion\Shell Extensions\Approved

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\Browser Helper Objects

HKCU\Software\Microsoft\Internet Explorer\UrlSearchHooks

HKLM\Software\Microsoft\Internet Explorer\Toolbar

HKLM\Software\Microsoft\Internet Explorer\Extensions

HKLM\System\CurrentControlSet\Services

HKLM\System\CurrentControlSet\Services

HKLM\System\CurrentControlSet\Control\Session Manager\BootExecute

HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options

HKLM\System\CurrentControlSet\Control\Session Manager\KnownDlls

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\UIHost

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Notify

HKCU\Control Panel\Desktop\Scrnsave.exe

HKLM\System\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9\Catalog_Entries

HKLM\System\CurrentControlSet\Services\WinSock2\Parameters\NameSpace_Catalog5\Catalog_Entries

HKLM\SYSTEM\CurrentControlSet\Control\Print\Monitors

HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SecurityProviders

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Authentication Packages

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Notification Packages

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Security Packages

HKLM\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order

A conversation with a McAfee engineer last week reminded me of something I have occasionally encountered over the years: Windows mail server admins who insist that file system AV isn’t necessary on their server. The logic for this has always escaped me and no mail server admin I have discussed this with can provide me with a reasonably sound argument of why AV shouldn’t be installed.

The arguments usually are something along the lines of “performance,” “it conflicts with x MTA” or “the vendor won’t support it.” Let’s look at each of these:

1. Performance: There’s no question that having AV running on your system can affect performance. It can affect performance on any system if the proper exclusions aren’t made. To use the argument that it shouldn’t be installed at all because of performance reasons is to argue that it shouldn”t be installed on any system. After all, everybody is concerned about performance.
2. It conflicts with my MTA: That’s what vendor-recommended exclusions are for. Of course, you don’t want your AV scanning your 500GB mail data file. Exclude it. But don’t go crazy and exclude entire partitions. Have justifications for your exclusions and use a scalpel, not a sledgehammer.
3. The vendor won’t support it: If the MTA vendor says that file system AV cannot be installed, find a different vendor. Seriously. They are taking the easy way out and putting your users at risk. Kick them to the curb and let them know why. Bad security choices by vendors need to have consequences for them to realize that they need to do better.

A worm spreading through administrative shares will be just as happy to devour your mail server as it will any other vulnerable Windows system. Mail servers generally need both file system and MTA-level AV for full protection. Anything less on a Windows system is risky behavior and may lead to very unhappy users.

Peter Handsteen, a.k.a “That Grumpy BSD Guy” recently observed a round of low and slow SSH password guessing attacks against his server. For those not familiar with the term “low and slow” this is an attack meant to be slow enough that IDS systems aren’t tripped but fast enough where the attacker has a decent chance of getting access. The interesting thing about this attack is that it is coming from several different sources and is likely a botnet design just for this purpose. There is about one attempt per minute against the root account. While this doesn’t sound like much, it’s enough for an attacker to cycle through 1440 common passwords in a single day. And since each few attempts come from a different source, it would be difficult to block at the network level. Admins who use passwords like root, test, admin and password are likely to fall victim.

OSSEC detects the invalid logins, but a cursory look at the length of time between each failed attempt reveals that it is not likely to trip the “SSHD brute force” rule (ID 5712). This means that while OSSEC will detect this, the threshold is low enough where you may not get an e-mail alert. This makes sense, since most people would not want an alert for just a couple of failed logins every minute or two.

So what can we do to defend against these types of attacks?

• Well, for starters, the basics of choosing good passwords apply. There is no vulnerability being exploited here other than poor password practices. This is in effect a people problem. Choose stong passwords and audit them regularly.
• It’s also a poor practice to login as root directly, especially thorugh SSH. Adding ‘PermitRootLogin no’ to sshd_config will prevent exploitation through SSH even with a poorly chosen password.
• Invalidate the password for root by locking the account, or just placing something like ‘*’ in the password field of the shadow file. Then give your non-privileged account access through sudoers. To attain root, simply type ‘sudo su -’ and the password of your non-privileged account.
• “Tune down,” then “tune up.” Tune your IDS to the point at which the alerts you’re getting are meaningful and need to be seen in near real-time. After you have done that (which could take several months in a large environment), use the GUI interface of the tool to see what it thinks isn’t important enough to alert you on. The tool’s idea of importance may be different than your own.

Exploitation from attacks like this are entirely preventable. Using layers of security mean that even if one defense fails, another steps in to hopefully take its place. The key is to implement the layers before the system is placed into production, then manage it well throughout its life.

Although I have yet to get any presents, they tell me that October is National Cyber Security Awareness Month. Where can you get in on the festivities?

• The Obama administration has declared it, according to Network Security News. Although this is a bit like Al Gore declaring that he invented the Intenet. October has been Cyber Security Month for awhile, Mr. President.
• Microsoft reminds you to stay patched, turn on your firewall and use antivirus. That’s got to be good for a warm fuzzy, at least.
• The Department of Homeland Security (aka the “Mess with us and we’ll send your sorry behind to Gitmo” people) remind us that this is the sixth annual event. Someone from DHS should call Obama to let him know they already have dibs.
• Finally, (or it could be that I just stopped following links) Educause has links to some forms, posters and an event calendar one can use at their NCSAM parties. If only they had these when I was a kid. Who needs Twisted Sister posters when we can espouse the virtues of safe computing?

Here’s wishing you safe computing!

Prior to Windows Vista, Microsoft’s default posture in Windows was to let the user have admin rights. That meant that not only did the user have admin rights, but so did the malware when it took advantage of one of the several juicy Windows vulnerabilities.

Doing your daily computing as an administrative user is dangerous, but running without admin can also be problematic, since many poorly written Windows applications assume the user has administrator rights. Using browsers, e-mail clients and instant messaging applications as admin is just asking for trouble.

Michael Howard from Microsoft saught to find a middle ground where one could stay logged in as admin but run certain applications in a non-admin context. The result is DropMyRights. Michael describes DropMyRights:

DropMyRights is a very simple application to help users who must run as an administrator run applications in a much-safer context—that of a non-administrator. It does this by taking the current user’s token, removing various privileges and SIDs from the token, and then using that token to start another process, such as Internet Explorer or Outlook. This tool works just as well with Mozilla’s Firefox, Eudora, or Lotus Notes e-mail.

If you’re still using a version of Windows prior to Vista, DropMyRights is a worthwhile tool for your desktop security.

And for those running Linux, none of this need apply. You’re already running as a non-privileged user.

If you could only choose one security measure to implement, what would it be? To me, the answer is simple: backup.

Without backup, you may not be able to recover at all from a computer compromise, virus incident, accidentally deleted file, or hard drive melt down. It may mean the end of your business, the end of your livelihood and even by extension, losing close relationships due to the resulting stress.

Fortunately, we have a plethora of protections to choose from, but if you’re ignoring backup then everything else may be moot.

If you have backup issues in your company, suggest that it is an information security risk of the highest order. You may get some strange looks, but if you educate people that protecting information involves more than just antivirus and firewalls, they may see the wisdom in your words.

Backup often and test restores. Anything less is playing with fire.