Immutable Security

Awhile back, I posted about how certain versions of Windows always have the capability to lose logs. I encourage you to read the full post to understand the issues involved, then come back here and continue reading.

The basic problem is that Windows loads the full log into a shared memory space, and if it’s too big, then logs will be silently dropped. That’s why it is very important to have a centralized logging solution with logs sent in real-time or pretty close to it.

So how does centralized logging affect the local retention settings? To know that, you have to make certain assumptions about log size, average number of events generated per day, and so-on. Let’s start with this quote from Microsoft’s Threats and Countermeasures Guide:

Although there is no simple equation to determine the best log size for a particular server, you can calculate a reasonable size. The average event takes up about 500 bytes within each log, and the log file sizes must be a multiple of 64 KB. If you can estimate the average number of events that are generated each day for each type of log in your organization, you can determine a good size for each type of log file.

For example, if your file server generates 5,000 events per day in its Security log and you want to ensure that you have at least 4 weeks of data available at all times, then you would want to configure the size of that log to about 70 MB. (500 bytes * 5000 events/day * 28 days = 70,000,000 bytes.)

70MB doesn’t sound so bad. It’s probably well within the architectural limits of this memory-mapped I/O thing, but even that may be too big.

A better way of calculating what the local log size should be is to consider the recovery time objective and recovery point objective of the centralized log server. That is, consider how long it might take you to get your log server back online in the event of a disaster, then base your local Windows logging settings on that.

For example, if your recovery point objective is three days, you want to make sure that the local Windows logs will collect for at least three days, but not much more than that. Ideally, you also want to make sure that the solution you’re using can forward the logs collected during the downtime, or have a high-availability solution as part of the plan.

Remember, the goal is to get the local log sizes as small as possible, not as large as possible. You want to make sure you’re not losing any logs, and the two best ways to achieve that are to use centralized logging and try to avoid maxing out the services.exe process. This is counter-intuitive, but when you think it through, you’ll realize that it makes sense.

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!

OSSEC 2.3 is out. It has some interesting new features such as the ability to run a command and capture the output like a log. This really opens up possibilities. I’d also like to get some time to explore how this can possibly be abused. We’ll see.

The Dovecot support I added awhile back is in this release, as well as a few miscellaneous rules and bug reports. All in all, things seems to be pretty stable. Get it here. and don’t forget to thank Daniel for all the hard work!

A conversation with a McAfee engineer last week reminded me of something I have occasionally encountered over the years: Windows mail server admins who insist that file system AV isn’t necessary on their server. The logic for this has always escaped me and no mail server admin I have discussed this with can provide me with a reasonably sound argument of why AV shouldn’t be installed.

The arguments usually are something along the lines of “performance,” “it conflicts with x MTA” or “the vendor won’t support it.” Let’s look at each of these:

1. Performance: There’s no question that having AV running on your system can affect performance. It can affect performance on any system if the proper exclusions aren’t made. To use the argument that it shouldn’t be installed at all because of performance reasons is to argue that it shouldn”t be installed on any system. After all, everybody is concerned about performance.
2. It conflicts with my MTA: That’s what vendor-recommended exclusions are for. Of course, you don’t want your AV scanning your 500GB mail data file. Exclude it. But don’t go crazy and exclude entire partitions. Have justifications for your exclusions and use a scalpel, not a sledgehammer.
3. The vendor won’t support it: If the MTA vendor says that file system AV cannot be installed, find a different vendor. Seriously. They are taking the easy way out and putting your users at risk. Kick them to the curb and let them know why. Bad security choices by vendors need to have consequences for them to realize that they need to do better.

A worm spreading through administrative shares will be just as happy to devour your mail server as it will any other vulnerable Windows system. Mail servers generally need both file system and MTA-level AV for full protection. Anything less on a Windows system is risky behavior and may lead to very unhappy users.

Tools like AIDE and the original Tripwire have long been used in the Unix world for so-called intrusion detection. They let you know when a file has changed by presenting you with an old and new checksum, but usually don’t provide much more info than that.

While knowing that a file changed is useful, how much inherent value is there in simple file hash reports and alerts? In my opinion, very little.

These old tools don’t provide any context about the change. They don’t tell you who changed it, what happened just before and after the change, and what in the file actually changed. At best, you’re left with a bewildered goose-chase trying to figure out what happened.

To make matters worse, a well administered systems is supposed to change. If critical system files aren’t changing, you’re probably not patching. And if you are patching, you’re getting these alerts all the time and will eventually ignore them. The more systems you have, the worse the problem is.

Does that mean there is no value at all in integrity checks? Of course not. The value lies in two things:

1. Getting alerts for rare but important events. Knowing about a file added to the init.d directory or the Windows run key in the registry might be valuable.
2. Having the checksums available on a secured system for forensics.

Security is always a balancing act. Getting flooded with information about low value events is just as bad, if not worse, than not getting any alerts. Alerts should be meaningful and should, at the very least, cause you to stand up and take notice of something that isn’t normal.

Addendum: Complete IDS systems are also sometimes lacking in context. It probably took three or four tries just to post this due to content in the post that the IDS thought was a web attack. :)

Peter Handsteen, a.k.a “That Grumpy BSD Guy” recently observed a round of low and slow SSH password guessing attacks against his server. For those not familiar with the term “low and slow” this is an attack meant to be slow enough that IDS systems aren’t tripped but fast enough where the attacker has a decent chance of getting access. The interesting thing about this attack is that it is coming from several different sources and is likely a botnet design just for this purpose. There is about one attempt per minute against the root account. While this doesn’t sound like much, it’s enough for an attacker to cycle through 1440 common passwords in a single day. And since each few attempts come from a different source, it would be difficult to block at the network level. Admins who use passwords like root, test, admin and password are likely to fall victim.

OSSEC detects the invalid logins, but a cursory look at the length of time between each failed attempt reveals that it is not likely to trip the “SSHD brute force” rule (ID 5712). This means that while OSSEC will detect this, the threshold is low enough where you may not get an e-mail alert. This makes sense, since most people would not want an alert for just a couple of failed logins every minute or two.

So what can we do to defend against these types of attacks?

• Well, for starters, the basics of choosing good passwords apply. There is no vulnerability being exploited here other than poor password practices. This is in effect a people problem. Choose stong passwords and audit them regularly.
• It’s also a poor practice to login as root directly, especially thorugh SSH. Adding ‘PermitRootLogin no’ to sshd_config will prevent exploitation through SSH even with a poorly chosen password.
• Invalidate the password for root by locking the account, or just placing something like ‘*’ in the password field of the shadow file. Then give your non-privileged account access through sudoers. To attain root, simply type ’sudo su -’ and the password of your non-privileged account.
• “Tune down,” then “tune up.” Tune your IDS to the point at which the alerts you’re getting are meaningful and need to be seen in near real-time. After you have done that (which could take several months in a large environment), use the GUI interface of the tool to see what it thinks isn’t important enough to alert you on. The tool’s idea of importance may be different than your own.

Exploitation from attacks like this are entirely preventable. Using layers of security mean that even if one defense fails, another steps in to hopefully take its place. The key is to implement the layers before the system is placed into production, then manage it well throughout its life.

Prior to Windows Vista, Microsoft’s default posture in Windows was to let the user have admin rights. That meant that not only did the user have admin rights, but so did the malware when it took advantage of one of the several juicy Windows vulnerabilities.

Doing your daily computing as an administrative user is dangerous, but running without admin can also be problematic, since many poorly written Windows applications assume the user has administrator rights. Using browsers, e-mail clients and instant messaging applications as admin is just asking for trouble.

Michael Howard from Microsoft saught to find a middle ground where one could stay logged in as admin but run certain applications in a non-admin context. The result is DropMyRights. Michael describes DropMyRights:

DropMyRights is a very simple application to help users who must run as an administrator run applications in a much-safer context—that of a non-administrator. It does this by taking the current user’s token, removing various privileges and SIDs from the token, and then using that token to start another process, such as Internet Explorer or Outlook. This tool works just as well with Mozilla’s Firefox, Eudora, or Lotus Notes e-mail.

If you’re still using a version of Windows prior to Vista, DropMyRights is a worthwhile tool for your desktop security.

And for those running Linux, none of this need apply. You’re already running as a non-privileged user.

One of the big problems in information security is how to effectively teach small businesses safe data handling. They’re too small to have dedicated security budgets and they can’t be expected to publish volumes of security policies; yet, they have needs to handle information safely above and beyond what a normal consumer has to deal with.

NIST attempts to fill this gap with the Small Business Information Security: The Fundamentals guide. In the guide they detail what a small business should minimally be concerned with, along with some extra measures they may want to take.

While it has a little ways to go (it is a draft, after all), it’s a great start to filling this much needed void. Check it out and see if they manage to walk the fine line by making security simple, yet effective enough for small business.