• The Shadowserver foundation is comprised of a group of volunteer security professionals who gather information about Internet-based crime. One of the more interesting projects is a compilation of how various antivirus vendors fare against 0-day threats. How does your vendor hold up?
• Logs are not much good if you can’t trust them. Maintaining log integrity is vital to a robust incident response process. Here is a great article on how to protect your logs from tampering. It’s not fool-proof, but it can go a long way.
• Information security is a profession that necessitates a solid ethical foundation. Security professionals are often trusted with the most sensitive of data. This presentation, from the Honeynet Project, tackles some of the more thorny situations about performing ethical research.
• Looking for a really awesome way to store and compare your Cisco configs? Rancid, or the Really Awesome New Cisco confIg Differ, may be just the tool for you. It stores Cisco configs in CVS and can let you know something changed. By the way, OSSEC is also capable of something very similar.
• Are you looking to use virtualization in your PCI program? It can be done, but like most technologies, has to be approached carefully. This guide will show you some of the things that need to be considered.

That’s it for today. Have a great weekend!

Of course, with any new technologies there are risks. Sometimes the risks are completely new and sometimes they are the same risks we have been dealing with for years — just repackaged in a slightly different way.

When most people think of virtualization security, they think of things like breaking out of the guest machine onto the host. This would be a very serious attack if successful because it has the potential to leverage one attack into something that can compromise several hosts. Even malware is getting into the business. The Blue Pill proof-of-concept used a small hypervisor to intercept calls to and from the guest, thereby controlling it according to desires of the attacker.

These are real threats, but for most organizations they do not represent the most serious threats to virtualized hosts. I was reminded of this last week when I read about Jason Cornish, a former IT employee of drug maker Shionogi. Jason had a beef with his former employer and he took it out by logging in and deleting 88 virtualized hosts. That’s the functional equivalent of a tornado taking out a small data center. He was able to do this simply because of a password which was not changed upon his termination, thereby allowing him remote access.

What’s the lesson here? The point is that we cannot forget the fundamentals of information security when implementing new technologies. That means doing the non-sexy stuff like changing default passwords, revoking access immediately for terminated employees and sending logs to a centralized log server. In the virtual world, we might review the access levels of the virtual administrators to make sure they are appropriate, separate hosts of different sensitivities onto different hypervisors and get alerted when a virtual machine is added, deleted or moved.

We do need to keep an eye on evolving virtual security threats, but we also have to keep the other eye on the fundamentals. We need both.

The basic problem is that Windows loads the full log into a shared memory space, and if it’s too big, then logs will be silently dropped. That’s why it is very important to have a centralized logging solution with logs sent in real-time or pretty close to it.

So how does centralized logging affect the local retention settings? To know that, you have to make certain assumptions about log size, average number of events generated per day, and so-on. Let’s start with this quote from Microsoft’s Threats and Countermeasures Guide:

Although there is no simple equation to determine the best log size for a particular server, you can calculate a reasonable size. The average event takes up about 500 bytes within each log, and the log file sizes must be a multiple of 64 KB. If you can estimate the average number of events that are generated each day for each type of log in your organization, you can determine a good size for each type of log file.

For example, if your file server generates 5,000 events per day in its Security log and you want to ensure that you have at least 4 weeks of data available at all times, then you would want to configure the size of that log to about 70 MB. (500 bytes * 5000 events/day * 28 days = 70,000,000 bytes.)

70MB doesn’t sound so bad. It’s probably well within the architectural limits of this memory-mapped I/O thing, but even that may be too big.

A better way of calculating what the local log size should be is to consider the recovery time objective and recovery point objective of the centralized log server. That is, consider how long it might take you to get your log server back online in the event of a disaster, then base your local Windows logging settings on that.

For example, if your recovery point objective is three days, you want to make sure that the local Windows logs will collect for at least three days, but not much more than that. Ideally, you also want to make sure that the solution you’re using can forward the logs collected during the downtime, or have a high-availability solution as part of the plan.

Remember, the goal is to get the local log sizes as small as possible, not as large as possible. You want to make sure you’re not losing any logs, and the two best ways to achieve that are to use centralized logging and try to avoid maxing out the services.exe process. This is counter-intuitive, but when you think it through, you’ll realize that it makes sense.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!

The Dovecot support I added awhile back is in this release, as well as a few miscellaneous rules and bug reports. All in all, things seems to be pretty stable. Get it here. and don’t forget to thank Daniel for all the hard work!

The arguments usually are something along the lines of “performance,” “it conflicts with x MTA” or “the vendor won’t support it.” Let’s look at each of these:

1. Performance: There’s no question that having AV running on your system can affect performance. It can affect performance on any system if the proper exclusions aren’t made. To use the argument that it shouldn’t be installed at all because of performance reasons is to argue that it shouldn”t be installed on any system. After all, everybody is concerned about performance.
2. It conflicts with my MTA: That’s what vendor-recommended exclusions are for. Of course, you don’t want your AV scanning your 500GB mail data file. Exclude it. But don’t go crazy and exclude entire partitions. Have justifications for your exclusions and use a scalpel, not a sledgehammer.
3. The vendor won’t support it: If the MTA vendor says that file system AV cannot be installed, find a different vendor. Seriously. They are taking the easy way out and putting your users at risk. Kick them to the curb and let them know why. Bad security choices by vendors need to have consequences for them to realize that they need to do better.

A worm spreading through administrative shares will be just as happy to devour your mail server as it will any other vulnerable Windows system. Mail servers generally need both file system and MTA-level AV for full protection. Anything less on a Windows system is risky behavior and may lead to very unhappy users.

While knowing that a file changed is useful, how much inherent value is there in simple file hash reports and alerts? In my opinion, very little.

These old tools don’t provide any context about the change. They don’t tell you who changed it, what happened just before and after the change, and what in the file actually changed. At best, you’re left with a bewildered goose-chase trying to figure out what happened.

To make matters worse, a well administered systems is supposed to change. If critical system files aren’t changing, you’re probably not patching. And if you are patching, you’re getting these alerts all the time and will eventually ignore them. The more systems you have, the worse the problem is.

Does that mean there is no value at all in integrity checks? Of course not. The value lies in two things:

1. Getting alerts for rare but important events. Knowing about a file added to the init.d directory or the Windows run key in the registry might be valuable.
2. Having the checksums available on a secured system for forensics.

Security is always a balancing act. Getting flooded with information about low value events is just as bad, if not worse, than not getting any alerts. Alerts should be meaningful and should, at the very least, cause you to stand up and take notice of something that isn’t normal.

Addendum: Complete IDS systems are also sometimes lacking in context. It probably took three or four tries just to post this due to content in the post that the IDS thought was a web attack. :)

OSSEC detects the invalid logins, but a cursory look at the length of time between each failed attempt reveals that it is not likely to trip the “SSHD brute force” rule (ID 5712). This means that while OSSEC will detect this, the threshold is low enough where you may not get an e-mail alert. This makes sense, since most people would not want an alert for just a couple of failed logins every minute or two.

So what can we do to defend against these types of attacks?

• Well, for starters, the basics of choosing good passwords apply. There is no vulnerability being exploited here other than poor password practices. This is in effect a people problem. Choose stong passwords and audit them regularly.
• It’s also a poor practice to login as root directly, especially thorugh SSH. Adding ‘PermitRootLogin no’ to sshd_config will prevent exploitation through SSH even with a poorly chosen password.
• Invalidate the password for root by locking the account, or just placing something like ‘*’ in the password field of the shadow file. Then give your non-privileged account access through sudoers. To attain root, simply type ‘sudo su -’ and the password of your non-privileged account.
• “Tune down,” then “tune up.” Tune your IDS to the point at which the alerts you’re getting are meaningful and need to be seen in near real-time. After you have done that (which could take several months in a large environment), use the GUI interface of the tool to see what it thinks isn’t important enough to alert you on. The tool’s idea of importance may be different than your own.

Exploitation from attacks like this are entirely preventable. Using layers of security mean that even if one defense fails, another steps in to hopefully take its place. The key is to implement the layers before the system is placed into production, then manage it well throughout its life.