Immutable Security

Logrotate is a fantastic little utility for, well, rotating logs. It has several options available. It can rotate by date, file size, set specific owners on rotated files, and it even gives you the capability to run pre and post-rotate scripts.

Getting it to work on a centralized log server, however, can be a bit tricky. The issue lies in the unique way these log servers are often configured. Rather than a static set of directories, log servers often dynamically create directories based on the hosts that are logging to them. It’s not uncommon to see a directory structure something along the lines of /mnt/logs/2010/May/18/host/messages. While logrotate can use wildcards in the directory structure definition, it gets a bit hairy when trying to define the rotated log directory.

There are two options for where to put the rotated logs. You can put them in another directory, or in the same directory as the rotated files. Another directory is usually the better option, as rotated logs in the same directory could be rotated again and again, unless you are careful with the way logs are created.

So what happens if the directory you specify for the rotated log files doesn’t exist (remember, the log server is creating them dynamically based on the hosts that report in)? In that case, logrotate will complain and refuse to rotate your logs.

By now you might be thinking, “Why, I’ll just use that handy pre-rotate option. Then I can have it run a script prior to the rotation.” Sorry, no dice. Logrotate will simply check the config prior to even starting and refuse to do anything useful.

The solution is to use a script, completely outside the scope of logrotate, to create these rotated logs directories just before logrotate runs. Let’s say all remote hosts log to /mnt/logs/yyyy/mmm/dd/<hostname>/<log-name>. Your logrotate.conf file might look like this:

/mnt/logs/*/*/*/* {
weekly
dateext
compress
maxage 365
olddir rotated

}

This would rotate all logs within that directory structure to the relative directory called “rotated.” But of course we need the rotated directory to exist. So let’s create a small script to do that:

#!/bin/bash

# Create “rotated” directory so logrotate has somewhere to put the rotated files.
# Logrotate will not do this by itself, even when putting the script in the “prerotate” section

for i in /mnt/logs/*/*/*; do
if ! [ -d $i/rotated ]; then
/bin/mkdir $i/rotated
/bin/chown root.log_analyst $i/rotated
/bin/chmod 550 $i/rotated
fi
done

In the above script, we’re checking to see if the directory exists, and if it doesn’t, we create it. Then we set proper permissions, owner and group in order to ensure integrity of the rotated log directory. I am assuming a group of log_analyst, so change as appropriate.

The final step it to ensure the script runs just before the logrotate happens. In RHEL, logrotate is called from within the cron.daily directory. So we’ll just name the script make_rotated.sh and link it in this directory in such a way that it runs before logrotate:

[root@hostname log]# ll /etc/cron.daily/ | grep logrotate
total 64
lrwxrwxrwx 1 root root 31 Apr 8 15:19 01_pre-logrotate -> /usr/local/sbin/make_rotated.sh
-rwx—— 1 root root 180 Feb 26 2009 logrotate

There are certainly other ways to solve this problem, but this seems to work well. Until next time…

For those that did not see me at the Rochester Security Summit last year, and who would like to see me present my OSSEC in the Enterprise presentation, I will be giving it again at the ISSA Ft. Worth chapter meeting, next Wednesday. Details are as follows:

Meeting Location: Devry University, 301 W. Commerce Street, Fort Worth, TX 76102

Phone: 817-810-9114

Parking is up to you, there is a parking garage, lots & meters.

FWT Consulting will be sponsoring the meeting & providing lunch, so please RSVP to steve.streiffert <a t> gmail.com by Monday, May 17, 2010.

Encrypted is good. Clear-text is bad. AV is good. Not having AV is bad. Those are the messages we have been receiving and teaching for some time now. And while they do contain sound security advice, it is important to also consider the context of the situation at hand.

A colleague of mine recently requested a firewall exception for FTP traffic. The security policy did not normally allow clear-text traffic, which is generally a good policy to have, but in this case it was justified.

He was trying to get DAT updates for AV on a Linux box (we’ll get to that in a minute), and anonymous FTP was the best way to do it.

Consider this:

1. There was already an FTP service running, so adding an additional service would mean one more to patch, secure, etc.
2. The access allowed was only anonymous, making the issue of clear-text password transition moot.
3. Only read access was allowed, thereby mitigating file integrity issues.
4. Since these were ultimately publicly available DAT files, file confidentiality was not an issue.

After careful examination, it was clear that clear-text FTP was the best option.

Antivirus on Linux is another area where the cure may be worse than the disease. Although you’re not likely to win over your auditor friends, sometimes it can be demonstrated that running AV on Linux is a bigger threat than not running it.

Antivirus software is software, just like every other kind of software. It is prone to security failures and attacks. Is an AV agent with a history of vulnerabilities more of a liability on a system which is not likely to be a target of viruses? Perhaps, perhaps not. Again, it depends a lot on context.

Context helps us to make good security decisions–decisions that are based on risk, reason and facts, not necessarily best-practices or requirements. Keep an open mind.

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

Letting people go is rarely a feel-good experience. Even if everything goes smoothly, there are always human emotions involved, and those emotions make some people unpredictable.  It’s important, therefore, to make sure extra attention and detail is paid to certain areas before and during a layoff. Here are five things to watch out for in your logs and monitoring tools.

1. Outbound connections to a single IP. If you have the benefit of knowing what normal network traffic patterns look like, watch for variations and patterns that may indicate a back-door connection being initiated from within the network. Does outbound traffic on one port spike at around 5:00 PM and stay that way for an extended period of time? Does it look encrypted? Is it using a common port like 53, normally reserved for DNS, but does not seem to be DNS traffic? These could all be indications of a back-door or unauthorized remote access channel.
2. New and changed scheduled tasks/cron jobs. Pay special attention to scheduled tasks and cron jobs and the scripts they run. Monitor changes to those scripts. Look for new and changed tasks and cron jobs. For Windows tasks, refer to IDs 602, 4698, 4699, 4700, 4701 and 4702. For ‘nix, look for changes in the cron log. Scheduled tasks and cron jobs are often used to plant logic bombs, which can execute even months after an employee has left.
3. E-mail to competitors and media. E-mail going to domains which are not considered “friendly” to the organization may indicate that someone has a hatchet to bury. In particular, if there are several of these organizations listed as recipients in the same e-mail, that could be someone trying to sling mud or disseminate confidential information.
4. Data leakage. Let’s get one thing straight. If someone wants to steal data, chances are they will be able to do so undetected if they are smart enough. Even so, people don’t always take the time to cover their tracks. Some things to check: proxy and mail gateway logs for interesting content types (i.e. attachments), USB drives being plugged in, and the good ol’ physical equipment walking out the door.
5. New and Changed Accounts. Some people aren’t too good at letting go. They may try to maintain access into the environment. This may involve either a password change or a new account. People know their account may be disabled, so they may try to create a new, innocuous looking account to maintain access. Perhaps they will try to use a service account with a well-known password. Whatever the case, pay special attention to accounts.

Perhaps the best thing to do is to level with your staff as much as is possible. Treating people with respect and dignity could go much further than any technical measure you can dream up.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.

OSSEC is one of those tools that continues to surprise me with its ability to perform low-level and important security tasks. In fine Unix tradition, individual parts of OSSEC can often be persuaded to do your bidding in ways not previously conceived. Put another way, it’s nice to hack. :)

Today, I’m going to show you how to use OSSEC to detect sensitive info within flat files. This example will demonstrate American social security numbers, but the same logic could be used to detect clear-text credit card numbers (a violation of the PCI DSS standard), or other personally identifiable information which should not be in clear-text.

We’re going to rely on the rootcheck functionality of OSSEC. Rootcheck does all kinds of nifty things, one of which is to scan within files for patterns. Normally, you would expect to look for things like insecure system configuration settings. We’re going to use it to find possible social security numbers.

First, create a test file containing a properly formatted SSN in a location that OSSEC is monitoring:

echo 123-45-6789 > /var/www/html/example.com/www/customer_data

Next, add the following to /var/ossec/etc/shared/system_audit_rcl.txt:

# Detect possible SSNs
[Possible Unencrypted Social Security Number Detected] [any] []
d:$web_dirs -> r:^\. -> r:\d\d\d-\d\d-\d\d\d\d;

Now, let’s create an alert so we get a heads up on what OSSEC finds (change the rule ID as needed):

<rule id=”100024″ level=”12″>
<if_sid>516</if_sid>
<match>Unencrypted Social Security Number</match>
<description>Possible Unencrypted Social Security Number Detected</description>
</rule>

Restart OSSEC:

/var/ossec/bin/ossec-control restart

Finally, we’ll initiate a scan on the local system:

/var/ossec/bin/agent_control -r -u 000

And here is what the resulting alert looks like:

OSSEC HIDS Notification.

2010 Jan 12 18:25:43
Received From: hostname->rootcheck
Rule: 100024 fired (level 12) -> “Possible Unencrypted Social Security Number Detected”
Portion of the log(s):

System Audit: Possible Unencrypted Social Security Number Detected. File: /var/www/html/example.com/www/customer_data.

–END OF NOTIFICATION

One of the nice things about the alert above is that it doesn’t actually include the contents of the file, so you don’t have to worry about clear-text data traversing your mail server.

There are also a few caveats to keep in mind.

First, it is slightly prone to false-positives. Since the OSSEC regex library is designed for speed, rather than full regex support, it’s not possible to use something like \b to define a word boundary. This means that while the regex will match SSNs like 123-45-6789, it would also match something like 00123-45-678911. It also wouldn’t match a delimiter other than the dash. You might be able to mitigate some of these risks if you’re sure how the data would be formatted (but then again, if you’re sure you have clear-text data on the server why do you need this :) ?)

Second, it won’t find data in databases, which is probably where it is likely to be stored. It’s really only useful against flat-files.

Finally, there’s currently no facility to save this change when upgrading, such as you can already do with local_rules.xml. Make sure you have a backup of system_audit_rcl.txt handy that you can restore.

Despite these shortcomings, if you are already using OSSEC then this can be an excellent way to quickly add some rudimentary data checking to the OSSEC scans. It shouldn’t be relied on exclusively and the absense of an alert doesn’t necessarily mean you’re safe, but for a 5-minute change and in the absence of a budget for a more robust tool, this might lead to quite a surprise!

Thanks to Daniel Cid for help in getting this working, and of course for the excellent tool that OSSEC is.

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!

OSSEC 2.3 is out. It has some interesting new features such as the ability to run a command and capture the output like a log. This really opens up possibilities. I’d also like to get some time to explore how this can possibly be abused. We’ll see.

The Dovecot support I added awhile back is in this release, as well as a few miscellaneous rules and bug reports. All in all, things seems to be pretty stable. Get it here. and don’t forget to thank Daniel for all the hard work!