• The Shadowserver foundation is comprised of a group of volunteer security professionals who gather information about Internet-based crime. One of the more interesting projects is a compilation of how various antivirus vendors fare against 0-day threats. How does your vendor hold up?
• Logs are not much good if you can’t trust them. Maintaining log integrity is vital to a robust incident response process. Here is a great article on how to protect your logs from tampering. It’s not fool-proof, but it can go a long way.
• Information security is a profession that necessitates a solid ethical foundation. Security professionals are often trusted with the most sensitive of data. This presentation, from the Honeynet Project, tackles some of the more thorny situations about performing ethical research.
• Looking for a really awesome way to store and compare your Cisco configs? Rancid, or the Really Awesome New Cisco confIg Differ, may be just the tool for you. It stores Cisco configs in CVS and can let you know something changed. By the way, OSSEC is also capable of something very similar.
• Are you looking to use virtualization in your PCI program? It can be done, but like most technologies, has to be approached carefully. This guide will show you some of the things that need to be considered.

That’s it for today. Have a great weekend!

Of course, with any new technologies there are risks. Sometimes the risks are completely new and sometimes they are the same risks we have been dealing with for years — just repackaged in a slightly different way.

When most people think of virtualization security, they think of things like breaking out of the guest machine onto the host. This would be a very serious attack if successful because it has the potential to leverage one attack into something that can compromise several hosts. Even malware is getting into the business. The Blue Pill proof-of-concept used a small hypervisor to intercept calls to and from the guest, thereby controlling it according to desires of the attacker.

These are real threats, but for most organizations they do not represent the most serious threats to virtualized hosts. I was reminded of this last week when I read about Jason Cornish, a former IT employee of drug maker Shionogi. Jason had a beef with his former employer and he took it out by logging in and deleting 88 virtualized hosts. That’s the functional equivalent of a tornado taking out a small data center. He was able to do this simply because of a password which was not changed upon his termination, thereby allowing him remote access.

What’s the lesson here? The point is that we cannot forget the fundamentals of information security when implementing new technologies. That means doing the non-sexy stuff like changing default passwords, revoking access immediately for terminated employees and sending logs to a centralized log server. In the virtual world, we might review the access levels of the virtual administrators to make sure they are appropriate, separate hosts of different sensitivities onto different hypervisors and get alerted when a virtual machine is added, deleted or moved.

We do need to keep an eye on evolving virtual security threats, but we also have to keep the other eye on the fundamentals. We need both.

Yeah, that’s right. Call the ISSA Ethics Committee. Call ISC(2). Call the President. I have been a bad, bad infosec professional. Do what you must, but I have a right to a defense. Allow me to explain my side of the story…

When I was just a young lad starting out in the security field, I had a pretty firm idea of what I thought it meant to be secure. I attended the right conferences, read the hardening guides and dutifully regurgitated what I had learned. Why would anyone choose to be insecure? I thought. That just didn’t make sense.  Security was about correctness. To be less-than-secure was like a mistake on an exam. It needed to be corrected.

As I went along, I learned that the take-it-or-leave-it approach had a few drawbacks. People didn’t really think the same way I did. They had their own jobs to worry about. When security got in the way, it was easier to route around it. When everyone was able to do their jobs every day, the “hackers are coming” argument became less and less effective.

Eventually, I noticed something interesting. The people that I had developed good relationships with were a bit more amenable to my suggestions. When I listened to the the challenges they faced and explained the benefits of what I was trying to do, they saw things in a different light. I made a bit more progress.

Today, I still try to build these important relationships. And when dealing with people who are not security minded, I am much more likely to make compromises. When asked to exclude a location from antivirus scanning that I know doesn’t need to be excluded, I might offer to make exclusions on read only, so that anything newly written will be scanned. When hardening a system, I might purposely leave out a setting that I think will likely cause the administrators to experience a lot of trouble and subsequently develop a negative association with security. I might even recommend that a risk not be addressed at this time, so as to focus limited resources on more important issues.

As infosec practitioners, we are the diplomats of our profession. We want others to come to us with security concerns, not try to route around us. We want to be partners with those we serve. And just like any other form of diplomacy, that involves compromise.

It’s true that attacks do get more sophisticated. As technology gives us new options to communicate, we have to consider the security implications of using that new technology. We might need to update our tactical strategy to deal with these threats, but our fundamental strategy likely doesn’t need to be changed.

Security engineering principles transcend technology. If the principles are sound, they will survive new technology with little need for change. Core principles such as least privilege, compartmentalization, kneed-to-know and fail-safe apply even to threats that we don’t know about yet.

So, don’t sweat the new stuff. When considering new threats, remember that they are likely just old threats, repackaged. Keep your eye on the ball. Look at your existing countermeasures and, if you have done a good enough job applying the core principles, you may find that you are already protected.

Model likely threat scenarios. What would it mean to incident response if a worm did take out the cell network? Had you already considered that a natural disaster might occur and then implemented something like walkie-talkies? If so, then congratulations–you just protected yourself against the worm threat.

For myself, I’ll just keep doing what I have been doing. Oh, and while I have your attention: “Get off my lawn!”

Getting it to work on a centralized log server, however, can be a bit tricky. The issue lies in the unique way these log servers are often configured. Rather than a static set of directories, log servers often dynamically create directories based on the hosts that are logging to them. It’s not uncommon to see a directory structure something along the lines of /mnt/logs/2010/May/18/host/messages. While logrotate can use wildcards in the directory structure definition, it gets a bit hairy when trying to define the rotated log directory.

There are two options for where to put the rotated logs. You can put them in another directory, or in the same directory as the rotated files. Another directory is usually the better option, as rotated logs in the same directory could be rotated again and again, unless you are careful with the way logs are created.

So what happens if the directory you specify for the rotated log files doesn’t exist (remember, the log server is creating them dynamically based on the hosts that report in)? In that case, logrotate will complain and refuse to rotate your logs.

By now you might be thinking, “Why, I’ll just use that handy pre-rotate option. Then I can have it run a script prior to the rotation.” Sorry, no dice. Logrotate will simply check the config prior to even starting and refuse to do anything useful.

The solution is to use a script, completely outside the scope of logrotate, to create these rotated logs directories just before logrotate runs. Let’s say all remote hosts log to /mnt/logs/yyyy/mmm/dd/<hostname>/<log-name>. Your logrotate.conf file might look like this:

/mnt/logs/*/*/*/* {
weekly
dateext
compress
maxage 365
olddir rotated

}

This would rotate all logs within that directory structure to the relative directory called “rotated.” But of course we need the rotated directory to exist. So let’s create a small script to do that:

#!/bin/bash

# Create “rotated” directory so logrotate has somewhere to put the rotated files.
# Logrotate will not do this by itself, even when putting the script in the “prerotate” section

for i in /mnt/logs/*/*/*; do
if ! [ -d $i/rotated ]; then
/bin/mkdir $i/rotated
/bin/chown root.log_analyst $i/rotated
/bin/chmod 550 $i/rotated
fi
done

In the above script, we’re checking to see if the directory exists, and if it doesn’t, we create it. Then we set proper permissions, owner and group in order to ensure integrity of the rotated log directory. I am assuming a group of log_analyst, so change as appropriate.

The final step it to ensure the script runs just before the logrotate happens. In RHEL, logrotate is called from within the cron.daily directory. So we’ll just name the script make_rotated.sh and link it in this directory in such a way that it runs before logrotate:

[root@hostname log]# ll /etc/cron.daily/ | grep logrotate
total 64
lrwxrwxrwx 1 root root 31 Apr 8 15:19 01_pre-logrotate -> /usr/local/sbin/make_rotated.sh
-rwx—— 1 root root 180 Feb 26 2009 logrotate

There are certainly other ways to solve this problem, but this seems to work well. Until next time…

]]> http://www.immutablesecurity.com/index.php/2010/05/19/using-logrotate-with-centralized-log-servers/feed/ 0 http://www.immutablesecurity.com/index.php/2010/05/13/ossec-in-the-enterprise-wednesday-may-19-2010/ http://www.immutablesecurity.com/index.php/2010/05/13/ossec-in-the-enterprise-wednesday-may-19-2010/#comments Thu, 13 May 2010 18:31:00 +0000 Michael Starks http://www.immutablesecurity.com/?p=496 For those that did not see me at the Rochester Security Summit last year, and who would like to see me present my OSSEC in the Enterprise presentation, I will be giving it again at the ISSA Ft. Worth chapter meeting, next Wednesday. Details are as follows:

Meeting Location: Devry University, 301 W. Commerce Street, Fort Worth, TX 76102

Phone: 817-810-9114

Parking is up to you, there is a parking garage, lots & meters.

FWT Consulting will be sponsoring the meeting & providing lunch, so please RSVP to steve.streiffert <a t> gmail.com by Monday, May 17, 2010.

A colleague of mine recently requested a firewall exception for FTP traffic. The security policy did not normally allow clear-text traffic, which is generally a good policy to have, but in this case it was justified.

He was trying to get DAT updates for AV on a Linux box (we’ll get to that in a minute), and anonymous FTP was the best way to do it.

Consider this:

1. There was already an FTP service running, so adding an additional service would mean one more to patch, secure, etc.
2. The access allowed was only anonymous, making the issue of clear-text password transition moot.
3. Only read access was allowed, thereby mitigating file integrity issues.
4. Since these were ultimately publicly available DAT files, file confidentiality was not an issue.

After careful examination, it was clear that clear-text FTP was the best option.

Antivirus on Linux is another area where the cure may be worse than the disease. Although you’re not likely to win over your auditor friends, sometimes it can be demonstrated that running AV on Linux is a bigger threat than not running it.

Antivirus software is software, just like every other kind of software. It is prone to security failures and attacks. Is an AV agent with a history of vulnerabilities more of a liability on a system which is not likely to be a target of viruses? Perhaps, perhaps not. Again, it depends a lot on context.

Context helps us to make good security decisions–decisions that are based on risk, reason and facts, not necessarily best-practices or requirements. Keep an open mind.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

1. Outbound connections to a single IP. If you have the benefit of knowing what normal network traffic patterns look like, watch for variations and patterns that may indicate a back-door connection being initiated from within the network. Does outbound traffic on one port spike at around 5:00 PM and stay that way for an extended period of time? Does it look encrypted? Is it using a common port like 53, normally reserved for DNS, but does not seem to be DNS traffic? These could all be indications of a back-door or unauthorized remote access channel.
2. New and changed scheduled tasks/cron jobs. Pay special attention to scheduled tasks and cron jobs and the scripts they run. Monitor changes to those scripts. Look for new and changed tasks and cron jobs. For Windows tasks, refer to IDs 602, 4698, 4699, 4700, 4701 and 4702. For ‘nix, look for changes in the cron log. Scheduled tasks and cron jobs are often used to plant logic bombs, which can execute even months after an employee has left.
3. E-mail to competitors and media. E-mail going to domains which are not considered “friendly” to the organization may indicate that someone has a hatchet to bury. In particular, if there are several of these organizations listed as recipients in the same e-mail, that could be someone trying to sling mud or disseminate confidential information.
4. Data leakage. Let’s get one thing straight. If someone wants to steal data, chances are they will be able to do so undetected if they are smart enough. Even so, people don’t always take the time to cover their tracks. Some things to check: proxy and mail gateway logs for interesting content types (i.e. attachments), USB drives being plugged in, and the good ol’ physical equipment walking out the door.
5. New and Changed Accounts. Some people aren’t too good at letting go. They may try to maintain access into the environment. This may involve either a password change or a new account. People know their account may be disabled, so they may try to create a new, innocuous looking account to maintain access. Perhaps they will try to use a service account with a well-known password. Whatever the case, pay special attention to accounts.

Perhaps the best thing to do is to level with your staff as much as is possible. Treating people with respect and dignity could go much further than any technical measure you can dream up.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)