Immutable Security

When many companies think about log management, they immediately jump to SIEMs, or Security Information and Event Managers. You’ll also find many in the infosec community who jump to the conclusion that a SIEM is the right solution for the problem, when they haven’t really even taken the time to set requirements, expectations and understand what it is they are trying to achieve.

I was reminded of this today when a client needed a refresh and storage update of their pre-existing SIEM solution. The quote was $77,000 (USD). They are collecting logs from about 300 servers and running daily reports of successful and failed logins. That’s it.

I couldn’t help to think, what is so special about this solution that they need to spend $77,000 for another log collector appliance (which is really just a Dell server with Windows installed on it) and about 2TB of storage? If they are paying this much for a simple upgrade, what was their initial investment? If it’s like most companies who buy SIEMs, they didn’t get out of the room for under six figures.

To be clear, a good SIEM does more than simply collect logs and allow one to run reports. It can correlate log and vulnerability information, elevate alerts based on asset value, identify suspicious patterns and integrate with ticketing and incident response systems.

The question is: who is sufficiently advanced in their security program to make use of such capabilities? Furthermore, what regulations, contracts or risk programs require this level of sophistication? The answer, based on my experience, is very, very few.

Does that mean having all of this wouldn’t be a good thing? Of course not. In a perfect world, security ninjas would be able to make these things sing and dance in all kinds of cool ways. But most of us don’t live in that world. We live in a world where management gives us conflicting priorities, doesn’t adequately staff the security team and turns down our requests to limit user access.

So I am here to tell you that you probably don’t need a SIEM. And here are the top five reasons why I think so:

1. They are ridiculously expensive. Sure, the cost of security should be commensurate with the risk. But when you’re talking about laying down six or seven figures for a tool, you better be damn sure it is going to pick up your toys and call you sweetheart.
2. The requirements often don’t necessitate it. But I have SoX, HIPAA and PCI requirements! Bah! I have been in all of those environments and, in most cases, a SIEM was nowhere to be found. Most of them had simple syslog servers with daily reports sent out by scripts. They passed the audits just fine.
3. It may never be fully deployed. SIEMs are complex beasts. They are also often buggy, frail and slow. To properly deploy, integrate and test a SIEM can take a team months or even years. The most often line I hear with SIEM administrators is, “We haven’t gotten to that part yet.”
4. They often provide little value. If the cost were high and the value was high, maybe the price wouldn’t be so much of an issue. But the bang for the buck you get is often very low beyond what you could do with a solution that costs one-tenth of the price, or less. I have seen many SIEMs that don’t even alert you to multiple failed logins to a single host. You have to configure these yourself. That means each analyst at each company becomes a research analyst, and has to do their own threat modeling. You should get the experience and expertise of the SIEM vendor from day one.
5. You’ll experience vendor lock-in. Now that you have all of this data in the SIEM, how do you get it out should you want to change course? or will you be forced to pay the piper when you need an upgrade?

Now, obviously, I have come down pretty hard on SIEM vendors here. I expect someone will come along and want to refute my assertions. I most definitely welcome that.

There’s a missing market in the log management world. It lies somewhere between solutions like OSSEC and syslog-ng, and Arcsight. It’s a market which would offer intelligent log collection, analysis, correlation and alerting, combined with valuable reports made available through a nice front-end. And there’s no reason a product like this should cost more than five-thousand dollars, at most.

Note to prospective SIEM employers if I am ever looking for work–just kidding! :)

Here’s a little secret that the sales guys of the million-dollar SIEMs are probably going to gloss over. Most of them do not offer a way to encrypt logs in transit end-to-end. Worse, many of them use downright silly methods to collect logs on common operating systems, such as installing an FTP client on every host and sending them along as a big batch every so often.

For some reason, this is seen as acceptable since it’s usually on the internal network. At the same time, we give mean looks to anyone who even mentions the use of a clear-text protocol like telnet.

Here’s another little secret: many of the big SIEM vendors use syslog-ng as the log daemon on their appliances. It’s reliable and secure and generally works very well.

What if there was a way to add encrypted log transit to the expensive SIEM, or to the other log management solution you already have in place? What if you can’t use or don’t need the full functionality of a log analysis solution like OSSEC, but would like the ability to use encrypted log transport? Read on, my friend, because I’m going to show you exactly how to do that, and it can be done in just a few minutes.

1. Install OSSEC in server mode on the existing syslog server from the vendor. I won’t go into detail here on how to install OSSEC, as it’s well documented elsewhere. These systems are generally just minimal Linux installations, so if you are lacking a compiler then you may need to install it on a like system and move OSSEC over. If your vendor won’t let you do this, ask them why you are paying them gobs of money without even the basic functionality of encrypted log transit!
2. Install OSSEC agents on the hosts you want to collect logs from. Configure the agents to collect the logs you are interested in and add them to the manager. Don’t forget to restart the OSSEC manager after adding the agents.
3. Add the following lines to /opt/syslog-ng/etc/syslog-ng.conf under the “source s_local” section:
   # Read OSSEC queue
   unix-dgram(“/var/ossec/queue/ossec/queue”);
4. Edit /var/ossec/etc/ossec-control and modify the two variables which are responsible for starting the OSSEC daemons. After modification, they should look like this:
   DAEMONS=”ossec-remoted”
   SDAEMONS=”ossec-remoted”
5. Finally, stop both syslog-ng and OSSEC. Start syslog-ng first, then start OSSEC.

The reason this works is due to the way OSSEC is designed. In fine Unix tradition, OSSEC separates the tasks into individual daemons that do only one job and do it well. In this case, ossec-remoted is writing to a Unix socket that syslog-ng is reading from in real-time. OSSEC is simply handing off the message to the local syslog daemon, at which point the message continues on its way as it normally would.

I used to use Snare, Zebedee and OSSEC for secure log delivery and file integrity. Now, it can all be handled with one open-source agent, already designed for secure and reliable log delivery.

There is a downside, which is primarily due to me not taking this to the next level and looking into the issue. In this configuration, you will not receive a notification if the OSSEC agent disconnects, since there is nothing to act on the absence of the heartbeat OSSEC sends. I’m pretty sure that can be solved by taking it up a notch and including one or two more daemons, combined with a stripped down decoder and rules file.

So the next time your expensive SIEM vendor tells you there’s no way to encrypt the logs in transit, tell them you can set it up in under an hour. Then send them a bill for your time.