Immutable Security

I have a dirty little secret. It doesn’t have anything to do with the NSA, a leaked memo or pink leotards. But it’s a secret just as earth-shattering, just as awe-inspiring and just as potentially devastating as any other well-hidden secret that has been revealed. Intrigued? Ok, here it is: I have sometimes knowingly agreed to, or even recommended less than secure solutions. Let me give you a minute to collect yourselves.

Yeah, that’s right. Call the ISSA Ethics Committee. Call ISC(2). Call the President. I have been a bad, bad infosec professional. Do what you must, but I have a right to a defense. Allow me to explain my side of the story…

When I was just a young lad starting out in the security field, I had a pretty firm idea of what I thought it meant to be secure. I attended the right conferences, read the hardening guides and dutifully regurgitated what I had learned. Why would anyone choose to be insecure? I thought. That just didn’t make sense.  Security was about correctness. To be less-than-secure was like a mistake on an exam. It needed to be corrected.

As I went along, I learned that the take-it-or-leave-it approach had a few drawbacks. People didn’t really think the same way I did. They had their own jobs to worry about. When security got in the way, it was easier to route around it. When everyone was able to do their jobs every day, the “hackers are coming” argument became less and less effective.

Eventually, I noticed something interesting. The people that I had developed good relationships with were a bit more amenable to my suggestions. When I listened to the the challenges they faced and explained the benefits of what I was trying to do, they saw things in a different light. I made a bit more progress.

Today, I still try to build these important relationships. And when dealing with people who are not security minded, I am much more likely to make compromises. When asked to exclude a location from antivirus scanning that I know doesn’t need to be excluded, I might offer to make exclusions on read only, so that anything newly written will be scanned. When hardening a system, I might purposely leave out a setting that I think will likely cause the administrators to experience a lot of trouble and subsequently develop a negative association with security. I might even recommend that a risk not be addressed at this time, so as to focus limited resources on more important issues.

As infosec practitioners, we are the diplomats of our profession. We want others to come to us with security concerns, not try to route around us. We want to be partners with those we serve. And just like any other form of diplomacy, that involves compromise.

It was inevitable. Cloud services are popping up everywhere and it was only a matter of time before log-based services started to appear. But does that mean the cloud is the right place for your logs? What are the key indicators of success for such services? What can we learn from years of logging on private networks? If I were to design a cloud-based logging service, here are the top things I would consider:

• Avoid legacy syslog transport at all costs. Plain-text, UDP syslog is so last-century. The only reason we (security types) tolerate it is because we usually have no choice. Most of the network devices force our hand. To support syslog as a transport method over the Internet is asking for all sorts of trouble. There will be no way to completely eliminate log spoofing, log sniffing and lost logs. Customers serious about logging will simply not tolerate such chaos.
• Support syslog message format at all costs. Notice the difference here. I don’t want to see the syslog transport method being used over the Internet. Logs are not radio station streams. I do want to see the syslog message format supported. Any service who does not support the syslog message format will not be successful.
• Secure the transport. Again, we need to address the problem of log spoofing, log sniffing and lost logs. This necessitates TCP for trasmission over UDP, authentication of the log source and encryption of the data. There are a couple of ways this can be implemented: a VPN gateway can be established with the log provider or a log-collecting agent can be installed on one or more local machines. In both scenarios, the transport of the logs is addressed.
• Implement flexible delivery options. Not all logs can or should be sent in real time. Firewall and proxy logs, in particular, are very chatty, but can also be very important indicators of an attack. Some mainframe logs have to be submitted as batch FTP transfers. Some systems push their logs and some need to be polled. Some logs are only produced when a monthly or quarterly report runs. The point is that we need to consider multiple delivery options. A successful cloud offering will allow for log traffic shaping, queueing, real-time and batch transfers.
• Protect the application layer. Logs are pretty useless unless you can index, search and correlate them. This means you need a robust application to navigate the millions of messages you might be sending. It’s inevitable that people will attempt to attack the application layer. Not only does this mean securing the site against traditional web attacks, but also remote log injection. Logs are volatile and anything might show up. That gives the attacker a log of flexibility. Can the bad guy drop the customers table with a well-formed SQL query put into the username field of a local server? It’s certainly within the realm of possibility.
• Meet the traditional requirements. Log management programs are often driven by compliance needs. It is imperative that traditional requirements for log management programs can be met. This means we need controls around who accesses these logs, when they accessed them, from where, and why. This includes employees of the cloud service. How long will they be stored online and offline? What happens in the event of a subpeona or a breach? Customers may even demand that the cloud service providers, themselves, be audited under something like ISO 27001.
• Start planning for interoperability. I haven’t seen any efforts in the area of cloud log service interoperability, but I suspect this will become a pain point in the future. If there is anything we have learned from vendor lock-in in the past, it is that the customer’s needs are not best served by this approach. I, for one, would be extremely wary of any vendor that did not allow me to export all of my logs in a common industry-accepted format. What if the service were to go out of business? Would I lose years of valuable log data?

While this is not an exhaustive list, I do believe that now is the time to start a dialogue. Rather than try to stop new technology and services, let’s find ways to make them work effectively. Let’s talk about the risks and address them. Let’s learn from the mistakes of the past and prevent the same mistakes in the future.

When I went searching for a better interest rate for my emergency fund, I ran across a bank that offered over 5%, with relatively few restrictions. I thought this might be a good bank to work with.

So I set up an account, and immediately noticed some things that had me concerned from an information security perspective. The password length was limited to six characters, e-mails to their support contact resulted in a bounce message from two people who apparently no longer worked there, and they were leaking other information that gave me details about their internal network. This was all noticed without actively probing them in any way at all. Finally, when I brought the issues to their attention, the best response I got was that they were planning a system upgrade sometime around September.

In that case, the actual price to me of taking an action to protect my information security, assuming I had invested $25,000 with them, would have been almost $100 a month in lost interest.

Cost does not necessarily equate to price, though. Price can be expressly measured in very finite terms, where cost is often a collection of values. In the example above, we might say that the price is around $100 a month, but the cost could include stress, time and aggravation from a resulting security breach.

I am reminded of this lately with the recent news surrounding the backscatter x-ray machines currently in operation at US airports, combined with the “enhanced” pat-down procedures for those that refuse to go through the machine. I don’t think its too strong to say that we are left with the choice of having a revealing picture taken of us, or being fondled by a TSA agent.  This is the trade-off we are supposed to accept for enhanced security.

This might be an acceptable trade-off if the technology and procedures significantly reduced the risk of dying in a terrorist attack. But according to reason.com, the risk of dying in a plausible attack is actually far lower than the risk of dying by crossing the street, and this was before the machines. And when asked about the technology, Rafi Sela, a security expert at Ben Gurion airport in Tel Aviv, Israel, had this to say:

“I don’t know why everybody is running to buy these expensive and useless machines. I can overcome the body scanners with enough explosives to bring down a Boeing 747″

On the surface, it would seem that the machines simply aren’t worth it.

Now ask yourself how you would feel if the system stopped a major attack, and it could be shown that no other countermeasures would have stopped it? How would you feel if someone close to you was killed by a terrorist?

Or maybe the question is, how would you feel about your three-year old daughter being photographed by one of these machines?

These provocative questions go beyond the statistics to the essence of what we are–human beings. They at the same time expose our fears and our fallacies; our fear of attack and the fallacy of not being able to accurately intuit risk.

We’re being asked to pay a higher and higher price for our safety and security, but at what cost? Are these trade-offs worth it? Have we taken a rational look at the alternatives? Are we being led by fear, allowing our liberties to be usurped by largely ineffective security measures? Will our children accept this as normal and routine? How far will it go? Before you answer, could you have envisioned this even five years ago?

At what point will the answer simply be, “No!” Security always has trade-offs. What are you willing to give up in the name of security?

Yesterday, I blogged about how we could beat OSSEC up, or, to put it more accurately, the people and protocols behind it. Today, we’re going to discuss how we can fight back against the bullies. For this post to make sense, you’ll need to read the first one. Go ahead, I’ll wait…

Ok. Put your gloves on. Here are some countermeasures and considerations for all of the risks we discussed previously.

1. Alert manipulation. Recall that alert manipulation is the process by which alerts are used against us.
   1. Alert tickling. The idea with alert tickling is to send just enough normal-looking alerts to create a window for the attacker to go undetected. There are a couple of ways to defend against this. One is active response. By default, anything that is level 6 or higher can trigger an active response such as blocking the IP for 15 minutes. The cool thing about active response is that it will continue to work even if your alerts are delayed until the next hour. This means the attacker might still be blocked regardless of when you get the e-mail. Another countermeasure is to increase the default number of alerts per hour to something like 9999. Of course, this opens you up to the next risk of alert flooding. Perhaps the best countermeasure is to configure the do_not_delay option in ossec.conf. Adding something like this would save all e-mails over 12 until the next hour, except for those that appear to be an attack:

   <email_alerts>
   <email_to>secanalyst@example.com</email_to>
   <group>attacks</group>
   <do_not_delay />
   </email_alerts>

   2. Alert flooding. With alert flooding, the attacker would like to swamp you with so many alerts that you just can’t bear to read them all. Besides tuning, the best option to protect against alert flooding is to keep the email_maxperhour setting at a reasonable level. If you choose to go that route, be sure to note the risks that can introduce and plan accordingly.
2. Killing the OSSEC processes. I am a firm believer in the mantra that if someone has administrative access to your system, they can do whatever they want. Ultimately, there is nothing you can do to completely protect against this risk. On the other hand, not all administrative users are savvy enough to be able to subvert the ossec processes. A well-written SeLinux policy just might be enough to hold them back. Also, recall that I talked about the non-malicious attacker. Non-technical safeguards such as background checks, job rotation and common workstation areas might be enough of a deterrent for those situations.
3. Log injection. With log injection, the attacker wants to control the log output. There are two main types of log injection: remote and unauthenticated. They can also be combined for greater efficiency.
   1. Remote log injection. The basic defense against remote log injection is to first assume that all input is malicious. Nothing from the user can be trusted, even if your OSSEC installation is on a private network. The second line of defense is to write your OSSEC decoders in such a way as to restrict where in the message input can be coming from. Referring again to Daniel’s paper, Attacking Log Analysis Tools, we can see that adding a ^ (to denote the beginning of the line) or a $ (to denote the end of a line) is often enough.
   2. Unauthenticated log sources. The obvious countermeasure to unauthenticated log sources is to not use them. However, while that is the best defense, it is rarely practical in a modern network. Sometimes you are simply faced with the choice of getting logs through syslog, or not at all. Given the choice, I would choose to get the logs. One approach is to design an out-of-band network where your unauthenticated network devices can send their logs. That approach is rather expensive and not very pragmatic for most people. The more practical approach is to enable IP spoofing protection on your network devices, such as in this example. Be aware, though, that it may not protect against attacks within the same broadcast domain.
4. Traffic manipulation. This attack can be difficult to prevent. Not only does it involve the direct manipulation of traffic, as is the case with something like ARP poisoning, but also intentional or unintentional flooding. Common countermeasures include NAC protection, blocking DoS attacks upstream and not using unauthenticated log sources.
5. Active response manipulation. Active response can prevent an attack from becoming a breach. It can also enable an attacker to block legitimate traffic. OSSEC accounts for this by giving you the white_list option for use in ossec.conf. Any IP or range added to this option won’t be blocked by active response. Keep in mind that a trusted host may be compromised first, with the attacker looking to elevate the access. The safest option is to not whitelist anything. OSSEC also includes a timeout for responses, so if you are accidentally blocked, you can try again in about 15 minutes.

By now you may be starting to realize that the default configuration of OSSEC is very sane and has taken most of these risks into account. At the same time, it gives you enough flexibility to meet your needs and to make your own risk decision. The important thing is to consider the possible consequences of changing a default setting and be comfortable with the additional risks it may present.

No discussion about the effectiveness of a security monitoring tool would be complete without exploring ways to defeat that tool. While this may seem self-defeating, it is my belief that an honest perspective about strengths and weaknesses of the tools we use leads to better risk decisions and ultimately, better tools.

OSSEC can be abused in a number of ways. Many of those are not the fault of OSSEC, but rather they speak to weaknesses in basic protocols or the analyst using the tool. Here are some of the ways an attacker may try to abuse OSSEC:

1. Alert manipulation. Alerts can be manipulated in a few different ways. The basic idea is to attack the alerting mechanism in such a way as to throw the defender off the trail. Here are a few of those ways:
   1. Alert tickling. By default, OSSEC will send no more than 12 alerts per hour. This is to prevent an attacker from flooding you with a sea of alerts. This leaves the potential for the bad guy to intentionally engage in something, maybe from a different IP, that looks like noise. You’ll chalk it up as nothing to worry about and the attacker will have a window to proceed while being undetected.
   2. Alert flooding. In this attack, the point is to flood you with as many alerts as possible. The attacker may know that you have raised the default number of alerts per hour, and will count on you not reading them all. A savvy attacker will begin the attack with simple stuff that looks like noise, hoping you will read the first few alerts and delete the remaining 1,000.
2. Killing the OSSEC processes. If your attacker is an administrator, then it is easy to simply kill the processes. Anything logged while the OSSEC processes aren’t running will not be sent once OSSEC is restarted. It is valid to say that if one has administrative access, then all bets are off. Understand, however, that not all attackers are outright malicious or untrusted. Your attacker could simply be an administrator who doesn’t want to follow the change control process, and stops OSSEC before making the unauthorized change.
3. Log injection. When the attacker can manipulate the log file to their liking, we call it log injection.
   1. Remote log injection. While SQL injection is well-understood, it’s log-based sister, remote log injection, is not widely discussed. The basic premise is the same. Anything coming from a user should not be trusted, as it could be malicious. As Daniel demonstrates in his paper, Attacking Log Analysis Tools, when the regexes responsible for parsing the message are not well-written, it gives the attacker the possibility to manipulate the log monitoring program in his favor.
   2. Unauthenticated log sources. By default, OSSEC sends all logs encrypted with blowfish. In addition, a counter is added to each log to prevent log replaying (an attack in itself). However, OSSEC also supports syslog, because, well, let’s face it, that’s sometimes the only option. As you may know, syslog messages can be spoofed. Since the protocol is UDP, the attacker does not have to worry about completing the connection. So, if an attacker can spoof the source IP, he can cause you all sorts of grief. For a great example of how to spoof syslog, check out the paper, The Ins and Outs of System Logging Using Syslog.
4. Traffic manipulation. At attacker who can interfere with the traffic between the OSSEC agent and the manager might be able to keep the message from being received. If the message isn’t received, the manager can’t correlate and alert on the log.
5. Active response manipulation. Active response can be a double-edged sword. On one hand, it can prevent an attack from becoming a breach. On the other hand, when combined with unauthenticated log sources, the attacker might be able to spoof traffic and block legitimate hosts from accessing your host.

Understanding the risks leads to the inevitable question, “how do I protect myself?” In part II, we’ll discuss some countermeasures for these attacks, as well as the trade-offs for these countermeasures.

There are four or five responses to risk, depending on who you ask. They are: mitigate, accept, transfer, reduce, and sometimes, ignore. Ignoring a risk is just a lame way of burying your head in the sand and pretending it doesn’t exist. Let’s just say there are four legitimate responses to risk.

What we usually hope for is mitigation and reduction. These responses deal with risk directly, but most importantly, they show the greatest amount of responsibility in dealing with the problem.

Transferring risk can be a legitimate response or it can be a way to essentially ignore it. It can be legitimate when there is a thoughtful decision to transfer to another party, and that party accepts. This can be an insurance policy, for example. It is illegitimate when transferring it to another party is a way of ignoring it, essentially just another way of saying, “it’s not my problem.”

That leaves acceptance. Acceptance means, “I know that this might happen, and if it does, I am prepared to accept the consequences of my decision.” The problem with acceptance is that it is rarely tied to consequences.

I see this problem in the real world all the time. Security professionals present risks to senior management and they simply do not respond. They don’t respond unless there is an incentive to do so, and agreeing to put themselves on the line with little or no positive return doesn’t win many fans. If the senior executive fails to respond and the risk is realized, they can just say, “I never saw the risk analysis,” or worse, “they failed to effectively communicate the risk to me.”

We need two things to make risk acceptance work. One, the risk acceptor can not be in a position of authority over the risk advisor. This creates a situation where the advisor might water down the risk out of fear for his own job. Two, there have to be consequences that tie directly to the risk being realized and for choosing to not mitigate, transfer or reduce it. Essentially, this means that ignoring the risk becomes default acceptance with a particular consequence. In both cases, the consequences of acceptance (either by default or explicitly) need to be commensurate with the risk.

Consequences tend to motivate people. Risk acceptance without consequences is not risk acceptance at all.