Immutable Security

Everyone is doing it. Whenever the new year rolls around, security bloggers feel the urge to predict the year ahead. We invent new acronyms like APT (Advanced Persistent Threat), talk about mobile malware shutting down communications networks and warn about cyber attackers as if they were just waiting to pounce on some undefendable threat. I am here to tell you that none of that matters.

It’s true that attacks do get more sophisticated. As technology gives us new options to communicate, we have to consider the security implications of using that new technology. We might need to update our tactical strategy to deal with these threats, but our fundamental strategy likely doesn’t need to be changed.

Security engineering principles transcend technology. If the principles are sound, they will survive new technology with little need for change. Core principles such as least privilege, compartmentalization, kneed-to-know and fail-safe apply even to threats that we don’t know about yet.

So, don’t sweat the new stuff. When considering new threats, remember that they are likely just old threats, repackaged. Keep your eye on the ball. Look at your existing countermeasures and, if you have done a good enough job applying the core principles, you may find that you are already protected.

Model likely threat scenarios. What would it mean to incident response if a worm did take out the cell network? Had you already considered that a natural disaster might occur and then implemented something like walkie-talkies? If so, then congratulations–you just protected yourself against the worm threat.

For myself, I’ll just keep doing what I have been doing. Oh, and while I have your attention: “Get off my lawn!”

It was inevitable. Cloud services are popping up everywhere and it was only a matter of time before log-based services started to appear. But does that mean the cloud is the right place for your logs? What are the key indicators of success for such services? What can we learn from years of logging on private networks? If I were to design a cloud-based logging service, here are the top things I would consider:

• Avoid legacy syslog transport at all costs. Plain-text, UDP syslog is so last-century. The only reason we (security types) tolerate it is because we usually have no choice. Most of the network devices force our hand. To support syslog as a transport method over the Internet is asking for all sorts of trouble. There will be no way to completely eliminate log spoofing, log sniffing and lost logs. Customers serious about logging will simply not tolerate such chaos.
• Support syslog message format at all costs. Notice the difference here. I don’t want to see the syslog transport method being used over the Internet. Logs are not radio station streams. I do want to see the syslog message format supported. Any service who does not support the syslog message format will not be successful.
• Secure the transport. Again, we need to address the problem of log spoofing, log sniffing and lost logs. This necessitates TCP for trasmission over UDP, authentication of the log source and encryption of the data. There are a couple of ways this can be implemented: a VPN gateway can be established with the log provider or a log-collecting agent can be installed on one or more local machines. In both scenarios, the transport of the logs is addressed.
• Implement flexible delivery options. Not all logs can or should be sent in real time. Firewall and proxy logs, in particular, are very chatty, but can also be very important indicators of an attack. Some mainframe logs have to be submitted as batch FTP transfers. Some systems push their logs and some need to be polled. Some logs are only produced when a monthly or quarterly report runs. The point is that we need to consider multiple delivery options. A successful cloud offering will allow for log traffic shaping, queueing, real-time and batch transfers.
• Protect the application layer. Logs are pretty useless unless you can index, search and correlate them. This means you need a robust application to navigate the millions of messages you might be sending. It’s inevitable that people will attempt to attack the application layer. Not only does this mean securing the site against traditional web attacks, but also remote log injection. Logs are volatile and anything might show up. That gives the attacker a log of flexibility. Can the bad guy drop the customers table with a well-formed SQL query put into the username field of a local server? It’s certainly within the realm of possibility.
• Meet the traditional requirements. Log management programs are often driven by compliance needs. It is imperative that traditional requirements for log management programs can be met. This means we need controls around who accesses these logs, when they accessed them, from where, and why. This includes employees of the cloud service. How long will they be stored online and offline? What happens in the event of a subpeona or a breach? Customers may even demand that the cloud service providers, themselves, be audited under something like ISO 27001.
• Start planning for interoperability. I haven’t seen any efforts in the area of cloud log service interoperability, but I suspect this will become a pain point in the future. If there is anything we have learned from vendor lock-in in the past, it is that the customer’s needs are not best served by this approach. I, for one, would be extremely wary of any vendor that did not allow me to export all of my logs in a common industry-accepted format. What if the service were to go out of business? Would I lose years of valuable log data?

While this is not an exhaustive list, I do believe that now is the time to start a dialogue. Rather than try to stop new technology and services, let’s find ways to make them work effectively. Let’s talk about the risks and address them. Let’s learn from the mistakes of the past and prevent the same mistakes in the future.

Since version 1.2, OSSEC has generated daily, chained MD5/SHA1 checksums of the alerts log, and if logall is enabled, the archives.log. As Daniel notes in his blog post, this can provide you with a mechanism to prove that a particular log was not tampered with. If you compare a hash that was e-mailed to you with one on the manager, you can reasonably say the log has not been tampered with.

Or can you? What if the e-mail server was on the same network as the manager? What if they were running the same OS version and had the same vulnerability exposure? Can you still say the hash has not been tampered with?

One way to have a high degree of confidence in the integrity of the hash is to get it out to as many people as possible. The more remote, unrelated computers that have a copy of the hash, the higher degree of confidence and non-repudiation in that hash. Since the hash is a one-way function and does not contain the actual log data, the confidentiality of your logs remains intact.

Here are just a few ways this can be done.

• Twitter. Tweet your hashes, maybe even to an account set up especially for this. Make sure they are public. Here’s an example of what I mean.
• Facebook. Set up a facebook page for your hashes and make it public.
• Your web site. Set up a cron job to update a page daily that has all of your hashes. Make sure it gets indexed.

Again, the more unrelated sites that have these hashes, the higher degree of assurance you will have. Although I am not a lawyer, I can imagine a situation where one would make a very compelling argument that the logs were intact, because in order for the other side to repudiate that argument, they would have to prove that all of the external sources were also compromised and the hash modified.

What other ideas do you have for log integrity and non-repudiation? Speak up in the comments!

In a recent point/counterpoint over at techtarget.com, Marcus Ranum responded to Bruce Schneir on the issue of the risks of hiring hackers. Marcus said one thing in particular that struck me as very insightful. He writes:

We hear a lot about “thinking like a hacker,” but I think that’s largely nonsense — it’s really just a matter of “thinking like an engineer” and performing an in-line failure analysis along with your design analysis.

I think he’s mostly right.

Information security defenses are generally built based on compliance, the current new attack and logical analysis of the threats, in that order. But even when we perform a thoughtful risk analysis, we tend to over-emphasize the rare but sensationalized threats, and under-emphasize the mundane threats. Because information security is seen as a solution to the problem of other people actively attacking systems, we tend to design our countermeasures around those threats.

While it’s true that attacks are happening all the time, the vast majority of those are non-targeted attacks that can be resolved by simple countermeasures such as patching. What we tend to underestimate is loss from areas where information security practitioners have traditionally not had a foothold. We don’t design good countermeasures around forgetting to renew the domain name, a service provider going out of business, or someone hitting the delete key by mistake. But the end result is the same–a loss of confidentiality, integrity or availability.

So where does thinking like a hacker come in? Do we need to think like a hacker to protect against a hacker? I suppose that depends on your definition of what a hacker is. If that definition involves creative thinking, considering how things break, and coming up with solid solutions, then yes, thinking like a hacker is good. But then that is really just good security engineering.

Take this site for example. Before it went live, I played the “what if” game. I started to ponder the many ways this site could break. Along with considering the usual stuff like SQL injection and XSS, I wondered what would happen if my DNS provider suddenly went offline. I assumed that someone would steal the server from the data center, or that a government subpoena might inadvertently include this site along with the site they were targeting. I assumed I would turn off some of the security measures temporarily to see if something started to work again. I then assumed I would forget to turn them back on.

I designed countermeasures for all of these scenarios to a certain extent. At some point it wasn’t worth going any further–it is a blog after all. The point is that I didn’t so much design to be attacked as I designed it to be resilient to failure. Does that mean it is fail-proof? Absolutely not. The site could still be defaced. The server could crash and I might have a problem with restoring the backup. I might get hit by a bus. Despite all this, I am comfortable with the level of security engineering I put into the site.

To sum this up, thinking like a hacker is not that different than thinking like a security engineer. It’s two sides of the same coin. To consider how systems can be resilient to failure (the security engineer), we need to consider how they will break (the hacker).

And sometimes, just sometimes, they break in spectacular ways.

For those that did not see me at the Rochester Security Summit last year, and who would like to see me present my OSSEC in the Enterprise presentation, I will be giving it again at the ISSA Ft. Worth chapter meeting, next Wednesday. Details are as follows:

Meeting Location: Devry University, 301 W. Commerce Street, Fort Worth, TX 76102

Phone: 817-810-9114

Parking is up to you, there is a parking garage, lots & meters.

FWT Consulting will be sponsoring the meeting & providing lunch, so please RSVP to steve.streiffert <a t> gmail.com by Monday, May 17, 2010.

Encrypted is good. Clear-text is bad. AV is good. Not having AV is bad. Those are the messages we have been receiving and teaching for some time now. And while they do contain sound security advice, it is important to also consider the context of the situation at hand.

A colleague of mine recently requested a firewall exception for FTP traffic. The security policy did not normally allow clear-text traffic, which is generally a good policy to have, but in this case it was justified.

He was trying to get DAT updates for AV on a Linux box (we’ll get to that in a minute), and anonymous FTP was the best way to do it.

Consider this:

1. There was already an FTP service running, so adding an additional service would mean one more to patch, secure, etc.
2. The access allowed was only anonymous, making the issue of clear-text password transition moot.
3. Only read access was allowed, thereby mitigating file integrity issues.
4. Since these were ultimately publicly available DAT files, file confidentiality was not an issue.

After careful examination, it was clear that clear-text FTP was the best option.

Antivirus on Linux is another area where the cure may be worse than the disease. Although you’re not likely to win over your auditor friends, sometimes it can be demonstrated that running AV on Linux is a bigger threat than not running it.

Antivirus software is software, just like every other kind of software. It is prone to security failures and attacks. Is an AV agent with a history of vulnerabilities more of a liability on a system which is not likely to be a target of viruses? Perhaps, perhaps not. Again, it depends a lot on context.

Context helps us to make good security decisions–decisions that are based on risk, reason and facts, not necessarily best-practices or requirements. Keep an open mind.

Awhile back, I posted about how certain versions of Windows always have the capability to lose logs. I encourage you to read the full post to understand the issues involved, then come back here and continue reading.

The basic problem is that Windows loads the full log into a shared memory space, and if it’s too big, then logs will be silently dropped. That’s why it is very important to have a centralized logging solution with logs sent in real-time or pretty close to it.

So how does centralized logging affect the local retention settings? To know that, you have to make certain assumptions about log size, average number of events generated per day, and so-on. Let’s start with this quote from Microsoft’s Threats and Countermeasures Guide:

Although there is no simple equation to determine the best log size for a particular server, you can calculate a reasonable size. The average event takes up about 500 bytes within each log, and the log file sizes must be a multiple of 64 KB. If you can estimate the average number of events that are generated each day for each type of log in your organization, you can determine a good size for each type of log file.

For example, if your file server generates 5,000 events per day in its Security log and you want to ensure that you have at least 4 weeks of data available at all times, then you would want to configure the size of that log to about 70 MB. (500 bytes * 5000 events/day * 28 days = 70,000,000 bytes.)

70MB doesn’t sound so bad. It’s probably well within the architectural limits of this memory-mapped I/O thing, but even that may be too big.

A better way of calculating what the local log size should be is to consider the recovery time objective and recovery point objective of the centralized log server. That is, consider how long it might take you to get your log server back online in the event of a disaster, then base your local Windows logging settings on that.

For example, if your recovery point objective is three days, you want to make sure that the local Windows logs will collect for at least three days, but not much more than that. Ideally, you also want to make sure that the solution you’re using can forward the logs collected during the downtime, or have a high-availability solution as part of the plan.

Remember, the goal is to get the local log sizes as small as possible, not as large as possible. You want to make sure you’re not losing any logs, and the two best ways to achieve that are to use centralized logging and try to avoid maxing out the services.exe process. This is counter-intuitive, but when you think it through, you’ll realize that it makes sense.

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.