Immutable Security

In a recent point/counterpoint over at techtarget.com, Marcus Ranum responded to Bruce Schneir on the issue of the risks of hiring hackers. Marcus said one thing in particular that struck me as very insightful. He writes:

We hear a lot about “thinking like a hacker,” but I think that’s largely nonsense — it’s really just a matter of “thinking like an engineer” and performing an in-line failure analysis along with your design analysis.

I think he’s mostly right.

Information security defenses are generally built based on compliance, the current new attack and logical analysis of the threats, in that order. But even when we perform a thoughtful risk analysis, we tend to over-emphasize the rare but sensationalized threats, and under-emphasize the mundane threats. Because information security is seen as a solution to the problem of other people actively attacking systems, we tend to design our countermeasures around those threats.

While it’s true that attacks are happening all the time, the vast majority of those are non-targeted attacks that can be resolved by simple countermeasures such as patching. What we tend to underestimate is loss from areas where information security practitioners have traditionally not had a foothold. We don’t design good countermeasures around forgetting to renew the domain name, a service provider going out of business, or someone hitting the delete key by mistake. But the end result is the same–a loss of confidentiality, integrity or availability.

So where does thinking like a hacker come in? Do we need to think like a hacker to protect against a hacker? I suppose that depends on your definition of what a hacker is. If that definition involves creative thinking, considering how things break, and coming up with solid solutions, then yes, thinking like a hacker is good. But then that is really just good security engineering.

Take this site for example. Before it went live, I played the “what if” game. I started to ponder the many ways this site could break. Along with considering the usual stuff like SQL injection and XSS, I wondered what would happen if my DNS provider suddenly went offline. I assumed that someone would steal the server from the data center, or that a government subpoena might inadvertently include this site along with the site they were targeting. I assumed I would turn off some of the security measures temporarily to see if something started to work again. I then assumed I would forget to turn them back on.

I designed countermeasures for all of these scenarios to a certain extent. At some point it wasn’t worth going any further–it is a blog after all. The point is that I didn’t so much design to be attacked as I designed it to be resilient to failure. Does that mean it is fail-proof? Absolutely not. The site could still be defaced. The server could crash and I might have a problem with restoring the backup. I might get hit by a bus. Despite all this, I am comfortable with the level of security engineering I put into the site.

To sum this up, thinking like a hacker is not that different than thinking like a security engineer. It’s two sides of the same coin. To consider how systems can be resilient to failure (the security engineer), we need to consider how they will break (the hacker).

And sometimes, just sometimes, they break in spectacular ways.

For those that did not see me at the Rochester Security Summit last year, and who would like to see me present my OSSEC in the Enterprise presentation, I will be giving it again at the ISSA Ft. Worth chapter meeting, next Wednesday. Details are as follows:

Meeting Location: Devry University, 301 W. Commerce Street, Fort Worth, TX 76102

Phone: 817-810-9114

Parking is up to you, there is a parking garage, lots & meters.

FWT Consulting will be sponsoring the meeting & providing lunch, so please RSVP to steve.streiffert <a t> gmail.com by Monday, May 17, 2010.

Encrypted is good. Clear-text is bad. AV is good. Not having AV is bad. Those are the messages we have been receiving and teaching for some time now. And while they do contain sound security advice, it is important to also consider the context of the situation at hand.

A colleague of mine recently requested a firewall exception for FTP traffic. The security policy did not normally allow clear-text traffic, which is generally a good policy to have, but in this case it was justified.

He was trying to get DAT updates for AV on a Linux box (we’ll get to that in a minute), and anonymous FTP was the best way to do it.

Consider this:

1. There was already an FTP service running, so adding an additional service would mean one more to patch, secure, etc.
2. The access allowed was only anonymous, making the issue of clear-text password transition moot.
3. Only read access was allowed, thereby mitigating file integrity issues.
4. Since these were ultimately publicly available DAT files, file confidentiality was not an issue.

After careful examination, it was clear that clear-text FTP was the best option.

Antivirus on Linux is another area where the cure may be worse than the disease. Although you’re not likely to win over your auditor friends, sometimes it can be demonstrated that running AV on Linux is a bigger threat than not running it.

Antivirus software is software, just like every other kind of software. It is prone to security failures and attacks. Is an AV agent with a history of vulnerabilities more of a liability on a system which is not likely to be a target of viruses? Perhaps, perhaps not. Again, it depends a lot on context.

Context helps us to make good security decisions–decisions that are based on risk, reason and facts, not necessarily best-practices or requirements. Keep an open mind.

Awhile back, I posted about how certain versions of Windows always have the capability to lose logs. I encourage you to read the full post to understand the issues involved, then come back here and continue reading.

The basic problem is that Windows loads the full log into a shared memory space, and if it’s too big, then logs will be silently dropped. That’s why it is very important to have a centralized logging solution with logs sent in real-time or pretty close to it.

So how does centralized logging affect the local retention settings? To know that, you have to make certain assumptions about log size, average number of events generated per day, and so-on. Let’s start with this quote from Microsoft’s Threats and Countermeasures Guide:

Although there is no simple equation to determine the best log size for a particular server, you can calculate a reasonable size. The average event takes up about 500 bytes within each log, and the log file sizes must be a multiple of 64 KB. If you can estimate the average number of events that are generated each day for each type of log in your organization, you can determine a good size for each type of log file.

For example, if your file server generates 5,000 events per day in its Security log and you want to ensure that you have at least 4 weeks of data available at all times, then you would want to configure the size of that log to about 70 MB. (500 bytes * 5000 events/day * 28 days = 70,000,000 bytes.)

70MB doesn’t sound so bad. It’s probably well within the architectural limits of this memory-mapped I/O thing, but even that may be too big.

A better way of calculating what the local log size should be is to consider the recovery time objective and recovery point objective of the centralized log server. That is, consider how long it might take you to get your log server back online in the event of a disaster, then base your local Windows logging settings on that.

For example, if your recovery point objective is three days, you want to make sure that the local Windows logs will collect for at least three days, but not much more than that. Ideally, you also want to make sure that the solution you’re using can forward the logs collected during the downtime, or have a high-availability solution as part of the plan.

Remember, the goal is to get the local log sizes as small as possible, not as large as possible. You want to make sure you’re not losing any logs, and the two best ways to achieve that are to use centralized logging and try to avoid maxing out the services.exe process. This is counter-intuitive, but when you think it through, you’ll realize that it makes sense.

Over at the Apache blog, you’ll find a nice and detailed incident report on the recent, successful attack on Apache.org. I thought it might be worth a few minutes to share my thoughts on their write-up.

First, I would like to say that the level of transparency in this response is truly commendable. Rather than sweep this under the rug, they have chosen to share the details of what happened, why it happened (more on that in a moment) and what their plans are to, hopefully, prevent future breaches.

I encourage you to read the entire post, because it is a good account of how actual, real-world attacks happen. Targeted attacks take advantage of trust (both in people and machines), shared and weak passwords, too much privilege and an assortment of other security 101 vulnerabilities.

The attacks consisted of a XSS vulnerability, brute-force logins, a shortened URL, password sniffing, password re-use and social engineering. Pretty typical stuff, really.

What I find most interesting about this report is the emphasis on technical countermeasures in the What are we Changing? section, when the attack succeeded primarily due to vulnerabilities in human beings.

• The Infrastructure Team members clicked on a cloaked and untrusted link, which launched a XSS attack.
• A brute force login succeeded against a poorly chosen password. But prior to it being successful, no one seemed to be getting alerts on so many failed login attempts.
• They once again exploited the Infrastructure Team by getting them to log in with a password that the team members, themselves, did not choose.
• They took advantage of cached passwords on the server.
• Slicehost didn’t respond to the attack when notified, which enabled one host to continue its attack against someone else.

These are all people issues. It’s the same stuff that we security types have been trying to hammer into the brains of people for years now. There are certainly technical countermeasures which could have helped, but this was an attack on people.

It’s easy for us to play armchair quarterback and be critical of the response, however that is not my intention, Rather, it is my intent to simply cast another light on the response so we can all learn and secure our assets more effectively.

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.

Here’s a little secret that the sales guys of the million-dollar SIEMs are probably going to gloss over. Most of them do not offer a way to encrypt logs in transit end-to-end. Worse, many of them use downright silly methods to collect logs on common operating systems, such as installing an FTP client on every host and sending them along as a big batch every so often.

For some reason, this is seen as acceptable since it’s usually on the internal network. At the same time, we give mean looks to anyone who even mentions the use of a clear-text protocol like telnet.

Here’s another little secret: many of the big SIEM vendors use syslog-ng as the log daemon on their appliances. It’s reliable and secure and generally works very well.

What if there was a way to add encrypted log transit to the expensive SIEM, or to the other log management solution you already have in place? What if you can’t use or don’t need the full functionality of a log analysis solution like OSSEC, but would like the ability to use encrypted log transport? Read on, my friend, because I’m going to show you exactly how to do that, and it can be done in just a few minutes.

1. Install OSSEC in server mode on the existing syslog server from the vendor. I won’t go into detail here on how to install OSSEC, as it’s well documented elsewhere. These systems are generally just minimal Linux installations, so if you are lacking a compiler then you may need to install it on a like system and move OSSEC over. If your vendor won’t let you do this, ask them why you are paying them gobs of money without even the basic functionality of encrypted log transit!
2. Install OSSEC agents on the hosts you want to collect logs from. Configure the agents to collect the logs you are interested in and add them to the manager. Don’t forget to restart the OSSEC manager after adding the agents.
3. Add the following lines to /opt/syslog-ng/etc/syslog-ng.conf under the “source s_local” section:
   # Read OSSEC queue
   unix-dgram(”/var/ossec/queue/ossec/queue”);
4. Edit /var/ossec/etc/ossec-control and modify the two variables which are responsible for starting the OSSEC daemons. After modification, they should look like this:
   DAEMONS=”ossec-remoted”
   SDAEMONS=”ossec-remoted”
5. Finally, stop both syslog-ng and OSSEC. Start syslog-ng first, then start OSSEC.

The reason this works is due to the way OSSEC is designed. In fine Unix tradition, OSSEC separates the tasks into individual daemons that do only one job and do it well. In this case, ossec-remoted is writing to a Unix socket that syslog-ng is reading from in real-time. OSSEC is simply handing off the message to the local syslog daemon, at which point the message continues on its way as it normally would.

I used to use Snare, Zebedee and OSSEC for secure log delivery and file integrity. Now, it can all be handled with one open-source agent, already designed for secure and reliable log delivery.

There is a downside, which is primarily due to me not taking this to the next level and looking into the issue. In this configuration, you will not receive a notification if the OSSEC agent disconnects, since there is nothing to act on the absence of the heartbeat OSSEC sends. I’m pretty sure that can be solved by taking it up a notch and including one or two more daemons, combined with a stripped down decoder and rules file.

So the next time your expensive SIEM vendor tells you there’s no way to encrypt the logs in transit, tell them you can set it up in under an hour. Then send them a bill for your time.

I recently had the displeasure of visiting the hospital emergency room when a family member needed some urgent care. Thankfully, everything turned out OK, so while I was sitting there waiting (and waiting.. and waiting), I had the opportunity to observe some information security practices in action.

Doctors and nurses were making heavy use of laptops, both on portable carts and by simply (awkwardly) carrying them around. Clearly, a wireless network was in use. Of course I didn’t attempt any actual assessment other than observing how people handled information.

While we was seeing the patient, one doctor placed his laptop on a trash can with a slightly pitched lid. I noticed that the laptop was starting to fall while he was, well, being a doctor, and intercepted it before it could crash to the floor. He was just trying to do his job (being a doctor that is, not a techie).

The intake lady (I don’t know what her title was) handed me a bunch of legal agreements to be signed. These were basically consents for treatment and payment, without which I assume we would not have been seen.

The forms simply had big X marks where I was supposed to sign. I noticed that most of the areas where one had to make a choice were not filled in. For example: did I wish to allow electronic access to the records or not? If I followed the instructions of the intake lady I would have simply signed on the dotted line, while my silent choice could have easily been made for me later.

I opted out of access to electronic information, but as I was waiting (and waiting.. and waiting), I started to wonder if that was the wrong choice. The form asked if I wanted to allow authorized users access to electronic information. It did not speak to the storage of said information, which undoubtedly is still electronic. So by not allowing authorized users access to information which is likely already stored electronically, I may have simply made their job harder, while the bad guys, who don’t care about access controls and agreements, might have a crack at it anyway.

The information security problems I observed had little to do with technical security. One doctor was struggling with an awkward laptop. That laptop almost became a write-off and would have probably resulted in some downtime for his to access information (availability). Another problem had to do with consent to access information. While it seemed to be well-intentioned (and probably mandated by HIPAA), the net security effect of my decision didn’t seem to matter much.

Information security is about allowing people to do their job effectively and getting out of the way. It is our job as security professionals to study how other professions operate so that we can enable them to work effectively, with safety of information built into that workflow. It is also about finding the subtle nuances of the controls we put in front of people and thinking it through the entire way. It’s a game of “what ifs,” which often leads to surprising conclusions.

Pop quiz: how many places can an application install itself to ensure it will survive a Windows reboot? If you named the usual suspects like the run and autorun locations in the registry, you would be correct, but have mentioned only a small subsection of the many places. I’m going to resist the urge to rant about how insane it is to have so many possible places for an application (i.e. trojan) to ensure its survivial. Instead, I’ll just give you the most comprehensive list I have come across, courtesy of the Autoruns utility from Microsoft (formerly SysInternals):

C:\Documents and Settings\All Users\Start Menu\Programs\Startup

C:\WINDOWS\Tasks

HKLM\System\CurrentControlSet\Control\Terminal Server\Wds\rdpwd\StartupPrograms

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Userinit

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run

HKCU\Software\Microsoft\Windows\CurrentVersion\Run

HKLM\SOFTWARE\Classes\Protocols\Filter

HKLM\SOFTWARE\Classes\Protocols\Handler

HKCU\SOFTWARE\Microsoft\Internet Explorer\Desktop\Components

HKLM\SOFTWARE\Microsoft\Active Setup\Installed Components

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\SharedTaskScheduler

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\ShellServiceObjectDelayLoad

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellExecuteHooks

HKLM\Software\Classes\*\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\AllFileSystemObjects\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\Shellex\DragDropHandlers

HKLM\Software\Classes\Directory\Shellex\PropertySheetHandlers

HKLM\Software\Classes\Directory\Shellex\CopyHookHandlers

HKLM\Software\Classes\Folder\Shellex\ColumnHandlers

HKLM\Software\Classes\Folder\ShellEx\ContextMenuHandlers

HKLM\Software\Classes\Directory\Background\ShellEx\ContextMenuHandlers

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers

HKLM\Software\Microsoft\Windows\CurrentVersion\Shell Extensions\Approved

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\Browser Helper Objects

HKCU\Software\Microsoft\Internet Explorer\UrlSearchHooks

HKLM\Software\Microsoft\Internet Explorer\Toolbar

HKLM\Software\Microsoft\Internet Explorer\Extensions

HKLM\System\CurrentControlSet\Services

HKLM\System\CurrentControlSet\Services

HKLM\System\CurrentControlSet\Control\Session Manager\BootExecute

HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options

HKLM\System\CurrentControlSet\Control\Session Manager\KnownDlls

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\UIHost

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Notify

HKCU\Control Panel\Desktop\Scrnsave.exe

HKLM\System\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9\Catalog_Entries

HKLM\System\CurrentControlSet\Services\WinSock2\Parameters\NameSpace_Catalog5\Catalog_Entries

HKLM\SYSTEM\CurrentControlSet\Control\Print\Monitors

HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SecurityProviders

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Authentication Packages

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Notification Packages

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Security Packages

HKLM\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order