Immutable Security

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

Awhile back I signed up for the CouponMom.com newsletter (hey, who doesn’t like to save a few bucks), using a very unique and distinctive e-mail address used only for this purpose.

Awhile later, I started to get garden variety spam to this e-mail address (Viagra, etc).

There are a few reasons this could happen:

1. I have been compromised and the spammers think it would be clever to use that address.
2. Spammers start spamming that address as a matter of chance or because they think, “hey, this guy likes Coupon Mom, maybe he’ll like some male enhancement!”
3. Coupon Mom is supplementing her income by spamming or selling the data, which makes its way into spammer’s hands.
4. Coupon Mom has been compromised.

Usually, when this happens, it’s number 4.

I got to thinking, “hey, they might want to know there might be a problem. I should tell them.”

I fill out their contact form and wait. More than a week goes by with no response.

I try to post a cautionary word to the forum. More than a week goes by and I don’t pass moderation.

I fill out the form again, indicating that it would be better for them to investigate this and notify their members of a breach, if one happened, than it would be for me to speculate about it.

Finally, I get a response. The response, in part, states:

You must have signed up for a Google advertiser link on the site, since the email signups for my site are not shared with any other party.

I am sorry you have had this experience, but caution you against publicly slandering The Coupon Mom program and our member database as the source of the unsolicited email.

Can I say that the Coupon Mom database has been breached? Categorically, no. But I can say that there are symptoms which, in my opinion, should cause a reasonable person to take a closer look.

What’s the lesson here? When someone tells you of a potential problem with your security, don’t just assume you are impenetrable. That person may serve as an early warning of a serious problem you would want to be on top of.

My three-year old recently tried to social engineer her mother and I. The conversation went something like this:

Daughter: I want some milk.

Mom: You can’t have milk right now, honey. You have a cold. How about some juice?

Daughter: I don’t have a cold any more, I gave it to Grandma.

Nice try, little girl, but your evil plan is foiled!

Little does she realize that giving a cold to someone does not mean that you get rid of it.

Until next time, daughter. Until next time…

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.

Here’s a little secret that the sales guys of the million-dollar SIEMs are probably going to gloss over. Most of them do not offer a way to encrypt logs in transit end-to-end. Worse, many of them use downright silly methods to collect logs on common operating systems, such as installing an FTP client on every host and sending them along as a big batch every so often.

For some reason, this is seen as acceptable since it’s usually on the internal network. At the same time, we give mean looks to anyone who even mentions the use of a clear-text protocol like telnet.

Here’s another little secret: many of the big SIEM vendors use syslog-ng as the log daemon on their appliances. It’s reliable and secure and generally works very well.

What if there was a way to add encrypted log transit to the expensive SIEM, or to the other log management solution you already have in place? What if you can’t use or don’t need the full functionality of a log analysis solution like OSSEC, but would like the ability to use encrypted log transport? Read on, my friend, because I’m going to show you exactly how to do that, and it can be done in just a few minutes.

1. Install OSSEC in server mode on the existing syslog server from the vendor. I won’t go into detail here on how to install OSSEC, as it’s well documented elsewhere. These systems are generally just minimal Linux installations, so if you are lacking a compiler then you may need to install it on a like system and move OSSEC over. If your vendor won’t let you do this, ask them why you are paying them gobs of money without even the basic functionality of encrypted log transit!
2. Install OSSEC agents on the hosts you want to collect logs from. Configure the agents to collect the logs you are interested in and add them to the manager. Don’t forget to restart the OSSEC manager after adding the agents.
3. Add the following lines to /opt/syslog-ng/etc/syslog-ng.conf under the “source s_local” section:
   # Read OSSEC queue
   unix-dgram(”/var/ossec/queue/ossec/queue”);
4. Edit /var/ossec/etc/ossec-control and modify the two variables which are responsible for starting the OSSEC daemons. After modification, they should look like this:
   DAEMONS=”ossec-remoted”
   SDAEMONS=”ossec-remoted”
5. Finally, stop both syslog-ng and OSSEC. Start syslog-ng first, then start OSSEC.

The reason this works is due to the way OSSEC is designed. In fine Unix tradition, OSSEC separates the tasks into individual daemons that do only one job and do it well. In this case, ossec-remoted is writing to a Unix socket that syslog-ng is reading from in real-time. OSSEC is simply handing off the message to the local syslog daemon, at which point the message continues on its way as it normally would.

I used to use Snare, Zebedee and OSSEC for secure log delivery and file integrity. Now, it can all be handled with one open-source agent, already designed for secure and reliable log delivery.

There is a downside, which is primarily due to me not taking this to the next level and looking into the issue. In this configuration, you will not receive a notification if the OSSEC agent disconnects, since there is nothing to act on the absence of the heartbeat OSSEC sends. I’m pretty sure that can be solved by taking it up a notch and including one or two more daemons, combined with a stripped down decoder and rules file.

So the next time your expensive SIEM vendor tells you there’s no way to encrypt the logs in transit, tell them you can set it up in under an hour. Then send them a bill for your time.

OSSEC is one of those tools that continues to surprise me with its ability to perform low-level and important security tasks. In fine Unix tradition, individual parts of OSSEC can often be persuaded to do your bidding in ways not previously conceived. Put another way, it’s nice to hack. :)

Today, I’m going to show you how to use OSSEC to detect sensitive info within flat files. This example will demonstrate American social security numbers, but the same logic could be used to detect clear-text credit card numbers (a violation of the PCI DSS standard), or other personally identifiable information which should not be in clear-text.

We’re going to rely on the rootcheck functionality of OSSEC. Rootcheck does all kinds of nifty things, one of which is to scan within files for patterns. Normally, you would expect to look for things like insecure system configuration settings. We’re going to use it to find possible social security numbers.

First, create a test file containing a properly formatted SSN in a location that OSSEC is monitoring:

echo 123-45-6789 > /var/www/html/example.com/www/customer_data

Next, add the following to /var/ossec/etc/shared/system_audit_rcl.txt:

# Detect possible SSNs
[Possible Unencrypted Social Security Number Detected] [any] []
d:$web_dirs -> r:^\. -> r:\d\d\d-\d\d-\d\d\d\d;

Now, let’s create an alert so we get a heads up on what OSSEC finds (change the rule ID as needed):

<rule id=”100024″ level=”12″>
<if_sid>516</if_sid>
<match>Unencrypted Social Security Number</match>
<description>Possible Unencrypted Social Security Number Detected</description>
</rule>

Restart OSSEC:

/var/ossec/bin/ossec-control restart

Finally, we’ll initiate a scan on the local system:

/var/ossec/bin/agent_control -r -u 000

And here is what the resulting alert looks like:

OSSEC HIDS Notification.

2010 Jan 12 18:25:43
Received From: hostname->rootcheck
Rule: 100024 fired (level 12) -> “Possible Unencrypted Social Security Number Detected”
Portion of the log(s):

System Audit: Possible Unencrypted Social Security Number Detected. File: /var/www/html/example.com/www/customer_data.

–END OF NOTIFICATION

One of the nice things about the alert above is that it doesn’t actually include the contents of the file, so you don’t have to worry about clear-text data traversing your mail server.

There are also a few caveats to keep in mind.

First, it is slightly prone to false-positives. Since the OSSEC regex library is designed for speed, rather than full regex support, it’s not possible to use something like \b to define a word boundary. This means that while the regex will match SSNs like 123-45-6789, it would also match something like 00123-45-678911. It also wouldn’t match a delimiter other than the dash. You might be able to mitigate some of these risks if you’re sure how the data would be formatted (but then again, if you’re sure you have clear-text data on the server why do you need this :) ?)

Second, it won’t find data in databases, which is probably where it is likely to be stored. It’s really only useful against flat-files.

Finally, there’s currently no facility to save this change when upgrading, such as you can already do with local_rules.xml. Make sure you have a backup of system_audit_rcl.txt handy that you can restore.

Despite these shortcomings, if you are already using OSSEC then this can be an excellent way to quickly add some rudimentary data checking to the OSSEC scans. It shouldn’t be relied on exclusively and the absense of an alert doesn’t necessarily mean you’re safe, but for a 5-minute change and in the absence of a budget for a more robust tool, this might lead to quite a surprise!

Thanks to Daniel Cid for help in getting this working, and of course for the excellent tool that OSSEC is.

I recently blogged about security practices in a hospital environment that I was witness to. It was interesting to see how security worked (or perhaps didn’t work), rather than post about another standard, tool or best-practice.

Today, I bring you the story of Grandpa. Grandpa had a little information security incident and, well, it didn’t go so well. In fact, it went something like this:

Scene opens with me exiting out of the bedroom after a good night’s sleep…

“Grandpa thinks he has a virus,” my wife dutifully said.

“Coffee,” was the best reply I could muster.

You see, I knew that viruses were really no big deal. I’ve battled many a virus and have always been victorious.  I certainly knew that it could wait until after a cup o’ joe, so I relegated to call him a bit later.

I forgot.

While I was forgetting, Grandpa was in a mini-panic. Malware was telling him he was infected with all kinds of other malware, using a fake scan window of course, and that the only resource was to pay $49.95 for the program.

He bought it. Of course, that didn’t fix his problem.

Your first reaction might be something along the lines of, “what a dummy.” Maybe you’re thinking, “doesn’t he know not to click on links?” Perhaps you think he got what he deserved. While all of those responses may be justified, and maybe even correct given a certain perspective, let’s take another look at this.

Grandpa can’t hear. He is almost completely deaf. Since he grew up hearing, he doesn’t know sign language. He finds family functions to be extremely saddening and frustrating. When the little ones try to tell him something, he doesn’t know what to say because he didn’t hear a word they said.

Being a retiree, he also doesn’t have much income. In fact, there’s really nothing extra. Spending that $50 meant that he had to take $50 from another, probably essential, area. It wasn’t planned; it was more along the lines of, “I need to do this now!”

So, why did he pay this money when I would have helped him for free? Why did he fall victim to the scam? Why did he make an impulsive decision and not think it through?

The answer has nothing to do with how he has or hasn’t been educated about not clicking on untrusted links or installing untrustworthy software. He became a victim because, in that moment, he felt as if he had no other choice. His fight-or-flight response kicked in. He emotionally made a decision because he felt as if he was losing control over something very important. If the computer were no longer his, how would he communicate? What would he do with this time? How would he take his mind off of the multiple health problems that fills his thoughts?

I was able to fix his computer and even helped to get the charge reversed, but the point of this story is that security problems usually aren’t technical in nature. Obviously, trying to teach people to become security experts just to be able to have some fun or do their job effectively isn’t working. The approaches have to change.

We need to understand people better. The age of the security geek is passing. If we are to truly make progress on issues like this, the solutions will be less technical and more people-focused. If Grandpa had felt secure in knowing that I would help him no matter what, perhaps he would have made a different choice. If he had known that his computer would not be relinquished to this scammer unless he paid money, he might have taken awhile to watch a movie until I had called him back.

Security is about people. Let’s make that a mantra for 2010.

One of the nice things about having your own blog is that you get to warn others about companies to avoid. One such company is Blooms Today.

We ordered flowers from Blooms Today quite some time ago and recently they have taken to sending me “checks” in the mail. These are checks for about six dollars and change. There’s a catch, obviously. If you cash the check then you’re agreeing to sign up for some kind of expensive recurring service, for which they will take the liberty of charging your credit card almost $200.

The mailing looks like a check you might receive from your employer or maybe even as a tax refund. It’s one of those tear-across-the-sides-and-top type of envelopes.

It’s a shame that companies have to stoop to the level of baiting you with dishonest sleaze-ball tactics such as this. Instead of interesting me in future business, they have only served to ensure I will not only never order anything from them again, but also warn others about their shady business practices. If I feel particularly ornery, I might even just send it along to my state Attorney General.

Beware and be safe!

I recently had the displeasure of visiting the hospital emergency room when a family member needed some urgent care. Thankfully, everything turned out OK, so while I was sitting there waiting (and waiting.. and waiting), I had the opportunity to observe some information security practices in action.

Doctors and nurses were making heavy use of laptops, both on portable carts and by simply (awkwardly) carrying them around. Clearly, a wireless network was in use. Of course I didn’t attempt any actual assessment other than observing how people handled information.

While we was seeing the patient, one doctor placed his laptop on a trash can with a slightly pitched lid. I noticed that the laptop was starting to fall while he was, well, being a doctor, and intercepted it before it could crash to the floor. He was just trying to do his job (being a doctor that is, not a techie).

The intake lady (I don’t know what her title was) handed me a bunch of legal agreements to be signed. These were basically consents for treatment and payment, without which I assume we would not have been seen.

The forms simply had big X marks where I was supposed to sign. I noticed that most of the areas where one had to make a choice were not filled in. For example: did I wish to allow electronic access to the records or not? If I followed the instructions of the intake lady I would have simply signed on the dotted line, while my silent choice could have easily been made for me later.

I opted out of access to electronic information, but as I was waiting (and waiting.. and waiting), I started to wonder if that was the wrong choice. The form asked if I wanted to allow authorized users access to electronic information. It did not speak to the storage of said information, which undoubtedly is still electronic. So by not allowing authorized users access to information which is likely already stored electronically, I may have simply made their job harder, while the bad guys, who don’t care about access controls and agreements, might have a crack at it anyway.

The information security problems I observed had little to do with technical security. One doctor was struggling with an awkward laptop. That laptop almost became a write-off and would have probably resulted in some downtime for his to access information (availability). Another problem had to do with consent to access information. While it seemed to be well-intentioned (and probably mandated by HIPAA), the net security effect of my decision didn’t seem to matter much.

Information security is about allowing people to do their job effectively and getting out of the way. It is our job as security professionals to study how other professions operate so that we can enable them to work effectively, with safety of information built into that workflow. It is also about finding the subtle nuances of the controls we put in front of people and thinking it through the entire way. It’s a game of “what ifs,” which often leads to surprising conclusions.

Note: The following post applies to Windows versions prior to Vista. I have not researched how logging has changed in versions greater than Vista. I can only assume and hope that is has changed for the better in new versions.

I’m going to let you in on something that not many people realize. There’s nothing you can do, no setting you can configure and no possible way not to lose logs in Windows. Due to the way Windows is designed, there’s always a chance of missing logs on an otherwise running and functional system.  Let’s explore why this is. Hang on to your hat, we’re going into the weeds.

The Windows Eventlog service uses memory-mapped i/o and each active log is fully opened in memory. The eventlog service resides within the services.exe process, which, due to architectural limitations, only allocates 2GB to the process on a 32-bit system.

Within this 2GB, logs are allocated in 64k chunks. New logs take a 64k chunk of memory and take further chunks as needed, ultimately loading the entire log into memory.

So far this isn’t too earth shattering. Smart people know that although you can configure each log to grow to 4GB individually in Event Viewer (see KB 183097), because of this memory-mapped i/0 thing, limited memory is available. So, as the KB article suggests, they configure the log size to 300MB and “Overwrite as needed.” Depending on how busy the host is this could mean a retention of one day or one year.

Here’s where things get funky. Recall that the services.exe process has limited memory available to it (normally 2GB). You have taken that into account by configuring a smaller log size than what Windows says you can configure. You have “Overwrite events as needed” configured to ensure that you’re under this limit. But did you know that this process isn’t just used for Windows logging? The services.exe process also plays host to other services, each of which may use memory for data, dlls and so-on. All of those have to be taken into account when configuring the log size. If you don’t, you’ll hit that 2GB wall a lot sooner than you expect.

The question is: how do you know how much memory to plan for? How do you know what data other applications may load? How can you properly scope the memory needed for logging, other applications, their data and dlls? The short answer is, practically speaking, you can’t. There are just too many variables out of your control.

So what actually happens when Windows can’t allocate enough memory for logging via the services.exe process? Recall that logs are allocated in 64k chunks. Each allocation must be contiguous; if there is not a contiguous chunk of memory available, even if there is sufficient memory available, logging will fail. If allocation fails, it is a silent error. A log that is below its configured size, but which is not given another chunk of contiguous memory to work with (even if non-contiguous memory is available), will simply fail to log anything until another contiguous chunk is available. Logs will be missing and you will be none the wiser. A log that is already at its size limit will, indeed, “overwrite as needed,” and as long as those chunks of memory are available, logs won’t be missing.

Unless perhaps you have “Retain x days” configured.

“Retain x days” sounds like a pretty good idea, right? Your security policy states that you need to have 30 days of logging available online, so you configure Windows to retain 30 days of logs. Simple enough. The problem is that new events in a full log will overwrite events older than x days, but if there are no events older than x days, new events will be discarded. There will likely be unexplainable gaps in your logs.

Taking the above into consideration, what can you do to better your odds of not losing logs? Here are my recommendations:

1. Don’t use the “Retain x days” setting. Simply abandon it. It’s not worth the hassle and confusion. You’re better off not using it at all.
2. Use “Overwrite as needed.” Although this won’t keep you from losing logs, it’s the best of what is available.
3. Increase your RAM to more than 4GB. This will lesson the odds of you running out of memory before that 2GB limit is reached.
4. Use the x64 architecture. 64-bit systems offer a higher memory limit per process than 32-bit. This gives you more potential allocated memory per-process to work with (not to be confused with total system memory.)
5. Use the /3GB switch. Windows has an option which allows for applications on a 32-bit system to have an extra GB allocated to them, at the expense of one less GB for the operating system (4GB being the total available for 32-bit to work with). Instructions on how to do this can be found here.
6. Use a central logging server. This is perhaps the best countermeasure against losing logs. If logs are sent to a central server in real-time, a relatively low log size can be configured in the Event Viewer. This also lowers the chance of running into a memory limitation.

None of these suggestions can fix the underlying problem. The problem is rooted deep within the design of Windows and, to the best of my knowledge (please let me know if I am incorrect), there is nothing that can be done to absolutely ensure that logs won’t be lost on an otherwise healthy system. I imagine this has been corrected on Windows versions Vista and above. At least, I hope it has. For now, may you enjoy the Holiday season and may you have enough memory for proper logging!