Immutable Security

Letting people go is rarely a feel-good experience. Even if everything goes smoothly, there are always human emotions involved, and those emotions make some people unpredictable.  It’s important, therefore, to make sure extra attention and detail is paid to certain areas before and during a layoff. Here are five things to watch out for in your logs and monitoring tools.

1. Outbound connections to a single IP. If you have the benefit of knowing what normal network traffic patterns look like, watch for variations and patterns that may indicate a back-door connection being initiated from within the network. Does outbound traffic on one port spike at around 5:00 PM and stay that way for an extended period of time? Does it look encrypted? Is it using a common port like 53, normally reserved for DNS, but does not seem to be DNS traffic? These could all be indications of a back-door or unauthorized remote access channel.
2. New and changed scheduled tasks/cron jobs. Pay special attention to scheduled tasks and cron jobs and the scripts they run. Monitor changes to those scripts. Look for new and changed tasks and cron jobs. For Windows tasks, refer to IDs 602, 4698, 4699, 4700, 4701 and 4702. For ‘nix, look for changes in the cron log. Scheduled tasks and cron jobs are often used to plant logic bombs, which can execute even months after an employee has left.
3. E-mail to competitors and media. E-mail going to domains which are not considered “friendly” to the organization may indicate that someone has a hatchet to bury. In particular, if there are several of these organizations listed as recipients in the same e-mail, that could be someone trying to sling mud or disseminate confidential information.
4. Data leakage. Let’s get one thing straight. If someone wants to steal data, chances are they will be able to do so undetected if they are smart enough. Even so, people don’t always take the time to cover their tracks. Some things to check: proxy and mail gateway logs for interesting content types (i.e. attachments), USB drives being plugged in, and the good ol’ physical equipment walking out the door.
5. New and Changed Accounts. Some people aren’t too good at letting go. They may try to maintain access into the environment. This may involve either a password change or a new account. People know their account may be disabled, so they may try to create a new, innocuous looking account to maintain access. Perhaps they will try to use a service account with a well-known password. Whatever the case, pay special attention to accounts.

Perhaps the best thing to do is to level with your staff as much as is possible. Treating people with respect and dignity could go much further than any technical measure you can dream up.

Many years ago, after I had been using OSSEC in an enterprise setting for a few months, I noticed an interesting phenomenon. Administrators, many of whom I had forwarded “was this you?” alerts to, were now coming to me to rat on themselves.

I would be working away in my cubicle when someone would come up behind me. It went something like this:

“Hey, Mike. Just wanted to let you know what you’ll be seeing me <insert action here> in the logs. No cause for concern.”

“Thanks for the heads up,” I would reply.

The administrators knew they were being monitored, but didn’t exactly know the full details of the monitoring. They naturally assumed I would see what they were up to. In many cases I wouldn’t have known.

The vast majority of these folks were honest to begin with, but I can’t help think it assisted them in following process and being just a bit more transparent with what they were doing. Maybe it even dissuaded someone who was on the fence from doing something vindictive.

After seeing this in other environments, I think it deserves a term. At the risk of coining a term for something that has already been identified, I hereby declare this the “OSSEC Effect.” The definition (which could use some refinement) is as follows:

OSSEC Effect: The alteration of a computer user’s behavior when they know their actions are being monitored, but do not realize or understand the extent of the monitoring. Users will, without provocation, volunteer information they believe could be seen as questionable, whether the monitoring system would have known about it or not.

Spammers are weasels. They will try to spam every web application they can, and of course this blog is no exception. Usually, it’s something about Viagra or gambling, but lately they have been leaving rather lame and somewhat amusing jokes along with the spam content. Why not take a moment for a chuckle. For you, dear reader, I present the best of moderated comment spam jokes!

• How do you revive a drowning rodent?  Give it mouse-to-mouse resuscitation.
• What do you call cheese that isn’t yours?  Nacho cheese.  <– my personal favorite :)
• How would you clean a tuba?  Try a tuba toothpaste.
• What kind of flowers grow in outer space?  Sunflowers.
• What is the difference between a prizefighter and a man with a cold?  One knows his blows, and the other blows his nose!
• What do sea monsters eat for lunch?  Fish and ships.

Enjoy your day.

You set up a centralized logging server. Check. You installed the OSSEC manager to analyze your logs in real-time. Check. You even managed to implement high availability. Good going! Now your ready to start configuring clients. It should be as simple as installing an agent and pointing it to the log server, right? Maybe so, but don’t forget these other important steps to make the most of your logs.

1. Set the time to sync with at least one time source. Three is even better. If you don’t ensure the client has synchronized time, putting an event timeline together after a compromise is going to be that much more difficult. With properly synchronized time, patterns emerge that you might otherwise miss.
2. Set the auditing policy. Unless you tell your system what to audit, there may never be any logs to send to the log server. For a Windows domain, a well configured group policy can ensure consistency across the enterprise. For stand-alone Windows systems, a security template can serve the same purpose. For ‘nix systems, pay special attention to the facility and priorities in syslog.conf
3. Review services which listen on the network. Did someone install WinSSHd on the Windows server you configured for logging? If you only look at the usual three Windows logs, you could be missing important information about potential attacks. Make sure to review the output of “netstat -an” for clues as to what may be offering services on the host. Once identified, configure them for logging.

Finally, it pays to take a few moments to make sure logging is working as intended. Countless administrators and log analysts have been bitten by situations where they try to refer to logs after a breach, only to find they’re not there.

Oh, and about that whole ballet thing from yesterday, well, I have decided that pink is not my color after all. Now that April Fool’s day is over, let’s get back to business. :)

I have been fighting the bad guys for the better part of a decade now, and have been in IT for even longer. In all that time I have seen no real progress in security. Systems get hacked, they get rebuilt, and companies supposedly take it seriously. Wash, rinse, repeat.

So I’m done. I’m leaving IT entirely and have decided to pursue my life’s passion, ballet. No more will I be analyzing logs, helping with OSSEC or chasing down the latest worm. Instead, I will be twirling on my tip toes and sliding across Swan Lake.

I imagine there will be some who don’t understand, especially my choice to don a pink tutu in the traditional style. That’s fine. You can continue your futile charge into the black hole that is information security. As for myself, I will enjoy the gentle breeze across my ruffles and the warm embrace of my slippers.

When I first heard about this, I thought to myself, “Say it isn’t so. Tell me this is just a big misunderstanding. Tell me that my favorite place to buy cables at great prices wasn’t breached.” Alas, it seems to be true. Monoprice.com had a breach.

I wasn’t too concerned since all of my credit card numbers are unique and automatically generated, and all of the e-mail addresses I use for businesses are also unique, but just the same, I checked my statement. So far, all is well.

As a security guy, I suppose I should forever relegate monoprice.com to the vendor blacklist. After all, they must have been doing something wrong to allow the bad guys to get in, right? That may indeed be the case. They may have had security so bad you could drive a truck though it. Then again, maybe it was very good.

Breaches happen. It’s very, very hard to cover every known threat, let alone the unknown. Security professionals have to anticipate and protect against all threats, while the criminal only has to find one vulnerability. Most of the time it’s not even under our direct control. We are charged with the responsibility of security and get the blame when a breach happens, but encounter fierce resistance when we tell management what really needs to be done to properly secure a site.

Unlike Gexa Energy, who took almost a year to notify affected customer of a breach, monoprice.com took the bold move of prominently and publicly placing a warning on the front page of their web site.  They further went on to stop accepting orders while the breach investigation was continuing. Right now they’re trying to get back on their feet.

Preventing a breach is difficult enough, but the truer measure of an effective security program is how you respond to a breach. Do you issue carefully-crafted letters from the PR department or do you level with your customers? Do you attempt to shun responsibility or do you recognize your mistakes and learn from them?

It seems that monoprice.com recognizes its responsibility to its customers and is doing the right thing. That, combined with my personal risk-mitigation strategies, probably means I will do business with them again.

Sometimes security measures can be completely correct and at the same time, completely useless. Such is the case when viewing the logs of a user added to the local Administrators group on a Windows 2003 server.

Here is what an OSSEC syslog-formatted alert looks like in this case:

Mar 3 11:04:39 hostname ossec: Alert Level: 8; Rule: 18114 – Group account changed.; Location: (hostname) 172.16.0.1->WinEvtLog; user: username; WinEvtLog: Security: AUDIT_SUCCESS(636): Security: username: HOSTNAME: HOSTNAME: Security Enabled Local Group Member Added: Member Name: – Member ID: %{S-1-5-21-2533786115-122353884-3837542848-1670} Target Account Name: Administrators Target Domain: Builtin Target Account ID: %{S-1-5-32-544} Caller User Name: username Caller Domain: DOMAIN Caller Logon ID: (0×0,0×11529569) Privileges: -

Pop quiz: Tell me who was added to the Administrators group. If your answer is S-1-5-21-2533786115-122353884-3837542848-1670, then you are correct. Now tell me who that really is. Is it Bob from accounting or is it a l33t hacker? What will you tell your auditors? Without the Member Name, how can you know?

By now you may be thinking, “this is clearly an OSSEC bug.” That’s what I thought, too. But as it turns out, Snare exhibits the exact same behavior. And so does another commercial log aggregation tool I have used. So what’s going on here?

When viewing this alert in the Windows Event Viewer, the Member Name also displays the dash (-), but resolves the Member ID at the application layer (if it can). If there is no corresponding user, the SID is displayed. The Member Name is not in the raw log at all.

On one hand, this could be seen as correct behavior. After all, the administrator “Bob” from two years ago may not be the same administrator named Bob who was just hired last week. From a log integrity perspective, presenting the unique SID, which is never duplicated, is correct.

On the other hand, having the Member Name is entirely useful and relevant from the practicality and usability perspective. Who the heck is S-1-7-23-3394466182-97151736-2635146241-1084? Should I be concerned? I don’t know.

Just when you thought this was starting to make sense, a domain controller exhibits different behavior. In that case, the Member Name is populated. If it is a domain member or a stand-alone system, it is not.

To me, the correct behavior seems to be to write both the Member Name and the Member ID to the raw log. That way, the log analyst can clearly see that these were different Bobs. By correlating the timeline of the logs with other activities, we can put the pieces of the puzzle together.

Why did Microsoft not do this? I’d love to know. If you know, please enlighten us in the comments.

Awhile back I signed up for the CouponMom.com newsletter (hey, who doesn’t like to save a few bucks), using a very unique and distinctive e-mail address used only for this purpose.

Awhile later, I started to get garden variety spam to this e-mail address (Viagra, etc).

There are a few reasons this could happen:

1. I have been compromised and the spammers think it would be clever to use that address.
2. Spammers start spamming that address as a matter of chance or because they think, “hey, this guy likes Coupon Mom, maybe he’ll like some male enhancement!”
3. Coupon Mom is supplementing her income by spamming or selling the data, which makes its way into spammer’s hands.
4. Coupon Mom has been compromised.

Usually, when this happens, it’s number 4.

I got to thinking, “hey, they might want to know there might be a problem. I should tell them.”

I fill out their contact form and wait. More than a week goes by with no response.

I try to post a cautionary word to the forum. More than a week goes by and I don’t pass moderation.

I fill out the form again, indicating that it would be better for them to investigate this and notify their members of a breach, if one happened, than it would be for me to speculate about it.

Finally, I get a response. The response, in part, states:

You must have signed up for a Google advertiser link on the site, since the email signups for my site are not shared with any other party.

I am sorry you have had this experience, but caution you against publicly slandering The Coupon Mom program and our member database as the source of the unsolicited email.

Can I say that the Coupon Mom database has been breached? Categorically, no. But I can say that there are symptoms which, in my opinion, should cause a reasonable person to take a closer look.

What’s the lesson here? When someone tells you of a potential problem with your security, don’t just assume you are impenetrable. That person may serve as an early warning of a serious problem you would want to be on top of.

My three-year old recently tried to social engineer her mother and I. The conversation went something like this:

Daughter: I want some milk.

Mom: You can’t have milk right now, honey. You have a cold. How about some juice?

Daughter: I don’t have a cold any more, I gave it to Grandma.

Nice try, little girl, but your evil plan is foiled!

Little does she realize that giving a cold to someone does not mean that you get rid of it.

Until next time, daughter. Until next time…

I have seen so many Windows servers with countless administrators, that it doesn’t even surprise me anymore. There seems to be a large disconnect between what management perceives the security of these systems to be and what the security staff knows the situation to be. Security folks can appreciate the subtle nuances surrounding access control that no other group can (or cares to).  It is with this in mind that I offer you the following subtle ways people already may have administrator access to your Windows server. I like to call them “Administrators by Proxy”:

1. The shared password. This is probably the most common culprit. The “true” administrators (incorrectly) argue that they need to use the built-in administrator account. It’s easy to use and they all use it out of habit. Since the password is, well, shared, there’s really no way to know for sure who actually is using that admin account. Usually, it has been passed around for years such that even ex-employees have administrator.
2. The password spreadsheet or database. While not necessarily a bad idea in concept, the password list is only as secure as its weakest link. If the access control on the file is weak, everyone who has access to that file has administrator. If there is a shared password to open the file, well, see #1. If different accounts are in the file all with different access privileges, everyone with any access has administrator.
3. The service account. Do the DBAs insist they need a service account to be in the administrators group, and do they also insist on managing that account? Even if they have lesser-privileged accounts they normally use, they are administrators. Do you have scripts that login as an administrator? Anyone with access to that script (e.g. Backup Operators) has administrator access.
4. The nested group. Best-practices often dictate that groups should be used for proper administration. If groups are nested (e.g. Wintel is a member of Administrators, and DBAs are a member of Wintel), then those nested group members have administrator. Worse, this is difficult to audit, effectively.
5. The vulnerability. Still running NT? Everyone has administrator. Haven’t patched BackupExec or the latest Microsoft system-level vulnerability? Everyone has administrator. Have an administrator test account with a password of “test?” Everyone has administrator.

One could argue that other access controls mitigate the risk, but I think it’s disingenuous to argue that a financial system still on NT because it’s “too fragile” has any inherent access control at all.

It’s not easy to hunt all of these down and to champion the changes necessary in order to limit the actual administrators. But at the very least, it may be a worthwhile exercise in risk analysis.