Immutable Security

Logrotate is a fantastic little utility for, well, rotating logs. It has several options available. It can rotate by date, file size, set specific owners on rotated files, and it even gives you the capability to run pre and post-rotate scripts.

Getting it to work on a centralized log server, however, can be a bit tricky. The issue lies in the unique way these log servers are often configured. Rather than a static set of directories, log servers often dynamically create directories based on the hosts that are logging to them. It’s not uncommon to see a directory structure something along the lines of /mnt/logs/2010/May/18/host/messages. While logrotate can use wildcards in the directory structure definition, it gets a bit hairy when trying to define the rotated log directory.

There are two options for where to put the rotated logs. You can put them in another directory, or in the same directory as the rotated files. Another directory is usually the better option, as rotated logs in the same directory could be rotated again and again, unless you are careful with the way logs are created.

So what happens if the directory you specify for the rotated log files doesn’t exist (remember, the log server is creating them dynamically based on the hosts that report in)? In that case, logrotate will complain and refuse to rotate your logs.

By now you might be thinking, “Why, I’ll just use that handy pre-rotate option. Then I can have it run a script prior to the rotation.” Sorry, no dice. Logrotate will simply check the config prior to even starting and refuse to do anything useful.

The solution is to use a script, completely outside the scope of logrotate, to create these rotated logs directories just before logrotate runs. Let’s say all remote hosts log to /mnt/logs/yyyy/mmm/dd/<hostname>/<log-name>. Your logrotate.conf file might look like this:

/mnt/logs/*/*/*/* {
weekly
dateext
compress
maxage 365
olddir rotated

}

This would rotate all logs within that directory structure to the relative directory called “rotated.” But of course we need the rotated directory to exist. So let’s create a small script to do that:

#!/bin/bash

# Create “rotated” directory so logrotate has somewhere to put the rotated files.
# Logrotate will not do this by itself, even when putting the script in the “prerotate” section

for i in /mnt/logs/*/*/*; do
if ! [ -d $i/rotated ]; then
/bin/mkdir $i/rotated
/bin/chown root.log_analyst $i/rotated
/bin/chmod 550 $i/rotated
fi
done

In the above script, we’re checking to see if the directory exists, and if it doesn’t, we create it. Then we set proper permissions, owner and group in order to ensure integrity of the rotated log directory. I am assuming a group of log_analyst, so change as appropriate.

The final step it to ensure the script runs just before the logrotate happens. In RHEL, logrotate is called from within the cron.daily directory. So we’ll just name the script make_rotated.sh and link it in this directory in such a way that it runs before logrotate:

[root@hostname log]# ll /etc/cron.daily/ | grep logrotate
total 64
lrwxrwxrwx 1 root root 31 Apr 8 15:19 01_pre-logrotate -> /usr/local/sbin/make_rotated.sh
-rwx—— 1 root root 180 Feb 26 2009 logrotate

There are certainly other ways to solve this problem, but this seems to work well. Until next time…