Taming Log Noise with Logrotate and a Little Elbow Grease

Introduction to Log Noise

I’ve seen this go wrong when log noise gets out of hand - it’s like trying to find a needle in a haystack. With the complexity of Linux systems increasing, log noise has become a significant problem for sysadmins, self-hosters, and developers. Luckily, we have logrotate to help tackle this issue. It’s a standard tool on most Linux distributions, including Debian, Arch Linux, and Red Hat.

Understanding Logrotate

logrotate is a simple yet powerful tool that allows you to rotate, compress, and manage your system logs. It’s typically run as a daily cron job, and its configuration is defined in the /etc/logrotate.conf file and the /etc/logrotate.d/ directory. By default, logrotate will rotate logs weekly, keeping four weeks’ worth of logs. However, this can be customized to suit your specific needs. Don’t bother with the default settings if you have specific requirements - it’s easy to tweak the configuration to fit your use case.

Configuring Logrotate

To configure logrotate, you’ll need to edit the /etc/logrotate.conf file and the configuration files in the /etc/logrotate.d/ directory. For example, to rotate the system logs daily and keep seven days’ worth of logs, you can add the following lines to the /etc/logrotate.conf file:

daily
rotate 7

You can also specify custom rotation settings for specific logs, such as the Apache access log, by creating a configuration file in the /etc/logrotate.d/ directory. For example, to rotate the Apache access log daily and keep 14 days’ worth of logs, you can create a file called apache with the following contents:

/var/log/apache2/access.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        invoke-rc.d apache2 reload > /dev/null
    endscript
}

This configuration will rotate the Apache access log daily, keeping 14 days’ worth of logs, and will also reload the Apache service after rotation. The real trick is to find the right balance between log retention and disk space usage.

Reducing Log Noise with Logrotate

To reduce log noise, you can use logrotate to filter out unnecessary log messages. For example, you can use the notifempty directive to prevent logrotate from rotating empty logs. You can also use the delaycompress directive to delay the compression of logs until the next rotation cycle, which can help reduce the amount of disk space used by logs. In practice, I usually start with a simple configuration and then tweak it as needed to reduce log noise.

Another way to reduce log noise is to use the logrotate postrotate script to run a command that filters out unnecessary log messages. For example, you can use the postrotate script to run a command like grep -v "noise" /var/log/syslog > /var/log/syslog.filtered, which will filter out log messages containing the string “noise”. This is where people usually get burned - they forget to test their postrotate script, and it ends up causing more problems than it solves.

Security Considerations

When configuring logrotate, it’s essential to consider security implications. For example, you should ensure that the log files are owned by a secure user and group, such as root and adm, and that the permissions are set to 640. You should also ensure that the logrotate configuration files are owned by root and have permissions set to 600. Additionally, you should be aware of potential security risks associated with log rotation, such as the possibility of an attacker exploiting a vulnerability in the log rotation process to gain access to sensitive information. To mitigate this risk, you can use tools like SELinux to enforce strict security policies on your system.

Troubleshooting Logrotate

If you encounter issues with logrotate, you can check the /var/log/syslog file for error messages. You can also use the logrotate -d option to run logrotate in debug mode, which will provide more detailed output about the rotation process. For example, to run logrotate in debug mode, you can use the following command:

logrotate -d /etc/logrotate.conf

This will provide detailed output about the rotation process, including any error messages that may occur.

Additional Resources

For more information about logrotate, you can refer to the official logrotate documentation. You can also check the Debian documentation for more information about log rotation on Debian-based systems.