Taming Noisy Systemd Logs with Journalctl Filters and Log Rotation Tweaks

Introduction to Systemd Logs

I’ve seen systemd logs become overwhelming on many Linux systems, making it tough to find relevant information. Systemd is a core component of most modern Linux distributions, responsible for managing system services, devices, and processes. One of its key features is the ability to collect and manage system logs through the journald service, which stores log messages in a binary format.

Understanding Journalctl

The real trick is to use journalctl effectively. It’s the primary tool for interacting with systemd logs, allowing you to view, filter, and manage log messages. By default, journalctl will show you all log messages since the last boot, which can be a lot of data. To make sense of this data, we need to use filters.

Basic Filtering

Don’t bother with trying to sift through all the log messages at once. Instead, start with basic filters like priority. Systemd logs have several priority levels, including emerg, alert, crit, err, warning, notice, info, and debug. You can use the -p option to filter by priority. For example:

journalctl -p err

This will show you all log messages with a priority of err or higher.

Filtering by Unit

I usually start with filtering by unit when troubleshooting a specific service. You can use the -u option to filter by a specific systemd unit. For example:

journalctl -u sshd

This will show you all log messages related to the sshd unit.

Filtering by Time

In practice, filtering by time is also very useful. You can use the --since and --until options to narrow down the time range. For example:

journalctl --since=yesterday --until=1hourago

This will show you all log messages from yesterday to 1 hour ago.

Log Rotation Tweaks

This is where people usually get burned - not managing log files properly. While filtering helps to reduce the amount of log data, it’s also important to manage the log files themselves. Systemd provides a built-in log rotation mechanism, which can be configured using the journald.conf file. One useful tweak is to set the maximum size of the log files. You can do this by adding the following line to the [Journal] section of the journald.conf file:

SystemMaxUse=100M

This will set the maximum size of the log files to 100M.

Another useful tweak is to set the maximum number of log files to keep. You can do this by adding the following line to the [Journal] section of the journald.conf file:

SystemKeepFree=500M

This will set the minimum amount of free space to keep on the disk, and will automatically rotate the log files when this threshold is reached.

Security Considerations

When working with log files, security is a top concern. One potential risk is that log files can contain sensitive information, such as passwords or encryption keys. To mitigate this risk, you can use the journalctl --verify option to verify the integrity of the log files. For example:

journalctl --verify

This will check the log files for any signs of tampering or corruption.

You can also use the journald.conf file to set the permissions of the log files. For example:

Storage=volatile

This will store the log files in a volatile location, such as /run/log/journal, which is cleared on reboot.

For more information on systemd and journald, you can visit the systemd.io website, which provides extensive documentation and resources.

Practical Examples

Here are a few practical examples of how to use journalctl filters and log rotation tweaks:

To view all log messages related to the httpd unit, use the following command:

journalctl -u httpd

To view all log messages with a priority of err or higher, use the following command:

journalctl -p err

To set the maximum size of the log files to 500M, add the following line to the journald.conf file:

SystemMaxUse=500M

To verify the integrity of the log files, use the following command:

journalctl --verify