Taming Noisy systemd Logs with journald Configuration and Filtering

Taming Noisy systemd Logs with journald Configuration

I’ve seen this go wrong when you’re dealing with a barrage of system events - journald can be quite verbose by default. As a seasoned Linux administrator, you’re likely familiar with the systemd suite and its logging component, journald. In practice, this can lead to a noisy and overwhelming log output. To make sense of it all, you need to tame those logs with some careful journald configuration and filtering.

Understanding journald Configuration

The real trick is to understand how journald stores its configuration in the /etc/systemd/journald.conf file. This file contains various options that can be used to customize the logging behavior, such as adjusting the log level, specifying the maximum log size, and configuring log rotation. Don’t bother with trying to decipher the entire file at once - start by taking a look at the default configuration:

sudo journalctl -n

This command will show you the last few log entries. You can also use the --priority option to filter logs by priority level:

sudo journalctl -n --priority=err

This will show you only error-level logs, which is usually where you want to focus your attention.

Filtering Logs

In practice, filtering logs is one of the most useful features of journald. You can use the --unit option to filter logs by systemd unit:

sudo journalctl -n --unit=ssh

This will show you only logs related to the SSH service, which can be really helpful when troubleshooting connectivity issues. You can also use the --identifier option to filter logs by identifier:

sudo journalctl -n --identifier=kernel

This will show you only kernel-related logs, which can be useful when you’re trying to diagnose low-level system problems.

Configuring journald

To configure journald, you’ll need to edit the /etc/systemd/journald.conf file. I usually start with the [Journal] section, where you can set options like SystemMaxUse, SystemKeepFree, and LogLevel. For example, to set the log level to info, you can add the following lines:

[Journal]
SystemMaxUse=100M
SystemKeepFree=20%
LogLevel=info

After making changes to the configuration file, you’ll need to restart the journald service:

sudo systemctl restart systemd-journald

This is where people usually get burned - forgetting to restart the service after making changes. For more information on journald configuration, you can refer to the systemd documentation.

Practical Example

Let’s say you want to configure journald to log only error-level messages and above. You can add the following lines to the /etc/systemd/journald.conf file:

[Journal]
LogLevel=err

After restarting the journald service, you can verify the changes:

sudo journalctl -n

This will show you only error-level logs and above, which should help you cut through the noise and focus on the issues that really matter.