Taming systemd's Journal Size with Log Rotation and Persistent Journal Storage

Introduction to systemd’s Journal

I’ve worked with Linux systems for years, and one thing that’s always been important is managing system logs. Systemd’s journal is a great tool for this, providing a centralized logging solution that’s both robust and efficient. By default, the journal stores its data in a volatile, in-memory cache, and on disk in /var/log/journal/. However, I’ve seen this go wrong when the journal’s size grows rapidly, especially on systems with high log volumes. This can lead to performance issues and disk space consumption. To avoid this, you can use log rotation and persistent journal storage.

Understanding Journal Configuration

The real trick is understanding how the journal’s configuration works. It’s controlled by the /etc/systemd/journald.conf file, which allows you to customize various aspects of the journal, including its size, rotation, and persistence. To view the current journal configuration, you can use the following command:

journalctl -u systemd-journald

This will display the current journal configuration, including the maximum size of the journal and the rotation settings. Don’t bother with trying to parse the output manually, though - it’s easier to just edit the config file directly.

Configuring Log Rotation

To configure log rotation, you need to edit the /etc/systemd/journald.conf file. I usually start with setting the SystemMaxUse and SystemKeepFree parameters to control the maximum size of the journal and the amount of free space to maintain on the disk. For example:

[Journal]
SystemMaxUse=100M
SystemKeepFree=20M

This configuration sets the maximum size of the journal to 100M and maintains at least 20M of free space on the disk. In practice, you may need to adjust these values depending on your system’s specific needs.

Enabling Persistent Journal Storage

By default, the journal stores its data in a volatile, in-memory cache, and on disk in /var/log/journal/. However, you can enable persistent journal storage by creating a directory for the journal and setting the Storage parameter in the /etc/systemd/journald.conf file. To do this, run the following commands:

mkdir -p /var/log/journal
chown systemd-journal:root /var/log/journal

Then, edit the /etc/systemd/journald.conf file to set the Storage parameter:

[Journal]
Storage=persistent

This configuration enables persistent journal storage, and the journal will store its data on disk in /var/log/journal/.

Rotating the Journal

To rotate the journal, you can use the journalctl command with the --rotate option:

journalctl --rotate

This command will rotate the journal and create a new journal file. This is where people usually get burned - they forget to rotate the journal, and it grows out of control.

Vacuuming the Journal

To remove old journal entries and free up disk space, you can use the journalctl command with the --vacuum-size option:

journalctl --vacuum-size=100M

This command will remove old journal entries and free up disk space, maintaining a maximum size of 100M for the journal.

Security Considerations

When configuring the journal, it’s essential to consider security implications. For example, you should ensure that the journal directory and files have proper permissions and ownership to prevent unauthorized access. You can use the following command to set the permissions and ownership:

chmod 755 /var/log/journal
chown systemd-journal:root /var/log/journal

Additionally, you should consider encrypting the journal data to protect sensitive information. You can use tools like LUKS to encrypt the journal directory.

Troubleshooting

If you encounter issues with the journal, you can use the journalctl command with the --status option to view the current journal status:

journalctl --status

This command will display the current journal status, including any errors or warnings. For more information on systemd’s journal, you can refer to the systemd documentation.