Troubleshooting Failed Mounts in Emergency Mode with systemd

Introduction to Emergency Mode

I’ve seen this go wrong when a Linux system encounters a critical issue during boot - it drops into emergency mode. This mode provides a minimal environment for troubleshooting and repair, allowing you to diagnose and fix issues that prevent the system from booting normally. In this article, we’ll explore how to troubleshoot failed mounts in emergency mode with systemd.

Understanding Emergency Mode

Emergency mode is a special boot target in systemd that provides a basic environment for troubleshooting. When a system enters emergency mode, it means that an error occurred during the boot process, and the system was unable to mount the root filesystem or other critical filesystems. The real trick is to understand that, in emergency mode, you’ll have access to a root shell, but many system services will not be started.

Identifying the Issue

To troubleshoot a failed mount in emergency mode, you’ll need to identify the cause of the issue. Don’t bother with guessing - start by checking the system logs for error messages related to the mount failure. You can use the journalctl command to view the system logs:

journalctl -xb

This command will show you the boot logs, including any error messages related to the mount failure. In practice, this is usually where I start when trying to diagnose a mount issue.

Checking Filesystem Integrity

One common cause of mount failures is filesystem corruption. I usually start with fsck to check the integrity of the filesystem:

fsck /dev/sda1

Replace /dev/sda1 with the device name of the filesystem you want to check. If fsck reports any errors, you may need to run it with the -y option to automatically repair the filesystem:

fsck -y /dev/sda1

This is where people usually get burned - forgetting to check the filesystem integrity can lead to more problems down the line.

Checking Mount Options

Another common cause of mount failures is incorrect mount options. You can check the mount options for a filesystem by looking at the /etc/fstab file:

cat /etc/fstab

This file contains a list of filesystems and their corresponding mount options. Check that the mount options for the failed filesystem are correct. It’s easy to overlook a simple mistake in the fstab file, but it can cause a lot of issues.

Using systemd to Troubleshoot

Systemd provides several tools for troubleshooting mount failures. You can use the systemctl command to check the status of the mount unit:

systemctl status mount-unit

Replace mount-unit with the name of the mount unit you want to check. This command will show you the status of the mount unit, including any error messages.

Repairing the Filesystem

If you’ve identified the cause of the mount failure, you can try to repair the filesystem. If the filesystem is corrupted, you may need to run fsck with the -y option to automatically repair it. If the mount options are incorrect, you can edit the /etc/fstab file to correct them.

Rebooting the System

Once you’ve repaired the filesystem or corrected the mount options, you can try to reboot the system:

systemctl reboot

This command will restart the system and attempt to boot normally.

Additional Resources

For more information on troubleshooting mount failures with systemd, you can refer to the systemd documentation. Additionally, you can check the kernel documentation for information on filesystems and mount options.

Best Practices

To avoid mount failures in the future, it’s a good idea to regularly check the integrity of your filesystems and ensure that your mount options are correct. You can also use tools like systemd-oomd to monitor your system’s memory usage and prevent out-of-memory errors.