Troubleshooting

Introduction to Troubleshooting Linux Boot Issues

When a Linux system fails to boot, it can be a frustrating experience. I’ve seen this go wrong when you’re relying on your system for critical tasks or services. Luckily, with the advancements in Linux and its ecosystem, troubleshooting boot issues has become more streamlined, thanks in part to the features and tools provided by systemd and the Linux kernel itself.

Understanding systemd’s Debug Shell

The real trick is to get insight into what’s going wrong during the boot process. systemd, the system and service manager for Linux, offers a debug shell that can be incredibly useful for this. To access the debug shell, you can modify the kernel parameters during boot. For example, to enable the debug shell, you can append the following to your kernel parameters:

Introduction to DNS Leaks

I’ve seen DNS leaks compromise even the most secure Linux setups - it’s a common issue that can expose your online activities. When running a small Linux server, ensuring the security and integrity of your DNS setup is crucial. A DNS leak occurs when your system sends DNS queries to an unintended DNS server, potentially revealing your browsing history. In this article, we’ll focus on troubleshooting DNS leaks on a small Linux server using systemd-resolved.

Introduction to Linux Rescue and Recovery

I’ve seen this go wrong when a Linux system becomes unbootable - it can be a real nightmare. Whether it’s a failed update, a misconfigured kernel, or a corrupted filesystem, having a reliable method for recovery is crucial. One of the most effective ways to rescue a borked Linux boot is by using a USB rescue drive in combination with the chroot command. This approach allows you to access and repair your system from a safe environment.

Introduction to Background Jobs

I’ve seen this go wrong when running commands in the background using the ampersand (&) symbol at the end of a command. The command keeps running even after the terminal is closed, which can be useful, but sometimes these background jobs consume excessive system resources or cause other issues. In practice, identifying and managing these rogue processes becomes essential to prevent system instability or crashes.

Identifying Rogue Processes with pgrep

The real trick is using the pgrep command to identify processes based on their name, user, or other attributes. For example, to find all processes running with the name “httpd”, you can use:

Introduction to Debugging Linux Network Connectivity

When dealing with network connectivity issues in Linux, I’ve found the ss command to be one of the most useful tools in my toolkit. ss stands for “socket statistics” and is used to dump socket statistics. It can display stats for PACKET sockets, TCP sockets, UDP sockets, DCCP sockets, RAW sockets, Unix domain sockets, and more. In practice, this command has helped me identify and resolve a wide range of network connectivity issues.

Introduction to Local Port Conflicts

I’ve seen this go wrong when running a multi-service Linux host - local port conflicts can be a real headache. These conflicts arise when two or more services attempt to bind to the same port, causing one or more of them to fail. To troubleshoot local port conflicts, I usually start with the ss command and nftables on a Linux system.

Understanding Port Conflicts

A port conflict occurs when a service tries to bind to a port that is already in use by another service. This can happen when multiple services are configured to use the same port, or when a service is not properly configured to use a unique port. Don’t bother with the traditional netstat command - the ss command is a more modern replacement.

Introduction to systemd-resolved

I’ve seen systemd-resolved cause its fair share of issues with split DNS and local hostnames, but it’s actually a powerful tool for managing DNS resolution on Linux systems. As part of the systemd ecosystem, it aims to improve the DNS resolution process and provide better integration with other systemd services.

Understanding Split DNS

Split DNS, or split-horizon DNS, is a technique used to provide different DNS responses based on the client’s location. This is commonly used in environments where internal and external DNS zones need to be separated - think of a company with an internal DNS zone for its internal network and a separate external DNS zone for its public-facing services. Don’t bother with split DNS if you don’t need it, but if you do, systemd-resolved can be configured to handle it.

Introduction to Emergency Mode

I’ve seen this go wrong when a Linux system encounters a critical issue during boot - it drops into emergency mode. This mode provides a minimal environment for troubleshooting and repair, allowing you to diagnose and fix issues that prevent the system from booting normally. In this article, we’ll explore how to troubleshoot failed mounts in emergency mode with systemd.

Understanding Emergency Mode

Emergency mode is a special boot target in systemd that provides a basic environment for troubleshooting. When a system enters emergency mode, it means that an error occurred during the boot process, and the system was unable to mount the root filesystem or other critical filesystems. The real trick is to understand that, in emergency mode, you’ll have access to a root shell, but many system services will not be started.

Introduction to Initramfs Recovery

I’ve seen this go wrong when a Linux system fails to boot due to a broken initramfs - it can be a real headache, especially if you rely on your system for daily work or critical services. The recent updates to the Linux kernel and initramfs tools may have introduced compatibility issues for some users, making it crucial to understand how to recover from such failures.

Introduction to systemd’s Restart Policy

I’ve seen this go wrong when a service is not properly configured - systemd’s ability to automatically restart services that fail or terminate unexpectedly can be a double-edged sword. On one hand, it helps maintain system stability and availability. On the other hand, if not configured correctly, it can lead to service thrashing, where a service is repeatedly restarted in a short period, potentially causing more harm than good.