rsync scheduled jobs that stall on authentication: a journalctl‑first approach
When an rsync job launched by a systemd timer stops at the authentication step, the first place to look is the journal. The logs contain every attempt to open a connection, every key exchange, and every error message that the ssh daemon emits. With the right filters you can turn a vague “stalled” symptom into a concrete failure reason.
1. Know the unit that runs your rsync
Most modern distributions ship a template unit [email protected] that is started by a timer such as [email protected]. The instance name encodes the target host, e.g. [email protected]. Check the status:
systemctl status [email protected]
The output shows the unit’s name, its last start time, and any immediate exit codes. If the unit never reaches the “running” state, the journal will contain the reason.
2. Pull the rsync logs with journalctl
Once you know the unit, fetch its messages:
journalctl -u [email protected] -p err -S
-ulimits the output to the unit.-p errshows only error‑level entries, which is usually enough for authentication problems.-Ssorts by timestamp, newest first.
If the unit is a simple wrapper that calls rsync directly, the journal will contain the command line and any output that rsync writes to stdout or stderr. Look for lines that start with rsync: or rsync: auth: .
3. Correlate with the sshd logs
rsync uses SSH for remote authentication. The SSH daemon logs authentication events under the sshd unit. To see what happened on the remote side, run:
journalctl -u sshd -p err -S --since "2026-10-05 00:00" --until "2026-10-05 02:00"
Adjust the time window to match the timer’s schedule. Search for:
sshd[12345]: Authentication failure
sshd[12345]: Connection closed by remote host
If the client side was waiting for a password prompt, the server will log “Connection closed by remote host” or “Connection closed by 192.0.2.5 port 22”.
4. Use tags and priority filters
journalctl can filter by tag (-t) or priority (-p). If your rsync unit sets a tag, e.g. -t rsync, you can isolate it:
journalctl -t rsync -p err -S
If you’re not sure whether the unit uses a tag, search the unit file:
grep -i 'Tag=' /etc/systemd/system/[email protected]
5. Inspect the key exchange
A common cause of stalls is a missing or unreadable private key. The sshd log will contain:
sshd[12345]: Authentication refused: bad ownership or modes for private key
Check the key’s permissions on the client:
ls -l ~/.ssh/id_rsa
The file must be -rw------- (600). If the key is in a directory that is not owned by the user, ssh will refuse to use it.
6. Verify the SSH agent
If the rsync job relies on an SSH agent, the agent must be running in the same user session that starts the timer. Systemd can start an agent with a Service unit:
[Unit]
Description=SSH Agent for rsync
After=network.target
[Service]
Type=forking
ExecStart=/usr/bin/ssh-agent -s
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
Then the rsync unit can reference the socket:
Environment=SSH_AUTH_SOCK=/run/user/1000/ssh-agent.socket
If the agent is not running, the journal will show:
rsync[12345]: auth: cannot open private key
7. Check host key verification
When connecting to a new host, ssh will ask to confirm the host key. In a non‑interactive job, this will block indefinitely. The sshd log will show:
sshd[12345]: Connection closed by 192.0.2.5 port 22
On the client side, you’ll see:
rsync[12345]: auth: cannot open private key
or
rsync[12345]: auth: connection closed
The usual fix is to pre‑populate the known‑hosts file or disable strict host key checking in the rsync command line with -e "ssh -o StrictHostKeyChecking=no"—but that’s a trade‑off you should weigh carefully.
If you’ve followed these steps and the logs still don’t explain the stall, double‑check that the timer actually fires by looking at the timer’s status:
systemctl list-timers | grep rsync
Sometimes the timer is disabled, or the unit file has a typo that prevents it from starting. Once the timer is firing, the journal will give you the missing piece of the puzzle.
TAGS: rsync, systemd, ssh, troubleshooting, journalctl
See also
- How to Stop Docker‑Compatible Containers from Writing Sensitive Files to the Host with Rootless Mode
- Configuring systemd‑resolved to Use a Custom DNS Server Only on the VPN Interface
- Recovering from emergency mode after a bad /etc/fstab entry on Ubuntu 24.04: a single‑command root shell solution
- How to Give the Backup User Read‑Only Access to /var/log via ACLs Without Granting sudo
- Rootless Podman on a Raspberry Pi – Solving the “Cannot Bind Mount /home/user/data” Permission Issue