When /usr Shows 100 % Used but df Says Space Is Fine: Spotting Inode Exhaustion

Why Inode Exhaustion Happens

On a busy server you’ll often see df still showing plenty of space, but df -i screaming 100 % inode usage on a mount like /usr. The kernel will happily refuse to create any more files, even though there’s room on disk. The culprit is usually a deluge of tiny files—think logs, temp data, or a mis‑configured service that never cleans up.

Spotting the Problem Quickly

# Space vs. inodes
df -h /usr
df -i /usr

If the first command reports 20 GB free but the second is 100 % used, you’ve hit an inode shortage. Now find the directory that’s eating them.

# Count files per directory
sudo find /usr -xdev -type f -printf '%h\n' | \
    awk -F/ '{print $NF}' | sort | uniq -c | sort -nr | head

The list will point you to the worst offenders—usually something like /usr/lib/tmp or /usr/share/doc where a build system or package manager churns out many small objects.

Common Sources of Tiny Files

Source	Typical Path	Why it explodes
Package builds	`/usr/lib/build`	`make` generates thousands of object files
Package managers	`/var/cache/apt/archives`	Unpacked packages leave a lot of small files
Log rotation mis‑config	`/var/log`	Old logs never get pruned
Temp directories	`/tmp`, `/var/tmp`	Services that forget to clean up

Cleaning Up

Remove old logs

sudo journalctl --vacuum-size=200M
sudo find /var/log -type f -mtime +30 -delete

Clear package cache

sudo apt-get clean          # Debian/Ubuntu
sudo dnf clean all          # RHEL/CentOS

Delete build artefacts

sudo find /usr/lib/build -type f -delete

Use tmpfs for short‑lived data
```
sudo mount -t tmpfs -o size=512M tmpfs /usr/lib/tmp
```
Persist the mount in /etc/fstab so it comes back after reboot.
Automate cleanup with systemd-tmpfiles
Create /etc/tmpfiles.d/usr-lib.conf:
```
d /usr/lib/tmp 0755 root root 10d
```
This will delete files older than 10 days automatically. See systemd-tmpfiles(5) on systemd.io.

Preventing Future Exhaustion

Adjust inode density when you format a new filesystem. Ext4 defaults to one inode per 16 kB of data. If you’re going to store lots of tiny files, bump the density:
```
mkfs.ext4 -i 4096 /dev/sdx1
```
You’ll waste a bit of space on large files, but you’ll gain the ability to hold more small ones.
Set inode quotas on critical mounts. Add quota to /etc/fstab and run edquota -u <user>. Quotas keep a single user or service from swallowing all the inodes.
Rotate and purge logs with logrotate. The default Debian config keeps a limited number of rotated logs. Edit /etc/logrotate.d/* to match your retention policy. More detail is on debian.org.
Use tmpfs for /tmp and /var/tmp on systems that churn a lot of temporary files. That way the inode pressure stays in RAM.

Security Implications

Inode exhaustion is a classic denial‑of‑service vector. An attacker who can flood a filesystem with small files can lock out legitimate users. Mitigations:

Limit write access to /tmp and similar dirs with chmod 1777 and tmpfs mounts.
Apply quotas to stop a single process from draining all inodes.
Keep log rotation tight; unpurged logs not only waste space but can leak sensitive data.

Quick Checklist

df -i shows 100 % on /usr?
Identify heavy directories with find | awk.
Clean logs, caches, build artefacts.
Use tmpfs for transient data.
Adjust inode density if many small files are expected.
Enable quotas on critical mounts.
Verify logrotate and systemd-tmpfiles are configured.

By checking inode usage regularly and applying these practices, you keep /usr (and any other mount) healthy and avoid silent denial‑of‑service conditions.