Finding the Most Frequent Syslog Lines in /var/log/syslog with awk, sort, and head

Why you need it

Syslog is the first place you look when something goes wrong. In a busy server or a home lab, the same message can appear thousands of times a day. Knowing which lines dominate the log helps spot misconfigurations, repeated failures, or malicious activity that repeats silently. A quick frequency count is a low‑overhead sanity check that can turn a noisy log into a focused list of suspects.

Quick recipe

awk '{print $0}' /var/log/syslog | \
  sort | \
  uniq -c | \
  sort -nr | \
  head -n 20

The pipeline does three things:

1. awk '{print $0}' emits every line unchanged. It’s a no‑op that keeps the command chain uniform and lets us insert filters later without breaking the flow.
2. sort arranges the lines lexicographically; uniq -c then collapses adjacent duplicates and prefixes a count.
3. A second sort -nr orders the counts numerically in descending order.
4. head -n 20 keeps the output manageable.

The result is a list of the 20 most frequent syslog entries, each prefixed by how many times it appeared in the current file.

Tweaking the scope

• Exclude timestamps
  If you’re interested in the message body only, strip the first 15 characters (typical timestamp format) before sorting:

  awk '{sub(/^[^ ]+ [^ ]+ /, "", $0); print}' /var/log/syslog | \
    sort | uniq -c | sort -nr | head
  

• Case‑insensitive
  Add LC_ALL=C to the sort command or use awk’s tolower().

• Search a rotated archive
  Syslog is usually rotated by logrotate. To include the last 7 days, combine zcat with the pipeline:

  zcat /var/log/syslog.*.gz | awk … | sort … | head
  

Performance notes

For a 10 GB syslog, the pipeline can take several minutes on a modest CPU. The bottleneck is the two full passes over the data (sort and uniq). If you need a faster, approximate count, consider awk’s associative array:

awk '{cnt[$0]++} END{for (l in cnt) print cnt[l], l}' /var/log/syslog | \
  sort -nr | head

This keeps everything in memory, so it fails on very large logs unless you have enough RAM. For a 10 GB file, you’ll need at least 10 GB of free memory, which is rarely the case on a production host.

Security & hygiene

• File permissions
  /var/log/syslog is usually owned by root:adm with 640 permissions. A user who can read the file can see potentially sensitive information (e.g., passwords in misconfigured authentication attempts). Keep the log readable only by trusted groups, and rotate it regularly to limit exposure.

• Log integrity
  If you’re hardening the system, consider signing logs or using a dedicated log‑collector that verifies integrity. A tampered log can hide an attacker’s trail.

• Avoid leaking in scripts
  When piping logs to other tools, be careful not to expose the data to processes that run as less‑privileged users. Use sudo sparingly and keep the pipeline minimal.

See also

• Preventing Disk Space Woes on a Home NAS by Configuring Logrotate for Docker Container Logs
• Keep a 1 GB VPS from dying: a single cron job that purges /var/log/syslog after 30 days
• When firewalld blocks HTTP traffic even though port 80 is open: the services vs ports trap
• Stuck in emergency mode after adding an NTFS mount to /etc/fstab: how to fix it
• How to Stop Podman From Creating 0777 Permissions on Bind‑Mounted Directories When Running as a Non‑Root User