Turning journalctl into a Grafana Dashboard with Loki for a Home‑Lab Server
Systemd’s journal is a reliable source of operational data, but raw journalctl output is hard to sift through at scale. Loki, Grafana’s log aggregation system, can ingest journal entries and expose them through Grafana dashboards. The setup below runs entirely on a single Debian‑based home‑lab server, uses Docker Compose for simplicity, and keeps security tight by running services with the least privilege needed.
1. Prerequisites
- Debian 12 (or any distro with systemd ≥ 246)
- Docker Engine 27+ and Docker Compose 2+
sudoaccess
sudo apt update
sudo apt install -y docker.io docker-compose
sudo systemctl enable --now docker
2. Deploy Loki, Promtail, and Grafana
Create a directory for the stack and a docker‑compose.yml:
mkdir ~/homelab-logs
cd ~/homelab-logs
cat > docker-compose.yml <<'EOF'
version: '3.8'
services:
loki:
image: grafana/loki:2.9.4
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki-data:/data
- ./loki-config.yaml:/etc/loki/local-config.yaml
ports:
- "3100:3100"
promtail:
image: grafana/promtail:2.9.4
command: -config.file=/etc/promtail/promtail.yaml
volumes:
- /var/log:/var/log
- /run/systemd/journal:/run/systemd/journal
- /etc/machine-id:/etc/machine-id:ro
- ./promtail.yaml:/etc/promtail/promtail.yaml
cap_add:
- SYS_PTRACE # required for systemd‑journal read
security_opt:
- no-new-privileges:true
grafana:
image: grafana/grafana:10.4.0
environment:
- GF_SECURITY_ADMIN_PASSWORD=changeme
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
volumes:
grafana-data:
EOF
Loki configuration (loki-config.yaml)
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
log_format: logfmt
http_server_read_timeout: 30s
http_server_write_timeout: 30s
http_server_idle_timeout: 60s
http_server_max_header_size: 8192
http_server_max_response_size: 10MiB
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
storage_config:
boltdb_shipper:
active_index_directory: /data/index
cache_location: /data/cache
shared_store: filesystem
filesystem:
directory: /data/chunks
schema_config:
configs:
- from: 2020-10-15
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
Promtail configuration (promtail.yaml)
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: systemd-journal
journal:
json: true
max_age: 24h
path: /run/systemd/journal
tags:
- _SYSTEMD_UNIT
relabel_configs:
- source_labels: ['_SYSTEMD_UNIT']
target_label: 'job'
3. Spin it up
docker compose up -d
You should now have:
- Loki listening on
http://localhost:3100 - Promtail sending logs to Loki
- Grafana available at
http://localhost:3000(admin / changeme)
4. Wire Grafana to Loki
- Log in to Grafana.
- Go to Configuration → Data Sources → Add data source.
- Pick Loki, set URL to
http://loki:3100, and click Save & Test.
If you hit a firewall, expose the Loki port on the host or use a reverse proxy.
5. Build a simple dashboard
Create a new dashboard, add a Logs panel, and point it at your Loki data source. A quick query to see the last 5 minutes of logs from the sshd unit:
{job="sshd"} |~ "Accepted" | json | line_format "{{ .message }}"
You can tweak the query to filter by severity, service, or any other tag that Promtail extracts. The real trick is to let Promtail add useful labels (like _SYSTEMD_UNIT, hostname, __path__) so you can slice and dice later.
6. Harden the stack
- Least‑privilege Promtail – we already dropped
SYS_PTRACEand disabled new privileges. - Isolated data volumes – Loki and Grafana each write to their own Docker volumes.
- Disable auth – Loki is unauthenticated in this example, which is fine for a home‑lab. For a production setup, enable
auth_enabled: trueand add a user. - Regular backups – the Loki data directory is a Docker volume; back it up with
docker cpor a snapshot tool.
7. Keep it tidy
If you ever need to prune old logs, run:
docker exec loki loki -delete-older-than 30d
Or adjust reject_old_samples_max_age in the config for automatic eviction.
TL;DR – Drop Promtail into your Docker stack, point it at /run/systemd/journal, and let Loki surface the logs in Grafana. You get a searchable, visual view of your system logs without pulling journalctl into a shell every time.
Tags
linux, monitoring, grafana, loki, security
See also
- How to Fix a “Failed to Mount /home” Error in Emergency Mode Without Rebuilding initramfs
- Using find … -print0 | xargs -0 to safely process filenames with spaces and newlines
- Removing the ‘Permission Denied’ Error When Editing `/etc/hosts` as a Non‑Root User
- Using systemd Socket Activation to Keep a Lightweight HTTP Server Idle Until the First Request
- Fixing the “Failed to mount /home” error caused by a missing UUID in /etc/fstab