Turning journalctl into a Grafana Dashboard with Loki for My Home‑Lab Server

Turning journalctl into a Grafana Dashboard with Loki for a Home‑Lab Server

Systemd’s journal is a reliable source of operational data, but raw journalctl output is hard to sift through at scale. Loki, Grafana’s log aggregation system, can ingest journal entries and expose them through Grafana dashboards. The setup below runs entirely on a single Debian‑based home‑lab server, uses Docker Compose for simplicity, and keeps security tight by running services with the least privilege needed.

1. Prerequisites

  • Debian 12 (or any distro with systemd ≥ 246)
  • Docker Engine 27+ and Docker Compose 2+
  • sudo access
sudo apt update
sudo apt install -y docker.io docker-compose
sudo systemctl enable --now docker

2. Deploy Loki, Promtail, and Grafana

Create a directory for the stack and a docker‑compose.yml:

mkdir ~/homelab-logs
cd ~/homelab-logs
cat > docker-compose.yml <<'EOF'
version: '3.8'

services:
  loki:
    image: grafana/loki:2.9.4
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-data:/data
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:2.9.4
    command: -config.file=/etc/promtail/promtail.yaml
    volumes:
      - /var/log:/var/log
      - /run/systemd/journal:/run/systemd/journal
      - /etc/machine-id:/etc/machine-id:ro
      - ./promtail.yaml:/etc/promtail/promtail.yaml
    cap_add:
      - SYS_PTRACE   # required for systemd‑journal read
    security_opt:
      - no-new-privileges:true

  grafana:
    image: grafana/grafana:10.4.0
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
volumes:
  grafana-data:
EOF

Loki configuration (loki-config.yaml)

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info
  log_format: logfmt
  http_server_read_timeout: 30s
  http_server_write_timeout: 30s
  http_server_idle_timeout: 60s
  http_server_max_header_size: 8192
  http_server_max_response_size: 10MiB
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
storage_config:
  boltdb_shipper:
    active_index_directory: /data/index
    cache_location: /data/cache
    shared_store: filesystem
  filesystem:
    directory: /data/chunks
schema_config:
  configs:
    - from: 2020-10-15
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

Promtail configuration (promtail.yaml)

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: systemd-journal
    journal:
      json: true
      max_age: 24h
      path: /run/systemd/journal
      tags:
        - _SYSTEMD_UNIT
    relabel_configs:
      - source_labels: ['_SYSTEMD_UNIT']
        target_label: 'job'

3. Spin it up

docker compose up -d

You should now have:

  • Loki listening on http://localhost:3100
  • Promtail sending logs to Loki
  • Grafana available at http://localhost:3000 (admin / changeme)

4. Wire Grafana to Loki

  1. Log in to Grafana.
  2. Go to Configuration → Data Sources → Add data source.
  3. Pick Loki, set URL to http://loki:3100, and click Save & Test.

If you hit a firewall, expose the Loki port on the host or use a reverse proxy.

5. Build a simple dashboard

Create a new dashboard, add a Logs panel, and point it at your Loki data source. A quick query to see the last 5 minutes of logs from the sshd unit:

{job="sshd"} |~ "Accepted" | json | line_format "{{ .message }}"

You can tweak the query to filter by severity, service, or any other tag that Promtail extracts. The real trick is to let Promtail add useful labels (like _SYSTEMD_UNIT, hostname, __path__) so you can slice and dice later.

6. Harden the stack

  • Least‑privilege Promtail – we already dropped SYS_PTRACE and disabled new privileges.
  • Isolated data volumes – Loki and Grafana each write to their own Docker volumes.
  • Disable auth – Loki is unauthenticated in this example, which is fine for a home‑lab. For a production setup, enable auth_enabled: true and add a user.
  • Regular backups – the Loki data directory is a Docker volume; back it up with docker cp or a snapshot tool.

7. Keep it tidy

If you ever need to prune old logs, run:

docker exec loki loki -delete-older-than 30d

Or adjust reject_old_samples_max_age in the config for automatic eviction.


TL;DR – Drop Promtail into your Docker stack, point it at /run/systemd/journal, and let Loki surface the logs in Grafana. You get a searchable, visual view of your system logs without pulling journalctl into a shell every time.


Tags

linux, monitoring, grafana, loki, security


See also