Logs: journalctl, /var/log, and dmesg

general / published / guide #34 / ≈ 4 min read

When something doesn't work on a Linux system, the answer is almost always in a log somewhere. The trouble is there are three different places those logs might live, depending on the program and the distro. This guide is the tour.

The three log neighbourhoods

  1. journalctl — logs collected by systemd. Almost everything on a modern Linux distro is here.
  2. /var/log/ — plain text log files, the older Unix way. Still used by some programs (web servers, mail servers, package managers).
  3. dmesg — the kernel's own log: hardware messages, driver complaints, filesystem warnings.

You'll consult all three eventually.

journalctl, the first place to look

If your distro uses systemd (Debian, Ubuntu, Mint, Manjaro, Arch, Fedora, openSUSE — basically anything except FreeBSD or a few minimalist Linuxes), journalctl is the universal log viewer.

The everyday recipes

journalctl -u sshd               # logs for one service
journalctl -u sshd -f            # follow in real time (like tail -f)
journalctl -u sshd -n 50         # last 50 lines
journalctl -u sshd --since "1 hour ago"
journalctl -u sshd --since today

journalctl -b                    # only logs since the current boot
journalctl -b -1                 # the PREVIOUS boot (great for post-crash forensics)
journalctl --list-boots          # all stored boots with timestamps

journalctl -k                    # kernel messages only (= dmesg)
journalctl -p err                # only errors (or warning, info, debug)
journalctl -p err -b             # only errors, only this boot

-u = unit, -f = follow, -n = last N lines, -b = boot, -k = kernel, -p = priority.

Search by text

journalctl -g "out of memory"                 # regex grep
journalctl --since "10 min ago" | grep ssh    # if you prefer pipes

Disk usage and cleanup

journalctl --disk-usage                       # how much disk does the journal use?
sudo journalctl --vacuum-time=2weeks           # delete anything older than 2 weeks
sudo journalctl --vacuum-size=500M             # keep only the most recent 500 MB

By default the journal grows up to ~10% of your filesystem. If it's eating disk, vacuum it.

/var/log — the classic text files

Even on a systemd system, some programs still write to /var/log/. What's there varies by distro and what you've installed. Common files:

  • /var/log/syslog or /var/log/messages — general system log (Debian/Ubuntu / many others)
  • /var/log/auth.log — logins, sudo, sshd auth (Debian/Ubuntu); /var/log/secure on Red Hat family
  • /var/log/kern.log — kernel ring buffer copied to disk
  • /var/log/dpkg.log / /var/log/pacman.log — package operations (Debian / Arch)
  • /var/log/Xorg.0.log — X server (graphical session)
  • /var/log/apache2/, /var/log/nginx/ — web server access and error logs

Read them with less (don't use cat on a large log):

sudo less /var/log/syslog
sudo tail -f /var/log/nginx/error.log
sudo grep "Failed password" /var/log/auth.log

In less: / to search, n/N for next/previous, G for end, q to quit.

Why are some log files gzipped?

You'll see syslog, syslog.1, syslog.2.gz, … That's log rotation: the current log is the unsuffixed one; older copies are renumbered and eventually compressed. Read a compressed log without unpacking it first:

sudo zless /var/log/syslog.2.gz
sudo zgrep error /var/log/syslog.*.gz

dmesg — the kernel's own diary

The Linux kernel keeps a ring buffer of messages: which devices it noticed at boot, drivers loading, hardware errors, USB hot-plug events. View it with:

sudo dmesg                       # full buffer
sudo dmesg -T                    # with human-readable timestamps
sudo dmesg -w                    # follow new messages (like tail -f)
sudo dmesg | grep -i nvidia      # filter for a specific driver

Same content is available via journalctl -k on systemd systems.

The first-look routine when something is broken

  1. systemctl --failed — anything obvious crashed?
  2. journalctl -b -p err — what errors fired since the last boot?
  3. If you know which service: journalctl -u thatservice -n 50.
  4. If it looks hardware-y: sudo dmesg -T | tail -50.
  5. If it looks login/auth: sudo tail /var/log/auth.log or journalctl _COMM=sudo.

One of these almost always reveals the culprit.

One quick trick for catching errors as they happen

Open two terminals or split your tmux. In one, run:

sudo journalctl -f

In the other, reproduce your problem. Log lines appear in the first terminal the moment they happen.

Capping the journal permanently

The --vacuum-* commands above are one-shot — they prune right now, but the journal can grow back tomorrow. To put a hard ceiling that systemd enforces on every write, edit /etc/systemd/journald.conf:

sudoedit /etc/systemd/journald.conf

Find the commented #SystemMaxUse= line and replace it with a real value, e.g.:

SystemMaxUse=200M

Then restart the journal and prune anything that is already over the new cap:

sudo systemctl restart systemd-journald
sudo journalctl --vacuum-size=200M

Verify:

journalctl --disk-usage

From now on the journal stays under the cap by itself — no more periodic cleanup runs needed.

← back to index