Logs are important. They provide a wealth of information about what’s happening on your server, and when something goes wrong you’re likely to find useful information in the logs. In this article I briefly look at rsyslog but mainly focus on systemd-journald. I won’t cover querying the systemd journal – Please see checking logs using journalctl if you want to know how to get information from the journal.

Types of logs

There are three main types of logs on modern Linux systems:

  • Services can manage their own logs. For instance, Apache manages its own access and error log files.
  • There’s also rsyslogd. The daemon can collect messages from various programs, and it uses a configuration file (/etc/rsyslog.conf) to define logging rules. This makes it possible to for instance log messages to a database, perhaps on a different server.
  • Systems running systemd use systemd-journald. Logs are kept in a binary format and logs are queried using the above-mentioned journalctl utility.

rsyslog

On RHEL8-based systems rsyslog is configured to get messages from the systemd journal. This is done via the imjournal module, which is defined in the “modules” section of the /etc/rsyslog.conf file. By default, messages are written to the usual files in /var/log:

  • /var/log/messages stores general messages with a priority of info or higher, excluding mail-related messages.
  • /var/log/secure stores authentication messages, such as SSH logins.
  • /var/log/maillog is the mail server log for the Mail Delivery Agent (Dovecot on cPanel servers).
  • /var/log/cron lists cron jobs that were run.

As rsyslog gets its data from from the systemd journal these logs are more or less redundant. They are still useful though. By default systemd journald doesn’t permanently store logs – log entries don’t survive a reboot. The logs in /var/log are rotated, and therefore do survive a reboot.

journald log options

As said, by default the journal is cleared when a system is rebooted. This behaviour isn’t set in stone. In fact, journald is very configurable. The default options are listed in /etc/systemd/journald.conf and the defaults can be overridden by creating a file in /etc/systemd/journal.conf.d.

Storage options

The default storage option is auto. This option uses persistent logging if the directory /var/log/journal exists. So, to have persistent logs you can simply create the directory, and if you don’t want logs to survive a system reboot then you can delete the directory. If you want logs to be persistent then you can also use the persistent option. With this option, the /var/log/journal directory is created if doesn’t yet exist.

If you don’t want any journals then you can set the storage option to none. This option still lets rsyslog collect messages, so you will have the usual log files in /var/log. There’s also a volatile option, which stores data only in memory (under /run/log/journal).

Overriding defaults

To override the default settings you can create one or more files in /etc/systemd/journal.conf.d (you almost certainly need to create the journal.conf.d directory first). Here is an example of how to change the storage option to “persistent”:

# cat /etc/systemd/journal.conf.d/00-overrides.conf
[journal]
storage=persistent

After changing the default settings you need to restart the systemd-journald service:

# systemctl restart systemd-journald.service

Journal rotation

Journals can grow very large over time. The journal rotation is somewhat complicated, but the default settings do a good job of stopping logs from getting too large. There are four options that define rotation for persistent storage:

# grep ^#System /etc/systemd/journald.conf
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100

SystemMaxUse and SystemKeepFree define how much space may be used by journals. By default, SystemMaxUse is 10% of the file system. SystemKeepFree has a similar purpose; it defines how much space should be left for other uses (the default is 15%). Both settings also have a hard limit of 4GB. In other words, journals won’t consume more than 10% of the file system; leave 15% of space for other uses and don’t exceed 4GB.

SystemMaxFileSize and SystemMaxFiles define how large individual journals may be and how many files are kept at most. For SystemMaxFileSize, the default size is 1/8 of SystemMaxUse. As said, the settings are somewhat complicated.

It’s worth repeating that the options are for persistent storage only. If you use volatile storage instead then you can look at these equivalent options:

# grep ^#Runtime /etc/systemd/journald.conf
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100

Rotating and vacuuming logs

You can rotate logs manually using journalctl --rotate. This effectively starts a new journal under /var/log/journal (or /run/log/journal if you you use volatile storage).

There’s also a --disk-usage option which, as the name suggests, shows much space is used for journals:

# journalctl --disk-usage
Archived and active journals take up 600.0M in the file system.

And there are also options to remove old journal files. For instance, journalctl --vacuum-size lets you reduce the amount of disk space used by archived journal files:

# journalctl --vacuum-size=200M
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...09ed.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...9748.journal (40.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...859d.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...74aa.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...3053.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...dfd4.journal (40.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...748c.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...0105.journal (48.0M).
Vacuuming done, freed 368.0M of archived journals on disk.

# journalctl --disk-usage
Archived and active journals take up 232.0M on disk.

The main thing to note here is that the option only removes archived journals. It doesn’t touch active journals.

Further reading

As much as I dislike saying “RTFM”, the man pages for journald.conf and journalctl are a really good resource to learn more about systemd journals. Like all systemd man pages, the documentation is excellent:

If you prefer something less dry, there’s also good information on loggly.com.