Logging on Linux

7 May 2021

Logs are important. They provide a wealth of information about what is happening on your server, and when something goes wrong you are likely to find useful information in the logs. In this article I briefly look at rsyslog, which was the utility used most commonly until the advent of systemd. However, I will mainly focus on systemd-journald. I won’t cover querying the systemd journal – please see checking logs using journalctl if you want to know how to get information from the journal.

Types of logs

There are three main types of logs on modern Linux systems:

• Services can manage their own logs. For instance, Apache manages its own access and error log files.
• There is also rsyslogd. The daemon can collect messages from various programs, and it uses a configuration file (/etc/rsyslog.conf) to define logging rules. This makes it possible to for instance log messages to a database, perhaps on a different server.
• Systems running systemd use systemd-journald. Logs are kept in a binary format and logs are queried using the above-mentioned journalctl utility.

rsyslog

On RHEL8-based systems rsyslog is configured to get messages from the systemd journal. This is done via the imjournal module, which is defined in the “modules” section of the /etc/rsyslog.conf file. By default, messages are written to the usual files in /var/log:

• /var/log/messages stores general messages with a priority of info or higher, excluding mail-related messages.
• /var/log/secure stores authentication messages, such as SSH logins.
• /var/log/maillog is the mail server log for the Mail Delivery Agent (Dovecot on cPanel servers).
• /var/log/cron lists cron jobs that were run.

As rsyslog gets its data from from the systemd journal these logs are more or less redundant. They are still useful though. By default systemd journald doesn’t permanently store logs – log entries don’t survive a reboot. The logs in /var/log are rotated, and therefore do survive a reboot.

journald log options

As said, by default the journal is cleared when a system is rebooted. This behaviour isn’t set in stone. In fact, journald is very configurable. The default options are listed in /etc/systemd/journald.conf and the defaults can be overridden by creating one or more files in /etc/systemd/journal.conf.d.

Storage options

The default storage option is auto. This option uses persistent logging if the directory /var/log/journal exists. So, to have persistent logs you can simply create the directory, and if you don’t want logs to survive a system reboot then you can delete the directory. If you want logs to be persistent then you can also use the persistent option. With this option, the /var/log/journal directory is created if doesn’t yet exist.

If you don’t want any journals then you can set the storage option to none. This option still lets rsyslog collect messages, so you do still have the usual log files in /var/log. The final option to mention is volatile. With that option data is stored in memory only (under /run/log/journal).

Overriding defaults

To override the default settings you can create one or more files in /etc/systemd/journald.conf.d (you almost certainly need to create the journald.conf.d directory first). Here is an example of how to change the storage option to “persistent”:

# cat /etc/systemd/journald.conf.d/00-overrides.conf
[Journal]
Storage=persistent

After creating the file you need to restart the systemd-journald service:

# systemctl restart systemd-journald.service

The /var/log/journal is be created if it doesn’t already exist. You can run systemctl status systemd-journald to check if everything is logged to the directory. When you run the command as root the output includes the most recent log entries, and one of the entries shows which system journal is used – this should be a file under /var/log/journal. Alternatively, you can also run journalctl -u systemd-journald to view log entries for the service.

It is worth noting that the configuration files are case-sensitive. For instance, the [Journal] section name uses title-case. If you instead use lower-case (i.e. [journal]) then the section is ignored. You won’t get an error when you restart systemd-journald either. So, do always double-check that changes you make are picked up.

Journal rotation

Journals can grow very large over time. The journal rotation is somewhat complicated, but the default settings do a good job of stopping logs from getting too large. There are four options that define rotation for persistent storage:

# grep ^#System /etc/systemd/journald.conf
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100

SystemMaxUse and SystemKeepFree define how much space may be used by journals. By default, SystemMaxUse is 10% of the file system. SystemKeepFree has a similar purpose; it defines how much space should be left for other uses (the default is 15%). Both settings also have a hard limit of 4GB. In other words, journals won’t consume more than 10% of the file system; leave 15% of space for other uses and don’t exceed 4GB.

SystemMaxFileSize and SystemMaxFiles define how large individual journals may be and how many files are kept at most. For SystemMaxFileSize, the default size is 1/8 of SystemMaxUse. As said, the settings are somewhat complicated.

It’s worth repeating that the options are for persistent storage only. If you use volatile storage instead then you can look at these equivalent options:

# grep ^#Runtime /etc/systemd/journald.conf
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100

Rotating and vacuuming logs

You can rotate logs manually using journalctl --rotate. This effectively starts a new journal under /var/log/journal (or /run/log/journal if you you use volatile storage).

There’s also a --disk-usage option which, as the name suggests, shows much space is used for journals:

# journalctl --disk-usage
Archived and active journals take up 600.0M in the file system.

And there are also options to remove old journal files. For instance, journalctl --vacuum-size lets you reduce the amount of disk space used by archived journal files:

# journalctl --vacuum-size=200M
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...09ed.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...9748.journal (40.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...859d.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...74aa.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...3053.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...dfd4.journal (40.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...748c.journal (48.0M).
Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...0105.journal (48.0M).
Vacuuming done, freed 368.0M of archived journals on disk.

# journalctl --disk-usage
Archived and active journals take up 232.0M on disk.

The main thing to note here is that the option only removes archived journals. It doesn’t touch active journals.

Further reading

As much as I dislike saying “RTFM”, the man pages for journald.conf and journalctl are a really good resource to learn more about systemd journals. Like all systemd man pages, the documentation is excellent:

• journald.conf (5)
• journalctl (1)

If you prefer something less dry, there’s also good information on loggly.com.