Logs are important. They provide a wealth of information about what is happening on your server, and when something goes wrong you are likely to find useful information in the logs. In this article I briefly look at
rsyslog, which was the utility used most commonly until the advent of systemd. However, I will mainly focus on
systemd-journald. I won’t cover querying the systemd journal – please see checking logs using journalctl if you want to know how to get information from the journal.
There are three main types of logs on modern Linux systems:
rsyslogd. The daemon can collect messages from various programs, and it uses a configuration file (/etc/rsyslog.conf) to define logging rules. This makes it possible to for instance log messages to a database, perhaps on a different server.
systemd-journald. Logs are kept in a binary format and logs are queried using the above-mentioned
On RHEL8-based systems
rsyslog is configured to get messages from the systemd journal. This is done via the imjournal module, which is defined in the “modules” section of the /etc/rsyslog.conf file. By default, messages are written to the usual files in /var/log:
rsyslog gets its data from from the systemd journal these logs are more or less redundant. They are still useful though. By default
systemd journald doesn’t permanently store logs – log entries don’t survive a reboot. The logs in /var/log are rotated, and therefore do survive a reboot.
As said, by default the journal is cleared when a system is rebooted. This behaviour isn’t set in stone. In fact,
journald is very configurable. The default options are listed in /etc/systemd/journald.conf and the defaults can be overridden by creating one or more files in /etc/systemd/journal.conf.d.
The default storage option is auto. This option uses persistent logging if the directory /var/log/journal exists. So, to have persistent logs you can simply create the directory, and if you don’t want logs to survive a system reboot then you can delete the directory. If you want logs to be persistent then you can also use the persistent option. With this option, the /var/log/journal directory is created if doesn’t yet exist.
If you don’t want any journals then you can set the storage option to none. This option still lets
rsyslog collect messages, so you do still have the usual log files in /var/log. The final option to mention is volatile. With that option data is stored in memory only (under /run/log/journal).
To override the default settings you can create one or more files in /etc/systemd/journald.conf.d (you almost certainly need to create the journald.conf.d directory first). Here is an example of how to change the storage option to “persistent”:
# cat /etc/systemd/journald.conf.d/00-overrides.conf [Journal] Storage=persistent
After creating the file you need to restart the
# systemctl restart systemd-journald.service
The /var/log/journal is be created if it doesn’t already exist. You can run
systemctl status systemd-journald to check if everything is logged to the directory. When you run the command as root the output includes the most recent log entries, and one of the entries shows which system journal is used – this should be a file under /var/log/journal. Alternatively, you can also run
journalctl -u systemd-journald to view log entries for the service.
It is worth noting that the configuration files are case-sensitive. For instance, the [Journal] section name uses title-case. If you instead use lower-case (i.e. [journal]) then the section is ignored. You won’t get an error when you restart
systemd-journald either. So, do always double-check that changes you make are picked up.
Journals can grow very large over time. The journal rotation is somewhat complicated, but the default settings do a good job of stopping logs from getting too large. There are four options that define rotation for persistent storage:
# grep ^#System /etc/systemd/journald.conf #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #SystemMaxFiles=100
SystemMaxUse and SystemKeepFree define how much space may be used by journals. By default, SystemMaxUse is 10% of the file system. SystemKeepFree has a similar purpose; it defines how much space should be left for other uses (the default is 15%). Both settings also have a hard limit of 4GB. In other words, journals won’t consume more than 10% of the file system; leave 15% of space for other uses and don’t exceed 4GB.
SystemMaxFileSize and SystemMaxFiles define how large individual journals may be and how many files are kept at most. For SystemMaxFileSize, the default size is 1/8 of SystemMaxUse. As said, the settings are somewhat complicated.
It’s worth repeating that the options are for persistent storage only. If you use volatile storage instead then you can look at these equivalent options:
# grep ^#Runtime /etc/systemd/journald.conf #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #RuntimeMaxFiles=100
You can rotate logs manually using
journalctl --rotate. This effectively starts a new journal under /var/log/journal (or /run/log/journal if you you use volatile storage).
There’s also a
--disk-usage option which, as the name suggests, shows much space is used for journals:
# journalctl --disk-usage Archived and active journals take up 600.0M in the file system.
And there are also options to remove old journal files. For instance,
journalctl --vacuum-size lets you reduce the amount of disk space used by archived journal files:
# journalctl --vacuum-size=200M Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...09ed.journal (48.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...9748.journal (40.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...859d.journal (48.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...74aa.journal (48.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...3053.journal (48.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...dfd4.journal (40.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...748c.journal (48.0M). Deleted archived journal /run/log/journal/cc25...fabc/system@692f1...0105.journal (48.0M). Vacuuming done, freed 368.0M of archived journals on disk. # journalctl --disk-usage Archived and active journals take up 232.0M on disk.
The main thing to note here is that the option only removes archived journals. It doesn’t touch active journals.
As much as I dislike saying “RTFM”, the man pages for
journalctl are a really good resource to learn more about systemd journals. Like all systemd man pages, the documentation is excellent:
If you prefer something less dry, there’s also good information on loggly.com.