Systems that use systemd write log entries for the kernel and various services to a journal. These log entries are stored in a binary format and can be inspected via the journalctl
utility.
journalctl
doesn’t replace all log files. If you are debugging, say, an error with sending emails you still want to inspect the /var/log/exim_mainlog file. Similarly, for PHP errors the error_log files are still the place to go. You are most likely to use journalctl
when the output of a systemctl status
command is reporting an error.
As said, journal entries are stored in a binary format. Each entry has a large number of fields that can be displayed in several formats. By default, log entries look much like entries you find in traditional syslog log files, such as /var/log/messages. Here is an example of a journalctl
entry:
Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128
And here is the same entry displayed in JSON format:
{ "__CURSOR" : "s=a285c16fef31c7;i=b82;b=8d04cfd2dd066c;m=c5e8f;t=59341f0;x=11a8f0", "__REALTIME_TIMESTAMP" : "1569283177914864", "__MONOTONIC_TIMESTAMP" : "53113740943", "_BOOT_ID" : "8d04cfd2dd9744b888u8d3f9cb52066c", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_SYSTEMD_SLICE" : "system.slice", "_MACHINE_ID" : "21a223f3763983e62f89dc6a5b90c0d1", "_HOSTNAME" : "server.example.net", "_CAP_EFFECTIVE" : "1fffffffff", "_TRANSPORT" : "syslog", "SYSLOG_FACILITY" : "10", "SYSLOG_IDENTIFIER" : "sshd", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_SELINUX_CONTEXT" : "system_u:system_r:sshd_t:s0-s0:c0.c1023", "_CMDLINE" : "sshd: unknown [priv]", "SYSLOG_PID" : "17255", "MESSAGE" : "Invalid user admin from 59.148.43.97 port 54128", "_PID" : "17255", "_SOURCE_REALTIME_TIMESTAMP" : "1569283177913968" }
As you can see, the entry contains quite a few fields, including a boot ID (_BOOT_ID
) and priority (PRIORITY
). You can use these fields to filter the data, which is what we will look at now.
The --since
and --until
options lets you specify a timestamp in the format YYYY-MM-DD HH:MM:SS
. You don’t have to include the time, and if you do include the time you may leave out the seconds (i.e. you can use HH:MM
):
journalctl --since="2019-09-26" --until="2019-09-26 11:00"
You can also use strings such as “yesterday”, “today” and “10m ago”.
# journalctl --since "10m ago"
The -u
option is used to filter a unit. The unit is the systemd unit name. So, to return log entries for sshd you can use the following:
# journalctl -u sshd.service -- Logs begin at Mon 2019-09-23 10:14:25 BST, end at Thu 2019-09-26 12:03:22 BST. -- Sep 23 10:14:45 server.example.net sshd[1409]: Server listening on 0.0.0.0 port 22. Sep 23 10:14:45 server.example.net sshd[1409]: Server listening on :: port 22. Sep 23 10:15:26 server.example.net sshd[8823]: Invalid user vbox from 5.39.95.92 port 60978 Sep 23 10:15:26 server.example.net sshd[8823]: input_userauth_request: invalid user vbox [preauth] Sep 23 10:15:26 server.example.net sshd[8823]: Received disconnect from 5.39.95.92 port 60978:11: Normal Shutdown, Thank you for playing [preauth] ...
When we looked at the JSON output we mentioned that each journal entry has a priority field. The priority number indicates the importance of the message. A priority of <i>0</i> means we got an emergency, while <i>7</i> is a debug message:
The -p
option lets you filter messages with a particular priority. If, say, you want to see all errors you can use either of these commands:
# journalctl -p 3
# journalctl -p err
You are most likely to use priority option to suppress less relevant messages. If you want to view log entries with a priority of err or greater you can add the -b
option. This will return entries with a priority of err, crit, alert or emerg.
# journalctl -p err -b -- Logs begin at Mon 2019-09-23 10:14:25 BST, end at Thu 2019-09-26 12:20:01 BST. -- Sep 23 20:21:29 server.example.net sshd[14834]: error: maximum authentication attempts exceeded for root from 180.126.50.52 port 5166 ssh2 [preauth] Sep 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth] ...
The output of journalctl
includes the name of the service followed by the process ID (in square brackets). You can use the _PID
option to get all entries for a specific process ID:
# journalctl _PID=17255 -- Logs begin at Mon 2019-09-23 10:14:25 BST, end at Thu 2019-09-26 12:30:01 BST. -- Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128 Sep 24 00:59:37 server.example.net sshd[17255]: input_userauth_request: invalid user admin [preauth] Sep 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth] Sep 24 00:59:39 server.example.net sshd[17255]: Disconnecting: Too many authentication failures [preauth]
You can monitor logs as entries are being written using the -f
option. This works exactly like tail -f
. Of course, you can filter the unit you want to check at the same time. For example, here we are following entries for Dovecot:
# journalctl -f -u dovecot.service
And now that we have mentioned tail
we should also highlight the -n
option. As you might have guessed, this option works exactly like tail -n
– it displays the last x entries. For instance, to view the last three entries for the Dovecot service you can use:
# journalctl -u dovecot.service -n 3
Although the filtering options we covered are quite flexible they don’t necessarily return the data you want. For instance, when you are looking in to SSH login attempts you might want to filter entries based on the IP address shown in the MESSAGE
field. On Centos 7 and CloudLinux 7 servers you can do that by piping the output of journalctl
to a utility like grep
:
# journalctl -u sshd.service | grep 59.148.43.97 Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128 Sep 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth]
Centos 8 and CloudLinux 8 ship with a version of journalctl
that includes the -g
(or --grep
) option, which lets you grep the message field. That means that you can instead use this command:
# journalctl -u sshd.service -g 59.148.43.97
This article aimed to give a brief overview of journalctl
. There are many topic we haven’t covered, including storage options (journalctl logs aren’t necessarily persistent, which means that log entries don’t survive system reboots) and the various output formats. If you want to learn more, the official documentation can be found at freedesktop.org.