Checking logs using journalctl

Systems that use systemd write log entries for the kernel and various services to a journal. These log entries are stored in a binary format and can be inspected via the journalctl utility.

journalctl doesn’t replace all log files. If you are debugging, say, an error with sending emails you still want to inspect the /var/log/exim_mainlog file. Similarly, for PHP errors the error_log files are still the place to go. You are most likely to use journalctl when the output of a systemctl status command is reporting an error.

Filtering options

As said, journal entries are stored in a binary format. Each entry has a large number of fields that can be displayed in several formats. By default, log entries look much like entries you find in traditional syslog log files, such as /var/log/messages. Here is an example of a journalctl entry:

Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128

And here is the same entry displayed in JSON format:

{
        "__CURSOR" : "s=a285c16fef31c7;i=b82;b=8d04cfd2dd066c;m=c5e8f;t=59341f0;x=11a8f0",
        "__REALTIME_TIMESTAMP" : "1569283177914864",
        "__MONOTONIC_TIMESTAMP" : "53113740943",
        "_BOOT_ID" : "8d04cfd2dd9744b888u8d3f9cb52066c",
        "PRIORITY" : "6",
        "_UID" : "0",
        "_GID" : "0",
        "_SYSTEMD_SLICE" : "system.slice",
        "_MACHINE_ID" : "21a223f3763983e62f89dc6a5b90c0d1",
        "_HOSTNAME" : "server.example.net",
        "_CAP_EFFECTIVE" : "1fffffffff",
        "_TRANSPORT" : "syslog",
        "SYSLOG_FACILITY" : "10",
        "SYSLOG_IDENTIFIER" : "sshd",
        "_COMM" : "sshd",
        "_EXE" : "/usr/sbin/sshd",
        "_SYSTEMD_CGROUP" : "/system.slice/sshd.service",
        "_SYSTEMD_UNIT" : "sshd.service",
        "_SELINUX_CONTEXT" : "system_u:system_r:sshd_t:s0-s0:c0.c1023",
        "_CMDLINE" : "sshd: unknown [priv]",
        "SYSLOG_PID" : "17255",
        "MESSAGE" : "Invalid user admin from 59.148.43.97 port 54128",
        "_PID" : "17255",
        "_SOURCE_REALTIME_TIMESTAMP" : "1569283177913968"
}

As you can see, the entry contains quite a few fields, including a boot ID (_BOOT_ID) and priority (PRIORITY). You can use these fields to filter the data, which is what we will look at now.

Filtering by time

The --since and --until options lets you specify a timestamp in the format YYYY-MM-DD HH:MM:SS. You don’t have to include the time, and if you do include the time you may leave out the seconds (i.e. you can use HH:MM):

journalctl --since="2019-09-26" --until="2019-09-26 11:00"

You can also use strings such as “yesterday”, “today” and “10m ago”.

# journalctl --since "10m ago"

Filtering by unit

The -u option is used to filter a unit. The unit is the systemd unit name. So, to return log entries for sshd you can use the following:

# journalctl -u sshd.service
-- Logs begin at Mon 2019-09-23 10:14:25 BST, end at Thu 2019-09-26 12:03:22 BST. --
Sep 23 10:14:45 server.example.net sshd[1409]: Server listening on 0.0.0.0 port 22.
Sep 23 10:14:45 server.example.net sshd[1409]: Server listening on :: port 22.
Sep 23 10:15:26 server.example.net sshd[8823]: Invalid user vbox from 5.39.95.92 port 60978
Sep 23 10:15:26 server.example.net sshd[8823]: input_userauth_request: invalid user vbox [preauth]
Sep 23 10:15:26 server.example.net sshd[8823]: Received disconnect from 5.39.95.92 port 60978:11: Normal Shutdown, Thank you for playing [preauth]
...

Filtering by priority

When we looked at the JSON output we mentioned that each journal entry has a priority field. The priority number indicates the importance of the message. A priority of 0; means we got an emergency, while 7 is a debug message. In other words, the lower the number, the higher the priority:

• 0: emerg
• 1: alert
• 2: crit
• 3: err
• 4: warning
• 5: notice
• 6: info
• 7: debug

The -p option lets you filter messages with a particular priority. If, say, you want to see all errors you can use either of these commands:

# journalctl -p 3

# journalctl -p err

You are most likely to use the priority option to suppress less relevant messages. If you want to view log entries with a priority of err or greater you can add the -b option. This will return entries with a priority of err, crit, alert or emerg.

# journalctl -p err -b
-- Logs begin at Mon 2019-09-23 10:14:25 BST, end at Thu 2019-09-26 12:20:01 BST. --
Sep 23 20:21:29 server.example.net sshd[14834]: error: maximum authentication attempts exceeded for root from 180.126.50.52 port 5166 ssh2 [preauth]
Sep 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth]
...

Filtering by process

The output of journalctl includes the name of the service followed by the process ID (in square brackets). You can use the _PID option to get all entries for a specific process ID:

# journalctl _PID=17255
-- Logs begin at Mon 2019-09-23 10:14:25 BST, end at Thu 2019-09-26 12:30:01 BST. --
Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128
Sep 24 00:59:37 server.example.net sshd[17255]: input_userauth_request: invalid user admin [preauth]
Sep 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth]
Sep 24 00:59:39 server.example.net sshd[17255]: Disconnecting: Too many authentication failures [preauth]

Tailing logs

You can monitor logs as entries are being written using the -f option. This works exactly like tail -f. Of course, you can filter the unit you want to check at the same time. For example, here we are following entries for Dovecot:

# journalctl -f -u dovecot.service

And now that we have mentioned tail we should also highlight the -n option. As you might have guessed, this option works exactly like tail -n – it displays the last x entries. For instance, to view the last three entries for the Dovecot service you can use:

# journalctl -u dovecot.service -n 3

Manipulating output

Although the filtering options we covered are quite flexible they don’t necessarily return the data you want. For instance, when you are looking in to SSH login attempts you might want to filter entries based on the IP address shown in the MESSAGE field. On Centos 7 and CloudLinux 7 servers you can do that by piping the output of journalctl to a utility like grep:

# journalctl -u sshd.service | grep 59.148.43.97
Sep 24 00:59:37 server.example.net sshd[17255]: Invalid user admin from 59.148.43.97 port 54128
Sep 24 00:59:39 server.example.net sshd[17255]: error: maximum authentication attempts exceeded for invalid user admin from 59.148.43.97 port 54128 ssh2 [preauth]

Centos 8 and CloudLinux 8 ship with a version of journalctl that includes the -g (or --grep) option, which lets you grep the message field. That means that you can instead use this command:

# journalctl -u sshd.service -g 59.148.43.97

More information

This article aimed to give a brief overview of journalctl. There are many topic we haven’t covered, including storage options (journalctl logs aren’t necessarily persistent, which means that log entries don’t survive system reboots) and the various output formats. If you want to learn more, the official documentation can be found at freedesktop.org.