Log rotation on Linux

10 May 2021

In my article about logging on Linux I looked at how log rotation works with systemd-journald. This article looks at how log rotation works for services that manage their own logs.

Logrotate configuration

The logs for services such as httpd and sshd are rotated by the logrotate utility. Its configuration file is /etc/logrotate.conf:

# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

That is the entire configuration file! By default, logs are rotated weekly, and the system keeps up to four old logs. The date is appended to the file name, and log files are not compressed. These defaults are sensible. The one thing you might want to change is the compress directive. When the option is enabled rotated logs are compressed using gzip. This can save quite a bit of space (plain text log files compress very well). Of course, a downside is that you need to decompress logs before you can inspect them.

The configuration file includes files from the /etc/logrotate.d directory. This is where individual services specify their own directives. For instance, this is Apache’s configuration file:

# cat /etc/logrotate.d/httpd
/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
    endscript
}

The first line shows the location of the log file, including the file name (the asterisk is replaced with the actual file name). Directives in these files are less obvious than the ones in /etc/logrotate.conf. If you are curious, all the directives are described in the man page (man 1 logrotate).

One directive that is worth pointing out is postrotate. As the name suggests, the directive defines jobs that should be run after a log is rotated. In this case, the httpd configuration file is reloaded.

The logrotate cron

Finally, let’s briefly look at how the log rotation is done. The logrotate binary is called by a shell script in /etc/cron.daily. Any executables in the directory are run by anacron rather than cron. Anacron works like cron, but there are a few differences. Most importantly, anacron checks if there are any jobs that need to done once a day. So, you can’t set specific intervals like you can with cron.

Anacron jobs are configured in the /etc/anacrontab file. By default there is a daily, weekly and monthly job. All three use the run-parts command to execute all jobs in the specified directories. So, for cron.daily it runs all jobs in the /etc/cron.daily directory.

# tail -4 /etc/anacrontab
#period   delay  job-identifier  command
1	   5     cron.daily      nice run-parts /etc/cron.daily
7	  25     cron.weekly     nice run-parts /etc/cron.weekly
@monthly  45     cron.monthly    nice run-parts /etc/cron.monthly

Checking anacron jobs

You can check the jobs run via journalctl. The below output shows that anacron woke up at 14:01. Next, it scheduled when cron.daily and cron.weekly should be run. The delay prevents that lots of different jobs are triggered at the same time (you can tweak the delay in the /etc/anacrontab file).

# journalctl -u crond
...
Mar 29 14:01:02 centos8 anacron[3529]: Anacron started on 2021-02-11
Mar 29 11 14:01:02 centos8 anacron[3529]: Will run job `cron.daily' in 15 min.
Mar 29  14:01:02 centos8 anacron[3529]: Will run job `cron.weekly' in 35 min.
Mar 29  14:01:02 centos8 anacron[3529]: Jobs will be executed sequentially
Mar 29  14:16:02 centos8 anacron[3529]: Job `cron.daily' started
Mar 29  14:16:11 centos8 anacron[3529]: Job `cron.daily' terminated
Mar 29  14:36:02 centos8 anacron[3529]: Job `cron.weekly' started
Mar 29  14:36:02 centos8 anacron[3529]: Job `cron.weekly' terminated
Mar 29  14:36:02 centos8 anacron[3529]: Normal exit (2 jobs run)
...