Investigating memory usage with sar

31 May 2021

The free and top utilities are handy if you want information about the current memory usage on your system. If you want to view historic memory usage data then you can use sar instead.

Memory data in sar

sar displays 10 memory related fields. They include most of the fields shown by free and the summary display in top, but they are named differently. Also, sar doesn’t include an “available” column. If you want an estimate of how much memory was available at a specific time then you have to get your calculator out. I will show a quick way to calculate the available memory shortly. First, let’s look at the output.

The below data comes from a VPS that sent an alert about the memory usage at 16:40. For the last ten minutes, the memory usage had been above 80%:

$ sar -s 16:00:00 -r
Linux 3.10.0-1160.25.1.el7.x86_64 (server.example.com) 	25/05/21 	_x86_64_	(1 CPU)

            Free      Used                Buffers    Cached
16:00:01    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
16:10:01       212980    801800     79.01         0    293512   1247884    122.97    530056    109360         0
16:20:01       214984    799796     78.81         0    293556   1248208    123.00    528032    109352         4
16:30:01        86396    928384     91.49         0    293656   1367420    134.75    656212    109392         0
16:40:01        62124    952656     93.88         0    293716   1392280    137.20    680392    109372         4
16:50:02        85780    929000     91.55         0    277704   1386144    136.60    651592    117496        28
17:00:01       123456    891324     87.83         0    280468   1350968    133.13    613404    117512         0
17:10:01       183864    830916     81.88         0    280560   1291676    127.29    553212    117344         4
17:20:01       172656    842124     82.99         0    280644   1300148    128.12    564912    117288        16
17:30:01       178120    836660     82.45         0    280876   1294144    127.53    559736    116936         0
17:40:01       177592    837188     82.50         0    280944   1295684    127.68    560008    116928        24
17:50:01       237628    777152     76.58         0    281168   1242272    122.42    502200    114972        12
18:00:01       250152    764628     75.35         0    281268   1228828    121.09    492784    112268         0
18:10:01       248364    766416     75.53         0    281308   1231892    121.39    493840    112268         0
18:20:01       222704    792076     78.05         0    281872   1255296    123.70    520228    111224         0
Average:       175486    839294     82.71         0    284375   1295203    127.63    564758    113694         7

Even without understanding the meaning of all the columns you can see a trend. The %memused column shows there was a spike in memory usage at around 16:30.

Memory columns

To further investigate the issue you need to understand what the different columns mean. I will cover them all, but you mainly need to know the fields that are also shown by free.

• kbmemfree is the same as the free column in the output of free. It is memory that is not used for anything – not even caching.
• kbmemused is the sum of the used and buff/cache columns in free. So, some of this memory is used for the page cache and can be released. Also, the sum of kbmemfree and kbmemused is always equal to the total amount of memory.
• %memused is the percentage of used memory. The system used for this article has 1014780 kilobytes (roughly 1GB) of memory. At 16:10 it was using 801800 kilobytes, which is 79.01% of the system’s memory ((801800 / 1014780) * 100).
• The kbbuffers and kbcached columns correspond to the buffers and cache columns in free. Both are part of kbmemused. To get an idea of how much memory was used but not cached you can therefore substract the two columns from kbmemused.
• The kbcommit and %commit columns show how much memory the kernel has overcommited. As explained in the article about memory data in free and top, processes are promised more memory than the kernel can actually give them. That is typically fine, as it is extremely unlikely that all processes use all the memory assigned to them. So, it is normal for the percentage to be more than 100%.
• kbactive and kbinact show the amount of memory that was and wasn’t used recently. The latter column are memory pages the kernel might evict if the memory gets too high.
• And finally, kbdirty shows the amount of memory that is waiting to be written to the disk.

Available memory

In this case there wasn’t a major issue. The spike happened at 16:40. If you add up the free memory, buffers and cache then you see that the VPS still had plenty of available memory:

62124 + 0 + 293716 = 355840

I should note that not all the cached memory can be made available instantly, but it does give a clearer picture of the actual memory usage.

Formatting the sar output

Ideally, the sar output shows the total amount of memory; the amount of used and free memory; the amount of memory used for buffers and cache and the amount of available memory. You can get that information with a bit of awk magic:

$ sar -s 16:00:00 -e 18:10:00 -r \
| grep -E ^[[:digit:]] \
| awk '{if (NR>1) print $1, $2+$3, $3, $2, $5+$6, $2+$5+$6}'
16:10:01 1014780 801800 212980 293512 506492
16:20:01 1014780 799796 214984 293556 508540
16:30:01 1014780 928384 86396 293656 380052
16:40:01 1014780 952656 62124 293716 355840
16:50:02 1014780 929000 85780 277704 363484
17:00:01 1014780 891324 123456 280468 403924
17:10:01 1014780 830916 183864 280560 464424
17:20:01 1014780 842124 172656 280644 453300
17:30:01 1014780 836660 178120 280876 458996
17:40:01 1014780 837188 177592 280944 458536
17:50:01 1014780 777152 237628 281168 518796
18:00:01 1014780 764628 250152 281268 531420

The grep command filters out any lines starting with a digit. Next, I use awk to print all lines apart from the line that includes the column headers (NR>1). The last column in the output is the available memory. It’s a rough calculation (the amount of free memory plus the buffers and cache) but it gives an indication of the actual memory pressure.

You can make the output more readable by adding column headers and formatting the columns. Here, all the columns also use mebibytes rather than kibibytes:

$ sar -s 16:00:00 -e 18:10:00 -r \
| grep -E ^[[:digit:]] \
| awk 'BEGIN {OFS="\t"; print "Time", "Total", "Used", "Free", "Cache", "Avail"}; {if (NR>1) print substr($1,1,5), int(($2+$3)/1024), int($3/1024), int($2/1024), int(($5+$6)/1024), int(($2+$5+$6)/1024)}'
Time	Total	Used	Free	Cache	Avail
16:10	990	783	207	286	494
16:20	990	781	209	286	496
16:30	990	906	84	286	371
16:40	990	930	60	286	347
16:50	990	907	83	271	354
17:00	990	870	120	273	394
17:10	990	811	179	273	453
17:20	990	822	168	274	442
17:30	990	817	173	274	448
17:40	990	817	173	274	447
17:50	990	758	232	274	506
18:00	990	746	244	274	518