Checking Linux system performance with sar

Read Time:25 Minute, 56 Second

Sar is a system utility that gives us many ways to examine performance on a Linux system. It provides details on all aspects of system performance including system load, CPU usage, memory use, paging, swapping, disk usage, device load, network activity, etc.

The name “sar” stands for “system activity report,” and it can display current performance, provide reports that are based on log files stored in your system’s /var/log/sa (or /var/log/sysstat) folder, or be set up to automatically produce daily reports. It’s part of sysstat – a collection of system performance monitoring tools.

To check if sar is available on your system, run a command like this:

$ which sar

/usr/bin/sar

If not, you will need to install it with a command like “yum install sysstat” or “apt install sysstat”. You will then be able to run commands like these which collect performance details from your system:

$ sar -d 5 2

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU)

12:38:42 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util

12:38:47 PM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:38:47 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:38:47 PM sdb 0.40 0.00 6.40 0.00 16.00 0.01 8.50 0.34

12:38:47 PM sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:38:47 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:38:47 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util

12:38:52 PM sda 2.40 0.00 388.80 0.00 162.00 0.01 2.00 0.34

12:38:52 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:38:52 PM sdb 1.40 0.00 21.60 0.00 15.43 0.01 2.00 0.28

12:38:52 PM sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:38:52 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util

Average: sda 1.20 0.00 194.40 0.00 162.00 0.00 2.00 0.17

Average: sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: sdb 0.90 0.00 14.00 0.00 15.56 0.01 3.44 0.31

Average: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Note that this report shows device usage with activity displayed for /dev/sda and /dev/sdb. It includes two 5-second reports plus the averaging. If you delve into the sar man page, you’ll find explanations of each data column. In the above example, the fields included in the above example include:

DEV the device

tps Total number of transfers per second

rkB/s Number of kilobytes read from the device per second

wkB/s Number of kilobytes written to the device per second

dkB/s Number of kilobytes discarded for the device per second

areq-sz Average size (in kilobytes) of the I/O requests

aqu-sz Average queue length of the requests

await Average time (in milliseconds) for I/O requests

%util Percentage of elapsed time during which I/O requests were issued to the device

The next command pulls data on memory usage with three 5-second reports plus the averages.

$ sar -r 5 3

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU)

12:09:54 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty

12:09:59 PM 3089584 5231848 471144 7.75 35944 2257060 2126524 17.50 645168 2021508 48

12:10:04 PM 3089584 5231848 471140 7.75 35944 2257060 2126524 17.50 645168 2021552 48

12:10:09 PM 3089584 5231848 471140 7.75 35944 2257060 2126524 17.50 645168 2021560 144

Average: 3089584 5231848 471141 7.75 35944 2257060 2126524 17.50 645168 2021540 80

Using a wide terminal window will clearly make the details a lot easier to view.

Looking at current performance stats

The sar commands shown above are all using real time data. In other words, sar is collecting the data as directed and displaying it. No log files are involved. Older stats will not be available.

The default display is CPU. So, if you don’t specify an option, you’ll see something like this:

$ sar 10 2

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU)

10:17:50 AM CPU %user %nice %system %iowait %steal %idle

10:18:00 AM all 0.05 0.00 0.15 0.05 0.00 99.75

10:18:10 AM all 0.05 0.00 0.20 0.00 0.00 99.75

Average: all 0.05 0.00 0.18 0.03 0.00 99.75

To save the output from a command like this in a file, you could run the command like this:

$ sar -o sarfile 6 2

Keep in mind that the data file created (sarfile in this example) will not be a text file that you can display with a cat or more command. However, you can display it using a sar command that includes the file name as an argument:

$ sar -f sarfile

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU)

10:15:52 AM CPU %user %nice %system %iowait %steal %idle

10:15:58 AM all 0.17 0.00 0.17 0.25 0.00 99.42

10:16:04 AM all 0.08 0.00 0.33 0.00 0.00 99.58

Average: all 0.12 0.00 0.25 0.12 0.00 99.50

Looking at previous performance stats

You can easily set sar up to run automatically on your system, store its collected stats in daily files, produce daily reports and remove the oldest of the files after a given number of days (default is 28). This allows you to review performance on recent dates and compare what you’re seeing with current stats. This can often help you answers questions like “What’s different today?” — often the first question you’re likely to ask yourself when problems arise.

Once you’ve collected data using sar, you can look at it with sar commands like this command that extracts data from the sa14 data file:

$ sar -f /var/log/sa/sa14 -b

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/14/2021 _x86_64_ (2 CPU)

02:54:22 PM LINUX RESTART (2 CPU)

03:00:19 PM tps rtps wtps dtps bread/s bwrtn/s bdscd/s

03:10:19 PM 1.02 0.02 1.00 0.00 0.01 69.05 0.00

03:20:19 PM 1.05 0.02 1.03 0.00 0.01 75.03 0.00

03:30:19 PM 0.99 0.02 0.97 0.00 0.28 74.35 0.00

To do this, you need to first enable sar to run automatically and save performance stats in data files. The process for doing this is slightly different depending on which Linux distribution you are using.

Fedora

On Fedora, you would use commands like these to enable sar to collect daily stats:

# systemctl start sysstat.service

# systemctl enable sysstat.service

# systemctl status sysstat.service

After this, sar will store its data files in the /var/log/sa directory. A couple days after setting it up, you can expect to see something like this:

$ ls -l

total 1096

-rw-r–r–. 1 root root 103292 Jun 14 23:50 sa14 <== data files

-rw-r–r–. 1 root root 273544 Jun 15 23:50 sa15

-rw-r–r–. 1 root root 156240 Jun 16 13:30 sa16

-rw-r–r–. 1 root root 158740 Jun 15 00:07 sar14 <== daily reports

-rw-r–r–. 1 root root 420712 Jun 16 00:07 sar15

The numbers in the file names (e.g., 14) represent the day of the month. The three “sa” files (sa14, sa15 and sa16) are the data files for each of the three days and the sar14 and sar15 files are the performance reports for the first two days. On Fedora, you can determine how many days’ worth of data will be saved by using this command:

$ grep HISTORY /etc/sysconfig/sysstat

HISTORY=28

Ubuntu

To enable sar on Ubuntu and related systems, edit the /etc/default/sysstat file and change ENABLED=”false” to ENABLED=”true”.

ENABLED=”true”

The log files will be stored in /var/log/sysstat.

Viewing daily reports

To view any of the daily sar reports (e.g., sar15), you can simply use commands like more or cat. These files are simple text files.

Once a day has ended (shortly after midnight), a full-day report will be available with a name like “sar15”. While files like “sa14” are used to collect the raw data, files like “sar14” are text files that contain the daily reports. They contain quite a lot of performance details.

$ wc -l sar15

4701 sar15

$ more sar15

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 2021-06-15 _x86_64_ (2 CPU)

00:00:19 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle

00:10:19 all 0.07 0.00 0.12 0.05 0.00 0.01 0.01 0.00 0.00 99.74

00:10:19 0 0.03 0.01 0.05 0.02 0.00 0.00 0.01 0.00 0.00 99.88

00:10:19 1 0.10 0.00 0.20 0.08 0.00 0.01 0.01 0.00 0.00 99.60

00:20:19 all 0.07 0.00 0.10 0.04 0.00 0.00 0.01 0.00 0.00 99.77

00:20:19 0 0.01 0.00 0.02 0.02 0.00 0.00 0.00 0.00 0.00 99.94

00:20:19 1 0.12 0.00 0.18 0.07 0.00 0.01 0.01 0.00 0.00 99.60

00:30:19 all 0.06 0.00 0.10 0.06 0.00 0.01 0.01 0.00 0.00 99.76

…

To extract data from one of the daily data files (not the report files), you would use a command like this using the -f option to specify the data file to be used:

$ sar -f sa15 | head -22

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU)

12:00:19 AM CPU %user %nice %system %iowait %steal %idle

12:10:19 AM all 0.07 0.00 0.14 0.05 0.00 99.74

12:20:19 AM all 0.07 0.00 0.11 0.04 0.00 99.77

12:30:19 AM all 0.06 0.00 0.12 0.06 0.00 99.76

12:40:19 AM all 0.06 0.00 0.12 0.05 0.00 99.77

12:50:19 AM all 0.07 0.00 0.11 0.05 0.00 99.77

01:00:19 AM all 0.07 0.02 0.12 0.05 0.00 99.74

01:10:19 AM all 0.07 0.00 0.11 0.05 0.00 99.77

01:20:19 AM all 0.30 0.00 0.12 0.04 0.00 99.53

01:30:19 AM all 0.07 0.00 0.11 0.04 0.00 99.77

01:40:19 AM all 0.07 0.00 0.11 0.03 0.00 99.78

01:50:19 AM all 0.07 0.00 0.12 0.03 0.00 99.78

02:00:19 AM all 0.06 0.00 0.12 0.03 0.00 99.78

02:10:19 AM all 0.06 0.00 0.13 0.04 0.00 99.77

02:20:19 AM all 0.06 0.00 0.12 0.04 0.00 99.78

02:30:19 AM all 0.06 0.00 0.12 0.02 0.00 99.79

02:40:19 AM all 0.07 0.00 0.12 0.03 0.00 99.79

02:50:19 AM all 0.07 0.94 0.20 0.05 0.00 98.76

03:00:19 AM all 0.18 0.00 0.12 0.03 0.00 99.67

03:10:19 AM all 0.06 0.00 0.12 0.04 0.00 99.78

You can specify a start (-s) and end (-e) times in addition to the file name as in this next example.

$ sar -s 11:00 -e 12:00 -f sa15

Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU)

11:00:19 AM CPU %user %nice %system %iowait %steal %idle

11:10:19 AM all 0.06 0.00 0.17 0.05 0.00 99.73

11:20:19 AM all 0.05 0.00 0.17 0.05 0.00 99.73

11:30:19 AM all 0.05 0.00 0.17 0.05 0.00 99.73

11:40:19 AM all 0.18 0.00 0.17 0.05 0.00 99.60

11:50:19 AM all 0.06 0.02 0.18 0.05 0.00 99.69

Average: all 0.08 0.00 0.17 0.05 0.00 99.70

Options, options and options

You can use sar to display stats related to:

Paging (sar -B)

I/O and transfer rates (sar -b)

Block devices (sar -d)

Currently mounted file systems(sar -F)

Huge page utilization(sar -H)

Interrupts (sar -I)

Power management (sar -m)

Network statistics (sar -n)

Run queue and load average (sar -q)

Memory usage (sar -r)

Swap utilization (sar -S)

CPU utilization (sar -u)

Status of inode, file and other kernel tables (sar -v)

Swapping (sar -W)

Context switches (sar -w)

TTY device activity (sar -y)

In fact, the easiest way to get a listing of all of these options and what they represent is to ask sar for help. Notice that for some options, like networking, you have a choice of which variety of activity (e.g., TCP or sockets) you want to see.

$ sar –help

Usage: sar [ options ] [ <interval> [ <count> ] ]

Main options and reports (report name between square brackets):

-B Paging statistics [A_PAGE]

-b I/O and transfer rate statistics [A_IO]

-d Block devices statistics [A_DISK]

-F [ MOUNT ]

Filesystems statistics [A_FS]

-H Hugepages utilization statistics [A_HUGE]

-I { <int_list> | SUM | ALL }

Interrupts statistics [A_IRQ]

-m { <keyword> [,…] | ALL }

Power management statistics [A_PWR_…]

Keywords are:

CPU CPU instantaneous clock frequency

FAN Fans speed

FREQ CPU average clock frequency

IN Voltage inputs

TEMP Devices temperature

USB USB devices plugged into the system

-n { <keyword> [,…] | ALL }

Network statistics [A_NET_…]

Keywords are:

DEV Network interfaces

EDEV Network interfaces (errors)

NFS NFS client

NFSD NFS server

SOCK Sockets (v4)

IP IP traffic (v4)

EIP IP traffic (v4) (errors)

ICMP ICMP traffic (v4)

EICMP ICMP traffic (v4) (errors)

TCP TCP traffic (v4)

ETCP TCP traffic (v4) (errors)

UDP UDP traffic (v4)

SOCK6 Sockets (v6)

IP6 IP traffic (v6)

EIP6 IP traffic (v6) (errors)

ICMP6 ICMP traffic (v6)

EICMP6 ICMP traffic (v6) (errors)

UDP6 UDP traffic (v6)

FC Fibre channel HBAs

SOFT Software-based network processing

-q [ <keyword> [,…] | PSI | ALL ]

System load and pressure-stall statistics

Keywords are:

LOAD Queue length and load average statistics [A_QUEUE]

CPU Pressure-stall CPU statistics [A_PSI_CPU]

IO Pressure-stall I/O statistics [A_PSI_IO]

MEM Pressure-stall memory statistics [A_PSI_MEM]

-r [ ALL ]

Memory utilization statistics [A_MEMORY]

-S Swap space utilization statistics [A_MEMORY]

-u [ ALL ]

CPU utilization statistics [A_CPU]

-v Kernel tables statistics [A_KTABLES]

-W Swapping statistics [A_SWAP]

-w Task creation and system switching statistics [A_PCSW]

-y TTY devices statistics [A_SERIAL]

That’s a LOT of choices! You can, however, use the -A option to collect the data provided by all of the options included included in the string “bBdFHSvwWy”.

Wrap-Up

While sar is easy to install and use, it provides a lot of valuable details on how your system is performing and might help you drill down to the cause of a problem when one arises.

The only “problem” with sar is that it generates a LOT of data and you’ll have to become accustomed to using it and familiar with what routine performance on your system looks like before unusual performance or reasons for concern will jump out at you.

Source : Checking Linux system performance with sar | Network World

Don’t hesitate to contact us:

Geetika Technosoft Pvt Ltd
E-mail : crm@gtechnosoft.in
T: 1800 212 6124
#GeetikaTechnosoft
#ManagedITService
#ITExpertsConsultation
#GloablITServiceProvider
#quality
#trustworthiness
#networkworld
#linux
#checking
#system
#performance
#sar
#masterhost
#commands