sar (System Activity Reporter) is a command line tool used to collect, report, and analyze system activity information in Linux and other Unix-based operating systems.
In other words, it gathers information and provides insights on system activity, such as CPU utilization, memory usage, I/O operations, network activity, disk usage, power management, and much more, which helps in monitoring system resources and identifying performance issues.
Internally, sar depends on the sadc (system activity data collector), which captures and stores system activity data at regular intervals, usually every 10 minutes. Generally, the collected data is stored in binary log files, named saDD, where DD represents the day of the month. These files are typically kept in the /var/log/sa directory.
sar provides several options for reporting captured system activity. In this article, we'll explore different sar commands, which will help you retrieve and analyze your system's activity.
Install sysstat
sar is available as a component of the sysstat package, which means that the sysstat package is required to use it.
On Ubuntu and other Debian-based distributions
sudo apt install sysstat
On CentOs, Fedora, and RHEL
sudo dnf install sysstat
On Arch-based distributions
sudo pacman -S sysstat
Enable sysstat
After installing sysstat, you need to enable the sysstat service to ensure sysstat runs automatically after boot-up, and both sadc and sar become active as well. To enable sysstat, run:
sudo systemctl enable sysstat Synchronizing state of sysstat.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable sysstat Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /lib/systemd/system/sysstat.service. Created symlink /etc/systemd/system/sysstat.service.wants/sysstat-collect.timer → /lib/systemd/system/sysstat-collect.timer. Created symlink /etc/systemd/system/sysstat.service.wants/sysstat-summary.timer → /lib/systemd/system/sysstat-summary.timer.
Afterward, run the following command to start the sysstat service so that data collection can begin in the right away.
sudo systemctl start sysstat
Then, run the following command, which should produce an output similar to the one below to verify that sar is functional and running.
The status can be checked like this:
$ systemctl status sysstat
● sysstat.service - Resets System Activity Logs
Loaded: loaded (/lib/systemd/system/sysstat.service; enabled; preset: enabled)
Active: active (exited) since Sun 2023-06-04 15:18:48 IST; 18h ago
Docs: man:sa1(8)
man:sadc(8)
man:sar(1)
Main PID: 4831 (code=exited, status=0/SUCCESS)
CPU: 34ms
Jun 04 15:18:48 silver-VirtualBox systemd[1]: Starting Resets System Activity Logs...
Jun 04 15:18:48 silver-VirtualBox systemd[1]: Finished Resets System Activity Logs.
$
Checking sar version
sar -V sysstat version 12.5.2 (C) Sebastien Godard (sysstat <at> orange.fr)
sar Command Syntax and Examples
sar commands typically take the following form:
sar -[options] [interval] [count]
Here's a breakdown of what each parameter does:
- [options]: These are additional flags used to filter and customize the information displayed by sar.
- [interval]: This refers to the time interval between each data collection and is usually specified in seconds.
- [count]: This indicates the number of times the collected data is to be reported.
1. Get CPU usage metrics
The "-u" option is used to retrieve the CPU utilization statistics. When invoked without additional parameters, sar will generate a report, showcasing the CPU utilization statistics for the duration the system has been up for the day.
To get the CPU metrics, run:
$ sar -u 1 Linux 6.2.0-20-generic (enlightened) 05/06/23 _x86_64_ (4 CPU) 09:46:36 AM IST CPU %user %nice %system %iowait %steal %idle 09:46:37 AM IST all 11.14 0.00 15.95 0.51 0.00 72.41 09:46:38 AM IST all 11.39 0.00 10.89 0.25 0.00 77.47 09:46:39 AM IST all 11.66 0.00 21.34 0.00 0.00 67.00 09:46:40 AM IST all 17.56 0.25 9.92 3.05 0.00 69.21 09:46:41 AM IST all 22.96 0.00 17.35 0.00 0.00 59.69 09:46:42 AM IST all 17.37 0.25 16.63 0.00 0.00 65.76 09:46:43 AM IST all 12.28 0.00 11.78 0.50 0.00 75.44
Note that we have provided an interval of 1 sec, so metrics will be retrieved once every second and reported in a new line. This way sar is actually reporting in realtime by fetching the most recent values for individual metrics.
Get more metrics: Additionally, you can include the "ALL" parameter in the command, which tells sar to show a more comprehensive table, containing all the available CPU fields as demonstrated below.
$ sar -u ALL 1 Linux 6.2.0-20-generic (enlightened) 05/06/23 _x86_64_ (4 CPU) 09:48:13 AM IST CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 09:48:14 AM IST all 35.43 0.25 14.57 0.00 0.00 0.00 0.25 0.00 0.00 49.50 09:48:15 AM IST all 18.18 0.00 16.67 0.00 0.00 0.00 0.00 0.00 0.00 65.15 09:48:16 AM IST all 27.97 0.25 29.46 0.25 0.00 0.00 0.50 0.00 0.00 41.58 09:48:17 AM IST all 46.02 0.00 22.64 0.75 0.00 0.00 0.50 0.00 0.00 30.10 09:48:18 AM IST all 21.07 0.00 18.78 0.25 0.00 0.00 0.00 0.00 0.00 59.90 09:48:19 AM IST all 10.49 0.00 16.37 0.51 0.00 0.00 0.00 0.00 0.00 72.63
Logged / Pre-recorded values
If you do not specify any interval, then sar will fetch pre-recorded values from a log file, which is usually located at: /var/log/sysstat/
$ ls -la /var/log/sysstat/ total 120 drwxr-xr-x 2 root root 4096 Jun 5 07:09 . drwxrwxr-x 14 root syslog 4096 Jun 5 07:09 .. -rw-r--r-- 1 root root 30952 Jun 4 20:10 sa04 -rw-r--r-- 1 root root 30868 Jun 5 09:50 sa05 -rw-r--r-- 1 root root 40454 Jun 5 07:09 sar04 $
Note the file named sar04 above. That file contains data for all metrics in a single large file. You can view the contents by issuing the following command:
cat /var/log/sysstat/sar04
To view old metrics issue the command like this, without any interval.
sar -u Linux 5.19.0-42-generic (ubuntu) 27/05/2023 _x86_64_ (4 CPU) 00:00:00 CPU %user %nice %system %iowait %steal %idle 00:10:00 all 0.50 0.00 0.50 0.18 0.00 98.82 00:20:00 all 0.24 0.00 0.15 0.01 0.00 99.60 00:30:00 all 0.27 0.00 0.24 0.00 0.00 99.49 00:40:00 all 0.27 0.00 0.21 0.00 0.00 99.52 00:50:00 all 0.23 0.00 0.21 0.01 0.00 99.54 01:00:00 all 0.16 0.00 0.20 0.00 0.00 99.64 01:10:00 all 0.12 0.00 0.14 0.00 0.00 99.73 01:20:00 all 0.25 0.00 0.17 0.01 0.00 99.57 01:30:00 all 0.16 0.00 0.15 0.00 0.00 99.68 01:40:00 all 0.15 0.00 0.13 0.00 0.00 99.71
In the output above, sar displays the CPU metrics recorded every 10 minutes over a 2-hour period in a table, consisting of various CPU fields.
Enable background monitoring
At this point you might run into a problem if sar is not running the background to record system statistics. The output will look something like this:
$ sar -u Linux 5.19.0-43-generic (silver-VirtualBox) 04/06/23 _x86_64_ (4 CPU) 03:12:15 PM IST LINUX RESTART (4 CPU) 03:12:43 PM IST LINUX RESTART (4 CPU) 03:18:48 PM IST LINUX RESTART (4 CPU) $ sar -r Linux 5.19.0-43-generic (silver-VirtualBox) 04/06/23 _x86_64_ (4 CPU) 03:12:15 PM IST LINUX RESTART (4 CPU) 03:12:43 PM IST LINUX RESTART (4 CPU) 03:18:48 PM IST LINUX RESTART (4 CPU) $
It will just show entries named "LINUX RESTART" and no other rows of any data. This means that sar(sysstat) is not logging system metrics.
To fix this problem you have to enable background monitoring. This can be done in many ways.
On Debian/Ubuntu based systems run the command, and select Yes when prompted. It will enable the service.
dpkg-reconfigure sysstat
Alternatively you can edit the file: /etc/default/sysstat and set ENABLED to true and it should look like this
$ cat /etc/default/sysstat # # Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat # and /etc/cron.daily/sysstat files # # Should sadc collect system activity informations? Valid values # are "true" and "false". Please do not put other values, they # will be overwritten by debconf! ENABLED="true" $
Now restart the system.
By default sar collects data every 10 minutes
sar -u ALL Linux 5.19.0-42-generic (ubuntu) 27/05/2023 _x86_64_ (4 CPU) 00:00:00 CPU %user %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 00:10:00 all 0.13 0.00 0.43 0.18 0.00 0.00 0.07 0.36 0.00 98.82 00:20:00 all 0.04 0.00 0.13 0.01 0.00 0.00 0.02 0.20 0.00 99.60 00:30:00 all 0.04 0.00 0.21 0.00 0.00 0.00 0.03 0.23 0.00 99.49 00:40:00 all 0.05 0.00 0.19 0.00 0.00 0.00 0.02 0.22 0.00 99.52
The table shown in the output above contains the following information:
- timestamp: This column indicates the time of data collection. The values for each row under this column correspond to a specific time interval.
- CPU: This refers to the specific CPU core or processor from which data was captured. In this case, "all" indicates that data was aggregated from all available cores.
- %usr: This column shows the percentage of CPU utilization while executing user-level processes like your applications.
- %nice: This column represents the percentage of CPU utilization while executing user-level processes with a nice (lower) priority.
- %sys: This indicates the percentage of CPU time spent executing system-level (kernel) processes.
- %iowait: This column shows the percentage of time when the CPU was idle, waiting for pending I/O operations to complete.
- %steal: This indicates the percentage of time spent by a virtual CPU in a virtual machine, waiting while the hypervisor serviced another virtual machine.
- %irq: This column represents the percentage of CPU time spent handling interrupt requests from hardware devices.
- %soft: This column represents the percentage of CPU time spent handling interrupt requests from software sources.
- %guest: This column shows the percentage of CPU time spent running a virtual machine.
- %gnice: This indicates the percentage of CPU time spent running niced guest processes.
- %idle: This column represents the percentage of CPU time during which the CPU was idle, without any outstanding tasks.
In addition, sar allows you to specify an interval and the number of times a report is to be generated. For example, to generate the CPU metrics 4 times after every 5 seconds, run the following command:
sar -u 5 4 Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 08:50:14 CPU %user %nice %system %iowait %steal %idle 08:50:19 all 0.25 0.00 0.51 0.00 0.00 99.24 08:50:24 all 0.25 0.00 0.25 0.00 0.00 99.49 08:50:29 all 0.41 0.00 0.41 0.00 0.00 99.14 08:50:34 all 0.51 0.00 0.61 0.00 0.00 98.89 Average: all 0.35 0.00 0.44 0.00 0.00 99.20
Report for Specific CPU Core
Alternatively, you can tell sar to generate a report for a specific CPU core by adding the "-P" option instead and giving it a CPU identifier. However, it's important to note that CPU identifiers begin from 0, which means that in multi-core processors, 0 is the first processor, 1 is the second processor, and so on.
To generate the CPU metrics for the first processor, run the following command:
sar -P 0 Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 CPU %user %nice %system %iowait %steal %idle 05:40:14 0 1.09 0.80 1.58 1.13 0.00 95.40 05:50:00 0 0.20 0.00 0.21 0.16 0.00 99.44 Average: 0 0.64 0.40 0.90 0.64 0.00 97.42
Additionally, when "ALL" is passed as an argument to the "-P" option, sar will generate a report, which shows the global and individual statistics for the processors present on the machine, as seen below.
sar -P ALL Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 CPU %user %nice %system %iowait %steal %idle 05:40:14 all 1.13 0.49 1.45 1.32 0.00 95.61 05:40:14 0 1.09 0.80 1.58 1.13 0.00 95.40 05:40:14 1 1.31 0.50 1.39 1.48 0.00 95.32 05:40:14 2 0.93 0.36 1.25 1.60 0.00 95.86 05:40:14 3 1.17 0.31 1.58 1.06 0.00 95.87 05:40:14 CPU %user %nice %system %iowait %steal %idle 05:50:00 all 0.34 0.00 0.22 0.07 0.00 99.38 05:50:00 0 0.20 0.00 0.21 0.16 0.00 99.44 05:50:00 1 0.32 0.00 0.19 0.02 0.00 99.47 05:50:00 2 0.24 0.00 0.15 0.07 0.00 99.54 05:50:00 3 0.58 0.00 0.32 0.03 0.00 99.07
2. Get Queue Length and Load Average
The "-q" option is used to generate a report about the system load average and the number of tasks in the run queue.
To view the report, run the following command:
sar -q Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 05:40:14 0 611 0.20 0.83 1.04 0 05:50:00 0 610 0.10 0.25 0.62 0 06:00:21 0 610 0.11 0.21 0.42 0 06:10:19 0 611 0.13 0.19 0.31 0 Average: 0 610 0.14 0.37 0.60 0
The output produced shows the following information:
- timestamp: This column indicates the time of data collection.
- runq-sq: This column shows the number of tasks that are currently in the run queue waiting for run time, which in this case is 0.
- plist-sz: This column shows the total number of tasks in the task list.
- ldavg-1: This shows the system load average for the last minute. The load average refers to the average number of running tasks (R state) and tasks in uninterruptable sleep (D state) during a specific interval.
- ldavg-5: This indicates the system load average for the last 5 minutes.
- ldavg-15: This indicates the system load average for the last 15 minutes.
- blocked: This column shows the number of processes that are currently blocked, awaiting the release of resources.
3. Get Memory Metrics
The -r option generates reports on the memory utilized by the system, which is useful for identifying memory-related issues and tracking how the components of your system consume memory.
To monitor memory metrics in realtime we can use interval 1.
$ sar -r 1 Linux 6.2.0-20-generic (enlightened) 05/06/23 _x86_64_ (4 CPU) 10:15:00 AM IST kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 10:15:01 AM IST 2602828 10090552 19148936 58.45 494448 9555044 57417972 43.11 3615968 20133344 1696 10:15:02 AM IST 2597116 10084852 19153452 58.46 494456 9556232 57406944 43.10 3615980 20128360 2092 10:15:03 AM IST 2593840 10081596 19156524 58.47 494472 9556420 57407512 43.10 3615996 20130200 2144 10:15:04 AM IST 2593840 10081608 19161352 58.49 494476 9551588 57402888 43.09 3616008 20130284 2028 10:15:05 AM IST 2592580 10080348 19165816 58.50 494476 9548384 57393372 43.09 3616008 20121708 2444 10:15:06 AM IST 2591324 10079140 19168720 58.51 494484 9546728 57393640 43.09 3616008 20122488 1948 ...
To watch historic stats omit the interval and get data from the log files.
sar -r Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 05:40:14 881916 2702588 4678196 57.72 46596 1980972 14578932 142.90 2468276 3441380 16 05:50:00 877496 2701836 4678848 57.73 47360 1983752 14586908 142.98 2477940 3436880 0 06:00:21 875640 2704104 4676544 57.70 47848 1987344 14586916 142.98 2484140 3435436 0 06:10:19 676984 2585532 4793536 59.14 48808 2065052 14587308 142.99 2584132 3533152 0 06:20:05 668104 2580280 4798492 59.21 49228 2068340 14595788 143.07 2596396 3531060 172 06:30:21 623716 2574216 4804396 59.28 49500 2106284 14587732 142.99 2608804 3560084 0
After executing the command, sar generates a report similar to the output above, which contains a lot of information. Below is a breakdown of what each column represents:
- timestamp: This indicates the time of data collection.
- kbmemfree: This column shows the amount of memory not currently in use, expressed in kilobytes.
- kbavail: This column represents the estimated amount of memory in kilobytes available for starting new applications, which also takes the page cache and reclaimable slab into consideration.
- kbmemused: This column indicates the amount of memory used, expressed in kilobytes, excluding buffers and cache.
- %memused: This column shows the percentage of the total available memory that is in use.
- kbbuffers: This column represents the amount of memory used as buffers by the kernel, expressed in kilobytes.
- kbcached: This indicates the amount of memory used for caching system data, expressed in kilobytes.
- kbcommit: This column represents an estimate of the amount of memory required to handle the current workload, expressed in kilobytes.
- %commit: This column shows the percentage of committed memory relative to the total amount of memory. Furthermore, since the kernel often overcommits memory, this value can be greater than 100.
- kbactive: This column shows the amount of memory in kilobytes that is currently active and not eligible for reclamation unless absolutely necessary.
- kbinact: This indicates the amount of inactive memory in kilobytes, eligible to be reclaimed for other tasks.
- kbdirty: This column represents the amount of memory in kilobytes that contains data expected to be written to disk.
Similar to the "-u" option, the "ALL" parameter can be used with the "-r" option to display a more exhaustive table, showing all the available memory fields, including kbslab, kbkstack, kbvmused, and so on.
4. Get Paging Metrics
The "-B" option is used to generate reports on paging information, which gives insights into how your system manages memory, highlighting the rate at which frequently accessed data is read from and written to disk.
To view paging statistics, run:
sar -B Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 05:40:14 393.33 39.73 1256.14 1.73 977.72 0.00 0.00 0.00 0.00 05:50:00 5.22 6.04 5.31 0.00 11.75 0.00 0.00 0.00 0.00 06:00:21 4.62 11.80 5.43 0.00 18.66 0.00 0.00 0.00 0.00 06:10:19 26.77 235.41 3.71 0.00 27.25 0.00 0.00 0.00 0.00 06:20:05 4.77 7.64 6.70 0.00 10.45 0.00 0.00 0.00 0.00 06:30:21 54.89 14.06 5.24 0.00 27.29 0.00 0.00 0.00 0.00 06:40:21 95.13 4.55 4.07 0.00 10.14 0.00 0.00 0.00 0.00
The output produced shows the following information:
- timestamp: The time of data capture.
- pgpgin/s: This column shows the rate in kilobytes per second at which data is being paged in.
- pgpgout/s: This represents the rate in kilobytes per second at which data is being paged out.
- fault/s: This column shows the number of page faults that arise every second, including both minor and major faults. A page fault occurs when a process tries to access a memory page that hasn't been mapped or that isn't available in the physical memory.
- majflt/s: This column indicates the number of major faults per second.
- pgfree/s: This column shows the number of pages placed on the free list per second.
- pgscank/s: This column shows the number of pages the kernel swap daemon (kswapd) scans every second.
- pgscand/s: This indicates the rate at which the kernel directly scans pages for reclaimable memory.
- pgsteal/s: This represents the number of pages the system has reclaimed from the cache per second to meet its memory needs.
- %vmeff: This column measures the efficiency of page reclamation, calculated as pgsteal/pgscan.
5. Get I/O and Transfer Rate Metrics
To generate a report on the I/O operations and transfer rate, the "-b" option is used. This provides information about I/O activity on your system, including the utilization rate of block devices.
The following command tells sar to produce a report on I/O operations and transfer rate:
sar -b Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 tps rtps wtps dtps bread/s bwrtn/s bdscd/s 05:40:14 12.04 7.15 4.89 0.00 687.74 79.46 0.00 05:50:00 1.50 0.50 0.99 0.00 10.45 12.09 0.00 06:00:21 1.17 0.13 1.04 0.00 9.23 23.60 0.00 06:10:19 3.47 0.67 2.81 0.00 53.54 470.82 0.00 06:20:05 1.32 0.14 1.18 0.00 9.54 15.28 0.00 06:30:21 3.03 1.93 1.10 0.00 109.77 28.13 0.00
The report produced shows the following information:
- timestamp: The time of data collection.
- tps: This column shows the total number of transfer requests per second for all block devices, which includes both read and write requests.
- rtps: This indicates the total number of read requests per second sent to all block devices.
- wtps: This column represents the total number of write requests per second sent to all block devices.
- dtps: This column represents the total number of discard requests per second sent to all block devices.
- bread/s: This column shows the total amount of data read from block devices, represented in blocks per second. A block is equivalent to 512 bytes.
- bwrtn/s: This indicates the total amount of data written to block devices, represented in blocks per second.
- bdscd/s: This column shows the total amount of data discarded for all block devices in blocks per second.
6. Get Swap Space Metrics
The "-S" option is used to generate reports on the utilization of the swap space. The swap space is a reserved portion of the hard disk, which acts as an extension of the physical memory. When the operating system runs out of memory, it temporarily stores data in the swap space. This information is useful for identifying potential memory management problems in your system.
To obtain the swap space utilization statistics, run the following command:
sar -S Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 kbswpfree kbswpused %swpused kbswpcad %swpcad 05:40:14 2097148 0 0.00 0 0.00 05:50:00 2097148 0 0.00 0 0.00 06:00:21 2097148 0 0.00 0 0.00 06:10:19 2097148 0 0.00 0 0.00
The output generated contains the following information:
- timestamp: This shows the time of data capture.
- kbswpfree: This column indicates the available amount of free swap space in kilobytes.
- kbswpused: This column shows the amount of swap space used in kilobytes.
- kbswpcad: This column represents the amount of cached swap memory in kilobytes.
- %swpcad: This column shows the percentage of cached swap memory relative to the amount of swap space used.
7. Get Mounted Filesystem Metrics
The "-F" option is used to generate reports for currently mounted filesystems. It provides information on the amount of space consumed in each mounted filesystem.
To view the mounted filesystem statistics, run:
sar -F 2 1 Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 17:35:08 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM 17:35:10 78651 21062 21.12 26.25 6304917 215915 3.31 /dev/sda3 17:35:10 0 163 100.00 100.00 0 438 100.00 /dev/loop1 17:35:10 0 73 100.00 100.00 0 13370 100.00 /dev/loop4 17:35:10 0 62 100.00 100.00 0 11793 100.00 /dev/loop2 17:35:10 0 64 100.00 100.00 0 11954 100.00 /dev/loop0 17:35:10 506 6 1.19 1.19 0 0 0.00 /dev/sda2 17:35:10 0 127 100.00 100.00 0 0 0.00 /dev/sr0 17:35:10 0 3650 100.00 100.00 0 0 0.00 /dev/sr1 17:35:10 1475 0 0.00 6.29 98291 13 0.01 /dev/sdb1 17:35:10 432 20 4.43 12.18 127986 14 0.01 /dev/sdb2 17:35:10 218 0 0.01 8.03 63989 11 0.02 /dev/sdb3 Summary: MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM Summary: 78651 21062 21.12 26.25 6304917 215915 3.31 /dev/sda3 Summary: 0 163 100.00 100.00 0 438 100.00 /dev/loop1 Summary: 0 73 100.00 100.00 0 13370 100.00 /dev/loop4 Summary: 0 62 100.00 100.00 0 11793 100.00 /dev/loop2 Summary: 0 64 100.00 100.00 0 11954 100.00 /dev/loop0 Summary: 506 6 1.19 1.19 0 0 0.00 /dev/sda2 Summary: 0 127 100.00 100.00 0 0 0.00 /dev/sr0 Summary: 0 3650 100.00 100.00 0 0 0.00 /dev/sr1 Summary: 1475 0 0.00 6.29 98291 13 0.01 /dev/sdb1 Summary: 432 20 4.43 12.18 127986 14 0.01 /dev/sdb2 Summary: 218 0 0.01 8.03 63989 11 0.02 /dev/sdb3
The output produced shows the following information:
- MBfsfree: This column shows the total amount of free space available in the filesystem, expressed in megabytes.
- MBfsused: This represents the amount of space used in the filesystem, expressed in megabytes.
- %fsused: This column indicates the percentage of the filesystem used from the perspective of a privileged user.
- %ufsused: This column shows the percentage of the filesystem used from the perspective of an unprivileged user.
- Ifree: This column shows the total number of free file nodes in the filesystem.
- Iused: This represents the total number of used file nodes in the filesystem.
- %Iused: This column shows the percentage of file nodes used in the filesystem.
- FILESYSTEM: This indicates the filesystem identifier.
To omit the loop devices we can pipe the output into grep with the -v option.
$ sar -F 1 1 | grep -v "loop" Linux 6.2.0-20-generic (enlightened) 05/06/23 _x86_64_ (4 CPU) 10:23:06 AM IST MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM 10:23:07 AM IST 19883 75667 79.19 84.32 5013052 1237444 19.80 /dev/sda1 10:23:07 AM IST 138545 263534 65.54 70.64 26019374 195026 0.74 /dev/sdc1 10:23:07 AM IST 3097236 1671774 35.05 35.05 0 0 0.00 /dev/sdd2 Summary: MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM Summary: 19883 75667 79.19 84.32 5013052 1237444 19.80 /dev/sda1 Summary: 138545 263534 65.54 70.64 26019374 195026 0.74 /dev/sdc1 Summary: 3097236 1671774 35.05 35.05 0 0 0.00 /dev/sdd2 $
In the above output we can see the real disk drive partitions along with space usage data.
8. Get Block Device Metrics
The "-d" option is used to produce a report on the activity of block devices. This information shows the performance of each block device, which helps in identifying potential problems with storage devices.
To view the statistics for all block devices, run the following command:
sar -d Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 17:00:27 DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 17:10:12 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:10:12 sr1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:10:12 sda 0.46 0.22 2.85 0.00 6.62 0.00 1.13 0.05 17:10:12 sdb 0.06 0.01 26.20 0.00 433.08 0.00 0.43 0.00
The resulting output includes the following information:
- timestamp: The time of data collection.
- DEV: This column indicates the names of block devices.
- tps: This shows the total number of transfers per second sent to block devices.
- rkB/s: This represents the data read from each block device, expressed in kilobytes per second.
- wkB/s: This column shows the rate at which data is written to block devices in kilobytes per second.
- dkB/s: This column indicates the number of kilobytes discarded per second.
- areq-sz: This column shows the average size of I/O requests in kilobytes sent to block devices. This field was known as avgrq-sz in previous versions of sar and was previously expressed in sectors.
- aqu-sz: This represents the average queue length of requests sent to block devices. It was previously known as avgqu-sz.
- await: This column shows the average time it takes in milliseconds to serve I/O requests sent to block devices.
- %util: This represents the percentage of time the device was busy with I/O operations.
Additionally, sar will generate a report for a specific block device when the "--dev" option is included and has passed the device name as shown below:
sar -d --dev=sdc 05:30:27 DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 05:40:14 sda 11.90 340.30 39.73 0.00 31.93 0.11 9.46 4.89 05:50:00 sda 1.50 5.22 6.04 0.00 7.53 0.00 2.39 0.40 05:50:00 DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 06:00:21 sda 1.17 4.62 11.80 0.00 14.01 0.00 1.69 0.23 06:10:19 sda 3.47 26.77 235.41 0.00 75.46 0.01 2.50 0.79
9. Get Network Metrics
The "-n" option is used to produce a report on network activity. However, it requires a keyword value or a network type or "ALL", which encompasses all keywords. Acceptable keyword values include:
- DEV: For all network interfaces.
- EDEV: For errors on network interfaces.
- FC: For fibre channel traffic.
- ICMP: For ICMPv4 traffic.
- EICMP: For ICMPv4 network errors.
- ICMP6: For ICMPv6 traffic
- EICMP6: For ICMPv6 network errors.
- IP: For IPv4 traffic.
- EIP: For IPv4 error messages.
- IP6: For IPv6 traffic.
- EIP6: For IPv6 network errors.
- NFS: For NFS client activity.
- NFSD: For NFSD server activity.
- SOCK: For IPv4 sockets.
- SOCK6: For IPv6 sockets.
- SOFT: For software-based network processing.
- TCP: For TCPv4 traffic.
- ETCP: For TCPv4 network errors.
- UDP: For UDPv4 traffic.
To obtain the network metrics of all available network interfaces, run the following command:
sar -n DEV Linux 5.19.0-42-generic (ubuntu) 29/05/2023 _x86_64_ (4 CPU) 05:26:45 LINUX RESTART (4 CPU) 05:30:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 05:40:14 lo 0.10 0.10 0.01 0.01 0.00 0.00 0.00 0.00 05:40:14 ens33 0.63 0.31 0.26 0.15 0.00 0.00 0.00 0.00 05:40:14 mpqemubr0 0.22 0.27 0.08 0.16 0.00 0.00 0.02 0.00 05:40:14 tap-e863670417c 0.07 0.14 0.03 0.05 0.00 0.00 0.00 0.00 05:40:14 tap-85a8e535edf 0.02 0.07 0.00 0.01 0.00 0.00 0.00 0.00 05:40:14 tap-3a2af97fc7d 0.07 0.14 0.03 0.05 0.00 0.00 0.00 0.00 05:40:14 tap-68433156c50 0.07 0.13 0.03 0.07 0.00 0.00 0.00 0.01
The resulting output shows the following information:
- timestamp: This shows the time of data capture.
- IFACE: This column shows the name of the network interface from which data was collected.
- rxpck/s: This represents the total number of packets received per second in an interface.
- txpck/s: This represents the total number of packets transmitted per second in an interface.
- rxkB/s: This column shows the total number of kilobytes received per second in an interface.
- tkKB/s: This column indicates the total number of kilobytes transmitted per second in an interface.
- rxcmp/s: This represents the number of compressed packets received per second in an interface.
- txcmp/s: This column shows the number of compressed packets transmitted per second in an interface.
- rxmcst/s: This column indicates the number of multicast packets received per second in an interface.
- %ifutil: This column shows the percentage of the network interface utilized.
However, it is worth noting that the fields (columns) present in a network report depend on the selected keyword. The man page of the sar command provides a comprehensive list of all the possible fields of the network report output.
10. Get Power Management Metrics
The "-m" option is used to produce reports on power management statistics. Similar to the "-n" option, it requires a keyword to determine the statistics to be displayed. Acceptable keyword values are:
- CPU: This reports CPU-related statistics.
- FAN: This shows information about the fan speed.
- FREQ: This shows information about the CPU clock frequency.
- IN: This shows information about the voltage inputs.
- TEMP: This reports statistics on the device's temperature.
- USB: This shows a summary of the currently plugged-in USB devices.
- ALL: This is equivalent to providing all the previous keywords.
To generate power management statistics for the CPU, execute the following command:
sar -m CPU 5 2 Linux 5.19.0-42-generic (ubuntu) 29/05/2023 _x86_64_ (4 CPU) 23:46:37 CPU MHz 23:46:42 all 1190.39 23:46:47 all 1190.39 23:46:52 all 1190.39 23:46:57 all 1190.39 Average: all 1190.39
As you can observe from the output above, sar produces a general report for all the CPU cores available on the system. The output also contains the MHz field, which represents the instantaneous CPU clock frequency in MHz.
11. Get Historical Metrics
As mentioned earlier in the article, captured metrics are stored in log files with a naming convention of saDD, where DD represents the day of data collection. Thus, you can review statistics from a specific time interval of your choosing.
To obtain data for a designated timeframe, the "-s" and "-e" options are used with the corresponding sar command and provided at the start and end time range in a 24-hour format, respectively. For example, executing the following command will retrieve memory metrics captured between 8:15 AM – 9:10 AM today.
sar -r -s 08:15:00 -e 09:10:00 Linux 5.19.0-42-generic (ubuntu) 30/05/2023 _x86_64_ (4 CPU) 08:20:21 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 08:30:12 431072 2456416 4921848 60.73 51684 2178708 14613700 143.25 2661280 3691008 8 08:40:21 429312 2454812 4923436 60.75 51828 2178716 14613832 143.25 2661424 3690256 4 08:50:31 428584 2454268 4923944 60.75 51996 2178728 14614004 143.25 2661600 3690760 0 09:00:17 428108 2453988 4924204 60.76 52172 2178752 14614136 143.25 2661768 3691536 4 Average: 429269 2454871 4923358 60.75 51920 2178726 14613918 143.25 2661518 3690890 4
However, sar also allows you to retrieve data from previous days when you provide a relative day offset as an option in the command. Therefore, the option "-1" corresponds to the previous day, while "-2" would mean two days prior, and so on.
To retrieve the CPU metrics captured yesterday between 4 PM - 4:30 PM, run:
sar -u -1 -s 16:00:00 -e 16:30:00 Linux 5.19.0-42-generic (ubuntu) 29/05/2023 _x86_64_ (4 CPU) 16:00:03 CPU %user %nice %system %iowait %steal %idle 16:10:12 all 0.82 0.00 0.45 0.46 0.00 98.27 16:20:04 all 0.84 0.00 0.62 0.14 0.00 98.40 Average: all 0.83 0.00 0.53 0.30 0.00 98.34
Query Specific Log File
Alternatively, with the "-f" option, you can specify the log file that corresponds to the day from which you wish to obtain data. As noted earlier, these files are often stored in the directory "/var/log/sa." However, on Ubuntu, they are stored at "/var/log/sysstat."
The following command demonstrates how to retrieve the CPU metrics captured on the 28th day of the month.
sar -u -s 16:00:00 -e 16:30:00 -f /var/log/sysstat/sa28 Linux 5.19.0-42-generic (ubuntu) 28/05/2023 _x86_64_ (4 CPU) 16:00:00 CPU %user %nice %system %iowait %steal %idle 16:10:00 all 0.16 0.00 0.16 0.00 0.00 99.68 16:20:00 all 0.20 0.00 0.20 0.01 0.00 99.59 16:30:00 all 0.21 0.00 0.18 0.01 0.00 99.60 Average: all 0.19 0.00 0.18 0.01 0.00 99.63
Get All Metrics
The "-A" option is used to generate a report which shows all the collected data. It is equivalent to invoking the following sar command:
sar -bBdFHISvwWy -m ALL -n ALL -q ALL -r ALL -u ALL
To produce a report on all collected data, run:
sar -A
Save Generated Report
sar also provides a way to store your generated reports in custom files rather than relying on the default log files created by the sadc. The "-o" option is used with the corresponding sar command to save the produced report.
The command below illustrates how to save the CPU metrics captured 4 times over a 5-second interval:
sudo sar -u 5 4 -o /tmp/metrics
In the command above, the "-o" option is given a value "/tmp/metrics", which indicates where the file should be saved. It's worth noting that if no value is specified, the default sadc log file will be used.
After saving the generated report, the "-f" option can be used to read the resulting file as shown below:
sar -u -f /tmp/metrics/sa31 Linux 5.19.0-42-generic (ubuntu) 31/05/2023 _x86_64_ (4 CPU) 06:35:29 CPU %user %nice %system %iowait %steal %idle 06:35:34 all 22.61 0.00 1.79 0.06 0.00 75.48 06:35:39 all 11.21 0.00 4.64 0.08 0.00 84.07 06:35:44 all 9.90 0.00 8.56 2.67 0.00 78.87 06:35:49 all 15.11 0.00 19.86 0.40 0.00 64.63 Average: all 15.31 0.00 8.16 0.75 0.00 75.78
Conclusion
sar is a valuable tool that provides insights into system activity and performance. It offers extensive monitoring and reporting features, which allow you to assess your system's health and resource utilization, and easily identify potential performance issues.
Through the various options and parameters that sar provides, you can keep track of a wide range of system metrics, including CPU usage, memory utilization, power management, disk activity, network traffic, and much more.
Overall, sar is an essential tool for maintaining the smooth operation of your system.
