Search our site for answers
Help Docs Performance Server Optimization System activity reporting with sar

System activity reporting with sar

sar monitors Linux system activity (CPU, memory, I/O) over time. It's crucial for historical performance analysis and troubleshooting server bottlenecks.
Performance Troubleshooting
5 min read

sar (System Activity Report) It is a powerful command-line utility for collecting, reporting, and saving system activity information. Installed by default on many Linux distributions like CentOS, it’s particularly useful for tracking down the root cause of performance issues, especially those related to disk I/O, CPU, and memory. sar maintains its historical data in a series of files located in the /var/log/sa/ directory.

Viewing historical data

sar automatically collects data at regular intervals and stores it in daily files. To see a list of the previous month’s reports, you can use the ls command:

ll /var/log/sa/

Example Output:

-rw-r--r--  1 root root 484640 Aug  1 15:51 sa01
-rw-r--r--  1 root root 484176 Aug  2 23:50 sa02
-rw-r--r--  1 root root 228576 Aug  3 11:00 sa03
...
-rw-r--r--  1 root root 491873 Aug  1 23:53 sar01
-rw-r--r--  1 root root 491907 Aug  2 23:53 sar02

You’ll notice files named saXX (binary data files) and sarXX (text reports, if generated). XX represents the day of the month.

Basic usage: CPU activity

Running sar without any flags will show you a breakdown of CPU usage from the current day. This output is similar to what you might see from iostat‘s CPU section.

Command:

sar

Example Output:

12:30:01 PM       CPU    %user    %nice  %system  %iowait   %steal    %idle
12:40:01 PM       all      0.13      0.01      0.08      0.02      0.00     99.76
12:50:01 PM       all      0.13      0.02      0.07      0.02      0.00     99.76
01:00:01 PM       all      0.12      0.01      0.08      0.03      0.00     99.77
01:10:01 PM       all      0.22      0.01      0.09      0.05      0.00     99.63
Average:          all      0.15      0.01      0.08      0.03      0.00     99.73

This basic report is useful for quickly assessing overall CPU load and identifying if iowait (time spent waiting for I/O) is a significant factor.

Specific metrics with flags

sar offers numerous flags to retrieve specific types of system activity:

CPU load averages (-q)

The -q flag displays the CPU load averages over time, similar to the output from uptime or top.

Command:

sar -q

Example Output:

12:00:05 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
12:10:04 AM         0       385      5.12      7.26      7.90
12:20:15 AM         1       402     13.63      9.29      8.08
Average:            1       362      8.03      9.36      8.60
  • runq-sz: Run queue size (number of tasks waiting for CPU).
  • plist-sz: Number of tasks in the task list.
  • ldavg-1, ldavg-5, ldavg-15: Load average over the last 1, 5, and 15 minutes.
Memory and swap usage (-r)

Use the -r flag to see historical memory and swap usage, both in percentages and kilobytes. This helps in diagnosing RAM-related bottlenecks.

Command:

sar -r

Example Output:

12:30:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
12:40:01 PM     70076   1687976     96.01    191540    984628   1959852        68      0.00         0
Average:        69309   1688743     96.06    191548    984871   1959852        68      0.00         0
  • kbmemfree: Amount of free memory available.
  • kbmemused: Amount of used memory.
  • %memused: Percentage of used memory.
  • kbbuffers: Memory used by kernel buffers.
  • kbcached: Memory used by the page cache.
  • kbswpfree, kbswpused, %swpused: Free, used, and percentage of used swap space.
  • kbswpcad: Amount of cached swap memory.
Swap space utilization (-s)

The -S flag provides detailed information about swap space utilization.

Command:

sar -S

Example Output:

12:00:02 AM kbswpfree kbswpused  %swpused  kbswcad  %swpcad
12:10:01 AM   1884668    212480     10.13     3200     1.51
Average:      188468     212480     10.13     3200     1.51
Swap activity (-w)

To see actual page-in and page-out activity to and from swap space, use the -W flag. This helps identify if your system is actively swapping, which can indicate memory pressure.

Command:

sar -W

Example Output:

12:30:01 PM  pswpin/s pswpout/s
12:40:01 PM      0.00      0.00
Average:         0.00      0.00
  • pswpin/s: Number of pages swapped in per second.
  • pswpout/s: Number of pages swapped out per second.
Time range and processor specifics

sar also allows you to specify time ranges and monitor individual processors:

Specifying a time range (-s and -e )

The -s (start time) and -e (end time) flags can be used to specify a particular time range for your sar output. Both flags follow the format HH:MM:SS.

Command:

sar -s 09:00:00 -e 10:30:00
Per-processor usage (-P)

The -P flag allows you to show CPU usage for a specific processor, or all processors using -P ALL. You can also specify multiple processor IDs separated by commas (e.g., -P 0,1,2).

Command (All Processors):

sar -P ALL

Example Output (-P ALL):

12:00:05 AM     CPU    %user    %nice  %system  %iowait   %steal    %idle
12:10:04 AM     all     18.88      0.70      5.27     75.15      0.00      0.00
12:10:04 AM       0     18.88      0.70      5.27     75.15      0.00      0.00

Command (Specific Processors):

sar -P 0,1,2

Example Output (-P 0,1,2):

12:00:01 AM     CPU    %user    %nice  %system  %iowait   %steal    %idle
12:10:01 AM       0      3.63      0.00      0.29      0.05      0.00     96.03
12:10:01 AM       1      4.76      0.00      0.33      0.04      0.00     94.88
12:10:01 AM       2      0.97      0.00      0.07      0.00      0.00     98.96
Checking logs for a different day (-f)

While sar by default shows the current day’s data, you’ll often need to examine older logs. You can do this using the -f flag, specifying the path to the daily data file (saXX from /var/log/sa/).

Command:

sar -f /var/log/sa/sa01

You can combine the -f flag with other sar flags (like -W for swap activity or -r for memory) to get specific information from historical data:

Command:

sar -f /var/log/sa/sa01 -W
Helpful tip: Daily averages one-liner

Here’s a useful one-liner function to quickly see the sar information on a per-day average for the recorded month. Copy and paste this into your terminal:

testfunction () { sar -$1 |head -3;ls /var/log/sa/sa[0-9]*|xargs -I '{}' sar -$1 -f {}| grep -E -o '(../../....|Average:.*)'; }

Then, run the function with the desired sar flag (e.g., W for swap info or q for load average):

testfunction W

Or for load average:

testfunction q

Conclusion

sar is an incredibly versatile and powerful tool for in-depth system performance monitoring and historical analysis. By understanding its various flags and how to interpret its output, you can effectively track CPU, memory, and disk I/O trends, allowing you to proactively identify and resolve performance bottlenecks on your server. For any advanced diagnostics or assistance, Liquid Web’s Heroic Support® team is always available.

Was this article helpful?