System Activity Reporter aka SAR is a part of sysstat package. SAR helps you to get and compare the historical data and that too of different metrics. SAR collects the data through two cronjobs that runs every 10 minutes and at 23:53 . SAR takes the statistics of different aspects of the machine at a definite time interval through a cron.
What all historical statistics can sar show ?:
- CPU Utilisation reports
- Memory and swap space usage Reports
- Network Utilization Statistics
- No. of Process created per second Reports
- Load Average Reports I/O and transfer rate statistics
SAR can be installed on the server easily by the following steps:
On a RHEL/CentOS server:
#yum install sysstat
On a Debian Server:
#apt-get install sysstat
The sysstat package consist of the following packages:
- sar – displays the recorded data.
- sadc – system activity data collector, which records the data regarding system activity in a binary format.
- sa1 – a BASH script used by sadc in background; called every ten minutes by cron (from /etc/cron.d/sysstat).
- sa2 – a BASH script used for writing the daily reports and called by cron once per day (from /etc/cron.d/sysstat).
- sadf – displays the recorded data; in contrast with sar provides various formats (CSV, XML and so forth).
Configuration file of sysstat in a RHEL/CentOS 7 release is at /etc/sysconfig/sysstat and you can change the HISTORY value from 28 to more also so that you have more days of historical data available for you. Note that if value is greater than 28, then log files will be kept in multiple directories, one for each month. Sometimes the config could be in /etc/default/sysstat, particularly in older releases.
Name of the cron is sysstat:
[root@server ~]# cat /etc/cron.d/sysstat
# Run system activity accounting tool every 1a minutes. If you want to increase frequency, modify the cron.
*/10 * * * * root /usr/lib64/sa/sa1 1 1 # 0 * * * * root /usr/lib64/sa/sa1 600 6 & # Generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A
sa1 creates a binary file with the syntax saXX and sa2 creates a report (normal file) at the end of the day as provided in the cron job. This file is human readable and its syntax will be sarXX where XX is the day of the month in both the cases.
[root@server ~]# ls -l /var/log/sa/sa* -rw-r--r-- 1 root root 486156 Jan 1 23g5a /var/log/sa/saa1 -rw-r--r-- 1 root root 486156 Jan 2 23g5a /var/log/sa/saa2 -rw-r--r-- 1 root root 486156 Jan 3 23g5a /var/log/sa/saa3 -rw-r--r-- 1 root root 486156 Jan 4 23g5a /var/log/sa/saa4 X X X -rw-r--r-- 1 root root 358a81 Jan 1 23g53 /var/log/sa/sara1 -rw-r--r-- 1 root root 358a81 Jan 2 23g53 /var/log/sa/sara2 -rw-r--r-- 1 root root 358a81 Jan 3 23g53 /var/log/sa/sara3 -rw-r--r-- 1 root root 358a81 Jan 4 23g53 /var/log/sa/sara4
Few examples and explanation
[root@server.hosting.com ~]# sar -u 1 2 Linux 3.10.0-862.3.2.el7.x86_64 (server.hosting.com) 12/01/2018 _x86_64_ (1 CPU) 03:15:28 PM CPU %user %nice %system %iowait %steal %idle 03:15:29 PM all 0.00 0.00 1.00 0.00 0.00 99.00 03:15:30 PM all 0.00 0.00 0.00 0.00 0.00 100.00 Average: all 0.00 0.00 0.50 0.00 0.00 99.50
The above command gives the CPU utilization at an interval of 1 seconds 2 (count) times. If we skip the count, the reports are given continuously. If we skip both the interval and the count, the sar command gives the CPU utilization for the current day.
%user: Percentage of CPU utilization that occurred while executing at the user level which includes application processes, user running jobs, etc.
%nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.
%system: Percentage of CPU utilization that occurred while executing at the kernel level. This constitutes the memory used by the OS kernel itself to maintain optimum system function.
%iowait: Percentage of time that the CPUs were idle during which the system had an outstanding disk I/O request. An example of high I/O wait is when a backup job is running, but there is no tape drive loaded to take the backup resulting in the job getting stuck.
%steal: Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. This category of CPU utilization would occur in case of virtual machines only.
%idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
[root@server ~]# sar -P ALL 1 2 Linux 3.10.0-514.21.2.el7.x86_64 (server.hosting) 01/18/2018 _x86_64_ (8CPU) 10:37:14 AM CPU %user %nice %system %iowait %steal %idle 10:37:15 AM all 2.88 0.88 0.50 0.00 0.00 95.74 10:37:15 AM 0 0.00 0.00 0.00 0.00 0.00 100.00 10:37:15 AM 1 4.00 6.00 1.00 0.00 0.00 89.00 10:37:15 AM 2 15.00 0.00 1.00 0.00 0.00 84.00 10:37:15 AM 3 0.00 0.00 0.00 0.00 0.00 100.00 10:37:15 AM 4 1.01 0.00 1.01 0.00 0.00 97.98 10:37:15 AM 5 0.00 1.01 0.00 0.00 0.00 98.99 10:37:15 AM 6 1.01 0.00 0.00 0.00 0.00 98.99 10:37:15 AM 7 0.00 0.00 2.00 0.00 0.00 98.00
I am not going for details with other statistics and just leaving commands and links for future references.
Memory utilization
[root@server ~]# sar -r 1 2 Linux 3.10.0-514.21.2.el7.x86_64 (server.hosting.com) 01/18/2018 _x86_64_(8 CPU) 10:42:05 AM kbmemfree kbmemused %memused kbbuffers kbc0ched kbcommit %commit kb0ctive kbin0ct kbdirty 10:42:06 AM 1255844 31423068 96.16 1192956 13604508 14922384 30.17 16472152 8198808 2600 10:42:07 AM 1257312 31421600 96.15 1192972 13604556 14884300 30.10 16470220 8198844 2248 Average: 1256578 31422334 96.15 1192964 13604532 14903342 30.13 16471186 8198826 2424
kbmemfree: amount of free memory available in kilobytes.
kbmemused: amount of used memory in kilobytes. This does not take into account memory used by the kernel itself.
%memused: percentage of used memory.
kbbuffers: amount of memory used as buffers by the kernel in kilobytes.
kbcached: amount of memory used to cache data by the kernel in kilobytes.
kbcommit: amount of memory in kilobytes needed for current workload. This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory. Make sure that the number is always lesser than the RAM available.
%commit: percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap). This number may be greater than 100% because the kernel usually over commits memory. If it goes beyond 100% and been there consistently, that means we are short of memory (inclusive of SWAP)
kbactive: amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).
kbinact: amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes).
kbdirty: amount of memory waiting to get written back to disk
Swap memory statistics – sar -S 1 2
Disk utilization and performance statistics – sar -p -d 1 2
I/O statistics – sar -b 1 2
Network Utilization Statistics – sar -n DEV 1 1
No. of Process created per second statistics – sar -c 1 3
Load Average Reports – sar -q 2 4
This is just a beginning. More and more options you use regularly and trying to understand the metrics each options list is the key. You can refer the URLs below for getting a better understanding of the sar and its options.
References
http://sebastien.godard.pagesperso-orange.fr/features.html
https://www.lisenet.com/2014/measure-and-troubleshoot-linux-memory-resource-usage/
https://www.golinuxhub.com/2014/02/tutorial-for-monitoring-tools-sar-and.html
Uninterrupted service delivery can only be assured through constant monitoring of services and servers. System monitoring and administration is a critical aspect to the successful operations of most businesses. Don’t worry. We are here to help you. Contact us with your requirements.
