DStat Is Now PCP – Performance Co-Pilot

Performance Co-Pilot (PCP) is a compact assessment and evaluation tool that is used to collect a wide array of server metrics and examines current and prior operational data. It is designed to be easily customizable utilizing modern APIs and other resources used to extend its functionality. Redhat has taken many of the best features of DStat and coupled them with additional capabilities that extended its usefulness, and have created their own updated version called Performance Co-Pilot.

PCP (or Performance Co-Pilot) now differentiates between the displayed metrics from the data collected via a PMAPI (or a performance metrics API). This feature lets programmers write unique functions to collect data once and then reuse that same operation across different requests. This allows for improved reporting by abstracting the returned data via calls using the PMAPI instead of the visualized sources directly.

PCP pulls much of the same information as top, sar, iotop, iostat, iftop, vmstat, netstat, and ifstat. Unfortunately, many of these tools do not work well together, but PCP solves that issue. Its plugin framework also allows for the collection of performance data from a single server. Additionally, PCP uses a client-server architecture that allows it to monitor a single host from multiple clients or multiple hosts from a single client.

Installation

Let’s begin by installing PCP.

Redhat/CentOS

[root@host ~]# yum install -y pcp pcp-doc pcp-gui pcp-system-tools

Debian/Ubuntu

[root@host ~]# apt-get install pcp pcp-do pcp-gui pcp-system-tools
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libpcp-gui2 libpcp-import1 libpcp-mmv1 libpcp-pmda-perl libpcp-pmda3 libpcp-trace2 libpcp-web1 libpcp3 libpfm4 libqt5printsupport5 pcp-conf python3-pcp

Suggested packages:
  libpcp-import-perl pcp-doc
The following NEW packages will be installed:

  libpcp-gui2 libpcp-import1 libpcp-mmv1 libpcp-pmda-perl libpcp-pmda3 libpcp-trace2 libpcp-web1 libpcp3 libpfm4 libqt5printsupport5 pcp pcp-conf pcp-gui python3-pcp

0 upgraded, 14 newly installed, 0 to remove and 30 not upgraded.
Need to get 4,648 kB of archives.
After this operation, 18.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
…
…
…
Setting up python3-pcp (4.3.4-1build1) ...
Setting up pcp (4.3.4-1build1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/pmcd.service → /lib/systemd/system/pmcd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/pmlogger.service → /lib/systemd/system/pmlogger.service.
Created symlink /etc/systemd/system/multi-user.target.wants/pmie.service → /lib/systemd/system/pmie.service.
Created symlink /etc/systemd/system/multi-user.target.wants/pmproxy.service → /lib/systemd/system/pmproxy.service.
Processing triggers for libc-bin (2.30-0ubuntu2) ...
Processing triggers for systemd (242-7ubuntu3) ...
Processing triggers for man-db (2.8.7-3) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu1) ...
Processing triggers for mime-support (3.63ubuntu1) ...
Processing triggers for gnome-menus (3.32.0-1ubuntu1) ...
[root@host ~]# 

Git

[root@host ~]# git clone https://github.com/performancecopilot/pcp.git
cd pcp
[root@host ~]# ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
[root@host ~]# make
[root@host ~]# make install

Enable The Service and Start The Daemons

[root@host ~]# systemctl enable pmcd

Created symlink from /etc/systemd/system/multi-user.target.wants/pmcd.service to /usr/lib/systemd/system/pmcd.service.

[root@host ~]# 
[root@host ~]# systemctl enable pmlogger
Created symlink from /etc/systemd/system/multi-user.target.wants/pmlogger.service to /usr/lib/systemd/system/pmlogger.service.
[root@host ~]#

[root@host ~]# systemctl start pmcd
[root@host ~]# systemctl start pmlogger

Examples

System Info - Stats

[root@host ~]# pcp
Performance Co-Pilot configuration on host.bare-centos7.com:

 platform: Linux host.test-centos7.com 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64
 hardware: 2 cpus, 1 disk, 1 node, 1871MB RAM
 timezone: EST+5
 services: pmcd
     pmcd: Version 4.3.2-1, 9 agents, 1 client
     pmda: root pmcd proc pmproxy xfs linux mmv kvm jbd2
 pmlogger: primary logger: /var/log/pcp/pmlogger/host.test-centos7.com/20191216.08.50

(Use Ctrl+C to exit)

PCP-Atop

(Use Ctrl+C to exit)

pmiostat

[root@host ~]# pmiostat -t 2sec
# Device      rrqm/s  wrqm/s     r/s    w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await   %util
sda             0.00   20.46    0.00   5.99     0.00    97.82   16.333    0.040    6.75    0.00    6.75    1.40
sdb             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sdc             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sda             0.00   12.96    0.00   5.98     0.00    67.81   11.333    0.034    5.75    0.00    5.75    1.20
sdb             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sdc             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sda             0.00   22.92    0.00   7.48     0.00   143.52   19.200    0.043    5.80    0.00    5.80    1.79
sdb             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sdc             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sda             0.00   18.44    0.00   5.98     0.00    89.70   15.000    0.034    5.75    0.00    5.75    1.79
sdb             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
sdc             0.00    0.00    0.00   0.00     0.00     0.00    0.000    0.000    0.00    0.00    0.00    0.00
[root@host ~]# 

pmstat

[root@host ~]# pmstat
@ Mon Dec 16 09:36:05 2019
 loadavg                      memory      swap        io    system         cpu
   1 min   swpd   free   buff  cache   pi   po   bi   bo   in   cs  us  sy  id
    0.57 120320  1616m  1395m  3826m    0    0    0  281 1005 1536   2   1  98
    0.52 120320  1612m  1395m  3826m    0    0    0  746  938 1362   2   0  98
    0.48 120320  1556m  1395m  3859m    0    0    0  361 4757 7986  19   3  78
    0.44 120320  1563m  1395m  3840m    0    0    0  343 4139 6424  16   3  81
    0.57 120320  1541m  1395m  3853m    0    0    0  471 5071 8033  23   4  73
    0.52 120320  1555m  1395m  3846m    0    0    0  494 4384 7030  12   4  84

[root@host ~]# 

pcp dstat

The pcp dstat command is similar in nature to the older versions of dstat. It uses the options below:

pcp [ pcp options ] dstat [ -acdfghilmnpqrstvVy? ] [ -C cpus ] [ -D disks ] [ -I interrupts ] [ -N interfaces ] [ -o output-file ] [ -S swap-devices ] [ --bits ] [ --bw ] [ --color ] [ --float ] [ --integer ] [ --nocolor ] [ --noheaders ] [ --noupdate ] [ --list ] [ --pidfile pid-file ] [ -- plugin ] [ --all-plugins ] [ delay [ count ]]

The default output of the pcp dstat command looks like this.

[root@host ~]# pcp dstat
You did not select any stats, using -cdngy by default.
----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
  3   1  95   1   0|   0   244k| 972  1623 |   0     0 |1519  2235 
  3   1  96   0   0|   0     0 | 414   264 |   0     0 |1074  1393 
  2   0  97   1   0|   0   136k| 189   189 |   0     0 |1103  1640 
  2   1  96   0   0|   0     0 |6928  4328 |   0     0 |1341  1758 
  4   1  94   1   0|   0   176k| 444   700 |   0     0 |1496  2065 
  2   1  97   0   0|   0    12k| 209   275 |   0     0 | 999  1360 
  3   1  96   1   0|   0   120k|2375  2423 |   0     0 |1386  1973 
  2   1  97 307   0|   0  8190B|   0     0 |   0     0 |1106  1532 
  3   1  94   2   0|   0   900k| 358   547 |   0     0 |1492  2110 
  2   1  96   0   0|   0     0 | 531  1695 |   0     0 |1267  1882 
  7   7  85   1   0|   0   296k| 190   178 |   0     0 |3634  5890 

A full view of nearly all the metrics literally spans two widescreen monitors.

Performance Metrics Domain Agents (PMDAs)

The default installation of PCP provides the metrics for your cpu, file system, per-process, network, swap, memory, disk, interrupts, nfs/rpc and others. These metrics are processed using the platform PMDA - specifically the pmda_linux.so file in Linux. There are also corresponding pmda files for Mac and Windows.

PMDAs in the pcp package includes:

• apache - monitor apache web server stats
• cisco - monitor Cisco router stats
• dbping - query any database, extract response times
• elasticsearch - monitor an elasticsearch cluster
• kvm - monitor kernel-based virtual machine stats
• mailq - monitor the mail queue
• memcache - monitor memcache server stats
• mmv - export memory-mapped value stats from an application
• mounts - keep track of mounted file systems
• mysql - monitor MySQL relational databases
• oracle - monitor Oracle relational databases
• postgres - monitor PostGreSQL relational databases
• process - keep an eye on critical processes/daemons
• roomtemp - monitor room temp (needs suitable probe)
• rsyslog - monitor the reliable system log daemon
• sendmail - monitor sendmail statistics
• shping - ping critical system services, extract response times
• trace - for instrumenting arbitrary applications, see pmtrace(1)
• txmon - transaction and QOS monitoring
• statsd - StatsD protocol data collector
• sample - for testing
• simple - example src code if you want to write a new PMDA
• trivial - even easier src code for a new PMDA.

None of the PMDAs are setup by default. You select one of the many PMDAs needed and then run the install script found in the corresponding /var/lib/pcp/pmdas folders.

pmchart

Open View

Default SCSI Disk Traffic View

Default Live CPU and Load Average View

Start Collecting Info

To enable historical data collection, use the following commands:

[root@host ~]# systemctl enable pmlogger
[root@host ~]# systemctl startpmlogger

The configuration for pmlogger can be found in /etc/pcp/pmlogger. The default interval for collecting data is 1 minute, and archives are located in /var/log/pcp/pmlogger/hostname.

Plugins

Plugins were a huge part of the original dstat, and in carrying on this vital work, Redhat has implemented this feature via the use of its PMAPI functionality. Now, all of the PCP dstat metrics that are displayed are from 'plugins' defined and outlined in config files. This provides an even greater advantage by making it easier to produce and build new plugins for PCP's dstat.

Exported Info

Previously, dstat only allowed the export of data to a CSV file. With dstat now being integrated into PCP, dstat now has the ability to export collected data into the following formats:

• pcp2elasticsearch
• pcp2graphite
• pcp2influxdb
• pcp2json
• pcp2spark
• pcp2xlsx
• pcp2xml
• pcp2zabbix

With the integration of dstat and PCP, the added benefit of utilizing PCP’s own 20-year-old archive format is the increased usability and functionality of the 'pmlogger' program which now runs as a service.  When a PCP client tool uses the PMAPI (as this dstat implementation does), it also gains access to metrics stored in archives without having to implement the functionality itself. This means even the brand-new dstat implementation can display previously gathered values as if they were live, using PCP standard flags to specify the timeframe of metrics to display.

Graphical Output

PCP graphical charts work like a native client program that presents data that has been collected from localhost, a remote server, or from archived files. Multiple report types are presented, including the “Overview Report,” which provides a very nice console type view of the system. The graphics in the overview are very functional and easy to use for investigation.

Custom Widgets

You can implement the GUI feature in PCP by modifying the “enableCustomWidgetFeature” setting to true in the /src/app/config.js file and then rebuilding. Next, if you open the Dashboard dropdown, and then select “Custom Chart” or “Custom Table” from under the Custom section, this will generate a brand-new chart panel to your dash.

Commands

Here is a shortlist of some of the commands available for use with PCP. To see a full listing of all of the available options visit the Performance Co-Pilot home page.

[root@host ~]# man PCPIntro
[root@host ~]# man pcp
[root@host ~]# man pmrep
root@host ~]# man pmcd
root@host ~]# man pminfo
root@host ~]# man pmie

See It In Action Today!

Do you have a Liquid Web Dedicated server, Private Cloud or a Liquid Web Server Cluster that requires constant monitoring? Reach out to one of our experienced Linux System Administrators today to get more information.

If you are experiencing problems setting this software up, give us a call today at 800.580.4985, or open a chat or ticket with us and one of our knowledgeable technicians will provide more info!