Load Investigation Basics
Overview
This guide provides the foundational knowledge required to troubleshoot server load issues effectively. It’s designed to help you understand key metrics, identify performance bottlenecks, and apply practical solutions.
Load average
Load average represents the number of processes waiting for CPU resources. It is calculated over 1, 5, and 15-minute intervals. You can find these metrics in /proc/loadavg, and they are displayed by commands like w and uptime.
Example output from w:
[root@cpanel ~]# w
12:50:18 up 325 days, 50 min, 1 user, load average: 0.44, 0.14, 0.08A high load is relative to CPU availability. A load average of 4.0 on a single-core CPU is problematic, while the same on a quad-core system may not be.
Sources of load
Server load can originate from:
- CPU Contention
- Memory Exhaustion and Swapping
- Disk I/O Bottlenecks
Run top to identify the type of load:
top - 13:18:02 up 325 days, 1:18, 1 user, load average: 0.01, 0.04, 0.05
Tasks: 127 total, 1 running, 125 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0.5 us, 0.2 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stPress 1 in top to display per-CPU stats:
us: user processessy: system/kernel processesni: niced user processesid: idle timewa: I/O waithi: hardware interruptssi: software interruptsst: stolen time (virtualization)
Other helpful top commands:
i: toggle idle processesd: change update frequency
CPU
Common causes of high CPU usage:
- Heavy PHP scripts
- MySQL queries
- Compression tasks (e.g., GZIP)
Apache
- Use
MaxRequestWorkersto limit concurrent processes. - Prefer MPM Event over Prefork.
PHP
- With suPHP/CGI, PHP limits are tied to Apache limits.
- With PHP-FPM, use
pm.max_childrento control per-account PHP processes. - PHP-FPM is more efficient than suPHP or CGI due to its persistent process model and support for Opcache.
MySQL
- High CPU from MySQL often involves table-level locks (common in MyISAM).
- Convert to InnoDB for better concurrency.
- Tune
innodb_buffer_pool_sizecarefully; excessive values can cause OOM.
Disk I/O
Indicators of high disk I/O include high wa (I/O wait) values in top.
Common causes:
- Swapping due to RAM exhaustion
- MySQL writing temporary tables
- Heavy email queue
- Scheduled backups
Use iotop to identify disk-intensive processes.
Solutions:
- Adjust InnoDB buffer pool size
- Reduce pigz process count in WHM > Tweak Settings (for cPanel backups)
Dealing with malicious or high-traffic IPs
If you identify attacking or abusive IPs:
- With Cloudflare:
- Enable Under Attack mode
- Block offending IPs via Cloudflare (never block Cloudflare IPs!)
- Ensure
mod_remoteipis installed and configured
- Without Cloudflare:
- We recommend enabling Cloudflare
- Or determine if another CDN is being used and apply appropriate protections
Useful commands to identify abused URLs:
cPanel
grep -s $(date +%d/%b/%Y): /usr/local/apache/domlogs/* | grep GET | awk '{print $7}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -n15
grep -s $(date +%d/%b/%Y): /usr/local/apache/domlogs/* | grep POST | awk '{print $7}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -n15InterWorx
grep "27/Dec/2021:07:4" /home/*/var/*/logs/transfer* | awk '{print $7}' | sort | uniq -c | sort -rn | headPlesk
grep -E '08/Jan/2020:12:(4|5)' /var/www/vhosts/*/logs/proxy_access{_,_ssl_}log | cut -d/ -f5 | sort | uniq -c | sort -rn | headMalware and spamming
Both malware and spamming can dramatically impact server load.
Steps:
- Run a malware scan (ImunifyAV or Imunify360)
- Review and act on scan results
- For Secure Server Plus customers, reach out to the support team and escalate to the Security Team if needed
Check the mail queue:
- If large, a spam investigation is required
- Investigate compromised email accounts or abused web forms
- Use CAPTCHA to mitigate spam from vulnerable forms
Summary
This guide offers practical insights into diagnosing and resolving server load issues by analyzing CPU, memory, and disk I/O metrics. It explains how to interpret load averages, identify high-resource processes using tools like top and iotop, and mitigate performance problems stemming from PHP, MySQL, backups, or abuse. It also provides actionable steps to handle malicious traffic, utilize CDN protections like Cloudflare, and address malware or spamming activity. With the right tools and awareness, you can maintain a healthy, high-performing server environment.