Diagnosing high server load

High Server Load; An Issue You'll Eventually Face

If you manage a Linux server, it's almost certain that one day you'll encounter messages like "the server is very slow" or "the site won't come up." The first thing that comes to mind is high server load. But the problem is that "load" is a vague number; a number that, if you don't know how to read it, might have you searching for the culprit in the wrong direction for hours.

The goal of this article is to clarify exactly this: how to correctly interpret load average, how to find the culprit process, and most importantly, how to determine whether the problem is CPU-related or I/O (disk input/output) related. This distinction is half the solution. Because treating a CPU-bound server is completely different from treating an I/O-bound server.

Reading Load Average; A Number Often Misinterpreted

When you run the uptime or top command, you'll see output similar to this:

load average: 4.52, 3.87, 3.21

These three numbers represent the average load over the last 1 minute, 5 minutes, and 15 minutes, respectively. But what exactly does "load" mean? In Linux, load average is the number of processes in the run queue (runnable) plus the number of processes waiting for I/O (uninterruptible sleep). This point is very important: load is not only about the CPU; a process waiting to read from disk also counts towards this number.

What is a "Good" Number?

A simple rule of thumb: compare the load average with the number of CPU cores. If the load equals the number of cores, the CPU is almost fully busy. If the load exceeds the number of cores, a wait queue forms, and server responsiveness drops.

For example, on a 4-core server, a load of 4 means the CPU is busy but there isn't a serious wait queue yet. A load of 8 means double the capacity, and users will notice significant delays. However, this rule doesn't hold for I/O-bound scenarios; a 4-core server with a load of 6 might have an almost idle CPU, with the problem stemming from the disk.

Step One: Finding the Culprit Process with top and ps

Once you've confirmed high load, the first tool is top. But just looking at the %CPU column isn't enough. My suggested order is:

1. Run the top command and press the P key to sort by CPU usage.
2. Press the M key to sort by memory usage (sometimes a RAM issue manifests itself as high load).
3. Pay attention to the STATE column: if a process is in the D state (uninterruptible sleep), it means it's waiting for I/O.

For a quicker, scriptable view, use ps:

ps -eo pid,ppid,user,stat,%cpu,%mem,cmd --sort=-%cpu | head -20

This command shows the top 20 processes by CPU usage. If you want to see output sorted by I/O, pidstat -d is a better tool (part of sysstat):

pidstat -d 2 5

This command shows the read and write rates in KB/s for each process every 2 seconds, 5 times. If you see a process like mysqld or php-fpm constantly reading from disk, the problem is I/O, not CPU.

Distinguishing CPU from I/O; The Most Important Skill in Diagnosing High Server Load

This section is the heart of the article. If you make this distinction correctly, you're halfway there. We'll examine three scenarios.

Scenario 1: CPU-bound (Problem with the Processor)

Symptoms: In top, a process's %CPU column is near 100% or higher (for multi-threaded processes). The wa column is low (below 5%). Load average is high and usually near or above the number of cores.

Real-world example: A PHP script with an infinite loop or a MySQL query that doesn't use an index and scans the entire table. In this case, the solution is code optimization or adding CPU resources, not upgrading the disk.

Scenario 2: I/O-bound (Problem with the Disk)

Symptoms: In top, the wa column is high (e.g., above 20%). Processes are in the D state. Load average is high, but the %CPU of processes is low (below 30%).

To confirm, use iostat:

iostat -x 2 3

Look at the %util column. If it's near 100%, the disk is saturated. The await column also shows the average disk response time in milliseconds; a value above 20ms for SSD and above 100ms for HDD means the disk is struggling.

Scenario 3: Combined (Both)

Sometimes the problem is combined; for example, a PHP-FPM process both consumes CPU and writes heavy logs. In this case, first resolve the I/O issue (since it's usually cheaper), then re-check the load.

More Advanced Tools for Precise Root Cause Analysis

If the basic tools aren't enough, try these:

atop; A More Complete View than top

atop also allows you to view history. Press the d key to go to the disk view and c for the CPU view. An interesting point: atop shows CPU usage on a "per-core" basis, making it easier to identify when a single-threaded process is saturating one core.

strace; Tracing System Calls

If you've found the culprit process but don't know why it's consuming so much I/O, attach strace to it:

strace -p PID -f -e trace=file,read,write -o /tmp/strace.log

After a few seconds, press Ctrl+C and examine the log file. If you see it constantly opening and closing a specific file, the problem is in the application's design.

Practical Solutions for Each Scenario

After diagnosis, it's time to act. These are the most common solutions:

If the Problem is CPU

• Optimize the application code: cache query results, use opcache for PHP, and ensure proper indexing in MySQL.
• Apply resource limits: with systemd, you can set a CPU cap. For example, for a service named myapp:

[Service]
CPUQuota=50%

This means the service can use at most half of one core.

If the Problem is I/O

• Reduce I/O volume: configure log rotation, cache queries that frequently read from disk.
• Use I/O scheduling: for HDDs, set the scheduler to deadline or mq-deadline to reduce latency.
• If you have a VPS with shared storage, check your guaranteed I/O rate (IOPS). In these cases, upgrading your plan or migrating to an NVMe disk can work wonders.

In this regard, if your server is hosted on cloud infrastructure, ServerNet offers options with NVMe disks and dedicated IOPS suitable for I/O-intensive workloads; but always make sure before upgrading that the problem is truly I/O-related and not the application code.

Summary; Final Checklist for Diagnosing High Server Load

To reach a conclusion quickly, keep this order in mind:

1. Run uptime and compare the load average with the number of cores.
2. Open top and pay attention to the %CPU, wa, and STATE columns.
3. If wa is high, confirm disk saturation with iostat -x.
4. Find the culprit process with ps or pidstat -d.
5. If needed, use strace to root-cause the process's behavior.
6. Apply the solution appropriate for the scenario (CPU or I/O) and monitor the load again.

Diagnosing high server load is a gradual skill. Each time you run this checklist, you'll get faster and rely less on guesswork. One final tip: always back up your current server configuration before any action, and apply changes one at a time so you can revert if something goes wrong.