Demystifying htop and top on Linux (2019)

Originally published by peteris.rocks

For a long time, the metrics displayed in htop were a mystery to me. For instance, I used to think that a load average of 1.0 on a dual-core machine automatically meant 50% CPU usage. Since the best way to master a topic is to teach it, I've put together this guide.

---

🛠️ Understanding System Uptime

When you look at htop (or run the uptime command), you see a summary of the system's current state.

Running the command: $ uptime 12:17:58 up 111 days, 31 min, 1 user, load average: 0.00, 0.01, 0.05

How does the system track this?

The uptime utility retrieves its data from the kernel. To find out exactly how, we can use strace to monitor system calls. Since strace sends its output to stderr, we must redirect it to stdout using 2>&1.

# Using strace to see which files uptime accesses
strace -e open uptime 2>&1 | grep uptime

The output reveals that the program opens: open("/proc/uptime", O_RDONLY) = 3

If we examine the contents of /proc/uptime, we see two numbers: 9592411.58 9566042.33

Note: On multi-core systems, the idle time can actually be higher than the total uptime because it is an aggregate of all CPUs.

Why use a program instead of reading the file? While /proc/uptime provides raw data for scripts, the uptime command formats that data into a human-readable string.

---

📈 Decoding Load Average

The uptime command also displays three numbers representing the load average. These are sourced from /proc/loadavg.

$ cat /proc/loadavg
0.00 0.01 0.03 1/120 1500

Breaking down the columns:

• Columns 1-3: The average system load over the last 1, 5, and 15 minutes, respectively.
• Column 4: The ratio of running processes / total processes.
• Column 5: The most recent Process ID (PID) assigned.

Process IDs (PIDs)

Every process gets a unique ID. These generally increase over time unless the system runs out of IDs and begins recycling them.

• PID 1: Always belongs to /sbin/init, the first process started at boot.

"Running" vs. "Sleeping"

Not every process is "running." Let's test this:

1. Idle Process: If you run sleep 10 &, the process is created, but it is sleeping (waiting). It does not count toward the "running" number in /proc/loadavg.
2. Active Process: If you run a CPU-intensive task, like generating random bytes: $ cat /dev/urandom /dev/null & Now, the running process count will increase.

---

🧮 The Mathematics of Load

The load average isn't a simple arithmetic mean; it is an exponentially damped moving average.

According to Wikipedia:

All three values average the system load since startup, but they decay at different rates.

For the 1-minute average, the calculation is roughly: $\text{Load}_1 = (0.63 \times \text{Load}_{\text{last minute}}) + (0.37 \times \text{Load}_{\text{since startup}})$

This means the 1-minute average is mostly influenced by the last 60 seconds, but it still carries a "memory" of previous activity.

Load vs. CPU Utilization

To understand if your system is overloaded, you must know your core count (use nproc or check the top-left of htop).

Crucial Nuance: Load includes both running processes and uninterruptible processes (those waiting for disk I/O or network responses).

Because of this, you might see a high load average even if CPU usage is low—this usually indicates an I/O bottleneck. For instantaneous CPU metrics, tools like mpstat are more precise.

---

🏁 Summary Checklist

• Understand /proc/uptime (Total vs. Idle time)
• Understand /proc/loadavg (1, 5, 15 min averages)
• Differentiate between Running and Sleeping processes
• Relate Load Average to the number of CPU cores
• Recognize that Load $\neq$ CPU usage due to uninterruptible states