PostgreSQL and the OOM Killer: Why We Use Strict Memory Overcommit

By Burak Yucesoy, Principal Software Engineer

Over the last 15 years, our team has designed and managed five different PostgreSQL services. Despite the evolution of the platform, one specific configuration has remained a non-negotiable constant: strict memory overcommit.

In this technical deep dive, we explore how strict overcommit prevents the Linux Out-of-Memory (OOM) killer from triggering catastrophic database failures. We will also discuss a peculiar three-character kernel bug that briefly disrupted this strategy and reveal the specific heuristic we use to calculate memory limits.

---

Why PostgreSQL Can't Tolerate the OOM Killer

To understand the danger, we first need to understand how Linux handles memory. Linux employs a strategy where processes can allocate more virtual memory than the system has physical RAM.

When a program calls malloc(), the kernel doesn't immediately hand over physical pages; it simply reserves a slice of virtual address space. Physical memory is only consumed when the process actually writes to or reads from that space (a "page fault"). The kernel gambles on the assumption that not every process will use all its allocated memory simultaneously.

The OOM Killer's Logic

When the gamble fails and physical memory runs dry, the kernel invokes the OOM Killer. This mechanism selects a process—usually the one consuming the most memory—and terminates it to save the rest of the system.

For a standard application, this is a minor inconvenience:

1. Process is killed.
2. Process restarts.
3. Connection is re-established.

However, PostgreSQL is not a standard application.

The PostgreSQL Architecture Risk

PostgreSQL utilizes a "postmaster" (the supervisor process) that forks a dedicated backend process for every single client connection. These backends rely on shared memory segments for:

• Shared buffers
• WAL (Write-Ahead Log) buffers
• Lock tables
• General shared state

The critical issue is that shared memory has no transactional guarantees at the OS level. If the OOM killer terminates a backend process while it is mid-write to a shared buffer, that segment can be left in an inconsistent state.

The Danger: A half-written page in the shared buffers results in silent data corruption.

To protect the integrity of the data on disk, the postmaster is designed to be paranoid. If it detects that any child process has been killed unexpectedly, it assumes the shared memory is tainted. To prevent this corruption from reaching the permanent storage, the postmaster terminates all other active backends.

Consequently, only one connection is lost $\rightarrow$ the entire database crashes. If the system had a high write volume, the subsequent crash recovery (replaying WAL files) can lead to significant downtime.

---

Strict Overcommit: Fail Early, Not Catastrophically

To avoid the OOM killer's "nuclear option," we can change how the kernel handles memory requests via the vm.overcommit_memory setting.

Comparison of Overcommit Modes

By using Mode 2, the kernel will refuse a malloc() request if it exceeds the commit limit. Instead of the kernel killing a random process later, the specific application request fails immediately. PostgreSQL can handle a failed memory allocation much more gracefully than it can handle a SIGKILL from the OOM killer.

---

The Investigation: A Three-Character Bug

While strict overcommit is our gold standard, we recently encountered a scenario where we had to disable it. Through a process of elimination, we found a kernel bug.

The Discovery Process

• Discovery: We noticed unexpected allocation failures despite having available headroom.
• Narrowing It Down: We isolated the issue to specific kernel versions.
• Fleet-Wide Analysis: We compared behavior across different hardware and kernel patches.

The culprit turned out to be a bug in the kernel's memory accounting—essentially a "three-character" error in the code—that caused the kernel to miscalculate the available commit limit, leading to premature allocation failures. We were forced to revert to Mode 0 until the kernel was patched.

---

Setting the Commit Limit

When using vm.overcommit_memory = 2, you must define the vm.overcommit_ratio. This determines how much of the physical memory (plus swap) can be committed.

Our Heuristic

We don't simply set it to 100%, as the OS and other background processes need breathing room. We use the following formula:

$\text{Commit Limit} = (\text{Total RAM} \times 0.8) + 2\text{GB}$

Why this specific formula?

1. The 80% Rule: This ensures that the vast majority of RAM is available for PostgreSQL, while leaving $20\%$ for the kernel, SSH, monitoring agents, and other overhead.
2. The +2 GB Buffer: On small instances, $20\%$ of RAM might be too little (e.g., on a 2GB RAM machine, $20\%$ is only 400MB). Adding a flat 2GB ensures that small VMs remain stable.

Implementation Checklist

To implement this on a Linux server, the following steps are taken:

• Set overcommit mode to strict: sysctl -w vm.overcommit_memory=2
• Calculate the ratio based on the formula above.
• Apply the ratio: sysctl -w vm.overcommit_ratio=X
• Persist settings in /etc/sysctl.conf.

# Example for a system with 16GB RAM
# Limit = (16 * 0.8) + 2 = 14.8 GB
# Ratio = (14.8 / 16) * 100 = 92.5%
sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=92

Conclusion

In the world of managed databases, predictability is better than optimism. While Linux's default overcommit settings are designed to maximize memory utilization for general-purpose workloads, they are dangerous for PostgreSQL. By enforcing strict memory limits, we trade the occasional failed allocation for the guarantee that our database will not suffer a catastrophic, fleet-wide crash due to the OOM killer.