Big Fish: Analyzing a 37-Year-Old OpenBSD Kernel Flaw

Author: Andrew Griffiths | Date: October 16, 2026 | Tags: #openbsd #madbugs

While Calif is well-known for publishing offensive security research, we also dedicate significant effort to defensive engineering. This includes helping open-source communities generate patches, improving cryptographic implementations, and conducting internal security audits for our clientele.

Earlier this year, we collaborated with the OpenBSD team to report several vulnerabilities and provide remediation strategies.

The Legacy Vulnerability

One specific flaw was particularly striking: it originated roughly 37 years ago in the ancestral BSD source code. The issue centered on a shared memory attachment counter that was defined as a signed 16-bit short.

The Mechanics of the Overflow

Because the counter was a signed 16-bit integer, its maximum positive value was $32,767$. By repeatedly forking processes that maintained shared memory attachment records, an attacker could push this counter beyond its limit, causing it to wrap around to a negative value.

This integer overflow created a window for a race condition, leading to a use-after-free (UAF) scenario. When paired with a separate out-of-bounds (OOB) write within the kernel, this chain allowed for local privilege escalation (LPE) to root.

These vulnerabilities were subsequently patched in errata 026 and errata 018.

---

The Exploit Chain

We successfully achieved local root access on OpenBSD—even with KARL, SMAP, and SMEP active—without needing an initial information leak. The attack vector followed this precise sequence:

Execution Steps

• Counter Wrap: Map 127 SysV shared memory segments $\rightarrow$ call munmap() $\rightarrow$ fork 259 processes to force the attach count to wrap to $-32,643$.
• Heap Grooming: Use sysctl(KERN_MALLOCSTATS) to monitor kb_total and kb_totalfree, ensuring precise control over page allocations.
• The Race: Utilize IPC_RMID in one process while simultaneously attaching a "size-confused" shared memory segment via a helper process.
• Socket Alignment: Align IP socket allocations and set IP_MULTICAST_TTL to 2. This forces the allocation of a persistent 16-byte struct ip_moptions.
• OOB Write: Trigger a conditional $-1$ OOB write to overwrite the membership fields.
• PTE Manipulation: Call close() on the socket. This triggers atomic decrements that modify the PTE self-referential mapping twice, flipping it from user-accessible to supervisor-accessible.
• Code Execution: Overwrite a predictable kernel pointer to redirect execution flow.
• Finalization: Restore the corrupted pointer and clean up shared memory accounting to spawn a root shell.

Technical Summary

The exploit essentially functioned as a write-what-where primitive:

---

Analysis of Mitigations

OpenBSD's KARL (Kernel Address Randomized Link) randomizes function and data addresses by relinking the binary. However, the memory allocators themselves remain deterministic.

The Weakness

Both the best-fit and boot-time allocators lack base address randomization. This means early boot allocations are predictable across reboots on identical hardware. The exploit targeted the lo0 and enc0 ioctl handlers—using one for control flow and the other for restoration.

Proposed Defensive Enhancements

To prevent similar attacks, we suggest the following improvements:

1. Safe-Unlinking

Implement checks for doubly linked list invariants before performing unlink writes. If an invariant is broken, the system should trigger an abort() (user mode) or panic() (kernel mode).

• Standard Unlinking $\rightarrow$ Safe-Unlinking (Already present in FreeBSD/NetBSD sys/queue.h).

2. PTE Map Randomization

Randomize the allocation address of the self-referential PTE map. This can be achieved by using a random byte from .openbsd.randomdata to select a PML4 slot: $\text{Slot} = 320 + (\text{random\_byte} \pmod{127})$

This makes PTE_BASE a runtime expression, resulting in bases ranging from: 0xffffa00000000000 $\rightarrow$ 0xffffdf8000000000 (at 512-GiB intervals).

Alternative: Mark the PTE as read-only and toggle CR0.WP only during necessary modifications.

3. Non-Deterministic Allocation

The kernel allocator could be hardened via:

• Boot-time randomization: Randomizing the base address of memory allocations.
• Runtime randomization: Randomizing the order of allocations (though this may introduce performance overhead).

4. Compiler-Level Protections

Utilizing the Array Bounds Sanitizer (a component of UBSan) can eliminate OOB accesses to fixed-size arrays.

// Example of what UBSan prevents
void vulnerable_func(int index) {
    int arr[10];
    arr[index] = 0; // UBSan catches if index < 0 or index >= 10
}

Calif has already recommended this to FreeBSD, having privately reported 21 distinct issues that the array bounds sanitizer would have successfully intercepted.