Tracking Down a Go Runtime Bug on 32-Bit Embedded Systems

Published on 22.08.2026 by the DE embedded team

While our primary focus usually involves security engineering and the lower layers of the Linux software stack, we frequently find ourselves debugging applications operating at a much higher level. Depending on the case, the root cause might be buried in the Linux kernel or located elsewhere entirely.

In this article, we detail our journey of discovering and resolving a bug within the Go runtime.

The Problem

A client reported that a Go-based application was experiencing intermittent crashes on their embedded Linux hardware. Every crash shared the same fatal signature:

runtime: netpoll: eventfd ready for 5 fatal error: runtime: netpoll: eventfd ready for something unexpected ...stack trace...

Initially, we suspected a bug within the application logic. However, a closer look suggested that an internal assumption within Go's netpoll mechanism was being violated.

The crash originates from src/runtime/netpoll_epoll.go:

if ev.Events & ^EPOLLIN != 0 { 
    println("runtime: netpoll: eventfd ready for", ev.Events) 
    throw("runtime: netpoll: eventfd ready for something unexpected") 
}

In this specific execution path, the netpoll logic expects EPOLLIN to be the sole active event. If any other event is reported, the runtime panics. This raised a critical question: Why would epoll return events other than EPOLLIN when the code specifically requested only EPOLLIN?

Initial Research

We started by searching for the error message, which led us to a Go project issue tracker. Other users had reported the same crash, noting that it typically occurred in applications that had been running for extended periods. Interestingly, the Go maintainers had already dismissed one previous attempt to fix the bug.

From the community discussion, we gathered a vital clue: the error exclusively appeared on 32-bit ARM and i386 Linux systems, regardless of the kernel version.

The Investigation

Since the issue was widespread but unsolved, we decided to tackle it. Our debugging process followed these steps:

• Hypothesize that epoll behaves differently on 32-bit architectures.
• Review netpoll usage in src/runtime/netpoll_epoll.go.
• Analyze integer conversions and 32-bit pitfalls (assisted by an LLM).
• Inspect the memory layout of epoll_event data.

We quickly discarded the idea that the Linux kernel's epoll implementation was flawed, as it is core, generic code. Instead, we focused on how Go interacts with it.

Analyzing epoll_event

Go's netpoll utilizes the data field of the epoll_event structure. Linux allows the kernel to store an 8-byte "cookie" which is returned to user space when an event fires, allowing applications to associate metadata with the event.

The Go runtime uses this ev.Data field to distinguish between an event fd and a socket fd. It does this by comparing ev.Data to the memory address of its internal netpollEventFd object:

if * (**uintptr)(unsafe.Pointer(&ev.Data)) == netpollEventFd { ... }

We discovered that ev.Data can hold one of two things:

1. A raw pointer to netpollEventFd.
2. A tagged pointer to a pollDesc object (per-socket), which includes a sequence counter called fdseq to handle recycled objects.

Mixing raw pointers and tagged pointers in a single field seemed suspicious.

The "Aha!" Moment

The bug lies in how Go handles tagged pointers on 32-bit platforms. To fit both the address and the tag into an 8-byte word, Go uses the following layout:

The Collision Logic

When the runtime checks if an event is an eventfd, it casts ev.Data to a uintptr. On a 32-bit system, a uintptr is only 4 bytes.

Therefore, the comparison logic effectively performs this operation: $\text{Comparison} = \text{ev.Data}[0:4] \equiv \text{netpollEventFd Address}[0:4]$

Because the fdseq (the tag) is stored in the lower 4 bytes of a tagged pointer, the runtime is actually comparing the socket sequence number against the eventfd address.

As soon as the fdseq increments to a value that matches the memory address of netpollEventFd, the runtime mistakenly identifies a socket event as an eventfd event. This breaks the internal assumption that the event must be EPOLLIN, triggering the crash.

Why is this rare?

This bug is constrained by several factors:

• Endianness: It only occurs on little-endian 32-bit systems. On big-endian systems, the uintptr cast would read the upper 4 bytes (the actual address).
• Architecture: On 64-bit systems, the comparison spans the full 8 bytes, preventing aliasing.
• Time: The fdseq must grow large enough to hit the address of netpollEventFd.

On a standard 32-bit ARM Linux system, netpollEventFd is typically located in a read-only section within the first 3 MiB of the address space. We verified this using pmap:

$ pmap `pidof netpoll_test`
204: /opt/netpoll_test 00010000 2696K r-x-- netpoll_test
002c0000 2192K r---- netpoll_test
004f0000 180K rw--- netpoll_test

Consequently, the application must create and recycle approximately 3 million pollDesc objects before the fdseq reaches the collision point and crashes the system.

A Test Case

To prove this, we developed a standalone... (article ends)