Resolving a Kubelet Memory Leak in Kubernetes 1.36

Author: Mike Robbins
Date: June 30, 2026

Recently, I encountered a subtle Go lifecycle bug within the Kubernetes kubelet that caused a context leak during every startPodSync operation.

The Warning Signs

The issue surfaced in a modest, single-node test cluster—not a production environment—which I had recently upgraded to v1.36 via DigitalOcean's managed DOKS service.

Interestingly, my preference for cost-saving actually helped me find the bug faster. Because the node only had $2 \text{ GiB}$ of RAM (approximately $2147 \text{ MB}$), the memory pressure became apparent quickly. In a larger node, this leak would have remained hidden for much longer.

Initial Observations

• Alerts indicated that Pods were restarting.
• kubectl top pods suggested the Pods were healthy. (The Pods weren't the problem).
• Application memory usage was stable and well below defined limits.

"Peeling back the Kubernetes facade, I opened up a root shell on the node itself."

By running htop and top, I discovered the culprit: the kubelet process was consuming an ever-increasing amount of memory. While a systemctl restart kubelet provided a temporary fix, the leak persisted.

---

The Investigation Process

To find the root cause, I followed this debugging workflow:

• Identify memory growth via htop.
• Capture a heap profile using Go's pprof.
• Analyze object counts vs. total memory space.
• Trace the leak to specific Go functions.
• Use AI tools to locate the regression in the source code.

Capturing the Heap

Since Kubernetes is written in Go, I utilized the pprof package to dump the heap memory of the running process:

kubectl get --raw "/api/v1/nodes/${NODE}/proxy/debug/pprof/heap?debug=0" "kubelet_pprof_heap.pb.gz"

I then analyzed the data using go tool pprof -top.

Analyzing the Data

1. By Object Count

I first looked at inuse_objects to see what was being created in massive quantities.

2. By Total Memory Space

Next, I checked inuse_space to see the actual RAM impact.

go tool pprof -top -sample_index=inuse_space kubelet_pprof_heap.pb.gz

The results were staggering: nearly a million contexts were occupying the bulk of the memory. The trace pointed directly toward SyncPod and WaitForAttachAndMount.

---

Finding the Regression

The debugging path can be visualized as follows:

Despite being unfamiliar with the internal implementation of the kubelet, the data was clear. I initially wondered if exec-based liveness or readiness probes were the cause, but I used Codex (an AI coding tool) to navigate the codebase.

By asking the right questions about the WaitForAttachAndMount logic, Codex quickly steered me toward the specific code change that introduced the regression.

Summary of the Bug

The leak was a classic Go lifecycle error: a context was being created but not properly cancelled, leading to a buildup of cancelCtx objects every time the kubelet performed a pod sync.