Task Failed Successfully: Saturating NIC and Disk Bandwidth

A journey from wondering "how" to wondering "why."

By Yao Meng | June 21, 2026 | 21 min read #System #TLB #RDMA #io_uring

---

0. Task Failed Successfully

The era of Artificial Intelligence has arrived with a velocity that caught most of us off guard. Specifically, agentic coding has fundamentally overhauled my daily professional workflow.

To be perfectly candid, I haven't manually authored a single line of production code in quite some time. Despite this, the software I oversee continues to execute across clusters comprising hundreds of HPC servers, operating at peak efficiency.

Of course, abstaining from writing code (or even skipping a granular review) doesn't mean I'm simply randomly poking around like a "monkey typing" at a keyboard. The engineering rigor remains:

• Analyze complex requirements
• Refine architectural designs with the AI agent
• Develop prototype demos
• Execute mock experiments
• Study results from small-scale deployments
• Iterate on discovered bottlenecks
• Maintain a rigorous, comprehensive testing pipeline

The "Monkey Typing" Paradox: With AI and agentic coding, the velocity of development has accelerated. Often, the code is generated faster than a human can fully comprehend its internal logic.

Recently, I prompted my agent to optimize a system's performance. The AI rapidly pivoted the system from roughly 50% throughput to total saturation. This article isn't about why the AI "failed successfully," but rather a detailed walkthrough of the analysis and debugging process used to achieve this optimization.

---

Optimizing a Demo: 1 NIC and 8 Disks

To isolate the performance variables, let's abstract the system into a simplified model. We want to focus on the optimization rather than business logic.

The Architecture

The system consists of a single thread performing the following sequence:

1. Issue 1 MiB random direct I/O reads across 8 NVMe drives.
2. Transfer that data to a remote host using RDMA WRITE.

Hardware Specifications:

• Storage: 8 NVMe drives (Max throughput: $7 \text{ GiB/s}$ per drive).
• Network: 1 NIC providing $400 \text{ Gb/s}$ of bandwidth.
• IOMMU: Configured in passthrough mode (no translation for I/O devices).

The Implementation

The AI agent constructed a basic event loop: Client Request $\rightarrow$ Server polls RDMA CQ $\rightarrow$ Submit reads via io_uring $\rightarrow$ Poll CQEs $\rightarrow$ RDMA WRITE back to client.

Baseline Performance

This setup is designed to eliminate interference. Every component—except the NIC we are testing—has significant headroom. The theoretical maximum for the NIC is $\approx 46.6 \text{ GiB/s}$ (calculated as $400 \text{ Gb/s} \div 8 \div 1.07$ approx).

Performance Metrics Table:

Surprisingly, the system hits a ceiling at an I/O depth of 16, reaching only about half of the NIC's potential bandwidth.

---

The Deep Dive: Profiling the Bottleneck

To understand why we stalled at $\approx 22.7 \text{ GiB/s}$, I used perf to profile the system at an I/O depth of 16.

The flamegraph revealed that 81.62% of the total CPU cost was spent within io_submit_sqes.

The Cost of Direct I/O

Since the demo utilizes Direct I/O, the kernel must construct DMA metadata from the user-space buffer for the block device. The most expensive operations in this path are:

1. __bio_iov_iter_get_pages: Converts the iov into bio pages.
2. pin_user_pages_fast: This is the critical step. It translates a virtual address range into an array of struct page pointers and pins them to prevent the VM from reclaiming, migrating, or swapping them during DMA.
3. folio_* updates: Because the NVMe device DMA-writes directly into user-space pages, those pages must be marked dirty. The kernel updates the folio (a unified abstraction for physically contiguous pages), managing reference counts, locking, and mapping.

The Mathematical Burden

The overhead is staggering when scaled:

• Per Read: A $1 \text{ MiB}$ read using $4 \text{ KiB}$ pages involves $\frac{1024 \text{ KiB}}{4 \text{ KiB}} = 256$ pages.
• Per Second: At $20,000$ to $50,000$ reads/sec, the system processes: $\text{Pages/sec} = (2 \times 10^4 \text{ to } 5 \times 10^4) \times 256 \approx 5 \text{ to } 13 \text{ million pages/sec}$

If physical memory is fragmented, this triggers massive amounts of folio metadata updates and atomic refcount increments, leading to expensive cross-core cache-line ownership transfers.

---

The Solution: Registered Buffers

To eliminate the per-I/O cost of processing user-space buffers, we can use io_uring_register_buffers(3). This function allows us to register buffers in advance, shifting the metadata preparation out of the critical path.

What io_uring_register_buffers(3) does upfront:

• Validation: Checks address ranges, alignment, and length limits for the iovecs.
• Pre-Pinning: Performs the GUP (Get User Pages) process and pins the pages for the entire duration of the registration.
• Metadata Caching: Constructs and stores io_mapped_ubuf structures on the kernel side.

By doing this, we bypass the expensive page-table walks and folio updates during every single I/O submission.