Optimizing the Xen dom0 I/O Path for NUMA Awareness

---

Introduction: Bridging the Gap

In the previous installment of this series, we analyzed the inherent costs of Xen's paravirtual I/O when running on Non-Uniform Memory Access (NUMA) hardware. The primary culprits were:

1. Foreign-mapped pages lacking NUMA metadata.
2. Kernel threads (kthreads) being placed arbitrarily by the scheduler.
3. A fundamental "information gap" where the dom0 kernel remained oblivious to the host's actual physical topology.

"The fix requires two distinct approaches: a structural overhaul and a component-specific implementation."

This article fulfills that promise, while also detailing two unexpected hurdles encountered during development.

---

The First Hurdle: Correcting dom0 Memory Placement

Before implementing NUMA awareness, Edera discovered a critical flaw in how dom0 handles memory. In standard Xen, the boot process trims the host's BIOS memory map (the E820) to match the memory allocated to dom0 via dom0_mem.

The Problem: Linear Allocation

Stock Xen processes the E820 map in physical address order. Because RAM regions are typically grouped by NUMA node in address order, dom0 simply grabs memory from the first available nodes until its quota is met.

Example Scenario:

• Host: 128 GiB RAM, 8 NUMA nodes.
• Config: dom0_mem=35%.
• Result: Nodes 0, 1, and 2 are completely filled; Nodes 3 through 7 contain zero no dom0 pages.

If Edera had proceeded without fixing this, they would have been lying to the kernel synthesizing a topology that the kernel could not actually utilize, as there would be no physical memory backing the remote nodes.

The Solution: Proportional Distribution

Edera replaced the linear "grab" with a proportional allocation strategy. This is implemented in two distinct phases:

1. Pass One: Calculate the total available host RAM.
2. Pass Two: Assign each memory region a share of the dom0_mem budget based on the node's proportion of total RAM.

To ensure the total memory equals exactly dom0_mem, they utilized a Bresenham-style algorithm to handle rounding remainders.

$\text{Allocation per Node} \approx \text{Total Host RAM} \times \left( \frac{\text{dom0\_mem}}{\text{Total Host RAM}} \right)$

// Conceptual logic for proportional trim
for (region in e820_map) {
    uint64_t share = (region.size * dom0_mem_total) / host_ram_total;
    apply_trim(region, share + remainder);
    remainder = calculate_new_remainder();
}

---

Implementing dom0 NUMA Awareness

To achieve true NUMA awareness, a Xen domain requires three specific pieces of topological data:

The Gap in Upstream Xen

While upstream Xen can synthesize SRAT and SLIT for PVH and HVM domUs (via XEN_DOMCTL_setvnumainfo), it fails in two areas:

• It does not synthesize x2APIC IDs in CPUID (an extension Edera added previously).
• It provides none of these three components for dom0, leaving dom0 to see a single, flat NUMA node.

The Edera Implementation

Edera extended the synthesis of all three components to dom0, provided the following conditions are met:

• Dom0 must be in PVH mode.
• dom0_vcpus_pin=1 must be set (1:1 hard-pinning of vCPUs to pCPUs).
• The number of dom0 vCPUs must equal the number of host pCPUs.
• The host must possess more than one physical NUMA node.

Because of the 1:1 pinning, the logic is simplified: the vNUMA node for vCPU N is identical to the host's NUMA node for pCPU N.

The result of this implementation:

• SRAT: Describes one proximity domain per host NUMA node.
• SLIT: Uses actual inter-node distances rather than the generic Linux defaults (10 local / 20 remote).
• x2APIC: Matches the synthesized topology using Edera's existing domU machinery.

---

Final Discovery: The Toolstack Bug

After the primary work was complete, Edera uncovered a secondary issue during memory-bandwidth benchmarking. They found a memory-placement bug within their own toolstack. This bug had existed silently since the introduction of multi-vnode vNUMA but only became apparent under the extreme stress of bandwidth testing.