The Burden of x86 Emulation

Welcome to our inaugural feature article.

In this piece, we are diving deep into a persistent challenge regarding x86 emulation that impacts every single application we attempt to run. The core of the issue can be summarized by one technical concept with massive implications: Emulating the x86 Total Store Ordering (TSO) memory model.

Attempting to replicate this specific memory model on the weak ordering architecture defined by ARM is a complex, multi-layered problem. In the following sections, we will explore the various hurdles we've encountered and the strategies we've used to solve them—or, in some instances, why a perfect solution remains elusive.

Grab a snack and a warm beverage; this is going to be a detailed journey.

---

The Foundations of ARMv8.0-a: Is Memory Access Really That Simple?

Before we can discuss the "workarounds," we must first define the problem. At its core, a memory model is a set of guidelines that dictate how memory accesses (reads and writes) behave in relation to one another. These rules govern how loads and stores interact, whether the program is running on a single thread or across multiple threads.

While several memory models exist across different hardware, we are primarily concerned with two extremes:

Comparison of Memory Models

A Note on Terminology

It is vital to distinguish between $\text{consistency}$ and $\text{atomicity}$. While they are closely linked, they are the same thing not the same, nor is one always guaranteed by the other.

Understanding x86 TSO

Because TSO is so rigid, a programmer can safely assume that once a memory store occurs, it is coherently visible to every other processor in the system. Furthermore, if a memory load occurs, any stores that logically preceded it will have been completed or made visible.

Essentially: $\text{Store} \rightarrow \text{Load}$ ordering is preserved. This is intuitive; you write data, and it is "there" for others to see.

Understanding ARM's Weak Model

ARM takes a different approach to prioritize power efficiency and speed. By default, standard loads and stores are not strictly coherent across different processors.

• The Efficiency Gain: It is computationally "expensive" for hardware to constantly snoop other caches or invalidate cachelines across cores.
• The Trade-off: When a store executes, that specific cacheline isn't immediately broadcast to other processors. Consequently, a processor loading data might not see the most recent update written by another core.

---

Solving the Consistency Gap

In a multi-threaded environment, this lack of coherence would be disastrous without safeguards.

1. The Old Way (ARMv7 and earlier): Developers used memory barrier instructions to force ordering. However, these were performance killers.
2. The Modern Way (ARMv8.0-a): ARM introduced specialized instructions:
   • load-acquire
   • store-release

In the world of C++, these map directly to std::atomic's memory_order_acquire and memory_order_release.

Pedantic Note: While FEX often uses the terms atomic-load and atomic-store interchangeably with these instructions, technically, they are not "atomic operations" in the strictest sense, but rather tools for memory ordering.

The RCsc Model

ARM refers to this as the "Release Consistency sequentially consistent (RCsc)" model. The logic is as follows:

• load-acquire instructions must be observed sequentially (no reordering).
• store-release instructions must also be observed sequentially while maintaining "barrier-ordered-before" semantics.

This allows ARM to avoid the heavy cost of full memory barriers.

---

The FEX-Emu Implementation Strategy

When we began emulating x86-TSO on ARMv8.0-a, we started with a straightforward mapping strategy:

By converting every x86 load into a load-acquire and every x86 store into a store-release, FEX achieves the same memory semantics as x86. In fact, we are actually being more strict than x86 requires, simply because there was no "middle-ground" instruction that matched the behavior exactly.

The Performance Penalty

As you might expect, this approach is incredibly expensive. ARM CPUs were designed with the assumption that acquire/release instructions would be relatively rare. When these instructions suddenly make up the vast majority of the executed code, performance plummets.

Benchmarking the Impact

To quantify this, we use microbenchmarks. We start with a "best-case" scenario:

• No complex edge cases.
• Standard memory access patterns.
• Baseline hardware performance measurement.

The resulting data (as seen in our internal graphs) shows a stark contrast between the theoretical baseline performance of the hardware and the actual performance when forced into this strict emulation mode. The "Load" and "Store" columns represent the peak efficiency the hardware should achieve, but the TSO emulation prevents the CPU from ever reaching those heights.