Deep Dive into C++ Asymmetric Fences

Author's Note: This post explores the intricacies of the Linux membarrier() system call and the conceptual framework of Asymmetric Fences, inspired by a deep dive into the Folly library.

While exploring the synchronization primitives within the Folly codebase to sharpen my understanding of concurrency, I stumbled upon a term that sounded familiar yet distinct: the Asymmetric Thread Fence.

I was already well-versed in the purpose of std::atomic_thread_fence, but this specific construct piqued my curiosity. What exactly does it do? This question led me down a technical rabbit hole to uncover the implementation details of asymmetric thread fences, particularly how they operate on Linux.

We will begin by examining the C++ proposal and then descend into the low-level implementation.

---

📜 The C++ Proposal: P1202R0

Before diving into the "how," we must understand the "why." According to the P1202R0 proposal, the goal is to optimize specific patterns of concurrent algorithms.

The Core Concept

Many concurrent algorithms can be bifurcated into two distinct execution paths:

1. The Common Path: Executed frequently.
2. The Uncommon Path: Executed rarely.

Both paths typically require fences (or non-relaxed memory orders) to ensure correctness. However, on various hardware platforms, we can optimize the common path by shifting the synchronization burden. By placing an even stronger fence (stronger than memory_order_seq_cst) on the uncommon path, we can use a lighter fence on the common path.

Key Insight: If the performance gain on the common path outweighs the increased overhead of the heavy fence on the uncommon path, the overall system throughput increases.

Real-World Applications in Folly

This technique is already utilized in several high-performance components within Folly:

• folly/synchronization/HazptrDomain.h $\rightarrow$ Hazard Pointers
• folly/synchronization/detail/ThreadCachedReaders.h $\rightarrow$ RCU (Read-Copy-Update)
• folly/executors/ThreadPoolExecutor.cpp $\rightarrow$ Thread Pool Executor

---

🛠️ Analyzing the Mechanics: Dekker’s Example

To illustrate the difference, let's look at a modified version of Dekker's algorithm.

1. The Symmetric Approach

In a standard symmetric implementation, both paths use the same strength of fence:

atomic_int x { 0 }, y { 0 };
int r1, r2;

// Path A
x.store(1, memory_order_relaxed);
atomic_thread_fence(memory_order_seq_cst); 
r1 = y.load(memory_order_relaxed);

// Path B
y.store(1, memory_order_relaxed);
atomic_thread_fence(memory_order_seq_cst); 
r2 = x.load(memory_order_relaxed);

Memory Model Analysis

Under the C++ memory model, several constraints emerge:

• Thread Start: Constraints $(1) \rightarrow (2)$ and $(1) \rightarrow (5)$ are established by the synchronizes-with relationship during thread construction.
• Coherence: Constraints $(4) \rightarrow (5)$ and $(7) \rightarrow (2)$ are coherence-orderings.
• Total Order: Since memory_order_seq_cst fences are used, the operation $(3)$ must precede $(6)$ in the single total order $S$, and vice versa.

2. The Asymmetric Transformation

We can optimize this by using symmetric fences implementing an asymmetric strategy:

// Globals: atomic_int x { 0 }, y { 0 }; int r1, r2;

// --- FAST PATH ---
x.store(1, memory_order_relaxed);
asymmetric_thread_fence_light(); // Inhibits only compiler reordering
r1 = y.load(memory_order_relaxed);

// --- SLOW PATH ---
y.store(1, memory_order_relaxed);
asymmetric_thread_fence_heavy(); // Stronger than seq_cst fence
r2 = x.load(memory_order_relaxed);

Logic Flow Visualization

---

🔍 Implementation Details

The Asymmetric Light Fence (Compiler Barrier)

The asymmetric_thread_fence_light is the simpler of the two. Its primary job is to prevent the compiler from reordering instructions, without necessarily issuing a heavy hardware instruction.

Platform Implementations

Understanding the memory Clobber

In GCC, the asm syntax follows this structure: asm asm-qualifiers ( AssemblerTemplate : OutputOperands [ : InputOperands [ : Clobbers ] ])

The "memory" clobber is the critical part here. It informs the compiler that the assembly code may read or write to memory locations not explicitly listed in the operands. Consequently:

1. Flush: The compiler must flush all modified register values back to memory before the asm block.
2. Reload: The compiler cannot assume that values previously read from memory remain valid; it must reload them after the asm block.

Essentially, this creates a read/write memory barrier at the compiler level.

Summary Checklist

• Understand the trade-off between common and uncommon paths.
• Identify the role of P1202R0 in the C++ memory model.
• Distinguish between compiler barriers (Light) and hardware barriers (Heavy).
• Recognize the use of "memory" clobbers in GCC to prevent reordering.