The Case of the Vanishing Atomic Update on Loongson CPUs

TL;DR: In February 2026, while packaging the normaliz library for Debian on a LoongArch server, developer Wang Miao encountered a mysterious infinite loop during software tests. The culprit wasn't the code, but a hardware bug: the CPU's atomic addition instruction occasionally failed to be atomic. After an AI-assisted reproduction effort, Loongson developed a firmware fix with negligible performance impact, slated for release by October 1st.

---

🕒 Timeline of Events

• February 2026: Initial discovery of the normaliz timeout during Debian packaging.
• Spring 2026: Attempted manual reduction to a minimal example (Failed/Shelved).
• August 2026: Investigation resumed using AI-driven reproduction.
• Late August: Root cause identified as a CPU erratum.
• September 2026: Loongson provides test firmware; fix verified.
• October 1, 2026: Expected public release of the firmware update.

---

🛠️ Origins: The loong13 Project

The issue surfaced within loong13, a community-led effort to port the Debian 13 stable distribution to the LoongArch architecture. Wang Miao, a maintainer of this project, noticed that the built-in tests for normaliz (a mathematical software package) were consistently timing out.

Because the build process was hanging, the package was initially skipped. However, since normaliz is a dependency for several other critical packages, a permanent solution was required.

Initial Hypotheses

Given that LoongArch utilizes a weak memory model, the team first suspected:

1. Hidden race conditions.
2. Memory-ordering violations.

These are common issues when porting software to weak-ordering architectures. However, the reality turned out to be far more surprising.

---

🔍 The First Round of Investigation

The team dove into the normaliz source code. The problematic logic can be distilled into the following structure:

func ( std :: list std :: vector int LatticePoints ) {
    size_t nr_to_match = LatticePoints.size(); // Total target
    size_t nr_points_matched = 0;             // Global accumulator

    while ( true ) {
        size_t nr_points_done_in_this_round = 0; 

        #pragma omp parallel {
            auto P = LatticePoints.begin();
            size_t ppos = 0;

            #pragma omp for
            for ( ppp = 0; ppp < nr_to_match ) {
                if ( skip_remaining ) continue;
                
                // Logic to advance P to ppp...

                if (( * P )[ 0 ] == 0 ) continue; // Already processed

                #pragma omp atomic
                nr_points_matched++; 

                #pragma omp atomic
                nr_points_done_in_this_round++;

                ( * P )[ 0 ] = 0; // Mark as processed
            }
        }
        // The loop should exit when all points are matched
        if ( nr_points_matched == nr_to_match ) break; 
    }
}

Logic Flow Analysis

The program processes a list of LatticePoints in parallel. Because some points are skipped in certain passes, the while(true) loop ensures the program keeps iterating until every single point is processed.

The termination condition is defined by the LaTeX expression: $\text{nr\_points\_matched} = \text{nr\_to\_match}$

The Symptom

Using gdb, the team observed a paradox:

• Observation A: Every element in the LatticePoints list had been marked as processed ((*P)[0] == 0).
• Observation B: The counter nr_points_matched had stopped increasing.
• Observation C: $\text{nr\_points\_matched} < \text{nr\_to\_match}$.

This discrepancy created an infinite loop.

Comparison of Counters

The team noticed that two counters, which were incremented atomically side-by-side, were drifting apart:

---

🧬 Deep Dive: From Software to Silicon

The team systematically ruled out software failures:

1. Memory Ordering: The code doesn't use atomic variables to guard other data; it only reads/writes the atomics themselves. Thus, it is logically sound.
2. OpenMP Implementation: They checked the disassembly to see how #pragma omp atomic was handled.

The compiler was using the following LoongArch64 instruction: amadd.d (Atomic Add Doubleword)

To verify if the issue was specific to the OpenMP implementation, they introduced std::atomic counters as a control group.

The Discovery

After failing to create a manual reproduction, the team used AI to iteratively generate a minimal reproduction case. Within two days, they had a stable reproducer that proved the amadd.d instruction was occasionally failing to perform the addition atomically.

This was a CPU erratum.

---

🏁 Resolution

Once the evidence was presented to Loongson, the response was rapid:

• Turnaround: Within two weeks, Loongson identified the hardware flaw.
• Solution: A firmware update was developed.
• Impact: The fix results in almost zero performance degradation.

The team confirmed the fix using the provided test firmware. Users of LoongArch servers can expect the official update to be available before the National Day (October 1st) holiday.