A Contract-Grade Verifier for LLM-Generated GPU Kernels

Authors: Rishi Shah & Rishav Shrestha
arXiv Identifier: 2608.12700 | Submitted: August 13, 2026
Categories: Machine Learning (cs.LG), Hardware Architecture (cs.AR), Distributed Computing (cs.DC)

---

🚨 The Problem: The Illusion of Correctness

Current systems that utilize Large Language Models (LLMs) to synthesize GPU kernels often claim impressive success rates. However, these metrics are based on rigorous verification dangerously loose testing.

Typically, a kernel is deemed "correct" if it passes a single test: running a few random inputs at one specific shape and checking if the output is "close enough" to a reference. This approach is flawed because it ignores several critical failure modes:

• Numerical Edge Cases: A kernel might return a standard number when the mathematically correct answer should be $\text{NaN}$ or $\infty$.
• Non-Determinism: Results may fluctuate between different runs.
• Shape Fragility: The kernel may crash or fail when the input dimensions change.
• Precision Drift: Accumulation in fp16 may diverge from an fp32 reference in ways that a simple threshold ignores.

---

🛠️ The Solution: A Contract-Grade Verifier

To address these gaps, the authors developed a high-rigor instrument designed to audit kernels through twelve adversarial gates. Each gate represents a specific property that a robust kernel must satisfy.

"Several of these gates are tolerance-free, ensuring that a failure cannot be dismissed by simply adjusting a threshold."

Verification Logic Flow

---

📊 External Audit Results

The verifier was used to audit 2,638 kernels that had already been flagged as "correct" by a public system's internal harness. The results reveal a significant gap in reliability:

Verifier Validation

To ensure the verifier itself was accurate, the authors employed four independent defense mechanisms:

• Positive Control: Achieved a $7/7$ success rate on known-good kernels.
• Calibration: Performed a comprehensive threshold-calibration sweep.
• Cross-Reference: Found $98.5\%$ agreement with the reference benchmark's own internal correctness logic.
• Manual Review: Conducted a stratified hand-audit of the results.

---

🚀 Internal Application: Blackwell Native Backward

The authors applied their verifier to a kernel they developed: the first native Blackwell tcgen05 training backward kernel for the Gated-Linear-Recurrence (GDN) family.

Notably, this includes the reverse-state stage, a component that most current systems still handle via a slower fallback mechanism.

Technical Implementation Details

The kernel was validated using a double-precision oracle to ensure mathematical integrity:

// Target Architecture: NVIDIA Blackwell
// Feature: tcgen05 Training Backward
// Family: Gated-Linear-Recurrence (GDN)
void gdn_backward_kernel(float* grad_output, float* state, ...) {
    // Implementation of reverse-state stage
    // Verified against double-precision oracle
}

Following verification, this kernel was successfully used to train five different members of the GDN family.

---

🏁 Conclusion

The perceived progress in LLM-driven kernel generation is largely an artifact of weak testing. The "correctness signal" is far thinner than reported. By implementing tolerance-free contracts, the industry can close the gap between appearing to work and actually being correct.

Reference: doi: 10.5281/zenodo.21563213