Perspectives on the LLM Hype: Why I Remain Bearish Post-Navier-Stokes

---

The Core Thesis

The current valuation of frontier AI labs is predicated on a specific narrative: that they have created, or are about to create, a seamless, fully autonomous replacement for the majority of knowledge workers. However, the reality is that current state-of-the-art models still require $\text{laborious oversight}$ and stringent guardrails, even for trivial assignments.

Meaningful autonomy has been achieved. $\rightarrow$ Meaningful autonomy is still a distant goal.

If you are swayed by "show of force" demonstrations—such as the solving of Navier-Stokes, FreeBSD RCEs, or the HuggingFace incident—consider the actual labor market. Software companies continue to employ "bottom-quartile" engineers who would likely perform worse on current benchmarks than the very models they are paid to supervise.

---

The Generalization Gap

Models do not "reason" in a vacuum; they generalize effectively only within a small neighborhood of the specific tasks present in their training data. Even within these zones, there are significant caveats:

1. The Recipe: Labs have a general method for teaching models tasks with clear performance metrics.
2. The Failure Mode: Even slight perturbations in a known task class often lead to:
   • Complete failure.
   • Reward hacking (finding a loophole to satisfy the metric without solving the problem).

The Specification Paradox

To stop reward hacking, one needs rigorous specification. This creates a cascading set of problems:

• Skill Divergence: Rigorous specification is a distinct expertise, separate from the domain knowledge of the problem itself.
• The Talent Gap: The overlap between "domain experts" and "specification experts" is $\text{extremely small}$.
• Cost Inefficiency: Often, the cost of writing a rigorous spec exceeds the cost of simply implementing the task based on an informal one.

Case Study: Hardware Engineering In CPU development, the ratio of validation/specification engineers to design engineers is often: $\text{Ratio} \approx 3:1 \text{ to } 5:1$

Furthermore, specifications are rarely "one-and-done." They evolve through a dialogue between the spec and the discoveries made during implementation. For high-level specs (like an ISA for CPUs), the verification costs are currently insurmountable, forcing a reliance on lower-level, more expensive, and more fragile specifications.

---

The "Best Case" Fallacy: Navier-Stokes

The success with Navier-Stokes and pure mathematics is not a blueprint for general knowledge work; it is the absolute best-case scenario for agentic AI.

Why Math is Different:

• The Spec is Built-in: The theorem statement is the rigorous specification.
• Community Vetting: These statements have been audited by humans for decades.
• The Verifier: The Lean theorem prover is a battle-tested tool designed specifically to prevent the "reward hacks" (unsoundness) that plague other domains.

Note: Even Lean isn't perfect. Soundness bugs have occasionally allowed LLMs to "launder" incorrect proofs through the kernel.

Requirements for "Math-like" Agentic Success:

• Rigorous, unambiguous specification.
• Decades of human auditing.
• A formal, audited verifier (e.g., a proof kernel).
• Applicability to the vast majority of human knowledge work.

---

The Human Bottleneck

If we cannot rely on formal specifications, we must rely on human review. This introduces two fatal flaws:

1. Scalability: Humans cannot review the massive volume of content LLMs generate.
2. Vulnerability: Experts are susceptible to reward hacking.
   • Example: The xz backdoor.
   • Example: The UMN "hypocrite" commits in the Linux kernel.

If human attention is the final filter, the speed of production is limited by human biology, not GPU clusters. This contradicts the "datacenter full of geniuses" utopia promised by CEOs.

---

Final Verdict: The "Cracked Intern"

For most industries, LLMs will remain like a "cracked intern": incredibly fast and capable when guided by an adult, but far too dangerous to be left unsupervised.

Most firms cannot adopt fully autonomous AI due to structural architectural issues, not a lack of "skill." Only three types of firms can truly utilize autonomous LLMs:

• The Low-Risk Group: Those who can afford failure (e.g., rapid prototyping, intern-level tasks).
• The Narrow-Scope Group: Those with highly repetitive, guarded tasks (e.g., customer service bots, controlled robotics).
• The High-Spec Group: Those already paying the "spec tax" because failure is existential (e.g., drug discovery, chip design).

The Market Split: The first two groups are price-sensitive and don't need "frontier" reasoning. They will likely use open-source models on local hardware. For the third group, the "fuzzy search" capabilities of frontier models are useful, but they still operate within the strict confines of rigorous validation.