The End of Sluggish Software

Recently, a viral tweet suggested that critics of LLM-generated "bloatware" will eventually be proven wrong once we use AI to rewrite everything in highly optimized assembly. While we aren't quite at the stage of writing every line in assembly, we are reaching a tipping point. As Nolan Lawson suggested regarding software testing—that you can now essentially choose your bug count—we are seeing a similar trend with performance.

The Democratization of Optimization

Historically, squeezing every drop of performance out of a system required a rare, specialized skill set. This made high-level optimization prohibitively expensive a luxury reserved for the most lucrative or massive-scale projects. Now, that barrier has collapsed.

"Completely agree... Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome." — Marc Brooker

This shift allows us to revive "weird" techniques from the demoscene—where developers fought for every byte and cycle on specific hardware. One classic example involved using actual executable code as textures to maximize cache locality.

The "Hard Part" of Coding

There is a common meme that "writing the code was never the hard part." While that holds true in some fields, in others, the implementation was the primary bottleneck.

• The JIT Barrier: Just-In-Time (JIT) compilers are incredibly powerful for speed, yet they remain rare. This suggests that, historically, the effort to implement one outweighed the perceived benefit.
• The AI Catalyst: LLMs have lowered the entry barrier, making the creation of a JIT compiler accessible to more developers.
• Ambitious Architecture: Databases were once the most difficult software to build; AI now allows us to be far more ambitious in our architectural goals.

---

Case Study: The FRE Regex Engine

To test this theory, let's look at FRE, a regex engine developed using an agentic loop.

The Development Process

The engine was created by an AI agent that spent a month iteratively improving performance using the rebar benchmark suite.

Initially, FRE was heavily overfit to the rebar suite. Once a holdout benchmark was introduced, the agent generalized the optimizations, resulting in acceptable performance across the board.

The AOT Experiment

We noticed that the native Ahead-of-Time (AOT) compiled version of FRE excelled at longer searches. This led to a hypothesis: why not run the native compiler in a background thread while ripgrep uses its standard matcher, then switch to the native code once compilation finishes?

The Trade-off:

• Short Queries: $\downarrow$ Performance (due to thread overhead).
• Long Queries: $\uparrow$ Performance (due to native execution).

Since a search taking minutes is a much bigger problem than one taking seconds, this is a winning trade.

Performance Results

The implementation was trivial; a few prompts to an agent handled the "code surgery" and benchmarking against real codex history.

While a $7\%$ increase isn't revolutionary, the fact that it was achieved in minutes of human effort is.

---

Beyond Regex: The Indexing Problem

Admittedly, building a native compiler for regex is a bit silly if the real solution is simply to index the text.

I previously worked on BitFunnel (the Bing search index), which won the Best Paper Award at SIGIR for its fast text ingestion. Most current local indexers only target code directories, but the real performance killer is when ripgrep hits massive temporary directories or the entire disk.

If I were operating at an AI lab with SOTA models on Cerebras chips (which offer massive tokens-per-second throughput), I would likely build a custom, full-disk indexer.

Evolution of Complexity

The difference in scale is stark:

• Open Source BitFunnel: Contains a bytecode interpreter and one JIT.
• Bing Version: Utilizes multiple JIT compilers.

In the past, this level of optimization was a massive corporate undertaking. Now, for many of these projects, the following is actually true:

• Design custom architecture
• Implement multiple JITs
• Optimize for specific hardware
• Finish by Sunday night

Conclusion: The cost of performance has plummeted. We no longer have to accept "slow" as the default state of software.