The Paradox of Automating AI Away

"OpenAI researchers are effectively automating themselves away by improving their AI." — Andrej Karpathy

Currently, I am utilizing Anthropic's Fable to develop Beagle SCM. It is an undeniably brilliant model, capable of identifying minute "nits" within massive codebases, generating tickets, and implementing fixes.

However, brilliance does not equal reliability. Just yesterday, the model managed to commit the build/ directory into a project—once twice.

The Core Conflict: Brilliance vs. Determinism

Because of the inherent nature of Large Language Models (LLMs), these types of errors are not "bugs" that will simply vanish as the models evolve. They are systemic. Consider the contrast between an LLM and a tool like Ragel (a parser generator):

Despite my explicit instructions—written in ALL CAPS—to $\text{NOT PARSE ANYTHING MANUALLY, EVER}$, the LLM attempts it anyway. Consequently, I must periodically command it to scan the codebase specifically to find and delete any instances of manual parsing.

The "Sandwich" Architecture

To manage an LLM that is simultaneously expensive, slow, and clumsy—yet brilliant—you must wrap its non-determinism in a rigid framework.

$\text{System} = \text{Deterministic Tools} + \text{LLM} + \text{Formal Workflow}$

The goal is to optimize the experience:

• Increase execution speed.
• Ensure the model sees only relevant data at the precise moment.
• Reduce clumsiness through constraints.
• Implement self-correction loops.

Malleable Automation

The strategy becomes truly powerful when the tools and processes are malleable. We can create a feedback loop where the AI optimizes its own environment:

1. Pattern Recognition: If Claude performs a specific sequence of actions frequently $\rightarrow$ Automate it.
2. Error Correction: If the model fails a specific task repeatedly $\rightarrow$ Automate the verification step.

In essence, we allow the LLM to automate itself away, replacing its own inconsistent reasoning with simple, reliable, deterministic tools.

Implementation in Beagle SCM

In Beagle SCM, LLMs are empowered to script their own routines using JavaScript. The architecture is split:

• The Core: Heavy lifting is written in C (rarely modified).
• The Layers: The tooling layer (bottom of the sandwich) and the workflow layer (top) are written in JavaScript, loading code from the filesystem in a node_modules-style approach.

For example, an LLM might generate a routine like this:

// Example of an LLM-generated routine for Beagle SCM
async function verifyNoBuildDir(commit) {
    const files = await beagle.getFiles(commit);
    if (files.some(f => f.startsWith('build/'))) {
        throw new Error("Deterministic Check Failed: build/ directory detected!");
    }
}

Imagine the potential of git hooks that can tokenize source files across nearly any language, analyze commit and file histories, and cross-reference links—essentially granting the AI access to every internal data point git can reach.