Quantifying "Slop": The Challenge of Code Quality in the AI Era

From: Sebastian (sebastian@earendil.com)
To: You
Date: Thursday, September 10, 2026
Subject: If coding is solved, what now?: Measuring the sloppiness of code

While Large Language Models (LLMs) have nearly mastered the art of generating syntactically and functionally correct code, the journey doesn't end with a passing test suite.

The real issue is that formal correctness $\neq$ good design. AI often introduces redundant abstractions, duplicates logic, or simply makes poor architectural choices. This phenomenon is what we call "slop."

The Crisis of "Vibe-Coding"

Many developers practicing "vibe-coding" (relying on AI to iterate rapidly) have noticed a disturbing trend: as new features are added, the total Lines of Code (LOC) explode exponentially.

This leads to a critical erosion of human agency. When a project grows by millions of lines of code per month, it becomes mathematically impossible for a human maintainer to keep the system in their head.

Some believe that autonomous agents will simply manage this complexity for us. The reality is that agents are just as susceptible to slop as the models that generate the code.

---

The Search for a Quantitative Metric

Coming from a physics background, I prefer an experimental, data-driven approach. Upon joining Earendil, I attempted to find a standardized way to measure "sloppiness." My initial research process looked like this:

• Review academic literature on software metrics.
• Analyze industry standards for AI-generated code.
• Audit current "AI Coding Agent" marketing claims.
• Find a universally accepted "slop" metric (Still pending).

I found the current industry state to be disappointingly "vibes-based." I was inundated with buzzwords:

• "End-to-end coding agents"
• "AI that doesn't just suggest—it ships"
• "Human-level evaluation at zero cost"

The reason for this superficiality is that verifiability is easy, but taste is hard. It is simple to run a hidden test to see if code works (a clear reward signal), but measuring if code is sloppy requires human intuition.

The Evaluation Spectrum

---

Deep Dive: The SlopCodeBench Metrics

To move beyond "vibes," I looked into SlopCodeBench, which proposes two primary metrics to identify LLM-generated slop.

1. Verbosity

This measures the density of redundant or unnecessary code. $\mathrm{Verbosity}=\frac{|\text{AST-Grep flagged lines}\cup\text{clone lines}|}{\mathrm{LOC}}$

2. Erosion

Erosion measures the concentration of "mass" within a few overly complex functions. First, we define the mass of a function $f$: $\mathrm{mass}(f)=CC(f)\sqrt{\mathrm{SLOC}(f)}$ (Where $CC$ is Cyclomatic Complexity and $SLOC$ is Source Lines of Code).

Then, Erosion is the ratio of mass in "complex" functions (where $CC > 10$) relative to the total mass: $\mathrm{Erosion}=\frac{\sum_{f:\,CC(f) > 10}\mathrm{mass}(f)}{\sum_f\mathrm{mass}(f)}$

The Results: Humans vs. Agents

The data reveals a stark contrast between established human repositories and agent-generated code:

Conclusion: AI-generated code is roughly twice as verbose and eroded as human-written code. My own "vibe-coded" projects mirrored this, sometimes hitting verbosity levels of $0.4$ and erosion of $0.75$.

---

Why Agents Can't Fix Themselves

To understand why agents struggle with slop, we can visualize the difference between standard benchmarks and SlopCodeBench:

Unlike traditional benchmarks that provide all instructions upfront, SlopCodeBench uses iterative rounds where the model's context is wiped between checkpoints. This mimics real-world development, proving that agents cannot effectively manage the long-term architectural decay they create.

# Example of "Slop" logic:
# Instead of a simple mapping, an agent might create 
# three unnecessary classes and a factory pattern 
# for a task that requires a single dictionary.
def solve_problem(data):
    # Human: return {k: v for k, v in data.items() if v > 0}
    # Agent: (creates 50 lines of boilerplate abstractions)
    pass