Autoresearch, Claude, and Constrained Optimization

By Elliot C Smith | Published July 1, 2026

Introduction

It is common to encounter bold assertions that AI is currently performing the workload of dozens of human employees. However, I maintain a healthy level of skepticism toward any claims of productivity gains that lack empirical evidence.

While this exploration shares some thematic DNA with the "loops" discourse prevalent on X, that overlap is purely incidental. Recently, I embarked on a project inspired by Andrej Karpathy's concept of 'Autoresearch'.

My goal was to identify a challenge that wasn't a standard numerical optimization or machine learning problem, yet still possessed a quantifiable measure of success. I chose this path because it mirrors the reality of most professional software projects: you have a specific metric you need to move (up or down) and a way to track it.

"For example, a common real-world constraint might be: 'We cannot allow the page load time for this specific feature to exceed 500ms.'"

In my experience, the path from ignorance to success is rarely a smooth, gradient-based optimization like you find in ML. Instead, it is an iterative cycle:

1. Implement a change.
2. Test it in a production-like environment.
3. Analyze performance.
4. Decide on the next pivot.

This process is fraught with risk; it is easy to chase a path that leads to a global optimum local optimum without realizing it. I wanted to gain intuition on how to assign large, mostly unsupervised blocks of work to AI agents. While tools like Ralph Loops or the /goal command in Claude Code exist, my approach differed by utilizing a quantifiable success number bounded by pass-fail constraints.

The Chosen Problem: File Compression

To keep the experiment manageable, I selected file compression. The objective and constraints are elegantly simple:

I didn't prioritize execution speed, but the 300-second cap was essential to prevent the unsupervised agent from getting trapped in infinite loops.

Furthermore, compression allows for easy benchmarking against established industry tools. While I didn't expect to invent a world-leading algorithm, comparing a "home-cooked" AI version against professional libraries provides a critical data point: At what point does the value of an in-house, AI-generated solution outweigh the risks of external supply chain attacks?

---

Methodology

Problem Setup

To be clear, this was a proof-of-concept regarding viability, not a benchmark of a specific model's intelligence.

• Model Used: Claude Code (Sonnet 4.6) with default settings.
• Source Code: Available at https://github.com/smitec/agent-compression

I utilized Rust for the implementation. Rust's strict type system acted as an implicit constraint, ensuring the agent didn't arbitrarily change function signatures.

Initial Workflow

• Create project scaffold.
• Implement "stub" functions for compress and decompress (initially just copying bytes).
• Develop basic unit tests to ensure a bit-perfect round trip for strings and files.

At the start, the code "worked," but it provided actual compression zero compression.

Data Acquisition and Entropy

I curated a dataset of public domain samples including:

• Video files
• Audio files
• Text documents
• Randomly generated data of varying sizes

Since many of these were already compressed, I converted them into less-compressed formats. This ensured a mix of high and low entropy. A successful algorithm should significantly shrink low-entropy data while leaving high-entropy data largely untouched (without increasing the size meaningfully).

The Benchmarking Loop

The evaluation script followed this logic:

The script generated two files:

1. debug.csv: Detailed per-file results.
2. results.csv: Key metrics for successful improvements.

The primary metric for success was defined by the following LaTeX formula:

$\text{Compression Metric} = \frac{\sum \text{Total Compressed Bytes}}{\sum \text{Total Original Bytes}}$

I considered using the average percentage of compression across the set, but decided on the total ratio instead (the implications of which I will discuss later).

Visualizing the Progress

The following images illustrate the evolution of the agent's attempts:

Evolution of the algorithm over iterations:

Compression ratios categorized by file type:

The trade-off between execution speed and compression ratio: