Recreating Minecraft Is Not a Benchmark

By Kuber Mehta | September 06, 2026

The recent launch of GPT Astra followed a predictable pattern. Within an hour, my social media feeds were saturated with the same handful of "impressive" feats:

• Generating a Minecraft clone via a single prompt.
• Self-portraits rendered in MS Paint.
• An SVG of a pelican riding a bicycle.
• A physics-accurate simulation of a ball bouncing in a rotating cube.
• An SVG-based game controller.

At first glance, these appear to be complex, visual challenges that push the boundaries of AI. However, I’ve begun categorizing these as demo-benchmarks. They are visually intuitive enough for the general public to grasp, yet finite enough that a developer can ensure the next model is "perfect" at them.

The Illusion of Capability

The fundamental issue is that these tests no longer provide an honest assessment of a model's intelligence. Because the targets are static, AI labs can simply optimize their models to ace these specific tasks before release.

"A test you can perfect on a schedule measures preparation instead of capability."

It isn't necessarily a "fail" on the part of the labs—it's basic marketing. A viral video of a pelican sells a product far better than a technical whitepaper. When you have a fixed target and a few months of development, the result is a breakthrough "solved pelican."

The Cycle of "Scheduled Shock"

The Data Leakage Problem

This phenomenon isn't limited to viral videos; it extends to formal open evaluations. We often see smaller models that feel less capable in real-world use outperforming "smarter" models on platforms like Artificial Analysis.

For example, consider the relationship between model size and performance: $\text{Capability} \neq \text{Benchmark Score}$

Specifically, Thinking Machines’ Inkling Small—which possesses less than $\frac{1}{3}$ of the parameters of its flagship counterpart—scored nearly identically on the Artificial Analysis Intelligence Index. It even surpassed the larger model on:

1. GPQA Diamond
2. SciCode
3. Humanity’s Last Exam

This happens because public, static test sets inevitably leak into the training data or influence fine-tuning. The "shock" we feel at a launch is often just a scheduled marketing event.

Finding a Real Benchmark

If static tests are broken, how do we actually measure progress? The solution is holdout evaluations—tests where the model cannot "study" the answers beforehand.

We already see this approach in:

• LiveBench: Rotates its question sets.
• ARC-AGI: Maintains a private testing set.
• Humanity’s Last Exam: Keeps a portion of the test hidden.

The "Real Benchmark" Checklist

• Is the test set private?
• Does the test rotate frequently?
• Is it resistant to training-set leakage?
• Does it measure reasoning over memorization?

Why the Pelican Still Wins

If holdout evals are superior, why do we still obsess over the Minecraft clones? Because the average social media user doesn't want to read a research paper. They want to see that the bicycle finally has pedals. A demo-benchmark communicates a "win" in seconds.

Ultimately, if a model boasts high scores but fails to assist with your actual professional workflow, that gap is the only metric that truly matters.

def evaluate_model(model):
    if model.can_do_viral_demo():
        return "Marketing Success"
    elif model.solves_unseen_problems():
        return "Actual Intelligence"
    else:
        return "Overfitted"