The Arrival of AI Superforecasters

By Scott Alexander (Astral Codex Ten)

At a recent annual conference for prediction markets, it became clear that the field has undergone a massive transformation. Prediction markets have moved from being an obscure hobby to a multi-billion dollar industry, shifting from a legal gray area to a space where the President’s own son serves as an advisor.

During the event, I encountered two striking examples of AI success:

1. A startup founder whose AI scaled a mere $35 into $2 million on Kalshi within seven months.
2. Another founder claiming to outperform the stock market by 25% using a market-neutral strategy. While luck is always a factor, they've seen similar dominance on Polymarket and Kalshi.

For those who enjoy "extending the lines on graphs," the prediction was always that AI would surpass the elite human forecasters around 2026 or 2027. We are now seeing that moment arrive. It doesn't look like a cinematic event; it looks like bots generating massive profits and comfortably beating the S&P 500.

---

What Exactly is an "AI Superforecaster"?

An AI superforecaster isn't just a raw LLM. It is a frontier model (such as Claude or ChatGPT) wrapped in a "scaffold."

A scaffold is essentially a programmatic layer that guides the AI through a rigorous research methodology. Instead of a single prompt, the scaffold manages:

• Iterative research loops.
• The deployment of specialized subagents.
• Strategic prompts and tool usage.

The result is a process that is slower and more expensive than a standard chat, but significantly more accurate.

---

Case Study: The "Common Cold" Test

I tested a tool from FutureSearch (the firm beating the stock market) with a specific query: What is the probability that the rate of common colds will be reduced by 50% by the year 2040?

The AI's Workflow

The AI didn't just guess; it executed a plan:

• Deploy three distinct subagents.
• Analyze 16 different websites.
• Investigate the scalability of ASHRAE Standard 241 for residential air cleaning.

After five minutes and a cost of $8 in credits, it returned a probability of 7%.

The Reasoning

The AI provided a detailed breakdown of why the goal is so unlikely:

The Conjunctive Chain of Failure:

1. Biological Complexity: There are over 200 cold-causing viruses. Even with 50 years of effort, vaccines have failed; as of late 2025, experts didn't see a vaccine arriving within 5 years.
2. Behavioral Resistance: COVID-era lockdowns barely dented rhinovirus rates, and they bounced back instantly.
3. Financial/Regulatory Hurdles: Current roadmaps suggest commercialization is 5–7+ years away, leaving almost no time for Phase 3 trials and FDA approval by 2040, especially with a budget under $500M.
4. Compliance: People are unlikely to use nasal sprays daily for a "mild" illness.
5. Infrastructure: Retrofitting air cleaners is expensive, and a 2025 RCT on germicidal UV showed no significant infection reduction.
6. Measurement Issues: The US lacks the surveillance to even prove a 50% reduction, as the 2020–2026 baseline is skewed by the pandemic.

---

Comparing the Forecasters

To see if this was an outlier, I compared FutureSearch's result with another AI and a human expert.

The alignment across these three diverse sources is remarkable.

---

Measuring Progress at Scale

How do we actually prove AI is better? We cannot use a simple "accuracy percentage" because the difficulty of questions varies. If the question is "Will the sun rise?", a 100% success rate is meaningless.

Instead, we use head-to-head matching against known benchmarks:

• The General Public: Baseline.
• CIA Analysts: Professional baseline.
• Metaculus Community: A "wisdom of crowds" aggregate.
• Metaculus Pros: The elite human superforecasters.

The Metaculus Data

According to Metaculus, base models (like Gemini 3.1 in May 2026) were rapidly approaching the "Community Prediction" level. However, they were still trailing the "Pro" humans.

But there is a catch: the base models aren't the ceiling. Metaculus found that scaffolding provides a massive boost.

# The "Scaffold Boost" Logic
base_model_performance = "May 2026 Level"
scaffold_advantage = "9 months of progress"
effective_performance = base_model_performance + scaffold_advantage 
# Result: A scaffolded AI today = A base model in February 2027

If we take the growth trajectory of base models and add the $\approx 9$-month advantage provided by scaffolding, the conclusion is inevitable: The AI superforecasters have arrived.

(Example of performance trajectories)