Analyzing the Knowledge Horizons of Claude and GPT

By Shrivu Shankar

We can uncover clandestine details regarding the development of frontier AI models by "probing" them with highly specific, curated queries. By analyzing how these models respond to niche data, we can derive estimates for several hidden variables:

• Parameter Counts: Using Incompressible Knowledge Probes to score models on obscure facts.
• Dataset Composition: Utilizing Data Mixture Inference to see how models decompose tokens.
• Training Schedules: Estimating timelines via date-specific and self-identification queries (the primary focus of this analysis).

Disclaimer: All findings presented here are estimates. Because there is a lack of public "ground truth" data, some of these speculations may be inaccurate.

---

The Architecture of Model Training

The process of creating massive LLMs has generally converged into a three-tier pipeline.

The Three-Stage Pipeline

1. Pre-training: The model consumes a gargantuan volume of general internet data to create a base "auto-complete" engine.
2. Capability Enhancement: The base model is refined using "textbook-quality," domain-specific data to improve long-context comprehension and specialized skills.
3. Post-training: The model is molded into an "assistant" persona, focusing on reasoning, personality, and the ability to call tools.

While post-training is increasingly compute-heavy to boost reasoning, the most resource-intensive phase is the creation of the pre-training checkpoint (e.g., claude-super-secret-2026-11-01-base.cpkt).

Organizational Dynamics

• Pre-training Teams: Manage multi-month runs. These typically result in major version jumps (e.g., $\text{GPT-4} \rightarrow \text{GPT-5}$).
• Capability/Post-training Teams: Experiment with the latest base model. These updates usually appear as minor version increments.
• Distillation: High-performing post-trained models are often distilled into families (e.g., Fable, Opus, Sonnet, Haiku or Sol, Terra, Luna).
• Wait for 100% completion $\rightarrow$ Labs may release models from "half-baked" checkpoints if the $N+1$ version is already outperforming the completed $N$ version.

---

Deciphering Knowledge Timelines

To estimate when a pre-training checkpoint was finalized, I developed a "Historical Quiz" using daily facts sourced from Wikipedia (e.g., events from 2025 in the United States). I presented models with an 8-way multiple-choice test.

The Methodology

By plotting the error rate over time, we can identify the point where the model loses its "signal." I use the midpoint of the curve to determine the cutoff, as the slope is often smoothed by the model's ability to guess near-future events or due to undersampling of recent data.

$\text{Knowledge Cutoff} \approx \text{Midpoint of Error Rate Increase}$

Key Findings:

Note on Luna: While Luna may appear to "predict the future," this is an artifact of a high overall error rate caused by low reasoning effort. Even when probing coding package versions, the cutoff remains around January 2026.

---

Self-Reported vs. Factual Dates

I also tested how models identify their own "current" date.

• The Diagonal: If a model falls on the dashed diagonal, its self-perception matches its factual knowledge.
• Below the Diagonal: The model believes it is living in the past.
• The OpenAI Exception: Recent OpenAI models were excluded because the API automatically injects the current date into the system prompt.

Interpreting the "Vertical Strips"

In the data, some model families (e.g., GPT-4.1 nano $\rightarrow$ mini $\rightarrow$ Opus 4.7 $\rightarrow$ Sonnet 5) form vertical lines. If we treat the X-axis as the pre-training corpus and the Y-axis as post-trained behavior, these strips represent:

• A constant pre-training base.
• Increasingly sophisticated post-training on recency-biased datasets.
• Potential distillation processes.

---

Summary Checklist of Probing Techniques

• Fact-based Quizzes: Determine the "signal loss" date.
• Self-ID Queries: Compare perceived date vs. actual knowledge.
• Token Analysis: Infer dataset mixtures.
• Niche Fact Scoring: Approximate parameter counts.

# Example of a conceptual probe request
probe_request = {
    "prompt": "Which of the following occurred on Oct 14, 2025?",
    "options": ["Event A", "Event B", "Event C", "Event D", "Event E", "Event F", "Event G", "Event H"],
    "target_metric": "accuracy_rate"
}