Sites that block AI training crawlers mostly ignore the answer time bots
The robots.txt Blind Spot: Why Sites Ignore Answer-Time AI Bots
"Your
robots.txtis a 2023 war memorial." — Sitedex Insights Data Study
A recent analysis of the top 10,000 websites reveals a striking trend: while many site owners rushed to block AI training, they almost entirely overlooked the bots that power real-time AI answers.
The Great Panic of 2023
The data suggests that most AI-related rules in robots.txt files weren't the result of a long-term strategy, but rather a sudden reaction.
Specifically, 323 sites (out of 861 dated rules) added restrictions in a single three-month window in late 2023. This spike occurred immediately after the launch of GPTBot and the subsequent lawsuit filed by The New York Times.
Timeline of Reaction
- August 2023: OpenAI releases
GPTBot. - October 2023: The New York Times sues OpenAI.
- Q4 2023: Massive surge in
robots.txtupdates (the "panic window"). - 2024-2026: A slow drip of updates for newer crawlers.
Roughly half of all GPTBot rules were established within six months of the bot's debut. While many of these sites returned to add rules for ClaudeBot (671 sites) or PerplexityBot (540 sites), they continued to focus on training crawlers.
The Two-Bot Architecture
There is a fundamental distinction in how AI companies interact with the web. Most site owners treat AI as a single entity, but in reality, vendors deploy two distinct types of bots:
Comparison of Bot Roles
| Bot Type | Purpose | Timing | Impact |
|---|---|---|---|
| Training Crawler | Reads the web in bulk | Months/Years before output | Builds the model's knowledge |
| Answer-Time Fetcher | Fetches specific pages | Instantaneous | Provides live citations/answers |
The "Blind Spot" in the Rulebook
As we move into an era of "pay-per-answer" models—highlighted by Cloudflare's new system for charging AI bots in real-time and the development of x402 payment rails—the distinction becomes critical.
We can rely on "vibes" to understand this. Instead, Sitedex looked at the actual robots.txt declarations of the top 10,000 sites.
The Data: Training vs. Answering
For sites that fully blocked a vendor's training bot, how many addressed the answer-time bot?
| Vendor | Training Bot Blocked | Answer-Time Bot(s) | Said Nothing (Blind Spot) | Explicitly Allowed |
|---|---|---|---|---|
| Anthropic | ClaudeBot | Claude-User, Claude-SearchBot | 71% (598/840) | 1.9% (16/840) |
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User | 53% (493/932) | 4.0% (37/932) |
| Perplexity | PerplexityBot | Perplexity-User | 50% (229/461) | 0.9% (4/461) |
Analysis of the Gap
The vast majority of sites that shut out training crawlers have no rule at all for the bots that generate live answers. This means the "ideal" publisher strategy—blocking the model from learning from your data while still allowing the bot to send referral traffic via live answers—is almost non-existent in practice.
Intent vs. Silence
It is important to note that silence in a robots.txt file doesn't always mean a lack of awareness. Since most people don't write Allow rules for bots they are happy to let in, the "Said Nothing" column represents an upper bound on the blind spot.
However, the few who do act with intent are notable. For example, Forbes uses a specific configuration:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
This explicit stance—no training, yes answers—is the rare exception in a landscape where most sites are still fighting a war that ended in 2023.