Status Report: Residential Proxies and the AI Scraping Crisis

Following a report published in early 2025 titled "Fighting the AI scraper bot scourge," which detailed the aggressive harvesting of web data for Large Language Model (LLM) training, the situation has only deteriorated. Over a year later, the scale of the problem is expanding.

The "open web" is becoming increasingly fragile as shadowy entities hammer sites with unprecedented intensity. The core question remains: Where is this traffic originating, and how can it be stopped?

The Mechanics of Residential Proxies

Modern scraper attacks do not originate from a few centralized servers. Instead, they utilize a massive, distributed array of sources. It is common to observe coordinated strikes involving millions of distinct IP addresses over a few hours, where the frequency of hits per IP is remarkably low:

$\text{Hits per IP} \approx 2 \text{ to } 3$

To evade detection, these bots use fictional User-Agent strings to mimic legitimate human browsers. While there are tells—such as the fact that these bots typically ignore CSS files or images—by the time a site administrator identifies the pattern, the specific IP address has already vanished and will not return.

The Infrastructure Flow

This traffic is routed through residential and mobile networks via a command-and-control (C2) architecture:

In most instances, the owners of these devices are completely unaware that their hardware is being used as a proxy.

Categorizing the Operators

The entities managing these residential-proxy networks generally fall into three categories:

1. The Criminal Element

These operators deploy malware to hijack systems.

• Case Study: Earlier this year, Google dismantled a network known as IPIDEA.
• Impact: Following the shutdown, LWN experienced a significant period of peace temporary lull in scraper traffic.
• Vectors: Recently, media-streaming devices have emerged as primary targets, either because they are compromised during manufacturing or possess critical security vulnerabilities.

2. The "Pseudo-Legitimate" Services

Some companies operate in a gray area, claiming their IP addresses are "ethically sourced."

• Bright Data: A prominent example that openly markets its ability to bypass traffic limits and access controls.
• The "Free" Trap: They offer free VPNs; the "cost" is that the user allows the company to route third-party traffic through their device.
• SDK Monetization: Some providers offer libraries for app developers, paying them to essentially hijack their users' network connections. (LWN was even approached by one such company to run an advertisement for their SDK—a conversation that ended very quickly).

3. Frontier AI Companies

The high-profile labs building the world's most famous models also scrape, but their methods differ:

• They generally identify themselves in the User-Agent field.
• They typically respect robots.txt directives.
• While they are repetitive (scraping old articles as if they might have changed), they do not use the "million-IP" stealth tactic.

Summary of Scraper Types

The Hidden Risks

It is a dangerous assumption to believe these networks are only used for scraping public text.

"To assume that this type of access would only be used for scraping would be naive at best."

Because these operators can execute code on millions of devices, they potentially have a gateway into any private network those devices are connected to.

The Unanswered Question: Who is Paying?

While the frontier AI companies are the most visible, it is unclear who is funding the residential-proxy networks. There is currently no direct evidence that the major AI labs are the clients of these stealth networks.

However, given their history of opacity regarding training data and their general disregard for content creators, such a revelation would be unsurprising. Furthermore, for every public-facing model, there are likely countless "undercover" models being developed by corporations desperate to win the AI arms race.

Bot Detection Checklist

• Check for missing CSS/Image requests
• Analyze IP rotation frequency
• Verify User-Agent consistency
• Identify C2 node origin (Difficult)