Creepy Crawlies

By Konstantin Ryabitsev | August 29, 2026

I have frequently voiced my frustrations regarding "AI crawlers," but I now have the empirical data to illustrate the actual damage they are causing.

To put it simply: the impact is severe enough to create a persistent "background radiation" of system load. We are permanently sacrificing a significant portion of our capacity to generate output that serves only one purpose: feeding a machine learning model.

TL;DR: We are currently spending more CPU cycles rendering commits for scrapers than we do for all legitimate traffic combined—including git clone operations.

Across five geo-distributed nodes, there are consistently 14 CPU cores doing nothing but transforming git commits into HTML.

---

Why is git.kernel.org a Target?

The development of the Linux kernel is an open book. Between the clonable git repositories and the real-time discussion archives, it is a goldmine for Large Language Models (LLMs). This data is not only accessible but easily filtered to ensure it is "pure," pre-AI content.

"Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease."

Because the entire history of kernel commits is guaranteed to be LLM-free, it is incredibly valuable training data.

The Most Inefficient Method Possible

We encourage cloning. We want the data to be distributed so that the history is preserved regardless of our own existence.

• The Easy Way: Just run git clone. You get the entire history instantly.
• The Archive Way: You can even clone the entirety of the LKML.

One would assume that "Artificial Intelligence" would prioritize efficiency. Instead, these bots employ the most absurd method imaginable: rendering every single commit as HTML and then parsing that HTML.

The Scale of the Problem

While the backend is efficient (most forks share the same objects), scrapers don't care. They crawl 922 duplicates of the same 1.48 million commits. Because cgit allows for patches, plain renders, and arbitrary diffs, the number of valid URLs for a single fork is practically infinite.

This was fine when the internet was for humans or bots that respected robots.txt. Now, it is a nightmare.

---

The Escalation War

We tried to stop them. It became a game of cat-and-mouse:

• User-Agent Blocking: Initially, bots identified themselves. We blocked them.
• Browser Spoofing: Bots began pretending to be standard browsers (Chrome, Firefox).
• IP Banning: We banned IPs that behaved inhumanly (e.g., scraping an 8-year-old abandoned fork).
• ASN Blocking: Bots moved to Google Compute Engine; we banned the entire Autonomous System Number.
• Residential Proxies: Bots shifted to millions of random mobile/residential IPs.

The final stage was the "locust" approach: a swarm of unique IPs making 4-5 requests each and then vanishing. Banning them was useless; it only bloated our firewall rules with IPs that would never return. This is the result of the "proxy SDK monetization" industry—where even your smart TV might be part of a botnet.

---

Implementing a "Computational Tax"

About a year ago, we tried to flip the economics of scraping by forcing bots to perform "throwaway math." We implemented a Proof-of-Work (PoW) challenge called Anubis.

The bot must find a nonce such that: $\text{sha256}(\text{IP} + \text{Secret} + \text{Nonce}) \rightarrow \text{starts with 4 zeros}$

The Results:

1. Initial Phase: Extremely effective. Bots gave up immediately.
2. Adaptation Phase: Bots returned and began solving the difficulty 4 challenge.
3. Escalation Phase: We moved to difficulty 5. This takes a few seconds on a mobile device and makes the hardware physically warm.

Current Status

Today, git.kernel.org handles roughly 6 million daily requests for random commits.

• 66% are blocked by the Anubis challenge.
• 33% are solving the math and successfully reaching the site.

The fact that 33% of requests are willing to burn significant CPU cycles just to access a commit suggests that our data is incredibly valuable to these models. It is nearly impossible to distinguish between a dedicated human and a sophisticated bot, but the patterns suggest the latter.

# Conceptual logic of the Anubis challenge
def solve_challenge(ip, secret, difficulty=4):
    nonce = 0
    prefix = '0' * difficulty
    while True:
        attempt = f"{ip}{secret}{nonce}"
        hash_result = hashlib.sha256(attempt.encode()).hexdigest()
        if hash_result.startswith(prefix):
            return nonce
        nonce += 1