I posted to HN, and then my website was flooded with AI bot traffic
The HN Effect: When AI Bots Crash the Party
I recently shared my project on Hacker News, and the aftermath was a chaotic flood of AI bot traffic.
The Rise and Fall of a Viral Post
For several years, I have been developing sprinklz.io, a highly customizable feed reader. One of its standout features is the ability to deprioritize specific topics, allowing users to curate their news experience.
About a month ago, I posted the site to HN. The response was positive—roughly 100 upvotes! It seems there is a significant appetite for tools that let people mute the constant noise surrounding LLMs.
However, the success came with a price: my website started lagging my website became incredibly slow.
The Investigation
Initially, I assumed the latency was just the result of a sudden spike in human popularity. Like most developers, I felt a mix of pride and a desire to harden my infrastructure. But the slowness persisted and even intensified days later. The metrics were baffling:
- User Activity: A small number of authenticated, active users.
- Server State: Exorbitantly high CPU load.
I implemented top-level logging for every endpoint to track request volume per URL path. I discovered something strange: the majority of requests weren't even attempting to fetch news content. Since my site is a Single-Page Application (SPA), this meant most "visitors" weren't even executing the JavaScript required to render the page.
Identifying the Culprits
By analyzing the aggregate NGINX logs, I found that nearly 50% of all traffic originated from a single user. This entity described itself as a service that monitors website changes to generate "AI slop" newsletters. Their method of "listening" involved querying my site dozens of times per minute.
Other visitors included standard crawlers:
| Bot Name | Purpose |
|---|---|
Claude bot | AI Training/Indexing |
OpenAI bot | AI Training/Indexing |
Googlebot | Search Indexing |
Random Scanner | Searching for /wp-content/phpinfo.php (I don't use WordPress) |
The Recovery Plan
To mitigate the onslaught, I implemented a robots.txt file to block AI scanners while permitting search engines. I even used Claude to simulate vulnerability scans to ensure my site wasn't leaking sensitive data.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Googlebot
Allow: /
The real takeaway, however, was that my site needed to be faster. I want to handle a high volume of Queries Per Second (QPS), represented mathematically as:
Infrastructure Checklist
- Implement
robots.txt - Audit vulnerability paths
- Optimize backend response times
- Improve scaling under heavy load
If you're interested, sprinklz.io is completely free. It allows you to define arbitrary customizations and filters (like removing all mentions of "AI") beyond just sorting by HN points.
Making the Emacs Web Browser "Great Again"
When I mention the Emacs web browser (eww), people are usually shocked that Emacs even has one. Their second reaction is usually, "I guess that counts as a browser."
eww is designed for a terminal-based, text-only environment. While it excels at simple, 1990s-style HTML, it fails miserably against the modern web's chaos of heavy JavaScript and complex visual styling.
The Solution: LLM-Powered Simplification
The goal is to transform modern, bloated webpages into old-school, simple HTML that eww can actually handle.
The Workflow:
- Load the page in Firefox to handle all JS.
- Use an LLM to write a reusable program that converts the current DOM into simplified text.
- Inject that script back into the browser.
- Send the resulting source to Emacs.
Results
This allows me to browse sites that were previously inaccessible in eww. For example, the documentation for Clojure's mapcat function:
Figure 1: The standard view.
Figure 2: The simplified view.
While the initial script generation takes a moment, subsequent browsing is fast and rarely requires new LLM calls. The project is open-source at https://github.com/sstraust/simpleweb.
Final Reflections on Productivity
"Technology makes systems more productive, but it does not, in fact, make you in particular more productive."
The same tools that make computers efficient also make it easier for others to exploit our time and attention. I noticed this personally: I found myself spending three hours a day watching StarCraft 2 replays, wondering why I was still awake at 2:00 AM.
Interestingly, this wasn't a problem for me until the late 2010s. I've had internet access since the mid-2000s and a smartphone shortly after, yet the struggle with attention only intensified recently. It's possible I've simply become less resilient...