I Built a 500k-Domain Search Engine for Makers in a Weekend for $10

Sunday, 2:00 AM. I realized that every search query I actually cared about—weird art projects, personal zines, indie software, and portfolios—was being buried under a mountain of corporate documentation and SEO-optimized garbage.

Driven by late-night inspiration, I opened my terminal and mapped out a plan. I figured that even if I stored $1\text{ KB}$ of data per site, the total would be:

$\text{10 million domains} \times 1\text{ KB} = 10\text{ GB}$

I concluded that $40\text{ GB}$ was a perfectly manageable limit. By Wednesday lunch, I had successfully catalogued 560,183 homepages, emptied my queue of interesting targets, and decided to call it a day.

---

The Executive Summary

If you're short on time: for roughly $10, a temporary GPU rental, and a few hours of oversight, you can create a personal search index of several hundred thousand websites that takes up less than a gigabyte of disk space.

---

🎯 The Vision: What I Was (and Wasn't) Building

The goal wasn't to "index the internet." Instead, I wanted to find people doing things—poetry, hardware, code, and small theaters—without getting lost in docs.company.com.

The Blueprint:

1. A crawler that strictly targets homepages.
2. A small, local LLM to analyze each page and generate a name, a brief summary (2-3 sentences), a category, and tags.
3. A search interface featuring fuzzy matching for flexible querying.

To prevent "feature creep" (and to keep AI agents from over-complicating the build), I explicitly forbade the following:

• IP scanning
• Redis integration
• Storing full HTML pages
• Recrawl scheduling
• Multi-tenant architecture

Note: I didn't build a "block" feature for categories; I simply added a checkbox to the UI to filter them out during search.

---

⚙️ The Technical Architecture

The system consisted of four distinct processes. Three ran on my local machine, and one ran on a rented GPU (which never accessed the database directly).

System Components

Data Flow

The "Scratch Buffer" Strategy

I treated page text as a temporary buffer. The raw HTML lived in the domain's database row only while "in progress." Once the model finished summarizing, the text was wiped.

Why this matters: If you store raw text for 40 million rows, you'll run out of space instantly. I only needed the text for the few seconds the LLM was reading it.

---

🛠️ The Struggle: Fighting the "Corporate Web"

The first version worked quickly, but by Sunday afternoon, I realized I had indexed the wrong web. My seed list was biased, and the crawler lacked a "philosophy" on what to follow.

I didn't want to hard-block corporate sites because a boring corporate page might link to a brilliant personal blog. Instead, I implemented a weighted queue.

# Conceptual logic for category-priority.txt
priorities = {
    "portfolio": 1.0,
    "zine": 1.0,
    "software": 0.8,
    "corporate": 0.1,
    "docs": 0.05
}

I spent the weekend tuning category-priority.txt, watching the results, and wiping the database twice to purge low-quality data.

Bug Fixes & Lessons Learned

• The "Short Summary" Bug: Some sites had their category label (e.g., academic-profile) shoved into the summary field.
  • Fix: If a summary is suspiciously short, trigger a retry with a stricter prompt.
• The "Hallucination" Bug: A GoDaddy parked page with no content was labeled as a "furry community" site because the word "furry" was in the URL.
  • Rule: Visible Text $\gg$ Page Title $\gg$ Domain Name. If a page is empty or parked, skip the LLM entirely.

---

🌐 The "Tumblr" Problem

By Sunday night, I hit a new wall: Tumblr and Neocities. These platforms were producing so many pages that they threatened to swallow the entire index.

I couldn't block them—Neocities is exactly the "maker" aesthetic I wanted. Instead, I implemented a root-domain cap:

The Rule: Always allow the main domain, but once a root domain produces $> 100$ subdomains, stop adding new ones to the queue.

This single rule instantly deleted over 45,000 queued pages. While Tumblr still contributed 9,000+ pages, it no longer dominated the index.

By the end of the weekend, the project shifted from "indexing everything" to a focused mission: finding the people making things for a community.