Show HN: Pulpie – Models for Cleaning the Web

Pulpie: Pareto-Optimal Models for Cleaning the Web Feyn Open Source Blog | Research Field Note Authors: Bhavnick Minhas, Shreyash Nigam Affiliation: Feyn Labs Published: June 25, 2026 Estimated Reading Time: 13 minutes

---

Introduction

We are excited to unveil Pulpie, a suite of Pareto-optimal models designed specifically to extract primary content from HTML pages.

The core value proposition of Pulpie is simple: it achieves state-of-the-art (SOTA) extraction quality while reducing costs to one-twentieth of previous benchmarks. Our most compact variant, pulpie-orange-small, achieves a 0.862 ROUGE-5 F1 score on the WebMainBench. This effectively matches the performance of Dripper (the current industry leader at 0.864), despite Pulpie being significantly smaller—210M parameters compared to Dripper's 600M.

Performance Metrics at a Glance

Pulpie operates as an encoder that classifies every HTML block as either content or boilerplate in a single forward pass.

Calculation based on L4 instance pricing at \0.39/\text{hr}$.

By drastically lowering the cost barrier, Pulpie enables high-fidelity web extraction at a scale previously deemed impossible, which we believe will revolutionize pre-training and context window management.

---

The Bottleneck: Why Extraction Matters

"Language models essentially consume the web twice."

During our research, we observed that roughly 70% of blocks on a standard HTML page consist of "noise" (boilerplate), such as:

• Navigation menus
• Advertisements
• Sidebars
• Footers

The quality of the extraction process directly impacts model performance at both the training and inference stages.

Impact on Pre-training

According to AICC (Ma et al., 2025), cleaner extraction leads to better models. Their team compared two corpora from the same Common Crawl snapshot:

1. One using standard extraction.
2. One using model-based extraction.

The model trained on the latter saw a 1.08 percentage point increase in average accuracy across 13 different benchmarks. Remarkably, this approach outperformed models trained on FineWeb and RefinedWeb—datasets famous for their rigorous filtering. This proves that complex deduplication superior extraction is a high-leverage way to improve data quality.

Impact on Data Integrity

Heuristic-based tools often fail where model-based extractors succeed, particularly with technical content.

Furthermore, research from ICML 2023 indicates that even a single irrelevant passage can derail a model's reasoning. Therefore, noise-free context equals higher accuracy and efficiency.

---

Cleaning on a Budget

We categorize current extraction methods into two primary philosophies:

1. Structure-based Extractors: These rely on surface-level signals (tags, DOM structure, text density). While fast, they are easily fooled. For example, a navigation table and a data table look identical to an algorithm that only counts cells.
2. Reading Extractors: These use transformers to label blocks based on actual semantic content. However, they are traditionally slow because they require the full model to be read from memory for every single label.

Pulpie's Innovation: We maintain the "reading" approach but shift the bottleneck from memory bandwidth to compute. By utilizing an encoder architecture, we label all blocks in one forward pass.

Quality vs. Cost Trade-off

---

The Technical Pipeline

The Pulpie workflow follows these steps:

• Simplify: Strip unnecessary HTML attributes.
• Segment: Divide the page into discrete blocks.
• Tokenize: Convert text to tokens.
• Pack: Group blocks into chunks of $\le 8,192$ tokens for single-pass processing.

Training the Model

To train Pulpie, we required a massive, high-quality labeled dataset:

1. Sampling: 16,670 English pages from Common Crawl (one per domain).
2. Segmentation: Used MinerU-HTML for block splitting.
3. Labeling: Initial labels generated by DeepSeek V3.2.
4. Refinement: Filtered out corrupted pages, leaving 15,880.
5. Verification: Ran Dripper 0.6B as a second labeler. We only kept pages where both labelers agreed on $\ge 70\%$ of blocks, resulting in a final set of 14,959 pages.

The Teacher Model

We created a "Teacher" by fine-tuning EuroBERT-2.1B.

Training Configuration:

{
    "learning_rate": 2e-5,
    "effective_batch_size": 8,
    "loss_function": "class-weighted cross-entropy",
    "hardware": "4x A100"
}

Note: Class weights were set inversely to the 28.6% content rate to mitigate class imbalance.

The Teacher achieved a 0.873 ROUGE-5 F1 on WebMainBench. While accurate, it was too slow for production, leading us to the distillation phase.

Knowledge Distillation

We distilled the 2.1B Teacher into two smaller versions: Pulpie Orange Base (610M) and Pulpie Orange Small (210M).

The loss function used for distillation was a weighted combination: $\text{Loss} = 0.7 \cdot \text{KL-Divergence}(\text{Teacher}, \text{Student}) + 0.3 \cdot \text{Cross-Entropy}(\text{Labels}, \text{Student})$ (Calculated at a temperature of $T=2.0$)

Final Results Comparison