Boosting ParadeDB's Search Performance

By Ming Ying | October 1, 2026

Recently, PlanetScale introduced TIN, a specialized full-text search extension for Postgres. In their announcement, they highlighted significant performance gains over a specific subset of ParadeDB’s features—specifically, document counting and BM25-ranked searches.

While we are thrilled to see more investment in Postgres search (proving that users desperately want to query relational data via text search) and appreciate the engineering effort behind TIN, we were particularly pleased to see them use our own benchmarker tool for their tests.

The Reality Check: Let's be honest—TIN was incredibly fast. In every single PlanetScale benchmark, it outperformed ParadeDB 0.25 by at least 8x.

Faced with that gap, we had only one choice: complain put on our performance optimization hats and get to work.

The Results: Two Weeks Later

After a fortnight of intense optimization, we re-ran the benchmarks using the same machine types and the StackExchange dataset. Note that since TIN is proprietary, it remains hosted on PlanetScale.

Benchmark Configuration

• Dataset: StackExchange Top K search (150M documents)
• Concurrency: 8 clients
• Duration: 5-minute execution
• Conditions: Read-only, warm-cache

Note: TIN utilized dense_ratio=2 with search elision disabled. Results represent a centered 9-second moving average.

The most compelling part of this result isn't just that we closed the gap, but how we did it. PlanetScale argued that their speed came from a fundamental architectural choice: using Postgres' internal ctid fields as document identifiers. We discovered that we could surpass that performance through optimizations that had almost nothing to do with document identification.

---

Understanding Text Search Architecture

To understand the debate, we first need to look at how search indices work.

Sequential Scan vs. Indexed Search

The core of any search index is the postings list: a list of document IDs containing a specific term. For example, if the word "database" appears in documents 2 and 4, the postings list is simply [2, 4].

The DocId vs. ctid Conflict

ParadeDB is powered by Tantivy, which uses sequential u32 identifiers (referred to here as DocId) based on the order of insertion. However, Postgres uses a ctid, which is a physical pointer to a row's location:

$\text{ctid} = (\text{block\_number}, \text{slot\_number})$

Example: (190, 17) means the row is in slot 17 of block 190.

The TIN Argument: PlanetScale claims that by using ctid directly as the identifier, they eliminate the need for a mapping table between DocId and ctid. They argue this enables:

1. More efficient visibility checks.
2. Faster bitmap operations.
3. Reduced data reads by organizing postings around Postgres pages.

The ParadeDB Counter-Argument: While we do map DocId $\rightarrow$ ctid, we defer this lookup until the final Top K documents are identified. In our profiling, these lookups are negligible and cannot account for an 8x performance difference.

---

Optimization 1: Eliminating Random Access in BM25

We began by analyzing a simple query on a smaller (28.7M doc) Hacker News dataset:

EXPLAIN (ANALYZE, BUFFERS) 
SELECT id, title, by 
FROM hn_items 
WHERE title @@ 'database' 
ORDER BY pdb.score(id) DESC 
LIMIT 10;

We noticed TIN touched significantly fewer Postgres pages. Using our new page-access attribution feature, we found the culprit:

Page Access Distribution:

• Fieldnorms: 1,513 accesses (83%)
• Everything else: 311 accesses (17%)

The "Fieldnorm" Problem

BM25 requires "fieldnorms" (the length of the indexed field) to normalize scores. In Tantivy, these are stored in a separate array indexed by DocId.

While this is efficient in memory-mapped storage, in Postgres, it caused a disaster: reading a sequential postings list required jumping all over the fieldnorm array, triggering ~1,500 distinct page reads for a single query.

The Solution

We changed the storage layout. Instead of a separate global array, we now store a fieldnorm array directly alongside each postings list, ordered by the same DocId sequence.

Our Optimization Checklist:

• Identify high page-access overhead.
• Trace overhead to fieldnorm lookups.
• Relocate fieldnorms to be co-located with postings.
• Enable sequential reads for scoring.

By doing this, scoring now reads fieldnorms sequentially alongside the postings, drastically reducing random I/O.