Rethinking Search Indexing for Object Storage

Date: October 15, 2026 | Category: engineering Part 1 of a technical series detailing the architecture of Infino.

At Infino, we are developing a high-performance retrieval layer designed to power AI agents at scale. Our goal is to provide one of the fastest full-text search implementations available [1]. Because agents interact with a mix of structured, semi-structured, and unstructured data, our engine must seamlessly integrate:

• SQL (Structured queries)
• Vector Search (Semantic similarity)
• BM25 (Full-text search)
• Hybrid (Any combination of the above)

Since most enterprise data already lives in object stores—which provide inherent high availability and massive scalability—we chose to build our search index directly on top of S3, GCS, and Azure Blob Storage.

The Paradigm Shift: Disk vs. Cloud

Traditionally, search engines are designed with the assumption that the index resides on a local disk. For example, Lucene (the foundation for most modern search engines) assumes that the CPU and the index are physically adjacent, making random seeks relatively inexpensive.

However, when the index moves to the cloud, the physics change. A random read is no longer a disk seek; it is a network round-trip.

The Cost Model Comparison

The fundamental difference in performance and pricing is summarized below:

Key Insight: In an object-store architecture, the scarcest resource is not CPU cycles or total bytes transferred, but the number of requests made to the storage API.

---

Architecture Overview

Infino utilizes a specialized structure to minimize latency. The system is organized into superfiles, which are essentially Parquet files that embed Infino's vector and full-text indexes.

High-Level Data Flow

Design Goals Checklist

• Minimize HTTP GET requests.
• Maintain compatibility with standard Parquet readers.
• Enable atomic updates to the index.
• Support efficient pruning of irrelevant data.

---

Implementation Details

1. One File per Index Unit

Search indexes are typically built as immutable units. You ingest a batch of documents, create a self-contained index, and later merge these units in the background.

• In Lucene, this unit is a segment.
• In Infino, this unit is a superfile.

While a Lucene segment consists of multiple files (e.g., .tim for dictionaries, .tip for index, .doc for postings), opening one on S3 would require multiple GET requests.

The Infino Approach: A superfile is a single Parquet file. We splice the full-text and vector indexes into byte regions immediately preceding the Parquet footer. Because the footer contains the exact byte offsets, we can retrieve any specific section using a single ranged GET request.

2. The Manifest

To avoid the "cold start" cost of opening every superfile, we use a Manifest. The manifest tracks all superfiles in a supertable and stores:

1. The Parquet footer for every superfile.
2. The index headers required for term lookups.
3. Column statistics (e.g., $\min$, $\max$, and $\text{null\_count}$).

Why this matters: Normally, a reader would need one GET for the footer and one for the header before searching. By caching the manifest, we reduce the "cold cost" per superfile: $\text{Cost: } \underbrace{1 \text{ GET (Footer)} + 1 \text{ GET (Header)}}_{\text{Traditional}} \rightarrow \underbrace{1 \text{ GET (Actual Data)}}_{\text{Infino}}$

Additionally, column stats allow the engine to skip entire superfiles. For example, if a query filters for created_at > '2026-10-01', any superfile where $\max(\text{created\_at}) < \text{'2026-10-01'}$ is ignored entirely.

3. The Term Index

Even with a manifest, checking every superfile for a specific word (like "refund") would require at least one GET per file. With thousands of superfiles, this is prohibitively expensive.

Infino solves this with a global Term Index. This index maps terms to the specific superfiles containing them and stores:

• Document Count: How many docs in that superfile contain the term.
• Score Bound: The maximum possible BM25 score for that term in the file: $\text{Score Bound} = \max(\text{BM25 score for term } t \text{ in superfile } S)$
• Byte Offset: The exact location of the postings within the superfile.

To keep the term index manageable, it is split into slices based on term ranges. A query only fetches the specific slices relevant to its search terms, rather than the entire index.

# Conceptual logic for superfile pruning
if term in term_index:
    for superfile in term_index[term]:
        if superfile.score_bound > current_top_k_threshold:
            fetch_postings(superfile.byte_offset)

By combining the manifest and the term index, Infino ensures that the engine only performs network requests for data that is mathematically guaranteed to contribute to the final result.