Asymmetric Quantization: Achieving Near-Lossless Retrieval with a 97% Storage Drop

Late interaction architectures, such as Wholembed v3, significantly enhance retrieval precision. Unlike traditional methods that squash an entire document into a single vector, late interaction preserves granular detail by generating multiple embeddings per document. Depending on the text's complexity, a single document might be represented by hundreds or even thousands of vectors.

Mixedbread Search utilizes silo, a specialized retrieval engine designed for multimodal late interaction at a scale of billions of documents. silo manages vectors for over 2.5 billion documents within object storage, hydrating them into high-speed tiers as queries demand.

The Scaling Challenge

At this magnitude, every single byte added to a document's footprint is multiplied billions of times. This directly impacts:

• Operational Cost: Higher storage expenses per document.
• Cold-Start Latency: Increased time to load shards.
• I/O Overhead: More bytes read per individual query.

The goal is to resolve the tension between making the system expensive keeping costs low while maintaining the high-fidelity quality that makes late interaction viable.

---

Quantization: Making Multi-Vector Storage Practical

Quantization involves converting high-precision floating-point vectors into lower-precision formats, such as int8 or 1-bit binary signs. The objective is to shrink the payload size without sacrificing ranking accuracy.

While single-vector embeddings (e.g., 3072 dimensions in fp32) require about 12 KiB per document, a multi-vector setup (e.g., 786 vectors of 128 dimensions) is roughly 33x larger when uncompressed.

Storage Comparison Matrix

Note: These figures represent raw vector payloads; actual production indexes include metadata and document IDs.

By utilizing binary document vectors, a 786-token multi-vector document is only $\approx 2\%$ larger than a standard single-vector fp32 embedding. This allows users to achieve late interaction quality at the cost of single-vector storage.

This approach builds on previous research: ColBERTv2 demonstrated that aggressive compression preserves quality, and PLAID showed that retrieval latency could be optimized through pruning.

---

Why Asymmetric Quantization?

The core logic is based on the lifecycle of data:

1. Documents are stored long-term $\rightarrow$ Compression is critical.
2. Queries are transient and processed once $\rightarrow$ Compression is negligible.

The Philosophy: We keep the query at a higher precision to maintain ranking intelligence, while binarizing the document vectors to slash storage costs.

If we were to binarize the query as well, we would lose the magnitude information essential for ranking, leading to a significant drop in quality.

Performance Summary

• Storage Reduction: 32x decrease (from 393 KiB to 12.28 KiB).
• Storage Savings: $\approx 97\%$.
• Quality Retention: Minimal loss (NDCG@10 dropped from $90.26 \rightarrow 89.65$).

---

The Scoring Trick

Binary document vectors are not just smaller; they allow for mathematical shortcuts.

Hardware Acceleration

For int8 $\times$ int8 operations, modern ARM CPUs provide optimized support via NEON dot-product instructions. The AArch64 kernel utilizes:

• SDOT: Accumulates sixteen int8 multiplications into int32 lanes.
• vaddvq_s32: Performs a horizontal reduction of the result.

Mathematical Optimization

When document dimensions are stored as sign bits $b_i \in \{-1, +1\}$, the dot product $\mathbf{q} \cdot \mathbf{b}$ can be rewritten to avoid expensive multiplications:

$\mathbf{q} \cdot \mathbf{b} = \sum_i q_i b_i = 2 \sum_{i\,:\,b_i = +1} q_i - \sum_i q_i$

In plain English: To calculate the score, we only need the sum of the query values where the document bit is positive, and the total sum of the query.

By implementing this "scoring trick," the system achieves the efficiency of binary storage without the typical quality penalties associated with symmetric binarization.