RAG Is Simpler Than You Think

By Rafael Pierre | June 10, 2026

"Nowadays, most people seem to over-engineer their RAG stack."

Many developers immediately dive into the deep end with vector databases, complex reranking pipelines, and embeddings. However, the end-user often just needs a simple answer, like "How do I reset my password?" The key to successful engineering is selecting the correct tool for the specific problem, rather than the most complex one.

---

🛠 Decision Factors

Before choosing a "recipe," evaluate your system based on these five criteria:

---

Recipe 1: The MVP — Full-Text Search Only

This approach relies on the classic BM25 algorithm. It is highly effective when:

• Users search using specific keywords (e.g., "pandas merge dataframe").
• Your data contains highly specific, proprietary terminology.

The Advantage: You bypass the "chunking nightmare." When you start with embeddings, you are immediately forced to solve the chunk_size and overlap puzzle. With full-text search, you don't.

---

Recipe 2: Agentic Query Rewriting

Many "semantic search" failures are actually just query formulation problems. Users might say "fix bugs," while the documentation uses the term "debugging."

The Strategy

Use a lightweight LLM (like GPT-4o-mini) to transform a messy user query into a clean, keyword-optimized search string.

Capabilities of the Rewriter:

• Stopword Removal: "How do I..." $\rightarrow$ $\emptyset$
• Synonym Expansion: "car" $\rightarrow$ "car automobile vehicle"
• Domain Translation: "speed up code" $\rightarrow$ "optimize performance"
• Decomposition: "read CSV and plot" $\rightarrow$ ["read CSV", "plot data"]

Cost: \approx \0.001$ per query.

The Agentic Loop

Instead of a single pass, you can implement a feedback loop:

def agentic_search(query, max_iterations=3):
    for i in range(max_iterations):
        # Rewrite query for optimization
        optimized = query_rewriter.rewrite(query, iteration=i)
        # Execute keyword search
        results = bm25_search(optimized)
        # Evaluate the quality of the retrieved docs
        quality = evaluate_results(results, query)
        
        if quality > threshold:
            return results
            
        # Learn from failure and try again
        query = refine_based_on_feedback(query, results, quality)
    return results

The "Proprietary Terminology" Problem

General embedding models are trained on the open web. If your company has a tool called "Atlas," a general model sees: $\text{Atlas} \rightarrow \text{Greek Mythology/Maps}$ (Similarity $\approx 0.15$)

By using a system prompt for query rewriting, you can force exact matches:

system_prompt: "Preserve these internal terms exactly: Atlas (data framework), Mercury (messaging), Zeus (auth)."

Result: Atlas batch jobs $\rightarrow$ Perfect BM25 Match $\checkmark$

---

Recipe 3: Hybrid Search (Sparse + Dense Reranking)

This is a two-stage pipeline: use BM25 to grab a wide net of candidates (top 50-100), then use embeddings to rerank the top 10.

When to deploy this:

• Users are asking conceptual questions (e.g., "find alternatives to X").
• You have data proving that query rewriting isn't enough.
• Your corpus is relatively stable.

The Math (Cost): Using text-embedding-3-small at \0.02 per 1M tokens: $$\text{50 docs} \times \text{500 tokens} = 25,000 \text{ tokens}$$ $$\text{Cost} = 25,000 \times \0.00002 \approx $0.0005 \text{ per query}$$

Note: The primary trade-off here is latency, not cost. Also, the chunking problem returns.

---

Recipe 4: On-The-Fly Embedding

Pre-embed everything and store it in a vector DB. Instead: Embed only the candidates retrieved by the search engine at runtime.

When to use this:

1. High Churn: More than 10% of your documents change daily.
2. Experimentation: You want to test different embedding models without re-indexing.

The Economics (1,000 queries/day, 50 docs/query):

• Monthly Cost: \approx \15$
• Storage: \0$ (Text only)
• Latency: $200\text{--}500\text{ms}$
• Freshness: Instant

The "Model Deprecation" Nightmare: When OpenAI moved from text-embedding-ada-002 to text-embedding-3, anyone who pre-embedded 10 million documents had to:

1. Re-embed all 10M docs.
2. Update the vector database.
3. Run regression tests.
4. Manage the cutover period.

With On-The-Fly Embedding, you simply change one API call.