Benchmarking Retrieval for Agents on Messy Real-World Company Knowledge

By Finn Bauer | October 2, 2026

The landscape of information retrieval is shifting rapidly across all sectors. For instance, Cursor recently pivoted away from embedding-based code searches, opting instead for a combination of grep and indexed search. Simultaneously, other developers are replacing traditional rerankers with cutting-edge models like Jev.

One of the most critical assets for AI agents is company knowledge—a chaotic mix of:

• Technical documentation
• Support tickets
• Chat logs (Slack/Teams)
• Internal wikis (Confluence/Notion)
• Source code

The Core Problem: Most engineering teams are flying blind; they cannot determine which retrieval strategy is most effective because they lack the means to measure performance against actual, "messy" production data.

Introducing the Company Knowledge Bench

Kapa provides a platform that indexes diverse company data sources into a single, searchable knowledge base that agents can query for context. Because retrieval is the heartbeat of our product, we are constantly iterating on our approach.

To guide this evolution, we developed the Company Knowledge Bench: a dataset of 1,000 evaluation cases meticulously annotated from real-world production data.

---

Performance Analysis: Retrieval Score vs. Latency

We tested seven different retrieval implementations. The results highlight a significant trade-off between accuracy and speed.

Visualizing the Trade-off

Key Takeaways:

• A frontier model using only grep (Sol) can match a tuned modern pipeline (0.61), but it is $\approx 5\times$ slower.
• An optimized agentic retriever (Kapa Deep) achieves the highest accuracy (0.65) in roughly five seconds.

$\text{Note on Bias:}$ Since retrieval is our core product, all tested retrievers were developed internally and utilized documents processed through our own ingestion pipeline.

---

Why Standard Benchmarks Fall Short

Public benchmarks are often too sterile or narrow. They typically focus on a single domain (e.g., medical or legal) or rely on synthetic data that doesn't mirror reality. Real-world company knowledge involves diverse personas and query styles:

1. Diverse Use Cases

2. Varied Query Nature

Queries aren't always clean. They often fall into two categories:

• Human-generated: Messy strings containing multiple questions, pasted stack traces, or vague references to previous conversations.
• Agent-generated: Precise, decomposed searches (e.g., webhook retry backoff config) created by tools like Claude Code.

---

Defining "Good" Retrieval

To measure success, we define the goal of a retriever as: $\text{Goal} = \min(\text{Chunks}) \text{ such that } \text{Answer is Complete} \land \text{Quality is Maximized}$

The Company Knowledge Bench evaluates three primary properties:

1. Completeness: The retrieved set must contain everything the agent needs. If it's incomplete, the answer will be wrong or partial.
2. Efficiency: Adding more chunks is not always better. Extra, irrelevant chunks increase costs and introduce "noise" that hinders the model's reasoning.
3. Quality/Authority: When multiple sources provide the same answer, the retriever must prioritize the most authoritative one.
   • Reference Page $\rightarrow$ Tutorial $\rightarrow$ Blog Post
   • Recent Slack Thread $\rightarrow$ 3-year-old Confluence Page

Strict Benchmark Rules:

• Must cover every reasonable interpretation of a query.
• Must provide all documented solutions to allow the agent to explain trade-offs.
• Must prioritize "strong evidence" (e.g., an explicit statement that a feature is $\text{NOT}$ supported).

---

Anatomy of an Eval Case

Each evaluation case consists of a production query, a snapshot of the corpus at that moment, and a retrieval criterion. The criterion is a boolean expression defining which chunks are valid.

Example Case:

• Query: "Which plans include SSO, and how do I turn it on?"
• Retrieval Criterion:

{
  "operator": "AND",
  "operands": [
    "chunk_1", 
    "chunk_2"
  ]
}

• Retriever Result: chunk_1 (Pricing page) $\rightarrow$ FAIL (Missing chunk_2 regarding the "How-to" steps).