The Bag of Decisions Reranker

Published October 8th, 2026

The Core Premise: What happens if we stop treating search queries as a "bag of terms" and start treating them as a "bag of decisions"?

Traditionally, search engines look for keyword overlaps. However, in a recent experiment, I explored a different path. Instead of simply asking a model (like Jev) a binary question—"Is this document relevant for {query}?"—I decided to let an LLM construct a customized relevance rubric for every single query.

🛠️ The Methodology

The process begins by prompting an LLM to generate a series of yes/no questions. If the answer to a question is "yes," it indicates that the document is relevant to the user's intent.

Example: "desk for kids"

For this specific query, the LLM might generate the following criteria:

1. Is the document about desks designed for children?
2. Does the document offer products or information on kids’ desks?
3. Does it mention child-sized or age-appropriate desk dimensions or ergonomics?
4. Does it reference desks for homework, study, or activities for kids?
5. Are features specific to children’s desks (e.g., height-adjustable for kids, safety/rounded edges, storage for school supplies) discussed?
6. Does it include buying guides, reviews, or comparisons of kids’ desks?
7. Does it provide pricing or availability for children’s desks?
8. Does it mention desks for specific child age ranges (e.g., toddlers, elementary, teens) as kids’ desks?
9. Does it cover where to buy or how to choose a desk for kids?

Technical Implementation

Instead of a single relevance score, we now have a bag of questions. We can use a decision model to evaluate a document against this rubric:

document = """ This adjustable study desk is perfect for kids ages 5–12. """

response = client.decide( 
    state=document, 
    questions={ 
        "question_1": Noul( instructions="Is this document about desks designed for children?" ), 
        "question_2": Noul( instructions="Does the document offer products or information on kids’ desks?" ), 
        # ... other questions
    }, 
)

To calculate the final rerank score, we sum the probabilities of "yes" for every question in the bag. Mathematically, the score is represented as:

$\text{Total Score} = \sum_{i=1}^{n} P(\text{yes})_i$

The implementation looks like this:

total_probability = 0.0 
for question_id, answer in response.answers.items(): 
    probability = answer.noul 
    print(f"{question_id}: {probability:.2f}") 
    total_probability += probability 

# total_probability is now the final reranker score

📊 Experimental Results

I compared three different strategies across two e-commerce datasets: Wayfair WANDS and Amazon ESCI.

1. BM25: The baseline keyword search.
2. Jev Reranker: A simple "Is this relevant?" prompt.
3. Bag of Decisions: The multi-question rubric approach.

Performance Metrics (NDCG)

🧠 Analysis & Logic Flow

Essentially, this method performs per-query feature creation. We use the LLM to identify what should be important, and then use the decision model to measure if those criteria are actually met.

Reflections and Future Work

While the results are promising, there is room for improvement:

• Redundancy: Some generated questions are repetitive.
• Diversity: We need better prompts to ensure the LLM provides more incisive and varied criteria.

Current Goals:

• Refine LLM prompts for better question diversity.
• Test on domain-specific decision models.
• Scale the experiment to larger datasets.

🎓 Learn More

If you're interested in this and other modern retrieval techniques, join Doug Turnbull and Trey Grainger for their upcoming course:

AI-Powered Search — Modern Retrieval for Humans & Agents Starting in November!

---

About the Author: Doug Turnbull Connect with Doug via: Twitter | LinkedIn | Newsletter | Bsky