Trusting the Consensus: When LLM Judges Agree, Is It Actually True?

"Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives."

By Krishna Balasubramanian and Sasha Podkopaev | August 26, 2026

---

📌 Executive Summary

In the realm of "LLM-as-a-judge," we often rely on multiple models to verify a result. However, the quantity of agreement is often mistaken for the quality of evidence. This research, presented at ICML 2026, introduces a way to filter out "echo chambers" among AI judges.

Key Takeaways

• The Problem: Agreement counts are artificially inflated when judges share the same training data, model architecture, or prompt templates.
• The Innovation: Use of Ising models for dependence-aware label aggregation.
• The Result: A $\text{9\% to 14\%}$ boost in accuracy compared to weighted-majority-vote baselines.
• The Application: Tested across three critical domains:
  1. Relevance Classification
  2. Toxicity Detection
  3. Summarization Assessment

---

🔍 The Illusion of Agreement

Imagine a standard RAG (Retrieval-Augmented Generation) pipeline:

1. User $\rightarrow$ Asks a question.
2. System $\rightarrow$ Retrieves a text snippet.
3. LLM Judge $\rightarrow$ Determines if the snippet is relevant.

To minimize errors, developers often employ a panel of judges. The logic seems simple: More votes = More truth. However, this assumes each judge is an independent observer.

The Correlation Trap

If you have ten judges and eight agree, it looks like a landslide. But what if those eight judges are all versions of the same model family (e.g., Llama-3 variants)? They likely share the same blind spots.

In the diagram above, Judge 3 and 4 are correlated. Their agreement isn't two independent pieces of evidence; it's one piece of evidence repeated twice.

---

🛠️ The Solution: Dependence-Aware Aggregation

The authors propose treating a judge panel as a network. Instead of a simple tally, they use Ising models to distinguish between independent evidence and shared mistakes.

The Mathematical Intuition

Rather than treating the probability of a correct label $y$ as a simple sum, the model accounts for the dependency $\omega$ between judge $i$ and judge $j$:

$P(\text{label}) \propto \exp\left( \sum_{i} \alpha_i s_i + \sum_{i,j} \omega_{ij} s_i s_j \right)$

Where:

• $\alpha_i$ represents the individual reliability of judge $i$.
• $\omega_{ij}$ represents the correlation (dependence) between judges.
• $s$ represents the judge's signal.

Logic Implementation (Pseudo-code)

def aggregate_labels(votes, correlation_matrix):
    # Instead of: return max(votes, key=votes.count)
    # We apply the Ising-based weighting:
    adjusted_weights = calculate_independence(votes, correlation_matrix)
    final_decision = weighted_sum(votes, adjusted_weights)
    return final_decision

---

📊 Performance Comparison

The new method was tested against the "Weighted Majority Vote" (where judges are weighted by their known historical accuracy).

---

🚀 Practical Guidance for AI Engineers

If you are building an LLM-as-a-judge pipeline, don't just count votes. Follow this checklist:

• Statistically Evaluate Diversity: Check if your judges are too similar (e.g., all from the same provider).
• Inspect Clustering: Look for patterns where specific models always agree regardless of the truth.
• Adjust Confidence: Report confidence levels based on independent evidence, not total vote count.

---

🖼️ Research Context

This work is part of Amazon's broader commitment to advancing Conversational AI and Machine Learning.

Related Research Areas: