Using LLM for confidence scoring is a bad idea
The Fallacy of LLM-Generated Confidence Scores
By Amrut Joshi | October 20, 2026
Modern LLM-based agents are incredibly versatile. They can handle a wide array of complex duties, such as:
- Writing executable code
- Summarizing academic literature
- Organizing travel itineraries
- Populating digital forms
Because they are so capable, many developers simply ask the LLM to provide a probability or a confidence score for its output. However, there is a fundamental problem: there is no logical reason to trust that number.
This article explores why relying on an LLM to self-assess its own confidence is a flawed strategy.
Defining "Confidence"
To understand why LLMs fail at this, we must first define what a confidence score actually represents. Consider a meteorologist who states:
"I am 70% confident that it will rain tomorrow."
This isn't a guess; it's a prediction based on specific inputs. The forecaster's certainty is derived from two primary sources:
- Real-time data: Current air pressure, humidity, wind patterns, and radar imagery.
- Historical data: Decades of records showing how often rain occurred under similar conditions.
In a perfect world, determining this percentage is an opinion a calculation. We can imagine this as a formal rule:
def calculate_confidence(historical_baseline, current_evidence):
# Update the baseline probability with new evidence
return update_probability(historical_baseline, current_evidence)
If two experts possess the same data and apply the same rule, they must reach the same confidence level. Discrepancies only arise if their data differs or one of them misapplies the rule.
The Impact of Time
If asked about the weather two weeks from now, the forecaster might only be able to say, "I am 30% confident," because 30% is the seasonal average. As the date approaches and more data arrives, that number shifts toward or .
Measuring Calibration
How do we verify if a confidence score is actually accurate? We cannot do it with a single event (since it either rains or it doesn't). Instead, we look at calibration.
If a forecaster claims 70% confidence on 80 different days, it should rain on approximately 56 of those days.
Example Calibration Table
| Confidence Level | Total Days Predicted | Days it Actually Rained | Actual Rate |
|---|---|---|---|
| 10% | 120 | 13 | 11% |
| 30% | 90 | 26 | 29% |
| 70% | 80 | 57 | 71% |
| 90% | 75 | 67 | 89% |
In this table, the forecaster is calibrated because the predicted confidence closely matches the actual outcome rate.
Note: A forecaster who always predicts the seasonal average (30%) would also be calibrated, but they would be useless because they aren't using current information to refine their score.
Requirements for Trustworthy Confidence
For a score to be reliable, it must meet these criteria:
- It must reflect the volume and strength of the underlying evidence.
- It must be validated against actual outcomes for similar queries.
How LLMs Actually "Score" Confidence
An LLM does not have an internal "certainty meter." It generates text one token at a time. When you ask, "How confident are you?" and it responds with 0.85, it isn't reading a gauge; it is simply predicting that 0.85 is the most likely string of characters to follow that question.
The Logic Gap
The LLM's output fails both requirements for reliability:
- No Evidence Summary: The number is a text continuation, not a mathematical summary of the evidence used to reach the answer.
- Wrong Calibration: While LLMs are calibrated to predict the next token in a sequence, they are not calibrated to ensure the numerical value they write matches their actual probability of being correct.
The Resulting Failures
Because the score is just text, it is unstable:
- Inconsistency: Asking the same question twice might yield
0.85the first time and0.7the second, despite the information remaining identical. - Suggestibility: If you ask "Are you sure?", the model often changes the score simply because the conversation flow suggests a change is needed.
- Contextual Bias: If the model has already written "This is definitely the correct answer," it will likely output a high confidence score to maintain linguistic consistency.
A Better Way: Measuring Confidence Externally
If we cannot trust the LLM to tell us how confident it is, we must build an external system to determine it. We should create a rule-based system that evaluates observable data.
Case Study: Customer Support Agent
Imagine an agent that searches a help center to answer user queries. Instead of asking the LLM "How sure are you?", we track these metrics:
Observable Metrics to Track:
- Historical Success: Is this a query type the agent typically handles well?
- Consistency: If we generate the answer twice, are the results identical?
By using these "environmental" signals—the AI equivalent of air pressure and humidity—we can derive a confidence score that is actually based on evidence rather than a hallucinated number.