What is a confidence threshold?
The line that decides which of an agent's decisions run on their own and which go to a person.
Confidence threshold — A confidence threshold is the score below which an automated action is routed to a person instead of executed. It lets an agent in auto mode handle the clear cases and hand over the unclear ones.
Why it matters
Not every decision an agent makes is equally certain. Classifying a reply as "not interested" might be obvious from the wording; classifying a vague one-line reply might not be. A confidence threshold gives every decision a score and compares it against a set line: above the line, the agent proceeds on its own; below it, the decision goes to a person to review.
The score a model reports about its own certainty is not reliable on its own. A model can sound equally confident about a right answer and a wrong one, so the threshold should be calibrated against real outcomes: pull a sample of past decisions, check which ones the agent got right at each score range, and set the cutoff where accuracy actually drops off, rather than picking a round number like 80%. A threshold set too low sends confident-sounding mistakes straight through; one set too high routes almost everything to a person and defeats the point of automating it. Reviewing the threshold periodically, as the agent handles more cases, keeps it matched to how the agent is actually performing rather than how it performed on day one.
How a confidence threshold is set
- 1
The agent scores each decision
A number representing how certain it is about the classification or action.
- 2
Past decisions are sampled and checked
Against known outcomes, across a range of scores.
- 3
A cutoff is chosen from that data
The score below which accuracy drops enough to warrant review.
- 4
Decisions are routed by the cutoff
Above it, the agent proceeds; below it, a person reviews.
- 5
The threshold is revisited over time
As the agent's accuracy at each score range changes with more data.
The mistake to watch for
Questions
- How is a confidence threshold different from a guardrail?
- A guardrail is a fixed rule, like a daily send cap, that never changes based on the decision at hand. A confidence threshold is a variable cutoff applied to each decision's own certainty score, routing only the unclear ones to a person.
- Where does the confidence score come from?
- Typically from the model itself or from a separate scoring step over its output, then calibrated against a sample of past decisions with known correct outcomes.
- Should the threshold be the same for every kind of decision?
- No. A low-risk classification and a customer-facing send warrant different thresholds, since the cost of a wrong automated action is not the same for both.