How do I measure if my AI agent actually works?
Measure per action class: approval rate, edit rate, rejection rate, time saved per item, and downstream outcome. Cost per accepted action is the number that decides whether to keep a feature running.
The short answer
- Measure an AI agent's performance per action class, not as one overall number: approval rate (how often output is accepted), edit rate (how often it needs a meaningful change first), rejection rate (how often it is rejected outright), time saved per item, and downstream outcome once the action has played out.
- Downstream outcome is the metric that matters most and is measured least: reply rate for a drafted email, accuracy confirmed later for a classification, actual close rate for a suggested next action. Approval alone only shows a reviewer thought it looked reasonable at the time.
- Cost per accepted action, what it actually costs to produce one output a person keeps and uses, ties usage cost to real value and is the number that should decide whether a feature is worth its price, not a raw usage total on its own.
- 'Hours saved,' when it comes from a vendor rather than your own measurement, is usually an estimate built on assumptions about how long a task would take a person, not a number derived from your team's actual before-and-after time. Treat it skeptically unless you measured it yourself.
Why per-action-class measurement beats one overall score
A single blended score for an agent, like an overall approval rate, obscures more than it reveals once an agent does more than one kind of task. An agent might have a 95% approval rate on drafting summaries and a 40% approval rate on suggesting stage changes, and a blended 80% number hides exactly the distinction that matters: one action class is working well, the other is not ready to be trusted with less oversight.
Breaking metrics out by action class turns "is this agent good" into a more useful question: which of this agent's jobs are working, and which still need review. That is also the same breakdown needed to decide which action classes are ready to move from review to automatic, so the same measurement serves both questions.
The metrics and how to compute them
| Metric | Formula | Target |
|---|---|---|
| Approval rate | Items approved as-is ÷ total items produced | Higher is better; watch for it masking a high edit rate |
| Edit rate | Items requiring a meaningful edit before approval ÷ total items | Lower is better; a rough starting bar is under 5% before considering promotion |
| Rejection rate | Items rejected outright ÷ total items | As close to zero as possible for a class considered for promotion |
| Time saved per item | Estimated manual time ÷ time actually spent reviewing | Meaningfully positive; a wash is not worth the review overhead |
| Downstream outcome | Reply rate, confirmed accuracy, or close rate after the fact | Comparable to or better than your team's own unassisted baseline |
| Cost per accepted action | Total cost of runs for a feature ÷ number of outputs actually kept and used | The number that decides whether a feature earns its cost |
Turning the metrics into a decision
These metrics only earn their keep if someone actually looks at them on a schedule and acts on what they show. A feature with a low approval rate and a high cost per accepted action is a candidate to fix, retrain the prompt, adjust the guardrails, or cut. A feature with a high approval rate and strong downstream outcomes is a candidate to promote from review toward more autonomy, following the same clean-period logic used to move an action class to auto.
Disclosure: SalesCrew is our product, and its audit log and cost tracking are built to support exactly this kind of measurement: every tool call is logged with the agent, action, result and cost, which is the raw data these metrics are computed from, per action class rather than as one blended agent score.
'Hours saved' self-reported by the vendor is not a measurement
Questions
- Why measure per action class instead of per agent overall?
- An agent's overall approval rate can hide a lot: strong performance on one task masking weak performance on another. Measuring per action class shows exactly which parts of an agent's job are working and which need attention, rather than one blended number that averages the two together.
- What is 'downstream outcome' measuring, specifically?
- The actual result after the action, not only whether a person approved it. A drafted email getting approved says the draft looked reasonable; a higher reply rate on approved drafts says the draft actually worked. The second is the harder, more meaningful number.
- Why is 'hours saved' from a vendor not a real measurement?
- Because it is usually an estimate made by the vendor, based on assumptions about how long a task would have taken a person, not a number derived from your own team's actual before-and-after time. Measure it yourself against your own baseline, or treat the vendor's figure as marketing.