How do I measure if my AI agent actually works?
Measure per action class: approval rate, edit rate, rejection rate, time saved per item, and downstream outcome. Cost per accepted action is the number that decides whether to keep a feature running.
Want to see it first? Book a demo
The short answer
- Measure an AI agent per action class, not as one overall number. Track approval rate (how often output is accepted), edit rate (how often it needs a real change first) and rejection rate (how often it's thrown out). Add time saved per item and the downstream outcome once the action has played out.
- Downstream outcome matters most and gets measured least. Think reply rate for a drafted email, accuracy confirmed later for a label, and close rate for a suggested next step. Approval alone only shows a reviewer thought it looked reasonable at the time.
- Cost per accepted action is what it costs to produce one output a person keeps and uses. It ties usage cost to real value. That number, not a raw usage total, should decide whether a feature is worth its price.
- 'Hours saved' from a vendor, rather than your own count, is usually a guess about how long a task would take a person. It isn't drawn from your team's real before-and-after time. Treat it skeptically unless you measured it yourself.
Why per-action-class measurement beats one overall score
A single blended score for an agent, like an overall approval rate, obscures more than it reveals once an agent does more than one kind of task. An agent might have a 95% approval rate on drafting summaries and 40% on suggesting stage changes. A blended 80% hides the split that matters. One action class works well. The other isn't ready for less oversight.
Breaking metrics out by action class turns "is this agent good" into a more useful question: which of this agent's jobs are working, and which still need review. That is also the same breakdown needed to decide which action classes are ready to move from review to automatic, so the same measurement serves both questions.
The metrics and how to compute them
| Metric | Formula | Target |
|---|---|---|
| Approval rate | Items approved as-is ÷ total items produced | Higher is better; watch for it masking a high edit rate |
| Edit rate | Items requiring a meaningful edit before approval ÷ total items | Lower is better; a rough starting bar is under 5% before considering promotion |
| Rejection rate | Items rejected outright ÷ total items | As close to zero as possible for a class considered for promotion |
| Time saved per item | Estimated manual time ÷ time actually spent reviewing | Meaningfully positive; a wash is not worth the review overhead |
| Downstream outcome | Reply rate, confirmed accuracy, or close rate after the fact | Comparable to or better than your team's own unassisted baseline |
| Cost per accepted action | Total cost of runs for a feature ÷ number of outputs actually kept and used | The number that decides whether a feature earns its cost |
Turning the metrics into a decision
These metrics only earn their keep if someone actually looks at them on a schedule and acts on what they show. A feature with a low approval rate and a high cost per accepted action is a candidate to fix, retrain the prompt, adjust the guardrails, or cut. A feature with a high approval rate and strong downstream outcomes is a candidate to promote from review toward more autonomy, following the same clean-period logic used to move an action class to auto.
Disclosure: SalesCrew is our product. Its audit log and cost tracking are built for exactly this kind of measurement. Every tool call is logged with the agent, action, result and cost. That's the raw data these metrics come from, per action class rather than as one blended score.
'Hours saved' self-reported by the vendor is not a measurement
Questions
- Why measure per action class instead of per agent overall?
- An agent's overall approval rate can hide a lot: strong performance on one task masking weak performance on another. Measuring per action class shows exactly which parts of an agent's job are working and which need attention, rather than one blended number that averages the two together.
- What is 'downstream outcome' measuring, specifically?
- The actual result after the action, not only whether a person approved it. A drafted email getting approved says the draft looked reasonable; a higher reply rate on approved drafts says the draft actually worked. The second is the harder, more meaningful number.
- Why is 'hours saved' from a vendor not a real measurement?
- Because it is usually an estimate made by the vendor, based on assumptions about how long a task would have taken a person, not a number derived from your own team's actual before-and-after time. Measure it yourself against your own baseline, or treat the vendor's figure as marketing.