How do I test an AI agent before trusting it?
Run it in draft mode on real data for two weeks. Score its outputs against what a person would have done. Measure edit rate and rejection rate per action class, and promote only the classes that pass. Test on your data, not the vendor's demo.
The short answer
- Run a new agent in draft mode, producing output for review but never sending or acting unattended, on your own real data for roughly two weeks before deciding whether to trust it further.
- Score its output against what a person would have done, not against an abstract standard of correctness. The comparison that matters is whether it saves the reviewer real time and produces something close to what they would have written or decided.
- Measure edit rate and rejection rate per action class throughout the test. Promote only the specific classes that clear a reasonable bar, not the agent as a whole based on one kind of task.
- A vendor's demo, run on clean, curated data, proves the agent can perform under ideal conditions. It proves nothing about how it will handle your mix of ambiguous contacts, half-finished deals and inconsistent notes.
Why draft mode is the right way to test, not a shortcut
It is tempting to skip straight to review mode, letting the agent produce real drafts that a person approves. That seems close enough to a real test. The gap is subtle but real. In review mode, a reviewer under time pressure can start approving quickly once early items look fine. That biases the test toward looking successful before enough volume has passed to reveal a pattern. In draft mode, nothing the agent produces has any live effect, whether approved or not. That removes the pressure. A reviewer can score honestly without worrying that a rejection blocks something time-sensitive.
Real data matters for the same reason a demo does not prove anything. Your CRM's contacts include duplicate names, threads that reference three different deals, and notes written in shorthand only your team understands. An agent has to handle that mess to be useful. A test on curated demo data will not reveal how it handles the mess until it is already live.
Metrics to track during the test, and a starting threshold
| Metric | How to measure | Pass threshold to start with |
|---|---|---|
| Edit rate | Share of drafts a reviewer had to meaningfully change before approving | Under roughly 5% to consider promoting the action class |
| Rejection rate | Share of drafts a reviewer rejected outright | Zero for the specific action class being tested |
| Time saved per item | Reviewer's estimate of time saved versus doing the task unassisted | Meaningfully positive; a wash is not worth the added review step |
| Downstream outcome | Reply rate for drafts, accuracy for classifications, checked after the fact | Comparable to or better than the team's own baseline |
What to do with the results
A two-week test on real data, scored against these metrics, gives an honest basis for a decision that is otherwise just a feeling. Does this agent's drafting save real time? Or does it produce output that looks fine at a glance but needs substantial editing every time? The metrics also make the decision specific to an action class rather than a verdict on the agent overall. A strong result on drafting says nothing about how the same agent would perform on a different kind of task.
Disclosure: SalesCrew is our product, and every agent ships in review mode by default. That works as this kind of test in production. Drafts are produced. A reviewer approves, edits or rejects with a reason. The approval queue tracks exactly the edit and rejection signal this testing depends on, per agent and per action class.
Testing on a sandbox with clean data proves nothing about your CRM
Questions
- Why does a vendor's demo not prove anything about your setup?
- A demo runs on clean, curated data chosen to show the product well. Your CRM has messy contact names, ambiguous threads, half-finished deals and years of inconsistent notes. An agent that performs well on a demo has been tested on none of the conditions it will face.
- What counts as 'real data' for this kind of test?
- Your actual CRM records, inbound threads and contacts, run through the agent in draft mode so nothing is sent or changed, only produced for review. The point is exposing the agent to the genuine mess of your data, not a sanitized subset.
- Should every action class go through the same two-week test?
- The two weeks is a starting point, not a fixed rule. Higher-stakes action classes, like anything that creates a deal or books a meeting, deserve a longer or stricter test than something reversible and internal, like tagging.