How Do You Set Acceptance Criteria for an AI Step That Is Only Right 92% of the Time?

Detailed Explanation

An AI step that gets 92 out of 100 examples right may be useful, unsafe, or both. The number does not tell you which 92 it got right, what the eight mistakes were, or whether a person would catch them before anything happened. That is why "92% accurate" is a weak acceptance criterion on its own.

For deterministic software, a buyer can often specify an exact input and expected output. A probabilistic system needs a more careful agreement because the same model can perform differently across document types, customer language and unusual cases. The acceptance test should therefore describe the operating conditions as well as the target result.

This sits one level below a general automation requirements brief. The brief says what the system must do. The acceptance criteria say what evidence will prove that the delivered version does it well enough to use.

Write the test before the result exists

Agree on these items before the supplier runs the final test:

1. The exact task. "Read invoices" is too broad. "Extract supplier name, invoice number, invoice date, total and GST from Australian supplier invoices" can be tested.
2. The test set. Use examples from the business's real operating conditions, with duplicates and personal information removed where appropriate. Keep the final test cases separate from examples used to configure prompts or tune the system.
3. The scoring rule. State whether every field must be correct for a document to pass, or whether fields are scored separately. A supplier and buyer can produce very different percentages from the same results if this is left open.
4. Important categories. Report common invoices, scanned invoices, credit notes and handwritten annotations separately if each appears in the real workload. An overall average can hide a category the system handles badly.
5. Error severity. Define which errors are cosmetic, which require correction, and which make the workflow unsafe to release.
6. The operational control. State which outputs can proceed automatically, which need review, and what sends a case to a person.
7. The remedy. Record whether a failed test leads to correction and retest, reduced scope, a price adjustment, or rejection under the contract.

The Australian Government Contract Management Guide describes acceptance as the buyer determining whether a deliverable meets the contract requirements. It also recommends inspecting against the contract specification and following the agreed remediation process when the standard is not met. The same discipline applies to a small automation project even when the contract is much shorter.

A worked example

Suppose an AI step extracts five fields from emailed purchase orders. The proposal says it is "92% accurate". A usable acceptance schedule could instead say:

The figures in that example are illustrative, not a universal standard. A business should set the sample and thresholds around its own volume, variation and risk. The important change is that both sides can now run the same test and reach the same conclusion.

Treat different errors differently

An overall accuracy figure gives a one-character typo and a wrong payment instruction the same weight. The business consequences are plainly different.

Split the acceptance result by error type. For a customer-email drafting step, tone that needs a quick edit may be a low-severity error. Inventing a refund promise or disclosing another customer's information should be a release blocker. For document extraction, a missed internal reference might be correctable, while a wrong amount that passes into payment approval needs a stronger control.

This is also where the human review rule belongs. Sampling AI output is defensible only after the business has measured performance by category and knows which cases must still receive full review.

Test the system you will actually operate

NIST's AI Risk Management Framework calls for documented test sets, metrics and methods, and for performance to be demonstrated under conditions similar to deployment. In practical terms, do not accept a demonstration built from the supplier's clean examples if production will receive scans, forwarded email chains and inconsistent customer descriptions.

Run the acceptance test on the complete workflow, including validation rules, escalation and the human review screen. A model can score well in isolation while the live workflow still sends uncertain results down the automatic path. General pre-launch workflow testing remains necessary after the AI component has met its own acceptance criteria.

Things to Consider

• Keep a blind final set. If the builder has already seen and adjusted the system around every test example, the result measures familiarity with that set rather than performance on new work.
• Name the judge. The person scoring the result should understand the process and have authority to accept or reject it. Do not leave the final decision to the builder's own account manager.
• Record disagreements. Some outputs are genuinely ambiguous. Define how disputed cases are resolved and whether they are excluded, passed or failed.
• Make the criterion observable after launch. The same categories and error definitions should feed monitoring once the system is live, rather than disappearing after the invoice is approved.
• Allow a narrower safe release. If one category fails, the right answer may be to automate the reliable categories and route the rest to a person, not to accept the whole system or abandon it entirely.

Common Mistakes

• Putting a percentage in the proposal without defining the denominator. "92%" could mean fields, documents, tasks or subjective ratings. Each produces a different result.
• Testing only the common, clean cases. This produces a high average while saying little about the exceptions that create the most work or risk.
• Changing the scoring rule after seeing the result. Agree on the test and remedies before delivery so neither side can move the standard when the outcome is inconvenient.
• Accepting the model but not testing the workflow. Escalation, validation and review can fail even when the model's isolated result looks strong.
• Treating every error as equal. A useful acceptance plan distinguishes tolerable correction work from errors that must block release.