An AI feature is not trustworthy because its model ranks well or its demo feels fluent. Trust comes from repeated evidence that the complete product performs a specific job, stays inside policy, exposes uncertainty, and recovers safely when it cannot proceed.
Evaluate the task, not the model
Begin with the user job and define what an acceptable result enables the user to do. A support draft might need to resolve the stated issue, remain grounded in account data, obey refund policy, and avoid inventing commitments. These are product criteria, not model aesthetics.
Build a representative set from real patterns: ordinary cases, ambiguous requests, policy edges, missing context, adversarial inputs, and historically expensive failures. Small, reviewed sets are more valuable than large synthetic sets nobody trusts.
The unit of quality is the completed user task—not the elegance of the generated sentence.
Measure quality in layers
Use deterministic checks for structure, permissions, required citations, and forbidden actions. Use reviewed rubrics for semantic quality. Sample human review to calibrate automated graders and inspect experiences a score cannot fully express.
Track failure categories separately. A single average can hide a severe policy regression behind an improvement in tone. Release decisions should reflect consequence, not merely statistical significance.
Build evaluation into the delivery pipeline
Put important AI behavior behind a typed, observable boundary. Record the policy decision, confidence route, evaluation version, and result metadata without retaining sensitive content unnecessarily.
Run a fast regression suite in pull requests and a broader suite before release. Monitor production acceptance, corrections, escalations, latency, and cost. Offline evaluation protects known behavior; production evidence reveals what the test set has not learned yet.
type TaskResult = {
answer: string
confidence: number
passesPolicy: boolean
sources: string[]
}
const result = await runTask(input, context)
if (!result.passesPolicy) return requestHumanReview()
if (result.confidence < threshold) return showSafeFallback()
recordEvaluation(result)
return presentWithSources(result)$ npm run evaluate:ai -- --suite support-drafts
✓ 128 cases completed
✓ policy pass rate 100%
✓ task acceptance 94.2%
⚠ human review rate 6.8%Design recovery as carefully as success
When the system lacks evidence or confidence, it should not improvise certainty. Ask for missing context, present a bounded fallback, save a draft, or route the case to a person with the evidence already assembled.
A good recovery path preserves user effort and makes the system legible. It is not an embarrassing exception. It is part of the product contract.
A production evaluation gate
- Acceptance criteria describe the user task and policy boundaries.
- The test set includes common, edge, ambiguous, and adversarial cases.
- Automated graders are calibrated against expert human review.
- Regressions block release according to consequence.
- Low-confidence and failed cases have a useful recovery path.