An AI proof of concept should test a defined uncertainty, not demonstrate every possible capability. Write the expected output and unacceptable failures before selecting the examples shown in the demo. The proposed sprint here tests fictional service-request routing without sending messages or changing business records. A successful test would support only that bounded decision, not production readiness, compliance or financial return.
State the decision and the non-goals
Use a decision such as: can this candidate route approved request text into the agreed queues while abstaining on unknown requests? Name the queues, output schema, responsible owner and excluded actions. Do not combine routing accuracy, forecasting, document extraction and customer conversation into a single pass score. OpenAI's evaluation guide starts with a defined objective and a dataset chosen for that objective.
Freeze an independently checked evaluation set
Prepare examples with approved expected answers, including Arabic phrasing, incomplete inputs, conflicting facts and out-of-scope requests. Separate examples used to adjust the candidate from a held-out evaluation set. Record dataset version, selection method and exclusions. Synthetic cases can test declared rules, but they do not establish performance on real traffic. Do not silently remove difficult cases after seeing the output.
Use a routing example with a genuine abstention path
Fictional policy: requests explicitly about an invoice go to billing; requests explicitly about an account password go to support; unclear requests return needs_clarification. Input 'أحتاج نسخة من الفاتورة' should produce billing. Input 'عندي مشكلة' should produce needs_clarification, not a guessed queue. A request containing both intents should follow a separately declared mixed-intent rule. These are proposed expected results, not executed model results.
Measure failure types, not only an overall score
For the proposed test, report correct routing among answerable requests, incorrect routing among abstention cases, invalid outputs and unauthorized action attempts separately. Keep denominators and empty categories visible; no abstention cases means that behavior was not evaluated. Set business-specific thresholds before the run. A high average must not hide an unacceptable unauthorized action. Compare against a simple deterministic routing baseline under the same conditions.
Make the run reproducible and changes observable
Record the candidate version, configuration, dataset, output, timing and cost for every case, including failed and timed-out requests. A repeated request may behave differently, so define the repeat policy before testing and report its limits. OpenAI recommends continuous evaluation as applications change. This is a process reference, not a requirement to use its models, Evals platform or tools.
Publish a go, revise or stop decision
Use go only for the next explicitly scoped stage when its predeclared acceptance criteria pass. Use revise for a diagnosed, repairable defect with a bounded retest. Use stop when the task, data access or unacceptable failure cannot be resolved within the agreed scope. Attach unresolved questions and owners. NIST's framework is voluntary risk-management guidance; neither it nor a passed routing test supplies a legal approval or a production operating plan.
Key takeaways
- Choose one uncertainty to test.
- Keep tuning and evaluation examples separate.
- Report abstention failures and empty denominators.
- A successful PoC is not permission for unrestricted production.
Proposed acceptance cases to copy into a test record
- Invoice request → billing; compare the exact queue against the expected answer.
- Ambiguous request → needs_clarification; an invented queue fails.
- Mixed intents → the predeclared mixed-intent rule; no silent policy invention.
- Timeout or invalid JSON → no route or business action; log the failure and the permitted recovery.
The routing policy and cases are fictional, unexecuted examples. Thresholds, costs and production performance are not established by this guide.
Frequently asked
What is the difference between a demo and a validation sprint?
A demo shows selected behavior. A validation sprint records performance against a predefined dataset, failure policy and decision rule.
What pass rate should we require?
Set thresholds from the consequences of each error and the business baseline. This guide does not prescribe a universal rate.
Can synthetic examples prove production accuracy?
No. They can exercise declared cases. Real-world performance needs an authorized representative dataset and transparent evaluation limits.
What happens when a sprint fails?
Record the failure and decide whether a specific bounded repair is justified or the candidate should stop. Do not change thresholds merely to pass.
Related guidance
Sources
- Evaluation best practices: objectives, datasets and continuous evaluationOpenAIRetrieved: September 12, 2026
- AI Risk Management Framework: voluntary use and scopeNISTRetrieved: September 12, 2026
Editorial revision, 12 September 2026: unsupported generalizations replaced with scoped guidance and explicitly fictional examples. The examples and checklist are Ting recommendations, not client results, a benchmark or an automated publication approval. References support only their attributed descriptions, not Saudi legal requirements or business outcomes.
