Skip to content

Verified Scenarios

A verified scenario preserves a piece of expert judgment so the team can check it again after the Agent changes.

The Agent contains instructions, sources, and tools. A scenario shows what those parts produce in a recognizable situation. You need both: instructions without cases leave behavior assumed, while cases without a defined job become a scattered checklist.

Start from real work

Choose scenarios that represent the next decision you need to make. Use four groups:

  • Core scenarios: Work the Agent should handle now
  • Boundary scenarios: Work that needs clarification, caution, or human review
  • Out-of-scope scenarios: Work the Agent should refuse, hand off, or route elsewhere
  • Action scenarios: Work that requires a tool, form, payment, booking, API call, or another Agent

Do not limit the set to ideal questions. A safe first scope also proves what the Agent does when information is incomplete or the request should remain with a person.

Match the case set to the decision

There is no universal first case count. Build the smallest set that provides enough evidence for the next decision.

Stage Decision Evidence focus
First behavior Is this job understood well enough to continue? One real situation and nearby boundaries
Controlled pilot Can a small audience use the approved scope? Frequent and consequential must-pass behavior, tools, and handoffs
Wider rollout Can the next version handle more work without breaking trusted behavior? New scope plus regression, boundary, tool, and handoff cases
Formal acceptance Do the parties agree on what was delivered? Locked Agent version, case set, Standards, must-pass cases, known limits, and result record

Select cases by frequency and consequence. A rare case may still be must-pass when getting it wrong would create a serious safety, financial, service, or trust problem.

Turn each scenario into a case

In Test Suite, a useful case has:

  • A realistic Input
  • Enough conversation or task context
  • A Standard that says what the response must do and must not do
  • A handoff condition when the Agent should not finish the work

Ideal Response is optional. Use it when an example materially improves review, but do not let one sample answer replace the underlying quality criteria.

Strong Standards are observable. Another reviewer should be able to inspect the result and decide pass or fail without inferring the author's intent.

Mark the behavior that must not regress

Treat a case as must-pass when a release should stop if it fails. Typical reasons include:

  • The behavior is central to the Agent's job.
  • The error would cause a high-consequence action or claim.
  • The Agent must hand off rather than continue.
  • The behavior is part of an approved pilot or acceptance boundary.

Not every case has to block release. Separate diagnostic coverage from release conditions so the team can see both what is still being explored and what must remain correct.

Diagnose before changing the Agent

When a case fails, determine which contract is wrong or incomplete:

  • Fix the Standard when it asks for the wrong result.
  • Fix Instructions when the decision rule is unclear.
  • Fix the Knowledge Base when the required source is missing, outdated, or hard to retrieve.
  • Fix the tool when the action or invocation boundary is wrong.
  • Keep a safe fallback when the situation is outside the approved scope.

Avoid adding a narrow instruction only to make one case pass unless that case represents a real general rule.

Expand by version

When you want the Agent to handle more work:

  1. Add the new situations and their Standards.
  2. Confirm which cases are must-pass for the next release.
  3. Update only the instructions, sources, or tools the failures show are necessary.
  4. Run the affected cases.
  5. Run broader regression cases when the change may affect trusted behavior.
  6. Publish a new version only after the release conditions are met.

This turns corrections into an accumulating professional asset instead of a sequence of prompt patches.

Keep ownership clear

  • Experts or owners approve the quality criteria and boundaries.
  • Operators capture real patterns and maintain cases.
  • Implementers connect sources and tools.
  • Release reviewers decide whether must-pass conditions are met.

One person may cover several roles, but the responsibility behind each decision should remain visible.