Skip to content

Verified Scenarios

Codeer is strongest when you build the agent and the important scenarios together.

The agent defines how it should answer, judge, and hand off. The scenarios show how it behaves on real questions. You need both sides at once: if you only write the agent, you are still guessing; if you write many cases without thinking about what the first version can actually do reliably, the scope becomes too scattered and hard to stabilize.

Start from the first scope

For the first version, choose a narrow set of scenarios that matter most. Usually 20 to 30 cases is enough.

Use four groups:

  • Core scenarios: questions the agent should answer now
  • Boundary scenarios: questions the agent should handle carefully, possibly by handing off or asking for more information
  • Out-of-scope scenarios: questions the agent should not answer yet and should refuse, hand off, or route to a form
  • Action scenarios: requests that need a tool, such as a form, payment, booking link, API call, or specialist agent

For a simple customer service agent, the first set might include:

  • A customer asks for the refund policy
  • A customer asks for a refund guarantee the business has not approved
  • A customer bought through a reseller and the policy is unclear
  • A customer wants a person to contact them
  • A customer asks for legal, medical, or competitor advice outside the supported scope

The goal is not to cover every possible question. The goal is to define the first scope you can understand, test, and improve.

You can ask Copilot to draft this first set from the current agent, then have an operator review and add the boundaries that truly matter. This keeps the agent capability and scenario scope aligned instead of making the first version too broad.

Turn each scenario into a case

In Test Suite, each important scenario should become a reusable case.

A useful case has:

  • a realistic user input
  • enough context for the agent to answer or hand off
  • a Standard that says what the AI response must do, must not do, and when it should hand off

Strong standards are checkable. Another operator should be able to read the AI response and decide whether it passed without guessing what you meant.

After you create the first cases, run them immediately. The purpose is not to prove the agent is perfect. The purpose is to find where it is unstable now.

Verify boundaries and out-of-scope behavior

Out-of-scope behavior is part of the product experience.

Do not only test questions the agent should answer. Also test questions it should not answer yet. A safe first version should know when to:

  • say the request is outside the supported scope
  • hand off to a human
  • ask the user to submit structured information through a form
  • avoid promises, prices, guarantees, or advice that your team has not approved

This is how you launch a narrow agent without pretending it can handle everything.

Even when the agent refuses or hands off, those conversations are still valuable. They become a real user query set that helps you decide which scope to expand next.

Keep knowledge small at first

Do not upload every document just because it exists.

Start with the smallest source of truth needed for the first important scenarios. Add more knowledge only when a failed case proves the missing knowledge is the real problem.

When a case fails, diagnose the cause before adding content:

  • If the Standard asks for the wrong thing, fix the Standard.
  • If the behavior rule is unclear, fix Instructions.
  • If the agent lacks required facts, add or clean the relevant Knowledge Base content.
  • If the agent needs an action, add or tighten the right tool.
  • If the scenario is outside the first scope, keep it on a safe fallback and add it to a later version.

Expand scope by version

Passing cases define the scope you can trust today.

For the first launch, keep the verified scope small. Route everything else to a safe fallback such as human handoff, refusal, or a request form.

When you want to expand:

  1. Add the new scenarios to Test Suite.
  2. Define the Standard for each scenario.
  3. Update the agent, knowledge, or tools only where the failed cases show a need.
  4. Run the affected cases.
  5. Run the broader suite if the change could affect existing behavior.
  6. Publish a new version only when the expanded scope is stable.

This creates a version-by-version path from a small controlled agent to a broader service layer.

Two ways to follow the method

You can follow this method directly in Codeer, or use another AI assistant to help draft the working material.

Path Use it when How it works
Codeer UI plus your own AI assistant You already use ChatGPT, Claude, Gemini, or another assistant for planning Ask it to draft scenarios, standards, and possible fixes, then review and enter the final version in Codeer
Codeer Skill workflow You want a guided workflow for scope, cases, debugging, and improvement Contact ian@codeer.ai if you want access when this guided workflow becomes available

The method is the same in both paths: define the scenario set, verify the AI response against standards, launch only trusted scope, and expand through versions.