Skip to content

Review Conversations and Improve

Real conversations show what the Agent did in the user's actual context. Use them to handle the immediate need first, then decide whether the new judgment should change a future version.

The operating loop is:

Observe -> Handle -> Preserve -> Change -> Verify

Step 1: Handle the conversation that needs attention

Open Conversations and start with items marked Needs human reply, high-consequence situations, or messages with Improve feedback.

If a user is waiting, respond before turning the issue into an optimization task. Write the reply yourself or generate an AI draft, review it, and send it through the original Channel when available.

→ Reply to Conversations

Step 2: Inspect what happened in context

Read the complete conversation and identify:

  • The job the user was trying to complete
  • The information available to the Agent
  • The action or answer the Agent produced
  • The quality or boundary decision that was missed
  • Whether the problem belongs to the current scope

Feedback such as “too generic” is a signal, not yet a diagnosis. Translate it into observable behavior: which question should have been asked, which claim should have been avoided, which next step was missing, or when handoff should have happened.

Step 3: Preserve the situation with Add Case

Use Add Case when the situation should remain checkable after the Agent changes.

In Case Detail:

  • Keep the context required to understand the job.
  • Confirm Input represents the behavior to rerun.
  • Add an Ideal Response only when an example improves review.
  • Write a small Standard that another reviewer can score without guessing.

A useful Standard might require the response to ask one missing question, avoid a specific unsupported promise, and hand off when the source does not resolve the situation.

Step 4: Diagnose the smallest relevant cause

Determine whether the issue is in:

  • The expected behavior or Standard
  • Instructions
  • Knowledge Base content or retrieval
  • Tool configuration or invocation boundary
  • The decision to include the situation in the current scope

Copilot can help inspect the case and current Agent configuration. Ask for the smallest pattern-level fix rather than a rule that only repeats the exact case wording.

AI draft feedback does not publish an Agent change

Dismissing or refining an AI draft can improve the next draft for that conversation. It does not replace updating and verifying the Agent when the underlying behavior should change for future users.

Step 5: Change and verify

If the situation belongs in the supported scope:

  1. Open AI agents and update only the relevant instruction, source, or tool.
  2. Use Apply to save the draft change.
  3. Run the new case in Test Suite.
  4. Run related must-pass cases.
  5. Run broader regression cases when the change may affect behavior that already passed.
  6. Publish the new version only after the release conditions are met.

If the situation does not belong in the current scope, keep the approved fallback and leave the case for a later version.

Step 6: Monitor the result

After publishing, return to Conversations and look for the same pattern in new traffic. A passing case proves the checked behavior under the case conditions; production review shows whether the real situation is recurring or changing.

Next steps