Skip to content

Feedback and Improve

This step starts after stakeholders have already tried the published experience and left rough feedback. The fastest operator loop is: review the thread in Histories, ask Copilot what happened, turn the risky reply into an eval case with Add Case, update the Agent, and run the saved case in Test Suite.

Step 1: Open the Stakeholder Thread in Histories

After stakeholders try the published Web channel, open Histories and find one thread worth improving:

  • A reply that felt safe but awkward
  • A reply that missed an important clarifying question
  • A reply that handed off too late
  • A message that already has stakeholder feedback attached

History thread with a real stakeholder feedback record attached to the pricing reply

Step 2: Open Copilot on the Thread and Ask What Went Wrong

On the history thread page, click the Copilot button in the bottom-right corner so the thread stays visible on the left. Then use the Copilot panel's Ask a question... box to ask what happened and what should change.

The stakeholder feedback does not need to be perfectly structured. In many teams, the stakeholder only points out what felt wrong and maybe why. The operator and Copilot do the diagnosis from there.

For example:

A stakeholder left this feedback on the pricing answer:
"The answer is safe, but it still sounds a bit generic. Make it shorter, state earlier that pricing comes from a human, and offer the human callback sooner."

Why did the Agent behave this way, and what instruction should I update in the Agent Editor?

Ask Copilot to help you answer questions like:

  • Which instruction caused the behavior
  • What part of the rule is too vague or too broad
  • How the instruction should be rewritten
  • What a better response would have sounded like
  • What a strong Standard should check later in Test Suite

History thread with the Copilot panel open while the operator asks about stakeholder feedback and reviews the analysis

Step 3: Use Add Case in Histories and Draft the Standard with Copilot

While the thread, feedback, and Copilot analysis are still visible, click Add Case on the risky reply.

Do not leave Histories yet. This is the best moment to turn the real failure into a reusable eval case.

Ask Copilot to help you write a short, checkable Standard for the case. For example:

Please turn this failure into 2 or 3 short `Standard` bullets for `Test Suite`.
They should be specific enough that another operator can score pass or fail without guessing.

In Case Detail:

  • Keep the attached conversation so the case includes the earlier context
  • Check the Input and make sure it matches the risky question you want to rerun
  • Update Ideal Response so it reflects the improved behavior you want
  • Use Copilot's suggestion to write a small Standard
  • Click Create

Case Detail opened from Histories after clicking Add Case on the risky pricing reply

Step 4: Return to Editor and Update the Agent

After the case is saved, go back to Edit Agents, open Consultation Desk, and update only the relevant part of Instructions.

When the change is ready:

  1. Click Apply.
  2. Review the updated instruction.
  3. Click Publish again so the web client uses the new version.

Consultation Desk in the editor, ready for instruction updates and republishing

Keep the Change Narrow

Small, targeted fixes are easier to verify. If you change too many things at once, it becomes harder to tell what actually improved.

Step 5: Run the Saved Case in Test Suite

Instead of bouncing back to Live Test, go straight to Test Suite and run the saved case against the updated version.

You are checking whether:

  • the case now passes, or at least moves clearly in the right direction
  • the Standard matches the behavior you actually want
  • the fix is narrow enough that you still understand why it worked

From this point on, the operating pattern is simple:

  1. Share the Agent with a few stakeholders
  2. Review rough feedback in Histories
  3. Ask Copilot what went wrong
  4. Save important threads as eval cases with Add Case
  5. Update the Agent in Editor
  6. Run those cases in Test Suite

This is how the Agent gets safer and more useful over time.

Test Suite showing the saved case and its latest result

Celebrate the First Improvement Loop

You have now completed the full beginner loop: publish an Agent, collect real stakeholder feedback, save one reusable eval case, improve the instructions, and validate the fix in Test Suite. That is the foundation for every later iteration.

When to Add More Structure

As the pilot grows:

Next Steps