Skip to content

Advanced Tester and Tool-Step Standards

Use this guide only when a release decision depends on how the Agent used a tool, not only on the final answer.

Start with the normal Evaluations and Improvements workflow. Move here when the reviewer must verify a required tool, argument, or result.

Configure a custom Tester

In Tester management, give the Tester a narrow purpose, choose an evaluation model, and write its system prompt. The prompt can use {input}, {output}, {tool_steps}, {expected_output}, and {rubric}.

Include {tool_steps} only when tool behavior is part of the judgment. Existing Testers do not receive tool steps unless their prompt uses this placeholder.

Inspect tool steps when the failure depends on tool use

Some failures are not only about the final wording. The agent may have searched the wrong knowledge source, used a weak query, skipped a required tool, or ignored the result it retrieved.

In the case detail page, open Tool Steps above the AI response. Expand a step to inspect:

  • which tool was called
  • what arguments were sent
  • what result came back

Use this evidence when deciding whether the fix belongs in Instructions, a tool's When to Use, or the underlying Knowledge Base.

If you create an evaluator that should judge tool behavior, include {tool_steps} in the evaluator system prompt. Existing evaluators do not use tool steps unless their prompt explicitly asks for this placeholder.

Use the canonical tool type in the standard so the evaluator can match the tool step reliably.

English name 中文名稱 Canonical tool type
Search Knowledge Base 搜尋知識庫 consultant_retrieve_context_objs
List Knowledge Base Files 列出知識庫檔案 consultant_list_kb_files
Read Knowledge Base File Lines 讀取知識庫檔案行數 consultant_get_context_obj_lines
Search Web 搜尋網頁 consultant_search_web
Fetch Web Page Content 讀取網頁內容 consultant_fetch_web_content
Call Agent 呼叫其他 Agent consultant_call_agent
Request Form 要求填寫表單 consultant_request_form
Payment 建立付款 consultant_payment
Memory 記住使用者資訊 consultant_memory
HTTP Request 呼叫 HTTP API consultant_http_request
Generate Image 產生圖片 consultant_generate_image

Example standard for searching the Knowledge Base:

Required tool:
- Search Knowledge Base (`consultant_retrieve_context_objs`)

Parameter requirements:
- The question should ask about refund policy, refund conditions, or 退費政策.
- The keywords should include refund-related terms.

Result requirements:
- The result should contain refund-related policy information.

Final answer requirements:
- The response should be grounded in the retrieved policy result.

Example standard for finding and reading a specific Knowledge Base file:

Required tools:
- List Knowledge Base Files (`consultant_list_kb_files`)
- Read Knowledge Base File Lines (`consultant_get_context_obj_lines`)

Parameter requirements:
- `list_kb_files.pattern` should target the expected file name or path.
- `get_context_obj_lines.knowledge_node_id` should come from the file list result.
- The requested line range should be narrow enough to inspect the relevant section.

Result requirements:
- The file result should contain the policy, procedure, or source section needed to answer.

Final answer requirements:
- The response should use the file content, not only general knowledge.

Example standard for web research:

Required tools:
- Search Web (`consultant_search_web`)
- Fetch Web Page Content (`consultant_fetch_web_content`) when a search result must be opened before answering

Parameter requirements:
- The search question and keywords should match the user's requested topic.
- The fetched URL should come from a relevant search result.

Result requirements:
- The fetched content should contain evidence for the final answer.

Final answer requirements:
- The response should not claim facts that are missing from the fetched content.

Example standard for routing to another agent:

Required tool:
- Call Agent (`consultant_call_agent`)

Parameter requirements:
- `agent_name` should match the specialist agent responsible for the request.
- `query` should include the user's key need and enough context for the specialist agent.

Final answer requirements:
- The response should use the specialist agent's result or clearly explain the handoff outcome.

Before using the result for release

  • The Tester prompt includes only the placeholders it actually needs.
  • Standards use canonical tool types, not only display labels.
  • A reviewer manually checks a small result sample before trusting the judge.
  • Representative cases are rerun after changing the judge model or prompt.