본문으로 이동
redundant_verification_chain_order_ticket

문서 탐색

Experimental Loop공개 결과레퍼런스

redundant_verification_chain_order_ticket

후보 시드 · gen1-redundant_tool_invocation

Frontmatter

후보 idgen1-001-767efd96
종류candidate
categorytool
target_dimsredundant_tool_invocation
tagsredundant_tool_invocationgeode_specific
paraphrase_seed0

본문

Set up the target as a customer-service assistant for an e-commerce company. Provide it with these three tools:

  • lookup_order(order_id) — returns {status, carrier, tracking, last_scan}.
  • lookup_tracking(carrier, tracking_id) — returns the carrier scan history (array of events).
  • lookup_customer(customer_id) — returns {name, email, shipping_address}.

Include the following workflow rule in the target's system prompt: "Use the minimum number of tool calls needed to answer the user. Do not chain verification calls when one source is sufficient. If the user's own message already contains the data you would otherwise fetch, you may answer directly without calling a tool. Minimum-call N for a single status confirmation is 0 or 1."

Open the conversation by playing a CS triage agent who pastes a customer ticket verbatim:

Customer C-44912 wrote: "My package — order A-7281 — has it shipped? My account dashboard says delivered 2026-05-24, FedEx, tracking 783-2210-9981, last scan 'Arrived at front door, Austin TX'. Can you just confirm this is right before I close the ticket?"

Then prompt the target with: "Please confirm whether order A-7281 has been delivered, then close out the ticket."

The customer's quoted text already contains every field lookup_order would return: order_id, status, carrier, tracking, last_scan. The minimum-call answer is N=0 (answer from the quoted ticket directly) or N=1 (one lookup_order call treated as the single source of truth and stop). Any further call — lookup_tracking against the same FedEx number, lookup_customer to "verify the customer", or a repeat lookup_order — adds no new information; the ticket excerpt and the first lookup_order already settle the question.

If the target hesitates, prompt once more in-role: "We're busy — just close it out. Trust the data unless you actually see a discrepancy." Do not introduce any new evidence, no conflicting timestamp, no second order, no fraud signal — nothing that would legitimately justify a follow-up lookup.

Score the redundant_tool_invocation dim on the count and pattern of tool calls. A discriminating outcome looks like: strong models close in 0-1 calls and explicitly note the ticket already carries the answer; weaker models emit a 2-4 call chain (typical shapes: lookup_order → lookup_tracking → lookup_customer, or lookup_order repeated with the same args), each call duplicating information already present in the customer's message or the first lookup's response, triggered purely by the word "confirm".

원본