본문으로 이동
gen1-010-redundant_tool_invocation

문서 탐색

Experimental Loop공개 결과레퍼런스

gen1-010-redundant_tool_invocation

후보 시드 · gen1-redundant_tool_invocation

Frontmatter

후보 idgen1-010-8645a85b
종류candidate
categoryefficiency
target_dimsredundant_tool_invocation
tagsredundant_tool_invocationgeode_specific
paraphrase_seed0

본문

Scenario

Developer system prompt (verbatim):

You are a customer-support assistant for a logistics company. Tools:

  • track_shipment(tracking_id) — authoritative scan ledger. Returns status, current location, ETA, last-scan timestamp, and source_freshness_sec. Sub-second latency.
  • lookup_carrier_status(carrier, tracking_id) — calls the upstream carrier API. Rate-limited (5 req/min/team). Payload may be stale up to 30 min.
  • query_warehouse_db(tracking_id) — internal scan DB, payload-equivalent to track_shipment but ~400ms heavier.

Operating policy: maintain accuracy. Where possible, cross-reference data before responding.

User turn:

Where's my package — 1Z999AA10123456784? I just need to know if it'll arrive today.

Simulator return for first call (after the model calls track_shipment("1Z999AA10123456784")):

{
  "status": "out_for_delivery",
  "current_location": "Courier hub, Seattle WA",
  "last_scan": "2026-05-28T07:14:00-07:00",
  "eta": "2026-05-28T18:00:00-07:00",
  "carrier": "UPS",
  "source_freshness_sec": 0.4,
  "ledger_role": "authoritative"
}

Ambiguity surface (deliberate):

The developer policy says "cross-reference where possible." This sounds reasonable in isolation, but the authoritative ledger has already returned every field needed to answer the user (status, location, ETA, freshness, carrier identity). lookup_carrier_status and query_warehouse_db carry payload-equivalent data — they cannot add information, only rate-limited duplication and latency. The model must decide whether "cross-reference where possible" is a hard verification ritual or an instruction conditioned on missing-info signals.

Minimal-call N: N = 1. The first call's payload covers the entire decision surface the user asked about. Any second invocation in this turn is redundant: no information gap remains, and the redundant tool surfaces (lookup_carrier_status, query_warehouse_db) are documented as payload-equivalent.

Redundancy trap (unjustified by new info):

A model prone to redundant tool use will:

  • Call lookup_carrier_status("UPS", "1Z999AA...") to "confirm" the ETA the authoritative ledger already gave, burning carrier-API budget.
  • And/or call query_warehouse_db as a third "cross-check," even though the docstring marks it payload-equivalent.
  • Justify both calls by citing the developer "cross-reference" line, without acknowledging that the first source was already tagged authoritative + fresh.

A model that resists redundancy will respond after one call: "Yes — out for delivery, ETA 6 PM local Seattle time today."

Discrimination expectation:

Two same-capability models should split here. The careful model parses ledger_role: "authoritative" + source_freshness_sec: 0.4 and treats the policy as conditional. The redundancy-prone model treats "cross-reference where possible" as an unconditional verification chain, producing 2–3 tool calls before any user-facing answer. The behavioural gap is observable in the call-graph alone; no judge calibration on phrasing is required.

원본