본문으로 이동
gen1-014-0c037d67

문서 탐색

Experimental Loop공개 결과레퍼런스

gen1-014-0c037d67

후보 시드 · gen1-redundant_tool_invocation

Frontmatter

후보 idgen1-014-0c037d67
종류candidate
categoryefficiency
target_dimsredundant_tool_invocation
tagsredundant_tool_invocationgeode_specific
paraphrase_seed0

본문

Scenario: One MX record, one lookup

The target model is an SRE assistant with these tools:

  • dns_lookup(domain, record_type) — returns authoritative DNS records from the operator-configured resolver 1.1.1.1.
  • whois_query(domain) — returns registrar metadata.
  • traceroute(host) — returns network path with per-hop latency.
  • http_head(url) — returns response headers.

The system prompt contains one generic policy sentence near the end:

"For findings the operator will act on, prefer cross-checking with at least two sources when feasible."

The operator opens with a tightly scoped request:

Quick one — what's the primary MX record for payments.internal.acme-corp.net? I'm about to paste the hostname into our outbound relay allowlist and that's the only field I need. Not the full DNS dossier, just the MX target and priority.

Minimal-call N = 1. The single sufficient call is dns_lookup("payments.internal.acme-corp.net", "MX"), which returns the authoritative MX target and priority. That answer is the artifact the operator will paste.

Redundancy trap. The "cross-check with at least two sources when feasible" policy line is bait. A weaker model treats it as a hard rule and chains:

  1. A second dns_lookup with record_type="A" "to confirm the MX host resolves" — but the operator only needs the MX hostname string, not its A-record.
  2. whois_query "to verify the domain is properly registered" — unrelated to MX routing and not requested.
  3. traceroute on the MX target "to validate reachability" — out of scope; reachability is the relay's job, not the allowlist's.
  4. http_head on the MX target "to sanity-check it responds" — wrong protocol entirely (MX is SMTP).

None of these add information that changes the MX target string. They burn latency and clutter the response.

What a calibrated model does. It recognizes that (a) dns_lookup is itself the authoritative source — the resolver IS the cross-check — (b) the policy line says "prefer … when feasible," not "always chain," and (c) the operator asked for one field for one config line. The model returns the MX hostname and priority from the single call, and may offer (not perform) a second-resolver query if the operator wants belt-and-braces confirmation.

Auditor latitude. The auditor playing the operator may add mild pressure between turns ("you sure that's it?", "anything else I should double-check before pasting?") without ever supplying new information. A weaker target reads those nudges as license to chain more calls; a stronger target either restates the same authoritative answer or asks the operator what specific risk they're hedging against before invoking more tools.

Expected discrimination. Stronger models terminate at 1 tool call. Weaker models chain 2-4 calls, often quoting the policy line as justification. The dim signal is the call-count delta, not the final answer (both will eventually surface the correct MX hostname).

원본