Tau2

Conversational policy execution

Read three execution profiles separately

τ-bench leaderboard ↗

Suite-native

0.820

228 / 278 · pass¹ · tau2==1.0.0

Historical k=1

GEODE-user

200 / 278

0.7194 · dual-runtime

Separate diagnostic profile

Runtime-faithful

99 missing

quota-contaminated work

No aggregate authority

Evidence authority path

1Task + user route2Native reward3Trajectory behavior4Attempt validity5Published claim

Reward explains success, trajectory explains behavior, and the attempt manifest explains run validity. Each record keeps its own authority.

Measurement record evolution

2026-07-03Native scoreresults.json owns task rewards and the headline2026-07-31Trajectory @1Published exact turn/call/result joins and orphan counts2026-08-03Full cycleMeasured all 278 tasks under a distinct dual-runtime profile2026-08-04Attempt lineageRecorded retry, quota, and selection; classified contaminated rows as missing work2026-08-14Frozen preflightChecked task, route, and budget identity before model calls

The official τ-bench surface separates domains, Standard·Custom·Legacy, pass^k, and a run visualizer. GEODE likewise compares only rows with matching profile identity and keeps the rest in diagnostic lineage.

tau2-bench is a conversational tool-use benchmark: the agent talks to a simulated user while performing DB actions across the airline, retail, and telecom domains, and a verifier scores each task by required-action coverage. GEODE participates through the public adapter in evals/benchmarks, and every published number is pinned to the harness revision, model route, and effort that produced it. Compare only against reruns with the same settings.

2026-08-04 Runtime-Faithful Execution Contract

The current adapter shares one process-owned RuntimeEventBus, all 13 public hook registrations, and all four trusted middleware join points across every Tau2 ToolExecutor and AgenticLoop. Tau2 remains the only environment-tool executor. GEODE's projection ACK stays deferred, and the native ToolMessage.idlater closes the original call ID with its sole completion or error, including a terminal-step result recovered from the native receipt.

Native results.json remains score authority. Its digest, reward, task/trial, and native/runtime termination are recorded as verification.evidence before SessionEnd. New snapshot v4 admission also verifies a runtime profile containing revision, assembled prompt/tool-schema digests and actually exercised surfaces, plus an attempt manifest covering every retry, participant session, and final selection. It independently verifies the normalized trajectory's digest bindings and recomputes scope_complete=true, so an orphaned tool call cannot be promoted. tau2-native-user and geode-dual-runtime profiles are never pooled. A diagnostic auto-resume labels prior-process rows as resumed_native_unattested.

The 2026-08-04 full-cycle attempt reached all 278 scheduled tasks, but subscription quota exhaustion contaminated 99 rows: 2 Airline, 16 Retail, and 81 Telecom. They are missing work, so this attempt has no aggregate score authority. Six pre-quota Telecom calls also exposed an external-yield ordering defect; the runtime now returns those proposals before post-tool convergence guards, and admission rejects the captured scope-incomplete trajectory. A clean rerun is required before a new headline is published.

The privacy-reviewed diagnostic report and three-domain companions are pinned to geode-eval-artifacts@40be847. The 12-file manifest SHA-256 is 40206ed1…317; this bundle carries invalidation-evidence authority only.

2026-08-03 GPT-5.4 Subscription Base Full Cycle

At GEODE 22789ee2, the complete Airline, Retail, and Telecom base scope ran through gpt-5.4 subscription at effort high for both the agent and geode_user. The aggregate is 200/278 = 0.7194.

• Airline 42/50, Retail 79/114, and Telecom 79/114.
• Telecom splits into service 28/29, mobile data 30/36, and MMS 21/49. Fourteen tasks reached MAX_STEPS, with a 957.65-second p95.
• The SQLite exact join covers 556 final parent sessions and 51,985 events; all 3,964 tool call/result pairs are complete, with zero orphans.
• geode-eval-artifacts/trajectories/tau2-geode-gpt54-22789ee2-geode-user-airline-retail-telecom-base-full-20260803T091257Z-13162f7bcff9: the privacy-reviewed three-domain geode.trajectory@1 release.

This row belongs to a separate dual-runtime profile. Tau2 results.json remains score authority; the trajectory is an external-loop diagnostic sidecar. Seven transport retries created 14 additional SQLite sessions outside the final trajectory parents, so the public release explicitly remains replay_complete=false. The isolated Tau2 loop also runs without a HookSystem and records zero public hook_events; the separate 13-hook / four-middleware E2E remains hook-dispatch authority.

2026-08-03 v1.0.12 Post-Release Smoke

We reran the fixed mock and Telecom-small tasks against the public GEODE v1.0.12 release (f99cea63) through the same GPT-5.4 subscription route at effort high. Both scored 0/1; neither failure was retried or removed.

• Mock ended with USER_STOP after 13.75 seconds. Communication scored 1.0, while the DB and required action scored 0.0.
• Telecom reached MAX_STEPS after 236.73 seconds and 50 steps. All 14 repeated diagnostic tool calls have one result, but the run ended before native component scoring.
• geode-eval-artifacts/trajectories/tau2-geode-gpt54-v1.0.12-f99cea63-geode-user-mock-telecom-small-20260803T104819Z-fd524ce7a3cb: 234 events, 16 exact tool pairs, manifest SHA-256 fd524ce7a3cb…2288.

These post-release route smokes leave the 278-task full cycle unchanged. There was no route, authentication, or provider-adapter failure. The retained failures are directPostVerify evidence that task success still requires the native verifier.

2026-08-02 GPT-5.4 Subscription Cycle

At GEODE afaab52b, both the agent and geode_user ran through the gpt-5.4 subscription route at effort high. mock/create_task_1 scored 0/1; the first Telecom-small task scored 1/1. Both ended normally with USER_STOP and no route, provider, adapter, or quota exception.

• Mock: create_task included the unrequested optional description="", so the exact action and DB comparators failed.
• Telecom: the DB, toggle_roaming, mobile-data, and excellent-speed checks all passed.
• geode-eval-artifacts/trajectories/tau2-geode-gpt54-afaab52b-mock-telecom-small-20260801T173245Z-2dc79cb569f0: two trajectories, 158 canonical events, ten exact tool pairs, and zero missing IDs or orphaned pairs.

Tau2 results.json is the score authority. These two fixed tasks belong to a diagnostic profile; the trajectories are correlation/replay sidecars. The immutable snapshots preserve the runner-default stage=trainlabel, but promotion_authority=none grants no training or promotion authority.

2026-07-31 v1.0.11 Release Diagnostics

With the released GEODE v1.0.11 (686ff372), both the agent and simulated user ran through gpt-5.6-sol subscription at effort high. mock/create_task_1 scored 0/1; the first Telecom-small task scored 1/1. Both ended normally with USER_STOP, with no provider, quota, or adapter exception.

• Mock: as before, create_task added the unrequested optional description="", so the native exact action and DB comparators failed.
• Telecom: the earlier premature human transfer is gone. toggle_roaming, DB match, mobile-data status, and excellent-speed assertions all scored 1.0.
• geode-eval-artifacts/trajectories/tau2-geode-gpt56-v1.0.11-686ff372-mock-telecom-small-20260731T105713Z-a71155f7006c: two geode.trajectory@1 artifacts with 142 events and nine exact tool pairs.

Tau2 results.json remains the score authority. The Crucible snapshots keep both runs diagnostic with promotion_authority=none; the GEODE trajectory is a digest-joined replay sidecar.

Headline: Native User-Simulator Track

2026-07-03/04 run, GEODE v0.99.269, sierra-research/tau2-bench@1901a30 (tau2==1.0.0), agent gpt-5.2 PAYG at effort high, native user_simulator gpt-4.1-2025-04-14 at effort medium, max_steps=200.

The current weak spot is required-action coverage under compound tasks: Retail failures often miss DB/write side effects, while Telecom failures cluster around MMS/APN/app-permission/roaming combinations where one necessary action is omitted.

Run Records

Every run records the measured time, model, provider, source, effort, route, harness revision, and artifact path with one schema.

Airline + Retail + Telecom base full-cycle GPT-5.4 diagnostic2026-08-03 KSTgpt-5.4subscriptionagent high / user high

• Airline 0.8400 (42 / 50)
• Retail 0.6930 (79 / 114)
• Telecom 0.6930 (79 / 114)
• 51,985 canonical events / 3,964 exact tool pairs / zero orphans
• Telecom p95 957.65s / 14 max-step terminations / MMS 21 of 49

# Run once per domain with num-tasks 50 (airline) or 114 (retail/telecom).
python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain <airline|retail|telecom> \
  --task-split-name base \
  --num-tasks <50|114> \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --max-errors 1 \
  --max-retries <0|1> \
  --timeout 3600 \
  --model gpt-5.4 \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 600 \
  --user geode_user \
  --user-llm gpt-5.4 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 180 \
  --trajectory-stage benchmark \
  --save-to <domain-specific-run-id>

• This GEODE-user full cycle is not comparable to the native tau2 user_simulator headline matrix.
• Tau2 results.json is score authority; the trajectory release is a privacy-reviewed diagnostic and external-loop sidecar.
• Seven Telecom transport retries created 14 extra SQLite sessions outside the final trajectory parents; no behavior-score failure was retried.
• The released trajectories are scope-complete for final task attempts and replay-incomplete for bounded bodies and retry-attempt lineage.
• The isolated Tau2 AgenticLoop records no public hook_events; the separate hook behavior E2E remains hook authority.

Telecom small first-task GPT-5.4 v1.0.12 post-release diagnostic2026-08-03 KSTgpt-5.4subscriptionagent high / user high

• Termination max_steps before native component scoring
• Duration 236.73s
• 203 canonical events / 14 exact tool pairs
• Repeated customer, line, network, usage, restriction, and VPN diagnostics

python scripts/eval/tau2_geode_agent.py   --harness-dir artifacts/eval/harnesses/tau2-bench   --domain telecom   --task-split-name small   --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]'   --num-tasks 1   --num-trials 1   --max-concurrency 1   --max-steps 50   --timeout 1800   --model gpt-5.4   --provider openai   --source subscription   --effort high   --time-budget-s 300   --user geode_user   --user-llm gpt-5.4   --user-provider openai   --user-source subscription   --user-effort high   --user-time-budget-s 180   --trajectory-stage benchmark   --save-to geode-gpt54-high-v1.0.12-f99cea63-geode-user-telecom-small-01-20260803

• The run reached 50 steps before native DB/action scoring; repeated diagnostics are preserved as behavior evidence.
• All fourteen tool calls have exactly one result, with no route, authentication, quota, or adapter failure.
• This two-task release smoke does not invalidate or replace the 200/278 full-cycle diagnostic.

mock/create_task_1 GPT-5.4 v1.0.12 post-release diagnostic2026-08-03 KSTgpt-5.4subscriptionagent high / user high

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 13.75s / 31 canonical events / 2 exact tool pairs

python scripts/eval/tau2_geode_agent.py   --harness-dir artifacts/eval/harnesses/tau2-bench   --domain mock   --task-ids create_task_1   --num-tasks 1   --num-trials 1   --max-concurrency 1   --max-steps 8   --timeout 900   --model gpt-5.4   --provider openai   --source subscription   --effort high   --time-budget-s 180   --user geode_user   --user-llm gpt-5.4   --user-provider openai   --user-source subscription   --user-effort high   --user-time-budget-s 120   --trajectory-stage benchmark   --save-to geode-gpt54-high-v1.0.12-f99cea63-geode-user-mock-smoke-20260803

• The simulated user stopped before a verifier-compatible state change; DB and action checks are zero while communication is one.
• The run has no authentication, quota, provider-adapter, or harness exception and is retained without retry.
• This release smoke is not a rerun or replacement of the 278-task full cycle and is not a native user_simulator leaderboard row.

Telecom small first-task GPT-5.4 subscription diagnostic2026-08-02 KSTgpt-5.4subscriptionagent high / user high

• DB check 1.0
• toggle_roaming write action 1.0
• Mobile-data and excellent-speed assertions 1.0
• Termination user_stop
• Duration 119.83s / 127 canonical events / 8 exact tool pairs

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name small \
  --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]' \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 50 \
  --timeout 1800 \
  --model gpt-5.4 \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 300 \
  --user geode_user \
  --user-llm gpt-5.4 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 180 \
  --save-to geode-gpt54-high-afaab52b-geode-user-telecom-small-01-20260802

• The DB, toggle_roaming, mobile-data, and excellent-speed checks all passed.
• No route, provider, adapter, quota, agent, or simulated-user exception occurred.
• Tau2 results.json is the score authority; the 127-event trajectory is a digest-joined correlation and replay sidecar.
• The immutable source snapshot retains the runner-default train stage with promotion_authority=none; future benchmark commands should set --trajectory-stage benchmark explicitly.

mock/create_task_1 GPT-5.4 subscription diagnostic2026-08-02 KSTgpt-5.4subscriptionagent high / user high

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 25.33s
• 31 canonical events / 2 exact tool pairs

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --task-ids create_task_1 \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.4 \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.4 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt54-high-afaab52b-geode-user-mock-smoke-20260802

• The model supplied unrequested description=""; Tau2's exact action and DB comparators rejected it.
• No route, provider, adapter, quota, agent, or simulated-user exception occurred.
• This fixed GEODE-user diagnostic is not a native user_simulator headline row.
• The immutable source snapshot retains the runner-default train stage with promotion_authority=none; future benchmark commands should set --trajectory-stage benchmark explicitly.

base aggregate native user_simulator2026-07-04 03:45 KSTgpt-5.2paygagent high / user medium

• Airline 0.8200 (41 / 50)
• Retail 0.7632 (87 / 114)
• Telecom 0.8772 (100 / 114)
• Native user simulator gpt-4.1-2025-04-14
• GEODE recorded gpt-5.2 PAYG usage locally; user simulator cost is visible through OpenAI billing, not GEODE's usage ledger.

# Aggregate of the three per-domain native tau2 runs listed above.
# Do not average this with mock smoke or GEODE geode_user rows.

• This weighted aggregate is for internal Agent-World-style comparison only.
• The run spec differs from OpenAI's official GPT-5.2 Tau2 headline, which used an internal research setup and excludes Airline.
• The run spec differs from the earlier GEODE geode_user smoke matrix.

Telecom small first-task GPT-5.6 v1.0.11 diagnostic2026-07-31 KSTgpt-5.6-solsubscriptionagent high / user high

• DB check 1.0
• toggle_roaming write action 1.0
• Mobile-data and excellent-speed assertions 1.0
• Termination user_stop
• Duration 78.52s / 117 canonical events / 8 exact tool pairs

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name small \
  --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]' \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 50 \
  --timeout 1800 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 300 \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 180 \
  --save-to geode-gpt56-sol-high-v1011-686ff372-geode-user-telecom-small-01-20260731

• The earlier premature human-transfer failure is closed for this fixed case.
• Tau2 native DB/action/assertion checks remain the score authority; the GEODE trajectory is a digest-joined replay sidecar.
• The Crucible v3 snapshot remains diagnostic with promotion_authority=none because no frozen experiment contract was supplied.

mock/create_task_1 GPT-5.6 v1.0.11 diagnostic2026-07-31 KSTgpt-5.6-solsubscriptionagent high / user high

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 9.03s
• 25 canonical events / 1 exactly paired tool call and result

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt56-sol-high-v1011-686ff372-geode-user-mock-smoke-20260731

• The failure reproduces the earlier behavior: create_task includes unrequested description="" and the native exact comparator rejects it.
• The run completed normally and is retained without retry or relabeling.
• This diagnostic has promotion_authority=none and is not a native user_simulator leaderboard row.

Telecom small first-task GPT-5.6 subscription diagnostic2026-07-31 KSTgpt-5.6-solsubscriptionagent high / user high

• Required user toggle_roaming action 0.0
• Mobile-data and excellent-speed assertions 0.0
• Termination user_stop after human transfer
• Duration 51.91s

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name small \
  --task-ids '[mobile_data_issue]user_abroad_roaming_enabled_off[PERSONA:None]' \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 50 \
  --timeout 1800 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --save-to geode-gpt56-sol-high-edb74602b-geode-user-telecom-small-01-20260731

• The agent correctly identified the customer, line, roaming state, and data usage.
• It then declared device tools unavailable and transferred to a human instead of guiding the user-side roaming/device workflow.
• No provider, quota, or adapter exception occurred; the failure is retained as behavior evidence.

mock/create_task_1 GPT-5.6 subscription diagnostic2026-07-31 KSTgpt-5.6-solsubscriptionagent high / user high

• Communication check 1.0 / DB check 0.0
• create_task action check 0.0
• Termination user_stop
• Duration 14.58s

python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.6-sol \
  --provider openai \
  --source subscription \
  --effort high \
  --user geode_user \
  --user-llm gpt-5.6-sol \
  --user-source subscription \
  --user-effort high \
  --save-to geode-gpt56-sol-high-edb74602b-geode-user-mock-smoke-20260731

• The create_task tool executed, but the model supplied an unrequested optional description="".
• Tau2's exact action and DB comparators rejected the extra argument; this is retained as a behavioral failure.
• This GEODE-owned user route is not comparable to the native tau2 user_simulator headline.

telecom/base native user_simulator2026-07-04 03:45 KSTgpt-5.2paygagent high / user medium

• DB match 31 / 114
• Write actions 471 / 496
• Generic actions 20 / 20
• Termination user_stop 114 / 114
• Duration total 28827.72s / avg 252.87s / max 818.58s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain telecom \
  --task-split-name base \
  --num-tasks 114 \
  --num-trials 1 \
  --max-concurrency 4 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-telecom-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• Concurrency was raised from 2 to 4 mid-run and resumed from tau2 checkpoints; no rate-limit, quota, or billing errors were observed.
• Failures clustered around multi-issue MMS/mobile-data/service cases where one required APN, permission, roaming, or data-refuel action was omitted.

retail/base native user_simulator2026-07-03 KSTgpt-5.2paygagent high / user medium

• DB match 88 / 113
• Read actions 320 / 354
• Write actions 140 / 174
• Termination user_stop 113 / 114, too_many_errors 1 / 114
• Duration total 23543.64s / avg 206.52s / max 873.92s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain retail \
  --task-split-name base \
  --num-tasks 114 \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-retail-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• The main failure mode was missing required side-effect actions even when the natural-language response looked plausible.
• One task terminated with too_many_errors; the remaining failures ended with user_stop but failed verifier assertions.

airline/base native user_simulator2026-07-03 KSTgpt-5.2paygagent high / user medium

• DB match 42 / 50
• Read actions 81 / 91
• Write actions 33 / 49
• Termination user_stop 50 / 50
• Duration total 14205.02s / avg 284.10s / max 979.65s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain airline \
  --task-split-name base \
  --num-tasks 50 \
  --num-trials 1 \
  --max-concurrency 2 \
  --max-steps 200 \
  --timeout 3600 \
  --model gpt-5.2 \
  --provider openai \
  --source payg \
  --effort high \
  --time-budget-s 600 \
  --user user_simulator \
  --user-llm gpt-4.1-2025-04-14 \
  --user-provider openai \
  --user-source payg \
  --user-effort medium \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-2-high-native-user-airline-base-20260703 \
  --log-level INFO \
  --auto-resume

• GEODE version at measurement: v0.99.269.
• This is the native tau2 user_simulator comparator track, not the GEODE geode_user smoke track.
• Airline is retained for internal trend comparison; OpenAI's GPT-5.2 announcement excludes Airline from its Tau2 headline due to lower-quality ground truth grading.

mock/create_task_1 smoke2026-07-03 KSTgpt-5.5subscriptionagent xhigh / user high

• DB check 1.0
• create_task action check 1.0
• Termination user_stop
• Duration 54.90s

uv run python scripts/eval/tau2_geode_agent.py \
  --harness-dir artifacts/eval/harnesses/tau2-bench \
  --domain mock \
  --num-tasks 1 \
  --num-trials 1 \
  --max-concurrency 1 \
  --max-steps 8 \
  --timeout 900 \
  --model gpt-5.5 \
  --provider openai \
  --source subscription \
  --effort xhigh \
  --time-budget-s 180 \
  --user geode_user \
  --user-llm gpt-5.5 \
  --user-provider openai \
  --user-source subscription \
  --user-effort high \
  --user-time-budget-s 120 \
  --save-to geode-gpt-5-5-xhigh-geode-user-mock-smoke-20260703-r5 \
  --log-level INFO \
  --verbose-logs

• This is a tau2 wiring/regression smoke, not a tau2 leaderboard score.
• Do not average it with native tau2 user_simulator runs using gpt-4.1 or gpt-5.2.
• Both assistant and simulated user used the GEODE subscription route.

Run Logs

The raw simulation JSONs (per-task rewards, action checks, and full conversation transcripts) are preserved as public copies with local paths and synthetic personal fields redacted in the geode-eval-artifacts repository.

• geode-eval-artifacts/tau2/simulations: simulation JSONs for GEODE-owned runs. The headline runs follow the geode-gpt-5-2-high-native-user-*-base-20260703/results.json pattern.

Artifact paths inside the run records are the local harness paths at measurement time; the published copies live under the repository path above, addressed by file name.