BenchmarksReference

GEO visibility

A stage-aware F/R/C/P/A/Q/O measurement contract with frozen workloads, digest-bound receipts, and no aggregate GEO score.

On this page

What it verifies
Empty cells and comparators
Execution contract
Data and artifact joins
Current boundary
Evidence basis

What it verifies

GEO is a stage-aware evidence state machine. A diagnostic moves through preflight → live_observe; only a preregistered treatment claim continues to experiment. An earlier phase cannot prove a later one. An absent measurement remainsnot_measured, never zero.

One query run is one observation. The live profile produces 6 roots × (one original + three paraphrases) × K=5, or 120 observations. C uses all 120 as its denominator; R, P, A, and Q use only the observations or claims for which that stage's evidence exists.

Stage	Numerator: what is counted	Denominator: eligible population
`F`	URLs passing each preflight check	audited target URLs
`R`	runs whose retrieval list contains a target URL	24×K runs exposing a retrieval list
`C`	runs citing a target URL	all 24×K query runs
`P`	target-cited runs with visible_rank ≤ 3	target-cited runs
`A`	runs where a verifier confirms target use	target-cited runs with an A verdict
`Q`	supported target-linked claims	verifier-declared target-linked claims audited in full
`O`	observed referrals, engagement, or conversions	eligible first-party impressions, referrals, or sessions

Empty cells and comparators

These values describe a local diagnostic receipt from 2026-08-24; they carry diagnostic authority only. 0is an observed result; not_measured means no eligible denominator or receipt exists, and partial means only a submetric of that stage was measured.

State	What the gap means	Required comparator
`F · partial`	All 78 local URLs and 577/577 internal links passed, but the previous runner discarded the deployed 77/78 sitemap failure as an exception. The current runner preserves it as a partial receipt.	Identity check: local export ↔ public host with the same URL digest
`R/C/P`	Pages measured R 0/120, C 4/120, and P 4/4. R records zero retrievals across 120 observations.	Surface diagnostic: Pages ↔ GitHub repository (R 109/120, C 9/120)
`A/Q`	A was 4/4 and Q was 43/58; an earlier same-model repeat returned 35/54. Q remains partial because it covers claim support only.	Verdict calibration: independent verifier ↔ human sample over a frozen claim set
`O · not_measured`	No completed Search Console, referral, or conversion window exists, so no eligible denominator was created.	Outcome comparison: baseline ↔ treatment with the same window, queries, and engine
`Promotion · none`	This run is diagnostic; no comparator or intervention arm was preregistered.	Promotion comparison: frozen baseline ↔ treatment under the same index, budget, and window

The GitHub repository is a surface diagnostic comparatorused to locate an authority-transfer boundary. A content-effect claim requires a separate causal comparator: frozen baseline and treatment arms.

Execution contract

Freeze one root query and three paraphrases for each of six roots: 24 exact strings.
run-spec.json freezes the workload SHA-256 and model; live runs require a separate operator approval receipt for the same surface and exactly K=5.
The native result binds the run-spec digest plus adapter, provider, credential source, and model.
Every cell in the 24×K matrix resolves to one native receipt; JSON Pointers for search activation, retrieval, and citations must match the source bytes.
Absorption and quality remain unmeasured without a separate verifier receipt, producer/version/model, and digest-bound rubric.
Q is computed only when the verifier-declared target-linked claim universe matches the audited rows.
Two invalid claim or source-verifier responses produce a digest-bound failure receipt and leave only that observation's A/Q unmeasured; transport failure still stops the run.
Each Q support verdict retains the claim text and an exact quote present in the source receipt.
Failed public-host checks persist as partial receipts; O is a late analytics overlay that never rewrites native results.
v1 measures only claim support within Q, so the broader Q stage remains partial even with complete receipts.

uv run python scripts/eval/geo_collect.py \
  --run-spec <run-dir>/run-spec.json \
  --workload <run-dir>/workload.json \
  --site-preflight <run-dir>/site-preflight.json \
  --link-audit <run-dir>/link-audit.json \
  --host-preflight <run-dir>/host-preflight.json \
  --out <run-dir>/native-results.json

uv run python scripts/eval/geo_verify.py \
  --workload <run-dir>/workload.json \
  --native-results <run-dir>/native-results.json \
  --rubric <run-dir>/verifier-rubric.json \
  --adapter <verifier-adapter> \
  --model <verifier-model> \
  --effort medium \
  --claim-adapter <claim-extractor-adapter> \
  --claim-model <claim-extractor-model> \
  --claim-effort low \
  --producer-version <version-or-revision> \
  --out <run-dir>/verifier-results.json

uv run python scripts/eval/geo_visibility.py \
  --run-spec <run-dir>/run-spec.json \
  --workload <run-dir>/workload.json \
  --native-results <run-dir>/native-results.json \
  --verifier-results <run-dir>/verifier-results.json \
  --outcome <run-dir>/outcome.json \
  --out <run-dir>/geo-vector.json

Data and artifact joins

The vector does not copy trajectories or raw receipts. Oneattempts.jsonl row joins each file by relative path and SHA-256; analysis.json recomputes ratios from the measurement's numerator and denominator JSON Pointers.

run-spec	attempt kind	Authority
`native_results`	`native-result`	provider-native outcome
`measurement_results`	`measurement`	geode.geo-vector@1
`verifier_receipts`	`verifier-receipt`	independent A/Q judgement
`outcome_receipts`	`outcome-receipt`	completed first-party analytics
`trajectory`	`trajectory`	geode.trajectory@1 / release manifest

A trajectory is behavior evidence, not score authority. The publication manifest classifies every declared file as public or withheld, while the bundle gate checks schemas, digests, run ID, and trajectory-release scope.

uv run python scripts/eval/contract.py validate-run-bundle \
  <run-dir>/run-spec.json

Current boundary

Local export, sitemap, self-canonical, noindex, and internal-link evidence leaves F partial. It becomes measuredonly after a public-host receipt binds the same URL set and passes HTTP, HTML, canonical, and robots checks. A/Q need a separate source-aware verifier; O needs a completed first-party analytics window.

The current implementation preserves failed host preflights, verifies each Q claim against an exact source quote, and joins O after the immutable native result. Remaining gaps now represent missing deployment, independent-judgement, or first-party analytics evidence.

Native outcome, verifier judgement, GEODE trajectory, analysis, and publication manifest retain separate authority. Slash runs keep their native format; Inspect-produced records retain .eval ownership.

Slash typed state is an advisory workflow projection. Model-authored numerators and locators carry workflow authority only. Benchmark authority begins with a schema- and digest-validated native/vector/ verifier/outcome bundle.

Evidence basis

Moved from the GEODE repository’s documentation site. Source commit de077a4 · View the source on GitHub