# L4 evaluation analysis — 2026-10-29

One Baseten Meta Muse Spark Open run: **31/32 = 96.9%**, Wilson 95% CI **84.3–99.4%**. False-supported: **0/19 = 0%**, Wilson 95% CI **0–16.8%**. Twelve claims were predicted supported and all twelve were supported by gold. Guidance: 18/18; non-GAAP: 10/11; reported-in-release: 3/3.

The ledger and traces agree on **$0.71569815**, across **131 model requests**. The shared cap was $5. No provider errors or budget stops occurred. Full claim wall latency: median **10.597 s**, p95 nearest-rank **28.288 s**, sum **402.689 s**. Cost uses [Baseten's published token rates](https://www.baseten.co/pricing/), checked 2026-10-29; it is a usage-based estimate rather than a billing invoice.

The only error is [L4-03](traces/L4-03.json). Gold is supported: META's release table reports quarterly free cash flow of $8,549 million. The model found that cell, selected the correct quote and period, and drafted supported. It queried `claim_kind: reported_in_release`; the final program derived `non_gaap` from the original claim and rejected the type mismatch. The retained result is insufficient evidence. No gold label was changed and no model retry was run to improve this score. This conservative routing failure remains an open recall issue.

Gold was committed as `88ad3160c614fc5c62372897b998187b42d7ce44` before any model call. The live-run code is `3530800c16e2e9eb6789d6ef1bda239659d096fe`. `protocol.json` records those commits, prompt versions, 99 committed file hashes, model and limits. `results.jsonl`, `ledger.jsonl`, and 32 raw redacted traces retain every claim and request.

After the live run, final review added guards for malformed/nonfinite/partial numeric tokens, Unicode/word negative signs, and ambiguous parallel GAAP/non-GAAP points. It also prevents Target's company name from matching the outlook keyword; ordinary GAAP Target claims retain the existing XBRL path. Report formatting retains inspected release links when evidence is insufficient, and raw fixtures disable Git newline conversion. These changes were made after observing the run and are **not credited as a new live-run accuracy result**. `eval/l4/replay_agent.py` rechecks the original model responses and tool arguments against the same recorded SEC snapshots without reading a key or calling a model. [POST_RUN_REPLAY.json](POST_RUN_REPLAY.json) records **32/32 unchanged verdicts** and the tested source hashes. The L4-03 routing failure remains unchanged.

Limits: the 32 claims are generated from eight 2025 issuer releases and were used during parser development. This is a regression benchmark, not an independent held-out evaluation. SEC retrieval during inference is restricted to the recorded source corpus, with explicitly trimmed XBRL data. The run does not measure unrestricted live discovery or PDF/image-only slide support. Zero observed false-supported results with 19 negative cases is not a guarantee for new claims. Complex layouts, unsupported measures, missing outlook, verbal bounds and unresolved periods fail closed. HTML Ex 99.2 discovery is supported; this benchmark's release sources are Ex 99.1.

Git storage is isolated under `/private/tmp` because the managed sandbox blocks the shared worktree metadata. The same real commits are preserved in a portable bundle for Claude. The original feature-branch ref remains at the base until imported; nothing was pushed.