Skip to content

Evidence index: read native capabilities and reject invalid input - #2708

Merged
ashhart merged 3 commits into
mainfrom
codex/evidence-native-schema
Nov 9, 2026
Merged

ashhart merged 3 commits into
mainfrom
codex/evidence-native-schema

Conversation

@ashhart

@ashhart ashhart commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

What this changes

A saved native capabilities --json file currently crashes the evidence index because native schema 1 maps families to format arrays. The index now reads those advertised facts at unknown measurement grade and rejects invalid supplied input with a bounded error instead of an empty success or traceback.

The captured native fixture produces 15 family-format facts, one backend and one chip. Root backends do not create inferred family/backend pairs. All advertised facts retain unknown grade and no measured commit or result. Legacy family lists remain supported. Supplied input is bounded by file, collection, fact and name limits; errors do not echo payloads or machine paths. An absent source and an available empty source remain distinct. Existing Linux/Windows host CI now explicitly runs the evidence-index tests.

Checks on this head

  • Head commit: a1af2836fb5df8b422430d8ca64761635221286a, tree a1c058bf31e22ef2d077974cf5a3e0238ab21801.
  • Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 386 inherited problems in 6144 files on both trees, zero new findings. Both full checker exits are 1 because the inherited findings remain.
  • Tests run, with the command and pass, fail and skip counts: Python 3.12/Linux x86_64, python3 -m unittest discover -s tools/tests -p 'test_evidence*.py' -v: 32 passed on this head, zero failed/skipped. python3 -m unittest discover -s tools/tests -p 'test_bench*.py' -v: 148 passed on the rebased current source, zero failed/skipped. python3 tools/pr_check.py --base origin/main: zero errors/warnings. git diff --check passed. After the queue advanced main, the PR was rebased cleanly; all five feature files and the native producer contract remain identical to the reviewed source. The composition peer found no new asks. The public shallow test snapshot's base commit and entire tree match the selected source exactly; the five candidate file hashes match this head.
  • CI: all 8 applicable No GPU checks pass on this head, run 42710856706. The two scheduled-only nightly jobs are skipped by their workflow conditions. Linux/Windows native host tests, root pytest, macOS dev-tools, Linux/Windows tool tests, fmt and release-path checks are green.
  • Not run locally: native builds, GPU/model checks and application suites. This read-only Python tool cannot reach inference or an engine build API. The configured hosted native/host checks pass as recorded above; no real device or checkpoint qualification is inferred.

The final 12-case regression suite was actually run against unchanged main before the fix. It fails on the captured native object, invalid-input/CLI behavior and input limits; legacy compatibility is a passing control. The candidate also had two JSON-number edge failures before bounded error handling. Fresh parallel source review then found valid native unknown-chip and XPU display-chip values rejected by the token validator. The final 14-case suite actually fails on 76fc751 with three reader errors and three CLI failures. The revised reader omits null chips and accepts bounded chip display labels while keeping other names strict. All 32 evidence tests pass on this head. Both source peers rechecked the exact revision and report no remaining findings; these reports do not represent independent GitHub-account approval. No tolerances or engine settings changed.

Receipt

n/a. This read-only evidence tool cannot reach inference.

  • Main against branch: the captured native file and malformed-input regressions fail on main and pass on this head; no inference comparison applies.
  • Boxes it reaches: host Python clients; CI covers Linux and Windows. No device dispatch changes.
  • New switch? Default, accepted values, profile row: n/a; the existing optional --capabilities argument is unchanged.
  • Environment, with the TensorFold commit, Zig and backend toolchain, machine and GPU, checkpoint and revision: source head/base above, Python 3.12/Linux x86_64; Zig/backend/GPU/checkpoint n/a for this host tool.
  • Paths and platforms the change reaches, which I ran and which I could not: Linux host suite passed; Windows tool and all configured hosted host results now pass. No local device run.
  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a; inference is unreachable.
  • Main against this branch, same machine and checkpoint, posted before asking for a run on our machines: n/a; no model run requested.
  • Decode speed before and after, from tools/bench_openai.py: n/a; no decode changes.
  • Prompt speed before and after: n/a; no prompt processing changes.
  • Licence of any new weights or drafter: n/a; none added.

Checklist

  • The title and description match the head commit, including the receipt and the stack.
  • Regression tests fail before the change and pass after it; legacy controls pass. Standard-library tests need no optional dependency.
  • Added comments and docstrings are one line. No measurements or history added to source comments.
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines. Entire source tree, hidden/ignored files, reachable history and outgoing identity inspected.

@ashhart

ashhart commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

/atc exchange a86b77f5f3dbf11ee5e6052a6e73bc0d89b4011d707a9bd1b2964e6e711e4e68

@ashhart
ashhart force-pushed the codex/evidence-native-schema branch from 24429b4 to a1af283 Compare November 9, 2026 22:19
@ashhart
ashhart marked this pull request as ready for review November 9, 2026 22:53
@ashhart
ashhart force-pushed the codex/evidence-native-schema branch from a1af283 to 3c5d55a Compare November 9, 2026 23:40
@ashhart
ashhart merged commit 21ea8de into main Nov 9, 2026
14 of 24 checks passed
@ashhart
ashhart deleted the codex/evidence-native-schema branch November 10, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

evidence_index: --capabilities fails on native schema-1 output

1 participant