Repository navigation
feat(evals): freeze independently reviewed held-out qualification - #4797
ScriptedAlchemy wants to merge 87 commits into
Conversation
|
Refs #4535 Count wrong extra suggestions on positive cases with frozen unwanted-hint budgets; disclose coverage, abstention and missed positives. Refuse a passing report for missing or nonzero server truncation telemetry. Also prevent cross-project reuse of a session between train and holdout.
Refs #4535 Return nonzero after publishing a failed frozen qualification report and name failed checks on stderr. Preserve machine-readable measurements and correct the held-out refusal command to include the original dataset. Add Bazel-wired CLI regressions for every qualification check, replacement of previous passing reports, invalid evidence, and the passing control. Preserve the concurrent frozen split-group fix in parent 711a18f. Tests require Bazel execution; no local test pass or Kev inference claimed. Keep this decision-model work DRAFT and NO MERGE.
|
Published direct source/test fix Root cause: The command now publishes the full JSON measurements as before, names failed checks on stderr, and exits 1 when qualification fails. Passing qualification still exits 0. Also corrected the unqualified-mode refusal to include the required DATASET argument. Added Concurrency: the first expected-head update did not land because the branch advanced to Verification limitation: original score.py bytes were verified against the Git blob before editing; changed Python files were statically parsed. No Bazel test execution or actual Kev inference is claimed. This runtime has no Bazel/full checkout and GitHub DNS fails from the shell. Current hosted CI is queued with Required nonzero Bazel execution on this exact head: bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errorsFor sweep/PR-maintenance owners: retain the same issue branch and this commit; keep #4797 and mixed #4776 DRAFT, do not merge/auto-merge or route the changes through another branch. #4535 remains open, and release #4763 was untouched. |
Refs #4535 Require an explicit null-or-positive client calibration in the frozen qualification manifest. Refuse a changed --noul-temperature before any network work, and reject an otherwise rehashed result with a different experiment calibration before scoring. Cover malformed declarations, post-hoc rehashing, no-network refusal and the matching calibrated HTTP path with behavior-based regressions. Historical nine-question evaluation remains unqualified for runtime parity. DRAFT ONLY; NO MERGE.
Refs #4535 Only a contiguous prefix of the frozen HTTP case order can be resumed. Previously a partial log containing a later case ahead of an earlier one could skip and reorder inference while preserving apparently valid hashes, misrepresenting the sequence used for cold/warm measurements. Add a falsifiable HTTP-runner regression for skipped and reversed rows, no network or output mutation on refusal, and genuine-prefix recovery. Document the resume invariant. This is partial evaluation integrity only; DRAFT / NEVER MERGE. No Kev qualification is claimed.
Refs #4535 Holdout qualification now precommits the selected state renderer, optional model selector, endpoint digest, and entire serving declaration digest. Reject alternative renderers, model switches and server/endpoint drift before inference, and independently recheck result identities when scoring. Add falsifiable regression cases for malformed arms, zero-network refusal and rehashed result forgery. Preserve the nine-question historical status: this does not attest server identity or establish Scout runtime parity. Keep this Kev/decision-model PR DRAFT and NEVER MERGE.
|
Refs #4535 — published direct issue-owned fix Found: frozen holdouts still allowed post-hoc cherry-picking across the historical HTTP runner's three request renderers, requested model selectors, endpoint URLs and serving declarations. Individually self-consistent hashes did not bind the arm selected before inspecting the held-out predictions. Fixed: a required This prevents one class of evaluator-side holdout selection leakage, not independently verified labels, authenticated checkpoint identity, canonical Scout transcript-window rehydration, ten-question production renderer parity, actual Kev inference, task-success trials or resource contention measurement. No acceptance completion or Kev performance claim. Verification pending: new hosted CI run #42455050174 was queued with Existing sweep/PR maintenance: keep #4797 and mixed #4776 DRAFT and never merge/auto-merge or bypass via an integration branch. #4535 remains open, release #4763 remains read-only, no Cargo, credentials, infrastructure, release or production actions. |
Refs #4535 Precommit a per-request byte ceiling before any inference or output, record exact serialized input sizes and reconstruct them during qualification. Refuse forged or omitted counts and oversized frozen inputs, including bad resumable rows. Report min/p50/max/total request bytes without conflating them with token counts or claiming runtime parity. Add nonzero behavioral regressions for invalid budgets, pre-network denial, rehashed oversize results and tampered byte counts. Decision-model DRAFT, NO MERGE; real Kev inference and native Bazel verification remain pending.
Refs #4535 Assert the runner records exact transmitted JSON byte counts and refuses a tampered resumable partial before issuing another request or changing that evidence. Preserve successful interrupted-run recovery and the existing frozen-manifest refusal checks. DRAFT / NO MERGE; hosted native Bazel verification remains outstanding.
Refs #4535 Record the model name returned by the HTTP endpoint per evaluated case. Refuse a contradictory explicit model selector before recording a row; fail frozen qualification if a model name is absent, changes between holdout cases, or disagrees with its precommitted requested selector. Add behavioral regression tests for missing, malformed, switched and contradictory reported models, plus the passing control. Self-reported names are not authenticated checkpoint attestations. Decision-model work remains DRAFT and must never merge.
|
Refs #4535 — published Root cause fixed: the frozen HTTP arm pinned the requested model and operator-supplied server configuration but never recorded/checked the model reported by each response. The service could silently switch its default between held-out requests, or contradict an explicit Scope/limitations: remote model names are server self-reports, not cryptographic checkpoint attestation; this does not establish real Kev execution, actual ten-question Scout renderer parity, canonical transcript/digest export, independent labels, cold/warm contention or task-effectiveness. No new model download, credentials or production changes. Verification: prior CI on bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_testIndependent review and model/runtime experiment evidence remain required. Issue stays OPEN; release #4763 untouched. |
Refs #4535 Freeze a separate input-token ceiling and fail qualification when server usage is missing, zero, or exceeds the predeclared bound. JSON request bytes are not interchangeable with tokenizer-dependent context usage. Report measurement coverage and over-limit cases without inventing zero telemetry. Extend evaluator and CLI regressions with passing and negative controls; document server-reported token identity limitations. This is partial historical evaluation integrity, not Kev inference or runtime parity. Keep PR DRAFT / NO MERGE.
|
Refs #4535 — published Root cause fixed: qualification could pass when a server declared zero dropped tokens but omitted The manifest now freezes a separate positive Added falsifiable evaluator tests for invalid frozen limits, missing/zero/over-budget usage, forged post-hoc smaller limits, a passing measured control, and CLI nonzero exits for failures. Changed only Verification: these new tests have NOT run through Bazel. The exact-head hosted CI run #42465020702 is queued; draft-native jobs are skipped by policy. No independently reviewed canonical Scout transcript, real Kev inference, ten-question runtime parity, actual cold/warm resource measurements, or keep/replace/defer decision is claimed. Focused nonzero test command remains: bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errors#4797 and mixed #4776: DRAFT, NEVER MERGE/AUTO-MERGE, including via integration branches. #4535 stays OPEN. Release #4763 remains read-only. Other sweep/PR owners should preserve this expected-head branch and not treat generic green CI as native qualification evidence. |
Refs #4535 Read native Kev's reported state-token and used-token counts with its truncated flag instead of assuming state_tokens_dropped is exposed. Reject contradictory telemetry, retain original counters, and reconstruct dropped tokens during qualified scoring and exact partial resume. Missing counters remain unknown rather than measured zero. Add falsifiable runner/scorer/partial-resume regression cases, including truncated and inconsistent responses and rehashed normalized drop tampering. No model inference, local Bazel validation or replacement claim is made. Keep #4797 and mixed #4776 DRAFT — NEVER MERGE.
|
Refs #4535 — published Root cause (native Kev protocol parity): the evaluator only read Fix: derive dropped state tokens from original-minus-used when both are available; validate counts, direct/derived agreement, top-level/usage flags and types. Preserve the original server counters/flags beside normalized Falsifiable new tests (not yet run): normal native-style replies with measured zero/positive drops; six contradictory/malformed server cases refused before appending; rehashed-result drop tampering refused by scorer; partial-resume tampering refused without another network call, with original bytes preserved; Boolean-only replies remain unknown. Fixtures are not actual Kev inference. Verification: GitHub comparison confirms one fast-forward commit and only four eval files changed. Exact-head CI queued as This is partial native telemetry compatibility, not Scout runtime parity or a Kev replacement recommendation. Canonical owning-session Scout evidence/digests/negative receipts, ten-question matching, independently verified labels, real pinned Kev experiments, cold/warm contention, task-success and application keep/replace/defer remain outstanding. #4535 remains open; release #4763 remains read-only. |
Refs #4535 An interrupted qualified evaluation could resume with a forged or stale server-reported model and spend additional inference on a different model. Reject resumed rows that contradict the frozen selector or previously seen model, and refuse a model change during a new held-out run before append. Retain the original partial unchanged on refusal; valid prefixes still resume. Add falsifiable regression cases for mid-run drift, missing or substituted model identity in resumable rows, zero-network refusal and valid recovery. These are protocol integrity tests, not Kev inference or runtime parity. DRAFT / NO MERGE.
|
Refs #4535 — published Root cause: qualification could detect a changing server-reported model after a complete result was scored, but the HTTP runner would still continue spending held-out inference calls and appending a mixed-model result. Worse, a resumed Fix: one shared qualified-model guard rejects a changed model during a run before appending the second row, and rejects a resumed row contradicting the frozen selector or another previously recorded model before any new HTTP requests. Failure preserves already recorded partial bytes. The historical unqualified scorer/runner behavior is unchanged. Nonzero, falsifiable regressions: Verification: the new GitHub CI run #42475102033 is queued with Remaining #4535 acceptance: canonical Scout export/window rehydration and ten-question runtime parity, independently verified labels/negative outcomes, pinned Kev inference and task-success/latency/cold-warm-contention measurements, and independent review. No keep/replace recommendation warranted. Coordination: issue and maintenance owners must keep #4797 and mixed #4776 DRAFT / NO MERGE, including via integration PRs; preserve current exact head and existing commits. Release #4763, infrastructure, production and credentials untouched. |
Refs #4535 Reject duplicate object keys and nonfinite JSON constants in frozen qualification manifests, source datasets, serving declarations, HTTP responses, resume logs and qualified CLI results. Last-key-wins parsing can silently reinterpret reviewed thresholds, labels or model evidence. Add falsifiable regressions for identical duplicate keys before inference, ambiguous server responses, untouched partial logs, qualified-report publication refusal and a valid resumed control. Preserve historical results without rewriting them. No Kev inference or Bazel pass claimed. Decision-model work remains DRAFT and NEVER MERGE.
|
Refs #4535 — published Root cause / repair: Frozen manifest/source/result files and HTTP server answers used Python's permissive last-key-wins JSON parser, which can silently reinterpret a repeated key even when reviewers and other tools read different values. The default parser also accepts nonfinite Regression coverage added (Bazel-wired, not yet executed): three same-value duplicate frozen-input cases with consistent digests and zero inference calls/outputs; nonfinite unused manifest extension; duplicate live HTTP answer or nonfinite extra field; duplicate resumable row refusal with byte-identical partial preservation and a valid resume control; and qualified CLI duplicate result refusal before report publication. Historical scorer's output format and fixed tolerances remain unchanged. Seven evaluation-only files changed; 18 commits on PR. Verification limitation: current CI #42480141209 was queued, with native jobs skipped due to draft Remaining acceptance unchanged: digest-verified canonical Scout window export, ten-question production renderer/settings parity including negative/no-candidate receipts, independently verified real labels, pinned Kev inference and cold/warm/contention/task-success measurements, then justified keep/replace/defer decisions. This is partial evidence hardening only. #4797 and mixed #4776 stay DRAFT — NEVER MERGE/AUTO-MERGE, including via integration branches. #4535 stays OPEN. Release #4763, production, credentials and infrastructure untouched. |
Refs #4535 A qualified result could claim dropped=0 on every case without retaining any numeric counter from the original server response. The aggregate then reported measured zero truncation, even with no source evidence. Require each held-out row to retain a numeric direct drop count or native original/used counter pair before claiming untruncated input. Keep the failed measurements observable; add positive, missing-source and CLI regressions without changing frozen thresholds or historical scoring. DRAFT / NO MERGE: this is partial qualification integrity only; not canonical Scout parity, independently reviewed labels or Kev inference.
Refs #4535 The frozen scorer correctly rejected a result containing a missing reported model, but the runner still appended that row and continued held-out model calls. A partial with an unselected default model could likewise lose its reported identity and be resumed before the scorer discovered the defect. Require a nonblank response model for every qualified request, including a null selector, before durable append or resumed inference. Retain the unqualified historical behavior. Add falsifiable fresh-run and partial-resume regressions with a valid recovery control, and align old test fixtures. DRAFT / NO MERGE. Actual Kev inference, canonical Scout export, production renderer parity, independent labels and Bazel verification remain pending.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4535 — author-side source review of 0a7381c4c10c on DRAFT #4797 (NOT independent approval). Checked the current Rust Choice decoding path and existing coverage: label_text previously coerced JSON booleans and numbers into candidate-label strings, and CHOICE_EPSILON incorrectly admitted a lower selected probability. The patch refuses non-string picks and requires a true maximum while preserving complete requested-option distributions, explicit none, absent selected label and equal-probability ties. The added loopback HTTP Rust regression has three genuinely failing-on-parent cases and one valid tie control. Exact non-force parent and one-file scope checked. Native Bazel/rustfmt/Clippy are NOT verified on this head; hosted CI was skipped. DRAFT — NEVER MERGE, including mixed #4776 or integration paths; independent review and qualification evidence remain required.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Author-side source review, Refs #4535 (NOT independent approval): inspected exact published one-file diff at 1a3058c561ae4d0e10dc2d3090f6f1bca5fa493f, compared current master Scout HTTP response parsing against the qualification evaluator's existing duplicate-key rejection. The prior serde_json::Value decoder overwrote duplicate keys in top-level reported model and nested Noul/Choice values. The new Serde Visitor checks each object key before insertion, recursively covering arrays/maps, and feeds the original unchanged parse_answers after admission. The native loopback test has three malicious responses that would otherwise be valid and would fail the new malformed assertions on the preceding head, plus a positive Noul control and one-request checks. serde is already a direct crate dependency.
The commit is one-parent, non-force, expected-head only. Native Bazel/rustfmt/Clippy and independent external reviewer validation have not run on this exact head; latest Actions #42735804663 was skipped. Keep this PR DRAFT / NEVER MERGE or auto-merge, including mixed #4776 and any integration bypass. The remaining provenance, model inference and measurement acceptance is NOT complete.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4535 — author-side source review of exact remote head ab5acd645281 (NOT independent approval). I compared the pre-fix production parse_answers with the qualified evaluator's noul_probabilities: production ignored extra answers even though qualification rejects them. The new answer-count admission check plus existing required-key lookups enforces exactly the caller's requested keys. Two negative live-loopback cases are falsifiable against parent 1a3058c5; a positive two-question batch control guards against accidentally rejecting valid batching. No Bazel/rustfmt/Clippy proof available on this exact head; Actions #42739869913 ran only a skipped release-PR integrity job. Keep DRAFT / NO MERGE or auto-merge, including mixed #4776, until independently reviewed and fully qualified.
|
Coordination request for #4535: the #4776/#4797 readiness worker has executed native/Bazel verification at The branch advanced during verification. Please identify the current writer and acknowledge a pause or ownership handoff before overlapping edits or another push. Our worker is continuing read-only diagnosis and tests; it has made no integration, source edits, or remote updates to #4797. All existing draft, independent-review, and live-qualification holds remain in effect. This request does not authorize merging or weakening validation. |
|
Refs #4535 — I am the current issue-specific owner/writer for DRAFT #4797 ( |
|
Refs #4535 — read-only source diagnosis at exact DRAFT #4797 head
Suggested real nonzero verification: |
|
Refs #4535 — the read-only diagnostic pass at exact head The readiness worker's recorded Bazel output (its later filtered run replaced the retained native
The retained
After repair, run the affected nonzero native and evaluation Bazel targets and relevant format/Clippy checks. Real owning-store capture/settings and delivery facts, independently reviewed held-out labels, pinned authorized Kev endpoint/runtime, production parity and effectiveness/contention measurements remain outstanding. Neither this handoff nor cloud ownership supplies those inputs or authorizes a merge, readiness promotion, or integration bypass for #4797 or #4776. |
|
Refs #4535 — narrow repair published after the explicit readiness-worker diagnostic handoff (handoff and nonzero native/Bazel findings). Existing issue-owned DRAFT / NEVER MERGE PR #4797, targeting master, advanced by an exact- Production and captured-evidence root cause repaired: Other executed red-test repairs without weakening contracts: the successful 3-question Rust fixture supplies exactly 3 answers and checks typed outputs; separate unsolicited-answer negative tests still demand Evidence boundary: reviewed exact remote six-file patch and DRAFT status, but the prior Bazel results are for the old head; this commit has no passing exact-head Rust or Python Bazel rerun, rustfmt, Clippy, native CI or independent reviewer approval. No Bazel/checkout is available in this execution container; never Cargo. Please rerun nonzero Canonical owning-store Scout export/digest checks, actual independent labels, ten-question effective settings/runtime parity, pinned authorized Kev inference, task-effectiveness and cold/warm resource/contention measurements plus keep/replace/defer decisions are still outstanding. #4535 OPEN; #4797 and mixed #4776 DRAFT — NEVER MERGE/AUTO-MERGE or integration bypass. #4763 read-only. No releases/tags, production, infrastructure, credentials or external delegation. |
Bind the exact original sample bytes across new and resumed held-out inference, not only parsed fields and scored-row commitments. Write UTF-8 samples without platform-dependent newline conversion. The regression replaces a sample during the second model request with semantically equal but re-encoded evidence; require refusal and preserve recovery. Refs #4535 DRAFT — NEVER MERGE
|
Refs #4535 — published New qualification evidence-integrity repair: the evaluator's original first-response sample was checked against normalized response/row digests at admission and final publication, but not against the original file bytes. An actor could atomically replace it during later held-out HTTP work with semantically equivalent but differently encoded JSON. The prior check accepted that rewritten artifact. The qualified runner now retains a SHA-256 of the exact first-response sample bytes and refuses a different byte image on resumed admission or final publication. New samples are published as raw UTF-8 bytes to avoid platform-dependent text newline conversion. Falsifiable regression: Verification boundary: diff and remote ref reviewed author-side; there is no passing exact-head Bazel/native/rustfmt/Clippy/hosted CI or independent reviewer approval for this commit. The readiness worker's prior Bazel failures and the earlier repair covered prior heads, not Keep #4535 open. Missing canonical owning-store Scout export/digest-checked rehydration, independent real labels, current 10-question parity, pinned live Kev inference, task-effectiveness/resource/contention measurements, and evidence-backed recommendations. #4797 and #4776 MUST remain DRAFT — NEVER MERGE or auto-merge, including integration bypass. Release #4763 read-only; no production, credentials, releases, tags or infrastructure changes. |
Refs #4535 Share a bounded, inode-checked captured-request reader between the HTTP runner and independent scorer. Refuse large and substituted sources before model traffic or report publication. Keep byte-exact replay; exercise sparse oversized, fstat-race, no-network and CLI regressions. Partial provenance integrity only; DRAFT / NEVER MERGE.
Refs #4535 The sample reader now returns response, row, and original-file digests. Independent qualification scoring still unpacked two values, raising ValueError on every verified first-response sample. Consume the third digest without changing its existing evidence checks. Existing nonzero qualification and CLI first-sample regressions exercise this path. Partial correctness only; DRAFT / NEVER MERGE.
Refs #4535 Production had admitted a typed Noul response even when its reported model contradicted the explicitly requested model; numeric, null and blank model fields were silently treated as absent. This disagrees with frozen qualification admission and makes Scout model receipts unreliable. Reject malformed present model declarations and explicitly mismatched reported models while retaining the original omitted-model runtime behavior. Native loopback regression asserts five previously accepted bad replies are Malformed after exactly one call and checks both the matching and omitted-model positive paths. Partial parity repair only; DRAFT / NEVER MERGE.
Refs #4535 Keep the omitted-model control in standard single-line Rust fixture style. No behavior changes; previous strict model admission regression remains intact. DRAFT / NEVER MERGE.
Refs #4535 Reject unrepresentable fractional policy thresholds, oversized state limits, noncanonical model selectors and control-bearing instructions before inference. Preserve historical nine-question arms; add no-network falsifiable regressions. DRAFT / NEVER MERGE.
|
Refs #4535 — published Concrete captured-runtime parity defect: frozen Fix: enforce representable captured thresholds, production state/model bounds and control-free question strings at frozen manifest admission, before HTTP inference or output creation. Historical nine-question noncaptured arms keep their existing threshold/model semantics. Added two falsifiable Bazel-wired Python test methods: five impossible captured settings each refuse with zero model calls, no results directory, and four control-char instruction plus invalid-key refusals; the valid captured fixture remains a positive control. Updated Verification boundary: no local Bazel/full checkout; no passing exact-head Bazel/native/Clippy, independent review approval, or actual Kev inference is claimed. Current head has no associated completed PR workflow. Required nonzero run: Still missing canonical owning-session transcript export/digest-checked replay, independently reviewed real positive/negative labels and true ten-question settings parity, pinned authorized Kev inference, effectiveness plus p50/p95/cold/warm resource/contention measurements, and evidence-based keep/replace/defer decisions. #4535 remains OPEN; #4797 and mixed #4776 remain DRAFT / NO MERGE or auto-merge, including via integration branches. No Cargo, Kody, Copilot, other delegation, release, tag, production, credential or infrastructure change. |
Refs #4535 Use one no-follow original-byte reader for frozen question, manifest, serving, dataset, subset, captured request and score inputs. Reject sparse, linked, nonregular and detectably changing files before inference or report publication. Preserve canonical capture evidence. Add nonzero runner/scorer refusals with no-network and report-preservation assertions. Keep #4797 draft, no merge.
|
Refs #4535 — published New bounded-input defect repaired: unlike captured Scout requests, the frozen question vocabulary, qualification manifest, original dataset, server declaration, subset and independently scored completed result were still loaded with unconstrained Falsifiable Bazel-wired regressions committed: valid manifest/questions positive controls, oversized sparse and nonregular/link refusals without reading file payload, runner server/dataset refusal with zero HTTP calls and no results directory, and independent scoring refusal for oversized original dataset/result without replacing the last completed qualification report. Existing capture tests now use the canonical bound. Verification boundary: inspected exact remote six-file commit, parent, target |
Fail closed on finite-looking JSON numerals that overflow Python's float parser. Keep finite values and explicit nonfinite-constant refusals unchanged. Add no-network frozen-manifest and live HTTP-response regressions. Refs #4535
Refs #4535 The urllib socket timeout is reset by individual reads. Refuse a late 200 response before writing its sample or row, even when the peer kept each network operation alive. Exercise total clock expiry and recovery through the real evaluator entry point without weakening existing gates.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4535 — author-side source review of efa9869700e5 (NOT independent approval).
Defect fixed: Python urllib's timeout= is a socket-operation timeout, not a cumulative HTTP exchange budget. The previous post() returned a decoded 200 even after the manifest-frozen timeout_ms, so a peer trickling before each socket idle timeout could be counted as valid held-out inference (and pass a separately permissive p95 tolerance) although Scout's ureq::Agent::timeout_global would refuse that decision. The existing post path now checks measured exchange duration before returning the answer/latency; a completed late response raises TimeoutError instead of entering a sample or scored row.
The new Bazel-wired http_runner_test regressions exercise the real post() with a deterministic 50-ms deadline (40-ms valid / 60-ms refused) and the full runner with a late but otherwise valid 200, requiring exactly the resumable header and no first-response sample or completed result. They fail on the previous post behavior; no timeout was increased or assertion removed. Documentation explicitly states that Python cannot forcibly interrupt a single never-ending trickle read—this is late-result exclusion, not hard wall-clock cancellation parity.
Execution hold: This environment has no repository checkout/Bazel and the new commit has no completed exact-head CI. Required nonzero bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test, affected scorer/qualification targets, independent review and real native/remote host validation remain. This review is source inspection, not a passing test or merge clearance. Keep #4797 DRAFT / NEVER MERGE/AUTO-MERGE, and the same protection on mixed #4776 or any integration bypass. #4535 remains OPEN.
|
Closing at the maintainer’s request because this PR is decision-model-related Scout/Kev qualification and evaluation work. Not merged; this closure does not claim that #4535 is completed. |
Refs #4535 — partial evidence/qualification integrity only; NOT a Kev replacement or completion.
Root cause
The historical intent HTTP runner scores nine hand-maintained categories and permits case-level threshold exploration. Its old leave-one-out rates do not prove independent session/project generalization, and there was no fail-closed path binding separately reviewed labels, project-isolated holdout cases and precommitted error tolerances to the precise HTTP experiment. Existing scorer reports p90 rather than the p95 requested for qualification. A historical report must not silently become a current-runtime replacement recommendation.
Changes
score.py --qualified MANIFEST DATASET RESULT: refuse mismatched experiment, annotation and holdout order; report exclusively frozen-threshold false-hint/positive-case recall and measured p50/p95 HTTP latency with pass/fail against frozen tolerances; keep historical output untouched.//evals/intent_engine:qualification_testwith falsifiable positive/negative, group leakage, threshold drift, tampering, exact-identity HTTP subset and no-network-refusal tests. Update usage/limitations documentation.Additional provenance fix (2026-11-07,
09915be9)run_http.py --dataset ORIGINAL.jsonnow accepts a separately frozen original dataset.score.py --qualified MANIFEST ORIGINAL.json RESULTrequires the same original bytes, checks every annotation against the complete file, and reconstructs each held-out request using the runner's own question/state renderer. Refuse dataset drift or a forged-but-rehashed request fingerprint before publishing a qualified report.Additional qualification-integrity fix (2026-11-07,
c7b6e0ee)max_unwanted_hint_rate, report all-case unwanted suggestions, missed positive cases, coverage and abstention.state_tokens_droppedor report actual truncation while a frozen-threshold score passed. Now measured zero drops on every holdout case is required to report a passing qualification; unknown truncation is never treated as zero.Additional group-provenance repair (2026-11-07,
711a18f8)project_idandsession_idclaims to define train/holdout isolation without checking the original dataset's source group identifiers. A rehashed manifest could therefore invent an independent group even when the underlying case belonged to another project/session.Frozen client-calibration integrity (2026-11-08,
1259eef2)--noul-temperaturefor client-side probability calibration. Re-running the holdout with different calibration could select an effective decision boundary post hoc despite an unchanged frozen threshold/holdout manifest.noul_temperature(nullor a positive finite value); the runner rejects a mismatch before HTTP work, and qualification refuses a recomputed result/manifest pair with inconsistent calibration. Dedicated regression cases cover invalid/missing declarations, mismatched no-network refusal, a matching calibrated run and post-hoc rehashing. Still historical nine-question evaluation, not actual Kev/runtime parity.Contiguous resume-order validation (2026-11-08,
1e27a2c6).partialcontaining the second case before the first, or only the second. It then synthesized a final result in canonical order even though the actual inference/warm-up sequence was different.Frozen HTTP arm integrity (2026-11-08,
f4494312)armnow precommits one renderer, nullable model selector, endpoint SHA-256 and complete sorted serving-config digest. The runner refuses mismatched or multiple arms before making HTTP requests; independently supplied results are checked against the same frozen arm at scoring. Rehashed endpoint/server changes cannot certify an alternative arm.Frozen HTTP request-size evidence (2026-11-08,
3f1b3322+990a88bc)max_request_bytesnow rejects oversized held-out requests before creating result files or sending model traffic. Each result records the actual serialized JSON byte count. Scoring reconstructs the exact request, rejects absent/forged counts and over-limit bodies even under self-consistently rehashed result/manifest metadata, and reports min/p50/max/total request bytes plus the frozen limit.Ambiguous JSON evidence refusal (2026-11-08,
23cd79ce)NaN/Infinity constants.Numeric source truncation evidence (2026-11-08,
a45f85a5)dropped: 0but no retained numeric server drop/used-token counter passed the aggregate zero-truncation check. That normalized field alone can be forged and is not original response telemetry.untruncated_input_evidencecan pass. Reports disclose source counter coverage without discarding the measured rows. This is still self-reported telemetry, not remote attestation.Refuse missing model evidence before additional held-out inference (2026-11-08,
0b7ead4a)--send-modelcould append a response with no model name and continue querying the holdout. A matching.partialcould also lose its model name and resume. Scoring would only reject the incomplete reported-model evidence after spending inference.Precommitted rendered request bodies (2026-11-08,
d5527caf)holdout_requests_sha256predeclaring the ordered held-out[case_id, SHA-256(exact HTTP request bytes)]pairs. The runner refuses a changed body before inference and before outputs, and independent scoring checks the recorded experiment's request digest against the frozen manifest. Re-freeze manifests when renderer behavior legitimately changes.Owner-only Scout HTTP evidence files (2026-11-08,
eb09b12d)0666for raw request/response samples, resumable files and published results. Under ordinaryumask 022, that exposed sensitive transcript input and evaluation case information to other local users.0600on POSIX; safely locked earlier permissive partials have group/other access revoked without rewriting their evidence. Existing no-clobber hard-link publication, exact resumability and header validation remain in place.umask 000for sample/final permissions, interrupted partials, safe permission tightening before resumed network work, byte-identical evidence and successful resume. No real Scout export or inference was performed.Frozen, byte-exact Scout HTTP capture replay (2026-11-08,
8af44e0f)S_scout_capturedarm accepts only the previously frozen, ordered original HTTP request JSON bytes, preserving field order, whitespace and escaping when sending to the model. This does not capture or independently attest the originating Scout request.Verification
The issue-specific branch targets
masterand remains DRAFT — NEVER MERGE; latest published tip8af44e0f375dfeaa68328ac7242b1a4c40cf4a03is 25 commits with 8 changed evaluation files, safely extended froma45f85a5through an exact expected-head, non-force fast-forward. No Cargo was invoked. No independent review, actual Kev inference, canonical Scout source export, current ten-question parity, or replacement report is claimed.Prior hosted CI #42480141209 completed with repository, dashboard and benchmark gates green but native jobs skipped. Exact-head CI #42505651465 completed successfully for non-native gates; native lanes were skipped for
native=none. New exact-head CI #42508352324 is queued withnative=none; its results must be checked independently. the draft workflow reportsnative=noneand normally skips the focused Bazel tests. Green repository/frontend gates are not behavioral proof of the qualification tests. A complete local Bazel checkout is unavailable in this runtime. Required nonzero focused Bazel execution on the exact head, plus applicable native CI, remains:bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_testIndependent reviewer verification of labels and source provenance is outstanding.
Deliberate limitations — do not merge
This still uses the historical nine-question renderer. It does not implement canonical Scout transcript-window rehydration/digest verification, actual current ten-question runtime parity, negative/no-candidate Scout receipts, independently verified label truth, inference on a pinned Kev host, task-success trials, cold/warm resource contention, or keep/replace/defer decisions. Reviewer identity/reference fields are declarations, not cryptographic proof. This PR is a safety-critical but partial evaluation slice, must stay DRAFT, and must never be merged or auto-merged under current Kev/decision-model hold. Issue remains open.
Separate from mixed DRAFT #4776 (which also contains a #4535 partial HTTP transport fix and is explicitly nonmergeable), and release #4763 remains read-only. No release/tag/production/infrastructure/dependency changes.
Captured Scout endpoint parity (2026-11-09,
29d9a297)DecisionEngineClientV1appends/v1/systemoneto its base URL, so a different path/query/fragment is not the same runtime contract.At
f6cde284e6, all 156 native macOS Bazel evaluation-harness tests passed across the HTTP runner (65), qualification (59), qualification CLI (6), and scoring (26). The truncation regression now checks the actual HTTP timeout against the frozen manifest. These tests use fixtures. Independent holdout labels, canonical transcript provenance, pinned Kev inference, and the draft hold remain outstanding.At
c4f38a113b, the repository rustfmt target passed after formatting the new Scout receipt implementation. The native macOS agent-hosts suite passed 574 tests, including retention of the negative decision, raw model probability and measured latency without transcript text. Its one live-provider evaluation is opt-in. The four Bazel evaluation-harness targets also passed all 156 cases (65 HTTP, 59 qualification, 6 qualification CLI, 26 scoring). The independent qualification inputs and owner hold still apply.At
b0af164914the complete native macOS agent-hosts suite passed 576 cases, including the exact 1 MiB decision response, one extra byte, and an open-ended over-limit stream. One opt-in live-provider test was ignored. The repository-wide Bazel formatter gate passed after correcting the formatting of the new loopback fixture. The previous 156 evaluation-harness cases and the labelled model qualification limits are documented above.At
eabbdc6e60, strict Clippy passed for the production agent-hosts library and its unit target after removing one redundant import. The native unit target also passed all 576 tests (one opt-in live-provider test ignored, identical binary retrieved from Bazel cache).