Repository navigation
Description
Goal
Establish a reproducible Kev-4B baseline that can decide which existing TraceDecay decisions are worth replacing. Reuse the current SystemOne client and evaluation tools. The result should support a small number of measured replacements and identify the machinery each replacement would remove.
Current evidence
Source inspected at 90909cd9d0f2ced7be95d924ce574380303cef80:
- The existing decision client already supports Noul, Choice, Score, typed failure outcomes, bounded concurrency and deadlines. The intent pass batches its questions in one request.
- Runtime reference questions now contain ten intents, including
build_diagnosticsandatomic_edit. The checked-in benchmark has nine categories, includingrefactor_edit, and a different state renderer. The historical scores/thresholds do not qualify today's exact requests. - Plan 41 promises a digest-checked canonical-session exporter to Kev JSONL. No corresponding implementation was found in the inspected scripts/evals/CLI paths.
- Delivery analytics describe delivered candidates. No-candidate exits do not retain the same prediction/window receipt. Delivered-hint data alone misses negative decisions and missed opportunities.
- run_http.py and score.py already provide remote evaluation and useful classification/calibration metrics. Extend them instead of creating another harness.
The official Kev-4B card describes a model that scores supplied options without generating text. Its v1.0 checkpoint is 139fdd94, with temperature 2.41 and 8,192 validated context tokens; the larger serving limit is not equivalent to validated quality. Qualification must pin the actual checkpoint and serving configuration used.
Bounded work
- Freeze the actual runtime question vocabulary, instructions, renderer/settings identity, input limits and thresholds in an evaluation manifest. Reuse the existing canonical settings digest and production rendering contract; avoid a third hand-maintained question list.
- Implement the missing bounded export from existing Scout provenance and outcomes. Rehydrate through the owning transcript API, reproduce the original window bounds and verify its digest. Report missing, changed, pruned or expired evidence explicitly; do not silently approximate an old request or create a parallel transcript store.
- Keep predictions, observed action facts, delivery decisions, explicit feedback, acted/ignored behavior and independent correctness labels separate. Only verified labels may enter training data. Include independently labelled negatives, near neighbors, ambiguity and misleading instructions inside tool output.
- Split related examples by session/project, freeze held-out cases before choosing thresholds, and preserve the historical benchmark as historical evidence. Extend existing calibration/error/coverage metrics and the existing task-effectiveness harness where a replacement claims user benefit.
- Record checkpoint/server identity, backend/dtype/temperature, question/renderer/settings digests, input size, truncation/refusal, measured latency and available usage. Reject
.partialresumes from a different experiment identity; unknown costs remain unknown. - Run inference on a separately provisioned compatible host through the existing typed HTTP contract, with supported authenticated access where required. No model download or local execution in this cloud workspace is required.
- For candidate selection, evaluate option-order sensitivity and an explicit
noneoutcome. Before a Choice result can select a live candidate, tighten the existing Choice decoder around the caller's supplied option set and required probability contract; the currently used intent path is Noul-only.
Acceptance
Produce a reproducible baseline/report and a keep, replace or defer recommendation for each evaluated application. Freeze application-specific error tolerances before reading held-out results. Report task success, unwanted suggestions or missed work, coverage/abstention, p50/p95 latency, cold/warm resource costs and contention with the existing intent pass.
A positive replacement decision must identify the existing code/tests/calls that disappear. If the existing deterministic behavior is as effective and cheaper, retain it. Fine-tuning is a later option justified by measured residual errors and independently labelled data.
Relationship to existing work
This completes the concrete qualification/export gap in Plan 41, whose Phase 1 was merged in #4121. Reuse the completed outcome work in #3654, #3655, #4216 and #4219; this issue does not reopen their broad task-effectiveness scope. Application-specific labels, decisions and deletion targets belong in the linked exploration issues.
This issue is based on source inspection and primary model documentation. No new Kev inference, training or performance measurements were performed while filing it.
Application explorations
- explore(scout): replace generative hint rewriting with bounded Kev candidate selection #4536 — Scout candidate choice/abstention and rewrite-path removal.
- explore(automation): use Kev to diagnose skill work and avoid unnecessary writer calls #4538 — Skill-work diagnosis and unnecessary writer-call avoidance.
- explore(memory): use Kev to triage curation pages before general-agent review #4539 — Memory-page work triage and unnecessary review-call avoidance.
- explore(intent): separate observed actions from keyword inference before expanding Kev routing #4540 — Semantic intent arbitration and legacy hint consolidation.
Suggested order: establish this baseline, then prioritize Scout selection and the existing intent disagreement; evaluate background authoring/curation gates with separate missed-work tolerances.
Activity
fix(scout): validate choice distributions and picked labels
fix(scout): reject explicitly malformed answer type tags
fix(evals): publish complete HTTP results without partial files
fix(evals): preserve foreign files during link probes
fix(evals): atomically publish scored summaries
fix(evals): validate typed HTTP decisions before recording results
fix(evals): constrain HTTP decisions to the selected endpoint
fix(evals): reject malformed scoring evidence
fix(evals): reject inconsistent calibrated score evidence
fix(evals): verify HTTP score experiment fingerprints
docs(evals): correct provenance note and test newline bytes
fix(evals): serialize concurrent HTTP result resumes
feat(evals): support secure bearer authentication for HTTP qualification
perf(evals): avoid quadratic AUC and leave-one-out rescans
fix(evals): hold writer lock through completed publication
127 remaining items
fix(evals): consume complete qualified sample verification result
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published two focused, exact-parent, expected-head, non-force fast-forward commits to the existing DRAFT / NEVER MERGE #4797 targeting
master:4f90c1fc29dc: root cause was two unboundedsource.read()paths for frozen captured Scout evidence (HTTP inference runner and independent scorer). Both callers now use one original-byte, inode-checked loader with a cohort-derived upper bound and 64 MiB safety ceiling. Added falsifiable sparse-oversize, fstat/read-growth, zero-network/no-output and independent scoring CLI regressions; documented exact bytes and limits. Positive normal capture admission preserved.a2c3787ebbb4: fixed an actual qualified-sample scorer failure:validate_qualified_samplereturns three digest values butqualified_reportunpacked only two, raisingValueErrorwhenever first-response evidence was supplied. Updated the remaining caller. Existing positive sample and qualified CLI tests exercise this non-vacuously.Re-read current root AGENTS, current master Scout path, complete issue discussion, competing PRs and review/CI. Verified remote PR head
a2c3787eand DRAFT state. Verification limit: no Bazel-enabled checkout/native runner accessible; no executed exact-head Bazel Python/Rust tests, rustfmt/Clippy or independent approval is claimed. Exact-head hosted Release PR integrity was skipped, not qualification evidence. Author-side source review only.This is partial evidence admission and scorer correctness, not acceptance. Outstanding: canonical owning-store Scout transcript export and digest-checked rehydration incl negative decisions, genuinely independent real labels/holdouts, ten-question production settings/renderer parity, pinned actual Kev inference, task effectiveness, cold/warm p50/p95 resource and contention measurements, and per-application keep/replace/defer recommendations. #4535 stays OPEN. #4797 and mixed #4776 remain DRAFT — NEVER MERGE/AUTO-MERGE, including integration bypass. Release #4763 read-only; no Cargo, external delegation, releases/tags, secrets, production or infrastructure changes.
fix(scout): refuse contradictory reported model identities
style(scout): align model admission loopback fixture formatting
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published two exact-parent, expected-head, non-force fast-forwards to the existing issue-specific DRAFT / NEVER MERGE #4797 targeting master:
7dc3744f6796, followed by formatting-onlyda57b34fafcc. Re-read confirms remote headda57b34fafcc91f0b968cbc2736708f0f7f67600, 81 PR commits, open/DRAFT; mixed #4776 remains separately DRAFT / NO MERGE. Neither merge nor auto-merge is authorized, including integration bypasses.Concrete production/evaluator admission mismatch repaired:
DecisionEngineClientV1previously accepted an otherwise valid Noul decision even when the server'smodelcontradicted the explicitly requested selector. It also silently treated a present malformedmodel(null, Boolean, numeric, whitespace) as absent. The existing qualified HTTP evaluator refuses contradictory/missing reported model evidence, so runtime model receipts could describe a different identity than qualification. Production now rejects a malformed present model field and a contradictory named model. A genuinely omitted server model remains accepted as an unverified runtime response; no false server/checkpoint attestation was added.Falsifiable source regression: added native Rust loopback
reported_model_must_be_valid_and_match_explicit_selection, with one matching positive, five formerly accepted invalid responses that must now beMalformedafter exactly one HTTP exchange, and one omitted-model positive. No API introduction or compatibility shim; one production source/test file touched. Source/commit/fast-forward were verified, but independent review approval and execution of the nonzero Rust test remain outstanding. Latest exact-head Actions #42905457574 was only a skipped Release PR integrity workflow; no passing Bazel/native/rustfmt/Clippy was inferred. No Cargo.Not issue acceptance: canonical Scout owning-store export and digest-checked window rehydration; genuinely independently reviewed positives/negatives and frozen cohort; actual pinned Kev inference against a compatible authenticated HTTP host; current runtime question/settings, action-fact and delivery parity; p50/p95 effectiveness, cold/warm resource and contention; and per-application keep/replace/defer remain unproven. Keep #4535 OPEN and the dedicated owner ENABLED; #4797 and mixed #4776 DRAFT — NEVER MERGE/AUTO-MERGE, including indirect paths. Release #4763, production, credentials, infrastructure, tags and releases untouched; no external delegation.
fix(evals): reject impossible captured Scout runtime settings
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — actual source/test update published to existing DRAFT / NEVER MERGE #4797: commit
ccb21285b37b, exact-parent nonforce expected-head fast-forward. Confirmed 82 commits on open DRAFT PR, targeting master; onlyqualification.py,test_qualification.py, andHTTP_RUNNER.mdchanged. No competing issue-specific PR.Newly fixed false-parity admission: captured Scout qualification could freeze settings that production Rust never accepts/emits: non-integer-permille thresholds, state budgets >1 MiB, padded/control-bearing/oversized explicit model names, or control characters in question instructions. Admission now refuses these before HTTP inference or output. Two new Bazel-wired falsifiable tests exercise five invalid frozen-arm cases with zero HTTP calls/no result directory, instruction/key control-character rejection, and a valid control. Historical noncaptured threshold semantics remain intact; no legacy facade introduced.
No exact-head Bazel/native CI or independent approval is claimed in this runner (no Bazel/checkout; PR head has no completed CI run). Required nonzero Bazel evaluation targets and reviewer clearance remain. This is partial Refs, not acceptance: original owning-store window exporter/digest verification, verified real labels, ten-question effective settings parity, pinned actual Kev inference, task success and p50/p95 cold/warm resource/contention measurements plus per-application keep/replace/defer remain missing. Keep #4535 OPEN and the original owner enabled; #4797 and #4776 DRAFT / NEVER MERGE/AUTO-MERGE (including indirect integration). Release #4763 unchanged; no Cargo, Kody/Copilot/delegation, production, secrets or infrastructure.
fix(evals): bound all frozen qualification file inputs
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4535 — published
37ec2f0652feto existing DRAFT / NO MERGE #4797 with exact-parent/nonforce expected-head ref update. A new local-evidence admission gap was addressed: frozen manifest/questions, server configuration, dataset/subset and independent result scorer had unconstrained file reads despite the existing bounded Scout-capture reader. They now share one no-follow, size-bounded (64 MiB) held-inode reader, rejecting sparse, symlink/FIFO and detectably changing files. Duplicated captured-file reader was removed. Three new Bazel-wired behavioral tests cover valid controls, zero-network/no-output refusals, and preservation of a published scored report; existing capture tests migrated to the renamed canonical ceiling. Full source/review and verification caveats.Not a qualified Kev baseline: exact-head Bazel/nonzero native execution and independent review remain unverified; no compatible pinned Kev host, authenticated canonical transcript export, independent real labels, ten-question runtime settings parity, p50/p95 task/resource/contention evidence, or per-application keep/replace/defer result is established. #4535 remains OPEN, original issue owner remains active; #4797 and mixed #4776 stay DRAFT / NEVER MERGE/AUTO-MERGE including integration paths. Release #4763 untouched, no Cargo/delegation/production/credentials/infra writes.
fix(evals): reject overflowed JSON exponents before inference
ScriptedAlchemy commentedon Nov 11, 2026
Refs #4535 — published 8a4c08cc996b to the existing issue-specific DRAFT / NEVER MERGE #4797, targeting
master. Verified exact one-parent commit (parent37ec2f0652fe), four-file source/test/doc diff, expected-head non-force branch fast-forward, new remote PR head8a4c08cc996b, and continuing DRAFT state. Mixed #4776 remains separately DRAFT / NO MERGE; no new PR or integration bypass.New reproducible evidence-admission defect: Python's
json.loadsaccepts JSON numeric exponent1e309asfloat('inf')without calling the existingparse_constantrefusal. Thus an unused nested field in a frozen qualification manifest—or an HTTP response—could survive supposedly unambiguous, finite JSON admission. The stock decoder reproduction producedinf(nonfinite). The canonicalloads_unambiguous_jsonnow supplies aparse_floatthat refuses nonfinite overflow while preserving ordinary finite floats and the prior NaN/Infinity constant guard. This shared parser covers qualification manifests, source evidence, raw HTTP answers, partial replay and independent scoring; no additional parser or legacy path was created.Falsifiable regressions added to existing Bazel targets: a frozen manifest with an unreviewed nested
1e309must refuse before HTTP inference or results creation while a finite1e308control remains valid; a raw HTTP response with a nested1e309must refuse admission alongside existing duplicate-key and NaN tests. Updated HTTP runner documentation. Reviewed the exact published commit diff, with no weakened tests or gates.Verification still required:
bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:qualification_testmust execute nonzero tests on this head; no Bazel binary/checkout exists in the current runtime, so no passing Bazel/native/Clippy or independent review is claimed. No exact-head Actions run appeared immediately after publication, and draft PR CI may skip native lanes. Actual owning-transcript/digest-checked exporter, independent real labels, ten-question runtime parity, pinned authenticated Kev inference, task-effectiveness/resource/contention data and keep/replace/defer decisions remain acceptance blockers.Keep #4535 OPEN, dedicated owner ENABLED; #4797 and #4776 DRAFT / NEVER MERGE or AUTO-MERGE, including indirectly via integration. Release #4763, production, credentials, tags and infrastructure untouched. No Cargo, Kody, Copilot or external delegation.
fix(evals): reject HTTP responses completed after deadline
ScriptedAlchemy commentedon Nov 11, 2026
Refs #4535 — published
efa9869700e5directly to the existing DRAFT / NEVER MERGE #4797, targetingmaster. Verified exact one-parent child of8a4c08cc996b, expected-headforce=falsefast-forward, the remote PR head, and a three-file diff limited torun_http.py, its existing Bazel-wired test, and HTTP runner documentation. No duplicate PR: the only other #4535-linked open PR is the separately held mixed DRAFT #4776.New runtime-parity defect repaired: urllib's per-socket timeout could be repeatedly reset by slow response chunks, allowing a decoded HTTP 200 to be scored even after the manifest's frozen whole-request budget. In contrast, the production Scout client uses a global request deadline. The existing evaluator now rejects any completed HTTP response whose actual elapsed time exceeds the supplied request timeout, before it can create a raw sample or scored prediction row. Two falsifiable deterministic regressions cover accepted/late 50-ms HTTP exchanges and the full runner's no-sample/no-result retryable-header behavior. This does not forcibly interrupt a never-ending trickle read, as now explicitly documented; real end-to-end deadline behavior remains an outstanding limitation.
Verification: tests are committed but not executed by Bazel on this head; no passing CI/native or independent external review can be claimed. Exact-head PR CI had no run at recheck. Require nonzero
//evals/intent_engine:http_runner_testplus relevant qualification/scorer targets, host-native validation and independent review. Still missing full owning-store digest-checked Scout exporter/negative receipt rehydration, authenticated independent labels, ten-question runtime parity, pinned actual Kev inference, task effectiveness, p50/p95 cold/warm resource and contention data and keep/replace/defer findings. #4535 stays OPEN; original owner schedule remains enabled. #4797 and #4776 DRAFT / NO MERGE or auto-merge, including indirect integration. Release #4763 read-only; no Cargo, Kody/Copilot/external delegation, model asset or production/credential/infrastructure operations.ScriptedAlchemy commentedon Nov 11, 2026
Closing as not planned at the maintainer’s request to discontinue LLM/decision-model implementation and qualification work. This is a scope decision, not a claim that the proposed feature was delivered.