Skip to content

explore(kev): qualify decision replacements against current runtime inputs and canonical Scout evidence #4535

Closed as not planned

Description

@ScriptedAlchemy

Goal

Establish a reproducible Kev-4B baseline that can decide which existing TraceDecay decisions are worth replacing. Reuse the current SystemOne client and evaluation tools. The result should support a small number of measured replacements and identify the machinery each replacement would remove.

Current evidence

Source inspected at 90909cd9d0f2ced7be95d924ce574380303cef80:

  • The existing decision client already supports Noul, Choice, Score, typed failure outcomes, bounded concurrency and deadlines. The intent pass batches its questions in one request.
  • Runtime reference questions now contain ten intents, including build_diagnostics and atomic_edit. The checked-in benchmark has nine categories, including refactor_edit, and a different state renderer. The historical scores/thresholds do not qualify today's exact requests.
  • Plan 41 promises a digest-checked canonical-session exporter to Kev JSONL. No corresponding implementation was found in the inspected scripts/evals/CLI paths.
  • Delivery analytics describe delivered candidates. No-candidate exits do not retain the same prediction/window receipt. Delivered-hint data alone misses negative decisions and missed opportunities.
  • run_http.py and score.py already provide remote evaluation and useful classification/calibration metrics. Extend them instead of creating another harness.

The official Kev-4B card describes a model that scores supplied options without generating text. Its v1.0 checkpoint is 139fdd94, with temperature 2.41 and 8,192 validated context tokens; the larger serving limit is not equivalent to validated quality. Qualification must pin the actual checkpoint and serving configuration used.

Bounded work

  1. Freeze the actual runtime question vocabulary, instructions, renderer/settings identity, input limits and thresholds in an evaluation manifest. Reuse the existing canonical settings digest and production rendering contract; avoid a third hand-maintained question list.
  2. Implement the missing bounded export from existing Scout provenance and outcomes. Rehydrate through the owning transcript API, reproduce the original window bounds and verify its digest. Report missing, changed, pruned or expired evidence explicitly; do not silently approximate an old request or create a parallel transcript store.
  3. Keep predictions, observed action facts, delivery decisions, explicit feedback, acted/ignored behavior and independent correctness labels separate. Only verified labels may enter training data. Include independently labelled negatives, near neighbors, ambiguity and misleading instructions inside tool output.
  4. Split related examples by session/project, freeze held-out cases before choosing thresholds, and preserve the historical benchmark as historical evidence. Extend existing calibration/error/coverage metrics and the existing task-effectiveness harness where a replacement claims user benefit.
  5. Record checkpoint/server identity, backend/dtype/temperature, question/renderer/settings digests, input size, truncation/refusal, measured latency and available usage. Reject .partial resumes from a different experiment identity; unknown costs remain unknown.
  6. Run inference on a separately provisioned compatible host through the existing typed HTTP contract, with supported authenticated access where required. No model download or local execution in this cloud workspace is required.
  7. For candidate selection, evaluate option-order sensitivity and an explicit none outcome. Before a Choice result can select a live candidate, tighten the existing Choice decoder around the caller's supplied option set and required probability contract; the currently used intent path is Noul-only.

Acceptance

Produce a reproducible baseline/report and a keep, replace or defer recommendation for each evaluated application. Freeze application-specific error tolerances before reading held-out results. Report task success, unwanted suggestions or missed work, coverage/abstention, p50/p95 latency, cold/warm resource costs and contention with the existing intent pass.

A positive replacement decision must identify the existing code/tests/calls that disappear. If the existing deterministic behavior is as effective and cheaper, retain it. Fine-tuning is a later option justified by measured residual errors and independently labelled data.

Relationship to existing work

This completes the concrete qualification/export gap in Plan 41, whose Phase 1 was merged in #4121. Reuse the completed outcome work in #3654, #3655, #4216 and #4219; this issue does not reopen their broad task-effectiveness scope. Application-specific labels, decisions and deletion targets belong in the linked exploration issues.

This issue is based on source inspection and primary model documentation. No new Kev inference, training or performance measurements were performed while filing it.

Application explorations

Suggested order: establish this baseline, then prioritize Scout selection and the existing intent disagreement; evaluate background authoring/curation gates with separate missed-work tolerances.

Activity

  1. 127 remaining items

  2. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4535 — published two focused, exact-parent, expected-head, non-force fast-forward commits to the existing DRAFT / NEVER MERGE #4797 targeting master:

    • 4f90c1fc29dc: root cause was two unbounded source.read() paths for frozen captured Scout evidence (HTTP inference runner and independent scorer). Both callers now use one original-byte, inode-checked loader with a cohort-derived upper bound and 64 MiB safety ceiling. Added falsifiable sparse-oversize, fstat/read-growth, zero-network/no-output and independent scoring CLI regressions; documented exact bytes and limits. Positive normal capture admission preserved.
    • a2c3787ebbb4: fixed an actual qualified-sample scorer failure: validate_qualified_sample returns three digest values but qualified_report unpacked only two, raising ValueError whenever first-response evidence was supplied. Updated the remaining caller. Existing positive sample and qualified CLI tests exercise this non-vacuously.

    Re-read current root AGENTS, current master Scout path, complete issue discussion, competing PRs and review/CI. Verified remote PR head a2c3787e and DRAFT state. Verification limit: no Bazel-enabled checkout/native runner accessible; no executed exact-head Bazel Python/Rust tests, rustfmt/Clippy or independent approval is claimed. Exact-head hosted Release PR integrity was skipped, not qualification evidence. Author-side source review only.

    This is partial evidence admission and scorer correctness, not acceptance. Outstanding: canonical owning-store Scout transcript export and digest-checked rehydration incl negative decisions, genuinely independent real labels/holdouts, ten-question production settings/renderer parity, pinned actual Kev inference, task effectiveness, cold/warm p50/p95 resource and contention measurements, and per-application keep/replace/defer recommendations. #4535 stays OPEN. #4797 and mixed #4776 remain DRAFT — NEVER MERGE/AUTO-MERGE, including integration bypass. Release #4763 read-only; no Cargo, external delegation, releases/tags, secrets, production or infrastructure changes.

  3. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4535 — published two exact-parent, expected-head, non-force fast-forwards to the existing issue-specific DRAFT / NEVER MERGE #4797 targeting master: 7dc3744f6796, followed by formatting-only da57b34fafcc. Re-read confirms remote head da57b34fafcc91f0b968cbc2736708f0f7f67600, 81 PR commits, open/DRAFT; mixed #4776 remains separately DRAFT / NO MERGE. Neither merge nor auto-merge is authorized, including integration bypasses.

    Concrete production/evaluator admission mismatch repaired: DecisionEngineClientV1 previously accepted an otherwise valid Noul decision even when the server's model contradicted the explicitly requested selector. It also silently treated a present malformed model (null, Boolean, numeric, whitespace) as absent. The existing qualified HTTP evaluator refuses contradictory/missing reported model evidence, so runtime model receipts could describe a different identity than qualification. Production now rejects a malformed present model field and a contradictory named model. A genuinely omitted server model remains accepted as an unverified runtime response; no false server/checkpoint attestation was added.

    Falsifiable source regression: added native Rust loopback reported_model_must_be_valid_and_match_explicit_selection, with one matching positive, five formerly accepted invalid responses that must now be Malformed after exactly one HTTP exchange, and one omitted-model positive. No API introduction or compatibility shim; one production source/test file touched. Source/commit/fast-forward were verified, but independent review approval and execution of the nonzero Rust test remain outstanding. Latest exact-head Actions #42905457574 was only a skipped Release PR integrity workflow; no passing Bazel/native/rustfmt/Clippy was inferred. No Cargo.

    Not issue acceptance: canonical Scout owning-store export and digest-checked window rehydration; genuinely independently reviewed positives/negatives and frozen cohort; actual pinned Kev inference against a compatible authenticated HTTP host; current runtime question/settings, action-fact and delivery parity; p50/p95 effectiveness, cold/warm resource and contention; and per-application keep/replace/defer remain unproven. Keep #4535 OPEN and the dedicated owner ENABLED; #4797 and mixed #4776 DRAFT — NEVER MERGE/AUTO-MERGE, including indirect paths. Release #4763, production, credentials, infrastructure, tags and releases untouched; no external delegation.

  4. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4535 — actual source/test update published to existing DRAFT / NEVER MERGE #4797: commit ccb21285b37b, exact-parent nonforce expected-head fast-forward. Confirmed 82 commits on open DRAFT PR, targeting master; only qualification.py, test_qualification.py, and HTTP_RUNNER.md changed. No competing issue-specific PR.

    Newly fixed false-parity admission: captured Scout qualification could freeze settings that production Rust never accepts/emits: non-integer-permille thresholds, state budgets >1 MiB, padded/control-bearing/oversized explicit model names, or control characters in question instructions. Admission now refuses these before HTTP inference or output. Two new Bazel-wired falsifiable tests exercise five invalid frozen-arm cases with zero HTTP calls/no result directory, instruction/key control-character rejection, and a valid control. Historical noncaptured threshold semantics remain intact; no legacy facade introduced.

    No exact-head Bazel/native CI or independent approval is claimed in this runner (no Bazel/checkout; PR head has no completed CI run). Required nonzero Bazel evaluation targets and reviewer clearance remain. This is partial Refs, not acceptance: original owning-store window exporter/digest verification, verified real labels, ten-question effective settings parity, pinned actual Kev inference, task success and p50/p95 cold/warm resource/contention measurements plus per-application keep/replace/defer remain missing. Keep #4535 OPEN and the original owner enabled; #4797 and #4776 DRAFT / NEVER MERGE/AUTO-MERGE (including indirect integration). Release #4763 unchanged; no Cargo, Kody/Copilot/delegation, production, secrets or infrastructure.

  5. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4535 — published 37ec2f0652fe to existing DRAFT / NO MERGE #4797 with exact-parent/nonforce expected-head ref update. A new local-evidence admission gap was addressed: frozen manifest/questions, server configuration, dataset/subset and independent result scorer had unconstrained file reads despite the existing bounded Scout-capture reader. They now share one no-follow, size-bounded (64 MiB) held-inode reader, rejecting sparse, symlink/FIFO and detectably changing files. Duplicated captured-file reader was removed. Three new Bazel-wired behavioral tests cover valid controls, zero-network/no-output refusals, and preservation of a published scored report; existing capture tests migrated to the renamed canonical ceiling. Full source/review and verification caveats.

    Not a qualified Kev baseline: exact-head Bazel/nonzero native execution and independent review remain unverified; no compatible pinned Kev host, authenticated canonical transcript export, independent real labels, ten-question runtime settings parity, p50/p95 task/resource/contention evidence, or per-application keep/replace/defer result is established. #4535 remains OPEN, original issue owner remains active; #4797 and mixed #4776 stay DRAFT / NEVER MERGE/AUTO-MERGE including integration paths. Release #4763 untouched, no Cargo/delegation/production/credentials/infra writes.

  6. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4535 — published 8a4c08cc996b to the existing issue-specific DRAFT / NEVER MERGE #4797, targeting master. Verified exact one-parent commit (parent 37ec2f0652fe), four-file source/test/doc diff, expected-head non-force branch fast-forward, new remote PR head 8a4c08cc996b, and continuing DRAFT state. Mixed #4776 remains separately DRAFT / NO MERGE; no new PR or integration bypass.

    New reproducible evidence-admission defect: Python's json.loads accepts JSON numeric exponent 1e309 as float('inf') without calling the existing parse_constant refusal. Thus an unused nested field in a frozen qualification manifest—or an HTTP response—could survive supposedly unambiguous, finite JSON admission. The stock decoder reproduction produced inf (nonfinite). The canonical loads_unambiguous_json now supplies a parse_float that refuses nonfinite overflow while preserving ordinary finite floats and the prior NaN/Infinity constant guard. This shared parser covers qualification manifests, source evidence, raw HTTP answers, partial replay and independent scoring; no additional parser or legacy path was created.

    Falsifiable regressions added to existing Bazel targets: a frozen manifest with an unreviewed nested 1e309 must refuse before HTTP inference or results creation while a finite 1e308 control remains valid; a raw HTTP response with a nested 1e309 must refuse admission alongside existing duplicate-key and NaN tests. Updated HTTP runner documentation. Reviewed the exact published commit diff, with no weakened tests or gates.

    Verification still required: bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:qualification_test must execute nonzero tests on this head; no Bazel binary/checkout exists in the current runtime, so no passing Bazel/native/Clippy or independent review is claimed. No exact-head Actions run appeared immediately after publication, and draft PR CI may skip native lanes. Actual owning-transcript/digest-checked exporter, independent real labels, ten-question runtime parity, pinned authenticated Kev inference, task-effectiveness/resource/contention data and keep/replace/defer decisions remain acceptance blockers.

    Keep #4535 OPEN, dedicated owner ENABLED; #4797 and #4776 DRAFT / NEVER MERGE or AUTO-MERGE, including indirectly via integration. Release #4763, production, credentials, tags and infrastructure untouched. No Cargo, Kody, Copilot or external delegation.

  7. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4535 — published efa9869700e5 directly to the existing DRAFT / NEVER MERGE #4797, targeting master. Verified exact one-parent child of 8a4c08cc996b, expected-head force=false fast-forward, the remote PR head, and a three-file diff limited to run_http.py, its existing Bazel-wired test, and HTTP runner documentation. No duplicate PR: the only other #4535-linked open PR is the separately held mixed DRAFT #4776.

    New runtime-parity defect repaired: urllib's per-socket timeout could be repeatedly reset by slow response chunks, allowing a decoded HTTP 200 to be scored even after the manifest's frozen whole-request budget. In contrast, the production Scout client uses a global request deadline. The existing evaluator now rejects any completed HTTP response whose actual elapsed time exceeds the supplied request timeout, before it can create a raw sample or scored prediction row. Two falsifiable deterministic regressions cover accepted/late 50-ms HTTP exchanges and the full runner's no-sample/no-result retryable-header behavior. This does not forcibly interrupt a never-ending trickle read, as now explicitly documented; real end-to-end deadline behavior remains an outstanding limitation.

    Verification: tests are committed but not executed by Bazel on this head; no passing CI/native or independent external review can be claimed. Exact-head PR CI had no run at recheck. Require nonzero //evals/intent_engine:http_runner_test plus relevant qualification/scorer targets, host-native validation and independent review. Still missing full owning-store digest-checked Scout exporter/negative receipt rehydration, authenticated independent labels, ten-question runtime parity, pinned actual Kev inference, task effectiveness, p50/p95 cold/warm resource and contention data and keep/replace/defer findings. #4535 stays OPEN; original owner schedule remains enabled. #4797 and #4776 DRAFT / NO MERGE or auto-merge, including indirect integration. Release #4763 read-only; no Cargo, Kody/Copilot/external delegation, model asset or production/credential/infrastructure operations.

  8. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Closing as not planned at the maintainer’s request to discontinue LLM/decision-model implementation and qualification work. This is a scope decision, not a claim that the proposed feature was delivered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions