Skip to content

feat(repo): choice events for a candidate's install, import, use or removal - #787

Open
napetrov wants to merge 25 commits into
intel:mainfrom
napetrov:feat/trace-choice-events
Open

napetrov wants to merge 25 commits into
intel:mainfrom
napetrov:feat/trace-choice-events

Conversation

@napetrov

@napetrov napetrov commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds a choice event: an agent's install, import, use or removal of a candidate the run names (repo run --candidates FILE), written beside its command or edit event.
  • An install or removal counts only where the command's exit code is its own: a && chain with no pipe, no --dry-run, no --download-only, and no part after an exit or a replacing exec. Contract §4.1 lists what is missed.
  • Round 5 (review threads on 949755343): a conda channel prefix (conda-forge::scipy) is no part of the name; an exec with options; --report and other pip value options; probes stop at an executed exit or exec, skip false && … and || short circuits, read heredoc terminators by shell rules, ignore $((…)) and redirection targets, and no longer treat command -v, a wrapper's --help, or env as the program.
  • One change to existing events. A Copilot shell command's rc is its shellExecution.exitCode only. A result that carries only success now gets rc null where it got 0 or 1, so failed_commands and first_useful count such a command as neither failed nor useful. Results that carry shellExecution.exitCode are unchanged. Contract §4.1 and a changelog fragment say so.
  • Otherwise additive: one new event kind, written only when candidates are given; the counters and first_useful do not read choices.
  • Stacked on feat(repo): probe events for commands that ask the host for its facts #782 (probe events, commit d3360cd2ef), which is still open. This PR's base is main, so its diff also shows that commit until feat(repo): probe events for commands that ask the host for its facts #782 merges.

Closes #807

Evidence

  • make ci-local on 052e374999 (same tree as 2661870ca5): 21979 passed, 19 skipped, 3 failed. The 3 are timeout tests that load slows (test_process.py::test_sighup_kills_the_live_groups_too, test_repo_reach.py orphan kill, test_repo_reach_edges.py); all 191 pass when those files run alone.
  • Each review thread has a test that is red on the earlier head and green here: the test names are in the commit messages.
  • Probe and choice outputs for the reviewed inputs were checked with the probe functions on 949755343 and on this head.
  • Self-review (pr-review, fresh context, one pass on 949755343..56791a5b1a): no blocking finding. The last two commits add a test and reword the contract from its notes. Declined notes are under Merge Danger.
  • Golden pin in tests/test_batch_ledger_golden.py: ac034e664259 → d4546308cce5. No golden fixture bytes changed in this round.
  • Not exercised end to end: repo run needs a container runtime, which the review box lacks. Covered by unit tests and the CLI snapshot.

Merge Danger

  • Stacked: merge feat(repo): probe events for commands that ask the host for its facts #782 first. Until then the diff includes d3360cd2ef.
  • Copilot shell rc (see Summary): failed_commands and first_useful change for a result that has only success.
  • New --candidates FILE flag on repo run: CLI snapshot regenerated, and --candidates has a row in tests/data/cli_flags_allowlist.txt.
  • events.jsonl gains choice rows when candidates are given. Readers must accept the new kind.
  • Choices and probes are textual matches, not a shell parser. Known misses, declined in review: a && chain whose later part fails gives no install choice (the exit code cannot say which part failed); installs inside bash -c or (pip …) give none; make 2>&1 | exit; nproc and exec sleep 5 & nproc lose the later probe (an undercount); a heredoc terminator with a CRLF ending stays open, as bash does; an apostrophe inside double quotes before << on the same line can hide the heredoc, so its body is scanned; a stray brace, do or done turns off the exit cutoff for the script (a probe may then repeat); combined short flags (command -pv) are not parsed.
  • Copilot apply_patch and Codex file_change carry no added text, so they give install and removal choices only.

🤖 Generated with Claude Code

A command that runs nproc, lscpu, nvidia-smi, pip list, printenv, a
python -c one-liner calling os.cpu_count(), or a PowerShell
Get-CimInstance Win32_Processor gets a probe event after its command
event, one per pattern it matches. Patterns match the program the shell
runs, never argument text, so grep nproc README is no probe.

The defaults ship in repos/probes.py. A manifest's probes key and
repo run --probes FILE replace a target's list; [] turns it off.
Existing event kinds are unchanged.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e or removal

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The harness reports one exit code per command, so an install after `||`,
`;` or a pipe may not have run, or may have failed, while the command
still exits 0. Such installs now give no choice. A `--dry-run` installs
nothing and gives none. An import after `;` on the same line is seen,
and `uv add` and `uv remove` name their package like `uv pip` does.

Contract section 4.1 says which installs count and which do not.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 15, 2026 07:43
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 15, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi repo run` parameters changed:
    + added option `--candidates`
    + added option `--probes`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@exai-bot

exai-bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 8 issue(s) in this PR.

CI at 2661870: 29 passing, 3 skipped.

By severity: 8 medium

Severity Category Location Finding
medium bug probes.py:213-214 Must-fix — distinguish pipeline-local termination from termination of the parent shell.
medium bug probes.py:378-379 Must-fix — find the executable after leading redirections.
medium bug probes.py:245-247 Must-fix — track all heredocs opened by a command, not just the first.
medium bug runner.py:118 Must-fix — include the manifest's probe configuration in resume identity.
medium bug choices.py:227-228 Must-fix — consume uv's dependency-group option values.
medium bug choices.py:211-212 Must-fix — exclude uv project operations that do not synchronize the environment.
medium bug probes.py:246 Must-fix — do not assume every heredoc attached to a shell is executable script input.
medium bug probes.py:254-255 Must-fix — distinguish here-strings from heredoc openers.
Review details
  • Commit reviewed: 2661870 against merge base 47f89ed
  • Coverage: 22 files in 5 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 2.02M tokens (1.49M from cache), 2m6s
  • Tool calls: 140 (92 file_read, 22 code_search, 13 pr_threads, 6 file_read_diff, 4 code_comment, 2 local_check_results, 1 pr_comments)
  • Probe events and event emission tests: 5 files
  • Choice event extraction: 3 files
  • Repository evaluation configuration and runner integration: 7 files
  • Copilot shell return codes and trace handling: 4 files
  • Batch ledger and evaluation fixtures: 3 files
  • ⏭️ Skipped (overlap with history): 1 comment(s)

Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
@codecov-commenter

codecov-commenter commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.94585% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.75%. Comparing base (47f89ed) to head (2661870).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/repos/choices.py 92.73% 7 Missing and 6 partials ⚠️
dokimi/dokimi/repos/probes.py 97.61% 5 Missing and 3 partials ⚠️
dokimi/dokimi/repos/events.py 50.00% 2 Missing and 2 partials ⚠️
dokimi/dokimi/repos/manifest.py 66.66% 3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #787      +/-   ##
==========================================
- Coverage   95.76%   95.75%   -0.01%     
==========================================
  Files         416      418       +2     
  Lines       55624    56172     +548     
  Branches     9553     9681     +128     
==========================================
+ Hits        53267    53789     +522     
- Misses       1456     1472      +16     
- Partials      901      911      +10     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/repo_run.py 95.86% <100.00%> (+0.06%) ⬆️
dokimi/dokimi/repos/batch_ledger.py 95.48% <100.00%> (+0.10%) ⬆️
dokimi/dokimi/repos/cli_traces.py 97.70% <100.00%> (+0.38%) ⬆️
dokimi/dokimi/repos/runner.py 97.03% <100.00%> (+0.05%) ⬆️
dokimi/dokimi/repos/transcript.py 97.26% <100.00%> (+0.03%) ⬆️
dokimi/dokimi/repos/manifest.py 94.19% <66.66%> (-0.59%) ⬇️
dokimi/dokimi/repos/events.py 95.41% <50.00%> (-1.57%) ⬇️
dokimi/dokimi/repos/probes.py 97.61% <97.61%> (ø)
dokimi/dokimi/repos/choices.py 92.73% <92.73%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 2991eb7. The events are wired in the right place: they go to events.jsonl only, and the counters and first_useful do not read them. The matching behind them is textual, and its holes write false choice and probe events. The merge waits for the bot points below, which I reproduced, and for the risks listed here.

What I ran, on a Linux review host at 2991eb7:

  • make ci-local: 31 passed, 2 failed, 14 skipped; 20,570 tests passed; changed-line coverage 92%. The 2 failures are the gitleaks install (HTTP 504) and the commit scan that needs it, which do not depend on this PR.
  • Reproductions for the code findings, with no model or network call.
  • A mutation run over choices.py and probes.py against their test files: 33 of 59 probes.py mutants and about ten choices.py mutants pass the tests.

Findings that cannot be anchored inline:

  1. --probes and --candidates are recorded nowhere in a cell or batch. execution_record (runner.py:107, outside this diff) takes the profile, tools, delegation and binary digests only. Resume a batch with another --candidates or --probes file: check_batch_executions does not refuse it, finished cells are skipped, and the batch holds events.jsonl files built under two configurations with nothing to tell them apart. Suggestion: add both files' digests to the execution record, or refuse a resume whose files differ.
  2. The tests pin the matching rules thinly. Mutation check on choices.py: removing not event.get("denied") (choices.py:97), the (?!\w) after from name, the startswith(name + ".") submodule rule, the (?<![\w.]) lookbehind of _calls, and the [ in _REQUIREMENT_END all pass test_repo_choices.py. On probes.py: dropping sudo, nohup, exec or nice from _WRAPPERS, shrinking _READERS to cat, head, grep, letting _in_order match non-consecutive stages, dropping the (?<![\w.]) lookbehind or the \s*\( call requirement of the Python probe, and dropping .exe and backslash handling all pass. Nothing exercises runner.py:267-269 or repo_run.py either: a change that drops opts.candidates or opts.probes passes the suite. Add one case per rule, plus a run_cell test that checks events.jsonl holds a choice and a probe row.

Open bot points I confirm:

  • I agree with exai-bot that an unresolved call counts as a completed one. New evidence: pip install scipy with rc None gives an install choice and with rc 1 none; the suggested rc == 0 would also drop every Copilot shell command, since cli_traces.py:745 writes rc as None for good.
  • I agree with exai-bot that a pipe without spaces gives an install choice. New evidence: pip install scipy |cat with rc 0 gives an install choice for scipy, pip install scipy | cat gives none.
  • I agree with exai-bot that a semicolon in a comment starts an import. New evidence: added text # example; import sklearnex gives import and use for sklearnex; x = 1 # import sklearnex gives use only.
  • I agree with exai-bot that help requests count as installs. New evidence: pip install scipy --help and pip install -h scipy with rc 0 each give an install choice for scipy.
  • I agree with exai-bot that an option value is read as a package. New evidence: pip install --target scipy numpy, -r scipy, -c scipy numpy and --index-url scipy numpy each give an install choice for scipy.
  • I agree with exai-bot that a heredoc body is read as commands: cat <<'EOF' > s.sh followed by a line nproc emits a cpu probe although only cat ran.
  • I agree with exai-bot that quoting is ignored when substitutions are extracted: echo '$(nproc)' prints a literal and still emits a cpu probe.
  • I agree with exai-bot that nested substitutions are missed: make -j$(nproc --ignore=$(printf 1)) runs nproc and emits no probe.
  • I agree with exai-bot that a quoted pipe splits a command: echo '|' nproc runs only echo and emits a cpu probe.
  • I agree with exai-bot that wrapper options hide the wrapped program: sudo -n nproc emits no probe.
  • I agree with exai-bot that a search pattern counts as a file operand: grep -e /proc/cpuinfo README emits the /proc/cpuinfo probe.
  • I agree with exai-bot that a string in a Python one-liner counts as a call: python -c 'print("os.cpu_count()")' emits a probe.

Comment thread dokimi/docs/repo-evaluation-contract.md Resolved
Choice events:
- `pip install X |cat` and a quoted `'|'` no longer read as a pipeline
  by whitespace alone; quoting decides.
- `--help`, `-h`, `--version` give no install or removal choice.
- A value of `--target`, `-r`, `-c`, `--index-url`, `--name` ... is
  not a package operand.
- `sudo`, `time`, `nice` before the manager no longer hide the call.
- An import is read from code: a `#` comment is dropped, a `\`
  continuation is joined, `if x: import m` counts.

Probe events:
- A heredoc body is data unless a shell reads it.
- `$( )` and backticks inside single quotes or escaped are text;
  nested substitutions are walked.
- A quoted `|` is not a stage break.
- Wrapper options (`sudo -n`, `nice -n 5`, `time -p`) are skipped
  before the wrapped program.
- A path probe matches a file operand, not a search pattern or an
  option value.
- A `py:NAME` probe matches a call in the parsed `-c` code, not a
  string or a comment.

Tests pin each rule, the denied-call check and the `run_cell` wiring of
`--candidates` and `--probes`. The contract names Copilot's edit tools
among those that give no import or use choice.

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

Pushed 1 commit to feat/trace-choice-events (2991eb7..b3bd1d8): Waiting on @napetrov: 2 questions below.

Changes

  • pip.exe and python.exe -m pip name the manager.
  • Windows paths keep their backslashes before the program is read.
  • py:NAME matches the interpreter's own -c, then the parsed code.
  • exit and exec end the && chain.
  • An empty --candidates file is refused.
  • A manager behind sudo, time or nice is found.
  • _ran has a test for a denied call, and run_cell has one for its --candidates and --probes rows.
  • A continued or compound import line is an import.
  • Pipes, help flags, option values, comments, heredocs, quoted and nested substitutions, wrapper options and file operands are read as the shell reads them.

Needs @napetrov

  1. Should a command or edit with no result (rc None) count as a completed choice? (dokimi/dokimi/repos/choices.py:97)
    pip install scipy with rc None gives an install choice on the head. Requiring rc == 0 also drops every Copilot shell command, since cli_traces.py:745 writes rc as None for good.
    • A: Keep rc in (None, 0): interrupted Claude runs can show a choice that never took effect
    • B: Require rc == 0 and have Copilot report a real rc: no unresolved call counts, larger change in cli_traces.py
    • C: Require rc == 0 for Claude and Codex only: Copilot shell commands give no install choice
  2. Should a resume with a different --candidates or --probes file be refused, or only recorded in the execution record? (dokimi/dokimi/repos/runner.py:107)
    execution_record (runner.py:107) holds the profile, tools, delegation and binary digests only. Resume a batch with another file: check_batch_executions does not refuse it and the batch holds events.jsonl built under two configurations.
    • A: Add both files' digests to the execution record, so a differing resume is refused by the existing check
    • B: Record the digests and only warn on a differing resume

Threads and comments

  • Fixed: 8, each answered in its thread.
  • Fixed (comment by @antonwolfy): the tests pin the matching rules thinly; nothing exercises runner.py wiring – Added a case per rule named for denied, use, from name and submodule imports, wrappers, readers, the Python name boundary and consecutive stages, plus a run_cell test that checks events.jsonl holds a choice and a probe row.
  • Fixed (comment by @antonwolfy): Copilot's other edit tools give no import or use choice – The contract and the changelog fragment name Copilot's edit tools.

Not this PR's

  • Fails on main too: 3 tests in tests/test_repo_compare_interval.py – fails on main as well; the file is not touched by the PR.
Verified (12)
  • probe and choice rows reach events.jsonl (the test moved; it passes in its new file): passes.
  • denied call and unresolved wiring: passes.
  • installs that are no choice: passes.
  • wrapped installs give their choice: passes.
  • imports read as code: passes.
  • text the shell does not run gives no probe: passes.
  • commands the shell runs give their probe: passes.
  • probe and choice rows reach events.jsonl: passes.
  • stages must be consecutive: passes.
  • python probe name boundary: passes.
  • use matches at a name boundary: passes.
  • Full suite with main merged in: 21906 passed.

Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
- restore the `_package` definition: `pip install 'scipy >= 1.11'` raised NameError
- `exec pip install x` gives its choice; later `&&` parts still give none
- redirection operators and targets are no package or file operand; `< file` still reads
- a string literal no longer hides or fakes an import on its line
- `uv --quiet add x` gives its install choice
- `sudo bash -c 'nproc'` and an unquoted heredoc's `$(nproc)` give their probe
- `echo \| nproc` runs no pipeline
- grep/rg/sed/awk long pattern options and `-f` no longer turn a file into a pattern
- `exec true && pip install x` gives no choice; a quoted `'>'` is a word, not a redirection
- a command after `false ||` gives its probe; only `true ||` and `: ||` skip it
- `FOO=1 exec true && pip install x` gives no choice
- a quoted `'1'` before a redirection is a word, not a descriptor

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

🟡 Waiting on @napetrov · 1 commit on feat/trace-choice-events (b3bd1d8..5705149)

Next: @napetrov: answer the 2 questions below.

Changes

  • _package is defined again (the def line was lost on 5705149).
  • uv --quiet add scipy gives its install choice.
  • An exec of a non-package command ends the chain, and a quoted > or < is no redirection.
  • A command after || gives its probe unless the left side is true or :.
  • An assignment before exec still ends the chain.
  • A quoted numeric word before a redirection is no descriptor.

Needs @napetrov

  1. Should a command whose call has no result (rc None) count as run, or only rc 0? (dokimi/dokimi/repos/choices.py:104)
    pip install scipy with rc None gives an install choice. The suggested rc == 0 also drops every Copilot command, since cli_traces.py:745 writes rc None for good.
    • A: keep rc None as run: a cut-off Claude call can give a choice
    • B: require rc 0 and have cli_traces.py write rc 0 for a Copilot success: cut-off calls give none
  2. After ||, should a probe in the fallback count when the command's rc is 0? (dokimi/dokimi/repos/probes.py:192)
    false || nproc with rc 0 gives no cpu probe, because every part after || is skipped; true || nproc with rc 0 is the same text outcome and the shell never runs nproc there. The text and rc cannot tell them apart.
    • A: skip parts after ||, as now: a fallback that ran is missed
    • B: keep them: true || nproc gives a false cpu probe

Threads and comments

  • 8 fixed; each answered in its thread.
4 threads left as they are, and why
  • dokimi/dokimi/repos/probes.py:176 (exai-bot) – Held on the head: `test_commands_the_shell_runs_give_their_probe[make -j$(nproc --ignore=$(printf ….
  • dokimi/dokimi/repos/probes.py:195 (exai-bot) – Held on the head: test_text_the_shell_does_not_run_gives_no_probe[echo '|' nproc] and `[sysctl -a '|' grep ….
  • dokimi/dokimi/repos/probes.py:227 (exai-bot) – Held on the head: test_text_the_shell_does_not_run_gives_no_probe[grep -e /proc/cpuinfo README], `[grep ….
  • dokimi/dokimi/repos/probes.py:236 (exai-bot) – Held on the head: test_text_the_shell_does_not_run_gives_no_probe[python -c 'print("os.cpu_count()")'] and ….

Verified on the new head: ✅ 6/6 tests fail before the fix, pass after · ✅ full suite with main merged in: 21947 passed, 3 failures that main has too · ⚠️ second reader: 3 of its points still open

What each test shows
  • redirections, escaped pipes and pattern options give no probe: fails before, passes after.
  • wrapped shells, heredoc expansions, < reads, -f and a quoted > give their probe: fails before, passes after.
  • redirect targets, exec chains (also exec true && ...) and escaped pipes give no install choice: fails before, passes after.
  • exec, uv --quiet, quoted requirement give their install choice: fails before, passes after.
  • strings and comments do not decide an import: fails before, passes after.
  • exec pip uninstall gives its removal choice: fails before, passes after.
  • Left to GitHub's own checks: agent-quality.yml:external-catalog-smoke, ci.yml:static / pre-commit, ci.yml:static / Install gitleaks (unsupported host: needs installed native gitleaks at the workflow's pinned version), ci.yml:static / Scan commits (unsupported host: needs installed native gitleaks at the workflow's pinned version), ci.yml:fidelity / Install the pinned Claude Code, ci.yml:test-latest, ci.yml:package-smoke, ci.yml:windows-smoke.

Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
…rgets as the shell does

Thread: PRRT_kwDOVBg7g863v6ZT
Thread: PRRT_kwDOVBg7g863v6ZX
Thread: PRRT_kwDOVBg7g863v6ZZ
Thread: PRRT_kwDOVBg7g863v6Zd
Thread: PRRT_kwDOVBg7g863v6Zi
Thread: PRRT_kwDOVBg7g863v6Zl
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863nhD_: already fixed in 5705149 (pipe without spaces)
Thread: PRRT_kwDOVBg7g863nhEB: already fixed in 5705149 (semicolon in a comment)
Thread: PRRT_kwDOVBg7g863nhEC: already fixed in 5705149 (help and version requests)
Thread: PRRT_kwDOVBg7g863nhEE: already fixed in 5705149 (option values are not packages)
Thread: PRRT_kwDOVBg7g863nhEG: already fixed in 5705149 (heredoc body is data)
Thread: PRRT_kwDOVBg7g863nhEI: already fixed in 5705149 (quoted substitutions)
Thread: PRRT_kwDOVBg7g863nhEJ: already fixed in 5705149 (nested substitutions)
Thread: PRRT_kwDOVBg7g863nhEM: already fixed in 5705149 (quoted pipe)
Thread: PRRT_kwDOVBg7g863nhEN: already fixed in 5705149 (wrapper options)
Thread: PRRT_kwDOVBg7g863nhEP: already fixed in 5705149 (grep pattern options)
Thread: PRRT_kwDOVBg7g863nhES: already fixed in 5705149 (python -c strings)
Thread: PRRT_kwDOVBg7g863o6RG: already fixed in 5705149 (contract names Copilot edit tools)
Thread: PRRT_kwDOVBg7g863sWEA: already fixed in 5705149 (exec replaces the shell)
Thread: PRRT_kwDOVBg7g863sWEF: already fixed in 5705149 (comment and string scanner)
Thread: PRRT_kwDOVBg7g863sWEK: already fixed in 5705149 (redirect targets)
Thread: PRRT_kwDOVBg7g863sWEN: already fixed in 5705149 (wrapped shell scripts)
Thread: PRRT_kwDOVBg7g863sWEV: already fixed in 5705149 (unquoted heredoc expansions)
Thread: PRRT_kwDOVBg7g863sWEd: already fixed in 5705149 (escaped operators)
Thread: PRRT_kwDOVBg7g863sWEi: already fixed in 5705149 (output redirection is no read)
Thread: PRRT_kwDOVBg7g863sWEo: already fixed in 5705149 (grep --regexp and -f)
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A call with no result (rc None: a cut-off call, or a Copilot command, which
writes no rc) and a failed one both take no choice. The fake harness gets a
result flag for the events test, so its golden cell ids move with its bytes.

Thread: PRRT_kwDOVBg7g863nhD6
Thread: IC_kwDOVBg7g88AAAABmzVvzA#1
Thread: IC_kwDOVBg7g88AAAABm4kUeA#1
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The execution id takes the digest of each given file's parsed content, so a
resume with another file is another execution and check_batch_executions
refuses it. An unset file adds no key, so the ids of runs without one stay.

Thread: IC_kwDOVBg7g88AAAABmzVvzA#2
Thread: PRR_kwDOVBg7g88AAAABYEmcQg#1
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… passed

A bracketed extra on the package name and a -c that follows a script name
were not pinned by any test: dropping either left the suite green.

Thread: PRR_kwDOVBg7g88AAAABYEmcQg#2
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
They were appended to the opening line, so a comment after the operator hid
them: cat <<EOF # c, then $(nproc), gave no probe. They now go before <<.

Thread: PRRT_kwDOVBg7g863v6Zd
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Inside "..." the second backslash of `\\` was read as an escape of the closing quote,
so a pipe after it was lost (probe) or a pipeline read as one stage (choice).

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…idate and probe digests

The execution id hashes the parsed --candidates and --probes files, so a resume with
other ones is refused; the _ran docstring says a Copilot call with no exit code or
success flag is rc None.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…er them

Thread: PRR_kwDOVBg7g88AAAABYEmcQg
Thread: IC_kwDOVBg7g88AAAABmzVvzA
Thread: IC_kwDOVBg7g88AAAABm4kUeA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 5 must-fixes · reviewed 5705149

Next: @napetrov: fix the 5 must-fix findings and answer 1 open thread from exai-bot; then push for a re-review.

Blocking

  • Still holds on this head: thread by exai-bot – pip install scikit-learn-intelex from a Claude trace with no tool_result (rc None) still gives an install choice; _ran accepts rc None.

Must-fix

  • dokimi/dokimi/repos/probes.py:573 – env FOO=1 python train.py gives an env probe, and env FOO=1 nproc gives no cpu probe: env is not in _WRAPPERS.

    Input: env FOO=1 python train.py and env -i python train.py. Gives a probe with target env, pattern env; env FOO=1 nproc gives env and no cpu. Expected: no probe for the first two, since the shell runs python, and cpu for the last. Contract §4.1: patterns match the program the shell runs. printenv and bare env still count.

    Also at: dokimi/dokimi/repos/probes.py:576.

    Suggestion: Treat env as a wrapper when it is followed by a program (skip its NAME=value words and options), and keep the env probe for a bare env, env | grep X and env > f.

    Patch, run on 5705149: the failing case passes; tests and lint are no worse.

    Patch
    --- a/dokimi/dokimi/repos/probes.py
    +++ b/dokimi/dokimi/repos/probes.py
    @@ -49,10 +49,10 @@
     
     _READERS = frozenset({"cat", "head", "tail", "less", "more", "nl", "sed", "grep", "egrep", "fgrep", "awk", "tac",
                           "strings", "wc", "rg"})
    -_WRAPPERS = frozenset({"time", "nohup", "sudo", "command", "exec", "builtin", "nice"})  # each runs the command after it
    +_WRAPPERS = frozenset({"time", "nohup", "sudo", "command", "exec", "builtin", "nice", "env"})  # each runs the command after it
     _WRAPPER_VALUES = {"sudo": frozenset({"-u", "-g", "-h", "-p", "-r", "-t", "-C", "-D", "-R", "-T", "-U"}),
                        "nice": frozenset({"-n", "--adjustment"}), "exec": frozenset({"-a"}),
    -                   "time": frozenset({"-f", "-o", "--format", "--output"})}  # options that take the next word
    +                   "time": frozenset({"-f", "-o", "--format", "--output"}), "env": frozenset({"-u", "-C", "--unset", "--chdir"})}  # take the next word
     _PATTERN_READERS = frozenset({"grep", "egrep", "fgrep", "rg", "sed", "awk"})  # the first operand is a pattern or script
     _GREP_VALUES = {"-e", "-f", "-m", "-A", "-B", "-C", "--regexp", "--file", "--max-count", "--after-context",
                     "--before-context", "--context"}
    @@ -338,8 +338,10 @@
     
     def _views(words: list[str]) -> Iterator[Stage]:
         """The command a simple command runs: its own program, then the command each wrapper (``time``, ``sudo``...) runs."""
    -    for rest in command_words(words):
    -        yield _head(rest[0], rest[1:])
    +    views = list(command_words(words))
    +    for k, rest in enumerate(views):
    +        if k == len(views) - 1 or _program(rest[0]) != "env":  # ``env FOO=1 nproc`` runs ``nproc``, not a probe of ``env``
    +            yield _head(rest[0], rest[1:])
     
     
     def command_words(words: list[str]) -> Iterator[list[str]]:

Verified on 5705149: ✅ changed tests: 192 passed, fail on main · ✅ full suite with main merged in: 21947 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 4 edge inputs tried

6 checks behind it, each with its evidence
Area What holds How
Changed tests tests/test_repo_choices.py, tests/test_repo_probes.py and tests/test_repo_run_events.py pass on the head (192 passed) and fail to import on main tests on the head
Merge Merges onto main with no conflict merge onto main
Denied calls Removing not event.get("denied") from _ran fails test_a_denied_install_gives_no_choice broke the code, tests caught it
Probe layers A manifest layer replaces a target's defaults, a later --probes layer replaces it again, and [] turns the target off without touching other targets ran it, edge input (collections)
Text input Empty, whitespace, ß, combining marks, U+2028 and a 200000-character command text go through with_probes and with_choices without raising ran it, edge input (text)
Config file paths A missing --probes file, a directory given as --candidates, and a non-UTF-8 file each give ValueError naming the flag and path ran it, edge input (paths)

Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

Fixed in 5258517: option C, rc 0 needed; Copilot rc is follow-up #807.

Fixed in 7d9b70d: option A, both files' digests in the id; pinned.

Needs the author: 2 questions
Answered in #1 and #2. Recorded in 9497553.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

Fixed in 5258517: the same rc 0 rule as #1 above; pinned by a test.

Not changed. Probes read text, not exit codes per part: make \|\| nproc gives nproc, though make may succeed; true \|\| nproc gives none. Known miss: make && true \|\| nproc when make fails.

Needs the author: 2 questions
Answered in #1 and #2 (#2 is pushed back). Recorded in 9497553.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

Fixed in 7d9b70d: a resume with other files is refused (option A); pinned.

Fixed in a77dda9: bracket and -c after a script pinned; mutants fail.

Review: two findings that cannot be anchored inline
Answered in #1 and #2. Recorded in 9497553.

Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/choices.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
…port give no false choice

exec -l has no program and does not end the shell, so the install after it counts. conda's
--download-only fetches packages without linking them, and pip's --report takes a file name
as its value, so the next word is not a package.

Thread: PRRT_kwDOVBg7g863zaJY
Thread: PRRT_kwDOVBg7g863zaJf
Thread: PRRT_kwDOVBg7g863zaJk
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A denied command runs nothing. command -v and a wrapper's --help or --version run no program, and
env runs the program after its NAME=value words, so no probe is of env there. A part after a failed
&& or a succeeded || never runs, nor does anything after an exit or an exec with a program. A
heredoc ends only at its exact delimiter, and only an unquoted << opens one. $(( )) is arithmetic,
not a substitution. A redirection's target is no operand of a command pattern.

Thread: PRRT_kwDOVBg7g863zU5_
Thread: PRRT_kwDOVBg7g863zU6O
Thread: PRRT_kwDOVBg7g863zU6U
Thread: PRRT_kwDOVBg7g863zaJq
Thread: PRRT_kwDOVBg7g863zaJs
Thread: PRRT_kwDOVBg7g863zaJu
Thread: PRRT_kwDOVBg7g863zaJy
Thread: PRR_kwDOVBg7g88AAAABYJsZAA#2
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The success flag of a Copilot result is no exit code, so it no longer gives a shell command's rc.
Every CLI's choice needs rc 0 from the code the command returned; a read still counts a result
reported as failed. Refs intel#807.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A `||` chain whose first part is true skips the rest, and a call with no exit code gives no choice:
both are pinned by regressions on the head (9497553).

Thread: PRRT_kwDOVBg7g863zU6I: already fixed in 9497553
Thread: PRR_kwDOVBg7g88AAAABYJsZAA#1: already fixed in 9497553
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A conditional exit (cd x || exit 1) does not end the scan, nor does one in a subshell. The Copilot
shell rc rule is written into the contract, which gives null for a result with only success.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix, 1 earlier finding still fails · re-reviewed 9497553

Next: @napetrov: fix the 1 must-fix finding and the 1 earlier finding below and answer 7 open threads from exai-bot; then push for a re-review.

Blocking

  • Still fails: earlier finding – true || echo skipped || nproc now gives no probe.
  • Still holds on this head: thread by exai-bot – exec -l && pip install scipy with rc 0 gives no install choice; expected one, since commandless exec -l replaces nothing.
  • Still holds on this head: thread by exai-bot – conda install --download-only scipy with rc 0 gives an install choice.
  • Still holds on this head: thread by exai-bot – pip install --report scipy numpy gives an install choice for scipy, the report filename.
  • Still holds on this head: thread by exai-bot – false && nproc, exit 0; nproc and exec true && nproc each give a cpu probe.
  • Still holds on this head: thread by exai-bot – cat <<EOF with a EOF line then nproc gives a cpu probe, and echo "<<EOF" then nproc gives none.
  • Still holds on this head: thread by exai-bot – echo $((nproc + 1)) gives a cpu probe.
  • Still holds on this head: thread by exai-bot – pip --version > list gives a packages probe.

Verified on 9497553: ✅ changed tests: 223 passed, fail on main · ❌ full suite with main merged in fails at tests/test_repo_compare_interval.py::test_an_end_is_bounded_at_deltas_too_large_to_move_by_one[builtin], tests/test_repo_compare_interval.py::test_an_end_is_bounded_at_deltas_too_large_to_move_by_one[left_to_right], tests/test_repo_compare_interval.py::test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart[compensated] · ✅ merges onto main · 1 mutation tried · 1 edge input tried

3 checks behind it, each with its evidence
Area What holds How
Unresolved calls Changing rc") == 0 in _ran to rc") != 1 fails the suite, so a call with rc None giving a choice is pinned broke the code, tests caught it, edge input (none)
Changed tests tests/test_repo_choices.py, tests/test_repo_probes.py, tests/test_repo_run_events.py and tests/test_batch_ledger_golden.py pass on the head (223 passed) tests on the head
Merge Merges onto main with no conflict merge onto main

Comment thread dokimi/dokimi/repos/choices.py Resolved
A { ...; exit 1; } or a multi-line if may not run its exit, so the scan goes on past it.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
conda-forge::scipy gives no install choice for scipy: the channel
prefix stayed in the name. The name is now the part after the last
'::', then the version, extras and URL are cut as before.

Thread: PRRT_kwDOVBg7g8630I1c
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…_useful

The contract and a changelog fragment say that failed_commands and
first_useful read the Copilot shell rc, so a success-only result
is now null for them.

Refs intel#807

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Pins the strip_tabs branch of the heredoc body scan, and reword the
Copilot rc paragraph of the contract to state the rule.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRR_kwDOVBg7g88AAAABYKG9xw: already fixed in 052e374
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Already fixed in f913010: no exit code, no choice. Pinned.

Fixed in 26a6873: sudo and env wrappers probe their program.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

fix the must-fix and answer 7 threads
Each item is answered in its thread. Fixes: the commits after 9497553.

@napetrov

Copy link
Copy Markdown
Contributor Author

Round 5 on 949755343: head is now 2661870ca5.

  • The 7 exai-bot threads and the review's "1 must-fix" (conda-forge) are fixed, each with a red test first.
  • Review Make dokimi public #1 (rc None) and true || echo skipped || nproc were already fixed on 949755343; f9130103b3 marks them.
  • Copilot rc (Copilot's shell command results should carry a real exit code #807): kept, documented in the contract and a changelog fragment. It changes failed_commands and first_useful for a result with only success.

Comment on lines +211 to +212
if {"--dry-run", "--download-only"} & set(words) or _INFO_FLAGS & set(words): # no install: a dry run, a download, a help request
return None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — exclude uv project operations that do not synchronize the environment. Trigger: rc-0 uv add --no-sync scipy or uv remove --no-sync scipy, with scipy configured under install. Evidence: _manager_rest accepts uv add/remove, this guard does not exclude --no-sync, and the operand loop emits an install/remove choice. uv's --no-sync changes project metadata without installing or uninstalling the package, so exit code 0 does not confirm the environment change that the choice claims. Impact: the persisted timeline falsely reports installation/removal, even when scipy remains absent/present respectively. Exclude this mode specifically for uv project verbs and add negative regressions for both acts. Workaround: omit --no-sync when measuring actual installation/removal, or filter these rows downstream. Severity justification: medium because this is bounded measurement corruption, not a failure of the evaluated command.

Comment on lines +213 to +214
if flat and not conditional and not in_subshell and any(_ends_shell(stage) for stage in stages):
return # what follows an ``exit`` or a replacing ``exec`` that surely runs never runs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — distinguish pipeline-local termination from termination of the parent shell. Trigger: exit 0 | cat; nproc (also exec true | cat; nproc). Evidence: split_script leaves the pipeline in one non-subshell part, and this check returns because its first stage ends its own process; the parent shell actually continues to nproc. Impact: events.jsonl omits a real CPU inspection. Stop scanning only when exit/exec terminates the enclosing shell, and add pipeline regressions alongside the standalone termination tests. Workaround: put nproc in a separate tool call. Severity justification: medium, bounded but material loss of evaluation evidence, without changing command execution.

Comment on lines +245 to +247
m = _opener(line)
if m and not _SHELL_LINE.match(line):
end, expands, head, at = m.group(2), not m.group(1), len(out) - 1, m.start()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — track all heredocs opened by a command, not just the first. Trigger: cat <<A <<B\nignored\nA\nnproc\nB. Evidence: _opener returns only A; after its terminator the scanner resumes ordinary processing and retains nproc from B's body. split_script has no heredoc handling to compensate. The shell executes cat, not nproc. Impact: events.jsonl falsely credits CPU inspection. Queue the opening line's heredoc delimiters and consume each body in shell order, preserving quoted/unquoted expansion rules; add a multiple-heredoc regression. Workaround: use one heredoc per command or an edit tool. Severity justification: medium, bounded but material corruption of persisted probe evidence. This differs from the resolved single-delimiter quoting/indentation issue.

continue
out.append(line)
m = _opener(line)
if m and not _SHELL_LINE.match(line):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — do not assume every heredoc attached to a shell is executable script input. Trigger: bash -c 'cat' <<'EOF'\nnproc\nEOF. Impact: events.jsonl records a CPU probe although bash executes only cat, which prints the literal nproc text. Evidence: _SHELL_LINE matches the opening line regardless of -c, so this branch retains the body and _pipelines interprets its nproc line as a separate command. Shell help/version invocations have the same problem. Preserve heredoc ownership and inspect the body as script only when the shell actually reads its script from stdin; add a negative regression alongside the existing bash <<EOF positive case. Workaround: supply the data through a file instead of a heredoc on a bash -c invocation. Severity justification: medium, bounded but material false inspection evidence; execution itself is unchanged.

Comment on lines +254 to +255
for m in _HEREDOC.finditer(line):
before = line[:m.start()]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — distinguish here-strings from heredoc openers. Trigger: cat <<<EOF\nnproc (two shell commands). Impact: the real nproc inspection is omitted from events.jsonl. Evidence: _HEREDOC matches <<EOF starting at the second < in <<<EOF; _opener accepts it, and _without_heredocs then discards the following nproc line as an unterminated heredoc body. Here-strings have no subsequent body or terminator. Recognize the full redirection operator before selecting heredoc handling and add this as a positive probe regression. Workaround: express the here-string as printf '%s\n' EOF | cat before the nproc command. Severity justification: medium, bounded loss of supported host-inspection evidence rather than a shell execution failure.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same root, more forms. These open a heredoc where there is none, so the next commands are dropped: echo $((1 << n)) then nproc, and a comment # later: cat <<EOF then nproc, give no probe. These are not seen as heredocs, so the body is scanned: cat <<\EOF > s.sh and cat <<'my-eof' > s.sh with a body line nproc each give a cpu probe. #782's probes.py already reads <<< as a here-string.

Comment on lines +378 to +379
views = list(command_words(words))
for k, rest in enumerate(views):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — find the executable after leading redirections. Trigger: >/tmp/probe.out nproc or < /etc/os-release cat. Evidence: command_stages retains the leading operator and target; command_words yields that operator as the program and stops, so _views never exposes nproc or cat. Unlike the choice detector, this call does not first remove redirections. Impact: valid CPU/OS inspections produce no probe event. Normalize the program lookup while retaining input-redirection paths for path probes; cover both leading output and input redirections. Workaround: place redirections after the executable. Severity justification: medium, bounded but material omission of supported host-inspection evidence.

companions = {n: harness_sha256(beside / n) if (beside / n).is_file() else None
for n in opts.profile.bin_companions}
overlay = opts.profile.overlay if arm is not None and parse_arm(arm).base == OVERLAY_ARM else None
inputs = {"candidates": opts.candidates, "probes": opts.probes}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — include the manifest's probe configuration in resume identity. Trigger: run with --manifest-file /tmp/manifest.yaml and no --probes, then change only that manifest's probes (for example, cpu: [nproc] to cpu: []) and resume the same batch. Impact: finished cells retain the old probe rows, while newly scheduled cells use the new configuration, silently mixing incompatible event evidence. Evidence: this fingerprint includes only opts.probes, but run_cell applies ctx.manifest.probes too; _context pins only the source tree, exported eval directory, image and graders, none of which changes when this external file is edited. Workaround: use a new batch or force all affected cells to rerun. Severity: medium, because this is a bounded reproducibility failure in persisted probe evidence. Fingerprint the effective probe mapping (or pin the manifest probe layer separately), and add a resume test that changes only an external manifest's probes.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix, 2 nits · re-reviewed 2661870

Next: @napetrov: fix the 1 must-fix finding and answer 7 open threads from exai-bot; then push for a re-review. 2 nits optional.

Blocking

  • Still holds on this head: thread by exai-bot – uv add --no-sync scipy with rc 0 still gives an install choice for scipy, and uv remove --no-sync scipy a remove choice.
  • Still holds on this head: thread by exai-bot – exit 0 | cat; nproc still gives no cpu probe: nproc runs in the shell, so the probe is missed.
  • Still holds on this head: thread by exai-bot – cat <<A <<B with nproc in the body of B still gives a cpu probe.
  • Still holds on this head: thread by exai-bot – bash -c 'cat' <<'EOF' with nproc in its body still gives a cpu probe.
  • Still holds on this head: thread by exai-bot – cat <<<EOF followed by nproc still gives no cpu probe.
  • Still holds on this head: thread by exai-bot – >/tmp/probe.out nproc still gives no cpu probe.
  • Still holds on this head: thread by exai-bot – execution_record (dokimi/dokimi/repos/runner.py:118) takes only opts.probes, while run_cell also applies ctx.manifest.probes (runner.py:269), so a change to the manifest's probes keeps the same id.

Verified on 2661870: ✅ changed tests: 606 passed, fail on main · ❌ full suite on the head fails at ci.yml:static / schema_compat vs the PR base · ⚠️ conflicts with main · ⚠️ 12 broken variants of the changed code: 10 caught by the tests, 2 not · 2 mutations tried · 7 edge inputs tried

15 checks behind it, each with its evidence
Area What holds How
Changed tests the changed test files pass on the head: 606 passed, including the Copilot rc test and the choice and cpu probe cases tests on the head
Changed tests on main the same test files fail on main at import, since choices.py and probes.py are not there same tests on main
Install choice rules pip install scipy gives an install choice, and |cat, --dry-run, --help, and an rc of None each give none ran it
Exit cutoff guard removing and not conditional from the exit check (dokimi/dokimi/repos/probes.py:213) fails the probe tests broke the code, tests caught it
Manifest error path _parse_probes (dokimi/dokimi/repos/manifest.py:329) returns {} after it records the error, so a raise added after that return never runs ran it
Subprocess syntax error the except in _calls (dokimi/dokimi/repos/probes.py:453) returns False before any re-raise, so a raise added after it never runs ran it
Candidate file loading malformed candidate files ([], an empty map, {sp: null}, a wrong type, an unknown act, truncated YAML) raise ValueError naming --candidates and the problem ran it, edge input (structured)
Name matching names are matched as text: scipy with a combining mark and ß give no choice, plain scipy gives one ran it, edge input (text)
Paths a Windows-style pip.exe path gives the install choice, and a directory given as --candidates raises ValueError naming the flag ran it, edge input (paths)
Repeated names a name given twice gives one choice, and an empty candidate map gives no event ran it, edge input (collections)
Cli file errors repo run with a missing --candidates file exits 1 and names the flag and the path ran it, edge input (cli)
Missing exit code a None rc gives no install choice, and a None time passes through the timeline ran it, edge input (none)
Non-finite times -inf and nan times pass through choice_timeline unchanged ran it, edge input (numbers)
Copilot shell rc a Copilot shell result with only success gets rc None, and failed_commands does not count it, while an rc of 1 is still counted ran it
Shell structure the probes stop at an executed exit or exec, skip false && ..., read <<- terminators and skip $((...)), as the description says ran it

text = _without_heredocs(script)
flat = not _GROUPED.search(text) # a group or a block may hold an exit that does not run: then nothing ends the scan
for part, after, in_subshell, _outer in split_script(text):
conditional = before in ("||", "&&") # the part runs only if the one before it did, or did not, succeed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Must fix: false || exit 1; nproc gives a cpu probe, though the exit always runs and nproc never does.

Input: false || exit 1; nproc. On 2661870 it gives a cpu probe for nproc. Expected: no probe, since false fails, exit 1 runs and the shell stops before nproc. true && exit 0; nproc gives the same probe. conditional is set for every part after || or &&, so the exit does not end the scan even when the status before it is known.

Fix, run on 2661870: the failing case passes; tests and lint are no worse.

Suggested change
conditional = before in ("||", "&&") # the part runs only if the one before it did, or did not, succeed
conditional = before in ("||", "&&") and status is None # the part runs only if the one before it did, or did not, succeed

_SHELL_C = re.compile(r"-[a-z]*c[a-z]*") # ``-c``, ``-lc``: the next word is a script
_LEX_SKIP = frozenset({"(", ")", "{", "}"}) # grouping a PowerShell or shell expression puts around a command
_SEPARATORS = frozenset({"|", "||", "&&", ";", "&"}) # ``|`` splits stages; the rest end a command
_GROUPED = re.compile(r"[{}()]|\b(?:if|then|else|elif|fi|do|done|while|until|case|esac)\b") # a group or a block

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: exit 0; echo $(date); nproc gives a cpu probe: _GROUPED reads any ( as a group and turns the exit cutoff off.

Input: exit 0; echo $(date); nproc. On 2661870 it gives a cpu probe. Expected: none, since exit 0 ends the shell first. _GROUPED matches any parenthesis, so a command substitution switches the cutoff off the way a stray brace does, and the description's known misses name braces, do and done only.

Suggestion: Let $( ) parse like the other substitutions instead of matching ( in _GROUPED, or name it with the braces among the known misses.

("nproc", {"denied": True}), # a denied command ran nothing
("true || echo skipped || nproc", {}),
("command -v nproc", {}), # ``command -v`` only looks the program up
("sudo --help nproc", {}), # ``--help`` prints text and runs nothing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Removing -V from _INFO_OPTIONS leaves every probe test green, and sudo -V nproc then gives a cpu probe.

Input: sudo -V nproc. On 2661870 it gives no probe, since sudo -V prints the version and runs nothing. With -V taken out of _INFO_OPTIONS (dokimi/dokimi/repos/probes.py:53) it gives a cpu probe, and the suite still passes. Expected: the suite fails, as sudo --help nproc already does for --help.

Fix, run on 2661870: the tests now catch the broken variant; tests and lint are no worse.

Suggested change
("sudo --help nproc", {}), # ``--help`` prints text and runs nothing
("sudo --help nproc", {}), # ``--help`` prints text and runs nothing
("sudo -V nproc", {}), # ``-V`` prints the version and runs nothing

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 2661870. The events go only to events.jsonl, and the counters do not read them. But the import and use choices count an Edit's unchanged lines. Also, this branch's probes.py has drifted from #782's, so the two PRs need one plan before either merges. The details are in the line comments.

Agreed with @exai-bot: uv add --no-sync; an exit inside a pipeline; two heredocs on one line; a heredoc on bash -c; here-strings (more cases in its thread); a leading redirection; the manifest's probes missing from the execution id.

Agreed with @vchamarthi: false || exit 1; nproc; _GROUPED reading any (; the missing sudo -V test.

What I ran, at 2661870:

  • make ci-local --offline: 21 passed, 1 failed, 20 skipped; 21,929 tests passed. The failure is schema_compat: run_context.v1.json reached main with #785 after this branch's base, so it does not come from this PR and goes away on a rebase.
  • A repro through convert, with_probes, with_choices and first_useful for each line comment, and the same probe inputs through #782's probes.py.
  • 60,000 random shell strings through with_probes and with_choices: no exception.

Checked, no issues

  • Resume identity: the --candidates and --probes digests go into the execution id, so a resume with other files is refused. The golden id moved only because the fixture binary changed.
  • Exit-code rule for choices: a denied or failed call, or one with no result, gives no choice. Only && chains count, as the contract says.
  • Schema and CLI: repo_manifest.v1.json only widens (probes). The CLI snapshot and the flag allowlist are updated, and the new files have the SPDX header.

Comment on lines +243 to +249
def _added_text(args: Any) -> str:
"""The text an edit tool adds: the new side of each replacement, or the whole file of a write (choices.py)."""
if not isinstance(args, dict):
return ""
if isinstance(args.get("edits"), list): # MultiEdit
return "\n".join(str(e.get("new_string", "")) for e in args["edits"] if isinstance(e, dict))
return str(args.get("new_string") or args.get("content") or args.get("new_source") or "")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: An Edit's unchanged lines count as added text, so they give false import and use choices.

Claude's Edit repeats the lines around a change in new_string. With sklearnex: {import: [sklearnex], use: [patch_sklearn]}, this Edit gives an import and a use choice for sklearnex, though the agent added only numpy:

old_string: "from sklearnex import patch_sklearn\npatch_sklearn()\n"
new_string: the same two lines + "import numpy\n"

An Edit that deletes a line gives choices for the lines kept around it too. The contract says "never what the edit replaced"; test_only_the_added_side_of_an_edit_counts passes only because its two sides share no line.

Suggestion: count only the lines of new_string that old_string does not have (per edit of a MultiEdit, e.g. with a Counter of the old lines), and add a test where both sides share a line.

return found


def command_stages(part: str) -> tuple[list[list[str]], list[str]]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch's probes.py is no longer #782's: the two PRs now carry two different implementations.

The description says this diff holds #782's commit d3360cd until #782 merges, but both branches have rewritten probes.py since (about 490 changed lines between them). On 33 inputs from the review threads they disagree on 15: #782 handles if true; then nproc; fi, cat <<<EOF and >/tmp/o nproc; this branch handles false && nproc, exit 0; nproc and env FOO=1 nproc. choices.py imports command_stages, command_words and split_redirects, which #782 does not have.

Suggestion: keep one copy: fold #782 into this PR and close it, or take the probes out of here and rebase onto #782 once it merges (AGENTS.md: PRs are not stacked).

Comment on lines +386 to +405
def command_words(words: list[str]) -> Iterator[list[str]]:
"""The words of the command a simple command runs, from its program on, then those of the command each wrapper
(``time``, ``sudo -n``, ``nice -n 5``...) runs: its options are not the program."""
i = 0
while i < len(words):
while i < len(words) and _ASSIGN.match(words[i]): # NAME=value words set variables for the command after them
i += 1
if i >= len(words):
return
yield words[i:]
wrapper = _program(words[i])
if wrapper not in _WRAPPERS:
return
i += 1
while i < len(words) and words[i].startswith("-") and len(words[i]) > 1:
if words[i] in _INFO_OPTIONS or (wrapper == "command" and words[i] == "-v"): # looks up or prints: runs nothing
return
i += 2 if words[i] in _WRAPPER_VALUES.get(wrapper, ()) else 1


Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A probe after a shell keyword is missed: if true; then nproc; fi, for x in 1; do nproc; done and ! nproc give no probe.

command_words takes then, do or ! for the program. So if command -v nvidia-smi; then nvidia-smi; fi, a common way to check for a GPU, gives no gpu probe. #782 fixed this in 57508d3 (a keyword set and a test); the fix is not on this branch.

Suggestion: skip leading if, then, else, elif, do, while, until and ! in command_words, and take #782's test.

event["rc"] = code if isinstance(code, int) else (None if data.get("success") is None
else 0 if data.get("success") else 1)
failed = _failed_read(str(event["text"]), event["rc"])
event["rc"] = code if isinstance(code, int) else None # the code the command returned, or none: no success fallback

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Copilot command with no exit code now gives first_useful the verdict never, so "not measured" counts as a miss.

With rc null for a result that has only success, useful.hits never matches it and first_useful returns never, which first_useful_rate counts. A copilot-json make test with result {"success": true} was a hit before this PR and is never now; a {"success": false} one no longer counts in failed_commands, which reads 0.

Suggestion: give first_useful no verdict (like unobserved) when a matching command has rc null, and say in the fragment that older Copilot runs are not comparable on these two.


_READERS = frozenset({"cat", "head", "tail", "less", "more", "nl", "sed", "grep", "egrep", "fgrep", "awk", "tac",
"strings", "wc", "rg"})
_WRAPPERS = frozenset({"time", "nohup", "sudo", "command", "exec", "builtin", "nice", "env"}) # each runs the command after it

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timeout, uv run, stdbuf and doas are not wrappers here, so the command after them gives no probe or choice.

timeout 10 nvidia-smi, uv run python -c 'import os; os.cpu_count()' and bash -o pipefail -c nproc give no probe; timeout 600 pip install scipy gives no install choice. Agents wrap long commands in timeout often. dokimi/context/shell.py::strip_wrappers already skips timeout 300 and uv run, and repos/environment/risks.py has timeout in its _WRAPPERS.

Suggestion: add timeout (with its duration), stdbuf, doas and uv run to the wrappers, or reuse strip_wrappers; find -c after other options (bash -o X -c, python -W X -c).


def _package_call(words: list[str]) -> tuple[str, list[str]] | None:
"""The act and the package names of a package manager's install, add, uninstall or remove call."""
if {"--dry-run", "--download-only"} & set(words) or _INFO_FLAGS & set(words): # no install: a dry run, a download, a help request

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some short spellings give a wrong install choice, or none.

  • conda install -d scipy and mamba install -d scipy (-d is --dry-run) give an install choice.
  • uv pip install -n scipy and poetry add -n scipy give none: -n (--no-cache, --no-interaction there) is in _VALUE_OPTIONS, so scipy is read as its value.
  • py -m pip install scipy gives none: this _PYTHON has no py, while probes._PYTHON has it.

Suggestion: read -d as a dry run for conda and mamba, take -n as a value option for conda only, and reuse probes._PYTHON.

Comment on lines +268 to +270
# Probe events go to events.jsonl only: the counters, first_useful and skill usage score the trace's events
probed = with_probes(transcript.events, probes_for(ctx.manifest.probes, opts.probes))
shown = with_choices(probed, opts.candidates or {}, transcript.added_text) # choice events: choices.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An exception in the probe or choice matching fails the cell's grade.

with_probes and with_choices run inside the grading try of run_cell, so an error there sets grade to failed_outcome("host error after the agent ran ...") and skips events.jsonl and answer.txt. For example, python -c '1+1+...' with 20,000 terms makes ast.parse in probes._calls raise RecursionError, which it does not catch. No score reads these events, so they should never cost a cell its grade.

Suggestion: catch RecursionError and MemoryError in _calls, and write the trace's own events when the matching raises, as _process_or_null does for the metrics.

if part.startswith("("): # a subshell: its exit code is not the command's own
continue
stages, _inner = command_stages(part)
if len(stages) > 1: # a pipeline reports its last stage's exit code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: should pip install scipy 2>&1 | tail -5 give a choice marked "exit code unknown", instead of no event?

This is a common install form in Claude Code traces, and today it leaves nothing in events.jsonl, so a reader cannot tell "not measured" from "not installed". The command's output usually holds pip's Successfully installed scipy-... line, which could confirm it. If you keep it as is, please say in the contract that no install choice does not mean the agent did not install it, as it does for Copilot's edits.

Comment on lines +110 to +112
def choice_timeline(events: Sequence[dict[str, Any]]) -> list[tuple[float | None, str, str]]:
"""The ``(t, candidate, act)`` of each choice event, in the order the trace gives them."""
return [(e.get("t"), str(e["candidate"]), str(e["act"])) for e in events if e.get("kind") == "choice"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: choice_timeline is used only by a test. Nothing in the package calls it. Suggestion: drop it, or use it where a report reads the choices.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Copilot's shell command results should carry a real exit code

6 participants