Repository navigation
Conversation
A command that runs nproc, lscpu, nvidia-smi, pip list, printenv, a python -c one-liner calling os.cpu_count(), or a PowerShell Get-CimInstance Win32_Processor gets a probe event after its command event, one per pattern it matches. Patterns match the program the shell runs, never argument text, so grep nproc README is no probe. The defaults ship in repos/probes.py. A manifest's probes key and repo run --probes FILE replace a target's list; [] turns it off. Existing event kinds are unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e or removal Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The harness reports one exit code per command, so an install after `||`, `;` or a pipe may not have run, or may have failed, while the command still exits 0. Such installs now give no choice. A `--dry-run` installs nothing and gives none. An import after `;` on the same line is seen, and `uv add` and `uv remove` name their package like `uv pip` does. Contract section 4.1 says which installs count and which do not. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi repo run` parameters changed:
+ added option `--candidates`
+ added option `--probes`Updated on every push; the |
|
🔍 OpenCodeReview found 8 issue(s) in this PR. CI at By severity: 8 medium
Review details
|
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #787 +/- ##
==========================================
- Coverage 95.76% 95.75% -0.01%
==========================================
Files 416 418 +2
Lines 55624 56172 +548
Branches 9553 9681 +128
==========================================
+ Hits 53267 53789 +522
- Misses 1456 1472 +16
- Partials 901 911 +10
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 2991eb7. The events are wired in the right place: they go to events.jsonl only, and the counters and first_useful do not read them. The matching behind them is textual, and its holes write false choice and probe events. The merge waits for the bot points below, which I reproduced, and for the risks listed here.
What I ran, on a Linux review host at 2991eb7:
make ci-local: 31 passed, 2 failed, 14 skipped; 20,570 tests passed; changed-line coverage 92%. The 2 failures are the gitleaks install (HTTP 504) and the commit scan that needs it, which do not depend on this PR.- Reproductions for the code findings, with no model or network call.
- A mutation run over
choices.pyandprobes.pyagainst their test files: 33 of 59probes.pymutants and about tenchoices.pymutants pass the tests.
Findings that cannot be anchored inline:
--probesand--candidatesare recorded nowhere in a cell or batch.execution_record(runner.py:107, outside this diff) takes the profile, tools, delegation and binary digests only. Resume a batch with another--candidatesor--probesfile:check_batch_executionsdoes not refuse it, finished cells are skipped, and the batch holdsevents.jsonlfiles built under two configurations with nothing to tell them apart. Suggestion: add both files' digests to the execution record, or refuse a resume whose files differ.- The tests pin the matching rules thinly. Mutation check on
choices.py: removingnot event.get("denied")(choices.py:97), the(?!\w)afterfrom name, thestartswith(name + ".")submodule rule, the(?<![\w.])lookbehind of_calls, and the[in_REQUIREMENT_ENDall passtest_repo_choices.py. Onprobes.py: droppingsudo,nohup,execornicefrom_WRAPPERS, shrinking_READERStocat,head,grep, letting_in_ordermatch non-consecutive stages, dropping the(?<![\w.])lookbehind or the\s*\(call requirement of the Python probe, and dropping.exeand backslash handling all pass. Nothing exercisesrunner.py:267-269orrepo_run.pyeither: a change that dropsopts.candidatesoropts.probespasses the suite. Add one case per rule, plus arun_celltest that checksevents.jsonlholds a choice and a probe row.
Open bot points I confirm:
- I agree with exai-bot that an unresolved call counts as a completed one. New evidence:
pip install scipywithrcNonegives aninstallchoice and withrc1 none; the suggestedrc == 0would also drop every Copilot shell command, sincecli_traces.py:745writesrcasNonefor good. - I agree with exai-bot that a pipe without spaces gives an install choice. New evidence:
pip install scipy |catwithrc0 gives aninstallchoice forscipy,pip install scipy | catgives none. - I agree with exai-bot that a semicolon in a comment starts an import. New evidence: added text
# example; import sklearnexgivesimportanduseforsklearnex;x = 1 # import sklearnexgivesuseonly. - I agree with exai-bot that help requests count as installs. New evidence:
pip install scipy --helpandpip install -h scipywithrc0 each give aninstallchoice forscipy. - I agree with exai-bot that an option value is read as a package. New evidence:
pip install --target scipy numpy,-r scipy,-c scipy numpyand--index-url scipy numpyeach give aninstallchoice forscipy. - I agree with exai-bot that a heredoc body is read as commands:
cat <<'EOF' > s.shfollowed by a linenprocemits a cpu probe although onlycatran. - I agree with exai-bot that quoting is ignored when substitutions are extracted:
echo '$(nproc)'prints a literal and still emits a cpu probe. - I agree with exai-bot that nested substitutions are missed:
make -j$(nproc --ignore=$(printf 1))runsnprocand emits no probe. - I agree with exai-bot that a quoted pipe splits a command:
echo '|' nprocruns onlyechoand emits a cpu probe. - I agree with exai-bot that wrapper options hide the wrapped program:
sudo -n nprocemits no probe. - I agree with exai-bot that a search pattern counts as a file operand:
grep -e /proc/cpuinfo READMEemits the/proc/cpuinfoprobe. - I agree with exai-bot that a string in a Python one-liner counts as a call:
python -c 'print("os.cpu_count()")'emits a probe.
Choice events: - `pip install X |cat` and a quoted `'|'` no longer read as a pipeline by whitespace alone; quoting decides. - `--help`, `-h`, `--version` give no install or removal choice. - A value of `--target`, `-r`, `-c`, `--index-url`, `--name` ... is not a package operand. - `sudo`, `time`, `nice` before the manager no longer hide the call. - An import is read from code: a `#` comment is dropped, a `\` continuation is joined, `if x: import m` counts. Probe events: - A heredoc body is data unless a shell reads it. - `$( )` and backticks inside single quotes or escaped are text; nested substitutions are walked. - A quoted `|` is not a stage break. - Wrapper options (`sudo -n`, `nice -n 5`, `time -p`) are skipped before the wrapped program. - A path probe matches a file operand, not a search pattern or an option value. - A `py:NAME` probe matches a call in the parsed `-c` code, not a string or a comment. Tests pin each rule, the denied-call check and the `run_cell` wiring of `--candidates` and `--probes`. The contract names Copilot's edit tools among those that give no import or use choice. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
Pushed 1 commit to Changes
Needs @napetrov
Threads and comments
Not this PR's
Verified (12)
|
- restore the `_package` definition: `pip install 'scipy >= 1.11'` raised NameError - `exec pip install x` gives its choice; later `&&` parts still give none - redirection operators and targets are no package or file operand; `< file` still reads - a string literal no longer hides or fakes an import on its line - `uv --quiet add x` gives its install choice - `sudo bash -c 'nproc'` and an unquoted heredoc's `$(nproc)` give their probe - `echo \| nproc` runs no pipeline - grep/rg/sed/awk long pattern options and `-f` no longer turn a file into a pattern - `exec true && pip install x` gives no choice; a quoted `'>'` is a word, not a redirection - a command after `false ||` gives its probe; only `true ||` and `: ||` skip it - `FOO=1 exec true && pip install x` gives no choice - a quoted `'1'` before a redirection is a word, not a descriptor Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
🟡 Waiting on @napetrov · 1 commit on Next: @napetrov: answer the 2 questions below. Changes
Needs @napetrov
Threads and comments
4 threads left as they are, and why
Verified on the new head: ✅ 6/6 tests fail before the fix, pass after · ✅ full suite with main merged in: 21947 passed, 3 failures that main has too · What each test shows
|
…rgets as the shell does Thread: PRRT_kwDOVBg7g863v6ZT Thread: PRRT_kwDOVBg7g863v6ZX Thread: PRRT_kwDOVBg7g863v6ZZ Thread: PRRT_kwDOVBg7g863v6Zd Thread: PRRT_kwDOVBg7g863v6Zi Thread: PRRT_kwDOVBg7g863v6Zl Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863nhD_: already fixed in 5705149 (pipe without spaces) Thread: PRRT_kwDOVBg7g863nhEB: already fixed in 5705149 (semicolon in a comment) Thread: PRRT_kwDOVBg7g863nhEC: already fixed in 5705149 (help and version requests) Thread: PRRT_kwDOVBg7g863nhEE: already fixed in 5705149 (option values are not packages) Thread: PRRT_kwDOVBg7g863nhEG: already fixed in 5705149 (heredoc body is data) Thread: PRRT_kwDOVBg7g863nhEI: already fixed in 5705149 (quoted substitutions) Thread: PRRT_kwDOVBg7g863nhEJ: already fixed in 5705149 (nested substitutions) Thread: PRRT_kwDOVBg7g863nhEM: already fixed in 5705149 (quoted pipe) Thread: PRRT_kwDOVBg7g863nhEN: already fixed in 5705149 (wrapper options) Thread: PRRT_kwDOVBg7g863nhEP: already fixed in 5705149 (grep pattern options) Thread: PRRT_kwDOVBg7g863nhES: already fixed in 5705149 (python -c strings) Thread: PRRT_kwDOVBg7g863o6RG: already fixed in 5705149 (contract names Copilot edit tools) Thread: PRRT_kwDOVBg7g863sWEA: already fixed in 5705149 (exec replaces the shell) Thread: PRRT_kwDOVBg7g863sWEF: already fixed in 5705149 (comment and string scanner) Thread: PRRT_kwDOVBg7g863sWEK: already fixed in 5705149 (redirect targets) Thread: PRRT_kwDOVBg7g863sWEN: already fixed in 5705149 (wrapped shell scripts) Thread: PRRT_kwDOVBg7g863sWEV: already fixed in 5705149 (unquoted heredoc expansions) Thread: PRRT_kwDOVBg7g863sWEd: already fixed in 5705149 (escaped operators) Thread: PRRT_kwDOVBg7g863sWEi: already fixed in 5705149 (output redirection is no read) Thread: PRRT_kwDOVBg7g863sWEo: already fixed in 5705149 (grep --regexp and -f) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A call with no result (rc None: a cut-off call, or a Copilot command, which writes no rc) and a failed one both take no choice. The fake harness gets a result flag for the events test, so its golden cell ids move with its bytes. Thread: PRRT_kwDOVBg7g863nhD6 Thread: IC_kwDOVBg7g88AAAABmzVvzA#1 Thread: IC_kwDOVBg7g88AAAABm4kUeA#1 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The execution id takes the digest of each given file's parsed content, so a resume with another file is another execution and check_batch_executions refuses it. An unset file adds no key, so the ids of runs without one stay. Thread: IC_kwDOVBg7g88AAAABmzVvzA#2 Thread: PRR_kwDOVBg7g88AAAABYEmcQg#1 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… passed A bracketed extra on the package name and a -c that follows a script name were not pinned by any test: dropping either left the suite green. Thread: PRR_kwDOVBg7g88AAAABYEmcQg#2 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
They were appended to the opening line, so a comment after the operator hid them: cat <<EOF # c, then $(nproc), gave no probe. They now go before <<. Thread: PRRT_kwDOVBg7g863v6Zd Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Inside "..." the second backslash of `\\` was read as an escape of the closing quote, so a pipe after it was lost (probe) or a pipeline read as one stage (choice). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…idate and probe digests The execution id hashes the parsed --candidates and --probes files, so a resume with other ones is refused; the _ran docstring says a Copilot call with no exit code or success flag is rc None. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…er them Thread: PRR_kwDOVBg7g88AAAABYEmcQg Thread: IC_kwDOVBg7g88AAAABmzVvzA Thread: IC_kwDOVBg7g88AAAABm4kUeA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 5 must-fixes · reviewed 5705149
Next: @napetrov: fix the 5 must-fix findings and answer 1 open thread from exai-bot; then push for a re-review.
Blocking
- Still holds on this head: thread by exai-bot –
pip install scikit-learn-intelexfrom a Claude trace with no tool_result (rcNone) still gives aninstallchoice;_ranacceptsrcNone.
Must-fix
-
dokimi/dokimi/repos/probes.py:573–env FOO=1 python train.pygives anenvprobe, andenv FOO=1 nprocgives nocpuprobe:envis not in_WRAPPERS.Input:
env FOO=1 python train.pyandenv -i python train.py. Gives a probe with targetenv, patternenv;env FOO=1 nprocgivesenvand nocpu. Expected: no probe for the first two, since the shell runspython, andcpufor the last. Contract §4.1: patterns match the program the shell runs.printenvand bareenvstill count.Also at:
dokimi/dokimi/repos/probes.py:576.Suggestion: Treat
envas a wrapper when it is followed by a program (skip itsNAME=valuewords and options), and keep theenvprobe for a bareenv,env | grep Xandenv > f.Patch, run on
5705149: the failing case passes; tests and lint are no worse.Patch
--- a/dokimi/dokimi/repos/probes.py +++ b/dokimi/dokimi/repos/probes.py @@ -49,10 +49,10 @@ _READERS = frozenset({"cat", "head", "tail", "less", "more", "nl", "sed", "grep", "egrep", "fgrep", "awk", "tac", "strings", "wc", "rg"}) -_WRAPPERS = frozenset({"time", "nohup", "sudo", "command", "exec", "builtin", "nice"}) # each runs the command after it +_WRAPPERS = frozenset({"time", "nohup", "sudo", "command", "exec", "builtin", "nice", "env"}) # each runs the command after it _WRAPPER_VALUES = {"sudo": frozenset({"-u", "-g", "-h", "-p", "-r", "-t", "-C", "-D", "-R", "-T", "-U"}), "nice": frozenset({"-n", "--adjustment"}), "exec": frozenset({"-a"}), - "time": frozenset({"-f", "-o", "--format", "--output"})} # options that take the next word + "time": frozenset({"-f", "-o", "--format", "--output"}), "env": frozenset({"-u", "-C", "--unset", "--chdir"})} # take the next word _PATTERN_READERS = frozenset({"grep", "egrep", "fgrep", "rg", "sed", "awk"}) # the first operand is a pattern or script _GREP_VALUES = {"-e", "-f", "-m", "-A", "-B", "-C", "--regexp", "--file", "--max-count", "--after-context", "--before-context", "--context"} @@ -338,8 +338,10 @@ def _views(words: list[str]) -> Iterator[Stage]: """The command a simple command runs: its own program, then the command each wrapper (``time``, ``sudo``...) runs.""" - for rest in command_words(words): - yield _head(rest[0], rest[1:]) + views = list(command_words(words)) + for k, rest in enumerate(views): + if k == len(views) - 1 or _program(rest[0]) != "env": # ``env FOO=1 nproc`` runs ``nproc``, not a probe of ``env`` + yield _head(rest[0], rest[1:]) def command_words(words: list[str]) -> Iterator[list[str]]:
Verified on 5705149: ✅ changed tests: 192 passed, fail on main · ✅ full suite with main merged in: 21947 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 4 edge inputs tried
6 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Changed tests | tests/test_repo_choices.py, tests/test_repo_probes.py and tests/test_repo_run_events.py pass on the head (192 passed) and fail to import on main |
tests on the head |
| Merge | Merges onto main with no conflict | merge onto main |
| Denied calls | Removing not event.get("denied") from _ran fails test_a_denied_install_gives_no_choice |
broke the code, tests caught it |
| Probe layers | A manifest layer replaces a target's defaults, a later --probes layer replaces it again, and [] turns the target off without touching other targets |
ran it, edge input (collections) |
| Text input | Empty, whitespace, ß, combining marks, U+2028 and a 200000-character command text go through with_probes and with_choices without raising |
ran it, edge input (text) |
| Config file paths | A missing --probes file, a directory given as --candidates, and a non-UTF-8 file each give ValueError naming the flag and path |
ran it, edge input (paths) |
|
Fixed in 5258517: the same rc 0 rule as #1 above; pinned by a test. Not changed. Probes read text, not exit codes per part:
|
…port give no false choice exec -l has no program and does not end the shell, so the install after it counts. conda's --download-only fetches packages without linking them, and pip's --report takes a file name as its value, so the next word is not a package. Thread: PRRT_kwDOVBg7g863zaJY Thread: PRRT_kwDOVBg7g863zaJf Thread: PRRT_kwDOVBg7g863zaJk Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A denied command runs nothing. command -v and a wrapper's --help or --version run no program, and env runs the program after its NAME=value words, so no probe is of env there. A part after a failed && or a succeeded || never runs, nor does anything after an exit or an exec with a program. A heredoc ends only at its exact delimiter, and only an unquoted << opens one. $(( )) is arithmetic, not a substitution. A redirection's target is no operand of a command pattern. Thread: PRRT_kwDOVBg7g863zU5_ Thread: PRRT_kwDOVBg7g863zU6O Thread: PRRT_kwDOVBg7g863zU6U Thread: PRRT_kwDOVBg7g863zaJq Thread: PRRT_kwDOVBg7g863zaJs Thread: PRRT_kwDOVBg7g863zaJu Thread: PRRT_kwDOVBg7g863zaJy Thread: PRR_kwDOVBg7g88AAAABYJsZAA#2 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The success flag of a Copilot result is no exit code, so it no longer gives a shell command's rc. Every CLI's choice needs rc 0 from the code the command returned; a read still counts a result reported as failed. Refs intel#807. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A `||` chain whose first part is true skips the rest, and a call with no exit code gives no choice: both are pinned by regressions on the head (9497553). Thread: PRRT_kwDOVBg7g863zU6I: already fixed in 9497553 Thread: PRR_kwDOVBg7g88AAAABYJsZAA#1: already fixed in 9497553 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A conditional exit (cd x || exit 1) does not end the scan, nor does one in a subshell. The Copilot shell rc rule is written into the contract, which gives null for a result with only success. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix, 1 earlier finding still fails · re-reviewed 9497553
Next: @napetrov: fix the 1 must-fix finding and the 1 earlier finding below and answer 7 open threads from exai-bot; then push for a re-review.
Blocking
- Still fails: earlier finding –
true || echo skipped || nprocnow gives no probe. - Still holds on this head: thread by exai-bot –
exec -l && pip install scipywith rc 0 gives no install choice; expected one, since commandlessexec -lreplaces nothing. - Still holds on this head: thread by exai-bot –
conda install --download-only scipywith rc 0 gives an install choice. - Still holds on this head: thread by exai-bot –
pip install --report scipy numpygives an install choice forscipy, the report filename. - Still holds on this head: thread by exai-bot –
false && nproc,exit 0; nprocandexec true && nproceach give acpuprobe. - Still holds on this head: thread by exai-bot –
cat <<EOFwith aEOFline thennprocgives acpuprobe, andecho "<<EOF"thennprocgives none. - Still holds on this head: thread by exai-bot –
echo $((nproc + 1))gives acpuprobe. - Still holds on this head: thread by exai-bot –
pip --version > listgives apackagesprobe.
Verified on 9497553: ✅ changed tests: 223 passed, fail on main · ❌ full suite with main merged in fails at tests/test_repo_compare_interval.py::test_an_end_is_bounded_at_deltas_too_large_to_move_by_one[builtin], tests/test_repo_compare_interval.py::test_an_end_is_bounded_at_deltas_too_large_to_move_by_one[left_to_right], tests/test_repo_compare_interval.py::test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart[compensated] · ✅ merges onto main · 1 mutation tried · 1 edge input tried
3 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Unresolved calls | Changing rc") == 0 in _ran to rc") != 1 fails the suite, so a call with rc None giving a choice is pinned |
broke the code, tests caught it, edge input (none) |
| Changed tests | tests/test_repo_choices.py, tests/test_repo_probes.py, tests/test_repo_run_events.py and tests/test_batch_ledger_golden.py pass on the head (223 passed) |
tests on the head |
| Merge | Merges onto main with no conflict | merge onto main |
A { ...; exit 1; } or a multi-line if may not run its exit, so the scan goes on past it.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
conda-forge::scipy gives no install choice for scipy: the channel prefix stayed in the name. The name is now the part after the last '::', then the version, extras and URL are cut as before. Thread: PRRT_kwDOVBg7g8630I1c Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…_useful The contract and a changelog fragment say that failed_commands and first_useful read the Copilot shell rc, so a success-only result is now null for them. Refs intel#807 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Pins the strip_tabs branch of the heredoc body scan, and reword the Copilot rc paragraph of the contract to state the rule. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRR_kwDOVBg7g88AAAABYKG9xw: already fixed in 052e374 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
Round 5 on
|
| if {"--dry-run", "--download-only"} & set(words) or _INFO_FLAGS & set(words): # no install: a dry run, a download, a help request | ||
| return None |
There was a problem hiding this comment.
Must-fix — exclude uv project operations that do not synchronize the environment. Trigger: rc-0 uv add --no-sync scipy or uv remove --no-sync scipy, with scipy configured under install. Evidence: _manager_rest accepts uv add/remove, this guard does not exclude --no-sync, and the operand loop emits an install/remove choice. uv's --no-sync changes project metadata without installing or uninstalling the package, so exit code 0 does not confirm the environment change that the choice claims. Impact: the persisted timeline falsely reports installation/removal, even when scipy remains absent/present respectively. Exclude this mode specifically for uv project verbs and add negative regressions for both acts. Workaround: omit --no-sync when measuring actual installation/removal, or filter these rows downstream. Severity justification: medium because this is bounded measurement corruption, not a failure of the evaluated command.
| if flat and not conditional and not in_subshell and any(_ends_shell(stage) for stage in stages): | ||
| return # what follows an ``exit`` or a replacing ``exec`` that surely runs never runs |
There was a problem hiding this comment.
Must-fix — distinguish pipeline-local termination from termination of the parent shell. Trigger: exit 0 | cat; nproc (also exec true | cat; nproc). Evidence: split_script leaves the pipeline in one non-subshell part, and this check returns because its first stage ends its own process; the parent shell actually continues to nproc. Impact: events.jsonl omits a real CPU inspection. Stop scanning only when exit/exec terminates the enclosing shell, and add pipeline regressions alongside the standalone termination tests. Workaround: put nproc in a separate tool call. Severity justification: medium, bounded but material loss of evaluation evidence, without changing command execution.
| m = _opener(line) | ||
| if m and not _SHELL_LINE.match(line): | ||
| end, expands, head, at = m.group(2), not m.group(1), len(out) - 1, m.start() |
There was a problem hiding this comment.
Must-fix — track all heredocs opened by a command, not just the first. Trigger: cat <<A <<B\nignored\nA\nnproc\nB. Evidence: _opener returns only A; after its terminator the scanner resumes ordinary processing and retains nproc from B's body. split_script has no heredoc handling to compensate. The shell executes cat, not nproc. Impact: events.jsonl falsely credits CPU inspection. Queue the opening line's heredoc delimiters and consume each body in shell order, preserving quoted/unquoted expansion rules; add a multiple-heredoc regression. Workaround: use one heredoc per command or an edit tool. Severity justification: medium, bounded but material corruption of persisted probe evidence. This differs from the resolved single-delimiter quoting/indentation issue.
| continue | ||
| out.append(line) | ||
| m = _opener(line) | ||
| if m and not _SHELL_LINE.match(line): |
There was a problem hiding this comment.
Must-fix — do not assume every heredoc attached to a shell is executable script input. Trigger: bash -c 'cat' <<'EOF'\nnproc\nEOF. Impact: events.jsonl records a CPU probe although bash executes only cat, which prints the literal nproc text. Evidence: _SHELL_LINE matches the opening line regardless of -c, so this branch retains the body and _pipelines interprets its nproc line as a separate command. Shell help/version invocations have the same problem. Preserve heredoc ownership and inspect the body as script only when the shell actually reads its script from stdin; add a negative regression alongside the existing bash <<EOF positive case. Workaround: supply the data through a file instead of a heredoc on a bash -c invocation. Severity justification: medium, bounded but material false inspection evidence; execution itself is unchanged.
| for m in _HEREDOC.finditer(line): | ||
| before = line[:m.start()] |
There was a problem hiding this comment.
Must-fix — distinguish here-strings from heredoc openers. Trigger: cat <<<EOF\nnproc (two shell commands). Impact: the real nproc inspection is omitted from events.jsonl. Evidence: _HEREDOC matches <<EOF starting at the second < in <<<EOF; _opener accepts it, and _without_heredocs then discards the following nproc line as an unterminated heredoc body. Here-strings have no subsequent body or terminator. Recognize the full redirection operator before selecting heredoc handling and add this as a positive probe regression. Workaround: express the here-string as printf '%s\n' EOF | cat before the nproc command. Severity justification: medium, bounded loss of supported host-inspection evidence rather than a shell execution failure.
There was a problem hiding this comment.
Same root, more forms. These open a heredoc where there is none, so the next commands are dropped: echo $((1 << n)) then nproc, and a comment # later: cat <<EOF then nproc, give no probe. These are not seen as heredocs, so the body is scanned: cat <<\EOF > s.sh and cat <<'my-eof' > s.sh with a body line nproc each give a cpu probe. #782's probes.py already reads <<< as a here-string.
| views = list(command_words(words)) | ||
| for k, rest in enumerate(views): |
There was a problem hiding this comment.
Must-fix — find the executable after leading redirections. Trigger: >/tmp/probe.out nproc or < /etc/os-release cat. Evidence: command_stages retains the leading operator and target; command_words yields that operator as the program and stops, so _views never exposes nproc or cat. Unlike the choice detector, this call does not first remove redirections. Impact: valid CPU/OS inspections produce no probe event. Normalize the program lookup while retaining input-redirection paths for path probes; cover both leading output and input redirections. Workaround: place redirections after the executable. Severity justification: medium, bounded but material omission of supported host-inspection evidence.
| companions = {n: harness_sha256(beside / n) if (beside / n).is_file() else None | ||
| for n in opts.profile.bin_companions} | ||
| overlay = opts.profile.overlay if arm is not None and parse_arm(arm).base == OVERLAY_ARM else None | ||
| inputs = {"candidates": opts.candidates, "probes": opts.probes} |
There was a problem hiding this comment.
Must-fix — include the manifest's probe configuration in resume identity. Trigger: run with --manifest-file /tmp/manifest.yaml and no --probes, then change only that manifest's probes (for example, cpu: [nproc] to cpu: []) and resume the same batch. Impact: finished cells retain the old probe rows, while newly scheduled cells use the new configuration, silently mixing incompatible event evidence. Evidence: this fingerprint includes only opts.probes, but run_cell applies ctx.manifest.probes too; _context pins only the source tree, exported eval directory, image and graders, none of which changes when this external file is edited. Workaround: use a new batch or force all affected cells to rerun. Severity: medium, because this is a bounded reproducibility failure in persisted probe evidence. Fingerprint the effective probe mapping (or pin the manifest probe layer separately), and add a resume test that changes only an external manifest's probes.
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix, 2 nits · re-reviewed 2661870
Next: @napetrov: fix the 1 must-fix finding and answer 7 open threads from exai-bot; then push for a re-review. 2 nits optional.
Blocking
- Still holds on this head: thread by exai-bot –
uv add --no-sync scipywith rc 0 still gives an install choice forscipy, anduv remove --no-sync scipya remove choice. - Still holds on this head: thread by exai-bot –
exit 0 | cat; nprocstill gives nocpu probe:nprocruns in the shell, so the probe is missed. - Still holds on this head: thread by exai-bot –
cat <<A <<Bwithnprocin the body ofBstill gives acpu probe. - Still holds on this head: thread by exai-bot –
bash -c 'cat' <<'EOF'withnprocin its body still gives acpu probe. - Still holds on this head: thread by exai-bot –
cat <<<EOFfollowed bynprocstill gives nocpu probe. - Still holds on this head: thread by exai-bot –
>/tmp/probe.out nprocstill gives nocpu probe. - Still holds on this head: thread by exai-bot –
execution_record(dokimi/dokimi/repos/runner.py:118) takes onlyopts.probes, whilerun_cellalso appliesctx.manifest.probes(runner.py:269), so a change to the manifest's probes keeps the sameid.
Verified on 2661870: ✅ changed tests: 606 passed, fail on main · ❌ full suite on the head fails at ci.yml:static / schema_compat vs the PR base ·
15 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Changed tests | the changed test files pass on the head: 606 passed, including the Copilot rc test and the choice and cpu probe cases |
tests on the head |
| Changed tests on main | the same test files fail on main at import, since choices.py and probes.py are not there |
same tests on main |
| Install choice rules | pip install scipy gives an install choice, and |cat, --dry-run, --help, and an rc of None each give none |
ran it |
| Exit cutoff guard | removing and not conditional from the exit check (dokimi/dokimi/repos/probes.py:213) fails the probe tests |
broke the code, tests caught it |
| Manifest error path | _parse_probes (dokimi/dokimi/repos/manifest.py:329) returns {} after it records the error, so a raise added after that return never runs |
ran it |
| Subprocess syntax error | the except in _calls (dokimi/dokimi/repos/probes.py:453) returns False before any re-raise, so a raise added after it never runs |
ran it |
| Candidate file loading | malformed candidate files ([], an empty map, {sp: null}, a wrong type, an unknown act, truncated YAML) raise ValueError naming --candidates and the problem |
ran it, edge input (structured) |
| Name matching | names are matched as text: scipy with a combining mark and ß give no choice, plain scipy gives one |
ran it, edge input (text) |
| Paths | a Windows-style pip.exe path gives the install choice, and a directory given as --candidates raises ValueError naming the flag |
ran it, edge input (paths) |
| Repeated names | a name given twice gives one choice, and an empty candidate map gives no event | ran it, edge input (collections) |
| Cli file errors | repo run with a missing --candidates file exits 1 and names the flag and the path |
ran it, edge input (cli) |
| Missing exit code | a None rc gives no install choice, and a None time passes through the timeline |
ran it, edge input (none) |
| Non-finite times | -inf and nan times pass through choice_timeline unchanged |
ran it, edge input (numbers) |
| Copilot shell rc | a Copilot shell result with only success gets rc None, and failed_commands does not count it, while an rc of 1 is still counted |
ran it |
| Shell structure | the probes stop at an executed exit or exec, skip false && ..., read <<- terminators and skip $((...)), as the description says |
ran it |
| text = _without_heredocs(script) | ||
| flat = not _GROUPED.search(text) # a group or a block may hold an exit that does not run: then nothing ends the scan | ||
| for part, after, in_subshell, _outer in split_script(text): | ||
| conditional = before in ("||", "&&") # the part runs only if the one before it did, or did not, succeed |
There was a problem hiding this comment.
🔴 Must fix: false || exit 1; nproc gives a cpu probe, though the exit always runs and nproc never does.
Input: false || exit 1; nproc. On 2661870 it gives a cpu probe for nproc. Expected: no probe, since false fails, exit 1 runs and the shell stops before nproc. true && exit 0; nproc gives the same probe. conditional is set for every part after || or &&, so the exit does not end the scan even when the status before it is known.
Fix, run on 2661870: the failing case passes; tests and lint are no worse.
| conditional = before in ("||", "&&") # the part runs only if the one before it did, or did not, succeed | |
| conditional = before in ("||", "&&") and status is None # the part runs only if the one before it did, or did not, succeed |
| _SHELL_C = re.compile(r"-[a-z]*c[a-z]*") # ``-c``, ``-lc``: the next word is a script | ||
| _LEX_SKIP = frozenset({"(", ")", "{", "}"}) # grouping a PowerShell or shell expression puts around a command | ||
| _SEPARATORS = frozenset({"|", "||", "&&", ";", "&"}) # ``|`` splits stages; the rest end a command | ||
| _GROUPED = re.compile(r"[{}()]|\b(?:if|then|else|elif|fi|do|done|while|until|case|esac)\b") # a group or a block |
There was a problem hiding this comment.
🟡 Nit: exit 0; echo $(date); nproc gives a cpu probe: _GROUPED reads any ( as a group and turns the exit cutoff off.
Input: exit 0; echo $(date); nproc. On 2661870 it gives a cpu probe. Expected: none, since exit 0 ends the shell first. _GROUPED matches any parenthesis, so a command substitution switches the cutoff off the way a stray brace does, and the description's known misses name braces, do and done only.
Suggestion: Let $( ) parse like the other substitutions instead of matching ( in _GROUPED, or name it with the braces among the known misses.
| ("nproc", {"denied": True}), # a denied command ran nothing | ||
| ("true || echo skipped || nproc", {}), | ||
| ("command -v nproc", {}), # ``command -v`` only looks the program up | ||
| ("sudo --help nproc", {}), # ``--help`` prints text and runs nothing |
There was a problem hiding this comment.
🟡 Nit: Removing -V from _INFO_OPTIONS leaves every probe test green, and sudo -V nproc then gives a cpu probe.
Input: sudo -V nproc. On 2661870 it gives no probe, since sudo -V prints the version and runs nothing. With -V taken out of _INFO_OPTIONS (dokimi/dokimi/repos/probes.py:53) it gives a cpu probe, and the suite still passes. Expected: the suite fails, as sudo --help nproc already does for --help.
Fix, run on 2661870: the tests now catch the broken variant; tests and lint are no worse.
| ("sudo --help nproc", {}), # ``--help`` prints text and runs nothing | |
| ("sudo --help nproc", {}), # ``--help`` prints text and runs nothing | |
| ("sudo -V nproc", {}), # ``-V`` prints the version and runs nothing |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 2661870. The events go only to events.jsonl, and the counters do not read them. But the import and use choices count an Edit's unchanged lines. Also, this branch's probes.py has drifted from #782's, so the two PRs need one plan before either merges. The details are in the line comments.
Agreed with @exai-bot: uv add --no-sync; an exit inside a pipeline; two heredocs on one line; a heredoc on bash -c; here-strings (more cases in its thread); a leading redirection; the manifest's probes missing from the execution id.
Agreed with @vchamarthi: false || exit 1; nproc; _GROUPED reading any (; the missing sudo -V test.
What I ran, at 2661870:
make ci-local --offline: 21 passed, 1 failed, 20 skipped; 21,929 tests passed. The failure isschema_compat:run_context.v1.jsonreachedmainwith #785 after this branch's base, so it does not come from this PR and goes away on a rebase.- A repro through
convert,with_probes,with_choicesandfirst_usefulfor each line comment, and the same probe inputs through #782'sprobes.py. - 60,000 random shell strings through
with_probesandwith_choices: no exception.
Checked, no issues
- Resume identity: the
--candidatesand--probesdigests go into the execution id, so a resume with other files is refused. The golden id moved only because the fixture binary changed. - Exit-code rule for choices: a denied or failed call, or one with no result, gives no choice. Only
&&chains count, as the contract says. - Schema and CLI:
repo_manifest.v1.jsononly widens (probes). The CLI snapshot and the flag allowlist are updated, and the new files have the SPDX header.
| def _added_text(args: Any) -> str: | ||
| """The text an edit tool adds: the new side of each replacement, or the whole file of a write (choices.py).""" | ||
| if not isinstance(args, dict): | ||
| return "" | ||
| if isinstance(args.get("edits"), list): # MultiEdit | ||
| return "\n".join(str(e.get("new_string", "")) for e in args["edits"] if isinstance(e, dict)) | ||
| return str(args.get("new_string") or args.get("content") or args.get("new_source") or "") |
There was a problem hiding this comment.
Blocking: An Edit's unchanged lines count as added text, so they give false import and use choices.
Claude's Edit repeats the lines around a change in new_string. With sklearnex: {import: [sklearnex], use: [patch_sklearn]}, this Edit gives an import and a use choice for sklearnex, though the agent added only numpy:
old_string: "from sklearnex import patch_sklearn\npatch_sklearn()\n"
new_string: the same two lines + "import numpy\n"
An Edit that deletes a line gives choices for the lines kept around it too. The contract says "never what the edit replaced"; test_only_the_added_side_of_an_edit_counts passes only because its two sides share no line.
Suggestion: count only the lines of new_string that old_string does not have (per edit of a MultiEdit, e.g. with a Counter of the old lines), and add a test where both sides share a line.
| return found | ||
|
|
||
|
|
||
| def command_stages(part: str) -> tuple[list[list[str]], list[str]]: |
There was a problem hiding this comment.
This branch's probes.py is no longer #782's: the two PRs now carry two different implementations.
The description says this diff holds #782's commit d3360cd until #782 merges, but both branches have rewritten probes.py since (about 490 changed lines between them). On 33 inputs from the review threads they disagree on 15: #782 handles if true; then nproc; fi, cat <<<EOF and >/tmp/o nproc; this branch handles false && nproc, exit 0; nproc and env FOO=1 nproc. choices.py imports command_stages, command_words and split_redirects, which #782 does not have.
Suggestion: keep one copy: fold #782 into this PR and close it, or take the probes out of here and rebase onto #782 once it merges (AGENTS.md: PRs are not stacked).
| def command_words(words: list[str]) -> Iterator[list[str]]: | ||
| """The words of the command a simple command runs, from its program on, then those of the command each wrapper | ||
| (``time``, ``sudo -n``, ``nice -n 5``...) runs: its options are not the program.""" | ||
| i = 0 | ||
| while i < len(words): | ||
| while i < len(words) and _ASSIGN.match(words[i]): # NAME=value words set variables for the command after them | ||
| i += 1 | ||
| if i >= len(words): | ||
| return | ||
| yield words[i:] | ||
| wrapper = _program(words[i]) | ||
| if wrapper not in _WRAPPERS: | ||
| return | ||
| i += 1 | ||
| while i < len(words) and words[i].startswith("-") and len(words[i]) > 1: | ||
| if words[i] in _INFO_OPTIONS or (wrapper == "command" and words[i] == "-v"): # looks up or prints: runs nothing | ||
| return | ||
| i += 2 if words[i] in _WRAPPER_VALUES.get(wrapper, ()) else 1 | ||
|
|
||
|
|
There was a problem hiding this comment.
A probe after a shell keyword is missed: if true; then nproc; fi, for x in 1; do nproc; done and ! nproc give no probe.
command_words takes then, do or ! for the program. So if command -v nvidia-smi; then nvidia-smi; fi, a common way to check for a GPU, gives no gpu probe. #782 fixed this in 57508d3 (a keyword set and a test); the fix is not on this branch.
Suggestion: skip leading if, then, else, elif, do, while, until and ! in command_words, and take #782's test.
| event["rc"] = code if isinstance(code, int) else (None if data.get("success") is None | ||
| else 0 if data.get("success") else 1) | ||
| failed = _failed_read(str(event["text"]), event["rc"]) | ||
| event["rc"] = code if isinstance(code, int) else None # the code the command returned, or none: no success fallback |
There was a problem hiding this comment.
A Copilot command with no exit code now gives first_useful the verdict never, so "not measured" counts as a miss.
With rc null for a result that has only success, useful.hits never matches it and first_useful returns never, which first_useful_rate counts. A copilot-json make test with result {"success": true} was a hit before this PR and is never now; a {"success": false} one no longer counts in failed_commands, which reads 0.
Suggestion: give first_useful no verdict (like unobserved) when a matching command has rc null, and say in the fragment that older Copilot runs are not comparable on these two.
|
|
||
| _READERS = frozenset({"cat", "head", "tail", "less", "more", "nl", "sed", "grep", "egrep", "fgrep", "awk", "tac", | ||
| "strings", "wc", "rg"}) | ||
| _WRAPPERS = frozenset({"time", "nohup", "sudo", "command", "exec", "builtin", "nice", "env"}) # each runs the command after it |
There was a problem hiding this comment.
timeout, uv run, stdbuf and doas are not wrappers here, so the command after them gives no probe or choice.
timeout 10 nvidia-smi, uv run python -c 'import os; os.cpu_count()' and bash -o pipefail -c nproc give no probe; timeout 600 pip install scipy gives no install choice. Agents wrap long commands in timeout often. dokimi/context/shell.py::strip_wrappers already skips timeout 300 and uv run, and repos/environment/risks.py has timeout in its _WRAPPERS.
Suggestion: add timeout (with its duration), stdbuf, doas and uv run to the wrappers, or reuse strip_wrappers; find -c after other options (bash -o X -c, python -W X -c).
|
|
||
| def _package_call(words: list[str]) -> tuple[str, list[str]] | None: | ||
| """The act and the package names of a package manager's install, add, uninstall or remove call.""" | ||
| if {"--dry-run", "--download-only"} & set(words) or _INFO_FLAGS & set(words): # no install: a dry run, a download, a help request |
There was a problem hiding this comment.
Some short spellings give a wrong install choice, or none.
conda install -d scipyandmamba install -d scipy(-dis--dry-run) give an install choice.uv pip install -n scipyandpoetry add -n scipygive none:-n(--no-cache,--no-interactionthere) is in_VALUE_OPTIONS, soscipyis read as its value.py -m pip install scipygives none: this_PYTHONhas nopy, whileprobes._PYTHONhas it.
Suggestion: read -d as a dry run for conda and mamba, take -n as a value option for conda only, and reuse probes._PYTHON.
| # Probe events go to events.jsonl only: the counters, first_useful and skill usage score the trace's events | ||
| probed = with_probes(transcript.events, probes_for(ctx.manifest.probes, opts.probes)) | ||
| shown = with_choices(probed, opts.candidates or {}, transcript.added_text) # choice events: choices.py |
There was a problem hiding this comment.
An exception in the probe or choice matching fails the cell's grade.
with_probes and with_choices run inside the grading try of run_cell, so an error there sets grade to failed_outcome("host error after the agent ran ...") and skips events.jsonl and answer.txt. For example, python -c '1+1+...' with 20,000 terms makes ast.parse in probes._calls raise RecursionError, which it does not catch. No score reads these events, so they should never cost a cell its grade.
Suggestion: catch RecursionError and MemoryError in _calls, and write the trace's own events when the matching raises, as _process_or_null does for the metrics.
| if part.startswith("("): # a subshell: its exit code is not the command's own | ||
| continue | ||
| stages, _inner = command_stages(part) | ||
| if len(stages) > 1: # a pipeline reports its last stage's exit code |
There was a problem hiding this comment.
Question: should pip install scipy 2>&1 | tail -5 give a choice marked "exit code unknown", instead of no event?
This is a common install form in Claude Code traces, and today it leaves nothing in events.jsonl, so a reader cannot tell "not measured" from "not installed". The command's output usually holds pip's Successfully installed scipy-... line, which could confirm it. If you keep it as is, please say in the contract that no install choice does not mean the agent did not install it, as it does for Copilot's edits.
| def choice_timeline(events: Sequence[dict[str, Any]]) -> list[tuple[float | None, str, str]]: | ||
| """The ``(t, candidate, act)`` of each choice event, in the order the trace gives them.""" | ||
| return [(e.get("t"), str(e["candidate"]), str(e["act"])) for e in events if e.get("kind") == "choice"] |
There was a problem hiding this comment.
Nit: choice_timeline is used only by a test. Nothing in the package calls it. Suggestion: drop it, or use it where a report reads the choices.
Summary
choiceevent: an agent's install, import, use or removal of a candidate the run names (repo run --candidates FILE), written beside itscommandoreditevent.&&chain with no pipe, no--dry-run, no--download-only, and no part after anexitor a replacingexec. Contract §4.1 lists what is missed.949755343): a conda channel prefix (conda-forge::scipy) is no part of the name; anexecwith options;--reportand other pip value options; probes stop at an executedexitorexec, skipfalse && …and||short circuits, read heredoc terminators by shell rules, ignore$((…))and redirection targets, and no longer treatcommand -v, a wrapper's--help, orenvas the program.rcis itsshellExecution.exitCodeonly. A result that carries onlysuccessnow getsrcnullwhere it got0or1, sofailed_commandsandfirst_usefulcount such a command as neither failed nor useful. Results that carryshellExecution.exitCodeare unchanged. Contract §4.1 and a changelog fragment say so.first_usefuldo not read choices.d3360cd2ef), which is still open. This PR's base ismain, so its diff also shows that commit until feat(repo): probe events for commands that ask the host for its facts #782 merges.Closes #807
Evidence
make ci-localon052e374999(same tree as2661870ca5): 21979 passed, 19 skipped, 3 failed. The 3 are timeout tests that load slows (test_process.py::test_sighup_kills_the_live_groups_too,test_repo_reach.pyorphan kill,test_repo_reach_edges.py); all 191 pass when those files run alone.949755343and on this head.949755343..56791a5b1a): no blocking finding. The last two commits add a test and reword the contract from its notes. Declined notes are under Merge Danger.tests/test_batch_ledger_golden.py:ac034e664259→d4546308cce5. No golden fixture bytes changed in this round.repo runneeds a container runtime, which the review box lacks. Covered by unit tests and the CLI snapshot.Merge Danger
d3360cd2ef.rc(see Summary):failed_commandsandfirst_usefulchange for a result that has onlysuccess.--candidates FILEflag onrepo run: CLI snapshot regenerated, and--candidateshas a row intests/data/cli_flags_allowlist.txt.events.jsonlgainschoicerows when candidates are given. Readers must accept the new kind.&&chain whose later part fails gives no install choice (the exit code cannot say which part failed); installs insidebash -cor(pip …)give none;make 2>&1 | exit; nprocandexec sleep 5 & nproclose the later probe (an undercount); a heredoc terminator with a CRLF ending stays open, as bash does; an apostrophe inside double quotes before<<on the same line can hide the heredoc, so its body is scanned; a stray brace,doordoneturns off the exit cutoff for the script (a probe may then repeat); combined short flags (command -pv) are not parsed.apply_patchand Codexfile_changecarry no added text, so they give install and removal choices only.🤖 Generated with Claude Code