Repository navigation
Conversation
A command that runs nproc, lscpu, nvidia-smi, pip list, printenv, a python -c one-liner calling os.cpu_count(), or a PowerShell Get-CimInstance Win32_Processor gets a probe event after its command event, one per pattern it matches. Patterns match the program the shell runs, never argument text, so grep nproc README is no probe. The defaults ship in repos/probes.py. A manifest's probes key and repo run --probes FILE replace a target's list; [] turns it off. Existing event kinds are unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi repo run` parameters changed:
+ added option `--probes`Updated on every push; the |
|
OpenCodeReview: Commit af6d2a8 was not reviewed. This PR has reached its review limit (6 reviews). CI at `af6d2a8`: 13 still running, 4 passing, 1 skipped
3 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (9f90047): 4 medium</summary> Open threads from the bot: 0 </details> |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #782 +/- ##
==========================================
- Coverage 95.79% 95.74% -0.05%
==========================================
Files 432 433 +1
Lines 57204 57600 +396
Branches 9858 9951 +93
==========================================
+ Hits 54796 55147 +351
- Misses 1497 1524 +27
- Partials 911 929 +18
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at d3360cd. The additive design is sound: probe events are emitted after the command event, and process counters, first_useful and skill usage score the trace without them. Two risks and three nits are about the matcher, which disagrees with the contract text added in this PR.
What I ran, on a Linux review host at d3360cd:
make ci-local: 31 passed, 2 failed (gitleaks download returned HTTP 504 so Scan commits did not run), 14 skipped; test and changed-line coverage steps pass- reproductions for the matcher findings by calling
probes._matching(command, probes_for())with the default patterns, no model or network involved.
Checked, no issues
- Sign-off: the single commit d3360cd has a
Signed-off-by. - Schema:
probesis added torepo_manifest.v1.jsonas an optional key, so existing manifests still validate.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 2fe9ab0, which fix the three nits from my review:
d7ee57a:dokimi/tests/test_repo_probes.py:131: added strict xfail tests forecho '$(nproc)',sysctl -a '|' grep machdep.cpuand the quotedos.cpu_count()one-liner.4002160:dokimi/dokimi/repos/probes.py:212: options after a wrapper are skipped, with the value ofnice -nandsudo -u, sosudo -E lscpuandnice -n 5 nprocmatch; tests added.8a0885f:dokimi/dokimi/repos/probes.py:234: forgrep,egrep,fgrep,rg,sedandawkthe search expression is no longer a file operand; negative tests forgrep /proc/cpuinfo READMEandgrep -e /proc/meminfo READMEadded.
The review bot then flagged lines of those commits; fixed as well:
57508d3:dokimi/dokimi/repos/probes.py:213: a command afterif,then,else,elif,do,while,untilor!is now read as the program, with positive tests for anifbody and aforbody.2fe9ab0:dokimi/dokimi/repos/probes.py:246: the value of-m,-A,-Band-Cis no longer taken as the search expression, and the target of an output redirection is dropped from a command's words, with negative tests forgrep -m 1,cat >andcat 2>>.
make ci-local at 2fe9ab0: 31 passed, 2 failed (host: gitleaks download), 14 skipped.
Not approving yet. Still open: the two risks from my review.
vchamarthi
left a comment
There was a problem hiding this comment.
Reviewed 2fe9ab0: Changes needed: 5 must-fixes, 1 nit.
Blocking
- Open, not looked at in this review: thread by @antonwolfy. For @napetrov.
- Open, not looked at in this review: thread by @antonwolfy. For @napetrov.
- Not verified:
repo runwrites theproberows and passes--probesand the manifestprobeskey intoAgentOptions(dokimi/dokimi/repos/runner.py:266,dokimi/dokimi/commands/repo_run.py); only the unit tests reach them. Runrepo runwith a manifestprobeskey and with--probes FILEon a task whose agent runsnproc, then check thatevents.jsonlhas theproberow right after itscommandrow. For @napetrov. - 1 defect the reviewer is sure of but could not show by a run.
Verified on 2fe9ab0: 15 checks pass
| Area | What holds | How |
|---|---|---|
| Changed tests on head | tests/test_repo_probes.py passes on the head: 58 pass and 3 strict xfail cases, which are the quoted-text inputs |
tests on the head |
| Changed tests on main | On main the same test file fails to import dokimi.repos.probes, so it covers only the new module |
same tests on main |
| Full run with main | With main merged in, no test fails that main lacks: the three failing tests also fail on main, and the CLI snapshot and schema-check steps pass | full suite, main merged in |
| Merge | The PR merges onto main with no conflict | merge onto main |
| Grep word | grep nproc README gives no probe event (test_a_grep_of_the_word_is_no_probe passes) |
tests on the head |
| Wrapper options | The tests pin the skip of the value words of nice -n and sudo -u: removing the skip fails 2 tests |
broke the code, tests caught it |
| Layers | A run layer replaces the manifest layer, and an empty list turns a target off (test_a_run_layer_replaces_the_manifest_layer and test_a_manifest_replaces_a_target_and_an_empty_list_turns_it_off pass) |
tests on the head |
| Manifest key | A manifest probes key is read, and a bad one is a manifest error (test_a_manifest_probes_key_is_read_and_a_bad_one_is_a_manifest_error passes) |
tests on the head |
| Run file errors | A bad --probes file names its path and the cause: a truncated file, an unknown target gpus and cpu: with no list each raise ValueError with that message, and an empty mapping gives {} |
ran it, edge input (structured) |
| Pattern text | A pattern with a trailing carriage return is stripped to nproc, and a command with ß still gives the nproc probe event |
ran it, edge input (text) |
| Missing file | A missing --probes file and a directory given as one both raise ValueError that names the path |
ran it, edge input (paths) |
| Repeated pattern | A pattern listed twice gives one probe event |
ran it, edge input (collections) |
| Usage error | repo run --probes with no value exits 2 and names --probes on stderr |
ran it, edge input (cli) |
| None inputs | A None layer gives {}, no layer gives the defaults, and a command with text of None gives no probe event and is passed through unchanged |
ran it, edge input (none) |
| Quoting | Double quotes do not stop a substitution: echo "$(nproc)" gives the nproc probe event |
ran it, edge input (quoting) |
A backslash before a space outside quotes is an escape again, so a WSL path `/mnt/c/Program\ Files/...` keeps one word (self-review of this round). Substitutions are found by balanced parentheses; quoted text is no command, so '$(nproc)' and '|' inside quotes are words; a redirection's target is written, an input's is read; a python one-liner's options end at its script and -cCODE is a code; a shell launcher's -c script is read after time, env assignments and options; a Windows path keeps its backslashes; a call inside a string literal is no call; a probe file nested too deep is a ValueError. The rg search pin fails when rg leaves the searchers. Already fixed earlier in this PR: sudo -E lscpu (4002160) and a grep search expression that looks like a path (8a0885f). Thread: PRRT_kwDOVBg7g863m7XT Thread: PRRT_kwDOVBg7g863o9_8 Thread: PRRT_kwDOVBg7g863m7XW Thread: PRRT_kwDOVBg7g863m7XX Thread: PRRT_kwDOVBg7g863m7XY Thread: PRRT_kwDOVBg7g863m7XZ Thread: PRRT_kwDOVBg7g863o9_- Thread: PRRT_kwDOVBg7g863pFRV Thread: PRRT_kwDOVBg7g863pb2R Thread: PRRT_kwDOVBg7g863pb2V Thread: PRRT_kwDOVBg7g863qoo9 Thread: PRRT_kwDOVBg7g863qoo_ Thread: PRRT_kwDOVBg7g863qopB Thread: PRRT_kwDOVBg7g863qopD Thread: PRRT_kwDOVBg7g863qopE Thread: PRRT_kwDOVBg7g863qopJ Review: PRR_kwDOVBg7g88AAAABYEoIyQ Review: PRR_kwDOVBg7g88AAAABYFAfqQ Review: PRR_kwDOVBg7g88AAAABYFeYaQ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
Changes needed → addressed at 13e3d9c (the probe fixes are 658e2ac; the merge brings intel/main in).
|
Thread: PRRT_kwDOVBg7g863zrh3 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zriH Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zriW Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 3 must-fixes · re-reviewed 13e3d9c
Next: @napetrov: fix the 3 must-fix findings; then push for a re-review.
Verified on 13e3d9c: ✅ changed tests: 77 passed, fail on main · ✅ full suite with main merged in: 21843 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried
3 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Probe tests | tests/test_repo_probes.py passes on the head: 77 passed |
tests on the head |
| Merge | Merges onto main with no conflict | merge onto main |
| Option mutation | Replacing the _SEARCHERS guard in _operands makes the suite fail |
broke the code, tests caught it |
|
Round 4 is addressed at a445b7f, 815dbc9 and 6c007cf: the three new must-fix threads on Mixed-probe batches: the decision is to refuse them, through the probe-file digest that #787 puts in the execution id (its |
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/repos/runner.py
A PowerShell $x = assignment is skipped; python -Ic code and -Im module read the code or module after the flags; <<< is a string, so the word after it is no file the command reads. Thread: PRRT_kwDOVBg7g8630gp- Thread: PRRT_kwDOVBg7g8630gqG Thread: PRRT_kwDOVBg7g8630gqL Review: PRR_kwDOVBg7g88AAAABYKTqPw Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
Thread replies above: the three must-fixes of the 13e3d9c review are fixed on this branch, one commit each with a red test first. |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 9f90047. The design holds: probe events only go into events.jsonl, and every reader of it (process counters, first_useful, skill usage, the security scan, compare, context/consistency) skips a kind it does not know. The main gap is that a matcher that raises fails the whole cell; the others are about what the default patterns match. Details are in the line comments.
What I ran, at 9f90047:
make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only steps); the test suite: 22,092 passed, 72 skipped.- The trigger of every resolved thread through
probes._matchingwith the defaults: all 27 cases give what the thread asked for. run_cellthroughrepo runwith a fake harness, once as is and once withwith_probespatched to raiseRecursionError: the passing cell becomesgrader_failure, with noevents.jsonland noanswer.txt.- About 40 more commands against the defaults (
env, wrappers,pip3,sysctl,from os import cpu_count), with the results in the line comments.
Checked, no issues
- Schema and manifest:
probesis an optional key ofrepo_manifest.v1.json(the schema is only widened), a bad value is aManifestError, andto_dictleaves it out when it is empty. - Scores: the counters,
first_usefuland skill usage still scoretranscript.events, so no published number changes. - Docs and changelog: contract §2 and §4.1 list the defaults as
DEFAULT_PROBEShas them; the CLI snapshot has--probes; the fragment is under Added and says how to replace a target or turn it off.
A matcher that raises (a deep chain hits RecursionError in ast.unparse) now costs the cell its probe events only: the grade, answer.txt and events.jsonl are written as before, and trace_errors notes the failure. Each cell's meta.json records the effective probe patterns per target. Thread: PRRT_kwDOVBg7g864ZUdO Thread: PRRT_kwDOVBg7g864ZUdg Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… run env runs the program after its options and assignments, and is a probe only when no program follows. timeout, xargs, conda run, uv run, bash -o and pwsh -Command are read through, and a pattern spelled pip3 or python -m pip is matched as pip. The contract lists what is still not matched. Thread: PRRT_kwDOVBg7g864ZUdS Thread: PRRT_kwDOVBg7g864ZUdY Thread: PRRT_kwDOVBg7g864ZUdo Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e-probe-events Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/repos/transcript.py
A pattern that starts with env, conda run or uv run is read through the same launcher rule as a command, so "uv run pip list" matches the command "uv run pip list". PowerShell options with a value (-ExecutionPolicy) are skipped, and -com is -Command. The two run tests move to tests/test_repo_run_probes.py to keep test_repo_run.py under the file cap; _views takes its wrapper arguments from a helper (complexity cap). The golden meta keys gain "probes". Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
The default `sysctl -a | grep machdep.cpu` matched only that grep word, so `sysctl -n machdep.cpu.brand_string`, `sysctl -n hw.ncpu` and `sysctl hw.memsize` gave no probe. The default is now `sysctl`, which reads kernel facts in almost every use. Thread: PRRT_kwDOVBg7g864ZUdu Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
@vlad-perevezentsev @antonwolfy all threads are fixed and resolved (sysctl default is now bare |
Summary
probeevent: a command that asks the host for its facts (nproc,lscpu,nvidia-smi,pip list,Get-CimInstance Win32_Processor, apython -ccall ofos.cpu_count()) gets one beside itscommandevent.probes:key andrepo run --probes FILEreplace a target's patterns; an empty list turns it off.first_usefuland skill usage score the trace's events without probes.Evidence
make ci-localond3360cd2ef: rc=0, 20537 passed, 19 skipped, no FAILED.;&<>or a spaced path validated but never matched.envused as a launcher, heredoc bodies,grep --include GLOB(a separate-word glob), combined short flags such asgrep -rf FILE(pre-existing on base in each case). Fixed in the review rounds: wrapper options, compound-keyword bodies,$( )with parentheses and with quoted parentheses, quoted text, python options and-c, shell launchers, redirections, Windows paths, backslash-newline continuations, search option values (rg -g,grep -f).Merge Danger
events.jsonlgainsproberows for runs whose commands match a pattern. Existing kinds are unchanged; any reader of that file must accept the new kind.--probes FILEflag onrepo run(CLI snapshot regenerated) and a manifestprobeskey (repo_manifest.v1,additionalProperties: false).repo runneeds a container runtime, which the review box lacks. Covered by unit tests and--helponly.intel/mainat47f89eda1(no conflicts).--probesfile is another execution); it lands with feat(repo): choice events for a candidate's install, import, use or removal #787.🤖 Generated with Claude Code