Skip to content

feat(repo): probe events for commands that ask the host for its facts - #782

Open
napetrov wants to merge 19 commits into
intel:mainfrom
napetrov:feat/trace-probe-events
Open

napetrov wants to merge 19 commits into
intel:mainfrom
napetrov:feat/trace-probe-events

Conversation

@napetrov

@napetrov napetrov commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds a probe event: a command that asks the host for its facts (nproc, lscpu, nvidia-smi, pip list, Get-CimInstance Win32_Processor, a python -c call of os.cpu_count()) gets one beside its command event.
  • Targets: cpu, gpu, os, memory, packages, env, network. Each has defaults. A manifest probes: key and repo run --probes FILE replace a target's patterns; an empty list turns it off.
  • Additive only: one new event kind. Existing events are unchanged, and the process counters, first_useful and skill usage score the trace's events without probes.

Evidence

  • make ci-local on d3360cd2ef: rc=0, 20537 passed, 19 skipped, no FAILED.
  • Targeted run (probe, event, metric, manifest and runner tests): 746 passed.
  • Local self-review (pr-review, local mode, lane L2): two scan passes on the final head, no confirmed blocking. Fixed in review: process counters were reading probe events; patterns with ;&<> or a spaced path validated but never matched.
  • Declined as follow-ups, with probe output: env used as a launcher, heredoc bodies, grep --include GLOB (a separate-word glob), combined short flags such as grep -rf FILE (pre-existing on base in each case). Fixed in the review rounds: wrapper options, compound-keyword bodies, $( ) with parentheses and with quoted parentheses, quoted text, python options and -c, shell launchers, redirections, Windows paths, backslash-newline continuations, search option values (rg -g, grep -f).

Merge Danger

  • events.jsonl gains probe rows for runs whose commands match a pattern. Existing kinds are unchanged; any reader of that file must accept the new kind.
  • New --probes FILE flag on repo run (CLI snapshot regenerated) and a manifest probes key (repo_manifest.v1, additionalProperties: false).
  • Not exercised end to end: repo run needs a container runtime, which the review box lacks. Covered by unit tests and --help only.
  • The branch merges intel/main at 47f89eda1 (no conflicts).
  • Mixed-probe batches are not refused by this PR. The refusal is the probe-file digest in the execution id that feat(repo): choice events for a candidate's install, import, use or removal #787 adds (a resume with another --probes file is another execution); it lands with feat(repo): choice events for a candidate's install, import, use or removal #787.

🤖 Generated with Claude Code

A command that runs nproc, lscpu, nvidia-smi, pip list, printenv, a
python -c one-liner calling os.cpu_count(), or a PowerShell
Get-CimInstance Win32_Processor gets a probe event after its command
event, one per pattern it matches. Patterns match the program the shell
runs, never argument text, so grep nproc README is no probe.

The defaults ship in repos/probes.py. A manifest's probes key and
repo run --probes FILE replace a target's list; [] turns it off.
Existing event kinds are unchanged.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 15, 2026 06:24
Copilot AI balanced review requested due to automatic review settings November 15, 2026 06:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 15, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi repo run` parameters changed:
    + added option `--probes`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@exai-bot

exai-bot commented Nov 15, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit af6d2a8 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `af6d2a8`: 13 still running, 4 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Linux Python 3.11 still running
Linux Python 3.14 still running
Linux Wheels Python 3.11 still running
Linux Wheels Python 3.14 still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running

3 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (9f90047): 4 medium</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
@codecov-commenter

codecov-commenter commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.22055% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.74%. Comparing base (b62613a) to head (af6d2a8).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/repos/probes.py 88.26% 25 Missing and 19 partials ⚠️
dokimi/dokimi/repos/manifest.py 66.66% 3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #782      +/-   ##
==========================================
- Coverage   95.79%   95.74%   -0.05%     
==========================================
  Files         432      433       +1     
  Lines       57204    57600     +396     
  Branches     9858     9951      +93     
==========================================
+ Hits        54796    55147     +351     
- Misses       1497     1524      +27     
- Partials      911      929      +18     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/repo_run.py 95.83% <100.00%> (+0.03%) ⬆️
dokimi/dokimi/repos/events.py 97.53% <ø> (ø)
dokimi/dokimi/repos/runner.py 97.20% <100.00%> (+0.06%) ⬆️
dokimi/dokimi/repos/transcript.py 97.36% <100.00%> (ø)
dokimi/dokimi/repos/manifest.py 94.19% <66.66%> (-0.59%) ⬇️
dokimi/dokimi/repos/probes.py 88.26% <88.26%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at d3360cd. The additive design is sound: probe events are emitted after the command event, and process counters, first_useful and skill usage score the trace without them. Two risks and three nits are about the matcher, which disagrees with the contract text added in this PR.

What I ran, on a Linux review host at d3360cd:

  • make ci-local: 31 passed, 2 failed (gitleaks download returned HTTP 504 so Scan commits did not run), 14 skipped; test and changed-line coverage steps pass
  • reproductions for the matcher findings by calling probes._matching(command, probes_for()) with the default patterns, no model or network involved.

Checked, no issues

  • Sign-off: the single commit d3360cd has a Signed-off-by.
  • Schema: probes is added to repo_manifest.v1.json as an optional key, so existing manifests still validate.

Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a commit per nit, up to 2fe9ab0, which fix the three nits from my review:

  • d7ee57a: dokimi/tests/test_repo_probes.py:131: added strict xfail tests for echo '$(nproc)', sysctl -a '|' grep machdep.cpu and the quoted os.cpu_count() one-liner.
  • 4002160: dokimi/dokimi/repos/probes.py:212: options after a wrapper are skipped, with the value of nice -n and sudo -u, so sudo -E lscpu and nice -n 5 nproc match; tests added.
  • 8a0885f: dokimi/dokimi/repos/probes.py:234: for grep, egrep, fgrep, rg, sed and awk the search expression is no longer a file operand; negative tests for grep /proc/cpuinfo README and grep -e /proc/meminfo README added.

The review bot then flagged lines of those commits; fixed as well:

  • 57508d3: dokimi/dokimi/repos/probes.py:213: a command after if, then, else, elif, do, while, until or ! is now read as the program, with positive tests for an if body and a for body.
  • 2fe9ab0: dokimi/dokimi/repos/probes.py:246: the value of -m, -A, -B and -C is no longer taken as the search expression, and the target of an output redirection is dropped from a command's words, with negative tests for grep -m 1, cat > and cat 2>>.

make ci-local at 2fe9ab0: 31 passed, 2 failed (host: gitleaks download), 14 skipped.

Not approving yet. Still open: the two risks from my review.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2fe9ab0: Changes needed: 5 must-fixes, 1 nit.

Blocking

  • Open, not looked at in this review: thread by @antonwolfy. For @napetrov.
  • Open, not looked at in this review: thread by @antonwolfy. For @napetrov.
  • Not verified: repo run writes the probe rows and passes --probes and the manifest probes key into AgentOptions (dokimi/dokimi/repos/runner.py:266, dokimi/dokimi/commands/repo_run.py); only the unit tests reach them. Run repo run with a manifest probes key and with --probes FILE on a task whose agent runs nproc, then check that events.jsonl has the probe row right after its command row. For @napetrov.
  • 1 defect the reviewer is sure of but could not show by a run.
Verified on 2fe9ab0: 15 checks pass
Area What holds How
Changed tests on head tests/test_repo_probes.py passes on the head: 58 pass and 3 strict xfail cases, which are the quoted-text inputs tests on the head
Changed tests on main On main the same test file fails to import dokimi.repos.probes, so it covers only the new module same tests on main
Full run with main With main merged in, no test fails that main lacks: the three failing tests also fail on main, and the CLI snapshot and schema-check steps pass full suite, main merged in
Merge The PR merges onto main with no conflict merge onto main
Grep word grep nproc README gives no probe event (test_a_grep_of_the_word_is_no_probe passes) tests on the head
Wrapper options The tests pin the skip of the value words of nice -n and sudo -u: removing the skip fails 2 tests broke the code, tests caught it
Layers A run layer replaces the manifest layer, and an empty list turns a target off (test_a_run_layer_replaces_the_manifest_layer and test_a_manifest_replaces_a_target_and_an_empty_list_turns_it_off pass) tests on the head
Manifest key A manifest probes key is read, and a bad one is a manifest error (test_a_manifest_probes_key_is_read_and_a_bad_one_is_a_manifest_error passes) tests on the head
Run file errors A bad --probes file names its path and the cause: a truncated file, an unknown target gpus and cpu: with no list each raise ValueError with that message, and an empty mapping gives {} ran it, edge input (structured)
Pattern text A pattern with a trailing carriage return is stripped to nproc, and a command with ß still gives the nproc probe event ran it, edge input (text)
Missing file A missing --probes file and a directory given as one both raise ValueError that names the path ran it, edge input (paths)
Repeated pattern A pattern listed twice gives one probe event ran it, edge input (collections)
Usage error repo run --probes with no value exits 2 and names --probes on stderr ran it, edge input (cli)
None inputs A None layer gives {}, no layer gives the defaults, and a command with text of None gives no probe event and is passed through unchanged ran it, edge input (none)
Quoting Double quotes do not stop a substitution: echo "$(nproc)" gives the nproc probe event ran it, edge input (quoting)

Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/tests/test_repo_probes.py Resolved
A backslash before a space outside quotes is an escape again, so a WSL path
`/mnt/c/Program\ Files/...` keeps one word (self-review of this round).

Substitutions are found by balanced parentheses; quoted text is no command,
so '$(nproc)' and '|' inside quotes are words; a redirection's target is
written, an input's is read; a python one-liner's options end at its script
and -cCODE is a code; a shell launcher's -c script is read after time, env
assignments and options; a Windows path keeps its backslashes; a call inside
a string literal is no call; a probe file nested too deep is a ValueError.
The rg search pin fails when rg leaves the searchers.

Already fixed earlier in this PR: sudo -E lscpu (4002160) and a grep
search expression that looks like a path (8a0885f).

Thread: PRRT_kwDOVBg7g863m7XT
Thread: PRRT_kwDOVBg7g863o9_8
Thread: PRRT_kwDOVBg7g863m7XW
Thread: PRRT_kwDOVBg7g863m7XX
Thread: PRRT_kwDOVBg7g863m7XY
Thread: PRRT_kwDOVBg7g863m7XZ
Thread: PRRT_kwDOVBg7g863o9_-
Thread: PRRT_kwDOVBg7g863pFRV
Thread: PRRT_kwDOVBg7g863pb2R
Thread: PRRT_kwDOVBg7g863pb2V
Thread: PRRT_kwDOVBg7g863qoo9
Thread: PRRT_kwDOVBg7g863qoo_
Thread: PRRT_kwDOVBg7g863qopB
Thread: PRRT_kwDOVBg7g863qopD
Thread: PRRT_kwDOVBg7g863qopE
Thread: PRRT_kwDOVBg7g863qopJ
Review: PRR_kwDOVBg7g88AAAABYEoIyQ
Review: PRR_kwDOVBg7g88AAAABYFAfqQ
Review: PRR_kwDOVBg7g88AAAABYFeYaQ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

Review at d3360cd: the additive design is sound.
Thanks. Its findings are the threads above, all answered in 658e2ac.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

I pushed a commit per nit, up to 2fe9ab0.
Thanks, the nits are in d7ee57a and up to 2fe9ab0.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 5 must-fixes, 1 nit.
All six are answered above, fixed in 658e2ac.

@napetrov

Copy link
Copy Markdown
Contributor Author

Changes needed → addressed at 13e3d9c (the probe fixes are 658e2ac; the merge brings intel/main in).

  • Must-fixes from the review: all six, each in its own reply above.
  • Left open, not blocking: probe list not in execution_record (design, needs a decision); timeout as a wrapper, |&/|( tokens, and deep $( ) nesting (follow-ups).

Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Thread: PRRT_kwDOVBg7g863zrh3
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zriH
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zriW
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 3 must-fixes · re-reviewed 13e3d9c

Next: @napetrov: fix the 3 must-fix findings; then push for a re-review.

Verified on 13e3d9c: ✅ changed tests: 77 passed, fail on main · ✅ full suite with main merged in: 21843 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried

3 checks behind it, each with its evidence
Area What holds How
Probe tests tests/test_repo_probes.py passes on the head: 77 passed tests on the head
Merge Merges onto main with no conflict merge onto main
Option mutation Replacing the _SEARCHERS guard in _operands makes the suite fail broke the code, tests caught it

Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
@napetrov

Copy link
Copy Markdown
Contributor Author

Round 4 is addressed at a445b7f, 815dbc9 and 6c007cf: the three new must-fix threads on probes.py, each with a red test first.

Mixed-probe batches: the decision is to refuse them, through the probe-file digest that #787 puts in the execution id (its --probes digest in execution_id, so a resume with another probe file is another execution and is refused). This PR adds no code for it, so it lands with #787; until then a mixed batch is not refused here.

…ress

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/repos/runner.py
A PowerShell $x = assignment is skipped; python -Ic code and -Im module read the code or module
after the flags; <<< is a string, so the word after it is no file the command reads.

Thread: PRRT_kwDOVBg7g8630gp-
Thread: PRRT_kwDOVBg7g8630gqG
Thread: PRRT_kwDOVBg7g8630gqL
Review: PRR_kwDOVBg7g88AAAABYKTqPw
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ress

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 3 must-fixes
All three are fixed, each with a red test first, at 7746c59.

@napetrov

Copy link
Copy Markdown
Contributor Author

Thread replies above: the three must-fixes of the 13e3d9c review are fixed on this branch, one commit each with a red test first.
My earlier top comment named the wrong commits for them (a445b7f, 815dbc9, 6c007cf fix other threads). Head: 9f90047. intel/main is merged in, so the PR is mergeable again.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 9f90047. The design holds: probe events only go into events.jsonl, and every reader of it (process counters, first_useful, skill usage, the security scan, compare, context/consistency) skips a kind it does not know. The main gap is that a matcher that raises fails the whole cell; the others are about what the default patterns match. Details are in the line comments.

What I ran, at 9f90047:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only steps); the test suite: 22,092 passed, 72 skipped.
  • The trigger of every resolved thread through probes._matching with the defaults: all 27 cases give what the thread asked for.
  • run_cell through repo run with a fake harness, once as is and once with with_probes patched to raise RecursionError: the passing cell becomes grader_failure, with no events.jsonl and no answer.txt.
  • About 40 more commands against the defaults (env, wrappers, pip3, sysctl, from os import cpu_count), with the results in the line comments.

Checked, no issues

  • Schema and manifest: probes is an optional key of repo_manifest.v1.json (the schema is only widened), a bad value is a ManifestError, and to_dict leaves it out when it is empty.
  • Scores: the counters, first_useful and skill usage still score transcript.events, so no published number changes.
  • Docs and changelog: contract §2 and §4.1 list the defaults as DEFAULT_PROBES has them; the CLI snapshot has --probes; the fragment is under Added and says how to replace a target or turn it off.

Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
Comment thread dokimi/dokimi/repos/probes.py Resolved
A matcher that raises (a deep chain hits RecursionError in ast.unparse)
now costs the cell its probe events only: the grade, answer.txt and
events.jsonl are written as before, and trace_errors notes the failure.
Each cell's meta.json records the effective probe patterns per target.

Thread: PRRT_kwDOVBg7g864ZUdO
Thread: PRRT_kwDOVBg7g864ZUdg
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… run

env runs the program after its options and assignments, and is a probe
only when no program follows. timeout, xargs, conda run, uv run, bash
-o and pwsh -Command are read through, and a pattern spelled pip3 or
python -m pip is matched as pip. The contract lists what is still not
matched.

Thread: PRRT_kwDOVBg7g864ZUdS
Thread: PRRT_kwDOVBg7g864ZUdY
Thread: PRRT_kwDOVBg7g864ZUdo
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e-probe-events

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/repos/transcript.py
A pattern that starts with env, conda run or uv run is read through the
same launcher rule as a command, so "uv run pip list" matches the
command "uv run pip list". PowerShell options with a value
(-ExecutionPolicy) are skipped, and -com is -Command.

The two run tests move to tests/test_repo_run_probes.py to keep
test_repo_run.py under the file cap; _views takes its wrapper
arguments from a helper (complexity cap). The golden meta keys gain
"probes".

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

The main gap is that a matcher that raises fails the whole cell.
Fixed in a3672c1. The other points are answered in their threads, so the summary needs no change of its own.

The default `sysctl -a | grep machdep.cpu` matched only that grep word, so
`sysctl -n machdep.cpu.brand_string`, `sysctl -n hw.ncpu` and `sysctl hw.memsize`
gave no probe. The default is now `sysctl`, which reads kernel facts in almost
every use.

Thread: PRRT_kwDOVBg7g864ZUdu
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

@vlad-perevezentsev @antonwolfy all threads are fixed and resolved (sysctl default is now bare sysctl), CI green on af6d2a8. Ready for approval — #787 is stacked on it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants