Repository navigation
Conversation
… missed host The host probe now records a fingerprint from a registry of collectors (os, cpu, memory, gpu, python, tools, network, virt) under `collectors`, beside the container's unchanged keys. A failing collector reports its error and the others still run. Nothing identifying (host or user name, address, env value, interface name, absolute path) is recorded. `repo run --label-host` runs a task the host misses instead of marking it unsupported, and records per-requirement `host_match` in result.json and meta.json. `repo report` groups rows by host fingerprint. Block mode is the default and unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi repo run` parameters changed:
+ added option `--label-host`Updated on every push; the |
|
OpenCodeReview: Commit face922 was not reviewed. This PR has reached its review limit (6 reviews). CI at `face922`: 12 still running, 5 passing, 1 skipped
2 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (ed0ec8e): 1 medium</summary> Open threads from the bot: 0 </details> |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #781 +/- ##
========================================
Coverage 95.79% 95.79%
========================================
Files 432 433 +1
Lines 57204 57501 +297
Branches 9858 9909 +51
========================================
+ Hits 54796 55081 +285
- Misses 1497 1508 +11
- Partials 911 912 +1
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 2ee29fd. The collector registry is clean and the label mode wiring through run_batch, run_cell and meta.json holds up. The problems are in repo report grouping and in the path redaction the privacy claim rests on. Three bugs and four risks are worth fixing before merge, plus one nit; all are inline.
What I ran, on a Linux review host at 2ee29fd:
make ci-local: 33 passed, 0 failed, 14 skipped (the skips are the other-Python legs, semgrep and jobs that do not run on a PR).- Reproductions for the report and redaction findings, through
summarize,format_report,repo reportandscrub, on hand-built batch directories, offline. - Mutation runs against
tests/test_host_fingerprint.pyandtests/test_repo_host_label.py: several mutants of the new code pass the full suite (see the test comments). - The commit has a
Signed-off-by.
Checked, no issues
- Label mode:
_verdictsreturns no unsupported entries in label mode, so old unsupported markers are cleared and--forcetakes the normal path; a failed probe givesmet=Noneand a note, not a block. - Isolation:
collect()catches per collector, and spawns getEnv.only(PATH, SYSTEMROOT), so no environment value is forwarded. - Identity:
fingerprint_idignores thenetworkcollector, as the docs say. - Docs: the collector names,
collectors_version1 and the 300-character cut in the contract match the code.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 881673e, which fix the nit from my review:
881673e:dokimi/dokimi/repos/host_fingerprint.py:8anddokimi/docs/repo-evaluation-contract.md:707: both now say the Windows CPU model isplatform.processor(), which Windows fills fromPROCESSOR_IDENTIFIER.
make ci-local at 881673e: 33 passed, 0 failed, 14 skipped.
Not approving yet. Still open: the three bugs and the four risks from my review.
Fixes in the host fingerprint and its report rows: - scrub redacts quoted paths with spaces, UNC paths and ~ paths whole. - The Windows CPU model and vendor come from the registry, not from platform.processor(), which reads PROCESSOR_IDENTIFIER. - repo report refuses a malformed host_fingerprint or host_match with exit 2 instead of a traceback. - A started missing cell counts on its own host's row; an unstarted one only when the task, arm and model has a single host row. - Text rows name their host; an undecided labelled row is not "unmet". - Tests: IPv4 check skips version strings, every identity collector, vendor, min_cpus boundary, flag case and mixed verdicts are pinned. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
Pushed 1 commit to Changes
Needs @napetrov
Threads and comments
Not this PR's
Verified (9)
|
- scrub: close a quoted path on its own quote, run a bare Windows or UNC path to the end of the line, keep image names such as ghcr.io/a/b - report: a started cell whose meta.json host has no collectors counts on the unfingerprinted row, as its result.json does Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
🟢 Ready for re-review · 1 commit on Next: @antonwolfy: re-review the new head. Nothing needed from @napetrov. Changes
Threads and comments
Verified on the new head: ✅ tests of the fix: 2 passed · ✅ full suite with main merged in: 21831 passed, 3 failures that main has too · ✅ second reader: 1 raised, all fixed What each test shows
|
scrub() left "cannot open path:/home/jdoe/private" unchanged: the colon before the slash excluded the path from redaction, and no test pinned that exclusion. Drop the colon from the lookbehind; URLs, proxy addresses and image references still come back as written. Thread: PRRT_kwDOVBg7g863vUNS Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🟡 Not approved yet · reviewed f042010
Next: @napetrov: answer 1 open thread from exai-bot (below).
Blocking
- Still holds on this head: thread by exai-bot –
scrub("cannot open path:/home/jdoe/private")still returns the input unchanged on the head.
Verified on f042010: ✅ changed tests: 238 passed, fail on main · ✅ full suite with main merged in: 21846 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried
5 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Changed tests | The five changed test files pass on the head (238 passed, 2 skipped) and fail to import on main | tests on the head |
| Local suite | The whole local run passes with main merged in | full suite, main merged in |
| Merge | The head merges onto main with no conflict | merge onto main |
| Malformed host keys | _check_cell refuses host_fingerprint as {} or 5 and a host_match field named 42 with a CellFileError |
ran it, edge input (structured) |
| Flag case | Changing flag.lower() in have to flag in have in _match_fields fails tests/test_repo_host_label.py |
broke the code, tests caught it, edge input (text) |
Each item was fixed by an earlier commit of this PR and checked on head. Thread: PRRT_kwDOVBg7g863m2nt: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863m2nu: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863m2nw: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863m2nz: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863m2n2: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863m2n5: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQc: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQf: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQi: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQk: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQm: already fixed in 881673e Thread: PRRT_kwDOVBg7g863pBQr: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQt: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pBQw: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863pIBx: already fixed in a22fd3a Thread: PRRT_kwDOVBg7g863sCH1: already fixed in f042010 Thread: PRRT_kwDOVBg7g863sCH4: already fixed in f042010 Review: PRR_kwDOVBg7g88AAAABYEpmmA: already fixed in f042010 Review: PRR_kwDOVBg7g88AAAABYEwXWw: already fixed in f042010 Comment: IC_kwDOVBg7g88AAAABmy6k2g: PR description corrected in this round (body edit, no commit) Comment: IC_kwDOVBg7g88AAAABm3tl6w: already fixed in f042010 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
vchamarthi
left a comment
There was a problem hiding this comment.
🟡 Not approved yet · re-reviewed b4fdefe
Next: @napetrov: answer 1 open thread from exai-bot (below). 1 nit optional.
Blocking
- Still holds on this head: thread by exai-bot – On the head
cannot open /home/Jane Doe/privategivescannot open <path> Doe/private: an unquoted POSIX path stops at its first space, while/home/张三/privateis redacted whole.
Verified on b4fdefe: ✅ changed tests: 239 passed, fail on main · ✅ full suite with main merged in: 22115 passed · ✅ merges onto main ·
6 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Colon path redaction | cannot open path:/home/jdoe/private gives cannot open path:<path> on the head, the case added in 6d3945e |
ran it |
| Scrub on text | scrub returns an empty or spaces-only string as given, keeps the image name ghcr.io/intel/dokimi-grader:1, and redacts a bare POSIX path with non-ASCII characters whole |
ran it, edge input (text) |
| Colon case pinned | Restoring : to the lookbehind of _PATH fails test_scrub_redacts_quoted_unc_and_home_relative_paths_whole, so the colon case is pinned |
broke the code, tests caught it |
| Changed tests | The five changed test files pass on the head: 239 passed, 2 skipped, including the colon case and the path-redaction cases | tests on the head |
| Local ci | The whole local CI run passes with main merged in | full suite, main merged in |
| Merge | The head merges onto main with no conflict | merge onto main |
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A bare POSIX path now runs to the end of the line, as a bare Windows or UNC path already did, so the rest of a path with spaces is not left behind. The regression cases cover spaces, a Unicode name, a ~ path and a URL. Thread: PRRT_kwDOVBg7g8630nuX Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864FY2z Review: PRR_kwDOVBg7g88AAAABYTPWEA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ress Resolves report.py: keeps main's batch-field check and mixed-execution refusal alongside the host-fingerprint row key and host checks. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
scrub() missed a path behind a file: scheme (file:///home/Jane Doe/x leaked whole) and a forward-slash UNC path (//server/share/x leaked). A file: path keeps its scheme, a quoted one keeps its quotes, and a UNC path is redacted to the end of the line. http and https URLs are unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Follow-up for the other URL schemes (smb://, sftp://, ssh://) in the trace scrub: #853. |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at ac2d4d7. The collectors, the label mode and the fixes from the earlier rounds hold. What is left is in the line comments: a missing cell the per-host rows lose, a fingerprint that a single timeout changes, and repo compare pooling labelled cells with no warning.
What I ran, at ac2d4d7:
make ci-local --offline: 23 passed, 0 failed, 20 skipped (network jobs and other Python legs).summarize()on hand-built batches: a crashed cell on a host with no finished cell, and a legacy batch with one new-fingerprint crashed cell. Both rows showmissing=0.collect()with a fake runner: adocker --versiontimeout givespresentand a newfingerprint_id, and anOSErrorformaketurnstoolsinto an error.
Checked, no issues
- Earlier threads: the scrub cases (quoted, UNC,
~,file:, colon, spaces), the registry source for the Windows CPU, the_check_cellrefusals and the missing-cell placement all hold on this head. - Label mode:
_verdictsgives no unsupported entries with the flag, so old markers are cleared, and the block mode is unchanged without it. - Changelog and CLI: the fragment describes the flag and the row split, and the CLI snapshot and the flags allowlist match.
A cell that started on a host with no finished cell was in the batch-level missing list only; the row for its host does not exist, so no row showed it. It now counts as unplaced, so the single-row rule places it. Thread: PRRT_kwDOVBg7g864ZGf7 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A cell run with --label-host on a host that misses its task, or that could not be checked, was pooled into the delta with no warning. compare now names those tasks, as it does for a model mismatch. Thread: PRRT_kwDOVBg7g864ZGgO Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
An OSError from the spawn of a tool that `which` found (a Windows make.cmd shim) made the whole tools collector an error. The spawn failure is now a fact about the tool, as a failing --version already is. Thread: PRRT_kwDOVBg7g864ZGgT Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The ContainerError text (docker stderr, up to 500 characters) went into _eval/host.json, meta.json and unsupported.json with paths unchanged. It is scrubbed like the other probe errors. Thread: PRRT_kwDOVBg7g864ZGgZ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Without --label-host a cell still carries host_fingerprint and repo report still splits rows by it; the contract said nothing changes. Thread: PRRT_kwDOVBg7g864ZGgd Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
summarize was over the complexity cap (ratchet) with the rowless-host step inline. Thread: PRRT_kwDOVBg7g864ZGf7 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
There was a problem hiding this comment.
Not ready to merge. Nothing new from me at ac2d4d7, but the open points from @vlad-perevezentsev still hold at this head, and they are bugs and risks.
I agree with @vlad-perevezentsev's points on the lost missing cell, the id change after one timeout, the tools collector lost to an OSError, the unscrubbed probe error, repo compare and the stale contract line. New evidence: a docker --version timeout changes the id (45f9f04d6b0c to 65ffff68073a).
What I ran, on a Linux review host at ac2d4d7:
make ci-local: 34 passed, 0 failed, 14 skipped; changed-line coverage 94%- Reproductions of the points above with fake probes, offline.
- All non-merge commits carry a
Signed-off-by.
Checked, no issues
- Earlier fixes: the report refusals, the placement of started cells, the identity test per collector and the scrub cases from the previous round hold; I have no open thread.
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
…e left out A collector that still fails after one retry is left out of fingerprint_id, and tools count by presence, so a timed-out --version does not split one host into two rows. Thread: PRRT_kwDOVBg7g864ZGgG Thread: PRR_kwDOVBg7g88AAAABYn7Jtw Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
@vlad-perevezentsev @antonwolfy @vchamarthi all threads fixed and resolved (fingerprint now uses tool presence only; smb/sftp/ssh scrub tracked in #853), CI green on head. Ready for approval. |
Summary
A repo task can need a host the grader does not have (a thread race needs 3+ CPUs, an ISA path needs avx512f). Today such a task is marked unsupported on any host that misses it, and results do not record which machine ran them.
This PR:
os,cpu,memory,gpu,python,tools,network,virt) undercollectorsin the probe facts, withcollectors_version1; a failing collector reports{"error": ...}and the others still run;repo run --label-host: a task the host misses runs anyway, and each cell recordshost_match(per requirement met, unmet or unknown) inresult.jsonandmeta.json;repo reportgroups rows by host fingerprint and names the host on each row.Block mode (the default) is unchanged.
Evidence
tests/test_host_fingerprint.py: each collector per platform (Linux, macOS sysctl, Windows with fakes), failure isolation, container and VM detection, GPU, and privacy checks run with real and monkeypatched names and addresses.tests/test_repo_host_label.py: AVX-512 task on an AMD host (labelled miss), block-mode regression, Windows host, container, GPU host, report grouping, flag parsing.make ci-localon the head of this push (main merged in): 0 failed, 25 passed, 20 skipped (network-only steps skipped).pr-review, lane L2): no confirmed blocking finding; declined nits and two questions recorded in the review.Merge Danger
result.jsongains ahost_fingerprintkey (additive). Batches written before this change group undernull.repo runandeval checknow spawns up to 10 short version calls (git,docker,cmake,gcc,clang,make,nvidia-smi), each with a 10 s timeout, once per invocation.docs/reference/cli-interface.jsonis regenerated for the new--label-hostflag.(redacts the rest of the line (ratio 3 / 4 donebecomesratio 3 <path>). Intended and fail-safe.🤖 Generated with Claude Code