Skip to content

fix(cli): protect the selected baseline from report output - #986

Open
chrisknvidia wants to merge 2 commits into
NVIDIA:mainfrom
chrisknvidia:feat/christopherk/protect-consumed-baseline
Open

chrisknvidia wants to merge 2 commits into
NVIDIA:mainfrom
chrisknvidia:feat/christopherk/protect-consumed-baseline

Conversation

@chrisknvidia

@chrisknvidia chrisknvidia commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #984.

Scanning with an explicit or author-shipped baseline now rejects a report destination that aliases the consumed baseline before analysis starts. The preflight compares both the supplied path and the exact normalized destination used by the writer, closing the missing/../.skillspector-baseline.yaml overwrite reported in review. The original filesystem identity check remains for physical symlink/.. paths. Separate report destinations and intentional baseline generation/reuse remain supported.

Current head 7c125d56, based on main ea5aeef6. Current-head CI is running. The prior green run predates this fix and is not current-head validation.

Validation at this head:

  • 17 new regression/control cases cover explicit/shipped baselines, JSON/SARIF, missing/nested-missing components, non-directory parents, normalized symlink parents and original physical aliases. The corrected pre-fix selection reproduced 12 failures; the fixed targeted selection passes all 35 tests.
  • 630 affected tests passed, with one platform skip. Full Ruff lint/format and diff checks passed.
  • 94 actual source/noneditable-wheel CLI runs passed: 45 paired scenarios plus four baseline reuse scans. Rejections preserve baseline bytes; positive controls cover separate normalized destinations, strict exits, suppression, precedence and generation/reuse.
  • All 45 source/wheel behavior pairs matched; retained outputs include 12 JSON and six project-model-validated SARIF reports. All 148 package files and 158 installed distribution versions matched across frozen environments.
  • Independent code/security review found no actionable findings.

Validation used static CLI scans on macOS. No live model, customer production, Windows, hostile concurrent filesystem mutation or full integration suite was tested. Remote/ZIP cases exercise preflight rejection only. SARIF validation uses project models, not external full-schema/service ingestion. Native typed review bots were unavailable. Initial new-test message mismatches and a test-selection error were retained and corrected without weakening byte-preservation, rejection or no-scan assertions.

@chrisknvidia
chrisknvidia force-pushed the feat/christopherk/protect-consumed-baseline branch from fd4ac05 to 2f2cafc Compare November 4, 2026 17:00
@chrisknvidia
chrisknvidia marked this pull request as ready for review November 4, 2026 19:15
@chrisknvidia
chrisknvidia requested a review from rng1995 November 4, 2026 19:15
@chrisknvidia

Copy link
Copy Markdown
Collaborator Author

Ready for review at 2f2cafc2. All six hosted checks pass, including 19,281 tests in the configured unit selection. Validation also covers 543 focused tests and 54 real source/wheel CLI runs, plus two additional baseline-preservation controls. Coverage and limits are documented in the description.

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source review found one remaining path alias that can overwrite the selected baseline. I reviewed 7a17737dbdb8f2cde8015237a096ce676a1c68de...2f2cafc258523b2944bb5e9ee4659a767f469d3b; Greptile completed for the same commits with no additional findings. Runtime tests, builds, source/wheel verification, and corpus/provider scans were not performed. The inline finding is supported by the source trace and has not been reproduced at runtime.

Comment thread src/skillspector/cli.py Resolved

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested.

Value and readiness

Rejecting reports that alias a consumed baseline is the right fix for #984. The added check covers direct, relative, symlink and hardlink aliases, but the writer and preflight still disagree on a missing-component path, leaving the original data loss reachable.

Material findings

Blocker — src/skillspector/cli.py:930-931: compare the writer's normalized destination. For a valid skill/.skillspector-baseline.yaml with no skill/missing directory, skillspector scan skill --no-llm --format json --use-shipped-baseline -o skill/missing/../.skillspector-baseline.yaml reaches this new check. _validate_report_destination at lines 348-374 sees output.exists() as false and allows lstat's FileNotFoundError. Analysis then consumes the baseline, but write_text_no_follow calls _normalize_root_owned_alias; os.path.abspath collapses missing/.., so the report replaces the baseline at the normalized path. The same route applies to explicit --baseline. Use the writer's normalization for the preflight comparison, or reject the unresolved spelling consistently, and add a preservation assertion for the missing-component alias before any scan. This independently confirms the existing unresolved finding; no duplicate inline comment is needed.

PIC tradeoffs

None identified. The requested normalization closes the advertised baseline-preservation boundary without changing intentional baseline generation.

Verification and gaps

Read the complete two-file diff, exact-head baseline selection and output flow, writer and normalization helper, all 16 added parameter combinations, linked #984 and complete review/thread history. All six attached checks report success in run 41873275570. Those tests exercise existing aliases and do not include this missing-component form. The consequence follows from the source trace; no runtime reproduction is claimed.
Static source review only. No contributor code, tests, snippets, imports, builds, setup, hooks, scanner invocations or model calls were executed.

Reviewed head: 2f2cafc258523b2944bb5e9ee4659a767f469d3b.

Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scan report output can overwrite its selected suppression baseline

3 participants