Skip to content

test(onboard): keep validation test on source boundary - #5927

Merged
cv merged 2 commits into
mainfrom
codex/fix-onboard-test-src-import
Jun 28, 2026
Merged

cv merged 2 commits into
mainfrom
codex/fix-onboard-test-src-import

Conversation

@cv

@cv cv commented Jun 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Restores main's source/package test boundary after #5738 added a colocated test that constructed paths into dist/lib. The test now imports source directly and injects its credential/probe collaborators, removing the child-process and require.cache fixture while preserving the non-zero exit regression coverage.

Related Issue

Refs #5919

Changes

  • expose credential and endpoint probes through the validation helper's existing dependency boundary
  • replace the compiled-output child-process fixture with a direct Vitest source test
  • preserve assertions for process.exit(1), process.exitCode, fallback throwing, redacted diagnostics, and bypassed recovery prompting

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: testability refactor with no user-facing behavior change
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: CodeQL, CodeRabbit, and both PR Review Advisor checks passed with no blocking findings or inline comments
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Targeted evidence: focused Vitest test; npm run build:cli; npm run typecheck:cli; npm run checks; normal commit and push hooks (including the full CLI test hook). The source-level test is sufficient for this testability-only refactor; existing onboarding integration tests and the strict-tool-call probe driver retain caller/runtime coverage.


Signed-off-by: Carlos Villela cvillela@nvidia.com

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

InferenceSelectionValidationDeps gains three optional properties (getCredential, probeAnthropicEndpoint, probeOpenAiLikeEndpoint). createInferenceSelectionValidationHelpers resolves these to local variables that fall back to module-level imports, and all four validation functions now call those resolved variables. The integration test that spawned a Node subprocess is replaced with an in-process Vitest test using vi.spyOn/vi.fn.

Changes

Dependency injection and test rewrite

Layer / File(s) Summary
InferenceSelectionValidationDeps interface and resolver wiring
src/lib/onboard/inference-selection-validation.ts
Adds optional getCredential, probeAnthropicEndpoint, and probeOpenAiLikeEndpoint to the interface; createInferenceSelectionValidationHelpers derives resolveCredential, runAnthropicProbe, and runOpenAiLikeProbe by preferring injected deps over module defaults.
Resolver usage in validation functions
src/lib/onboard/inference-selection-validation.ts
Switches validateOpenAiLikeSelection, validateAnthropicSelectionWithRetryMessage, validateCustomOpenAiLikeSelection, and validateCustomAnthropicSelection from direct module-level function calls to the resolved local variables.
In-process Vitest test
src/lib/onboard/inference-selection-validation.test.ts
Replaces the spawned-Node-process test (temp files, require.cache patching) with a Vitest in-process test that injects mock probe/credential deps and asserts rejection message, process.exit(1), process.exitCode, promptValidationRecovery not called, and console.error output.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5738: Introduced or preserved the process.exit/process.exitCode behavior in validateOpenAiLikeSelection that this PR's new in-process test now directly asserts.

Suggested reviewers

  • ericksoa
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: keeping the onboard validation test on the source boundary.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-onboard-test-src-import

Comment @coderabbitai help to get the list of available commands.

@cv cv added chore Build, CI, dependency, or tooling maintenance area: ci CI workflows, checks, release automation, or GitHub Actions area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow labels Jun 28, 2026
@cv cv self-assigned this Jun 28, 2026
@cv cv added the v0.0.70 label Jun 28, 2026
@github-code-quality

github-code-quality Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the codex/fix-onboard-te... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/fix-onboard-te... a1a550b +/-
nemoclaw/src/se...cret-scanner.ts — 100% —
nemoclaw/src/commands/slash.ts — 100% —
nemoclaw/src/li...bprocess-env.ts — 100% —
nemoclaw/src/bl...eprint/state.ts — 98% —
nemoclaw/src/onboard/config.ts — 98% —
nemoclaw/src/bl...int/snapshot.ts — 97% —
nemoclaw/src/bl...print/runner.ts — 95% —
nemoclaw/src/co...ration-state.ts — 94% —
nemoclaw/src/bl...ate-networks.ts — 94% —
nemoclaw/src/index.ts — 94% —

TypeScript / code-coverage/cli

The overall coverage in the codex/fix-onboard-te... branch is 68%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/fix-onboard-te... a1a550b +/-
src/lib/actions...all/run-plan.ts — 80% —
src/lib/state/o...oard-session.ts — 79% —
src/lib/actions...dbox/rebuild.ts — 74% —
src/lib/state/sandbox.ts — 72% —
src/lib/shields/index.ts — 70% —
src/lib/onboard/preflight.ts — 69% —
src/lib/actions...licy-channel.ts — 59% —
src/lib/onboard...er-gpu-patch.ts — 59% —
src/lib/policy/index.ts — 52% —
src/lib/onboard.ts — 20% —

Updated June 28, 2026 18:29 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: No advisor follow-up required beyond maintainer review.
Open items: 0 required · 0 warnings · 0 suggestions · 0 test follow-ups
Since last review: 3 prior items resolved · 0 still apply · 0 new items found

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 4 test follow-ups
Since last review: 0 prior items resolved · 2 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Acceptance clause
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add or identify a package-contract or CLI-level non-interactive onboarding validation test that stubs a failed OpenAI-like endpoint probe and verifies the process exits with status 1.. The source test is focused and sufficient for the local dependency-injection refactor, but the touched path is onboarding/inference/credentials and the deleted test previously exercised a built package/process boundary.
  • PRA-T2 Runtime validation — In that runtime/package-boundary test, assert recovery prompting is not invoked and stderr includes only the summarized HTTP/curl probe status while excluding the API key or raw probe details.. The source test is focused and sufficient for the local dependency-injection refactor, but the touched path is onboarding/inference/credentials and the deleted test previously exercised a built package/process boundary.
  • PRA-T3 Runtime validation — Add or identify deterministic issue Complete the Vitest E2E cutover with job-variant parity evidence #5919 acceptance coverage once the issue clauses/comments are available, mapping each literal clause to the source test, package-boundary test, or existing onboarding integration coverage.. The source test is focused and sufficient for the local dependency-injection refactor, but the touched path is onboarding/inference/credentials and the deleted test previously exercised a built package/process boundary.
  • PRA-T4 Acceptance clause — Refs Complete the Vitest E2E cutover with job-variant parity evidence #5919 — add test evidence or identify existing coverage. The deterministic validation context reported linkedIssues: [], so no literal issue Complete the Vitest E2E cutover with job-variant parity evidence #5919 body clauses or comments were available to verify. The PR body reference is untrusted evidence only.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: onboard-negative-paths-e2e, inference-routing-e2e, cloud-onboard-e2e
Optional E2E: cloud-inference-e2e, openclaw-inference-switch-e2e

Dispatch hint: onboard-negative-paths-e2e,inference-routing-e2e,cloud-onboard-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • onboard-negative-paths-e2e: Directly exercises non-interactive onboarding failure behavior for invalid credentials/endpoints, matching the changed validation failure and exit-path contract.
  • inference-routing-e2e: Covers inference-provider validation classifications, including invalid-key and unreachable-endpoint behavior, downstream of the changed endpoint validation helper.
  • cloud-onboard-e2e: The changed helper is used during full hosted onboarding for NVIDIA Endpoints/OpenAI-like provider validation, so a real hosted onboard smoke is required to catch regressions not visible in unit tests.

Optional E2E

  • cloud-inference-e2e: Useful adjacent confidence that a hosted onboarded sandbox can still serve inference through inference.local after provider validation succeeds.
  • openclaw-inference-switch-e2e: Optional because the same validation helpers are relevant when switching OpenClaw inference providers, but this PR primarily changes onboard-time validation rather than the switch workflow.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/nightly-e2e.yaml
  • jobs input: onboard-negative-paths-e2e,inference-routing-e2e,cloud-onboard-e2e

@github-actions

github-actions Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: ubuntu-repo-cloud-openclaw
Optional Vitest E2E scenarios: ubuntu-repo-cloud-langchain-deepagents-code

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

  • ubuntu-repo-cloud-openclaw: The PR changes onboarding inference selection validation helpers used during non-interactive provider/model validation. The live-supported OpenClaw cloud scenario is the smallest dispatch that exercises the standard NVIDIA/OpenAI-like onboarding validation path through the Vitest scenario workflow.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional Vitest E2E scenarios

  • ubuntu-repo-cloud-langchain-deepagents-code: Optional adjacent coverage for the other live-supported cloud onboarding flavor that may also traverse inference endpoint validation, but it is not the primary OpenClaw path touched by this fix.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-langchain-deepagents-code

Relevant changed files

  • src/lib/onboard/inference-selection-validation.ts

@cv cv mentioned this pull request Jun 28, 2026
9 of 21 tasks
laitingsheng pushed a commit that referenced this pull request Jun 28, 2026
Cherry-pick of 23a9b6e from origin/codex/fix-onboard-test-src-import (PR #5927) so this branch can rebuild dist without tripping the no-test-dist-imports guard. Refactors the validation test to drive createInferenceSelectionValidationHelpers from src instead of spawning a child node process against dist artifacts.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
…t-src-import

# Conflicts:
#	test/package-contract/inference-selection-validation.test.ts
@cv

cv commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator Author

Refreshed this bootstrap onto the v0.0.69 release commit in signed merge commit a1a550b.

The only conflict was the tactical #5928 move of the compiled-output test into test/package-contract. I resolved it toward this PR's intended steady state:

  • the behavioral test is again co-located under src/ and imports source;
  • it runs in-process with explicit probe/credential dependencies;
  • the package-contract copy and child-process/dist harness are removed;
  • all v0.0.69 product and E2E stabilization changes are retained.

Post-resolution evidence:

  • focused CLI test: 1/1 passed;
  • source/package import boundary passed;
  • Vitest project disjointness passed (898 files / 8 projects);
  • clean CLI build passed;
  • CLI typecheck passed;
  • repository checks passed;
  • normal commit and push hooks passed;
  • both PR commits report Verified in GitHub.

The diff against current main remains limited to the three intended source/package-boundary files.

@cv

cv commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator Author

Fresh-head GPT advisor test-follow-up disposition:

  • PRA-T1/PRA-T2: existing coverage is the intended boundary. The deleted child process did not exercise a public package contract; it used require.cache surgery against three compiled internals solely to inject probe and credential behavior. The new source test directly drives the same non-interactive branch and asserts the rejection, process.exit(1), process.exitCode=1, no recovery prompt, and the exact credential-safe stderr summary. Node's mapping of process.exit(1) to status 1 is platform/runtime behavior, not a NemoClaw package invariant. Reintroducing a dist-backed duplicate would undo the source/package separation this bootstrap exists to restore.
  • PRA-T3/PRA-T4: Complete the Vitest E2E cutover with job-variant parity evidence #5919 is the GitHub-only migration acceptance ledger, not a product behavior clause for this three-file bootstrap. Its source/package trust-foundation requirement is satisfied by the import boundary and disjoint-project checks now passing. Domain parity and live paired-run evidence remain tracked in Complete the Vitest E2E cutover with job-variant parity evidence #5919 and are deliberately not duplicated as tests in this PR.

No additional test is warranted here. The focused source test, clean build, typecheck, import-boundary check, project-overlap check, repository checks, and fresh CI static-checks all pass.

@cv
cv merged commit 7f75831 into main Jun 28, 2026
42 checks passed
@cv
cv deleted the codex/fix-onboard-test-src-import branch June 28, 2026 20:26
@cjagwani cjagwani mentioned this pull request Jun 30, 2026
11 of 21 tasks
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
## Summary

Restores `main`'s source/package test boundary after NVIDIA#5738 added a
colocated test that constructed paths into `dist/lib`. The test now
imports source directly and injects its credential/probe collaborators,
removing the child-process and `require.cache` fixture while preserving
the non-zero exit regression coverage.

## Related Issue

Refs NVIDIA#5919

## Changes

- expose credential and endpoint probes through the validation helper's
existing dependency boundary
- replace the compiled-output child-process fixture with a direct Vitest
source test
- preserve assertions for `process.exit(1)`, `process.exitCode`,
fallback throwing, redacted diagnostics, and bypassed recovery prompting

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: testability refactor with no
user-facing behavior change
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: CodeQL, CodeRabbit, and
both PR Review Advisor checks passed with no blocking findings or inline
comments
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [x] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Targeted evidence: focused Vitest test; `npm run build:cli`; `npm run
typecheck:cli`; `npm run checks`; normal commit and push hooks
(including the full CLI test hook). The source-level test is sufficient
for this testability-only refactor; existing onboarding integration
tests and the strict-tool-call probe driver retain caller/runtime
coverage.

---

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant