Skip to content

ci: retain agent npm installation evidence - #1800

Open
lilly-luo wants to merge 2 commits into
mainfrom
isaac/lilly.luo/ci-agent-install-evidence-20261120
Open

lilly-luo wants to merge 2 commits into
mainfrom
isaac/lilly.luo/ci-agent-install-evidence-20261120

Conversation

@lilly-luo

@lilly-luo lilly-luo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What did you change, and why?

The main Pi/Linux shard failed before tests because npm returned 404 for the pi-coding-agent@1.6.0 tarball (job); later runs installed successfully. The existing evidence cannot distinguish publication from registry propagation issues.

Enable npm HTTP/timing logs. After failed installs, retain installation time, Node/npm versions, package specification and the complete installed global dependency tree for seven days. Preserve installation failures and archive only two explicit evidence files, excluding raw npm debug logs and configuration. Package versions and assertions stay unchanged.

How do you know it works?

  • Executed the workflow shell with external command substitutes for exits 0/1/42: context is retained and exact installer status preserved.
  • YAML parsing, artifact path/retention/guard checks, the real full-tree npm command, whitespace checks and commit/push secret hooks passed.
  • CI attempt 2 completed SUCCESS with all required checks green. The pre-existing advisory VS Code click failure is covered by test(ci): open the native VS Code model picker with Enter #1805, whose four native journeys passed on both Linux and Windows.

Local Python contract execution was blocked by internal PyPI proxy DNS resolution. Live failed-install artifact delivery was not reproduced. tests/README.md is unchanged.

Checklist before requesting review

  • Comment /ug-review and address the comments. The connector footer prevented exact matching; the supported ug-review label triggered the completed APPROVE review. Both threads are addressed and resolved. Polly reported no blockers/security issues; its optional test suggestion is answered.
  • Ordinary PR CI provided integration validation.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

/ug-review


This comment was generated with GitHub MCP.

@lilly-luo lilly-luo added the ug-review Run the automated UG review label Nov 20, 2026
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
@github-actions

Copy link
Copy Markdown
Contributor

UG review recommendation

RECOMMENDATION: APPROVE

Reasons:

  • The --depth=0 finding (medium) affects only how useful the CI diagnostics are. It doesn't break existing workflows or the build. Scoping to npm ls -g "$AGENT_NAME" --all --json is a worthwhile follow-up, not a blocker.
  • The redundant package != '' condition (low) is a maintainability nit with no change in behavior.
  • Uploading artifacts from all 12 shards on green runs is an optional storage and cost question. It isn't a correctness or safety issue.
  • None of the findings involve data loss, credential leakage, a reproducible bug, or missing regression coverage.

Findings considered: 2 (severities: low: 1, medium: 1); see the inline review comments.

Advisory only: this recommendation never approves or merges the pull request. The ready-to-approve label tracks it.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None found.

Security vulnerabilities

None found. The artifact includes two explicit evidence files, not npm debug logs, configuration, or environment dumps. No permissions or trust boundaries change.

Non-blocking notes

  • can we add stubbed install cases in tests/test_integration_contract.py for exits 0/1/42, retained context, and tolerated tree failure?

Approach

The approach is sound. Leaving installation unpiped preserves its exit status; collecting the tree separately allows that diagnostic to fail without masking installation failure.

Summary

Diff size (computed): 1 files; +23 / -1 lines (24 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 0 0 0 0.0%
Documentation 0 0 0 0.0%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 1 23 1 100.0%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

The workflow adds installation context, npm HTTP/timing output, a depth-zero global package tree, and seven-day artifact retention after success or failure.

The implementation is proportional to the diagnostic goal. No unnecessary scaffolding or concrete size reduction was identified. Package specifications and installation selection remain unchanged.

Tests

Test-by-test assessment

No tests or test-support files changed. Existing contracts protect workflow security and required-check propagation, but do not cover the new evidence handling.

Test / case Behavior protected Layer Needed? Action / rationale
tests/test_integration_contract.py::test_no_workflow_uses_pull_request_target Workflow event trust boundary Static contract keep Distinct security protection; passed through standalone execution, not pytest.
tests/test_integration_contract.py::test_security_scan_runs_before_every_job_that_runs_pr_code Security scanning precedes execution of PR code Static contract keep Passed against reconstructed workflow content; does not validate evidence handling.
tests/test_integration_contract.py::test_required_e2e_enforces_integration_result Required-check failure propagation Bash subprocess keep Six parameter cases passed through standalone execution; protects a separate failure boundary.

Offline execution of the changed installation script with stub commands confirmed exit codes 0, 1, and 42 remain unchanged and context is retained. This validates shell behavior, not live npm installation or artifact delivery.

The missing regression coverage belongs in offline Bash cases and focused workflow assertions, not another live agent E2E. Full pytest execution, Windows behavior, actual npm HTTP/timing output, and GitHub artifact delivery remain unverified. API and product-telemetry evidence are not applicable.

To verify delivery, inspect successful and failed installation runs: download the artifact, check both evidence files, confirm seven-day retention, and confirm installation errors still fail the installation step.

Scope

All changes support retaining installation evidence in .github/workflows/ci.yml. No unrelated changes or unresolved scope questions were found.


Automated review by Polly · workflow run

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Addressed both UG review threads in cec4c91 and resolved them. For Polly's non-blocking test note: the actual workflow shell was executed with external command substitutes for exits 0/1/42, and each retained context while preserving the installer status. YAML parsing also checks the explicit artifact paths, retention, and failure-only guards. I kept this declarative workflow change scoped to one file rather than adding Python tests that duplicate shell/GitHub runner semantics. Live artifact delivery on a failing install remains unverified; raw npm debug logs are excluded. tests/README.md remains unchanged.


This comment was generated with GitHub MCP.

@lilly-luo

lilly-luo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Current-head CI has four failures outside the changed npm installation step: Windows Cowork permission-dialog readiness, VS Code tooltip interception, CUJ7 upstream routing HTTP 422, and managed Codex HTTP 429 exhaustion. Recent main CI passed those jobs; the earlier PR head also passed CUJ7.

Follow-ups are open: #1802 records Cowork wait/snapshot evidence; #1803 retains sanitized routing request metadata; #1804 captures immediate click-timeout evidence while preserving the original click and state-recovery timeouts. CUJ7 passed all ten tests on the routing PR, and its live artifacts contain the intended metadata. The Cowork native diagnostic self-check has passed in both Windows jobs. The final VS Code diagnostic revision is undergoing fresh review.

CI attempt 1 has completed, and GitHub accepted the failed-jobs-only rerun on the same cec4c91 commit (run 44751086897, attempt 2). These changes do not claim to fix upstream rate limiting or the unresolved backend 422 cause.

@lilly-luo

lilly-luo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Follow-up CI artifacts explicitly report account-rate-limit HTTP 429 responses in Claude/Cowork relay jobs, including wrappers that otherwise look like UI timeouts. Codex jobs also exhaust native retries on HTTP 429; the exact quota behind those responses is unverified. These are separate from the npm diagnostic change.

A lasting capacity decision has a trade-off: increasing test-account/provider limits changes resource usage, while globally serializing integration runs increases queue time substantially. I am leaving both settings unchanged in this unattended run and recommend that the CI/account owners choose the capacity budget. The failed-only request re-executed the four failed direct jobs and downstream advisory jobs. Security Gate, Lint and Unit results inherited their original execution timestamps despite receiving fresh job IDs. Remaining retries will target confirmed failures after each workflow completes; the PRs preserve failure evidence and do not weaken assertions or add task retries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants