Skip to content

fix(pr-management): verify CI status check rollup belongs to PR head (part 1 of #2034) - #2061

Open
Kaap10 wants to merge 2 commits into
apache:mainfrom
Kaap10:fix/ci-rollup-head-oid-verification
Open

Kaap10 wants to merge 2 commits into
apache:mainfrom
Kaap10:fix/ci-rollup-head-oid-verification

Conversation

@Kaap10

@Kaap10 Kaap10 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • When a PR branch is updated or force-pushed, GitHub GraphQL's commits(last: 1) can lag behind or return a statusCheckRollup belonging to a prior commit rather than the PR's true headRefOid.
  • In cr-open.graphql and cr-pr.graphql, request oid on latest: commits(last: 1) { nodes { commit { oid statusCheckRollup ... } } }.
  • In pr_management.code_review.data.from_node():
    • Guard against lagging commit edges: verify commit_oid strictly matches headRefOid (case-normalized); on mismatch, fail closed by clearing rollup = {}.
    • Handle query skew defensively: if statusCheckRollup exists but commit_oid is absent (e.g. an older installed vetted-ops that lacks oid under latest), fail closed on rollup and set ci_outdated_query = True.
  • In disposition.pick():
    • When ci_outdated_query is set, block APPROVE with an explicit gate ("CI rollup cannot be verified — vetted-ops is out of date (latest commit lacks OID)") and provide an actionable ci_action ("upgrade vetted-ops to update GraphQL queries, then re-run disposition"), rather than silently showing "CI is not reported".

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • ruff check passes across tools/ and plugins/
  • ruff format --check passes across tools/ and plugins/
  • mypy passes (76 source files checked, 0 issues) in both tools/ and plugins/
  • pytest tools/pr-management/tests/code_review/ passes (241 passed)
  • Dual-tree parity verified: byte-for-byte identical match between tools/ and plugins/ for pr_management and vetted_ops
  • Added unit tests in test_data.py:
    • test_from_node_accepts_matching_head_and_commit_oid
    • test_from_node_discards_rollup_when_commit_oid_mismatches_head_ref_oid (lagging commit edge guard)
    • test_from_node_discards_rollup_and_flags_outdated_query_when_commit_oid_missing (query skew handling)
    • test_from_node_discards_rollup_when_head_ref_oid_missing
    • test_from_node_accepts_rollup_case_insensitively
  • Added integration tests in test_decisions.py:
    • test_pr_with_lagging_commit_edge_fails_closed_to_comment
    • test_pr_with_outdated_vetted_ops_query_gates_approval_with_upgrade_action

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Sandbox — no new unrestricted host access; network reach declared in the adapter
  • Vendor neutrality — placeholders (<PROJECT>, <tracker>, <upstream>, <security-list>) used in all skill / tool prose (the check-placeholders prek hook is the mechanical gate)
  • Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable
  • Write-access discipline — no autonomous outbound messages; drafts only, sent on confirmation
  • Privacy LLM — private content does not reach a non-approved LLM; redactor invoked where needed

Linked issues

Refs #2034 (part 1 of 2: Direction 1 fail-closed guard against lagging GraphQL commit edges, plus query-skew defense)

Notes for reviewers (optional)

…(part 1 of apache#2034)

* Query commit OID under latest: commits(last: 1) in cr-open.graphql and cr-pr.graphql
* In data.py from_node(), verify commit.oid == headRefOid case-insensitively and fail closed (discard rollup) on mismatch or missing OID
* When discarded, disposition.py blocks APPROVE with 'CI is not reported'
* Add regression tests in test_data.py and test_decisions.py

Refs apache#2034
@github-actions github-actions Bot added family:tools tools/* substrate:analytics Tool substrate: read-only metrics / dashboards / renderers substrate:sandbox Tool substrate: agent isolation, egress control, settings audit labels Nov 13, 2026

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a fail-closed check that the CI rollup belongs to the head commit, but the case it tests is not the one #2034 reported, and a missing oid (an older vetted-ops install) looks the same as an unreported CI.

Smaller observations

  • See inline comments on tools/pr-management/tests/code_review/test_data.py:82, tools/pr-management/src/pr_management/code_review/data.py:218.

Worth a second look from

Folks with the most context on these paths:

  • @potiuk — 6 recent commit(s) on the changed paths (frequent contributor, not a committer)

None of them have been notified — asking any of them for an
extra pass is the maintainer's call, and optional.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
Contributing guide.

… scoping

- In data.py, distinguish missing commit OID from mismatched OID when a rollup exists
- Flag ci_outdated_query on PR so disposition can gate with an explicit upgrade instruction
  rather than leaving maintainers guessing why CI is unreported
- Clarify test docstrings and naming to reflect Direction 1 (lagging commit edge guard)
  without overstating coverage of same-head stale rollups
- Sync dual-tree mirrors under plugins/magpie-security-tools
@Kaap10
Kaap10 requested a review from shahar1 November 13, 2026 06:12

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The head-commit check and the missing-oid handling look right and both earlier threads are addressed; two smaller points remain, about where the out-of-date-query signal shows up and the texts that describe the CI gates.

Smaller observations

  • See inline comments on tools/pr-management/src/pr_management/code_review/cli.py:446, tools/pr-management/src/pr_management/code_review/disposition.py:148.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
Contributing guide.

merge_method=pr.default_merge_method,
pending_not_required=pr.pending_not_required,
viewer_known=bool(pr.viewer),
ci_outdated_query=pr.ci_outdated_query,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainers are immediately informed

A missing OID fails closed to prevent unverified approvals, but surfaces an explicit "vetted-ops is out of date" approval gate and ci_action so maintainers are immediately informed.

ci_outdated_query reaches only disposition (Step 6). Under the same skew, context still prints CI: none reported (_ci_line reads only rollup_state) and returns "ci": {"state": null, "real_ci": false}, and queue ranks every PR as real-CI-never-ran with no announcement. The maintainer reads the wrong headline for the whole review and learns the cause at the last step. Have _ci_line return something like CI: cannot be verified (vetted-ops query out of date) when the flag is set, and carry the flag in the ci object; a queue announcement can follow in part 2.

ci_action = "approve the waiting workflow runs (pr-management-triage), then re-run disposition"
elif ci_outdated_query:
gates.append("CI rollup cannot be verified — vetted-ops is out of date (latest commit lacks OID)")
ci_action = "upgrade vetted-ops to update GraphQL queries, then re-run disposition"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spec-sync pre-check before pushing a functionality PR

Before pushing a PR that ships or changes a skill, tool, or mode — and before pushing a rebase of one onto a main that has moved — confirm tools/spec-loop/.last-sync is at (or near) the current main tip and that the affected specs reflect what actually shipped.

This adds a CI gate and a third meaning of ci_action, and two texts still describe the old set. tools/spec-loop/specs/pr-management-family.md lists the gates as "Failing CI, a rollup green from bot checks only, and workflows waiting for approval". plugins/magpie-pr-management/skills/code-review/review-flow.md (Step 6) tells the agent "When ci_action is set, the project's workflows have not run (waiting for approval, or bot checks only)", which is now wrong for the upgrade case: the agent would point the maintainer at workflow approval. Add the head-commit rule and the out-of-date-query gate to both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

family:tools tools/* substrate:analytics Tool substrate: read-only metrics / dashboards / renderers substrate:sandbox Tool substrate: agent isolation, egress control, settings audit

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants