Repository navigation
fix(ci): isolate review follow-up and self-host coalescing #2618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
e32c782
fix(ci): isolate review follow-up and self-host coalescing
seonghobae c3e689f
fix(review): recognize authenticated independent Noema App evidence
seonghobae 01f868d
Merge current central main while preserving reviewed repair boundaries
seonghobae b4c3e91
Merge current central main for isolated review repair
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
84 changes: 84 additions & 0 deletions
84
docs/doctoring/central-review-handoff-and-runner-20261009.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # Central review handoff and runner repair | ||
|
|
||
| ## Scope | ||
|
|
||
| This change repairs existing review control paths. It does not claim that the | ||
| organization's hosted workload migration, LiteLLM onboarding, or product | ||
| validation is complete. | ||
|
|
||
| - Central Noema handoff/base binding is already owned by | ||
| ContextualWisdomLab/.github#2111. A duplicate implementation was removed | ||
| from this change; this patch does not claim that repair is deployed. | ||
| - OpenCode's immediate post-approval scheduler invocation is dry-run, | ||
| merge-disabled, auto-merge-disabled, and branch-update-disabled. A formal | ||
| review is evidence, not permission for that job to mutate the reviewed PR. | ||
| - The coalescing tick is a trusted central-main schedule with no PR checkout. | ||
| It selects the already-admitted `CWL MCP remediation` group with | ||
| `self-hosted`, `linux`, and `x64` labels. The existing job-level enable flag | ||
| still prevents inert ticks from obtaining a runner. | ||
| - The coalescing invocation explicitly selects `--merge-mode disabled` as | ||
| well as disabling auto-merge and branch updates. Disabling auto-merge alone | ||
| does not disable a direct-merge mode. | ||
|
|
||
| The regular review-event and scheduled merge scheduler paths are unchanged. | ||
| This patch does not establish an end-to-end no-merge guarantee for a PR that | ||
| those separately authorized paths inspect. | ||
|
|
||
| ## Current operational evidence | ||
|
|
||
| GitHub check-run annotations for | ||
| `ContextualWisdomLab/psychometrics-commons#165`, head | ||
| `bb9028714d996871ed2588ea9a3aa83fd581af84`, reported that the hosted job could | ||
| not start because the account was locked due to a billing issue. The jobs | ||
| had runner id `0` and no executed steps. They are not executed test failures. | ||
|
|
||
| The product fix was previously verified locally, including branch coverage | ||
| 1724/1724. That evidence does not replace GitHub required checks, independent | ||
| review, or an admitted isolated self-hosted build lane. | ||
|
|
||
| An organization runner-group readback showed `CWL MCP remediation` already | ||
| admitted the coalescing workflow at central `refs/heads/main`, with two online | ||
| runners. Runtime availability is re-read before dispatch; this observation is | ||
| not a permanent capacity claim. | ||
|
|
||
| ## Remaining prerequisites | ||
|
|
||
| - Product PR execution must use a verified isolation boundary; do not send | ||
| untrusted builds to shared metadata/review runners. | ||
| - The scheduler recognizes only the canonical GitHub-authenticated Noema | ||
| App as an exception to blanket bot exclusion. It requires the real review | ||
| commit to match the exact head, a coherent publisher head/result/actor/App | ||
| credential receipt, and non-author identity after REST/GraphQL normalization. | ||
| The latest relevant formal review is resolved before receipt validation: | ||
| revocations or malformed latest approvals cannot resurrect older approvals. | ||
| This establishes independent-review eligibility only; GitHub's aggregate | ||
| approval policy, checks, security evidence, threads, and merge freshness | ||
| remain separate and unchanged. | ||
| - Current review configuration remains `orchestrator/free`; it is not a | ||
| literal `auto` model setting. An authenticated LiteLLM onboarding and | ||
| explicit gateway contract/policy update are required before claiming the | ||
| requested `auto` route. | ||
| - The public LiteLLM schema at `https://litellm.poinnetworks.net/openapi.json` | ||
| declares authentication for `/key/generate`; no unauthenticated key was | ||
| requested or issued. A dedicated inference credential must enter the | ||
| gateway's approved secret boundary, not source, logs, or review text. | ||
| - No hosted security or product-validation workflow was disabled by this | ||
| patch. Mandatory unavailable evidence stays non-passing until an equivalent | ||
| isolated replacement is verified. | ||
|
|
||
| ## Verification | ||
|
|
||
| Regression tests enforce assessment-only post-publication scheduler arguments | ||
| and pin the trusted coalescer's runner selection. Workflow shell syntax and | ||
| actionlint validate the modified callers. Two pre-existing CodeQL tests now | ||
| freeze their default-clock input so September fixtures do not expire under | ||
| October's real wall clock; production freshness thresholds are unchanged. | ||
|
|
||
| ## References | ||
|
|
||
| - GitHub. (n.d.). *Secure use reference*. | ||
| https://docs.github.com/en/actions/reference/security/secure-use | ||
| - GitHub. (n.d.). *Events that trigger workflows*. | ||
| https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows | ||
| - LiteLLM. (n.d.). *Self serve*. | ||
| https://docs.litellm.ai/docs/proxy/self_serve | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,160 @@ | ||
| """Recognize only authenticated exact-head Noema App review receipts.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| from copy import deepcopy | ||
|
|
||
| import pytest | ||
|
|
||
| from scripts.ci import noema_review_gate as publisher | ||
| from scripts.ci import pr_review_merge_scheduler_core as scheduler | ||
|
|
||
| HEAD = "a" * 40 | ||
| BASE = "b" * 40 | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def published_review(monkeypatch: pytest.MonkeyPatch) -> dict: | ||
| """Capture the real publisher without a network write or model call.""" | ||
| payloads = [] | ||
|
|
||
| def capture(_args, *, stdin=None): | ||
| payloads.append(json.loads(stdin)) | ||
| return "" | ||
|
|
||
| monkeypatch.setattr(publisher, "run", capture) | ||
| monkeypatch.setenv("NOEMA_REVIEW_TOKEN_SOURCE", "noema-review-github-app-refresh") | ||
| publisher.submit_review( | ||
| "ContextualWisdomLab/example", 7, | ||
| {"headRefOid": HEAD, "baseRefOid": BASE}, "cwl-noema-review[bot]", | ||
| { | ||
| "decision": "approve", "summary": "Changed-line review found no blocking regression.", | ||
| "findings": [], | ||
| "reviewed_lines": [{"path": "src/runtime.py", "line": 12, "side": "RIGHT", "analysis": "Invalid references are rejected before lookup."}], | ||
| "adversarial_validation": { | ||
| "status": "passed", "residual_risk": "Deployment evidence remains separate.", | ||
| "probes": [{"path": "src/runtime.py", "line": 12, "side": "RIGHT", "outcome": "falsified", "hypothesis": "Invalid references reach lookup.", "evidence": "Fixture test at src/runtime.py:12 returned rejected input."}], | ||
| }, | ||
| }, | ||
| ) | ||
| payload = payloads[0] | ||
| assert payload["commit_id"] == HEAD and payload["event"] == "APPROVE" | ||
| return scheduler.rest_review_node({"id": 17, "state": "APPROVED", "commit_id": HEAD, "user": {"login": "cwl-noema-review[bot]"}, "body": payload["body"]}) | ||
|
|
||
|
|
||
| def pull_request(reviews: list[dict], *, author="participant") -> dict: | ||
| """Build an authenticated API snapshot; independent gates retain their authority.""" | ||
| return {"author": {"login": author}, "headRefOid": HEAD, "baseRefOid": BASE, "reviewDecision": "APPROVED", "reviews": {"nodes": reviews}} | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("api_type,eligible", [("Bot", True), ("User", False)]) | ||
| def test_rest_conversion_preserves_authenticated_actor_type(published_review, api_type, eligible): | ||
| """Conversion must not turn an explicit REST User into an App by suffix.""" | ||
| converted = scheduler.rest_review_node({ | ||
| "id": 18, | ||
| "state": "APPROVED", | ||
| "commit_id": HEAD, | ||
| "user": {"login": "cwl-noema-review[bot]", "type": api_type}, | ||
| "body": published_review["body"], | ||
| }) | ||
| assert converted["author"]["__typename"] == api_type | ||
| assert scheduler.has_independent_current_head_approval(pull_request([converted])) is eligible | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("graphql_actor", [False, True]) | ||
| def test_actual_publisher_receipt_counts_as_independent_review(published_review, graphql_actor): | ||
| if graphql_actor: | ||
| published_review["author"] = {"login": "cwl-noema-review", "__typename": "Bot"} | ||
| pr = pull_request([published_review]) | ||
| assert scheduler.has_independent_current_head_approval(pr) | ||
| assert scheduler.merge_approval_block_reason(pr) is None | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("source", ["noema-review-github-app"]) | ||
| def test_authenticated_app_credential_sources(published_review, source): | ||
| published_review["body"] = published_review["body"].replace("noema-review-github-app-refresh", source) | ||
| assert scheduler.has_independent_current_head_approval(pull_request([published_review])) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("author", ["cwl-noema-review", "cwl-noema-review[bot]", "CWL-NOEMA-REVIEW"]) | ||
| def test_app_never_approves_itself(published_review, author): | ||
| assert not scheduler.has_independent_current_head_approval(pull_request([published_review], author=author)) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("actor", [ | ||
| {"login": "cwl-noema-review", "__typename": "User"}, | ||
| {"login": "cwl-noema-review"}, | ||
| {"login": "cwl-noema-review[bot]", "__typename": "User"}, | ||
| {"login": "another-bot[bot]", "__typename": "Bot"}, | ||
| {"login": "opencode-agent[bot]", "__typename": "Bot"}, | ||
| ]) | ||
| def test_only_canonical_authenticated_bot_qualifies(published_review, actor): | ||
| published_review["author"] = actor | ||
| assert not scheduler.has_independent_current_head_approval(pull_request([published_review])) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("commit", [None, {"oid": "b" * 40}, {"oid": "A" * 40}, {"oid": "short"}]) | ||
| def test_body_head_never_replaces_actual_exact_commit(published_review, commit): | ||
| published_review["commit"] = commit | ||
| assert not scheduler.has_independent_current_head_approval(pull_request([published_review])) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("old,new", [ | ||
| (f"- Head SHA: `{HEAD}`", f"- Head SHA: `{'b' * 40}`"), | ||
| ("decision=approve", "decision=comment"), | ||
| ("- Result: APPROVE", "- Result: COMMENT"), | ||
| ("- Actor: `cwl-noema-review[bot]`", "- Actor: `opencode-agent[bot]`"), | ||
| ("noema-review-github-app-refresh", "noema-review-pat"), | ||
| (publisher.NOEMA_REVIEW_FOOTER_MARKER, ""), | ||
| ]) | ||
| def test_incoherent_or_incomplete_receipt_fails_closed(published_review, old, new): | ||
| published_review["body"] = published_review["body"].replace(old, new) | ||
| assert not scheduler.has_independent_current_head_approval(pull_request([published_review])) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("state", ["CHANGES_REQUESTED", "DISMISSED", "APPROVED"]) | ||
| def test_latest_formal_policy_state_blocks_old_approval(published_review, state): | ||
| latest = deepcopy(published_review) | ||
| latest.update(state=state, body="Malformed receipt.", author={"login": "cwl-noema-review", "__typename": "Bot"}) | ||
| assert not scheduler.has_independent_current_head_approval(pull_request([published_review, latest])) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("head", ["", "short", "A" * 40]) | ||
| def test_noncanonical_snapshot_head_cannot_authorize_app_review(published_review, head): | ||
| pr = pull_request([published_review]) | ||
| pr["headRefOid"] = head | ||
| assert not scheduler.has_independent_current_head_approval(pr) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("mutation", ["missing_commit", "stale_commit", "wrong_api_type", "trailing_text", "duplicate_footer"]) | ||
| def test_latest_invalid_formal_receipt_never_falls_back(published_review, mutation): | ||
| latest = deepcopy(published_review) | ||
| if mutation == "missing_commit": | ||
| latest["commit"] = None | ||
| elif mutation == "stale_commit": | ||
| latest["commit"] = {"oid": "b" * 40} | ||
| elif mutation == "wrong_api_type": | ||
| latest["author"] = {"login": "cwl-noema-review", "__typename": "User"} | ||
| elif mutation == "trailing_text": | ||
| latest["body"] += "\nUntrusted continuation." | ||
| else: | ||
| latest["body"] += publisher.NOEMA_REVIEW_FOOTER_MARKER | ||
| assert not scheduler.has_independent_current_head_approval(pull_request([published_review, latest])) | ||
|
|
||
|
|
||
| def test_comment_does_not_revoke_formal_approval(published_review): | ||
| latest = deepcopy(published_review) | ||
| latest.update(state="COMMENTED", body="Additional observation.") | ||
| assert scheduler.has_independent_current_head_approval(pull_request([published_review, latest])) | ||
|
|
||
|
|
||
| def test_github_review_decision_remains_required(published_review): | ||
| pr = pull_request([published_review]) | ||
| pr["reviewDecision"] = "REVIEW_REQUIRED" | ||
| assert "repository approval policy is unsatisfied" in scheduler.merge_approval_block_reason(pr) | ||
|
|
||
|
|
||
| def test_human_approval_behavior_is_unchanged(): | ||
| review = {"state": "APPROVED", "author": {"login": "reviewer", "__typename": "User"}, "commit": {"oid": HEAD}} | ||
| assert scheduler.has_independent_current_head_approval(pull_request([review])) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
๐ Maintainability & Code Quality | ๐ก Minor | โก Quick win
๐ Supported by static analysis
๐ Script executed:
Repository: ContextualWisdomLab/.github
Length of output: 26998
Runner ๊ทธ๋ฃน๋ช ๊ณผ ๋ ์ด๋ธ์ ์ค์ ์ค์ ์ ๋ง์ถ์ธ์.
coalescing workflow๋
CWL CI isolated๊ทธ๋ฃน๊ณผcwlab-ci-isolated๋ ์ด๋ธ์ ์ฌ์ฉํฉ๋๋ค. ๋ฌธ์์ ๋ runner ๊ทธ๋ฃน ์ฐธ์กฐ๋ฅผ ๋ชจ๋ ๊ฐ์ ๊ฐ์ผ๋ก ์์ ํ์ธ์.์ ์ ์์
๐ค Prompt for AI Agents