Skip to content

Commit 5d5ca74

Browse files
committed
ci(collect): cap and scrub PR bodies, scope the write token, add timeouts
- Cap the PR body at 60,000 characters (GitHub rejects >65,536) and keep the full summaries in the run's artifacts; upload the promote output too. - Scrub scraped summaries and the Tier 0 report before they reach a body written by github-actions[bot]: drop "@", and break "#123", "GH-123" and URLs with a zero-width space so nobody is mentioned or cross-referenced. - Check out with persist-credentials: false and hand the token to the push step only. - Timeouts: 90 minutes for ingest, 240 for promote. Shallow checkouts and ubuntu-24.04 runners. - Pass workflow inputs to scripts through env (verify-promote, validate-data). - Docs: TechEngineBot's final check is TechAPI's final-check label; only TechAPI needs Actions allowed to create PRs; the coverage roadmap line no longer mentions a sticky issue.
1 parent f310f03 commit 5d5ca74

6 files changed

Lines changed: 106 additions & 29 deletions

File tree

‎.github/workflows/ingest.yml‎

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@ concurrency:
3333

3434
jobs:
3535
ingest:
36-
runs-on: ubuntu-latest
36+
runs-on: ubuntu-24.04
37+
timeout-minutes: 90
3738
env:
3839
CATEGORY: ${{ inputs.category }}
3940
LIMIT: ${{ inputs.limit }}
@@ -52,7 +53,10 @@ jobs:
5253
ref: develop
5354
path: TechAPI
5455
submodules: true
55-
fetch-depth: 0
56+
# The Tier 0 diff compares the new commit with develop's tip only.
57+
fetch-depth: 1
58+
# The write token is handed to the push step alone.
59+
persist-credentials: false
5660

5761
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
5862
with:
@@ -139,9 +143,15 @@ jobs:
139143
> "$GITHUB_WORKSPACE/verify-report.md" 2>&1 \
140144
|| echo "_Tier 0 report unavailable._" > "$GITHUB_WORKSPACE/verify-report.md"
141145
146+
# github-actions[bot] writes the body, so scraped text must not mention
147+
# anyone or cross-reference an issue: drop "@", break "#123"/"GH-123" and
148+
# URLs with a zero-width space. Colours such as #3fb950 are untouched.
142149
- name: Build PR body
143150
if: steps.commit.outputs.changed == 'true'
144151
run: |
152+
zw=$(printf '\342\200\213')
153+
ref="s/(#|[Gg][Hh]-)([0-9]+)(\$|[^0-9A-Za-z_])/\1$zw\2\3/g"
154+
scrub() { tr -d '@' | sed -E -e "$ref" -e "$ref" -e "s#://#:$zw//#g" -e "s/www\./www$zw./g"; }
145155
{
146156
echo "Closes #295"
147157
echo
@@ -154,18 +164,28 @@ jobs:
154164
echo
155165
echo "## Tier 0 verification (changed records)"
156166
echo
157-
cat verify-report.md
167+
scrub < verify-report.md
158168
echo
159169
echo "## Ingest summary"
160170
echo
161-
cat ingest-summary.md 2>/dev/null || true
171+
if [ -f ingest-summary.md ]; then scrub < ingest-summary.md; fi
162172
if [ -f enrich-summary.md ]; then
163173
echo
164174
echo "## Enrich summary"
165175
echo
166-
cat enrich-summary.md
176+
scrub < enrich-summary.md
167177
fi
168-
} > pr-body.md
178+
} > pr-body.full.md
179+
# GitHub rejects a PR body over 65,536 characters.
180+
python - <<'PY'
181+
from pathlib import Path
182+
body = Path("pr-body.full.md").read_text(encoding="utf-8", errors="replace")
183+
limit = 60000
184+
if len(body) > limit:
185+
note = "\n\n_Truncated; the full summaries are in the run's artifact._\n"
186+
body = body[: limit - len(note)] + note
187+
Path("pr-body.md").write_text(body, encoding="utf-8")
188+
PY
169189
170190
# One open PR per task: the fixed branch is cut from develop and
171191
# force-pushed each run; an open PR is edited in place. No comments.
@@ -176,7 +196,9 @@ jobs:
176196
GH_TOKEN: ${{ github.token }}
177197
run: |
178198
set -euo pipefail
179-
git push --force origin "HEAD:refs/heads/$BRANCH"
199+
auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0)
200+
git -c "http.https://github.com/.extraheader=AUTHORIZATION: basic $auth" \
201+
push --force origin "HEAD:refs/heads/$BRANCH"
180202
number=$(gh pr list --repo GetTechAPI/TechAPI --head "$BRANCH" --base develop \
181203
--state open --json number --jq '.[0].number // empty')
182204
if [ -n "$number" ]; then

‎.github/workflows/refresh.yml‎

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ concurrency:
2626

2727
jobs:
2828
refresh:
29-
runs-on: ubuntu-latest
29+
runs-on: ubuntu-24.04
3030
# A job that runs into GitHub's 6h ceiling is reported as *cancelled*,
3131
# not failed, so it never surfaces as a broken run. Stop short of it.
3232
timeout-minutes: 330
@@ -46,7 +46,10 @@ jobs:
4646
ref: develop
4747
path: TechAPI
4848
submodules: true
49-
fetch-depth: 0
49+
# The Tier 0 diff compares the new commit with develop's tip only.
50+
fetch-depth: 1
51+
# The write token is handed to the push step alone.
52+
persist-credentials: false
5053

5154
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
5255
with:
@@ -157,9 +160,15 @@ jobs:
157160
> "$GITHUB_WORKSPACE/verify-report.md" 2>&1 \
158161
|| echo "_Tier 0 report unavailable._" > "$GITHUB_WORKSPACE/verify-report.md"
159162
163+
# github-actions[bot] writes the body, so scraped text must not mention
164+
# anyone or cross-reference an issue: drop "@", break "#123"/"GH-123" and
165+
# URLs with a zero-width space. Colours such as #3fb950 are untouched.
160166
- name: Build PR body
161167
if: steps.commit.outputs.changed == 'true'
162168
run: |
169+
zw=$(printf '\342\200\213')
170+
ref="s/(#|[Gg][Hh]-)([0-9]+)(\$|[^0-9A-Za-z_])/\1$zw\2\3/g"
171+
scrub() { tr -d '@' | sed -E -e "$ref" -e "$ref" -e "s#://#:$zw//#g" -e "s/www\./www$zw./g"; }
163172
{
164173
echo "Closes #296"
165174
echo
@@ -172,19 +181,29 @@ jobs:
172181
echo
173182
echo "## Tier 0 verification (changed records)"
174183
echo
175-
cat verify-report.md
184+
scrub < verify-report.md
176185
echo
177186
echo "## Enrichment summaries"
178187
for f in enrich-*.md; do
179188
[ -f "$f" ] || continue
180189
echo
181190
echo "<details><summary>$f</summary>"
182191
echo
183-
cat "$f"
192+
scrub < "$f"
184193
echo
185194
echo "</details>"
186195
done
187-
} > pr-body.md
196+
} > pr-body.full.md
197+
# GitHub rejects a PR body over 65,536 characters.
198+
python - <<'PY'
199+
from pathlib import Path
200+
body = Path("pr-body.full.md").read_text(encoding="utf-8", errors="replace")
201+
limit = 60000
202+
if len(body) > limit:
203+
note = "\n\n_Truncated; the full summaries are in the run's artifact._\n"
204+
body = body[: limit - len(note)] + note
205+
Path("pr-body.md").write_text(body, encoding="utf-8")
206+
PY
188207
189208
# One open PR per task: the fixed branch is cut from develop and
190209
# force-pushed each run; an open PR is edited in place. No comments.
@@ -195,7 +214,9 @@ jobs:
195214
GH_TOKEN: ${{ github.token }}
196215
run: |
197216
set -euo pipefail
198-
git push --force origin "HEAD:refs/heads/$BRANCH"
217+
auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0)
218+
git -c "http.https://github.com/.extraheader=AUTHORIZATION: basic $auth" \
219+
push --force origin "HEAD:refs/heads/$BRANCH"
199220
number=$(gh pr list --repo GetTechAPI/TechAPI --head "$BRANCH" --base develop \
200221
--state open --json number --jq '.[0].number // empty')
201222
if [ -n "$number" ]; then

‎.github/workflows/validate-data.yml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,13 @@ jobs:
4343
- name: Validate seed JSON (§9.3, §15.3)
4444
env:
4545
TECHAPI_DATA_DIR: ${{ github.workspace }}/TechAPI/data
46+
BASE_SHA: ${{ inputs.base-sha }}
4647
run: |
47-
if [ -n "${{ inputs.base-sha }}" ]; then
48-
git -C TechAPI fetch --no-tags --depth=100 origin "${{ inputs.base-sha }}"
49-
git -C TechAPI merge-base "${{ inputs.base-sha }}" HEAD >/dev/null 2>&1 \
48+
if [ -n "$BASE_SHA" ]; then
49+
git -C TechAPI fetch --no-tags --depth=100 origin "$BASE_SHA"
50+
git -C TechAPI merge-base "$BASE_SHA" HEAD >/dev/null 2>&1 \
5051
|| git -C TechAPI fetch --no-tags --unshallow origin
51-
python -m app.validate --changed-since "${{ inputs.base-sha }}"
52+
python -m app.validate --changed-since "$BASE_SHA"
5253
else
5354
python -m app.validate
5455
fi

‎.github/workflows/verify-promote.yml‎

Lines changed: 42 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,13 @@ concurrency:
4040

4141
jobs:
4242
promote:
43-
runs-on: ubuntu-latest
43+
runs-on: ubuntu-24.04
44+
timeout-minutes: 240
4445
env:
4546
PYTHONIOENCODING: utf-8
47+
MAX_URLS: ${{ inputs.max_urls }}
48+
MAX_WIKIDATA: ${{ inputs.max_wikidata }}
49+
MAX_CROSSREF: ${{ inputs.max_crossref }}
4650
DRAFT_FLAG: ""
4751
BRANCH: collect/promote
4852
TITLE: "data(verify): promote records to verified"
@@ -57,7 +61,10 @@ jobs:
5761
ref: develop
5862
path: TechAPI
5963
submodules: true
60-
fetch-depth: 0
64+
# The Tier 0 diff compares the new commit with develop's tip only.
65+
fetch-depth: 1
66+
# The write token is handed to the push step alone.
67+
persist-credentials: false
6168

6269
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
6370
with:
@@ -79,20 +86,28 @@ jobs:
7986

8087
- name: Tier 1 source-URL liveness
8188
working-directory: TechAPI/TechEngine
82-
run: python -m app.verify check-urls --max "${{ inputs.max_urls }}"
89+
run: python -m app.verify check-urls --max "$MAX_URLS"
8390

8491
- name: Batched Wikidata source liveness
8592
working-directory: TechAPI/TechEngine
86-
run: python -m app.verify check-wikidata --max "${{ inputs.max_wikidata }}"
93+
run: python -m app.verify check-wikidata --max "$MAX_WIKIDATA"
8794

8895
- name: Tier 2 external cross-reference
8996
working-directory: TechAPI/TechEngine
90-
run: python -m app.verify crossref --max "${{ inputs.max_crossref }}"
97+
run: python -m app.verify crossref --max "$MAX_CROSSREF"
9198

9299
- name: Tier 3 promote (dry-run)
93100
working-directory: TechAPI/TechEngine
94101
run: python -m app.verify promote | tee "$GITHUB_WORKSPACE/promote-dry-run.md" >> "$GITHUB_STEP_SUMMARY"
95102

103+
- name: Upload promotion output
104+
if: always()
105+
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
106+
with:
107+
name: promote-output
108+
path: promote-dry-run.md
109+
if-no-files-found: ignore
110+
96111
- name: Tier 3 promote (apply)
97112
if: inputs.apply
98113
working-directory: TechAPI/TechEngine
@@ -165,9 +180,15 @@ jobs:
165180
> "$GITHUB_WORKSPACE/verify-report.md" 2>&1 \
166181
|| echo "_Tier 0 report unavailable._" > "$GITHUB_WORKSPACE/verify-report.md"
167182
183+
# github-actions[bot] writes the body, so scraped text must not mention
184+
# anyone or cross-reference an issue: drop "@", break "#123"/"GH-123" and
185+
# URLs with a zero-width space. Colours such as #3fb950 are untouched.
168186
- name: Build PR body
169187
if: steps.commit.outputs.changed == 'true'
170188
run: |
189+
zw=$(printf '\342\200\213')
190+
ref="s/(#|[Gg][Hh]-)([0-9]+)(\$|[^0-9A-Za-z_])/\1$zw\2\3/g"
191+
scrub() { tr -d '@' | sed -E -e "$ref" -e "$ref" -e "s#://#:$zw//#g" -e "s/www\./www$zw./g"; }
171192
{
172193
echo "Closes #296"
173194
echo
@@ -180,14 +201,24 @@ jobs:
180201
echo
181202
echo "## Tier 0 verification (changed records)"
182203
echo
183-
cat verify-report.md
204+
scrub < verify-report.md
184205
echo
185206
echo "## Promotion run"
186207
echo
187208
echo '```'
188-
cat promote-dry-run.md
209+
scrub < promote-dry-run.md
189210
echo '```'
190-
} > pr-body.md
211+
} > pr-body.full.md
212+
# GitHub rejects a PR body over 65,536 characters.
213+
python - <<'PY'
214+
from pathlib import Path
215+
body = Path("pr-body.full.md").read_text(encoding="utf-8", errors="replace")
216+
limit = 60000
217+
if len(body) > limit:
218+
note = "\n\n_Truncated; the full summaries are in the run's artifact._\n"
219+
body = body[: limit - len(note)] + note
220+
Path("pr-body.md").write_text(body, encoding="utf-8")
221+
PY
191222
192223
# One open PR per task: the fixed branch is cut from develop and
193224
# force-pushed each run; an open PR is edited in place. No comments.
@@ -198,7 +229,9 @@ jobs:
198229
GH_TOKEN: ${{ github.token }}
199230
run: |
200231
set -euo pipefail
201-
git push --force origin "HEAD:refs/heads/$BRANCH"
232+
auth=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0)
233+
git -c "http.https://github.com/.extraheader=AUTHORIZATION: basic $auth" \
234+
push --force origin "HEAD:refs/heads/$BRANCH"
202235
number=$(gh pr list --repo GetTechAPI/TechAPI --head "$BRANCH" --base develop \
203236
--state open --json number --jq '.[0].number // empty')
204237
if [ -n "$number" ]; then

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ Spins up Postgres 16, seeds from the mounted TechAPI checkout, serves on `:8000`
9090

9191
- [x] Split out from TechAPI; sibling-checkout data pipeline
9292
- [x] **Coverage gap detector** - diff curated dataset vs upstream catalogs
93-
and surface missing SKUs as a sticky weekly issue
93+
and list missing SKUs in an on-demand report (`coverage-report.yml`)
9494
([#1](https://github.com/GetTechAPI/TechEngine/issues/1))
9595
- [x] **Ingestion crawler** - scrape canonical sources and draft new SKUs for
9696
a TechAPI PR ([#2](https://github.com/GetTechAPI/TechEngine/issues/2));

‎docs/SPEC.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2003,14 +2003,14 @@ TechAPI의 **핵심 목표 중 하나**. TechPicks 외에 다양한 앱·플랫
20032003
3. 레포 간 `repository_dispatch` 중계와 서브모듈 자동 bump(`notify-*`, `bump-*`)는 없앤다.
20042004
4. `verify-status`는 TechAPI `dump-develop`에 합치고, `dump-refresh`는 `dump-develop`의 수동 전체 재생성으로 대체한다. 커버리지 리포트는 주 1회, TechEngine 레포 이슈만 갱신한다.
20052005
5. 스크레이핑 워크플로(`weekly-data.yml`)는 수동 실행으로 검증한 뒤에 cron을 켠다.
2006-
6. **예외 — TechEngineBot 최종 확인**: TechEngineBot 계정은 중요한 PR의 최종 확인자 역할로만 쓴다. 쓰기 권한이 있는 메인테이너(OWNER/MEMBER/COLLABORATOR)가 `/final-review` 댓글을 달 때만 TechAPI `final-review.yml`이 돌고, PR 하나에 댓글 하나(재실행 시 같은 댓글 수정)만 남긴다. 조회는 `GITHUB_TOKEN`으로 하고, `TECHENGINEBOT_TOKEN`(TechAPI 레포 한정 fine-grained 토큰, Pull requests: write)은 그 댓글에만 쓴다. 스케줄·push·PR 이벤트로 TechEngineBot이 움직이는 워크플로는 만들지 않는다.
2006+
6. **예외 — TechEngineBot 최종 확인**: TechEngineBot 계정은 중요한 PR의 최종 확인자 역할로만 쓴다. 메인테이너가 PR에 `final-check` 라벨을 붙일 때만 TechAPI `final-check.yml`이 돌고, PR 하나에 댓글 하나(재실행 시 같은 댓글 수정)만 남긴다. 조회는 `GITHUB_TOKEN`으로 하고, `TECHENGINEBOT_TOKEN`(TechAPI 레포 한정 fine-grained 토큰, Pull requests: write)은 그 댓글에만 쓴다. 스케줄·push·PR 이벤트로 TechEngineBot이 움직이는 워크플로는 만들지 않는다.
20072007
- **개정 (2026-11-01)**: TechAPI는 PR 검증을 자체 검증기로 직접 돌리므로, 2·4·5번을 다음으로 대체한다.
20082008
- PR 댓글 워크플로(`techapi-pr-validation-comment`, `techapi-verify-comment`)와 `refresh-data`를 삭제한다. `coverage-report`는 수동 실행·읽기 전용으로 바꾼다.
20092009
- 수집·검증(`weekly-ingest`, `weekly-refresh`, `verify-network`)은 `ingest.yml`, `refresh.yml`, `verify-promote.yml`로 바꾼다. TechAPI `collect.yml`이 `workflow_dispatch`로만 호출하는 PR 생성기다. 작업마다 고정 브랜치(`collect/*`)에 열린 PR을 최대 1개만 두고 제자리에서 갱신한다. 댓글은 달지 않으며, 게이트(`app.validate`, `integrity_check.py --strict`) 결과와 Tier 0 보고서는 PR 본문에 남긴다. 덤프는 커밋하지 않는다. 실행되는 엔진은 TechAPI가 서브모듈로 고정한 TechEngine 커밋이다.
20102010
- 크롤러는 브라우저를 흉내 내지 않는 `TechEngine-*` User-Agent로 자신을 밝힌다. 안티봇 챌린지는 "알 수 없음"으로 두고 `verified` 승급 근거로 쓰지 않는다. 검색엔진 결과 스크레이핑(GSMArena 후보 탐색용 DuckDuckGo/Bing)은 없앤다.
20112011
- 모든 action은 GitHub 소유 action을 SHA로 고정해 쓴다. TechEngineBot 최종 확인(6번)은 TechAPI 쪽에서 관리한다.
20122012
- **트레이드오프**: `GITHUB_TOKEN`이 만든 PR은 다른 워크플로를 트리거하지 않으므로, 주간 PR은 생성 전에 같은 작업 안에서 검증을 끝낸다. 포크 PR에는 토큰이 읽기 전용이라 댓글 대신 Job Summary로 결과를 남긴다. Projects 보드 필드 자동 입력(`pr-metadata`)은 없애고 GitHub Projects 내장 Auto-add를 쓴다.
2013-
- **필요 설정**: 두 레포 Settings → Actions → General에서 "Allow GitHub Actions to create and approve pull requests"를 켜야 PR 생성이 된다.
2013+
- **필요 설정**: PR은 TechAPI에서 TechAPI 토큰으로 만들어지므로 TechAPI만 Settings → Actions → General의 "Allow GitHub Actions to create and approve pull requests"를 켠다. TechEngine은 끈 채로 둔다.
20142014

20152015
---
20162016

0 commit comments

Comments
 (0)