Skip to content

Commit f310f03

Browse files
Seungpyo1007Seungpyo1007
committed
ci: rebuild ingest, refresh and promote as manual PR generators
Collection and verification stay, as reusable workflows that only TechAPI's manually dispatched collect.yml calls (ingest.yml, refresh.yml, verify-promote.yml, replacing weekly-ingest, weekly-refresh and verify-network). - The engine is the TechEngine commit TechAPI pins: TechAPI is checked out with submodules and its TechEngine installed. - Each task keeps one open PR on a fixed collect/* branch cut from develop, force-pushed and edited in place; no change means no PR. No comments. - data/ only; no dump. The gate (app.validate, integrity_check --strict) runs before the push, and its result and the Tier 0 report go in the PR body. - Plain git and gh instead of peter-evans/create-pull-request; GitHub-owned actions pinned by SHA. Timeouts, concurrency, --sleep and limits kept.
1 parent 7d89ee2 commit f310f03

7 files changed

Lines changed: 652 additions & 25 deletions

File tree

‎.github/workflows/ingest.yml‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
name: ingest
2+
3+
# Reusable, manual PR generator (ADR-017). Only TechAPI's collect.yml calls it,
4+
# from workflow_dispatch, so github.token is TechAPI's GITHUB_TOKEN and the
5+
# branch and PR come from github-actions[bot]. Nothing here runs on a schedule,
6+
# posts a comment, or uses a personal or bot-account token. Crawlers send the
7+
# honest TechEngine-* User-Agent and keep their --sleep and limits.
8+
#
9+
# Task: scrape upstream catalogs and draft missing SKUs into TechAPI data/.
10+
on:
11+
workflow_call:
12+
inputs:
13+
category:
14+
description: "Category to ingest (cpu, gpu or smartphone)"
15+
type: string
16+
default: cpu
17+
limit:
18+
description: "Max candidates per source"
19+
type: string
20+
default: "50"
21+
include_drafts:
22+
description: "Write incomplete records too (PR opened as draft)"
23+
type: boolean
24+
default: false
25+
26+
permissions:
27+
contents: write
28+
pull-requests: write
29+
30+
concurrency:
31+
group: collect-ingest-${{ inputs.category }}
32+
cancel-in-progress: false
33+
34+
jobs:
35+
ingest:
36+
runs-on: ubuntu-latest
37+
env:
38+
CATEGORY: ${{ inputs.category }}
39+
LIMIT: ${{ inputs.limit }}
40+
INCLUDE_DRAFTS: ${{ inputs.include_drafts }}
41+
DRAFT_FLAG: ${{ inputs.include_drafts && '--draft' || '' }}
42+
BRANCH: collect/ingest-${{ inputs.category }}
43+
TITLE: "feat(data/${{ inputs.category }}): ingest new SKUs"
44+
TECHAPI_DATA_DIR: ${{ github.workspace }}/TechAPI/data
45+
steps:
46+
# The TechEngine commit TechAPI pins as its submodule is the engine that
47+
# runs here, so collection, PR checks and the deploy share one engine.
48+
- name: Checkout TechAPI develop with its pinned TechEngine
49+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
50+
with:
51+
repository: GetTechAPI/TechAPI
52+
ref: develop
53+
path: TechAPI
54+
submodules: true
55+
fetch-depth: 0
56+
57+
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
58+
with:
59+
python-version: "3.12"
60+
61+
- name: Install the pinned TechEngine
62+
run: pip install -e ./TechAPI/TechEngine
63+
64+
- name: Run ingest
65+
working-directory: TechAPI/TechEngine
66+
run: |
67+
DRAFTS_FLAG=""
68+
if [ "$INCLUDE_DRAFTS" = "true" ]; then
69+
DRAFTS_FLAG="--include-drafts"
70+
fi
71+
python -m app.ingest \
72+
--category "$CATEGORY" \
73+
--limit "$LIMIT" \
74+
--data-root "$TECHAPI_DATA_DIR" \
75+
--summary "$GITHUB_WORKSPACE/ingest-summary.md" \
76+
$DRAFTS_FLAG
77+
78+
# Variant-safe benchmark backfill on existing CPU records (PassMark).
79+
# CPU-only; never overwrites, only fills nulls on exact heading matches.
80+
- name: Enrich benchmarks (PassMark, cpu only)
81+
if: env.CATEGORY == 'cpu'
82+
continue-on-error: true
83+
working-directory: TechAPI/TechEngine
84+
run: |
85+
python -m app.ingest.enrich \
86+
--data-root "$TECHAPI_DATA_DIR" \
87+
--limit "$LIMIT" \
88+
--min-year 2008 \
89+
--sleep 0.5 \
90+
--summary "$GITHUB_WORKSPACE/enrich-summary.md"
91+
92+
- name: Upload summaries
93+
if: always()
94+
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
95+
with:
96+
name: ingest-${{ inputs.category }}
97+
path: |
98+
ingest-summary.md
99+
enrich-summary.md
100+
if-no-files-found: ignore
101+
102+
# Commit locally first: the Tier 0 report scores the records this commit
103+
# changes against develop.
104+
- name: Commit data changes
105+
id: commit
106+
working-directory: TechAPI
107+
run: |
108+
set -euo pipefail
109+
git add data/
110+
if git diff --cached --quiet; then
111+
echo "No data changes; no PR."
112+
echo "changed=false" >> "$GITHUB_OUTPUT"
113+
exit 0
114+
fi
115+
echo "base=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
116+
git config user.name "github-actions[bot]"
117+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
118+
git switch -C "$BRANCH"
119+
git commit -q -m "$TITLE"
120+
echo "changed=true" >> "$GITHUB_OUTPUT"
121+
122+
# Gate: a PR opened with GITHUB_TOKEN triggers no checks, so the gate runs
123+
# here and its result goes in the PR body. Any failure stops the push.
124+
- name: Gate (app.validate + integrity_check --strict)
125+
if: steps.commit.outputs.changed == 'true'
126+
working-directory: TechAPI/TechEngine
127+
run: |
128+
set -euo pipefail
129+
python -m app.validate
130+
python integrity_check.py "$TECHAPI_DATA_DIR" --strict
131+
132+
- name: Tier 0 report for the changed records
133+
if: steps.commit.outputs.changed == 'true'
134+
working-directory: TechAPI/TechEngine
135+
env:
136+
BASE: ${{ steps.commit.outputs.base }}
137+
run: |
138+
python -m app.verify score --base "$BASE" --changed --no-cache --format md \
139+
> "$GITHUB_WORKSPACE/verify-report.md" 2>&1 \
140+
|| echo "_Tier 0 report unavailable._" > "$GITHUB_WORKSPACE/verify-report.md"
141+
142+
- name: Build PR body
143+
if: steps.commit.outputs.changed == 'true'
144+
run: |
145+
{
146+
echo "Closes #295"
147+
echo
148+
echo "Generated by TechEngine \`$GITHUB_WORKFLOW\` (run ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID})."
149+
echo "Add the \`final-check\` label for the final pass."
150+
echo
151+
echo "## Gate"
152+
echo "- \`app.validate\`: **passed**"
153+
echo "- \`integrity_check.py --strict\`: **passed**"
154+
echo
155+
echo "## Tier 0 verification (changed records)"
156+
echo
157+
cat verify-report.md
158+
echo
159+
echo "## Ingest summary"
160+
echo
161+
cat ingest-summary.md 2>/dev/null || true
162+
if [ -f enrich-summary.md ]; then
163+
echo
164+
echo "## Enrich summary"
165+
echo
166+
cat enrich-summary.md
167+
fi
168+
} > pr-body.md
169+
170+
# One open PR per task: the fixed branch is cut from develop and
171+
# force-pushed each run; an open PR is edited in place. No comments.
172+
- name: Push and open or update the PR
173+
if: steps.commit.outputs.changed == 'true'
174+
working-directory: TechAPI
175+
env:
176+
GH_TOKEN: ${{ github.token }}
177+
run: |
178+
set -euo pipefail
179+
git push --force origin "HEAD:refs/heads/$BRANCH"
180+
number=$(gh pr list --repo GetTechAPI/TechAPI --head "$BRANCH" --base develop \
181+
--state open --json number --jq '.[0].number // empty')
182+
if [ -n "$number" ]; then
183+
gh pr edit "$number" --repo GetTechAPI/TechAPI --title "$TITLE" --body-file ../pr-body.md
184+
else
185+
gh pr create --repo GetTechAPI/TechAPI --base develop --head "$BRANCH" \
186+
--title "$TITLE" --body-file ../pr-body.md $DRAFT_FLAG \
187+
--assignee Seungpyo1007 --assignee TechEngineBot \
188+
--label data --label enhancement \
189+
--milestone "Massive dataset rebuild (1989-2026)"
190+
fi

0 commit comments

Comments
 (0)