Skip to content

Commit ca165c8

Browse files
Seungpyo1007Seungpyo1007
committed
docs(readme): describe the health rules, the scaffolded files and the org's repositories
The README now lists what the health check actually inspects, the files the scaffolder writes and the steps for creating a satellite, with a short table of the organisation's repositories.
1 parent 6be9568 commit ca165c8

1 file changed

Lines changed: 114 additions & 44 deletions

File tree

‎README.md‎

Lines changed: 114 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -2,52 +2,75 @@
22

33
Repository management for the GetTechAPI organisation. Where
44
[TechEngine](https://github.com/GetTechAPI/TechEngine) processes the data,
5-
TechMachine looks after the repositories that hold it.
5+
TechMachine looks after the repositories that hold it. It does two things:
6+
a health check over every repository, and a scaffolder for new satellite
7+
data repositories.
68

7-
## Health check
8-
9-
`machine/health.py` runs when a person starts it — the `health` workflow's
10-
*Run workflow* button, or locally. For every repository in [`machine/repos.json`](machine/repos.json)
11-
it reads the newest completed run of each active workflow on each tracked
12-
branch, and it fetches the public endpoints the TechAPI homepage depends on.
9+
## The organisation's repositories
1310

14-
Anything other than `success`, `skipped` or `neutral` is reported —
15-
`cancelled` and `timed_out` included. That is deliberate: a job killed at
16-
GitHub's six-hour limit reports itself as *cancelled*, and several automations
17-
here failed that way for days without anyone noticing.
11+
| Repository | What it is |
12+
| --- | --- |
13+
| [TechAPI](https://github.com/GetTechAPI/TechAPI) | The dataset and the site |
14+
| [TechEngine](https://github.com/GetTechAPI/TechEngine) | The engine that processes the data; TechAPI pins it as a submodule |
15+
| [game-catalog](https://github.com/GetTechAPI/game-catalog) | Satellite data repository: games |
16+
| [cpu-engineering-samples](https://github.com/GetTechAPI/cpu-engineering-samples) | Satellite data repository: CPU engineering samples |
17+
| [.github](https://github.com/GetTechAPI/.github) | The organisation profile and shared community files |
18+
| TechMachine | This repository: health check and satellite scaffolder |
1819

19-
It queries each workflow separately rather than scanning "the last N runs",
20-
because weekly jobs scroll out of any fixed window behind the daily ones — and
21-
the weekly jobs are the ones that go quiet.
20+
## Health check
2221

23-
It also reads each active workflow file and reports any that uses a secret
24-
other than `GITHUB_TOKEN`. Automation here must act as `github-actions[bot]`,
25-
never as a person: workflows running under a personal token are how a
26-
maintainer's account got suspended. Disabling the workflow in the Actions
27-
settings clears the finding; a non-token secret a repository genuinely needs
28-
goes in that repository's `allowed_secrets` in `repos.json` — TechAPI's
29-
label-triggered final check is the one place TechEngineBot's token is allowed.
22+
`machine/health.py` runs when a person starts it: the `health` workflow's
23+
*Run workflow* button, or locally. It reads [`machine/repos.json`](machine/repos.json)
24+
and reports three kinds of problem.
25+
26+
1. **The latest completed run of each active workflow, per tracked branch.**
27+
Anything other than `success`, `skipped` or `neutral` is reported,
28+
`cancelled` and `timed_out` included. That is deliberate: a job killed at
29+
GitHub's six-hour limit reports itself as *cancelled*, and several
30+
automations here failed that way for days without anyone noticing. Each
31+
workflow is queried separately rather than scanning "the last N runs",
32+
because weekly jobs scroll out of any fixed window behind the daily ones,
33+
and the weekly jobs are the ones that go quiet. Disabled workflows are
34+
skipped, and so is GitHub's own *Dependabot Updates* workflow in a
35+
repository that no longer has a `.github/dependabot.yml` (or `.yaml`).
36+
2. **Secrets.** It reads each active workflow file on each tracked branch and
37+
reports any that uses a secret other than `GITHUB_TOKEN` or the
38+
repository's `allowed_secrets`. Automation here must act as
39+
`github-actions[bot]`, never as a person: workflows running under a
40+
personal token are how a maintainer's account got suspended. The one
41+
allowed secret is `TECHENGINEBOT_TOKEN`, which the label-triggered final
42+
check uses to post its comment. Disabling a workflow in the Actions
43+
settings clears its finding.
44+
3. **Endpoints.** It fetches each public URL listed under `endpoints` and
45+
reports one that is unreachable, is not JSON, or lacks its `expect` field.
46+
These are the files the TechAPI homepage depends on.
47+
48+
A repository that cannot be read is reported too, rather than passed over.
3049

3150
It only reads. The report goes to stdout and the run's step summary; it
32-
writes nothing to GitHub, mentions nobody, and runs on no schedule — an
51+
writes nothing to GitHub, mentions nobody, and runs on no schedule. An
3352
account here was suspended over automation, and a report nobody asked for is
34-
how that starts.
53+
how that starts. The run itself always ends green: the report is the output.
3554

3655
```bash
3756
python -m machine.health # print the report (GITHUB_TOKEN raises the rate limit)
3857
python -m pytest -q
3958
```
4059

41-
Adding a repository or endpoint is one entry in `machine/repos.json`.
60+
Adding a repository or endpoint is one entry in `machine/repos.json`: a
61+
`name`, the `branches` to watch and, where the final check runs,
62+
`"allowed_secrets": ["TECHENGINEBOT_TOKEN"]`. A repository with no workflows
63+
(the `.github` profile repository) is listed with its branch only.
4264

4365
## Final check
4466

4567
Adding the `final-check` label to a pull request asks
46-
[TechEngineBot](https://github.com/TechEngineBot) for a last look, in place of
47-
Copilot. The check job runs the tests (including a generated satellite
48-
repository), validates `repos.json`, and checks syntax and that every action
49-
is GitHub-owned and pinned by SHA. TechEngineBot then leaves **one** comment
50-
with the verdict; adding the label again edits it.
68+
[TechEngineBot](https://github.com/TechEngineBot) for a last look. The check
69+
job runs the tests (including a generated satellite repository), validates
70+
`repos.json`, and checks syntax and that every action is GitHub-owned and
71+
pinned by full commit SHA, both in this repository's workflows and in the
72+
ones the template generates (`tests/test_pinned_actions.py`). TechEngineBot
73+
then leaves **one** comment with the verdict; adding the label again edits it.
5174

5275
The check job runs the PR's code with no secrets and a read-only token. Only
5376
the comment job sees `TECHENGINEBOT_TOKEN`, and it posts nothing unless that is
@@ -57,27 +80,74 @@ as TechAPI's final check.
5780

5881
## Satellite repositories
5982

60-
Categories that do not belong in TechAPI live in their own repository (games:
61-
[game-catalog](https://github.com/GetTechAPI/game-catalog)). The split
62-
criterion is identity, not size — software and websites are tech data and
63-
stay in TechAPI.
83+
Categories that do not belong in TechAPI live in their own repository
84+
([game-catalog](https://github.com/GetTechAPI/game-catalog),
85+
[cpu-engineering-samples](https://github.com/GetTechAPI/cpu-engineering-samples)).
86+
The split criterion is identity, not size: software and websites are tech
87+
data and stay in TechAPI.
6488

65-
`machine/new_satellite.py` writes a new one from the layout game-catalog
66-
proved out: a streaming validator, a site build that publishes
67-
`summary.json` + `history.json` (never a listing of every record), CI, and
68-
licences.
89+
`machine/new_satellite.py` writes a new one from
90+
[`machine/satellite_template/`](machine/satellite_template):
6991

7092
```bash
71-
python -m machine.new_satellite --repo game-catalog --category game --title "Game catalog" --plural games --date-field release_date --range rating:0:5 --range metacritic:0:100 --out ../game-catalog
93+
python -m machine.new_satellite --repo game-catalog --category game \
94+
--title "Game catalog" --plural games --date-field release_date \
95+
--range rating:0:5 --range metacritic:0:100 --out ../game-catalog
7296
```
7397

74-
It only writes files. Creating the repository changes the organisation, so
75-
that is left to a person; the remaining steps are printed at the end,
76-
including adding `main` to the Pages environment's deployment branches —
77-
without it every deploy fails and leaves no log.
98+
### What it generates
99+
100+
```
101+
.github/CODEOWNERS # * @GetTechAPI/maintainers
102+
.github/pull_request_template.md
103+
.github/workflows/validate-data.yml # validator + tests on PRs and pushes to develop/main
104+
.github/workflows/deploy-pages.yml # builds and publishes the site from main
105+
.github/workflows/final-check.yml # the label-triggered check and TechEngineBot's comment
106+
.gitignore
107+
DATA_LICENSE.md # CC BY-SA 4.0 for the records
108+
LICENSE # MIT for the code
109+
README.md
110+
app/__init__.py
111+
app/validate.py # required fields, slug, dates, ranges, duplicates
112+
pyproject.toml
113+
site/.nojekyll
114+
site/build.py # summary.json + history.json, never a listing of every record
115+
site/index.html
116+
tests/test_validate.py
117+
data/<category>/ # empty; records go in <bucket>/<slug>.json
118+
```
78119

79-
The tests generate a repository and run *its* test suite and validator, so a
80-
template change that breaks generated repos fails here.
120+
The workflows follow the organisation's rules: GitHub-owned actions pinned
121+
by full commit SHA, least-privilege `permissions:`, no schedules, nothing
122+
that opens pull requests or issues, and no secret except
123+
`TECHENGINEBOT_TOKEN` in the final check's comment job. There are no issue
124+
templates; those are shared from the `.github` repository.
125+
126+
The tests generate a repository, compare its tree with the list above, and
127+
run *its* test suite and validator, so a template change that breaks or
128+
reshapes generated repositories fails here.
129+
130+
### Creating a satellite
131+
132+
The scaffolder only writes files. Everything else changes the organisation,
133+
so it is left to a person; the same list is printed after generating.
134+
135+
1. Generate the files with the command above.
136+
2. Create the repository: `gh repo create GetTechAPI/<repo> --public`.
137+
3. Push the scaffold to `develop`, then `develop` to `main`.
138+
4. Turn Pages on with the workflow as its source:
139+
`gh api -X POST repos/GetTechAPI/<repo>/pages -f build_type=workflow`.
140+
5. Add `main` to the Pages environment's deployment branches:
141+
`gh api -X POST repos/GetTechAPI/<repo>/environments/github-pages/deployment-branch-policies -f name=main`.
142+
Without it every deploy fails and leaves no log.
143+
6. Add the repository to `machine/repos.json` here, with
144+
`"allowed_secrets": ["TECHENGINEBOT_TOKEN"]`, and its `summary.json` URL
145+
to `endpoints`.
146+
7. Add it to `SATELLITES` in TechAPI's `site/src/scripts/techapi.js` so the
147+
homepage counts it.
148+
8. For the final check, create the `final-check` label and set the
149+
`TECHENGINEBOT_TOKEN` secret to a fine-grained token of TechEngineBot with
150+
*Pull requests: read and write* on that repository only.
81151

82152
## Branching
83153

0 commit comments

Comments
 (0)