diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c7235dd..8ab75f3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -8,9 +8,10 @@ jobs: check: runs-on: ubuntu-latest # a real Postgres for the library server's highlights tests (the production driver, postgres.js, passes parameters differently from PGlite) + # and for its runtime-login test (an owner migrates, a login with row rights only serves; logins are real only on a real server) services: postgres: - image: postgres:16 + image: postgres:18 env: POSTGRES_PASSWORD: postgres POSTGRES_DB: hltest @@ -31,6 +32,11 @@ jobs: run: pnpm --filter @chalk-format/library-server exec vitest run src/highlights.test.ts env: TEST_POSTGRES_URL: postgres://postgres:postgres@127.0.0.1:5432/hltest + - name: The library server on a real Postgres with an owner and a DML-only runtime login (MIGRATE_ON_START=false) + run: pnpm --filter @chalk-format/library-server exec vitest run src/migrate.test.ts + env: + # a superuser: the test makes its own database and logins, and drops them after + CHALK_LIBRARY_TEST_DB: postgres://postgres:postgres@127.0.0.1:5432/postgres - name: JSON Schema is up to date with the model run: pnpm gen:schema && git diff --exit-code schema/ - name: Shared lab library builds reproduce their keys (libraries/catalog.json) diff --git a/docs/guides/run-a-library-server.md b/docs/guides/run-a-library-server.md index 1d415c4..748aac3 100644 --- a/docs/guides/run-a-library-server.md +++ b/docs/guides/run-a-library-server.md @@ -50,6 +50,7 @@ starting point. Never commit the real `.env`. |---|---|---| | `DATABASE_URL` | `pglite:` (in memory) | `postgres://user:pass@host:5432/db?sslmode=require`, or `pglite:./dir` to keep PGlite's data in a folder. The Docker image needs Postgres (see below). | | `DB_SCHEMA` | `chalk_library` | Postgres only. The library keeps everything in this schema (tables, its migration record, `pg_trgm`), so it can share a database with an app. | +| `MIGRATE_ON_START` | `true` | `false`: the server creates nothing at start (no schema, extension or migration) and refuses to start unless the schema is at its migration step; `chalk-library migrate`, run with the owner login, brings it there. Lets the server run as a login with only row rights (see [Postgres](#postgres)). | | `STORAGE` | `disk` | `s3` or `disk`. | | `S3_ENDPOINT` | | e.g. `https://.r2.cloudflarestorage.com`, `http://minio:9000`, `https://s3.eu-central-1.amazonaws.com`. | | `S3_BUCKET` | | The bucket. | @@ -62,6 +63,7 @@ starting point. Never commit the real `.env`. | `PUBLIC_URL` | `/files/` | Where readers fetch the bytes: the bucket's public domain or CDN. With disk storage the server serves them itself at `/files/`. | | `SERVER_URL` | `http://localhost:` | This server's own public address, e.g. `https://library.example.org`. Disk storage puts it in upload links, and it is the `iss` of every access token, so set it in production and do not change it casually (tokens issued under the old value stop working). With `PUBLIC_READ=false` the store at `GET /v1/stores` names it; not set, the address the request reached is used (with `X-Forwarded-Proto`), and the answer is not shared-cacheable. | | `TOKENS` | none | Bootstrap tokens: `name:token,name:token`. A plain name is an author (reads and writes), a name ending in `!` an administrator (`admin!:…`). Compared in constant time. See [Users, sign-in and API keys](#users-sign-in-and-api-keys). | +| `HUMAN_ACCOUNTS` | `on` | `off`: the library has no user accounts. No sign-in or refresh (`400 unsupported_grant_type`), the user endpoints answer `404`, tokens users already have are refused; API keys (acting for people included) and `TOKENS` work. For a deployment whose people sign in to another app that acts for them ([below](#acting-for-a-person-chalk-actor)). | | `PUBLIC_READ` | `true` | `false`: searching, reading records **and the files' bytes** need a token too (any signed-in user or key; see [A private library](#a-private-library-public_readfalse)). Writing always needs one. | | `ACCESS_TOKEN_TTL` | `900` | Seconds an access token lasts. | | `REFRESH_TOKEN_TTL` | `2592000` (30 days) | Seconds a refresh token lasts (each use gives a new one). | @@ -87,10 +89,35 @@ exist). On first start the server: 1. creates `DB_SCHEMA` if it is missing; 2. points every connection's `search_path` at it (then `public`); -3. applies [`spec/1.2/registry.sql`](../../spec/1.2/registry.sql) once, recording it in - `chalk_library_schema`. +3. applies [`spec/1.2/registry.sql`](../../spec/1.2/registry.sql) once, then each later migration step, recording + them in `chalk_library_schema`. + +Nothing outside that schema is touched. + +**An owner login for migrations, a runtime login for the server.** With `MIGRATE_ON_START=false` the server +changes nothing in the database and checks at start that the schema has every migration step it knows; if not, it +stops with a message to run `chalk-library migrate`. Migrations are then a release step, run with a login that owns +the schema (`DATABASE_URL=postgres://px_library_owner:…@…/db chalk-library migrate`, in the image `node server.mjs +migrate`), before the new version starts. The server's own login needs only this (as a superuser or the database +owner, once): + +```sql +CREATE ROLE px_library_owner LOGIN PASSWORD '…'; +CREATE ROLE px_library_app LOGIN PASSWORD '…'; +CREATE SCHEMA chalk_library AUTHORIZATION px_library_owner; +CREATE EXTENSION IF NOT EXISTS pg_trgm SCHEMA chalk_library; -- the setup makes it: the owner may not create extensions +GRANT USAGE ON SCHEMA chalk_library TO px_library_app; +ALTER DEFAULT PRIVILEGES FOR ROLE px_library_owner IN SCHEMA chalk_library GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO px_library_app; +ALTER DEFAULT PRIVILEGES FOR ROLE px_library_owner IN SCHEMA chalk_library GRANT USAGE, SELECT ON SEQUENCES TO px_library_app; +-- tables the owner made before these defaults existed: +GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA chalk_library TO px_library_app; +GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA chalk_library TO px_library_app; +``` -Nothing outside that schema is touched. Search uses trigram similarity on titles, file names and tags, which works +`pg_trgm` can be installed once per database, in one schema: put it in the library's schema (the server finds it +through `search_path`). If another app of the same database already installed it in `public`, that works too. +The `user`, `key` and `signing-key` commands follow `MIGRATE_ON_START` like the server, so they run with either +login. Search uses trigram similarity on titles, file names and tags, which works in any script, including Bengali (no Bengali stemmer exists). ## S3-compatible storage @@ -103,7 +130,10 @@ only then makes it available. What the bucket needs: 1. **Public read at `PUBLIC_URL`.** - - **R2:** a custom domain on the bucket. Cloudflare's cache then serves it. + - **R2:** a custom domain on the bucket, plus a Cache Rule for its `c1/` paths (R2 sends no `Cache-Control`, so + Cloudflare does not keep the files without one): the exact settings are in the + [package README](../../packages/library-server/README.md#files-from-the-edge-an-r2-custom-domain-behind-cloudflares-cache). + A public library's `GET /v1/files/{key}` then redirects there too, instead of pre-signing the bucket. - **AWS:** a bucket policy or CloudFront. - **MinIO:** an anonymous download policy. - Set `PUBLIC_URL` to that address, with a trailing `/`. @@ -239,17 +269,19 @@ needs `GetObject` on the bucket, which it already has (it reads uploads back to ```bash chalk-library serve # the API (the default) +chalk-library migrate # create the schema and apply the migrations, then exit (the release step with MIGRATE_ON_START=false) chalk-library scan [prefix] # adopt media already in the storage; again adds only what is new chalk-library check --token T [--admin-token A] # conformance: does a server do what the registry API says? chalk-library check --user U [--admin-token A] # the same, signed in (password: CHALK_LIBRARY_PASSWORD, a prompt or stdin) chalk-library user add --roles reader # users (password: a prompt without echo, or the first line of stdin) chalk-library user list | disable | enable | password -chalk-library key add --roles author # API keys for machines; the key is printed once +chalk-library key add --roles author [--act-for] # API keys for machines; the key is printed once; --act-for: it may send Chalk-Actor +chalk-library key act-for on|off # allow or stop a key acting for people chalk-library key list | revoke chalk-library signing-key rotate # a new token signing key ``` -`user`, `key` and `signing-key` need only `DATABASE_URL` (and `DB_SCHEMA`), not the storage settings. +`migrate`, `user`, `key` and `signing-key` need only `DATABASE_URL` (and `DB_SCHEMA`), not the storage settings. From the repository use `npx tsx packages/library-server/src/cli.ts `. From a build use `node dist/server.mjs `. @@ -331,6 +363,50 @@ seconds (30) of its use, the same token gives the same new refresh token again, used, so a client whose answer was lost stays signed in. `POST /v1/auth/revoke {"refresh_token"}` signs out. `libraryAuth` in `@chalk-format/store` does all of this for a browser or an app (`signIn`, `token()`, `signOut`). +### Acting for a person (Chalk-Actor) + +An app whose people sign in to the app itself (a staff dashboard) can call the library with one API key and say on +each request who it acts for, so the library's records name the person ([registry API +§10.7](../../spec/1.2/registry-api.md#107-acting-for-a-person)): + +```bash +chalk-library key add dashboard --roles admin --act-for # or POST /v1/admin/keys {"name","roles","actFor":true}; PATCH /v1/admin/keys/{id} {"actFor"} later +``` + +The app's server sends `Chalk-Actor: base64url(JSON.stringify({ id: 'staff:', name: 'Rita Akter' }))` with +the key. The library then records the `id` wherever it records who did something (`uploaded_by`; the highlights' +`reviewed_by` and `rects_by`, in place of the body's `by`; the key that made a key), `?mine=1` means that person, +and the table `actors` keeps each id's latest name once a change made for them succeeded (`GET /v1/admin/actors`; +reads and refused requests record nobody). `GET /v1/auth/me` answers `actingFor`. The header from any other caller +is refused (`403 act_for_not_allowed`), a malformed one is `400`. Without the header the key is recorded under its +own name. + +`staff:` and `system:` ids are reserved for acting: a highlight write whose `by` starts with one, from a caller that +is not acting for someone, is refused (`400 actor_requires_act_for`). **So every writer that names a person or a job +needs an act-for key and sends `Chalk-Actor`**, a sync job included (`{"id":"system:library-sync","name":"Library +sync"}`), instead of putting that id in `by`. `POST …/import` is the exception: it restores links with their +original `reviewedBy`, and accepts `staff:`/`system:` reviewers only from an act-for key or an administrator. + +Such a key can write as anyone it names, so it stays on the app's server and never goes to a browser (the library +does not allow `Chalk-Actor` in CORS). With `HUMAN_ACCOUNTS=off` the library has no accounts of its own at all: +only such keys, machine keys and `TOKENS`. + +#### A deployment whose people sign in elsewhere (keys and settings) + +The settings: `HUMAN_ACCOUNTS=off`, `MIGRATE_ON_START=false` (the server's `DATABASE_URL` is the runtime login), and +`chalk-library migrate` with the owner login as the release step ([Postgres](#postgres)). The keys, one per writer, +each in that writer's secret store: + +| Key | Command | Sends `Chalk-Actor` | Recorded as | +|---|---|---|---| +| The staff dashboard's server | `key add dashboard --roles author --act-for` (`--roles admin` only if it withdraws assets or lists actors and keys) | `staff:` of the signed-in person, `system:` for its own jobs (`system:publish`) | that person or job | +| The library sync (media and highlights published from a computer or a job) | `key add library-sync --roles author --act-for` | `system:library-sync` | `system:library-sync` | +| CI that publishes books or media | `key add github-publish --roles author` | no | the key's name, `github-publish` | +| A server that only reads (`PUBLIC_READ=false`) | `key add web-reader --roles reader` | no | (reads record nothing) | + +A key made without `--act-for` can get it later (`key act-for on`, or `PATCH /v1/admin/keys/{id} +{"actFor":true}`). Keep one bootstrap administrator in `TOKENS` only as long as you need it to make these keys. + ### Making a library private (PUBLIC_READ=false) Turning reading off breaks every reader that has no credential, so do it in this order: diff --git a/packages/library-server/.env.example b/packages/library-server/.env.example index d7733af..76ac5b8 100644 --- a/packages/library-server/.env.example +++ b/packages/library-server/.env.example @@ -1,6 +1,9 @@ # A Chalk library server. Copy to .env (never commit .env). See src/config.ts for every setting. DATABASE_URL=postgres://user:password@host:5432/database?sslmode=require DB_SCHEMA=chalk_library +# false: the server changes nothing in the database at start and refuses a schema that is not current; run +# `chalk-library migrate` with the schema's owner login as a release step (the server's own login then needs row rights only) +MIGRATE_ON_START=true STORAGE=s3 # Cloudflare R2 (any S3-compatible service works: MinIO, AWS S3, Backblaze B2, …) S3_ENDPOINT=https://.r2.cloudflarestorage.com @@ -15,6 +18,8 @@ SERVER_URL=https://library.example.org # bootstrap tokens: name:token, comma separated; a plain name is an author, a name ending in ! an administrator. # Users and API keys: chalk-library user add … / key add … (see docs/guides/run-a-library-server.md) TOKENS=teacher:change-me,admin!:change-me-too +# off: no user accounts (no sign-in, refresh or user admin); API keys (also ones acting for people: key add --act-for) and TOKENS only +HUMAN_ACCOUNTS=on # encrypts the token signing keys in the database; required once there are users or PUBLIC_READ=false. # openssl rand -base64 32 (the CLI needs the same value) AUTH_KEY_SECRET= diff --git a/packages/library-server/README.md b/packages/library-server/README.md index 75eef8d..1bb25d6 100644 --- a/packages/library-server/README.md +++ b/packages/library-server/README.md @@ -19,14 +19,78 @@ Users and API keys (the library issues its own tokens; spec 1.2 registry API §1 ```bash DATABASE_URL=pglite:./.data/db npx tsx src/cli.ts user add alice --roles admin # asks for a password DATABASE_URL=pglite:./.data/db npx tsx src/cli.ts key add ci --roles author # prints the key once +DATABASE_URL=pglite:./.data/db npx tsx src/cli.ts key add dashboard --roles admin --act-for # may send Chalk-Actor (acts for its people) +DATABASE_URL=postgres://owner… npx tsx src/cli.ts migrate # the release step when MIGRATE_ON_START=false ``` +An app's server can act for its signed-in people with a key made `--act-for`: it sends `Chalk-Actor` (base64url +JSON `{"id": "staff:", "name"}`), and the library records the person's id (registry API §10.7). `staff:` and +`system:` ids are reserved for this: every writer that names a person or a job (a staff dashboard, a sync job sending +`system:library-sync`) needs its own `--act-for` key; a plain key sending such a `by` gets `400 actor_requires_act_for`. +`HUMAN_ACCOUNTS=off` runs the library with keys and `TOKENS` only (§10.8). The keys and settings of such a +deployment, one by one: [run a library server](../../docs/guides/run-a-library-server.md#a-deployment-whose-people-sign-in-elsewhere-keys-and-settings). + (PGlite allows one process at a time: stop `serve` first, or use the admin API.) Docker, from the repository root: `docker build -f packages/library-server/Dockerfile -t chalk-library-server .` It also keeps the **highlights** of books (registry API §11): the lines of a book's text and the questions linked to them, as records an editor and a matcher change one by one (`v1/books//lines`, `…/links`), so reading apps get them from the library. They are migration step 5 of its database (two tables, `book_lines` and `book_links`), applied -when the server starts. +when the server starts (or by `chalk-library migrate` with `MIGRATE_ON_START=false`). + +## Files from the edge: an R2 custom domain behind Cloudflare's cache + +A public library (`PUBLIC_READ` unset or `true`) names its bucket's public address in `GET /v1/stores` +(`PUBLIC_URL` + `c1/{h0}/{h1}/{h}`), and readers fetch files there. On S3 storage, `GET /v1/files/{key}` (the address +apps use for a single file: an `` poster, a lab's code) answers `302` to the same file under `PUBLIC_URL`, +kept `public, max-age=86400` (a key's bytes never change; a file adopted in place, which may move to another +copy, `public, max-age=300`); `410` for a withdrawn key, `404` for an unknown one. It pre-signs the bucket instead +(5 minutes, `private, max-age=240`, as a private library always does) for an SVG, which a public address would +send inline so that its script runs on that domain (pre-signed, it is an attachment), and when `PUBLIC_URL` is not +set or is a server's own `/files/` (only a disk library serves that path). A private library +(`PUBLIC_READ=false`) never sends anyone to `PUBLIC_URL`. Those settings for the reference deployment: `STORAGE=s3`, +`PUBLIC_URL=https://assets.pathshalax.com/` (trailing slash), `SERVER_URL=https://library.pathshalax.com`. + +For the bytes to come from Cloudflare's edge rather than from the bucket each time, on Cloudflare (R2 objects carry +no `Cache-Control`, so without a rule a custom domain answers `cf-cache-status: DYNAMIC`): + +1. **Public access through the custom domain only.** R2 → the bucket → Settings → *Public access* → *Custom + domains* → connect `assets.example.org` (a domain in the same Cloudflare account; it is proxied by itself). + *R2.dev subdomain*: *Disallowed* (it is rate-limited and never cached). +2. **Cache Rule** (the zone of that domain → Caching → Cache Rules → Create rule → *Edit expression*): + ``` + (http.host eq "assets.example.org" and starts_with(http.request.uri.path, "/c1/")) + ``` + *Eligible for cache*; Edge TTL *Ignore cache-control header and use this TTL* → 1 year, with a status code TTL + `404` → *No cache* (a key asked for before its upload finished, or adopted elsewhere, must not stay missing; + without status code TTLs on the plan, use 1 day); Browser TTL *Override origin* → 1 day (a browser cannot be + purged: this is how long it may keep a file that was withdrawn); Cache key + *Ignore query string*. Only `c1/` is matched: files adopted in place (`chalk-books/…`) can be replaced by the + tool that wrote them, so they are left to Cloudflare's defaults. +3. **CORS** (the bucket → Settings → CORS): readers fetch from the browser, sometimes with `Range`. + ```json + [{ "AllowedOrigins": ["*"], "AllowedMethods": ["GET", "HEAD"], "AllowedHeaders": ["Range"], + "ExposeHeaders": ["Content-Length", "Content-Range", "ETag"], "MaxAgeSeconds": 86400 }] + ``` + Keep `*` rather than a list of origins: with a list the bucket answers per origin (`Vary: Origin`), which + Cloudflare's cache ignores, so one site's answer would be served to another. Authors' uploads go to the bucket's + S3 endpoint (pre-signed `PUT`), not through this domain; their CORS rule is the one in the guide below. +4. **Withdrawing** a key deletes its object, but the edge keeps a copy for its 1-year TTL: purge it too (Caching → + Configuration → *Custom purge* → URL `https://assets.example.org/c1//

/`). Browsers are not reached + by a purge: one that fetched the file may show it for up to a day more (Browser TTL), and one that followed + `/v1/files` may follow the kept redirect for up to a day (5 minutes for a file adopted in place), after which it + gets the `410`. +5. **Optional, defence in depth** (Rules → Transform Rules → *Modify Response Header*, when + `http.host eq "assets.example.org"`): set `Content-Security-Policy: sandbox` and + `X-Content-Type-Options: nosniff`, so that nothing the bucket holds can ever run as a page on that domain + (``, `