diff --git a/AGENTS.md b/AGENTS.md index 0dfce6a..38ccc5d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,7 +88,7 @@ Use `.env.example`, `tests/api/routes/test_auth.py`, and See `README.md` for current auth/sync routes, local sandbox entry points and test lanes. For client/server contracts, use the workspace's `papyrus-sync-contract` skill and -`../.agents/skills/papyrus-sync-contract/references/contract-map.md` when developing +`../.agents/skills/papyrus-sync-contract/references/CONTRACT_MAP.md` when developing inside the Papyrus workspace. To build the dev sandbox assets without the Vite dev server: diff --git a/README.md b/README.md index 94d48fe..eaded29 100644 --- a/README.md +++ b/README.md @@ -49,9 +49,10 @@ provider-backed tests require explicit SMTP/Google configuration. [Managed acquisition routes](papyrus/api/routes/acquisition.py) use owner-scoped endpoints, jobs and rules. Provider adapters, job lifecycle and release tokens live in [the acquisition service](papyrus/services/acquisition/). The monitor -imports selected media and handles retry/cancellation. [OPDS relay setup](docs/opds-relay.md) -describes catalog transport and origin restrictions. -See [in-app support setup](docs/support.md) for GitHub issue submission configuration. +imports selected media and handles retry/cancellation. See the +[deployment guide](deploy/README.md#optional-services) for OPDS relay and support +submission setup. Wire schemas are defined by the generated API reference; sync +validation lives beside its service code and regression tests. ## Verification diff --git a/deploy/README.md b/deploy/README.md index c74c4b0..83ccb20 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -142,13 +142,35 @@ sign-in, book/media upload and sync from a Play-installed Android build. Monitor PowerSync replication slots: a 1 GB WAL retention cap protects disk but an extended outage can invalidate a slot and require a controlled resync. -## Updating and recovery +## Optional services + +### OPDS relay + +`OPDS_RELAY_ENABLED` enables catalog transport, including anonymous browsing. +Restrict it with `OPDS_RELAY_ALLOWED_HOSTS`, a JSON array of exact hostnames; +include any required redirect and cover-image hosts. An empty array permits all +public hosts. Private/LAN destinations remain blocked even when listed. + +Use HTTPS and allow the web app's origin in `CORS_ORIGINS`. The selected backend +receives protected catalog credentials, so do not log relay request bodies or +credential-bearing URLs. The relay does not upload guest libraries. Rate and +concurrency limits are per API worker; apply aggregate limits before scaling out. -The initial testing baseline is `0.0.1`. Its release workflow removes the unused -`1.0.0` container by its exact manifest digest after publishing the replacement. -It preserves shared manifests and refuses to delete versions with unrelated tags. -The package itself stays public. This cleanup uses the publishing repository's -package admin access through `GITHUB_TOKEN` and only runs for `0.0.1`. +### Support submissions + +Set `SUPPORT_GITHUB_TOKEN` to a fine-grained PAT restricted to the target repository +with Issues read/write and organization Projects read/write permissions. Attachment +uploads also require the token owner to have repository write access; GitHub App +installation tokens cannot upload them. Keep this token on the server. An empty +token disables submissions. Routing settings are listed in `production.env.example`. + +Reports and attachments become public on GitHub. A timeout can leave a created +issue or uploaded attachment without a confirmed client response; check GitHub +before retrying. Project routing failures are logged with the issue number and +need manual reconciliation; they are not persisted for retry. A project's visible +`viewerCanUpdate` value alone does not prove the PAT has Projects write permission. + +## Updating and recovery Deploy the backward-compatible server first, then roll out the client. Update `PAPYRUS_VERSION`, check out the corresponding source/config and install the web @@ -178,8 +200,8 @@ docker compose --env-file production.env -f compose.yml logs --tail 100 api powe ``` Do not expose logs containing tokens or environment values when asking for help. -The first release still needs public DNS, SMTP/OAuth setup, credentials and a -real-device sync/auth test; local container compilation is not a deployment test. +Verify public DNS, SMTP/OAuth, credentials and real-device sync/auth after +deployment; local container compilation is not a deployment test. References: - [Hetzner Cloud firewalls](https://docs.hetzner.com/cloud/firewalls/overview/) diff --git a/docs-tracking.md b/docs-tracking.md deleted file mode 100644 index c499e13..0000000 --- a/docs-tracking.md +++ /dev/null @@ -1,52 +0,0 @@ -# Reading tracking contract and rollout - -Tracking schema version 2 is advertised additively by `GET /v1/sync/settings`. -Goals/session/statistics REST endpoints now return owned persisted records and -ledger-derived progress. Existing metric strings remain valid; `reading_days` is -additive. Goal definition, activity, and period payloads are validated in the same -transaction/owner lock used by ordinary PowerSync uploads. - -`reading_goals`, `reading_activities`, and `goal_periods` store owner-scoped JSONB -payloads with UUID identities. Activity is immutable: correction/undo appends a -reversal referencing an owned original. Retries are idempotent. Shelf membership -and book-title snapshots survive book deletion. Activity/period deletion through -sync is rejected; deleting a goal retains rule/period history. Account deletion -still removes owned data. - -Progress unions overlapping time intervals and per-book/day content coverage, -counts distinct confirmed books, and applies creation cutoffs, pauses, captured -IANA timezone, Monday weeks, and historical rule revisions. Period history holds -rules rather than mutable counters, allowing delayed corrections to reproject it. -Shared fixtures in `tests/fixtures/tracking_projections.json` match Dart tests. - -## Deployment order - -1. Apply Alembic revision `b5c6d7e8f901` (`uv run alembic upgrade head`). This is - additive and preserves existing library tables/data. -2. Add the three tracking tables to the PostgreSQL PowerSync publication and grants. - Updated `deploy/bootstrap-powersync.sql` and `scripts/setup_local_powersync.sh` - include them. Existing deployments must run the updated bootstrap; creating - database tables alone does not update an existing publication. -3. Deploy the updated PowerSync stream configuration and server capability. -4. Release the integrated client after verifying the capability and transport. - -Older servers are supported by client local staging; unsupported tracking never -blocks ordinary library uploads. Versions remain unchanged in these development -PRs. Run server tests only against a separate test database; the fixtures recreate -tables. The tracking migration test verifies existing books survive and Alembic -metadata matches the upgraded schema. - -## Selected books (schema v2) - -A goal with `scope: "book"` may include `book_ids`, a canonical list of selected -book UUIDs. `scope_id` remains a selected book for legacy payload compatibility. -Omitting `book_ids` retains the original single-book meaning. Every selected book -is ownership-checked; changing the selection requires a replacement goal. -Completion targets cannot exceed the number of distinct selected books. - -The client omits the additive field for single-book goals and stages multiple-book -goals and their period snapshots separately on v1 servers. Ordinary tracking and -library uploads continue. Staged records promote when v2 is discovered. Deploy -the v2 server before releasing this client; use the updated client on all devices -to calculate multiple-book progress. Existing JSONB and SQLite payload columns -carry the selection; no additional database migration or stream changes are needed. diff --git a/docs/opds-relay.md b/docs/opds-relay.md deleted file mode 100644 index 3ead0dd..0000000 --- a/docs/opds-relay.md +++ /dev/null @@ -1,48 +0,0 @@ -# OPDS relay - -`POST /v1/opds/relay` retrieves catalog feeds, search descriptions, covers, and book files. The route is anonymous and does not access the database or persistent file storage. The client imports the returned bytes into its selected local or signed-in library. Deploy the client and backend changes together. - -The default API prefix is `/v1`; use the configured `API_PREFIX` if different. A request contains: - -```json -{ - "url": "https://www.gutenberg.org/ebooks/1342.epub.noimages", - "catalog_url": "https://www.gutenberg.org/ebooks/1342.opds", - "max_bytes": 268435456 -} -``` - -Protected catalogs may include `credentials: {"username": "reader", "password": "..."}` in the JSON body. The relay sends HTTP Basic credentials only to the `catalog_url` origin. It never forwards incoming Authorization or Cookie headers, upstream cookies, or upstream error-page bodies. Request bodies and credential-bearing URLs must not be captured by proxy/access logging. - -Successful responses stream the bytes with the upstream Content-Type, an optional Content-Length, and `X-OPDS-URL` containing the final upstream URL. Redirects are followed on the server, with validation at each hop. The client uses that final URL to resolve relative catalog links. Responses use `Cache-Control: no-store`, attachment disposition, `nosniff`, and a sandbox CSP. Errors use the existing API error envelope with code `OPDS_RELAY_ERROR`; temporary capacity errors include `details.retryable: true`. - -## Configuration - -- `OPDS_RELAY_ENABLED`: defaults to `true`; set `false` to disable this feature. -- `OPDS_RELAY_ALLOWED_HOSTS`: defaults to `[]`, allowing public hosts. Set a JSON array of exact hostnames to restrict access, for example `["www.gutenberg.org"]`. Include image/CDN/redirect hosts required by the permitted catalogs. Private addresses remain blocked even for listed hosts. -- `RATE_LIMIT_GENERAL`: existing per-IP request limit, also applied to the relay. -- `CORS_ORIGINS`: include the web app origin. `X-OPDS-URL` is exposed to browser clients by the API middleware. The client uses `PAPYRUS_API_BASE_URL` or the selected custom backend independently of login. - -Use HTTPS for deployed app-to-backend connections and protected upstream catalogs. The backend is trusted with catalog credentials. Configuring this relay does not enable PowerSync, require guest registration, or upload guest books. - -## Bounds and operation - -Only HTTP/HTTPS URLs without embedded credentials are accepted. Every resolved address must be public; private, loopback, link-local, shared, multicast, and IPv4-mapped IPv6 destinations are rejected. Each connection is pinned to a validated IP while retaining the original HTTP Host and TLS hostname verification. HTTPS-to-HTTP redirects are rejected. The relay does not use environment HTTP proxies. - -Each worker allows eight concurrent resources and eight DNS operations. The client shares a four-request queue across browsing, covers, and downloads, and retries temporary relay-capacity errors at most twice. The existing in-memory rate limiter and concurrency limits are per worker; deployments with multiple replicas should apply aggregate quotas at their reverse proxy if needed. - -Resources default to 8 MiB and cannot exceed 256 MiB. The service enforces both declared and actual byte counts, permits at most five redirects, bounds DNS and response opening to 30 seconds each, and interrupts stalled reads after 30 seconds. A five-minute deadline closes each resource even if a downstream client stops consuming it. OS DNS calls cannot be cancelled, so timed-out lookups continue in a bounded executor without admitting unlimited resolver work. - -Connections close on completion, disconnect, and failure. A limit or network error after response headers have been sent aborts the stream; the client treats that as an interrupted download and does not commit an incomplete import. Compression is requested as `identity`; unexpected upstream content encodings are rejected. The relay retrieves complete resources, without range/resume support or shared caching. - -Private LAN catalogs are deliberately unavailable through this anonymous relay. Authentication, licensing, and DRM requirements of upstream catalogs still apply. - -## Verification - -```sh -uv run pytest tests/services/test_opds.py tests/api/routes/test_opds.py -uv run ruff check papyrus/services/opds.py papyrus/api/routes/opds.py papyrus/schemas/opds.py -uv run mypy papyrus/services/opds.py papyrus/api/routes/opds.py papyrus/schemas/opds.py -``` - -Coverage includes destination validation, pinned TLS identity, scoped credentials, redirect limits, known/unknown size limits, truncated streams, slow DNS/headers/chunk framing, idle stream expiry, anonymous rate limiting, CORS metadata, and disconnect cleanup. The route tests do not need a running database. diff --git a/docs/reading-timer-sync.md b/docs/reading-timer-sync.md deleted file mode 100644 index fc6aaab..0000000 --- a/docs/reading-timer-sync.md +++ /dev/null @@ -1,65 +0,0 @@ -# Reading timer synchronization - -Signed-in devices share one active reading timer per account through the -`reading_timers` PowerSync stream. Guest timers remain local. The selected book, -stopwatch/countdown mode, recorded intervals, running timestamp and paused/finished -state are in the payload. Devices compute elapsed time from UTC timestamps locally. -There is no network write for each tick. Countdown elapsed time is capped at its -goal even when no device is open. - -## Concurrent and offline actions - -Every transition carries a unique payload version and the preceding `base_version`. -The upload handler locks the owner before checking that base version. The first -accepted transition wins. A stale branch and its descendants are acknowledged -without applying their changes, then PowerSync reconciles them to server state. -This follows PowerSync's [server-authoritative conflict resolution](https://docs.powersync.com/usage/lifecycle-maintenance/handling-update-conflicts). - -A partial unique index permits one active timer per owner. If two disconnected -devices start independent timers, the first uploaded start wins. The other session -is marked discarded. Offline edits remain usable locally until reconnection, but -cannot replace a more recent accepted branch. Device clocks should be synchronized. - -Finishing and saving writes the terminal timer, its reading activities and optional -book progress in one SQLite transaction and one upload batch. The server validates -these effects before committing. Conflicting or repeated saves skip the entire -batch, including progress updates. Terminal rows remain durable so later offline -writes cannot revive a saved or discarded session. Stable activity IDs use the -session UUID and interval index. - -Save batches must use `PUT` for every reading or completion activity. An activity -`PATCH` is rejected because a missing target would otherwise be silently skipped, -allowing the timer to become terminal without its history. Rejection leaves the -timer and book progress unchanged so a corrected batch can be retried. - -The old local `reading_timer` draft is moved to the new table once when its profile -opens. Guest and authenticated databases remain separate. Account/server switches -invalidate the old repository and subscription. - -## Rollout - -Deploy the server and replication changes before shipping the new client: - -1. Apply Alembic revision `c0d1e2f34567` (`uv run alembic upgrade head`). This adds a - table and indexes, leaving books and reading history untouched. -2. Add `public.reading_timers` to the PowerSync publication and grant the source - role SELECT access. The local setup script includes both changes. -3. Apply the updated `powersync/sync-config.yaml` and run the updated API. -4. Release the client that writes the new table. - -Older servers do not accept timer mutations. A new client must use the updated -server, otherwise its queued timer transactions remain pending. Timer save batches -are never split by tracking-capability discovery, because separating the ledger -from the terminal state would defeat exactly-once save handling. - -## Verification - -- `tests/services/test_reading_timer_sync.py` covers handoff, competing transitions, - simultaneous saves, retries, ownership, atomic rollback and deleted books. -- `tests/test_reading_timer_migration.py` verifies additive migration and metadata. -- Client persistence tests cover queued transaction boundaries, migration, guest - isolation, recovery and retained terminal states. -- `integration_test/powersync_timer_integration_test.dart` uses three isolated local - databases and two disposable accounts against live API and PowerSync services. - Run with `RUN_POWERSYNC_INTEGRATION=true` on macOS. It checks countdown handoff, - pause/resume, offline conflicts, competing saves, reset and account isolation. diff --git a/docs/support.md b/docs/support.md deleted file mode 100644 index 339d832..0000000 --- a/docs/support.md +++ /dev/null @@ -1,59 +0,0 @@ -# In-app support - -`POST /v1/support/issues` accepts bug reports and feature requests from guests and -signed-in readers. It creates a public issue in the configured repository (default -`PapyrusReader/client`). The request -is multipart: `payload` contains the JSON support fields and optional repeated -`attachments` parts contain images or videos. - -Set `SUPPORT_GITHUB_TOKEN` in the server environment (`production.env` for Compose) -to a fine-grained PAT restricted to that repository with **Issues: read and write**. -Also grant organization **Projects: read and write** for the Papyrus project. -The token owner must also have repository write access for media uploads. GitHub -App installation tokens cannot upload attachments. Keep the token out of client -builds. Without it, the endpoint returns 503. - -Routing is controlled by the `SUPPORT_GITHUB_*` environment variables shown in -`.env.example` and `deploy/production.env.example`: repository, assignee, project -owner/number, labels, issue types, and project field names/values. Set overrides in -the server's ignored `.env` or deployment `production.env`. These routing values -are ordinary configuration. Only `SUPPORT_GITHUB_TOKEN` is a secret. None of these -settings can be overridden by a support request. - -By default, bug reports receive the `bug` label and native `Bug` issue type. Feature requests -receive `enhancement` and `Feature`. Both are assigned to `Eoic` and added to the -Papyrus project (#1) with Status `Backlog` and Kind `Task`. Priority, milestone, -and requirement links are left for triage. Project access and field options are -checked before uploads or issue creation. The server sends the confirmed issue -response before applying project updates in a background task. Slow project updates -cannot delay that response. If updates fail, the server logs the issue number for -manual reconciliation. Background routing is not persisted or retried, so an abrupt -server shutdown can leave project metadata incomplete. - -Reading a public project and its `viewerCanUpdate` value only confirms visibility -and the token owner's role. It does not prove the token has Projects write access. -A `FORBIDDEN` project mutation is logged with the issue number and a permission -hint. Grant organization Projects read/write to the configured token, then add -any already-created reports to the project without resubmitting them. - -The form accepts up to five PNG, JPEG, GIF, WebP, MP4, MOV, or WebM files, each up to -10 MiB. The server checks sizes and media signatures before contacting GitHub. -The multipart request is bounded to 50 MiB plus 64 KiB of form overhead, including -chunked requests. Media is uploaded through the same native GitHub attachment -endpoint used by [GitHub CLI](https://github.com/cli/cli/blob/v2.102.0/internal/attachments/client.go), -then embedded in the issue. Original filenames are replaced with generic names. -Nothing is stored in the reader's library or exposed through library media routes. - -Submissions are limited to three per hour and ten per day per client IP using the -existing process-local limiter. Configure trusted proxy forwarding so requests -retain their client IP. Multiple API workers need a shared limiter or equivalent -edge limits before increasing deployment scale. - -Each GitHub request has a 30-second socket timeout. The client allows four minutes -for submission and never retries automatically. A failed submission preserves the -form and selected files. An attachment failure prevents issue creation, but files -already accepted by GitHub may remain public even if a later step fails. GitHub -does not provide an attachment deletion API. An issue-creation timeout may mean -GitHub created the issue without the client receiving confirmation. - -Tests mock GitHub and never create public issues or upload real attachments.