Skip to content

A native GitHub feed supplies team-owned tracker records - #786

Open
zlareb1 wants to merge 124 commits into
mainfrom
feat/tracker-github-feed
Open

zlareb1 wants to merge 124 commits into
mainfrom
feat/tracker-github-feed

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Related: #784, #837

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Operators can maintain GitHub issues, pull requests and comments in a team-owned ClickHouse tracker record with loom tracker-feed. Contract views apply observed repository visibility and hide content when that observation expires.

Why

Tracker search needs a record database. A scheduled feed supplies that record for teams that do not already import repository activity, while keeping copied content and erasure with the team that owns the database.

Scope and risk

The CLI checks its actual deployment storage authority and the organization held by its cell before record access. External connections recheck that deployment admission; record metadata never substitutes for deployment metadata. Exit closes both clients and the deployment Postgres pool.

The CLI initializes an empty dedicated database, configures an allowlisted repository and processes item, issue-comment and review-comment streams. Acknowledged writes and complete timestamp boundaries control cursor advancement; replays replace the same logical rows. Repository control versions advance beyond their persisted value so clock rollback cannot discard acknowledged configuration, hiding or checkpoint updates. Whole-millisecond publication versions increase across every repository and content table so late comments remain visible to existing tracker cursors.

Each content row stores its observed privacy. Views combine the most restrictive repository, item and stored-comment classification for both is_private and UUID identity, including when tables lag independently. Private-to-public transitions reset every stream cursor before public metadata is published; interrupted replay retains that reset. Stale public observations cannot widen access after another turn changes the configuration.

A record-wide lease covers initialization and publication, with uncertain sends retaining the claim through their bound. Provider reads run outside that lease. Owner, revision and watermark reads request sequential consistency. Writer admission accepts only ReplacingMergeTree and SharedReplacingMergeTree tables: the supported topology is a single self-hosted server or shared storage, not self-hosted replicated tables.

Writer credentials are separate from the serving reader and have verified direct grants limited to the record database and three system metadata tables. GitHub calls use GET on the fixed API origin, refuse redirects and validate identities. The operator attests read-only token permissions; a successful GET cannot certify absence of write grants. Tokens are sealed in the team's record and excluded from serving views and output.

Compatibility, rollout and rollback

  • Wire operations and defaults are unchanged. No deployment-schema migration or automatic worker is added.
  • Operator steps: create a dedicated record with separate writer/reader credentials, set the tracker organization, initialize it, register the read-only contract views, then configure and schedule each allowlisted repository. Installation tokens require external renewal and another configuration turn before expiry.
  • Sync turns are bounded to 120 seconds, 100 items per page and 2–100 pages per stream. Visibility expires after 15 minutes; it is a polling observation, not an instant permission mirror. Incremental lists do not detect deletions. The team stops all writers before erasing raw rows, comments and checkpoints or retiring a repository.
  • Rollback: stop the scheduled writer. The team retains ownership of its record and backups; expired observations hide its contract rows. Deleting a Loom namespace removes Loom's derived data, not the external record. No compatibility branch or data conversion is added.

Tests

Full final-head PR CI and a fresh independent whole-change review are required after this push. A prior head passed the unit shard containing the controlled-clock regression; that result is historical evidence, not final-head qualification. The coalescer burst regression controls its window clock and flushes explicitly: every existing request and verdict assertion remains unchanged, while runner delays cannot split the intended same-window burst. Prior-head CI artifacts confirm that all three native SQL regressions for configuration, hiding and cursor updates replace persisted future versions. The SQL tests exercise actual record tables and serving views; they do not measure Cloud replica timing. Provider responses are faked, and no live read-only GitHub probe has run.

The CLI admission repair passes 24 directly related authority, cell, target-routing, cleanup and source-contract checks; targeted typing reports no errors in the two CLI modules. The behavioral regressions first fail without the repair. Fixed memory, request, turn and observation ceilings retain their values.

Other regressions cover record ownership and isolation, direct and effective writer grants, pagination and incomplete boundaries, revision races, privacy lag, replay, late comments and publication ordering. A fake-clock regression proves that delayed responses cannot extend the receiving lease deadline; reproduce it with pytest tests/unit/test_external_ch.py::test_receiving_clock_never_extends_a_lease_deadline -q. Shared lease durations, send limits and guards are unchanged.

Canonical CLI help snapshots, generated references, the pipeline actor inventory and analyzer entries match the feature. The union with current main retains the canonical generated outputs; both CLI snapshot generation cases pass. Current main's complete translation-validation implementation and lifecycle tests are preserved exactly.

The composed Recall Lab repair reuses the score table until its results, settled query, opening callback, loading or visibility changes. A new regression checks draft edits without repeated score-data reads and immediate clearing. Its existing 100-score, fourfold-throttled browser guard retains the 200 ms budget. The earlier head measured 216 ms and 224 ms; the treatment result remains pending current-head CI. Local execution of this UI repair is unrun under the resource admission limit.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff). Independent source reviews and focused regression results accompany the change.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed.
  • Generated files are regenerated in this PR, not by hand: the SDK op tables, docs/reference/configuration.md and the docs/reference/ tables (python scripts/codegen/docs_reference.py), the reach manifest, schemas/schema.sql (make schema-dump) and the Codex plugin copies (python3 scripts/codegen/agent_plugin_sync.py).
  • A new migration is additive and numbered after the latest on main; no merged migration is edited.
  • A behaviour change to any file a plugin ships (hooks/, lib/, bin/, skills/, .mcp.json, the Codex plugin-defaults.json) bumps both plugin manifests' version, and no other open PR uses that version.
  • The frontend builds (npm --prefix frontend run build) and its tests pass, if frontend/ changed.
  • Docs under docs/ describe the system as it now is, and this description names any operator steps and wire changes.
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers; the history goes in this description (tests/unit/test_public_copy.py).
  • Commits are signed off (git commit -s), as CONTRIBUTING.md asks.

@zlareb1
zlareb1 force-pushed the feat/knowledge-push-sync branch from 77fd2f4 to f95a936 Compare November 17, 2026 21:20
@zlareb1
zlareb1 force-pushed the feat/tracker-github-feed branch from 5cba0c7 to 8dc2ef0 Compare November 17, 2026 21:20
@zlareb1
zlareb1 force-pushed the feat/knowledge-push-sync branch from f95a936 to 05f5636 Compare November 17, 2026 21:36
@zlareb1
zlareb1 force-pushed the feat/tracker-github-feed branch from 8dc2ef0 to c99dc7c Compare November 17, 2026 21:36
Keep generated drafts private until an administrator publishes their content. Register opt-in read-only query-history sources and bound generation by privacy thresholds and namespace caps.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the requested container name for both creation and cleanup. A failed creation leaves any existing container untouched.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reviewers can inspect the full body and evidence before promoting a definition. Describe private draft destinations and authenticated approval attribution.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Private definition drafts retain a supporting-account count without storing identities. Both exact lineage paths use that count while ordinary and assistant-rule candidates keep their existing accounting.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Private queue writes retain their proposal time and share a bounded writer lease with namespace deletion. Check erasure under that lease and retain uncertain sends so a delayed decision or model completion cannot restore erased evidence.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Resolve authorized tracker targets before narrowing capture consent and searching the other planes. Keep duplicate verdicts for scoped defect reports and leave memory and knowledge ranking unchanged for general tracker matches.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Empty duplicate verdicts remain useful evidence, while tag filters apply to memory and knowledge rather than tracker matches.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Older replicas neither preserve user policy nor protect private warnings. Activate after the fleet upgrade and settle and remove warning events before a maintenance rollback, then restore saved policy on upgrade.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the exact admin menu assertion aligned with the supported source actions.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Carry the connected store through registry resealing and its lifecycle fences. Keep timeout defaults in the configuration catalog, and align source contracts and generated snapshots.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Namespace name checks must not scan every retained source registration in an organization. Keep callback lookups and maintenance cursors on the same physical key prefix.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Record the official schema and statement catalog for namespace-scoped push lookups.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the feed and receiving-clock behavior while inheriting namespace-keyed source notifications.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The renewal lease permits work for forty percent of its lifetime. Reject provider and cleanup budgets that exceed that window while retaining the existing defaults and lease policy.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Inherit the source-push lease budget check while preserving feed behavior.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the merged mirror latch migration and place source push after it.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Clock rollback must not discard an acknowledged visibility, credential or cursor update.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Record the namespace-keyed registry and deployment latch in one official schema.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve receiving-clock and control-version bounds with namespace-keyed source notifications.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A permitted slash in an organization must not become a callback path separator.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve source callback identities while retaining the feed publication bounds.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep feed publication and control-version bounds while aligning the branch with main.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 changed the base branch from feat/knowledge-push-sync to main November 18, 2026 06:38
@zlareb1 zlareb1 closed this Nov 18, 2026
@zlareb1 zlareb1 reopened this Nov 18, 2026
Check the deployment authority and held organization before record work, keep external metadata separate, and close the Postgres pool on exit. Classify fixed CLI and privacy bounds without changing their values.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Index normalized table rows while preserving first-match offsets and unmatched row order. Use prefix maximum ends for protected-line lookups.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve external record publication while retaining deployment observation controls.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse indexed page metadata without changing translation request contents.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Parse each anchored destination once per validation or repair pass. Check fragmentless destinations without reading their bodies.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse page link validation across translation candidates.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The test assumes five submissions share a window, but a busy runner can
advance the real loop clock between them. Use the manual clock and flush
explicitly so every existing request and verdict assertion tests that premise.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Draft query edits rebuilt the score table before the debounced search ran.
Reuse the table until its result, loading or opening context changes, while
keeping query cancellation and immediate clearing unchanged.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse parsed destination bodies while rechecking file contents for each link. Keep parse failures visible for links without fragments.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant