Skip to content

feat(integrations): ingest provider activity and trigger trusted workflows - #156

Draft
tee8z wants to merge 2 commits into
mainfrom
tee8z/buzz-ingest
Draft

tee8z wants to merge 2 commits into
mainfrom
tee8z/buzz-ingest

Conversation

@tee8z

@tee8z tee8z commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and behavior

GitHub reviews/CI, Linear issues, and public Slack discussions need a searchable Buzz record. buzz-ingest verifies provider webhooks and publishes ordinary channel threads, while Buzz workflows consume trusted integration messages. Provider-specific HTTP handlers remain outside the relay.

Surface Result
GitHub PRs, reviews/comments, and check runs become replies under an external thread anchor.
Linear Issues/comments and label additions become replies; UUID label names and webhook fallback routing are configurable.
Slack Public-channel messages/edits become threads; DMs/private channels are excluded.
Workflows integration_event selects source/action and requires explicit trusted publisher keys. Metadata is available as trigger_* values.
flowchart LR
    P[Provider] -->|HMAC-verified webhook| I[buzz-ingest]
    I -->|strong reads and signed write-ahead plans| R[Buzz relay]
    I -->|kind 9 anchor and occurrence reply| R
    R --> W[Trusted integration workflow]
    R --> C[Buzz and Paseo conversation history]
Loading

Kind 30623 stores bounded, base64-encoded signed plans before messages. Provider delivery and canonical payload fingerprint coordinates recover the same event after partial failure, restart, or changed unsigned headers. Every lookup requests writer consistency. An expired pending event changes only after an authoritative empty ID lookup; read errors propagate.

The service requires one replica with Recreate. It bounds bodies, labels, request rates, relay responses, and delivery deadlines. All external callbacks use the existing HTTPS hostname; the chart adds no NAT or database. The image uses a read-only nonroot runtime and joins the existing native ECR build matrix.

Validation

  • Five real HTTP journeys cover signatures, timestamp freshness, public Slack gating, mapping, label UUIDs, lost acknowledgements, restart, replay, journal ordering, expired plans, and strong-read failures, concurrent authenticated bursts, and redirecting an existing Linear issue to a newly configured channel.
  • Service fmt, clippy -D warnings, and tests pass. Existing core integration tests and workflow tests also pass.
  • Removing signature verification, freshness checking, fingerprint write-ahead persistence, NIP-10 reply markers, or current route precedence each makes a regression fail.
  • Built service: health 200, disconnected-relay readiness 503, missing signature 401, authenticated GitHub ping 200. SIGTERM drains and exits 0.
  • BuildKit static checks and GitHub Actions actionlint pass.
  • just ci passes workspace Rust formatting/clippy, then stops at native Tauri clippy: this machine cannot locate glib-2.0.pc. Later repository lanes are unverified.

The container compile/runtime contract test and real provider/relay delivery remain deployment validation. No production resources or provider settings were changed.

Rollout and review

Companion deployment: https://github.com/voltagecloud/cd/pull/677

  1. Provision prod ECR repository buzz-ingest and grant the existing image publisher access.
  2. Enroll a dedicated relay member key and add it to each mapped channel.
  3. Configure the Kubernetes Secret, mappings, and trusted workflow publisher keys.
  4. Merge source, pin its published digest in the CD values, sync the manual Application, then enable provider webhooks.

The chart refuses its placeholder digest until an actual image is supplied. Full configuration and replay procedures are in crates/buzz-ingest/README.md and docs/nips/NIP-IG.md.

Independent review found missing reply markers and stale saved-route precedence. Both are fixed; regressions bind the production NIP-10 parser and current mapping through the HTTP router. A bounded 32-request admission queue also handles ordinary concurrent CI deliveries under the existing four-second deadline.

This PR remains draft pending human workflow confirmation. No review completion marker attestation is present.

@tee8z tee8z self-assigned this Nov 10, 2026
@coderabbitai

coderabbitai Bot commented Nov 10, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@tee8z

tee8z commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review of 349be4f16 against signed provider ingestion, ordinary channel threads, retry safety, and trusted workflow triggers.

Two concrete blockers:

  1. Imported occurrences are not replies. plan.rs adds only an e/root marker on both initial and refreshed display plans. buzz_core::nip10::ThreadMarkers::resolve() explicitly treats root-only events as top-level. Reviews, comments, and check results therefore appear outside their intended thread and skip reply/descendant counter updates. Add e/reply naming the anchor on both paths. Bind the HTTP journey assertion to the production NIP-10 parser, then verify materialized counters against the local relay.

  2. An old issue route overrides an explicit Linear webhook mapping. normalize::linear chooses .or(routed) before the configured webhook mapping. http::process also queries the old route even when a webhook mapping exists. After an operator redirects a webhook to channel B, a comment can continue going to saved channel A while A remains among other configured destinations. This contradicts the documented mapping precedence and can disclose the comment to the wrong channel membership. Prefer project/team, then explicit webhook, then saved route. Add a journey that changes the webhook destination while a prior route remains.

Optional improvements:

  • Queue a bounded number of authenticated deliveries behind the serialized plan writer. The current single try_acquire before body validation rejects normal concurrent CI bursts. Keep a bounded admission pool and the overall deadline. GitHub failed callbacks need manual or operated redelivery; do not imply automatic retries.
  • Ordinary Linear payloads carry stateId; retain it when enriched state.name is absent so state-change metadata is not lost.

The write-ahead delivery/fingerprint ordering is sound for the stated single-publisher deployment. Strong reads, signature validation, bounded bodies/plans/responses, expired-plan refresh after confirmed absence, and the dedicated publisher identity match the design.

Evidence: source review of normalization, request verification/admission, relay acknowledgements, and journal planning; comparison with the production NIP-10 parser and official Linear payload documentation. Existing HTTP journeys and clippy pass, but their fake relay did not classify threads. Live local relay acceptance is being exercised separately. No provider or production changes were made.

Minimalism: 9/10. Elegance: 8/10 until routing precedence has one consistent owner. Correctness: 7/10 until both blockers are fixed. Optional items do not gate readiness.

Signed-off-by: Tee8z <tee8z@protonmail.com>
@tee8z

tee8z commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Independent follow-up review of c20572585: the reply-semantics blocker is resolved. Both newly planned and refreshed occurrences carry NIP-10 root and reply markers. The HTTP regression resolves them through the production parser.

I also exercised the published service against a real local relay in an isolated database/community. A verified GitHub fixture created the durable plans, one anchor, one direct occurrence reply, and one trusted workflow response. Stored thread metadata has depths 0/1/2; the anchor has reply_count=1 and descendant_count=2. Repeated delivery and a changed unsigned delivery header preserved the three display event IDs. Invalid HMAC returned 401.

The bounded admission queue retains the four-second end-to-end deadline. Current explicit Linear mappings take precedence over saved issue routing, with a production-router regression. No remaining concrete blocker in this follow-up. Minimalism 9/10, elegance 9/10, correctness 9/10.

This was local agent validation. Human runtime confirmation and deployment validation remain outstanding; the PR stays draft.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant