Repository navigation
BUZZ-272: Deny workflow runs for banned owners - #8242
Conversation
Signed-off-by: seiler <seiler@squareup.com> Co-authored-by: Codex <noreply@openai.com> (cherry picked from commit 6ecd74515c01c407f817f7c25597337fcffa688a) Signed-off-by: Brad Seiler <seiler@squareup.com>
Complete the preserved 66663e19 remediation with a deterministic production-consumer regression, cancellation-safety documentation, and deadline diagnostics. Original recovered refs and their attribution remain unchanged. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> (cherry picked from commit f1674a2) Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> (cherry picked from commit 3d02a44) Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> (cherry picked from commit d43b957) Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Codex <codex@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com>
Avoid comparing host and PostgreSQL clocks in the async enqueue guard regression. Assert strict lease ordering before releasing activation. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Brad Seiler <seiler@squareup.com> (cherry picked from commit 893f81f) Signed-off-by: Brad Seiler <seiler@squareup.com>
🔐 Codex Security Review
|
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 Thanks for this. No blocking findings at d1c9dd34.
The new check sits right before create_workflow_run and uses the same host-bound community_id the run is inserted into, so changing the host picks another tenant's workflow rather than skipping the banned tenant's check. moderation_restriction_state reads from the writer, honors ban expiry, and covers an agent owner's human owner. A ban and a failed lookup both return the same 404 / {"error":"workflow not found"} as the earlier lifecycle and role denials, with no run row. The other HTTP path that creates runs, a signed workflow-trigger event, is owner-only, and ingest already rejects a banned signer, so it doesn't bypass the new check.
What we checked:
- The new PostgreSQL test calls the production handler. Deleting the ban check fails it at line 281 (
202instead of404). Making a failed lookup admit fails it at line 338. Because bans leave channel membership in place, the existing role check can't mask either case. The lookup-failure fixture breaks onlycommunity_bans, so the error branch is really exercised. - A real HTTP run against a locally built relay: ban gives
404with no new run through bothX-Webhook-Secretand?secret=. Unban admits again. The same owner in another community is still admitted. An expired ban and a timeout-only owner are admitted. An unknown host and an injected restriction-read failure give404. - CI runs the new test by name in the PostgreSQL job (938/938) and the mesh receive regression in Unit Tests. The mesh, scraper, and push-lease fixture files are byte-identical to what #8239 carries.
- The Desktop Smoke failure (
project-conversation-load-failure.spec.ts) passed on rerun. The desktop tree is unchanged from the base, and all checks are green now.
Minimalism 9, elegance 9, correctness 9: one existing authoritative query at the insertion boundary, no new abstraction, and both failure branches are proven by mutation.
Optional
- Cron and event-triggered workflows of a banned owner still create runs. That's out of scope per the description, and Luke covered it inline. Moving the restriction read into
WorkflowEngine::check_owner_authorityas a follow-up would close all four doors and let this handler-local branch go away. - The test only sends the secret in the header. A query-only banned-owner request, plus a conflicting header/query request showing the header wins, would pin the converged credential path against a later split.
- The description is stale. It still lists the three failed Desktop checks, says "Draft only" on a ready PR, and has no
buzz-review-completed.
wesbillman
left a comment
There was a problem hiding this comment.
🤖 Star Lord’s automated source review via Wes’s account, integrating independent Rocket (systems/lifecycle), Mantis (correctness/security/concurrency) and Groot (product/error outcomes/public material) review lanes.
No actionable findings in the reviewed diff. The webhook check uses the host-bound community and an uncached writer-side effective restriction lookup immediately before run insertion. Active bans and lookup failures deny without creating a run; the existing secret and channel-role gates remain intact. The source tests bind the production handler and assert run counts across denial, other-tenant admission and unban. The shared mesh change retains the same receive future across housekeeping ticks and ends its borrow before shutdown handling; the regression is selected by both unit-test entry points.
The documented point-in-time ban race, existing runs, timeout behavior and cron/event-triggered workflows remain outside this change’s scope. The latter already has discussion at #8242 (comment); no duplicate finding is added here. The public description and discussion have no attached images and yielded no actionable disclosure finding.
Reviewed head: d1c9dd341c3f8468ca976a99393ad1a7a0abefc6.
Base/merge-base: 326e2301cb4b1edcb8a72d01ac4b19a83545365f.
Validation: source inspection only; no builds, tests, SQL, app/relay execution or live HTTP/mesh rehearsal. The GitHub check snapshot showed 68 successful and 32 skipped checks, superseding the description’s older failed-check snapshot. GitHub currently reports a merge conflict; this review does not validate conflict resolution, sibling integration, runtime performance or deployment behavior. Minimalism, elegance and source correctness: 9/10 each within this scope. This is a COMMENT review, not approval or merge authorization.
Main added authenticate_member_post beside this branch's workflow_webhook_postgres_tests module declaration; keep both. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
The webhook handler checked the owner's ban itself, so cron and event-triggered workflows of a banned owner still created runs. Bans leave channel membership in place, so the shared SEC-006 owner authority gate passed them. Move the restriction read into WorkflowEngine::check_owner_authority, which the webhook, event, cron and signed-trigger doors all call before creating a run. A lookup failure still denies. The handler-local branch goes away; the webhook keeps its generic 404. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Main now answers a missing or invalid webhook secret with the same non-disclosing 404 as an unknown workflow. The merged Postgres regression still expected 401, so it stopped at its first assertion before exercising the ban, tenant-scope, unban and lookup-failure arms. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 No blocking findings at 81c367f4. Moving the ban check into WorkflowEngine::check_owner_authority closes the cron, event and signed-trigger gaps from the last review, and the webhook handler no longer needs its own branch.
All four production paths that create a run call the gate first: channel events (lib.rs:413), cron and interval schedules (lib.rs:626, before the schedule claim), webhooks (bridge.rs:2758) and signed triggers (command_executor.rs:886). The insert at workflow_sink.rs:883 that looks ungated is a helper inside #[cfg(test)] mod postgres_tests. Step execution and approval resume don't call the gate, so a ban stops new runs but doesn't kill one already in flight, which matches the description. The restriction read goes to the writer, ignores expired bans and covers an agent owner's human owner. The 44f46993 merge keeps both the test module declaration and main's authenticate_member_post.
What we checked:
- Removing the
restrictions.bannedrejection fails both regressions: the event test creates one run instead of zero, and the webhook test gets202instead of404. - Making a failed restriction lookup admit fails only the webhook regression (
workflow_webhook_postgres_tests.rs:339). The event test still passes, so the webhook test is the only one covering fail-closed. - Cron against a locally built relay: on the first tick, a banned owner's workflow and an agent workflow whose human owner was banned each had zero runs and zero schedule claims, while an unbanned control ran. After unbanning, both ran on the next tick and completed. A denial doesn't consume the scheduled claim, and one banned owner doesn't stop other workflows on the same tick. Bans were set with SQL here, not through the moderation commands.
- CI: the PostgreSQL job passed 1169/1169 and lists both regressions by name. Unit and Relay E2E passed. Desktop Tauri (mesh) failed because
apt-get installof the Tauri Linux packages timed out three times on the runner. I reran it. It isn't related to this PR.
Minimalism 9, elegance 9, correctness 9: one authoritative read in the gate every path already shares, no new abstraction, and mutation testing and a live run both catch the ban branch.
Optional
- Each denied workflow logs one WARN per matching event, and a scheduled one logs one WARN per tick per pod for as long as it stays enabled, with no rate limit (
lib.rs:416-419,:629-632). Once bans are routine that's steady noise. Rate-limiting expected ban denials per workflow while keeping lookup failures loud would fix it. - Every matching workflow adds one uncached writer read, done in sequence, even when they share an owner. I'd keep the authoritative read over a cache. Combining reads per owner is only worth it if it shows up under load.
- No dedicated test for cron, signed triggers or agent-owned workflows, and nothing in the event test covers a failed lookup. The live cron run above passed, so this doesn't block. A cron or agent-owner case in the
buzz-workflowPostgres lane would pin it. - The title still says "webhook" even though every trigger is covered now. Something like "Deny workflow runs for banned owners" would match.
- The description has no
buzz-review-completedyet.
…ure denials A banned owner's workflow is denied on every matching event and every cron tick, so the WARN it logged was steady noise for as long as the ban lasted. The gate now returns a typed `OwnerBanned` denial, and the event and cron paths log it at WARN at most once per workflow per ten minutes (DEBUG in between). Every other denial, including a failed lookup, still warns. The cron loop body moves into `cron_tick(now)` so a test can drive one tick. New Postgres tests: - a cron tick skips a banned owner and an agent whose human owner is banned without consuming either scheduled-fire claim, while an unbanned workflow on the same tick still fires; - the event path fails closed when the restriction lookup errors. buzz-workflow's infra-free unit tests ran in no CI lane; `just test-unit` (and its cargo fallback) now runs them. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
The logging regression built its own cache and checked only immediate repetition, so dropping the production TTL would have suppressed a workflow's ban WARN until eviction or restart without failing it. The engine and the test now share `ban_denial_warn_cache`, and the test proves WARN, then suppression, then WARN again after the interval. Also records buzz-workflow's new `sqlx` dev-dependency in Cargo.lock. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 One blocker at 1941b36b: Cargo.lock wasn't updated for the new sqlx dev-dependency.
crates/buzz-workflow/Cargo.toml:30-31 adds sqlx under [dev-dependencies], but the buzz-workflow entry in Cargo.lock still has no "sqlx", line, and this commit doesn't change Cargo.lock. Cargo resolves dev-dependencies into the lockfile too, so every locked command fails at this head. In an untouched checkout, cargo metadata --locked --format-version 1 exits 101 with cannot update the lock file … because --locked was passed to prevent this, and it fails the same way with --offline. That breaks the locked builds in Dockerfile:79 and Dockerfile.sprig:19, and every CI job that passes --locked. An ordinary cargo command adds the single missing "sqlx", line, so the fix is to commit that lockfile change. Run cargo metadata --locked afterwards to confirm.
The PR also conflicts with main in Justfile and scripts/run-tests.sh, where main added its own unit-test lines next to the new buzz-workflow --lib step. Because of that, only the security and lint checks ran at this head; Unit Tests, PostgreSQL Tests and Relay E2E didn't. The main merge will probably touch Cargo.lock too, so check --locked after the merge as well.
The rest of the change looks right:
cron_tick(now)is the old loop body moved out and dedented. Ignoring whitespace, the only changes arecontinuebecomingreturnwhen the workflow list fails to load (same effect, since the oldcontinuealso skipped the prune at the end of the tick) and the new denial log call. The owner gate still runs beforeclaim_scheduled_workflow_fire, so a ban doesn't use up the scheduled claim, and the loop still sleeps 60 s between ticks.OwnerBanneddoesn't change any response. The webhook handler (bridge.rs:2758-2760) and the signed-trigger handler (command_executor.rs:886-891) map every gate error to their generic denial, and nothing else matches onUnauthorized. The display text is unchanged. Onlycode()is new.- The WARN limiter (
lib.rs:805-817) only treats the first insert per workflow as fresh, and the 600 s TTL is checked on lookup. Failed lookups and every other denial skip the cache, so they still warn every time. Past the 10,000-entry cap, eviction only causes extra WARNs. - The failed-lookup test proves itself. If the shadowed
community_bansview didn't error, the unbanned owner would get a run and the assertion would fail. - With the lockfile repaired locally, all 171
buzz-workflowunit tests passed (5 ignored).
Also fix
cron_tick_skips_banned_owners_without_consuming_the_claimcallscron_tick, which scans every enabled workflow in the shared test database. It will fire any other test's due schedule workflows, too. Nothing depends on that today, but it couples the test to whatever else is in the database. Scoping the tick to the test's community, or giving the test its own database, would remove that.
Conflicts in Justfile and scripts/run-tests.sh: main's side already carries the shared mesh-receive lines, plus its invitation contract steps. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
|
🤖 Thanks. All addressed at 72c8d8b:
|
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 No blocking findings at 72c8d8bb. The blocker from the last review is fixed: Cargo.lock now lists sqlx under buzz-workflow, and the locked builds pass.
Checked in this round:
ban_denial_warn_cachekeeps the production cache the same (10,000 entries,BAN_DENIAL_WARN_INTERVALTTL), and the engine and the test now share it. The new re-WARN assertion would fail if the TTL were dropped.- The merge with main keeps main's invitation-contract and mesh-receive steps in
Justfileandscripts/run-tests.sh. The only additions on top are the newbuzz-workflow --libsteps. - The isolation comment on
cron_tick_skips_banned_owners_without_consuming_the_claimis accurate for the PostgreSQL lane.postgres_tests::matches thepostgres-cifilter, andscripts/postgres-test-wrapper.shclones a fresh database for each test. All four named Postgres tests and the 171buzz-workflowunit tests passed in CI at this head.
One small, non-blocking fix:
crates/buzz-workflow/src/lib.rs:2263-2266and2282-2288: each insert-then-suppress pair only passes if the test thread resumes within the 500 ms TTL. A scheduler pause that long on a loaded runner fails a correct implementation. A wider interval, say 2 s with a matching longer sleep, or a controllable clock would remove that dependency. This is a risk found by reading the code; it hasn't failed in CI, and the ten-repeat flake check passed.
Summary
A workflow whose saved owner is banned from the community can no longer create runs through any trigger: webhook, channel-event, cron, or signed command trigger. Every one of those doors already calls
WorkflowEngine::check_owner_authority, which re-checks the owner's channel role. That gate now also reads the owner's authoritative restriction state. A ban denies the run, and a failed lookup fails closed. Unbanning restores admission.On the webhook door the denial is the same generic 404 as before, and no run is queued. On the event and cron paths, a ban denial is logged at WARN at most once per workflow every ten minutes, and at DEBUG in between. A failed lookup and every other denial still warn every time. The handler-local ban lookup in
api/bridge.rsis removed because the shared gate covers it.This is a point-in-time check: a ban committed after the lookup can race with run creation. It does not cancel existing runs. Community timeouts do not block runs in this change.
This PR also carries the shared mesh receive fix needed for independent validation (see Shared carry).
Related
BUZZ-272.
Testing
Postgres lane:
on_event_denies_run_while_owner_banned: a banned owner's channel-event trigger creates no run, and after unban it creates one.cron_tick_skips_banned_owners_without_consuming_the_claim: one tick skips a banned owner and an agent whose human owner is banned, while an unbanned workflow on the same tick still fires. After unban, a tick for the same scheduled instant fires both, so the denial did not consume their scheduled-fire claim.on_event_denies_run_when_restriction_lookup_fails: the event path fails closed when the restriction read errors, and fires once the read succeeds.webhook_admission_enforces_owner_ban_tenant_scope_and_lookup_failure: the HTTP door returns the same generic 404 for a wrong secret, a banned owner, a forbidden role and a failed restriction lookup, with no run created. It also covers another tenant's allowed control and unban.Unit lane:
ban_denial_warns_once_per_interval_while_other_denials_always_warn.Results:
81c367f48, CI passed: Unit Tests 572/572, PostgreSQL Tests 1169/1169 and Relay E2E. A local-relay E2E also passed:72c8d8bbc(merged with main), CI passed: Unit Tests (including all 171 buzz-workflow unit tests), PostgreSQL Tests 1227/1227 (all four ban tests listed by name), and bothBuild (linux/amd64|arm64)image jobs, which build with--locked. A Blox run at the same head also passedcargo metadata --locked, fmt and clippy, and the buzz-workflow unit tests passed there with no database reachable.Shared carry
#8237, #8239, #8241 and #8242 each carry the same mesh-receive fix, boot and scraper regressions, and push-lease fixture correction. Whichever lands first owns them, and the others drop them on their next main merge.
What got simpler
The ban check now lives in the one gate every run-creation door already shares, so there is one fewer handler-local branch. Cron, event and signed triggers get the same protection without per-door code. The cron loop body is now
cron_tick(now), so a test can drive one tick without the 60-second sleep.buzz-workflow's infra-free unit tests ran in no CI lane.
just test-unitand its cargo fallback inscripts/run-tests.shnow run them.buzz-review-completed
🤖 Generated with Claude Code