Skip to content

feat(taskbroker): Retry activations that miss their heartbeat - #847

Open
evanh wants to merge 4 commits into
mainfrom
evanh/feat/broker-heartbeat
Open

evanh wants to merge 4 commits into
mainfrom
evanh/feat/broker-heartbeat

Conversation

@evanh

@evanh evanh commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

STREAM-2032

Depends on getsentry/sentry-protos#503

Activations with a heartbeat get a heartbeat_deadline when they start processing, and workers push it back with
the UpdateHeartbeat RPC. Upkeep retries any activation past its heartbeat_deadline the same way it handles a
missed processing deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh
evanh requested a review from a team as a code owner November 5, 2026 15:42
@linear-code

linear-code Bot commented Nov 5, 2026

Copy link
Copy Markdown

STREAM-2032

@evanh

evanh commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Ran locally against sqlite and postgres (devservices):

$ cargo test -q --lib heartbeat
running 6 tests
......
test result: ok. 6 passed; 0 failed

$ cargo test -q
test result: ok. 350 passed; 0 failed

$ cargo clippy --workspace --all-targets --all-features --no-deps -- -D warnings
(clean)

New metrics: grpc_server.update_heartbeat{result=ok|not_found|error} and upkeep.handle_processing_deadline.reset{reason=processing_deadline|heartbeat}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh

evanh commented Nov 5, 2026

Copy link
Copy Markdown
Member Author
$ cargo clippy --all-targets -q -- -D warnings
(no output)

$ cargo test -q --lib heartbeat
running 6 tests
......
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 344 filtered out

$ cargo test -q
test result: ok. 350 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Heartbeat tests run on sqlite and postgres. They cover BatchUpdateHeartbeat with a mix of live and missing ids: only missing comes back in not_found_ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh

evanh commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

BatchUpdateHeartbeatRequest is now repeated string ids. Bumped the git rev to 0383e4f.

$ cargo clippy --all-targets -q -- -D warnings
(no output)

$ cargo test -q
test result: ok. 350 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant