Skip to content

fix(metrics): drop connection on write timeout - #127

Open
bappisinsane wants to merge 1 commit into
jamesconwaydev:mainfrom
bappisinsane:fix/metrics-write-timeout
Open

bappisinsane wants to merge 1 commit into
jamesconwaydev:mainfrom
bappisinsane:fix/metrics-write-timeout

Conversation

@bappisinsane

Copy link
Copy Markdown

Fixes #124.

When a write times out, the metrics endpoint now aborts the transport so the buffered reply and the socket are dropped at once instead of waiting for the buffer to flush.

  • _handle aborts writer.transport on connection/timeout errors.
  • CHANGELOG entry under [Unreleased] -> Fixed.

@jamesconwaydev jamesconwaydev left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this one too. It has the same three gaps as #126, so my review there applies here as well:

  1. A test that fails on main and passes with your change. #124 describes it: monkeypatch WRITE_TIMEOUT down, serve a registry whose render() returns tens of megabytes, connect with a small SO_RCVBUF and don't read, then check the connection gets reset or closed.
  2. Sign-off for the dco check: git commit --amend -s --no-edit, then force-push.
  3. A smaller change. TimeoutError is already an OSError, so the except line can stay as it was. writer.transport always exists and abort() doesn't raise, so the body comes down to writer.transport.abort(), without the getattr, the suppress, the contextlib import or the pass.

For the CHANGELOG, say what a reader would notice, something like: "The metrics endpoint closes a connection whose client stops reading, instead of keeping it and its response open after the write timeout (#124)."

I've approved the CI run for this commit, and I'll approve the next one when you push.

@bappisinsane

Copy link
Copy Markdown
Author

Addressed all three points:

  1. Simplified to writer.transport.abort(), reverting the except line to the original except (ConnectionError, OSError) (TimeoutError is an OSError).
  2. Added Signed-off-by via git commit -s and force-pushed.
  3. Updated the CHANGELOG line per your suggestion.

On the test: I haven't yet added the write-timeout regression test; I'll follow up with it.

@jamesconwaydev

Copy link
Copy Markdown
Owner

Thanks, @bappisinsane. The code change is exactly what I was after, and so is the CHANGELOG line. What's holding it up now is the branch, plus the test you already mentioned.

The branch conflicts with main. 0.2.9 went out and moved everything under [Unreleased], so your CHANGELOG line needs a new ### Fixed heading there. GitHub won't start CI on a conflicting PR, which is why no checks show up and there's nothing for me to approve yet.

The first commit still has no sign-off. The dco check reads every commit in the PR, and only the second one has the line. Squashing fixes both problems in one go:

git reset --soft HEAD~2
git commit -s
git fetch https://github.com/jamesconwaydev/sentry-v2 main
git rebase FETCH_HEAD        # fix CHANGELOG.md, git add it, git rebase --continue
git push --force-with-lease

The test is the part I can't merge without. #124 describes what to check: a client that stops reading gets its connection closed after the write timeout. It should fail on main and pass with your change. Push it whenever you're ready, on this PR, and I'll approve the CI run once it's up.

@bappisinsane

Copy link
Copy Markdown
Author

Thanks for the detailed guidance. I'll squash both commits with sign-off, rebase onto main, fix the CHANGELOG heading under [Unreleased], and add the write-timeout regression test per #124. Will push with --force-with-lease and ping once CI is green.

@jamesconwaydev

Copy link
Copy Markdown
Owner

Nice work, @bappisinsane. The squash and rebase came out clean: dco passes, the CHANGELOG line landed in the right place, and I approved the CI run. lint, typecheck and test are all green on 3531729.

So the test is all that's left, and it doesn't have to be big. Start from test_server_header_timeout in tests/unit/test_metrics.py, then follow the recipe in #124. Before you push, run it once with your fix stashed. If it doesn't fail on main, it isn't catching the bug yet.

Push it to this branch and I'll approve the next run.

jamesconwaydev added a commit that referenced this pull request Nov 2, 2026
…#143)

F2 in `docs/security-review.md`. The metrics endpoint had no connection
limit, so any local user could hold connections open (5 s each) until
the daemon hit its 1024 fd limit, taking the control socket and sinks
down with it. The server now closes any connection past 16 open ones.

The new test opens two idle connections against a cap of 2 and checks
the third is closed at once (on `main` it waits for the header timeout
and the test times out), then that a scrape works once they're gone. Off
by default, so only hosts with `metrics.enabled = true` were exposed.
#127 still covers aborting a connection that stops reading.

Signed-off-by: James Conway <148910373+jamesconwaydev@users.noreply.github.com>
Signed-off-by: bappisinsane <108801304+bappisinsane@users.noreply.github.com>
@bappisinsane
bappisinsane force-pushed the fix/metrics-write-timeout branch from 3531729 to d9b69d5 Compare November 3, 2026 16:49
@jamesconwaydev

Copy link
Copy Markdown
Owner

Same here, @bappisinsane: the fix is right and CI is green on d9b69d5. It needs the same two things as #126:

  1. The regression test from Metrics endpoint keeps a connection open after its write timeout #124: a client that stops reading the response, and a check that the server drops the connection after the write timeout and _open comes back down. It should fail on main and pass here.
  2. The CHANGELOG line moved from [0.2.9] up to ### Fixed under [Unreleased], since 0.2.10 has shipped.

Same offer stands: if you'd rather I finish it, say so and I'll add the test on top, credit stays yours.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metrics endpoint keeps a connection open after its write timeout

2 participants