Repository navigation
fix(server): extend the http keep-alive window to 360 seconds - #863
Draft
Avicennasis wants to merge 1 commit into
Draft
Avicennasis wants to merge 1 commit into
Avicennasis wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FreeToken currently inherits uvicorn's 5-second HTTP idle timeout. Add
--http-keep-alive-s, default 360 seconds, and pass it to uvicorn in both normal and terminal-shell startup. Document that it must exceed the longest client pool idle timeout; a header-blind client pool can otherwise race the server's close and lose an unretried POST.The idea comes from Infernix PR #55, whose revision was cherry-picked as 9ee5e543, and the maintainer's 360-second follow-up. The subsequent buffered-close change corrects an earlier claim: Bun probes pooled sockets, but a FIN can arrive after the probe. This configuration change copies no donor code; both projects use Apache-2.0. It does not add Infernix's Keep-Alive advertisement, close buffer or independent HTTP-client ceiling. Uvicorn's
limit_concurrencyremains unset because it couples connections/tasks to request admission.Verified on Linux x86_64, Python 3.12.3, CPU-only torch 2.14.1+cpu, uvicorn 0.54.0. From the repository root:
export PYTHONPATH=python python -m pytest tests/server/test_api_server.py -q python -m pytest tests/server/ -q python -m pytest tests/ -q --continue-on-collection-errorsThe regression was written first: 4 failed on the base (both startup paths used 5 seconds; the override flag was unknown), then 4 passed after the change. It inspects real
uvicorn.Configobjects without binding a socket. The server subsystem has 660 passing tests. Full suite on the branch: 1669 passed, 20 failed, 919 skipped, one warning. On the base: 1665 passed, 20 failed, 919 skipped, one warning. All 20 failing IDs are identical and are the known CPU-only environment failures.python -m freetoken.cli serve --helpandgit diff --checkalso passed.Bun 1.4.4 ran two POSTs with a 10-second idle gap through FreeToken's actual startup with a CPU-only echo route and no model. Baseline returned HTTP 200 twice on different connections; both the new default and explicit
--http-keep-alive-s 360returned HTTP 200 twice on the same connection. This verifies the idle lifetime, not the timing race: the lost POST on main was NOT reproduced. Real small-modelft servebefore/after, GPU/checkpoint loading, 300-second pool expiry and live terminal-shell interaction are NOT VERIFIED.Based directly on 628b88f and does not depend on #334, #459, #565, #593 or draft #656; their file lists do not overlap. #592 overlaps only in
docs/cli.md, in the separate multimodal section. Its media flags are already on main, so this argument is added beside--port. No merge-tree stacking check was run.Maintainer question: is 360 seconds an acceptable HTTP idle default for Bun/opencode deployments? Operators serving longer-lived header-blind pools must choose a larger value with margin.