Skip to content

fix(server): extend the http keep-alive window to 360 seconds - #863

Draft
Avicennasis wants to merge 1 commit into
FlashML-org:mainfrom
Avicennasis:rm52586-keepalive-window
Draft

Avicennasis wants to merge 1 commit into
FlashML-org:mainfrom
Avicennasis:rm52586-keepalive-window

Conversation

@Avicennasis

Copy link
Copy Markdown

FreeToken currently inherits uvicorn's 5-second HTTP idle timeout. Add --http-keep-alive-s, default 360 seconds, and pass it to uvicorn in both normal and terminal-shell startup. Document that it must exceed the longest client pool idle timeout; a header-blind client pool can otherwise race the server's close and lose an unretried POST.

The idea comes from Infernix PR #55, whose revision was cherry-picked as 9ee5e543, and the maintainer's 360-second follow-up. The subsequent buffered-close change corrects an earlier claim: Bun probes pooled sockets, but a FIN can arrive after the probe. This configuration change copies no donor code; both projects use Apache-2.0. It does not add Infernix's Keep-Alive advertisement, close buffer or independent HTTP-client ceiling. Uvicorn's limit_concurrency remains unset because it couples connections/tasks to request admission.

Verified on Linux x86_64, Python 3.12.3, CPU-only torch 2.14.1+cpu, uvicorn 0.54.0. From the repository root:

export PYTHONPATH=python
python -m pytest tests/server/test_api_server.py -q
python -m pytest tests/server/ -q
python -m pytest tests/ -q --continue-on-collection-errors

The regression was written first: 4 failed on the base (both startup paths used 5 seconds; the override flag was unknown), then 4 passed after the change. It inspects real uvicorn.Config objects without binding a socket. The server subsystem has 660 passing tests. Full suite on the branch: 1669 passed, 20 failed, 919 skipped, one warning. On the base: 1665 passed, 20 failed, 919 skipped, one warning. All 20 failing IDs are identical and are the known CPU-only environment failures. python -m freetoken.cli serve --help and git diff --check also passed.

Bun 1.4.4 ran two POSTs with a 10-second idle gap through FreeToken's actual startup with a CPU-only echo route and no model. Baseline returned HTTP 200 twice on different connections; both the new default and explicit --http-keep-alive-s 360 returned HTTP 200 twice on the same connection. This verifies the idle lifetime, not the timing race: the lost POST on main was NOT reproduced. Real small-model ft serve before/after, GPU/checkpoint loading, 300-second pool expiry and live terminal-shell interaction are NOT VERIFIED.

Based directly on 628b88f and does not depend on #334, #459, #565, #593 or draft #656; their file lists do not overlap. #592 overlaps only in docs/cli.md, in the separate multimodal section. Its media flags are already on main, so this argument is added beside --port. No merge-tree stacking check was run.

Maintainer question: is 360 seconds an acceptable HTTP idle default for Bun/opencode deployments? Operators serving longer-lived header-blind pools must choose a larger value with margin.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant