Repository navigation
fix(mqtt): hold token renewal bounces for the cross-slot guard - #99
Merged
Merged
Conversation
A Wi-Fi drop cut every slot's session, and the link-recovery retry advanced each slot's backoff level. Sessions shorter than the 120 s stable clear never reset it, so a node whose link flapped every couple of minutes walked every slot up to the 300 s rung and toward the breaker while the broker was fine. At the link-down edge, before live clients are closed, a slot that was still connected or whose session ended within 20 s of the edge (either side, since the socket usually fails before the polled edge) is marked as a link-caused outage. Its recovery retry then holds the ladder where it was. Slots already failing before the drop, and any drop when the link had been up under 30 s, advance as before. Refusals, handshake failures and quick drops while the link stays up are unchanged.
The 10 s status poll reports the link-down edge after the socket has already died, so measuring the floor to the edge could excuse a link that really held for less than kLinkFlapFloorMs. Use the outage stamp when the slot has one inside the current link-up period.
A JWT renewal bounce and the corrected-clock reconnects started TLS handshakes without checking the cross-slot guard, so they could overlap another slot's handshake. A renewal that needs a bounce now waits, before minting or disconnecting, until the guard is idle and no handshake was launched in the pass. The corrected-clock path stages the fresh token and marks the slot owed; maintainSlotConnection() runs each owed reconnect through the same guard, one per window. Both arm the full guard as before.
…queued tokens A reconnect or breaker probe rejected locally armed the cross-slot guard without starting a handshake; with the backoff rolled back, a slot failing every 10 s could hold a waiting renewal off indefinitely. The guard now reverts to the last real handshake on a local rejection, and while any slot's renewal or corrected-clock bounce is waiting, other slots' backoff reconnects and probes stand aside so it gets the next idle window. A queued corrected-clock reconnect now re-mints first when its staged token has expired or aged into the renewal buffer, so it presents fresh credentials and is not followed by a second renewal bounce.
A slot still marked connected at the polled link-down edge has no outage stamp, so the floor measured the polled edge as uptime. Require the poll slack (10 s) in that case; a stamped session end keeps the 30 s floor.
…ttempt After a link outage a disconnected JWT slot can reconnect through the renewal or corrected-clock path instead of the backoff path. That left the link-retry flag set, so if the handshake failed before CONNACK the next backoff retry was treated as link recovery a second time. Clear the flag once the reconnect is accepted; a local rejection keeps it.
…pted The renewal and corrected-clock paths cleared the link-retry flag but left the outage attribution set. A second link-up edge without a new link-down edge re-armed the retry and the stale attribution held the ladder again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 15 s cross-slot guard keeps two TLS handshakes from overlapping (one handshake needs about 35 KB of internal heap, and boards without PSRAM run with 40-70 KB free). Two paths started handshakes without consulting it:
maintainSlotConnection()soft-disconnected and reconnected at once, even if another slot had started a handshake in the same pass or within the guard window. It armed the guard only afterwards.syncTimeWithNTP()reconnected or bounced every slot with a stale token in one loop, back to back, and never armed the guard.Fix
Both paths now take a turn through the guard.
clock_bounce_owed). Each marked slot is served inmaintainSlotConnection(), one per guard window. If its staged token has aged into the renewal buffer by then, it is re-minted first, so there is one handshake with fresh credentials.The decisions are pure functions in
MQTTConnectionPolicy.h(renewalStep,owedBounceStep,bounceHandshakeAllowed,owedDispatchNeedsMint). New state is bridge-task only: two bools per slot and one mask byte.Worst-case wait for a renewal is about 90 s with five slots, against the 300 s renewal lead of every shipped preset. A custom token lifetime of 600 s or less (60 s lead) can expire while waiting; the broker then closes the session and the reconnect goes out with a fresh token.
Unchanged: first setups, reconfigures, link-recovery retries, non-JWT slots, and backoff reconnects and breaker probes when no slot is waiting.
Testing
test_mqtt_connection_policy).heltec_v4_repeater_observer_mqtt,Heltec_v3_repeater_observer_mqtt.observer-firmware-dev, both images with the same scratch hooks (300 s token lifetime for custom slots, clock-back and NTP-sync commands; not in this PR). Slots 2 and 5 pointed at two local mosquitto brokers, slots 1, 3 and 4 left on their presets:token renewal waiting for the handshake guard, kept publishing, bounced 15 s after the firstreconnect queued; bounces 15 s apart, none skippedAll five slots stayed ok on both images and heap was unchanged across the runs.
Not tested: more than two slots colliding, TLS brokers on the renewal path (the local brokers were plain TCP), the yield of other slots' backoff reconnects while a slot waits, a staged token aging before its turn, V3, Station G3 and Ethernet boards. The host tests simulate the bridge's order of operations; they do not run
MQTTBridge.cpp.This and #98 both change the backoff-reconnect block in
MQTTBridge.cpp; the second to merge needs a conflict resolved.