Skip to content

fix(mqtt): hold token renewal bounces for the cross-slot guard - #99

Merged
agessaman merged 8 commits into
observer-firmware-devfrom
fix/mqtt-renewal-guard
Nov 12, 2026
Merged

agessaman merged 8 commits into
observer-firmware-devfrom
fix/mqtt-renewal-guard

Conversation

@agessaman

@agessaman agessaman commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Problem

The 15 s cross-slot guard keeps two TLS handshakes from overlapping (one handshake needs about 35 KB of internal heap, and boards without PSRAM run with 40-70 KB free). Two paths started handshakes without consulting it:

  • JWT renewal bounce. When a token neared expiry on a broker that enforces it, maintainSlotConnection() soft-disconnected and reconnected at once, even if another slot had started a handshake in the same pass or within the guard window. It armed the guard only afterwards.
  • Corrected clock. After an NTP correction, syncTimeWithNTP() reconnected or bounced every slot with a stale token in one loop, back to back, and never armed the guard.

Fix

Both paths now take a turn through the guard.

  • Renewal. If a bounce is needed while the guard is busy, the slot waits and nothing is touched: no token minted, throttle not stamped, no disconnect. The session keeps publishing. Once the guard is idle it mints, bounces and arms the guard. With the guard idle, a renewal runs in the same pass as before, and a renewal that needs no bounce never waits.
  • Corrected clock. The loop still mints and stages credentials but only marks the slot (clock_bounce_owed). Each marked slot is served in maintainSlotConnection(), one per guard window. If its staged token has aged into the renewal buffer by then, it is re-minted first, so there is one handshake with fresh credentials.
  • No starvation. A backoff reconnect or breaker probe that is rejected locally (no handshake launched) restores the guard to what it was. While a slot is waiting, other slots' backoff reconnects and breaker probes yield to it.

The decisions are pure functions in MQTTConnectionPolicy.h (renewalStep, owedBounceStep, bounceHandshakeAllowed, owedDispatchNeedsMint). New state is bridge-task only: two bools per slot and one mask byte.

Worst-case wait for a renewal is about 90 s with five slots, against the 300 s renewal lead of every shipped preset. A custom token lifetime of 600 s or less (60 s lead) can expire while waiting; the broker then closes the session and the reconnect goes out with a fresh token.

Unchanged: first setups, reconfigures, link-recovery retries, non-JWT slots, and backoff reconnects and breaker probes when no slot is waiting.

Testing

  • 765 native tests pass (15 new in test_mqtt_connection_policy).
  • Builds: heltec_v4_repeater_observer_mqtt, Heltec_v3_repeater_observer_mqtt.
  • Heltec V4 A/B against observer-firmware-dev, both images with the same scratch hooks (300 s token lifetime for custom slots, clock-back and NTP-sync commands; not in this PR). Slots 2 and 5 pointed at two local mosquitto brokers, slots 1, 3 and 4 left on their presets:
Test Without the fix With the fix
Two renewals due in one guard window bounces 8 s apart second slot logged token renewal waiting for the handshake guard, kept publishing, bounced 15 s after the first
Clock set back 2 h, then NTP sync bounces 1 s apart both logged reconnect queued; bounces 15 s apart, none skipped

All five slots stayed ok on both images and heap was unchanged across the runs.

Not tested: more than two slots colliding, TLS brokers on the renewal path (the local brokers were plain TCP), the yield of other slots' backoff reconnects while a slot waits, a staged token aging before its turn, V3, Station G3 and Ethernet boards. The host tests simulate the bridge's order of operations; they do not run MQTTBridge.cpp.

This and #98 both change the backoff-reconnect block in MQTTBridge.cpp; the second to merge needs a conflict resolved.

A Wi-Fi drop cut every slot's session, and the link-recovery retry advanced
each slot's backoff level. Sessions shorter than the 120 s stable clear never
reset it, so a node whose link flapped every couple of minutes walked every
slot up to the 300 s rung and toward the breaker while the broker was fine.

At the link-down edge, before live clients are closed, a slot that was still
connected or whose session ended within 20 s of the edge (either side, since
the socket usually fails before the polled edge) is marked as a link-caused
outage. Its recovery retry then holds the ladder where it was. Slots already
failing before the drop, and any drop when the link had been up under 30 s,
advance as before. Refusals, handshake failures and quick drops while the
link stays up are unchanged.
The 10 s status poll reports the link-down edge after the socket has
already died, so measuring the floor to the edge could excuse a link
that really held for less than kLinkFlapFloorMs. Use the outage stamp
when the slot has one inside the current link-up period.
A JWT renewal bounce and the corrected-clock reconnects started TLS
handshakes without checking the cross-slot guard, so they could overlap
another slot's handshake. A renewal that needs a bounce now waits, before
minting or disconnecting, until the guard is idle and no handshake was
launched in the pass. The corrected-clock path stages the fresh token and
marks the slot owed; maintainSlotConnection() runs each owed reconnect
through the same guard, one per window. Both arm the full guard as before.
…queued tokens

A reconnect or breaker probe rejected locally armed the cross-slot guard
without starting a handshake; with the backoff rolled back, a slot failing
every 10 s could hold a waiting renewal off indefinitely. The guard now
reverts to the last real handshake on a local rejection, and while any
slot's renewal or corrected-clock bounce is waiting, other slots' backoff
reconnects and probes stand aside so it gets the next idle window.

A queued corrected-clock reconnect now re-mints first when its staged token
has expired or aged into the renewal buffer, so it presents fresh
credentials and is not followed by a second renewal bounce.
A slot still marked connected at the polled link-down edge has no outage
stamp, so the floor measured the polled edge as uptime. Require the poll
slack (10 s) in that case; a stamped session end keeps the 30 s floor.
…ttempt

After a link outage a disconnected JWT slot can reconnect through the renewal
or corrected-clock path instead of the backoff path. That left the link-retry
flag set, so if the handshake failed before CONNACK the next backoff retry was
treated as link recovery a second time. Clear the flag once the reconnect is
accepted; a local rejection keeps it.
…pted

The renewal and corrected-clock paths cleared the link-retry flag but left the
outage attribution set. A second link-up edge without a new link-down edge
re-armed the retry and the stale attribution held the ladder again.
@agessaman
agessaman merged commit 34e658e into observer-firmware-dev Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant