Repository navigation
fix(mqtt): stop link outages from climbing the reconnect ladder - #98
Merged
Merged
Conversation
A Wi-Fi drop cut every slot's session, and the link-recovery retry advanced each slot's backoff level. Sessions shorter than the 120 s stable clear never reset it, so a node whose link flapped every couple of minutes walked every slot up to the 300 s rung and toward the breaker while the broker was fine. At the link-down edge, before live clients are closed, a slot that was still connected or whose session ended within 20 s of the edge (either side, since the socket usually fails before the polled edge) is marked as a link-caused outage. Its recovery retry then holds the ladder where it was. Slots already failing before the drop, and any drop when the link had been up under 30 s, advance as before. Refusals, handshake failures and quick drops while the link stays up are unchanged.
The 10 s status poll reports the link-down edge after the socket has already died, so measuring the floor to the edge could excuse a link that really held for less than kLinkFlapFloorMs. Use the outage stamp when the slot has one inside the current link-up period.
A slot still marked connected at the polled link-down edge has no outage stamp, so the floor measured the polled edge as uptime. Require the poll slack (10 s) in that case; a stamped session end keeps the 30 s floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every Wi-Fi outage pushed every MQTT slot one level up the reconnect backoff ladder, even though each reconnect after link-up succeeded.
The level rises just before each due reconnect and is cleared only by a session that stays up for 120 s. The link-recovery retry (
advanceBackoffForLinkRecovery) could not trip the breaker, but it still added a level. A node whose link dropped every couple of minutes therefore walked its slots to level 5 and toward the breaker with a healthy broker. On a Heltec V4, ten drops 150 s apart took one slot to level 5 withfailures at max: 2.Fix
At the link-down edge, before the bridge closes the live clients,
MQTTConnectionPolicy::outageCausedByLink()marks a slot whose outage belongs to the link:The link-recovery retry of a marked slot keeps its level and failure count where they were. The
reconnectinglog line adds, link outage: held.Unchanged on purpose:
Cost: one byte per slot and four bytes on the bridge, all written by the bridge task.
Testing
test_mqtt_connection_policy).heltec_v4_repeater_observer_mqtt,Heltec_v3_repeater_observer_mqtt.fbfb35d23, before the last commit added the 10 s poll allowance:Heap was flat across the 10 drops (126 K free, 115 K largest block after).
Not tested: Heltec V3, Station G3 and Ethernet boards on hardware; a session that ends more than 20 s before Wi-Fi reports the drop (it behaves as before); the 120 s clear of levels raised by flapping was not observed directly.