[selekkala]

Problem

During FRR failback from the backup path to the primary path, traffic experienced a transient loss even though the primary link, LAG forwarding, and L3 adjacency were already ready. The affected route uses a GR2 protection next-hop group (NHG) with a primary member and a standby member.

GR2 autonomous protection reversion depends on that protection NHG remaining programmed. The ASIC can switch between the primary and standby members only while the protection object and its standby relationship exist in hardware.

Root cause

The route manager derives the SAI NHG key from the currently resolved next-hop set. During primary resolution changes, the resolved set changes from primary plus backup to backup-only and then back to primary plus backup. Those two resolved sets produce different NHG keys.

The previous FBOSS behavior therefore treated a temporary primary resolution loss or recovery as a new NHG topology:

1. The old protection NHG was dereferenced and its standby child group and members were removed.
2. A new outer protection NHG was created.
3. The standby member and then the primary member were added to the new group.
4. The route was updated to use the replacement group.

This is a break-before-make sequence. While the old group had been removed and the replacement group was not yet complete, GR2 had no complete protection NHG on which to perform autonomous protection or reversion. Packets arriving during that programming gap were dropped.

The observed event order was:

• 22:28:57.565424 - primary link operational
• 22:28:58.571575 - LAG forwarding restored
• 22:28:58.584416 - L3 adjacency ready
• 22:28:58.597292 - old standby member removed
• 22:28:58.597764 - old child NHG removed
• 22:28:58.598314 - old outer NHG removed
• 22:28:58.602510 - new outer NHG created
• 22:28:58.604566 - new standby member added
• 22:28:58.606073 - new primary member added

The traffic hit was approximately 8.8 ms, with the outer protection NHG absent for approximately 4.2 ms. SAI operations completed successfully; the issue was the non-atomic FBOSS object replacement, not an SAI return-code failure.

Fix

• Store the resolved next-hop identity and group type on SaiNextHopGroupHandle when the group is created.
• During a protection-route update, obtain the client next-hop set as the source of truth for the intended primary and backup topology.
• Reuse the existing protection NHG and route next-hop handle when:
  • the client primary is unchanged;
  • the backup set is unchanged and non-empty; and
  • the new resolved set differs only because the primary is temporarily unresolved.
• Continue to use the existing managed next-hop subscriptions so the primary member can disappear and be recreated as neighbor resolution changes, without deleting the protection NHG itself.
• Create a new NHG for intentional topology changes, including primary removal or replacement, backup changes, or a missing backup path.

This preserves the GR2 protection object across transient resolution changes while preventing stale primary or backup paths from being retained after an intentional route update.

Tests

Added focused route-manager tests covering:

• reuse of the protection NHG across primary resolution loss and recovery;
• rejection of reuse when the client removes the primary; and
• rejection of reuse when the backup path changes.

Validation

• git diff --check passed.
• The GR2/G200 forwarding-stack build passed after the implementation correction with zero compilation failures.

[selekkala]

GR2 SAI failback validation

The live G200/GR2 failback was cross-checked against the SAI replay log. The
operations occurred in make-before-break order:

19:35:20.467018  create_next_hop_group                 rv: 0
19:35:20.467798  create backup next-hop-group member   rv: 0
19:35:20.469484  create primary next-hop-group member  rv: 0
19:35:20.471818  set MySID next-hop-group              rv: 0
19:35:20.473482  remove old next-hop-group member      rv: 0
19:35:20.473997  remove old next-hop-group             rv: 0

The replacement protection NHG and both members were programmed before the old
NHG was released. No SAI operation failed; all return values were rv: 0.
The final MySID state retained both the resolved primary and the backup path.
SAI online diagnostic packet loss remained 0, and the primary aggregate-port
output counters increased while traffic was running.

The focused route-manager test was strengthened in commit 302e4567c7 to
verify that the SAI top-level protection group and child hardware-protection
group remain present and unchanged across primary resolution loss and recovery.
The negative cases for intentional primary removal and backup changes remain
covered.

Local execution of the targeted test is blocked on the validation host because
the repository's buck2 wrapper requires the dotslash executable, which is
not installed there. git diff --check passed; CI should run the targeted
//fboss/agent/hw/sai/switch/tests:route_manager_test target.

[selekkala]

Updated the UT in commit 0da63bbfbe to verify the SAI protection-group member state across resolution changes:

• initial state: protection group has primary and backup members;
• primary unresolved: the same protection group remains and only the primary member is absent;
• primary restored: the same protection group has both members again.

The existing negative tests for intentional primary removal and backup changes remain unchanged.