robustness: reproduce #20221

[henrybear327]

Targeting bug fix #20229

Before the bug fix, the watch created with a future revision with progress notification enabled, can run into the situation where after receiving a progress notification, the watch receives events with revisions lowered than what the watch initial requested for.

    logger.go:146: 2026-03-03T21:26:18.977+0100 ERROR   Broke watch guarantee   {"guarantee": "reliable", "client": 19}
  []model.PersistedEvent{
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key3", Value: model.ValueOrHash{Value: "221"}},
+               Revision: 187,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key1", Value: model.ValueOrHash{Value: "219"}},
+               Revision: 188,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "220"}},
+               Revision: 188,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key14", Value: model.ValueOrHash{Value: "222"}},
+               Revision: 189,
+       },
+       {Event: model.Event{Type: "delete-operation", Key: "key0"}, Revision: 189},
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key0", Value: model.ValueOrHash{Value: "223"}},
+               Revision: 190,
+               IsCreate: true,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key3", Value: model.ValueOrHash{Value: "224"}},
+               Revision: 191,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key13", Value: model.ValueOrHash{Value: "225"}},
+               Revision: 192,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "226"}},
+               Revision: 193,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key0", Value: model.ValueOrHash{Value: "227"}},
+               Revision: 194,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key15", Value: model.ValueOrHash{Value: "228"}},
+               Revision: 195,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key20", Value: model.ValueOrHash{Value: "229"}},
+               Revision: 196,
+               IsCreate: true,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "231"}},
+               Revision: 197,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key3", Value: model.ValueOrHash{Value: "232"}},
+               Revision: 198,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key14", Value: model.ValueOrHash{Value: "233"}},
+               Revision: 198,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key13", Value: model.ValueOrHash{Value: "230"}},
+               Revision: 199,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key17", Value: model.ValueOrHash{Value: "235"}},
+               Revision: 200,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "236"}},
+               Revision: 200,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key18", Value: model.ValueOrHash{Value: "237"}},
+               Revision: 201,
+               IsCreate: true,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key17", Value: model.ValueOrHash{Value: "234"}},
+               Revision: 202,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key15", Value: model.ValueOrHash{Value: "238"}},
+               Revision: 203,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key15", Value: model.ValueOrHash{Value: "239"}},
+               Revision: 204,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key14", Value: model.ValueOrHash{Value: "240"}},
+               Revision: 205,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "241"}},
+               Revision: 206,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key15", Value: model.ValueOrHash{Value: "242"}},
+               Revision: 207,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key13", Value: model.ValueOrHash{Value: "243"}},
+               Revision: 208,
+       },
+       {Event: model.Event{Type: "delete-operation", Key: "key14"}, Revision: 208},
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key15", Value: model.ValueOrHash{Value: "244"}},
+               Revision: 208,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key1", Value: model.ValueOrHash{Value: "245"}},
+               Revision: 208,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key21", Value: model.ValueOrHash{Value: "248"}},
+               Revision: 209,
+               IsCreate: true,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key20", Value: model.ValueOrHash{Value: "246"}},
+               Revision: 210,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key0", Value: model.ValueOrHash{Value: "247"}},
+               Revision: 210,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "249"}},
+               Revision: 211,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key13", Value: model.ValueOrHash{Value: "250"}},
+               Revision: 212,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key10", Value: model.ValueOrHash{Value: "251"}},
+               Revision: 213,
+       },
+       {Event: model.Event{Type: "delete-operation", Key: "key20"}, Revision: 213},
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key21", Value: model.ValueOrHash{Value: "252"}},
+               Revision: 213,
+       },
+       {Event: model.Event{Type: "delete-operation", Key: "key10"}, Revision: 214},
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key18", Value: model.ValueOrHash{Value: "254"}},
+               Revision: 214,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key21", Value: model.ValueOrHash{Value: "253"}},
+               Revision: 215,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key17", Value: model.ValueOrHash{Value: "255"}},
+               Revision: 216,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key0", Value: model.ValueOrHash{Value: "256"}},
+               Revision: 217,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key22", Value: model.ValueOrHash{Value: "257"}},
+               Revision: 218,
+               IsCreate: true,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key22", Value: model.ValueOrHash{Value: "258"}},
+               Revision: 219,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key23", Value: model.ValueOrHash{Value: "260"}},
+               Revision: 220,
+               IsCreate: true,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key1", Value: model.ValueOrHash{Value: "261"}},
+               Revision: 221,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key13", Value: model.ValueOrHash{Value: "259"}},
+               Revision: 222,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key18", Value: model.ValueOrHash{Value: "262"}},
+               Revision: 223,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key23", Value: model.ValueOrHash{Value: "263"}},
+               Revision: 223,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key22", Value: model.ValueOrHash{Value: "264"}},
+               Revision: 223,
+       },
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key3", Value: model.ValueOrHash{Value: "265"}},
+               Revision: 224,
+       },
+       {Event: model.Event{Type: "delete-operation", Key: "key18"}, Revision: 225},
+       {
+               Event:    model.Event{Type: "put-operation", Key: "key13", Value: model.ValueOrHash{Value: "266"}},
+               Revision: 225,
+       },
+       {Event: model.Event{Type: "delete-operation", Key: "key1"}, Revision: 225},
  }
    logger.go:146: 2026-03-03T21:26:18.977+0100 ERROR   Watch validation failed {"duration": "317.958µs", "error": "broke Reliable - a sequence of events will never drop any subsequence of events; if there are events ordered in time as a < b < c, then if the watch receives events a and c, it is guaranteed to receive b"}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.61%. Comparing base (83fa5ca) to head (565e41a).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

see 56 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #21367      +/-   ##
==========================================
- Coverage   68.49%   65.61%   -2.88%     
==========================================
  Files         428      428              
  Lines       35291    35291              
==========================================
- Hits        24172    23156    -1016     
- Misses       9723    10729    +1006     
- Partials     1396     1406      +10     

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83fa5ca...565e41a. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[henrybear327]

Markdown issues fixed. CI should now be passing.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: henrybear327, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [serathius]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[henrybear327]

#17737 (comment) could help lower the reproduction time due to the fact that BlackholeUntilSnapshot is taking quite a long time to execute (our blackhole is leaking traffic)