Repository navigation
atelet: Reject stale snapshot generations - #2803
Conversation
| defer t.unlock(actorUID) | ||
| a, err := t.waitForDetachLocked(ctx, actorUID) | ||
| a, err := t.waitForDetachLocked(ctx, actorUID, func(a *actorSnapshots) error { | ||
| return a.checkGeneration(snap) |
There was a problem hiding this comment.
🤖 question 🟢 – The removed TODO aimed for "no commit replaces a snapshot with an older one", but CommitLocal and CommitDurable still don't compare generations. Two Checkpoints of the same actor (say generations 5 and 6) can both pass this check, because holds don't exclude each other and newest() doesn't see an in-flight Checkpoint's target. If generation 5 commits last, LatestLocal and LatestDurable roll back to it, and the prune then deletes generation 6's local directory. Is a concurrent Checkpoint of one actor ruled out elsewhere? If not, CommitLocal and CommitDurable could skip a snapshot older than what they hold, as CommitPromoted now does.
There was a problem hiding this comment.
Good catch. ateapi holds the actor lease during Checkpoint, but if a lease expires or a request times out while an earlier Checkpoint is still running on atelet, a newer one can arrive and overlap.
Updated both entry and commit to handle that:
BeginCheckpointandBeginPromotenow record their snapshot ona.inFlightuntilend()runs, andnewest()includesinFlightso an older generation cannot start while a newer one is running.CommitLocalnow skips snapshots when!a.localNeeded(snap)andCommitDurableskips when!a.uploadNeeded(snap), so a stalled older checkpoint finishing after a newer commit leavesLatestLocalandLatestDurablealone (andPruneLocalCheckpointskeeps the newer local directory).
4bda273 to
23f883c
Compare
Have Checkpoint and PromoteSnapshot reject a snapshot whose generation is below the actor's newest tracked snapshot on the node, or equal to it under a different snapshot UUID, with FailedPrecondition before doing any work. Commits also skip snapshots superseded while a checkpoint or promote ran.
23f883c to
46d5fae
Compare
Have Checkpoint and PromoteSnapshot reject a snapshot whose generation is below the actor's newest tracked snapshot on the node, or equal to it under a different snapshot UUID, with FailedPrecondition before doing any work.
Part of #2069