You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Recover interrupted checkpoint operations and allow failed checkpoint cleanup #2746
Follow-up from the review of #2742, separate from revision GC scheduling in #2744.
A checkpoint that reaches FAILED still retains prepared_revision, but BeginDeleteAgentInstanceCheckpoint only accepts READY and DELETING. The public delete path therefore reports NotFound for the failed record instead of cleaning it up. Retiring the template and deleting its source instance cannot release that revision.
An interrupted CREATING reservation is more disruptive: it retains the revision and blocks task admission and source-instance lifecycle changes. Retrying the original create request can resume it, but there is no automatic recovery when the caller disappears. A failed compensation can also leave the reservation in CREATING.
Implement recovery in the checkpoint lifecycle owner:
Allow failed checkpoints to be deleted through the public API, including the case where tag creation never succeeded or compensation already removed the tag.
Resume or compensate abandoned creation and deletion after restart. Coordinate recovery with in-flight create requests so a slow, live tag copy cannot be reaped solely because it is old.
Keep revision/context references until the lifecycle owner has established that cleanup is complete. Revision GC must continue to count every checkpoint row; filtering out FAILED or CREATING would bypass the safety invariant.
Preserve request idempotency, tag identity checks, and fork-reference protection. Do not hold database locks across Substrate calls.
Acceptance coverage: PostgreSQL and real Substrate tests for failed tag creation, failed compensation, lost responses, and restart between reservation/tag creation/finalization; deleting a failed checkpoint must release its references, abandoned reservations must stop blocking their source instance, and a concurrent live create or fork must remain safe.
I would like to take the remaining interrupted-checkpoint recovery work in #2746. #3199 already covers failed-checkpoint deletion; I will leave that behavior intact.
The follow-up will coordinate live Create/Delete requests and a restart-safe recovery worker through durable operation ownership, resume or safely compensate abandoned CREATING/DELETING operations, and preserve request idempotency, tag UID checks, fork protection, and revision/history retention. A slow live tag copy must not be reclaimed just because it is old, and database transactions will not span Substrate calls.
I will open a separate draft PR against main with PostgreSQL concurrency/restart coverage and Substrate E2E coverage. This is independent of my existing #3213 and #3302 PRs. Please let me know if someone is already taking the remaining scope.
Activity
fix: safely collect runtime revisions in a leader-elected runnable (#…
fix(checkpoint): allow failed checkpoint cleanup (kagent-dev#3199)
Jacob953 commentedon Nov 4, 2026
I would like to take the remaining interrupted-checkpoint recovery work in #2746. #3199 already covers failed-checkpoint deletion; I will leave that behavior intact.
The follow-up will coordinate live Create/Delete requests and a restart-safe recovery worker through durable operation ownership, resume or safely compensate abandoned CREATING/DELETING operations, and preserve request idempotency, tag UID checks, fork protection, and revision/history retention. A slow live tag copy must not be reclaimed just because it is old, and database transactions will not span Substrate calls.
I will open a separate draft PR against main with PostgreSQL concurrency/restart coverage and Substrate E2E coverage. This is independent of my existing #3213 and #3302 PRs. Please let me know if someone is already taking the remaining scope.