Skip to content

fix(session): allow explicit deletion after task settlement failure - #3213

Open
Jacob953 wants to merge 9 commits into
kagent-dev:mainfrom
Jacob953:fix/session-unpublished-boundary
Open

Jacob953 wants to merge 9 commits into
kagent-dev:mainfrom
Jacob953:fix/session-unpublished-boundary

Conversation

@Jacob953

@Jacob953 Jacob953 commented Oct 15, 2026 •

Copy link
Copy Markdown

When a runtime saves a terminal or waiting task boundary but its SettleTask acknowledgement never commits, the Session rejects further work and lifecycle operations, including deletion. This PR allows explicit deletion through the existing durable Actor deletion workflow.

Partially addresses #3010, case B, using the explicit-deletion alternative. Deletion discards the pending result; it does not recover the original conversation or publish its unconfirmed task boundary.

Behavior

  • Share the runtime admission query across deletion, dispatch, task writes and checkpoint creation. Explicit deletion may proceed despite unpublished events. Live dispatch reservations, claimed quiescence work and checkpoint creation remain blockers. Deletion intent survives backend failures for retry.
  • Serialize deletion admission and settlement under the Session row lock. If deletion starts first, late acknowledgement of an unpublished boundary returns ErrConflict (Aborted over gRPC), including after deletion preparation fails. It cannot publish task state/history or enqueue push notifications. Already published acknowledgements remain idempotent until final deletion.
  • Keep idle expiration blocked by unpublished events. Retain upstream's expiration rules for settled INPUT_REQUIRED and AUTH_REQUIRED tasks. No API or schema changes are introduced.

Validation

Synced with upstream main at d9cc3a82 and rechecked on October 29. Updated the regression fixtures for the current Substrate protobuf imports, explicit Actor JWT audience, and push-delivery send timeout; the explicit-deletion behavior is unchanged.

  • Full PostgreSQL database and migration race suites passed, including inline SQL preparation. Focused expiration, waiting-boundary and deletion/settlement ordering tests also passed.
  • Full Session service and gRPC race suites passed. Packages were run with -p=1. The gRPC suite required an isolated retry after local Docker reaper-startup and PostgreSQL port-mapping failures; no tests were excluded.
  • The Go ADK settlement-outage integration test now uses signed Actor JWTs and the production runtime authentication provider. It exercises the production Go ADK executor, A2A, TaskStore gRPC and PostgreSQL with an in-process model and test OIDC issuer. An injected settlement outage leaves private completed/public working state; explicit deletion is admitted while settlement is unavailable and late settlement is rejected. The fixture models egress credential replacement; it does not run a real egress gateway.
  • make -C go lint passed with 0 issues. The E2E package and all four affected packages compiled against the synced head.

From go/, with Docker and the repository's envtest assets configured:

go test -race -count=1 ./core/internal/database ./core/pkg/migrations
go test -race -count=1 ./core/internal/service/session ./core/internal/grpcserver

November 4 compatibility validation

The authenticated Go ADK fixture now sends the SDK's current A2A protocol version explicitly. It observes the injected failed settlement boundary instead of assuming four retries or cleanup within the request's 15-second deadline. Availability is restored only after DELETE admission, and late acknowledgement must remain Aborted. This is a test-only update; deletion semantics are unchanged.

The focused Go ADK race test passes on this branch. Combined with current main and #3519 in a temporary checkout, the full database, checkpoint, Session service and gRPC race suites pass. One combined checkpoint-suite attempt failed before PostgreSQL started because of a local Testcontainers reaper timeout; the isolated full retry passed, without excluding tests.

E2E scope and pending review

The current TestSessionDeletionStopsLiveRuntime covers real Actor deletion during a blocked model request, loss of a successful public deletion reply, and rejection of sends to the deleted context. It uses the shared harness matrix. It no longer stages private TaskStore state using unsigned Actor headers, which upstream authentication now rejects. Unpublished-boundary fault injection remains covered by the authenticated gRPC integration and PostgreSQL-backed workflow tests.

The current E2E test was compiled but not run against a live cluster. Earlier live unpublished-boundary results predate the JWT authentication change and do not validate this head. Upstream CI Build and Migration Immutability require maintainer approval to run. The harness-matrix run remains pending, as does maintainer agreement on allowing explicit deletion to supersede an unpublished boundary with the retained lifecycle guards.

AI assistance: OpenAI Codex assisted with implementation, testing and this PR description.

Keep explicit deletion available after a runtime loses its settlement callback. Preserve dispatch, checkpoint and claimed lifecycle exclusions. Verify failed deletion and a lost response after deletion with PostgreSQL-backed workflow tests.

Refs kagent-dev#3010. This restores deletion, not in-place conversation recovery.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
Reproduce exhausted Go ADK settlement retries through gRPC and PostgreSQL. Preserve deletion write exclusions and document the separate recovery gap for abandoned quiescence claims. Link the shared tracking issues and state the conditions required before relaxing recovery guards.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
Preserve idle expiration cleanup checks while allowing the explicit deletion escape for kagent-dev#3010 B. Reject late publication after deletion admission, including preparation failure, and retain idempotent published acknowledgements. Cover all boundary states, initial and updated boundaries, failure retries and settlement/deletion ordering.

Automatic publication and same-session recovery remain outside this deletion fix. AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added the bug Something isn't working label Oct 15, 2026
Use PostgreSQL clock cutoffs in the TTL regression to avoid host/container skew. Add shared E2E coverage for deleting staged terminal and waiting boundaries while a real Actor is running, including a lost public reply and rejected stale writes. Document the harness selectors.

Validated locally with the Go ADK harness on Kind/Substrate/gVisor, full affected race suites, 100 TTL repetitions, and Go lint. AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@Jacob953
Jacob953 marked this pull request as ready for review October 15, 2026 06:29
@Jacob953
Jacob953 requested a review from a team as a code owner October 15, 2026 06:29
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 15, 2026
@Jacob953
Jacob953 marked this pull request as draft October 15, 2026 07:02
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 15, 2026
@Jacob953
Jacob953 marked this pull request as ready for review October 15, 2026 07:19
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 15, 2026
Use one admission query for lifecycle operations, dispatch, task writes and
checkpoint creation. Allow unpublished boundaries only for explicit deletion,
while retaining claimed quiescence and live dispatch protections.

Remove fork tracking and unrelated cancellation references from source comments.
Verify explicit deletion still rejects live dispatch and checkpoint creation.

Refs kagent-dev#3010.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 16, 2026
Wait for the executor cleanup callback instead of counting telemetry flushes.
Join the test A2A server and check its exit error. Assert the precise public
task state after settlement retries fail.

Clarify the deletion fault fixture's effect ordering and verify its lifecycle
interface at compile time. Tighten comments and document the limit on
settlement acknowledgement idempotence after deletion.

Refs kagent-dev#3010.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 16, 2026
@Jacob953
Jacob953 requested a review from moezdil October 19, 2026 02:38
Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 23, 2026
Adapt the regression fixtures to Substrate API v1 imports, the explicit runtime JWT audience, and push-delivery send timeouts.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot removed the bug Something isn't working label Oct 29, 2026
Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Nov 4, 2026
@Jacob953
Jacob953 force-pushed the fix/session-unpublished-boundary branch from dc53728 to 3a60eca Compare November 4, 2026 13:05
@EItanya

EItanya commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Sorry for taking so long to get to this review, is this PR still necessary in light of #3530

@EItanya

EItanya commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

🤖 Comment left by Claude on behalf of @EItanya.

Following up on my question: I think #3530 covers this. On current main, DeleteSession with force: true skips the settled-runtime check, so a Session whose SettleTask acknowledgement never arrived can still be deleted. Finishing the deletion then publishes the saved but unacknowledged results (publishSavedSessionResults) rather than discarding them, and a concurrent late acknowledgement remains harmless. TestForceDeletionPreservesSavedBoundary and TestForceDeletionPreservesConcurrentRuntimeSettlement cover this case. Normal deletion intentionally still requires settlement, and force is the escape hatch.

The remaining way a Session can be blocked is a stranded CREATING checkpoint, which blocks even force deletion. #3519 is addressing that. Unless you see a case main doesn't handle, I'd suggest closing this one. Thanks for the thorough work here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants