Skip to content

api,cli: report task residency, actor UID, and outcome - #556

Draft
erain wants to merge 1 commit into
google:mainfrom
erain:task-residency-outcome
Draft

erain wants to merge 1 commit into
google:mainfrom
erain:task-residency-outcome

Conversation

@erain

@erain erain commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

ax get shows only a task's phase, and the phase can't tell the cases operators ask about apart. A task whose actor Substrate marked crashed reads Failed, the same as a task whose template AX could not create. A reverting actor reads Resuming. Nothing in TaskStatus names the actor in Substrate's own telemetry, so ax-server records can't be joined with Substrate logs and events. Part of #13; see also #346 and #502.

This PR adds three fields to TaskStatus and shows them in the CLI:

  • residency: where the task's actor is in its Substrate lifecycle. The values are queued (no actor yet), resuming, running, suspending, suspended, reverting, crashed, deleting, and deleted (the actor existed and is gone).
  • actorUid: the actor's Substrate UID, the ate.actor.uid on Substrate's logs and events. It changes if Substrate recreates the actor.
  • outcome: how the command ended: succeeded, failed, or cancelled. Nothing sets it yet. Command exit detection (Task reports phase=Running and Ready=True indefinitely after its command has exited #346) will, and that change only needs to write the field.

ax get tasks gains RESIDENCY and OUTCOME columns, and ax describe task gains Residency, Outcome, and Actor UID. phase is unchanged.

How the values are kept

Since #515, AX reads everything about the actor from Substrate on every get and list, and stores none of it. residency and actorUid are a deliberate exception. Planned state-change events (#13) will wrap the task store and emit an event whenever a write changes phase, residency, or outcome, so AX's own transitions have to show up as stored changes.

  1. CreateTask stores queued before it provisions the actor.
  2. The reconciler records the UID and residency from the Actor that CreateActor, SuspendActor, and ResumeActor return. SuspendActor now returns the actor; its response always carried one.
  3. DeleteTask stores deleting together with phase Terminating.
  4. Reads still overlay the live actor, so they never show a stale value. A wake by the router (AX keeps saying a task is Suspended after a request wakes it #502) shows running on the next get, and an actor deleted outside AX shows deleted. When Substrate can't be reached, phase is Unknown and residency keeps its last recorded value.

Keeping the stored residency in step with changes made outside AX, such as router wakes, is separate follow-up work and is not part of this PR.

Validation

  • Unit and integration: go test ./.... The new tests cover:
    • the residency and UID overlay on reads (crashed, reverting, a recreated actor's new UID, deleted outside AX, Substrate unreachable);
    • the sequence of residencies stored across create → resume → suspend → delete;
    • SuspendActor returning the actor;
    • the CLI table and describe output.
  • Postgres store: the Postgres-backed tests, plus a new round-trip test for the two stored fields, pass against postgres:17-alpine.
  • Live E2E: the Kind suite now also checks ax get tasks and ax describe task for residency, outcome, and an actor UID equal to the Substrate actor's.
    • Setup: I ran the full suite against a GKE 1.37 cluster with Agent Substrate v1.0.0-rc.1, installed from its release images, a 4-worker gVisor pool, and ax-server and the e2e task runner built from this branch.
    • Result: all 8 tests passed on a clean run. The first run lost its kubectl port-forward to ax-server partway through, and every later test then failed to connect. ax-server neither restarted nor logged errors, and all eight tests passed when rerun.

In the recording, a script drives the CLI against that cluster and fails on any unexpected state. It creates a task, resumes it, and reads the actor UID. It then deletes the worker pod hosting the actor: the task reads Failed with residency crashed. Resuming reverts the actor to running, and suspending gives suspended. The script and its setup notes are on a separate branch of my fork.

ax get and ax describe across create, resume, a crashed actor, and suspend

This change was written with help from an AI coding assistant.

Add residency, actor_uid, and outcome to TaskStatus and show RESIDENCY and
OUTCOME in `ax get tasks`, and Residency, Outcome, and Actor UID in
`ax describe task`.

Residency names where the task's actor is in its Substrate lifecycle:
queued, resuming, running, suspending, suspended, reverting, crashed,
deleting, or deleted. Unlike phase, it tells a crashed or reverting actor
apart from a task AX failed to provision. actor_uid is the actor's
ate.actor.uid, which Substrate's logs and telemetry carry.

The reconciler records both from the Actor that CreateActor, SuspendActor,
and ResumeActor return, and the store keeps them, so each change AX makes
is a stored change that state-change telemetry can observe. Reads still
overlay the live actor, so a wake by the router or a recreated actor shows
up on the next get. Phase is unchanged.

outcome is defined but not set yet; command exit detection (google#346) will
set it to succeeded, failed, or cancelled.
@erain
erain force-pushed the task-residency-outcome branch from 2bf0e6b to 05bda21 Compare November 10, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant