Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions docs/orchestration-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,55 @@ this run mode; ran in the foreground.`. `STEP_DISABLE_SUBAGENT_BACKGROUND=1` or
`[subagent] background = false` does the same everywhere, with a matching
notice.

## Background commands and process ownership

`run_command` with `run_in_background: true` starts a detached process tree for
long-lived work such as a dev server. The tree is registered in the process
registry under the calling session, with `kind: "bash"`.

Kind decides whether a tree counts as unfinished work. `subagent` and
`workflow` are the session's own work, so they count. `bash` is a side effect
the user inspects afterwards: a session with a running dev server is still a
finished session, and its row does not read as busy.

Ownership decides when the tree stops:

| Event | Background command |
| --- | --- |
| Switching which session is viewed | keeps running |
| Forking a session | keeps running; the new session starts with none of them |
| Reloading settings or extensions | keeps running |
| The owning session is deleted, replaced, or resumed away | stopped |
| The host exits | stopped |

Stopping goes through three layers, from narrowest to widest:

1. `releaseOwnerGracefully(ownerId)` sends `SIGTERM` to the tree's process
group, then `SIGKILL` after `PROCESS_TREE_TERM_GRACE_MS` (3s). The owning
session calls it from its `session_shutdown` handler. A command the user
interrupted uses the shorter `INTERRUPTED_COMMAND_GRACE_MS` (500ms) instead:
an Escape or an elapsed deadline must look stopped immediately, and the call
still waits for the tree to exit.
2. `releaseAllProcessesSync()` runs from a `process.exit` listener registered
inside the registry module. Every host inherits it, including hosts that do
not write their own shutdown path.
3. A detached parent-death supervisor holds one IPC lease per armed group.
When the channel closes — including when the host is `SIGKILL`ed — the
supervisor signals the group. No in-process handler can cover that case.
Every command arms it, foreground or background: both are spawned detached,
so both would otherwise outlive a killed host. The supervisor is one process
per host, so a foreground command costs a lease, not a process.

The registry also decides what the model and the UI can see. `bash_list` returns
the calling session's own commands with pid, status, and log path; `bash_kill`
stops one or all of them and refuses pids owned by another session. A fork
therefore starts with an empty list even though the parent's commands keep
running.

Log files survive their command: they are the only record of what a dev server
printed before it was stopped, so the registry keeps the `logPath` and does not
delete it.

## Cron scheduling and delivery

`cron_create` accepts numeric five-field local-time cron expressions. Fields
Expand Down
201 changes: 201 additions & 0 deletions packages/coding-agent/src/core/process-registry.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
/**
* Process ownership registry for detached child process trees.
*
* A coding session starts process trees it cannot wait on: subagent RPC children, background
* `run_in_background` commands, and the shells that `run_command` creates. Each of them must be
* reclaimed at a different moment, and the moment depends on *why* the process exists:
*
* - `subagent` and `workflow` are the session's own work. They count as unfinished work, and they
* die with the session that started them.
* - `bash` is a side effect the user inspects (a dev server, a watcher). It must survive a session
* switch or a fork, so it is *not* unfinished work, but it still dies when its session is
* deleted or the host exits.
*
* Three reclamation layers exist, from narrowest to widest:
*
* 1. `releaseOwner(ownerId)` — one session's trees. Used when a session is torn down.
* 2. `releaseAll()` — every tree this process owns. Registered on `process.exit`, so a host that
* never wrote its own shutdown path still reclaims children.
* 3. The parent-death guard in `./parent-death-guard.ts` — an external supervisor for `SIGKILL`,
* which no in-process handler can survive.
*
* Watch out: `trackDetachedChildPid` used to be a bare `Set<number>` in `utils/shell.ts`. Anything
* that still uses it keeps working through the compatibility wrappers exported there, but new code
* must register here so ownership and kind are known.
*/

import { killProcessTree, PROCESS_TREE_TERM_GRACE_MS, terminateProcessTreeGracefully } from "../utils/process-tree.ts";

/** Why a process tree exists. Determines whether it counts as unfinished session work. */
export type ProcessKind = "bash" | "subagent" | "workflow";

/** One registered process tree. */
export interface ProcessRegistration {
/** Process-group leader pid on POSIX; the wrapper's pid on Windows. */
pid: number;
kind: ProcessKind;
/** Session that owns the tree; `HOST_OWNER` for trees outside any session. */
ownerId: string;
/** Human-readable label, bounded by the caller. */
label: string;
/** Working directory the command ran in, when known. */
cwd?: string;
/** Log file capturing output, for background commands. */
logPath?: string;
startedAt: number;
/** True once a reclamation has been requested for this tree. */
releasing?: boolean;
}

/** Owner id for trees a host starts outside a session. */
export const HOST_OWNER = "\0host";

/** Grace window between the tree's SIGTERM and its SIGKILL. */
export const PROCESS_TERM_GRACE_MS = PROCESS_TREE_TERM_GRACE_MS;

/** Kinds that count as unfinished session work. `bash` deliberately does not. */
const UNFINISHED_WORK_KINDS: ReadonlySet<ProcessKind> = new Set<ProcessKind>(["subagent", "workflow"]);

/** Whether a kind represents work the session is still performing. */
export function isUnfinishedWork(kind: ProcessKind): boolean {
return UNFINISHED_WORK_KINDS.has(kind);
}

const registrations = new Map<number, ProcessRegistration>();
const listeners = new Set<(entries: readonly ProcessRegistration[]) => void>();

function notify(): void {
if (listeners.size === 0) return;
const snapshot = [...registrations.values()];
for (const listener of listeners) {
try {
listener(snapshot);
} catch {
// An observer must never break process reclamation.
}
}
}

/** Subscribe to registry changes; returns the unsubscribe function. */
export function subscribeToProcessRegistry(listener: (entries: readonly ProcessRegistration[]) => void): () => void {
listeners.add(listener);
return () => {
listeners.delete(listener);
};
}

/** Every registered tree, newest last. */
export function listProcesses(ownerId?: string): ProcessRegistration[] {
const all = [...registrations.values()];
return ownerId === undefined ? all : all.filter((entry) => entry.ownerId === ownerId);
}

/** Trees for one owner that count as unfinished work. */
export function listUnfinishedWork(ownerId: string): ProcessRegistration[] {
return listProcesses(ownerId).filter((entry) => isUnfinishedWork(entry.kind) && !entry.releasing);
}

/** Register one process tree. Re-registering the same pid replaces the previous record. */
export function registerProcess(input: Omit<ProcessRegistration, "startedAt"> & { startedAt?: number }): void {
if (!Number.isSafeInteger(input.pid) || input.pid <= 0) return;
registrations.set(input.pid, { ...input, startedAt: input.startedAt ?? Date.now() });
notify();
}

/** Drop one tree's record without signalling it (the caller already reaped it). */
export function forgetProcess(pid: number): void {
if (registrations.delete(pid)) notify();
}

/**
* Drop a tree's record without signalling it, but only when the record still describes what the
* caller registered. A pid can be reused after exit, so a late `exit` event from an old child must
* not remove a newer registration.
*/
export function forgetProcessIf(pid: number, matches: (entry: ProcessRegistration) => boolean): void {
const entry = registrations.get(pid);
if (entry && matches(entry) && registrations.delete(pid)) notify();
}

/**
* Kill one owner's trees immediately and forget them.
*
* Synchronous and unbounded: it exists for `process.exit`, where there is no event loop turn left to
* await an escalation. Callers that can wait must use {@link releaseOwnerGracefully} instead, so a
* shell's `trap ... TERM` handler still runs.
*/
export function releaseOwnerProcessesSync(ownerId: string): number {
let released = 0;
for (const [pid, entry] of [...registrations]) {
if (entry.ownerId !== ownerId) continue;
killProcessTree(pid);
registrations.delete(pid);
released += 1;
}
if (released > 0) notify();
return released;
}

/** Kill every registered tree immediately and forget them. For host exit. */
export function releaseAllProcessesSync(): number {
let released = 0;
for (const [pid] of [...registrations]) {
killProcessTree(pid);
registrations.delete(pid);
released += 1;
}
if (released > 0) notify();
return released;
}

/**
* Terminate one owner's trees and wait for the grace window.
*
* **Kind is the caller's decision.** This function releases every registration the owner has,
* whatever its kind, unless `only` narrows it. That is deliberate for session teardown, where all
* of the session's work should stop. It is wrong for a narrower caller: `bash_kill` promises to
* stop background commands, so it must pass the pids it means and leave the owner's subagent trees
* alone. Any new caller that means "one kind" has to say so explicitly:
*
* ```ts
* const pids = listProcesses(ownerId).filter((e) => e.kind === "bash").map((e) => e.pid);
* await releaseOwnerGracefully(ownerId, undefined, new Set(pids));
* ```
*
* `only` narrows pids, this function does not widen them back.
*
* Marks each tree released first, so a concurrent {@link releaseAllProcessesSync} on `process.exit`
* does not signal the same group twice. Returns after the escalation is *sent*, not after every pid
* is observed dead: a tree that ignores `SIGKILL` is not something this process can fix.
*/
export async function releaseOwnerGracefully(
ownerId: string,
graceMs: number = PROCESS_TERM_GRACE_MS,
only?: ReadonlySet<number>,
): Promise<number> {
const pending: Array<Promise<void>> = [];
for (const entry of registrations.values()) {
if (entry.ownerId !== ownerId || entry.releasing) continue;
if (only !== undefined && !only.has(entry.pid)) continue;
entry.releasing = true;
// `keepAlive`: the caller awaits `settled`, so the grace timer must keep the loop alive long
// enough to fire the escalation. An unref'd timer would let Node exit mid-wait.
pending.push(terminateProcessTreeGracefully(entry.pid, graceMs, { keepAlive: true }).settled);
}
if (pending.length === 0) return 0;
notify();
await Promise.all(pending);
for (const [pid, entry] of [...registrations]) {
if (entry.ownerId !== ownerId || !entry.releasing) continue;
if (only !== undefined && !only.has(pid)) continue;
registrations.delete(pid);
}
notify();
return pending.length;
}

// A host that never wrote its own shutdown path still reclaims its children. `prependListener`
// runs this before any other exit handler, including ones that throw.
process.prependListener("exit", () => {
releaseAllProcessesSync();
});
36 changes: 32 additions & 4 deletions packages/coding-agent/src/core/tools/bash.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ import { keyHint } from "../../render/keybinding-hints.ts";
import { truncateToVisualLines } from "../../render/visual-truncate.ts";
import { theme } from "../../theme/theme.ts";
import { waitForChildProcess } from "../../utils/child-process.ts";
import { armParentDeathGuard } from "../../utils/parent-death-guard.ts";
import {
getShellConfig,
getShellEnv,
killProcessTree,
INTERRUPTED_COMMAND_GRACE_MS,
type ShellConfig,
spawnShellChild,
terminateProcessTreeGracefully,
trackDetachedChildPid,
untrackDetachedChildPid,
} from "../../utils/shell.ts";
Expand Down Expand Up @@ -137,24 +139,45 @@ export function createLocalShellOperations(
stderr: "pipe",
});
if (child.pid) trackDetachedChildPid(child.pid);
// The child is spawned detached (its own process group), so it survives a host that dies
// without running its handlers. Arm the same guard a background command uses: one shared
// supervisor per host, so a foreground command costs a lease, not a process. Without this
// a `SIGKILL`ed host leaves the command — and everything it started — running.
const guard = child.pid ? armParentDeathGuard(child.pid, INTERRUPTED_COMMAND_GRACE_MS) : undefined;
void guard?.ready.catch(() => {});
let timedOut = false;
let timeoutHandle: NodeJS.Timeout | undefined;
// Stop and timeout send SIGTERM to the process group and escalate to SIGKILL after a
// grace window, so a shell `trap ... TERM` handler gets to run its own cleanup. The
// escalation is cancelled once the child actually exits; a late SIGKILL must never land
// on a reused pid. Re-arming (timeout then abort) replaces the previous escalation
// instead of leaking it.
//
// A foreground command uses a short window: the user pressed Escape, or the deadline
// already elapsed, so anything longer reads as a hang. The call still waits for the tree
// to exit, which is where a cleanup handler's time actually goes.
let cancelEscalation: (() => void) | undefined;
const stopGracefully = () => {
if (!child.pid) return;
cancelEscalation?.();
cancelEscalation = terminateProcessTreeGracefully(child.pid, INTERRUPTED_COMMAND_GRACE_MS);
};
const onAbort = () => {
if (child.pid) killProcessTree(child.pid);
stopGracefully();
};

try {
// Set timeout if provided.
if (timeoutMs !== undefined) {
timeoutHandle = setTimeout(() => {
timedOut = true;
if (child.pid) killProcessTree(child.pid);
stopGracefully();
}, timeoutMs);
}
// Stream stdout and stderr.
child.stdout?.on("data", onData);
child.stderr?.on("data", onData);
// Handle abort signal by killing the entire process tree.
// Handle abort signal by stopping the entire process tree.
if (signal) {
if (signal.aborted) onAbort();
else signal.addEventListener("abort", onAbort, { once: true });
Expand All @@ -170,6 +193,11 @@ export function createLocalShellOperations(
...(timeout === undefined ? {} : { timeoutSeconds: timeout }),
};
} finally {
// Disarm a pending escalation first: the process is gone, and its pid may be reused.
cancelEscalation?.();
// Release the guard lease too. A command that exited on its own must not be reported
// as a parent death, and the lease must not outlive the pid it names.
void guard?.disarm();
if (child.pid) untrackDetachedChildPid(child.pid);
if (timeoutHandle) clearTimeout(timeoutHandle);
if (signal) signal.removeEventListener("abort", onAbort);
Expand Down
32 changes: 32 additions & 0 deletions packages/coding-agent/src/features/step.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import type {
ExtensionUIContext,
InlineExtension,
} from "../core/extensions/types.ts";
import { releaseOwnerGracefully } from "../core/process-registry.ts";
import { BUILTIN_SLASH_COMMANDS } from "../core/slash-commands.ts";
import type { FeedbackIdentity } from "../step/feedback/types.ts";
import { STEP_INIT_PROMPT } from "../step/init-prompt.ts";
Expand Down Expand Up @@ -346,6 +347,11 @@ export function createStepExtension(options: StepExtensionOptions = {}): Extensi
// A "this session" Full Access grant must not carry into the next
// session; only the persisted acknowledgment does.
fullAccessAcknowledged = false;
// Background commands belong to the session that started them. A session switch or a
// fork never reaches this handler (it keeps the same session object and the old one
// stays open), so reaching here means the session is really going away: release exactly
// its own trees, and leave every other session's dev servers running.
releaseSessionProcesses(ctx);
if (nativeRetryOverridden && nativeRetryPreference !== undefined && ctx.setAutoRetryEnabled) {
try {
ctx.setAutoRetryEnabled(nativeRetryPreference);
Expand Down Expand Up @@ -522,6 +528,32 @@ export function createStepExtension(options: StepExtensionOptions = {}): Extensi
* persisted preset as `initialPreset` would otherwise outrank env aliases in
* Pi's resolver, so only inject it when no policy override is present.
*/
/**
* Release the background process trees one session owns.
*
* Called from `session_shutdown`, which fires only when the session is really going away: `/new`,
* resume, a deleted session, and quit. A session switch or a fork reuses the same session object
* and never reaches this handler, so a dev server the user is watching keeps running while they
* look at another conversation. That is the intended difference between "not visible" and "closed".
*
* Failure is reported, not thrown: a session that cannot stop one tree must still finish shutting
* down, and the host-exit sweep is the backstop.
*/
function releaseSessionProcesses(ctx: ExtensionContext): void {
let ownerId: string;
try {
ownerId = ctx.sessionManager.getSessionId();
} catch {
// A context from an already-replaced session cannot name its owner; the host sweep covers it.
return;
}
if (!ownerId) return;
void releaseOwnerGracefully(ownerId).catch(() => {
// The registry already marked the trees releasing; a failed wait must not surface as an
// unhandled rejection during teardown.
});
}

function resolvePermissionOptions(options: StepExtensionOptions): StepPermissionControllerOptions | undefined {
const explicit: StepPermissionControllerOptions = {
...options.permission,
Expand Down
Loading
Loading