diff --git a/docs-site/src/content/docs/fr/reference/cli/lifecycle.md b/docs-site/src/content/docs/fr/reference/cli/lifecycle.md index 27b4c6c6cea..fcd71d1fe5d 100644 --- a/docs-site/src/content/docs/fr/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/fr/reference/cli/lifecycle.md @@ -34,6 +34,10 @@ ocx start --socks5-off Arrête le proxy actif à partir de son PID, supprime le fichier de PID et rétablit le fonctionnement natif de Codex. Si un service d’arrière-plan géré est installé, `ocx stop` l’arrête d’abord afin qu’il ne puisse pas relancer le proxy. Le bouton **Stop** du tableau de bord Web exécute la même opération (`POST /api/stop`) sur tous les backends, sauf le Planificateur de tâches Windows : le wrapper peut y relancer le proxy après la fin de la tâche, donc le tableau de bord refuse avec `respawnable_service`, ne modifie rien et vous demande d'exécuter `ocx stop`. +Sur un client hub connecté, `ocx stop`, y compris via l’API, conserve le routage Codex, le catalogue et la configuration Grok appartenant à la connexion ; `ocx start` les dessert de nouveau. L’arrêt explicite d’un Remote Link Child rétablit au contraire, si la restauration partagée réussit, le routage natif de Codex, le catalogue et les métadonnées de reprise sauvegardées, et supprime la configuration Grok gérée. La connexion Home enregistrée reste conservée ; `ocx disconnect` la supprime. + +Pour réutiliser Home depuis Codex après cette restauration, lancez `ocx start` (ou `ocx service start` si un service est installé), puis `ocx sync --restart-codex`. Le démarrage seul ne réapplique pas le routage ; sync ne recrée pas le bloc Grok. Si l’appartenance de la connexion ne peut pas être confirmée avant l’arrêt du service, l’API d’arrêt renvoie HTTP 409 avec `sharedTeardown: "refused"` et laisse le proxy et le gestionnaire de service actifs. Attendez la fin de la connexion ou déconnexion en cours, puis réessayez. + ### `ocx restart` Lorsqu’un proxy est actif, demande précisément au PID et au port dont l’identité a été vérifiée de redémarrer sur place, attend la fin du drainage normal, puis vérifie qu’un autre PID d’exécution écoute sur le même port. Le routage géré et la supervision du service restent installés pendant toute l’opération. Une requête dont l’état est incertain est observée, et non rejouée comme lors d’une séquence distincte d’arrêt et de démarrage. diff --git a/docs-site/src/content/docs/guides/remote-link.md b/docs-site/src/content/docs/guides/remote-link.md index a63d26a25ee..374ac1cf15a 100644 --- a/docs-site/src/content/docs/guides/remote-link.md +++ b/docs-site/src/content/docs/guides/remote-link.md @@ -51,7 +51,11 @@ Connecting restarts OpenCodex on this computer. Codex turns that are already run The Child waits for its configured port while the old process releases it. If a CLI-managed restart still fails, run `ocx start` on the Child and check `~/.opencodex/restart-handoff.log`. In the desktop app, the app starts and supervises the replacement automatically. -`ocx stop` on a Child stops its proxy and leaves Codex routing, the catalog and the Grok config in place, so `ocx start` (or `ocx service start`) brings the Child back unchanged. If `ocx status` on the Child reports that Codex routing is native, run `ocx sync --restart-codex`; `ocx start` does not rewrite a Child's Codex routing. +An explicit `ocx stop` on a Remote Link Child stops its proxy and, when shared restoration succeeds, restores native Codex routing, its catalog and backed-up resume-history metadata and removes the managed Grok configuration. The saved Home connection is kept; use `ocx disconnect` to remove that connection. + +If a hub connection takes ownership while stopping, its routing and configuration are preserved. A connection still being created or disconnected can refuse restoration; let that operation finish, then retry `ocx stop`. The dashboard stop API returns HTTP 409 with `sharedTeardown: "refused"` when ownership cannot be confirmed and keeps the listener running. Signal cleanup also preserves configuration when ownership cannot be confirmed. + +To use Home from Codex again after restoration, start the Child with `ocx start` (or `ocx service start` for an installed service), then run `ocx sync --restart-codex`. Starting the runtime does not reapply a Child's Codex routing. Sync reapplies Codex settings when Codex is enabled for the saved connection, and the flag restarts Codex to load them; it does not recreate the Grok block. This is the explicit Link Child stop/start workflow; hub-client ownership and the separate restart, recycling, service-exit and sibling rules are unchanged. If **Child** says to pair this machine first, open this computer's configured literal-loopback HTTP dashboard, for example `http://127.0.0.1:`. The local pairing form appears only when pairing is missing and the dashboard and API use the same loopback origin. Copy the form's `ocx gui pair --origin "http://127.0.0.1:"` command, run it in a terminal on this computer, and paste the one-use code into the form. Use the exact origin shown in the form; a provider API key or admin token is not a pairing code. Missing pairing is separate from a configured-port mismatch. diff --git a/docs-site/src/content/docs/guides/web-dashboard.md b/docs-site/src/content/docs/guides/web-dashboard.md index c19a1c825b4..977ded28b85 100644 --- a/docs-site/src/content/docs/guides/web-dashboard.md +++ b/docs-site/src/content/docs/guides/web-dashboard.md @@ -453,6 +453,8 @@ The GUI is a thin client over the proxy's JSON management API. Useful endpoints | `GET` / `PUT /api/subagent-models` | Read or set the five featured `spawn_agent` override models. | | `POST /api/stop` | Stop the proxy/service, restore native Codex, and exit. Refused with `respawnable_service` on the Windows Task Scheduler backend, with `self_unload_service` when this proxy is itself the installed launchd/systemd job, and with `service_state_unknown` when the Task Scheduler state cannot be read; nothing is changed in any of those cases. A dashboard session is also refused, with `desktop_supervised`, while the OpenCodex desktop app runs the proxy. | +Stop rechecks connection ownership before restoring configuration. A new hub connection keeps its routing, catalog and Grok configuration. If ownership is still changing or cannot be confirmed, the API returns HTTP 409 with `sharedTeardown: "refused"` and keeps serving; finish the connection operation and retry. See the [Remote Link stop workflow](/guides/remote-link/). + :::tip Adding **Ollama Cloud** or another catalog provider from the dashboard copies its text-versus-vision classification into the saved provider config, so the [vision sidecar](/guides/sidecars/) diff --git a/docs-site/src/content/docs/ja/reference/cli/lifecycle.md b/docs-site/src/content/docs/ja/reference/cli/lifecycle.md index 2306116e743..f66b720fdbb 100644 --- a/docs-site/src/content/docs/ja/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/ja/reference/cli/lifecycle.md @@ -30,7 +30,11 @@ ocx start --socks5-off 実行中のプロキシを (PID によって) 停止し、PID ファイルを削除します。スタンドアロンの所有者ではネイティブ Codex を復元します。マネージド バックグラウンド サービスがインストールされている場合、`ocx stop` はそれを最初に停止するため、プロキシを再起動できません。Web ダッシュボードの **停止** ボタンは同じ処理 (`POST /api/stop`) を実行しますが、Windows タスク スケジューラだけは例外です。タスク終了後もラッパーがプロキシを再起動しうるため、ダッシュボードは `respawnable_service` で拒否し、何も変更せずに `ocx stop` の実行を促します。 -接続済みの hub クライアントまたは Remote Link Child では、`ocx stop`（成功した API 停止を含む）は接続が所有する Codex のルーティング、モデルカタログ、Grok 設定を保持します。`ocx start` はルーティングを再注入せず、その設定で再び処理します。Codex を自身のアカウントへ戻すには `ocx disconnect`、既にネイティブルーティングへ戻した後の再適用には `ocx sync --restart-codex` を使います。 +接続済みの hub クライアントでは、`ocx stop`（成功した API 停止を含む）は接続が所有する Codex のルーティング、モデルカタログ、Grok 設定を保持します。`ocx start` はルーティングを再注入せず、その設定で再び処理します。Codex を自身のアカウントへ戻すには `ocx disconnect`、既にネイティブルーティングへ戻した後の再適用には `ocx sync --restart-codex` を使います。 + +Remote Link Child を明示的に停止すると、共有設定の復元が成功した場合、ネイティブ Codex のルーティング、カタログ、バックアップ済みの再開履歴メタデータを復元し、管理対象の Grok 設定を削除します。保存済みの Home 接続は保持され、`ocx disconnect` で削除できます。Codex から Home を再び使うには、`ocx start`（サービス導入済みなら `ocx service start`）の後に `ocx sync --restart-codex` を実行してください。起動だけではルーティングは再適用されず、sync は Grok ブロックを再作成しません。 + +サービス停止前に接続の所有権を確認できない場合、停止 API は HTTP 409 と `sharedTeardown: "refused"` を返し、プロキシとサービスマネージャーを稼働させたままにします。接続または切断の処理が完了してから再試行してください。 接続済みクライアントの停止では、保護されたローカルのランタイム記録とリスナーの証明を照合します。クライアント用リスナーと通常プロキシの両方にこの確認が必要で、通常プロキシでは設定の保持機能も認証付きの読み取りで確認します。古いランタイムや確認できないランタイムは、サービスやプロセスを停止する前に拒否されます。監視元からランタイムを更新・再起動して再試行してください。 diff --git a/docs-site/src/content/docs/ko/reference/cli/lifecycle.md b/docs-site/src/content/docs/ko/reference/cli/lifecycle.md index 69166a87f50..f1467d731dc 100644 --- a/docs-site/src/content/docs/ko/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/ko/reference/cli/lifecycle.md @@ -54,7 +54,11 @@ ocx start --socks5-off 넘어가지 않습니다. 영수증에 근거한 지연 복원도 최종 복원과 영수증 정리를 부모가 담당하며, 부모의 공유 설정 복원이 실패하면 종료 실패로 남기고 미완료 영수증을 보존합니다. -연결된 hub 클라이언트나 Remote Link Child에서는 성공한 API 종료를 포함해 `ocx stop`이 연결 소유의 Codex 라우팅, 모델 카탈로그와 Grok 설정을 보존합니다. `ocx start`는 라우팅을 다시 주입하지 않고 보존한 설정으로 서비스를 재개합니다. Codex를 자체 계정으로 돌리려면 `ocx disconnect`를, 이미 기본 라우팅으로 복원했다면 다시 연결하려고 `ocx sync --restart-codex`를 실행하세요. +연결된 hub 클라이언트에서는 성공한 API 종료를 포함해 `ocx stop`이 연결 소유의 Codex 라우팅, 모델 카탈로그와 Grok 설정을 보존합니다. `ocx start`는 라우팅을 다시 주입하지 않고 보존한 설정으로 서비스를 재개합니다. Codex를 자체 계정으로 돌리려면 `ocx disconnect`를, 이미 기본 라우팅으로 복원했다면 다시 연결하려고 `ocx sync --restart-codex`를 실행하세요. + +Remote Link Child를 명시적으로 중지하면 공유 설정 복원에 성공한 경우 기본 Codex 라우팅, 카탈로그와 백업된 재개 이력 메타데이터를 복원하고 관리형 Grok 설정을 제거합니다. 저장된 Home 연결은 유지하며 `ocx disconnect`로 제거할 수 있습니다. Codex에서 Home을 다시 쓰려면 `ocx start`(서비스가 설치되어 있으면 `ocx service start`) 후 `ocx sync --restart-codex`를 실행하세요. 시작만으로 라우팅을 다시 적용하지 않으며, sync는 Grok 블록을 다시 만들지 않습니다. + +서비스 중지 전에 연결 소유권을 확인할 수 없으면 종료 API는 HTTP 409와 `sharedTeardown: "refused"`를 반환하고 프록시와 서비스 관리자를 실행 상태로 둡니다. 진행 중인 연결 또는 연결 해제가 끝난 뒤 다시 시도하세요. 연결된 클라이언트를 중지할 때는 보호된 로컬 런타임 기록과 리스너의 신원 증명을 대조합니다. 클라이언트 리스너와 일반 프록시 모두 이 검증이 필요하며, 일반 프록시는 인증된 조회로 설정 보존 기능도 확인합니다. 오래되었거나 검증할 수 없는 런타임은 서비스나 프로세스를 중지하기 전에 거절합니다. 런타임을 관리하는 앱이나 서비스에서 업데이트·재시작한 뒤 다시 시도하세요. diff --git a/docs-site/src/content/docs/reference/cli/lifecycle.md b/docs-site/src/content/docs/reference/cli/lifecycle.md index 08680d49d6e..bdb0fbc1d0f 100644 --- a/docs-site/src/content/docs/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/reference/cli/lifecycle.md @@ -84,10 +84,18 @@ ocx start --socks5-off Stop the running proxy (by PID), remove the PID file, and restore native Codex for standalone owners. If a managed background service is installed, `ocx stop` also stops it first so it cannot respawn the proxy. -On a connected client (a Remote Link Child or a hub client), `ocx stop` leaves Codex routing, its +On a connected hub client, `ocx stop` leaves Codex routing, its model catalog and the Grok config as they are, including after a successful API stop: the connection owns them, and `ocx start` serves them again without re-injecting routing. Use `ocx disconnect` to hand Codex back to its own account. If Codex on a connected client already reads native routing, `ocx status` names `ocx sync --restart-codex` to point it back. +An explicit stop on a Remote Link Child restores native Codex routing, catalog and backed-up +resume-history metadata and removes the managed Grok configuration when shared restoration succeeds. +The saved Home connection is kept; `ocx disconnect` removes it. To use Home from Codex again, +start the Child with `ocx start` (or `ocx service start` for an installed service), then run +`ocx sync --restart-codex`. Starting alone does not reapply routing; sync does not recreate the Grok block. +If connection ownership cannot be confirmed before service stop, the stop API returns HTTP 409 with +`sharedTeardown: "refused"` and keeps the proxy and service manager running. Let any pending +connect or disconnect finish, then retry. A connected stop verifies the listener against the protected local runtime record before changing its service. Both client-role listeners and ordinary proxies must prove that identity; an ordinary proxy must also advertise preservation across API, signal and service shutdown in an authenticated diff --git a/docs-site/src/content/docs/ru/reference/cli/lifecycle.md b/docs-site/src/content/docs/ru/reference/cli/lifecycle.md index 938634822ad..2ac1b18c144 100644 --- a/docs-site/src/content/docs/ru/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/ru/reference/cli/lifecycle.md @@ -54,7 +54,11 @@ ocx start --socks5-off установлена managed background service, `ocx stop` сначала останавливает и её, чтобы она не перезапустила прокси обратно. Кнопка **Stop** в веб-дашборде выполняет то же действие (`POST /api/stop`) на всех бэкендах, кроме планировщика заданий Windows: там обёртка может перезапустить прокси после завершения задачи, поэтому дашборд отказывает с `respawnable_service`, ничего не меняет и просит выполнить `ocx stop`. -На подключённом клиенте hub или Remote Link Child команда `ocx stop`, включая успешную остановку через API, сохраняет принадлежащие соединению маршрутизацию Codex, каталог моделей и конфигурацию Grok. `ocx start` снова обслуживает их, не внедряя маршрутизацию заново. Чтобы вернуть Codex к собственному аккаунту, выполните `ocx disconnect`; для повторного применения уже восстановленной нативной маршрутизации используйте `ocx sync --restart-codex`. +На подключённом клиенте hub команда `ocx stop`, включая успешную остановку через API, сохраняет принадлежащие соединению маршрутизацию Codex, каталог моделей и конфигурацию Grok. `ocx start` снова обслуживает их, не внедряя маршрутизацию заново. Чтобы вернуть Codex к собственному аккаунту, выполните `ocx disconnect`; для повторного применения уже восстановленной нативной маршрутизации используйте `ocx sync --restart-codex`. + +Явная остановка Remote Link Child при успешном восстановлении общей конфигурации восстанавливает нативную маршрутизацию Codex, каталог и сохранённые в резервной копии метаданные истории возобновления, а также удаляет управляемую конфигурацию Grok. Сохранённое соединение с Home остаётся; удалить его можно командой `ocx disconnect`. Чтобы снова использовать Home из Codex, выполните `ocx start` (или `ocx service start` для установленной службы), затем `ocx sync --restart-codex`. Один запуск не применяет маршрутизацию заново; sync не создаёт блок Grok повторно. + +Если владение соединением нельзя подтвердить до остановки службы, API остановки возвращает HTTP 409 с `sharedTeardown: "refused"`, оставляя прокси и менеджер службы работающими. Дождитесь завершения подключения или отключения и повторите попытку. При остановке подключённого клиента команда сверяет доказательство идентичности слушателя с защищённой локальной записью рантайма. Это требуется и для клиентского слушателя, и для обычного прокси; обычный прокси также должен подтвердить сохранение настроек через аутентифицированный запрос чтения. Старые или непроверенные рантаймы отклоняются до остановки службы или процесса. Обновите и перезапустите рантайм через управляющий супервизор, затем повторите команду. diff --git a/docs-site/src/content/docs/tr/reference/cli/lifecycle.md b/docs-site/src/content/docs/tr/reference/cli/lifecycle.md index 5b483838e1f..311c2436d39 100644 --- a/docs-site/src/content/docs/tr/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/tr/reference/cli/lifecycle.md @@ -54,6 +54,10 @@ ocx start --socks5-off geri yükleyin. Yönetilen bir arka plan servisi kuruluysa `ocx stop` proxy'yi yeniden oluşturamaması için önce onu da durdurur. Web kontrol panelinin **Durdur** düğmesi aynı eylemi (`POST /api/stop`) Windows Görev Zamanlayıcı dışındaki tüm arka uçlarda çalıştırır: orada görev bittikten sonra sarmalayıcı proxy'yi yeniden başlatabilir, bu yüzden panel `respawnable_service` ile reddeder, hiçbir şeyi değiştirmez ve `ocx stop` çalıştırmanızı ister. +Bağlı bir hub istemcisinde `ocx stop`, API üzerinden başarılı durdurma dahil, bağlantının sahip olduğu Codex yönlendirmesini, kataloğu ve Grok yapılandırmasını korur; `ocx start` bunları yeniden sunar. Remote Link Child açıkça durdurulduğunda ise paylaşılan geri yükleme başarılıysa yerel Codex yönlendirmesi, katalog ve yedeklenmiş oturum sürdürme geçmişi meta verileri geri yüklenir, yönetilen Grok yapılandırması kaldırılır. Kayıtlı Home bağlantısı korunur; `ocx disconnect` ile kaldırılır. + +Geri yüklemeden sonra Codex'ten Home'u yeniden kullanmak için `ocx start` (servis kuruluysa `ocx service start`), ardından `ocx sync --restart-codex` çalıştırın. Yalnızca başlatmak yönlendirmeyi yeniden uygulamaz; sync Grok bloğunu yeniden oluşturmaz. Servis durdurulmadan önce bağlantı sahipliği doğrulanamıyorsa durdurma API'si HTTP 409 ve `sharedTeardown: "refused"` döndürür; proxy ve servis yöneticisi çalışmaya devam eder. Devam eden bağlanma veya bağlantıyı kesme işlemi bitince yeniden deneyin. + ### `ocx restart` Bir proxy çalışırken bu tam onaylanmış PID ve porttan yerinde yeniden diff --git a/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md b/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md index 6a1a7ec602b..7c34574b37e 100644 --- a/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md @@ -30,7 +30,11 @@ ocx start --socks5-off 停止正在运行的代理（按 PID），移除 PID 文件；独立运行的所有者还会恢复原生 Codex。如果安装了受管后台服务，`ocx stop` 还会先停止该服务，这样它就无法重新拉起代理。Web 仪表盘的 **Stop** 按钮在多数后端执行同样的操作（`POST /api/stop`），但 Windows 任务计划程序除外：任务结束后包装器仍可能重新拉起代理，只有运行在代理之外的 stop 才能在恢复客户端配置前确认这个重启窗口，因此仪表盘会以 `respawnable_service` 拒绝、不做任何更改，并提示改用 `ocx stop`。 -对于已连接的 hub 客户端或 Remote Link Child，`ocx stop`（包括成功的 API 停止）会保留连接所拥有的 Codex 路由、模型目录和 Grok 配置。`ocx start` 使用这些保留的配置恢复服务，不会重新注入路由。要让 Codex 回到自己的账户，请运行 `ocx disconnect`；若已恢复为原生路由，要重新应用连接路由，请运行 `ocx sync --restart-codex`。 +对于已连接的 hub 客户端，`ocx stop`（包括成功的 API 停止）会保留连接所拥有的 Codex 路由、模型目录和 Grok 配置。`ocx start` 使用这些保留的配置恢复服务，不会重新注入路由。要让 Codex 回到自己的账户，请运行 `ocx disconnect`；若已恢复为原生路由，要重新应用连接路由，请运行 `ocx sync --restart-codex`。 + +显式停止 Remote Link Child 时，若共享配置恢复成功，会恢复原生 Codex 路由、目录和已备份的恢复会话历史元数据，并移除托管的 Grok 配置。已保存的 Home 连接会保留，可用 `ocx disconnect` 移除。要再次从 Codex 使用 Home，请先运行 `ocx start`（已安装服务时使用 `ocx service start`），再运行 `ocx sync --restart-codex`。仅启动不会重新应用路由；sync 不会重新创建 Grok 配置块。 + +若在停止服务前无法确认连接所有权，停止 API 会返回 HTTP 409 和 `sharedTeardown: "refused"`，并保持代理和服务管理器运行。请等待正在进行的连接或断开操作完成后重试。 停止已连接的客户端时，命令会依据受保护的本地运行时记录验证监听器的身份证明。客户端监听器和普通代理都必须通过此验证；普通代理还必须通过经过身份验证的读取确认配置保留能力。对于旧版或无法验证的运行时，命令会在停止服务或进程前拒绝操作。请通过管理该运行时的应用或服务更新并重启它，然后重试。 diff --git a/docs-site/src/content/docs/zh-tw/reference/cli/lifecycle.md b/docs-site/src/content/docs/zh-tw/reference/cli/lifecycle.md index 3504915bb17..ebb9af1eb57 100644 --- a/docs-site/src/content/docs/zh-tw/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/zh-tw/reference/cli/lifecycle.md @@ -30,6 +30,10 @@ ocx start --socks5-off 停止執行中的代理（依 PID）、移除 PID 檔案，並還原原生 Codex。若已安裝受管背景服務，`ocx stop` 也會先停止它，使其無法重新生成代理。網頁儀表板的 **Stop** 按鈕在多數後端執行相同動作（`POST /api/stop`），但 Windows 工作排程器除外：工作結束後包裝程序仍可能重新啟動 Proxy，因此儀表板會以 `respawnable_service` 拒絕、不做任何變更，並請你改用 `ocx stop`。 +對已連線的 hub 用戶端，`ocx stop`（包括成功的 API 停止）會保留連線擁有的 Codex 路由、目錄和 Grok 設定，`ocx start` 會再次提供服務。明確停止 Remote Link Child 時，若共用設定還原成功，則會還原原生 Codex 路由、目錄與已備份的續接歷程中繼資料，並移除受管理的 Grok 設定。已儲存的 Home 連線會保留，可用 `ocx disconnect` 移除。 + +要在還原後再次從 Codex 使用 Home，請先執行 `ocx start`（已安裝服務時使用 `ocx service start`），再執行 `ocx sync --restart-codex`。僅啟動不會重新套用路由；sync 不會重新建立 Grok 設定區塊。若在停止服務前無法確認連線擁有權，停止 API 會回傳 HTTP 409 與 `sharedTeardown: "refused"`，並讓 Proxy 與服務管理員繼續執行。請等待進行中的連線或中斷連線作業完成後再試。 + ### `ocx restart` 執行 `stop` 後接 `ensure`：停止代理／服務、還原原生 Codex、在背景啟動代理，並將即時連接埠同步回 Codex。 diff --git a/scripts/test-layout/layout-clients.json b/scripts/test-layout/layout-clients.json index a22eba46ab3..68e12a95b2c 100644 --- a/scripts/test-layout/layout-clients.json +++ b/scripts/test-layout/layout-clients.json @@ -5,6 +5,7 @@ "fix-plain-data-own-keys.test.ts": "lib", "fix-plain-data-shape.test.ts": "lib", "claude-native-fast-1m-ingress.test.ts": "claude-integration", + "stop-restoration-preflight.test.ts": "service", "claude-intercept-acl-barrier.test.ts": "claude-integration", "claude-config-root-writer-barrier.test.ts": "claude-integration", "claude-config-root-late-admission.test.ts": "claude-integration", diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index dcadae77ce8..62f70df7ba6 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -3,6 +3,8 @@ "root": "tests", "explicitShards": ["layout-clients.json", "layout-codex-collision.json", "layout-agent-assertion.json", "layout-chat.json", "layout-replay.json", "layout-adapters.json", "layout-jev.json", "layout-workbuddy.json", "layout-combos.json", "layout-credit-renewal.json", "layout-server-inbound.json", "layout-providers.json", "layout-responses.json", "layout-exec-boundary.json"], "explicit": { "service-desktop-recovery.test.ts": "service", "provider-input-modalities.test.ts": "providers", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "codex-provider-routing.test.ts": "codex-integration", "config-native-reasoning-retention.test.ts": "config", "server-native-reasoning-retention-merge.test.ts": "server", "server-native-reasoning-retention.test.ts": "server", "responses-retained-reasoning-stream-recovery.test.ts": "responses", "openai-native-hosted-search-history.test.ts": "adapters/openai", "responses-native-compact-search-history.test.ts": "responses", "config-hosted-search-projection.test.ts": "config", "gcp-credential-boundary.test.ts": "adapters/google", "gcp-sa-marker.test.ts": "adapters/google", "gcp-sa-adapter.test.ts": "adapters/google", "management-gcp-credential-input.test.ts": "server", "codex-preserve-signed-in.test.ts": "codex-integration", "cli-account-preservation.test.ts": "cli", "cli-provider-list-active.test.ts": "cli", "service-state-isolation.test.ts": "service", "compaction-history-framing.test.ts": "responses", "catalog-native-max-context.test.ts": "codex-integration", "protocol-argon-thought-boundary.test.ts": "responses", "service-log.test.ts": "service", "messages-native-empty-inline.test.ts": "claude-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "bridge-terminal-refusal.test.ts": "adapters", "loop-terminal-refusal.test.ts": "images", "web-search-terminal-refusal.test.ts": "web-search", "combo-terminal-refusal.test.ts": "routing", "server-terminal-refusal-wiring.test.ts": "server", "incomplete-reason-normalization.test.ts": "responses", "advisor-retention.test.ts": "advisor", "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "api-key-selection-cooldown.test.ts": "providers", "api-key-auto-pause-distribution.test.ts": "providers", "explicit-key-pause-oauth-boundary.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "coding-agent-windows-dispatch.test.ts": "adapters", "coding-agent-native-empty-argv.test.ts": "windows", + "stop-restore-history-ownership.test.ts": "service", + "cli-stop-refusal-transport.test.ts": "cli", "service-proxy-env.test.ts": "service", "openai-chat-standalone-bare-block.test.ts": "adapters/openai", "owner-registry-isolation.test.ts": "config", "server-combo-stream-required-tool.test.ts": "server", "combo-required-tool-stream-preflight.test.ts": "routing", "responses-auto-pause-replay-owner.test.ts": "responses", "responses-account-auto-pause.test.ts": "responses", "oauth-account-auto-pause.test.ts": "oauth", "responses-account-auto-pause-wire-owner.test.ts": "responses", "oauth-upsert-paused-api-key.test.ts": "oauth", "management-account-auto-pause.test.ts": "server", "cli-account-key-pause.test.ts": "cli", "oauth-account-auto-pause-revision.test.ts": "oauth", "config-account-auto-pause-revisions.test.ts": "config", "codex-history-pending-wal.test.ts": "codex-integration", "responses-anthropic-billing-send-budget.test.ts": "responses", "anthropic-billing-ownership.test.ts": "adapters/anthropic", "server-anthropic-billing.test.ts": "server", "server-anthropic-billing-boundaries.test.ts": "server", "router-anthropic-billing-failover.test.ts": "routing", "anthropic-billing-refusal.test.ts": "adapters/anthropic", "messages-request-id-headers.test.ts": "claude-integration", "bounded-file-read.test.ts": "lib", "responses-undeclared-tool-guard-inbound.test.ts": "responses", "explicit-empty-tool-catalog.test.ts": "responses", "explicit-empty-tool-catalog-history.test.ts": "responses", diff --git a/src/cli/index.ts b/src/cli/index.ts index c97a6c1ca5f..91d0e94b0ec 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -71,7 +71,6 @@ import { } from "../config/process-state"; import { claimPendingTeardown, - clearPendingTeardown, isPendingTeardownAbandoned, listPendingTeardowns, pendingTeardownPathFor, @@ -163,7 +162,8 @@ import { removeOwnedConfigAfterDesktopCleanup } from "./uninstall-client-state"; import { withProcessRuntimeProvenance } from "../lib/bun-runtime"; import { startArgv } from "../lib/self-launch-argv"; import { initializeNodeLauncherContext } from "./launcher-context"; -import { assertConnectedClientStopSupported, connectedClientStopSkipLine, preflightConnectedClientStop, preserveConnectedClientRoutingOnExit, restoreSharedClientStateAfterStop } from "./stop-restore"; +import { assertConnectedClientStopSupported, clearRestoredStopReceipts, connectedClientStopSkipLine, notOwnedTeardownIsAcceptable, preflightConnectedClientStop, preserveConnectedClientRoutingOnExit, restoreBoundaryOwnership, restoreSharedClientStateAfterStop, settleUnrestoredStopBoundary } from "./stop-restore"; +import { stopRestoreWriteGuard } from "./stop-restore-guard"; import { startClientRuntimeUnderOwnershipLease } from "./client-start-fence"; import { recoverStartStateUnderOwnershipLease } from "./start-owner-fence"; import { handledSignalExitCode } from "../lib/handled-signal-exit"; @@ -657,6 +657,7 @@ async function handleStart(options: { block?: boolean } = {}) { // Dashboard drain-and-restart (#563) must not tear down injection: the replacement // process expects Codex/Grok/env fences to still be in place. A sibling owns none of them. const teardown = preserveConnectedClientRoutingOnExit(decideStartExitTeardown({ sibling: siblingStart, recycling: isRecyclingForExit(), ocxService: process.env.OCX_SERVICE })); + const restoreGuard = stopRestoreWriteGuard(); if (teardown.revertSystemEnv) { try { revertSystemEnv(); } catch { /* best-effort */ } } @@ -665,15 +666,17 @@ async function handleStart(options: { block?: boolean } = {}) { if (teardown.restoreNativeCodex) { try { if (!codexHomeIsAbsent(getCodexHome())) { - const restored = restoreNativeCodex(); + const restored = restoreNativeCodex({ withWriteGuard: restoreGuard.run }); if (!restored.success) { cleanupSucceeded = false; console.error(`⚠️ Native Codex restore failed during shutdown: ${restored.message}`); } } } catch (error) { - cleanupSucceeded = false; - console.error(`⚠️ Native Codex restore failed during shutdown: ${error instanceof Error ? error.message : String(error)}`); + if (restoreGuard.boundary() !== "hub-owned") { + cleanupSucceeded = false; + console.error(`⚠️ Native Codex restore failed during shutdown: ${error instanceof Error ? error.message : String(error)}`); + } } } // Same ownership rule as `ocx stop`: if the installed service belongs to another home, the @@ -681,7 +684,8 @@ async function handleStart(options: { block?: boolean } = {}) { // left pointing nowhere. This guard also covers signal-driven exits, which is the path that // would otherwise bypass handleStop's gate entirely. if (teardown.stripGrokConfig && serviceEnvironmentOwnedHere()) { - try { stripGrokConfig(); } catch { /* best-effort restore */ } + try { restoreGuard.run(() => stripGrokConfig()); } + catch { if (restoreGuard.boundary() !== "hub-owned") cleanupSucceeded = false; } } return cleanupSucceeded; }; @@ -708,7 +712,7 @@ async function handleStart(options: { block?: boolean } = {}) { void (async () => { let shutdownSucceeded = false; try { - shutdownSucceeded = await drainAndShutdown(server, config.shutdownTimeoutMs ?? 5000); + shutdownSucceeded = await drainAndShutdown(server, config.shutdownTimeoutMs ?? 5000, () => { syncCleanup(); return undefined; }); } finally { const restored = syncCleanup(); // idempotent (cleaned-guard); also re-run by process.on("exit") process.exit(restored && shutdownSucceeded ? handledSignalExitCode(signal) : 1); @@ -1159,6 +1163,11 @@ async function handleStopUnlocked(snapshot?: GuardedStopSnapshot) { claimTeardown(exact ?? configuredEndpoint(), exact ? "exact" : "guessed"); const graceful = await stopProxy(pid, { deferSharedTeardownNonce: teardownNonce, + // "not-owned" is refused only for a positively identified Link connection, read at the + // POST: an old build answers it while preserving the loopback route this stop must + // restore. A connect committed mid-stop can make this a hub client whose answer is + // truthful, so the entry-time skip snapshot cannot stand in for this check. + acceptNotOwnedTeardown: notOwnedTeardownIsAcceptable(readClientConnectionState(), stoppingSibling), // Only an exact endpoint may direct the request; the configured fallback is a guess // good enough to record an obligation against, not to POST a stop to. runtimeEndpoint: exact ?? undefined, @@ -1436,44 +1445,30 @@ async function handleStopUnlocked(snapshot?: GuardedStopSnapshot) { const restoreBlocked = ownershipBlocked || inheritedBlocks || nativeRestoreHandledByProxy || stoppingSibling || connectedClientSkip !== null; const handedOver = stopHandsTeardownOver({ requested: handoverRequested, restoreBlocked, nonce: teardownNonce, stopFailed, recoveredCount: recoveredNonces.length, proxy: record.proxy, service: record.service, deferralHonored: deferralHonoredByProxy }); + // Ownership can change while the stop awaits; only the state at the restore boundary decides (#7736). + const boundary = restoreBlocked || handedOver ? "restore" : restoreBoundaryOwnership(); + if (boundary !== "restore") stopFailed = settleUnrestoredStopBoundary(boundary, record, teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces) || stopFailed; if (handedOver) announceStopHandover(); - else if (!restoreBlocked) { + else if (!restoreBlocked && boundary === "restore") { if (recoveredNonces.length > 0) { // A previous deferred stop died before restoring, and the probe says its endpoint is // not answering. That is the whole point of leaving the receipt behind. console.log("↩️ Finishing a shared teardown left unfinished by an earlier stop."); } const restore = await restoreSharedClientStateAfterStop(reportRetainedCodexProviderTable); - record.sharedTeardown = restore.historyDeferred ? "refused" : restore.other ? "failed" : "restored"; - if (restore.other) stopFailed = true; - else if (restore.historyDeferred) historyDeferredNonces = teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces; - else if (restore.historyOnly) historyOnlyFailure = true; - // The obligation is discharged whether or not history metadata finalized: config and - // catalog are what a client reads, and `restore.other` already fails the stop. - // - // Each nonce names its own file, so a clear can only ever remove the obligation it - // names — never one a concurrent stop wrote. Both this run's claim and every inherited - // receipt it proved discharged are released together. - // - // A history-preflight refusal is the exception: it restored nothing, so there is - // nothing to discharge. Clearing here would drop a real obligation on the floor and - // leave the client config pointing at a proxy that is gone, with nothing on disk - // saying so — which is the whole failure the receipt exists to prevent (#4718). - if (restore.historyDeferred) { - console.error(" The shared teardown was refused before it changed anything, so it is still owed."); - console.error(" Its receipt is preserved; run 'ocx stop' again once Codex is closed to retry the restore."); - } - if (!restore.other && !restore.historyDeferred) { - const discharged = teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces; - for (const nonce of discharged) { - // A receipt that survives its discharge re-triggers recovery forever, so a failed - // removal is surfaced rather than swallowed. - if (!clearPendingTeardown(nonce)) { - stopFailed = true; - record.receiptClearFailed = true; - console.error(`❌ The shared teardown finished, but its receipt could not be removed: ${pendingTeardownPathFor(nonce)}`); - console.error(" Remove it manually; otherwise every later stop and update will try to recover it again."); - } + if (restore.boundary) stopFailed = settleUnrestoredStopBoundary(restore.boundary, record, teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces) || stopFailed; + else { + record.sharedTeardown = restore.historyDeferred ? "refused" : restore.other ? "failed" : "restored"; + if (restore.other) stopFailed = true; + else if (restore.historyDeferred) historyDeferredNonces = teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces; + else if (restore.historyOnly) historyOnlyFailure = true; + // A history preflight refusal restored nothing, so the obligation survives (#4718). + if (restore.historyDeferred) { + console.error(" The shared teardown was refused before it changed anything, so it is still owed."); + console.error(" Its receipt is preserved; run 'ocx stop' again once Codex is closed to retry the restore."); + } + if (!restore.other && !restore.historyDeferred) { + stopFailed = clearRestoredStopReceipts(record, teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces) || stopFailed; } } } diff --git a/src/cli/stop-restore-guard.ts b/src/cli/stop-restore-guard.ts new file mode 100644 index 00000000000..b87bc749ada --- /dev/null +++ b/src/cli/stop-restore-guard.ts @@ -0,0 +1,48 @@ +import { withConfigMutationLockSync } from "../config"; +import { inspectStopRestoreOwnership } from "../client/restore-ownership"; +export { inspectStopRestoreOwnership, stopRestoreWriteGuard } from "../client/restore-ownership"; +import type { StopRunRecord } from "./stop-report"; +import { clearPendingTeardown, pendingTeardownPathFor } from "../config/pending-teardown"; + +type RestoreRecord = Pick; + +function refuse(record: RestoreRecord): true { + record.sharedTeardown = "refused"; + console.error("❌ The client connection could not be confirmed (it may be changing); restoration stopped and the obligation is kept. Rerun 'ocx stop'."); + return true; +} + +/** Only the named, discharged nonces are removed; C covers both ownership and unlink. */ +export function clearRestoredStopReceipts(record: RestoreRecord, nonces: string[], restored = true): boolean { + let entered = false; + try { + return withConfigMutationLockSync(() => { + entered = true; + const boundary = inspectStopRestoreOwnership(); + if (boundary === "unproven" || boundary === "restore" && !restored) return refuse(record); + if (boundary === "hub-owned") { + record.sharedTeardown = "skipped"; + console.log("↩️ A hub connection was committed while this stop ran; it owns Codex routing, its catalog and the Grok config, so restoration stopped."); + } + let failed = false; + for (const nonce of nonces) { + if (!clearPendingTeardown(nonce)) { + failed = true; + record.receiptClearFailed = true; + console.error(`❌ The shared teardown was discharged, but its receipt could not be removed: ${pendingTeardownPathFor(nonce)}`); + console.error(" Remove it manually; otherwise every later stop and update will try to recover it again."); + } + } + return failed; + }); + } catch { + if (entered) record.receiptClearFailed = true; + return refuse(record); + } +} + +export function settleUnrestoredStopBoundary( + boundary: "hub-owned" | "unproven", record: RestoreRecord, nonces: string[], +): boolean { + return boundary === "unproven" ? refuse(record) : clearRestoredStopReceipts(record, nonces, false); +} diff --git a/src/cli/stop-restore.ts b/src/cli/stop-restore.ts index ee460e90ce0..313b87a30b1 100644 --- a/src/cli/stop-restore.ts +++ b/src/cli/stop-restore.ts @@ -2,22 +2,32 @@ import { restoreNativeCodexAsync } from "../codex/inject"; import type { RetainedCodexProviderTable } from "../codex/inject/restore"; import { stripGrokConfig } from "../grok/inject"; import { readClientConnectionState, type ClientConnectionState } from "../client/state"; +import { withClientLifecycleSync } from "../client/lifecycle-lock"; +import type { RestoreBoundaryOwnership } from "../client/restore-ownership"; +export type { RestoreBoundaryOwnership } from "../client/restore-ownership"; +import { inspectStopRestoreOwnership, stopRestoreWriteGuard } from "./stop-restore-guard"; +export { clearRestoredStopReceipts, settleUnrestoredStopBoundary } from "./stop-restore-guard"; import { readPid, readRuntimePort } from "../config/process-state"; import { loadConfig } from "../config"; import { readBoundedResponseBody } from "../lib/bounded-body"; import { createLocalAttestationChallenge, LOCAL_ATTESTATION_CHALLENGE_HEADER, LOCAL_ATTESTATION_PROOF_HEADER, verifyLocalAttestationProof } from "../lib/local-management-attestation"; import { LOCAL_MANAGEMENT_READ_PATHS } from "../lib/local-management-capability"; -import { gracefulStopHost, isProcessAlive, ProxyOwnershipRefusedError } from "../lib/process-control"; +import { gracefulStopHost, isProcessAlive, ProxyOwnershipRefusedError, stopRequestFetch as directStopRequestFetch } from "../lib/process-control"; import { directLocalHttpFetch } from "../server/direct-local-http"; import { fetchBoundLocalManagementRead } from "../server/local-management-read-client"; +import { isSharedTeardownDeferredOnExit } from "../server/stop-teardown"; import { findLiveProxy, probeEndpointLiveness, proveLiveProxyOwnedByHome } from "../server/proxy-liveness"; import { observeGuardedManagerStopped } from "../service/guarded-manager-target"; import type { StartExitTeardown } from "./dispatch"; import type { StopOutcome } from "./stop-report"; -/** Re-read ownership when an ordinary runtime actually exits, including after API drain. */ +/** Honor accepted API deferral, otherwise re-read connection ownership on ordinary exit. */ export function preserveConnectedClientRoutingOnExit(teardown: StartExitTeardown): StartExitTeardown { - return readClientConnectionState().kind === "connected" + if (isSharedTeardownDeferredOnExit()) { + return { ...teardown, restoreNativeCodex: false, stripGrokConfig: false }; + } + const connection = readClientConnectionState(); + return connection.kind === "connected" && connection.value.transport !== "link" ? { ...teardown, restoreNativeCodex: false, stripGrokConfig: false } : teardown; } @@ -123,17 +133,61 @@ export async function preflightConnectedClientStop(approved?: ConnectedStopTarge * The line `ocx stop` prints in place of the shared teardown on a connected client, or null * when this home is not one (#7563). * - * A hub client or Remote Link Child gets its Codex routing and catalog from `ocx connect` / - * `ocx sync`, and its client runtime never writes them on start. The standalone teardown + * A hub client gets its Codex routing and catalog from `ocx connect` / `ocx sync`, and its + * client runtime never writes them on start. The standalone teardown * restored native Codex, rewrote the catalog and stripped the Grok block, and nothing on the * client put any of it back: `ocx start` brought the runtime up with Codex still native and * Grok unconfigured. Leaving them in place makes stop/start a round trip; `ocx disconnect` - * remains the teardown that owns them, with its own ownership checks. + * remains the teardown that owns them, with its own ownership checks. A Remote Link Child + * is deliberately excluded: its unauthenticated loopback route is unsafe after the listener exits. */ export function connectedClientStopSkipLine(state: ClientConnectionState): string | null { - if (state.kind !== "connected") return null; - const role = state.value.transport === "link" ? "Remote Link Child" : "hub client"; - return `↩️ Codex routing, its catalog and the Grok config stay as they are: this ${role}'s connection owns them, and 'ocx start' serves them again. To hand Codex back to its own account, run 'ocx disconnect'.`; + if (state.kind !== "connected" || state.value.transport === "link") return null; + return `↩️ Codex routing, its catalog and the Grok config stay as they are: this hub client's connection owns them, and 'ocx start' serves them again. To hand Codex back to its own account, run 'ocx disconnect'.`; +} + +/** + * Transport for the authenticated `POST /api/stop`. The target is always this machine's own + * runtime, so the request never consults HTTP(S)_PROXY/ALL_PROXY, whatever the connection + * state is or becomes while the stop awaits: an environment proxy must never receive the admin + * token. Restoration ownership is decided separately. + */ +export function stopRequestFetch(): typeof fetch { + return directStopRequestFetch as typeof fetch; +} + +/** + * Whether the stop caller may accept `sharedTeardown: "not-owned"` as teardown confirmation. + * + * Refused only for a positively identified Link connection: its unauthenticated loopback + * route is unsafe once the listener exits, only an old build answers `not-owned` for one, + * and it says so while preserving the route this stop must restore. Everything else — a + * sibling, a hub connection, or no connection at all — has nothing that answer skipped. + * Callers pass the state read at the POST, not at stop entry: a connect or disconnect + * committed while the stop is still awaiting decides the ownership the answer describes. + */ +export function notOwnedTeardownIsAcceptable(state: ClientConnectionState, stoppingSibling: boolean): boolean { + return stoppingSibling || state.kind !== "connected" || state.value.transport !== "link"; +} + +/** + * Re-read connection ownership at the parent restore boundary (#7736). A hub connection + * committed while the stop was in flight owns Codex routing, the catalog and Grok, so the + * stop must not restore over it. `ocx connect`/`ocx disconnect` publish connection state + * under the client lifecycle lock (L), so this early read takes L. Each subsequent write + * repeats the check under C via stopRestoreWriteGuard; the early read grants no write authority. + * A busy or failing lock proves nothing either way. + */ +export function restoreBoundaryOwnership(deps: { + read?: typeof readClientConnectionState; + withLifecycle?: typeof withClientLifecycleSync; +} = {}): RestoreBoundaryOwnership { + const read = deps.read ?? readClientConnectionState; + try { + return (deps.withLifecycle ?? withClientLifecycleSync)(() => inspectStopRestoreOwnership(read)); + } catch { + return "unproven"; + } } /** @@ -155,12 +209,14 @@ export function connectedClientStopSkipLine(state: ClientConnectionState): strin */ export async function restoreSharedClientStateAfterStop( reportRetainedCodexProviderTable: (retained: RetainedCodexProviderTable) => void, -): Promise<{ historyOnly: boolean; historyDeferred: boolean; other: boolean }> { +): Promise<{ historyOnly: boolean; historyDeferred: boolean; other: boolean; boundary?: "hub-owned" | "unproven" }> { let historyOnly = false; let historyDeferred = false; let other = false; + const guard = stopRestoreWriteGuard(); try { - const result = await restoreNativeCodexAsync(); + const result = await restoreNativeCodexAsync({ withWriteGuard: guard.run }); + if (guard.boundary() !== "restore") throw new Error("stop_restore_ownership_changed"); if (result.success) { console.log(`↩️ ${result.message}`); if (result.retainedCodexProviderTable) { @@ -190,16 +246,20 @@ export async function restoreSharedClientStateAfterStop( console.error(`⚠️ ${result.message}`); } } catch (error) { + const boundary = guard.boundary(); + if (boundary !== "restore") return { historyOnly: false, historyDeferred: false, other: false, boundary }; other = true; console.error(`⚠️ Native Codex restore failed: ${error instanceof Error ? error.message : String(error)}`); } // A refused or thrown Grok strip is actionable because it would point Grok at a dead proxy. try { - const grok = stripGrokConfig(); + const grok = guard.run(() => stripGrokConfig()); if (grok.changed) console.log(`↩️ ${grok.message}`); else if (!grok.ok) { other = true; console.error(`⚠️ ${grok.message}`); } } catch (error) { + const boundary = guard.boundary(); + if (boundary !== "restore") return { historyOnly, historyDeferred, other, boundary }; other = true; console.error(`⚠️ Grok config restore failed: ${error instanceof Error ? error.message : String(error)}`); } diff --git a/src/client/restore-ownership.ts b/src/client/restore-ownership.ts new file mode 100644 index 00000000000..e1d190cb9f0 --- /dev/null +++ b/src/client/restore-ownership.ts @@ -0,0 +1,45 @@ +import { withConfigMutationLockSync } from "../config"; +import { assertNoClientDisconnectPending, pendingClientConnectMayOwnToken, readClientConnectionState } from "./state"; + +export type RestoreBoundaryOwnership = "restore" | "hub-owned" | "unproven"; +export type RestoreOwnershipRefusal = Exclude; + +export class StopRestoreOwnershipError extends Error { + constructor(readonly boundary: RestoreOwnershipRefusal) { + super(`stop_restore_${boundary}`); + } +} + +/** Read only; the caller must exclude connection publication with L or C. */ +export function inspectStopRestoreOwnership(read = readClientConnectionState): RestoreBoundaryOwnership { + try { + if (pendingClientConnectMayOwnToken()) return "unproven"; + assertNoClientDisconnectPending(); + const state = read(); + if (state.kind === "invalid" || state.kind === "mismatched") return "unproven"; + return state.kind === "connected" && state.value.transport !== "link" ? "hub-owned" : "restore"; + } catch { return "unproven"; } +} + +/** C is reentrant inside N/C; this never acquires L or N. No lease crosses an await. */ +export function stopRestoreWriteGuard() { + let boundary: RestoreBoundaryOwnership = "restore"; + return { + boundary: () => boundary, + run(write: () => T): T { + let entered = false; + try { + return withConfigMutationLockSync(() => { + entered = true; + if (boundary === "restore") boundary = inspectStopRestoreOwnership(); + if (boundary !== "restore") throw new StopRestoreOwnershipError(boundary); + return write(); + }); + } catch (error) { + if (!entered) boundary = "unproven"; + else if (error instanceof StopRestoreOwnershipError) boundary = error.boundary; + throw error; + } + }, + }; +} diff --git a/src/codex/history-job.ts b/src/codex/history-job.ts index ea0a62d175f..95f58eefe55 100644 --- a/src/codex/history-job.ts +++ b/src/codex/history-job.ts @@ -27,6 +27,7 @@ import { currentHistoryDbBusyTimeoutMs, resolveExistingHistoryBackupPath } from import { spawnWorker } from "../lib/worker-embed"; import type { CodexHistoryFailureReason, CodexHistoryVerifiedNoopProof } from "./history-provider"; import { getCodexHome, resolveCodexStateDbPath } from "./paths"; +import type { RestoreOwnershipRefusal } from "../client/restore-ownership"; /** * Resolve the paths a history job needs, at CALL time. @@ -103,6 +104,7 @@ export interface CodexHistoryJobRequest { readonly canonicalBackupPath: string; readonly operation: CodexHistoryWorkerOperation; readonly expectedDesiredEnabled?: boolean; + readonly guardStopOwnership?: boolean; } export type CodexHistoryJobOutcome = @@ -111,6 +113,7 @@ export type CodexHistoryJobOutcome = | { readonly kind: "blocked"; readonly reason: "busy" | "database" | "unsafe-path" | "desired_disabled" | "desired_enabled" } | { readonly kind: "failed"; readonly reason: "worker-error" | "worker-died" | "timeout"; readonly message: string; readonly historyFailureReason?: CodexHistoryFailureReason; + readonly restoreBoundary?: RestoreOwnershipRefusal; /** Specific integrity condition when `historyFailureReason` is `"integrity"`. */ readonly historyIntegrityCode?: string; readonly rows?: number; readonly files?: number }; @@ -178,6 +181,8 @@ function isPlausibleWorkerResult( || message.reason === "desired_disabled" || message.reason === "desired_enabled"; case "error": return typeof message.message === "string" + && (message.restoreBoundary === undefined || (target?.operation === "restore-openai" + && (message.restoreBoundary === "hub-owned" || message.restoreBoundary === "unproven"))) && (message.rows === undefined || (Number.isSafeInteger(message.rows) && Number(message.rows) >= 0)) && (message.files === undefined || (Number.isSafeInteger(message.files) && Number(message.files) >= 0)) && ((message.rows === undefined && message.files === undefined) @@ -307,6 +312,7 @@ function classifyWorkerResult(result: HistoryWorkerResult): CodexHistoryJobOutco kind: "failed", reason: "worker-error", message: redactWorkerMessage(result.message), + ...(result.restoreBoundary ? { restoreBoundary: result.restoreBoundary } : {}), ...(result.reason ? { historyFailureReason: result.reason } : {}), ...(result.integrityCode ? { historyIntegrityCode: result.integrityCode } : {}), ...(result.rows !== undefined && result.files !== undefined @@ -438,6 +444,7 @@ export async function runCodexHistoryJob( canonicalStateDbPath: request.canonicalStateDbPath, canonicalBackupPath: request.canonicalBackupPath, ...(request.expectedDesiredEnabled === undefined ? {} : { expectedDesiredEnabled: request.expectedDesiredEnabled }), + ...(request.guardStopOwnership ? { guardStopOwnership: true } : {}), // A Worker is a fresh module realm: it would otherwise open state_5.sqlite with this // module's default rather than the timeout this process resolved. Production sends the // same codex-rs-matching 5s the Worker would have used on its own. diff --git a/src/codex/history-provider.ts b/src/codex/history-provider.ts index c236132cc4e..e02edf40964 100644 --- a/src/codex/history-provider.ts +++ b/src/codex/history-provider.ts @@ -1368,7 +1368,7 @@ export function syncCodexHistoryProvider( provider: CodexHistoryProvider, stateDbPath = resolveCodexStateDbPath(), backupPath = resolveExistingHistoryBackupPath(stateDbPath), - opts: { skipWhenProvablyNoop?: boolean } = {}, + opts: { skipWhenProvablyNoop?: boolean; withWriteGuard?: (write: () => T) => T } = {}, ): CodexHistorySyncResult { // Opt-in steady-state gate (Design B loopback callers only): default semantics of // this exported API are unchanged — legacy stop/restore paths never pass the flag. @@ -1377,7 +1377,10 @@ export function syncCodexHistoryProvider( return { rows: 0, files: 0 }; } try { - const retried = withHistoryRetryResult(() => syncCodexHistoryProviderUnsafe(provider, stateDbPath, backupPath)); + // The guard encloses the entire synchronous attempt: DB CAS/commit, every rollout + // append and manifest consumption/finalization. No ownership publication can interleave. + const write = () => syncCodexHistoryProviderUnsafe(provider, stateDbPath, backupPath); + const retried = withHistoryRetryResult(() => opts.withWriteGuard ? opts.withWriteGuard(write) : write()); return retried.ok ? retried.value : { rows: 0, files: 0, failed: true, failureReason: retried.reason }; } catch (error) { if (error instanceof CodexHistoryIntegrityError) { diff --git a/src/codex/history-worker.ts b/src/codex/history-worker.ts index 964510b08a0..06f69ecb620 100644 --- a/src/codex/history-worker.ts +++ b/src/codex/history-worker.ts @@ -26,6 +26,7 @@ import { withHistoryWriteSerialization } from "./history-lock"; import { loadConfig } from "../config"; import { shouldSyncCodexOnStart } from "./desired-state"; +import { stopRestoreWriteGuard, type RestoreOwnershipRefusal } from "../client/restore-ownership"; import { writeHistoryProviderTransition, writeLegacyOpenaiHistoryRecovery, @@ -63,6 +64,8 @@ export interface HistoryWorkerRunMessage { readonly canonicalBackupPath: string; /** When set, prove this transition's desired direction while H is held. */ readonly expectedDesiredEnabled?: boolean; + /** Stop restoration must revalidate connection ownership inside H, under C. */ + readonly guardStopOwnership?: boolean; /** * The parent realm's `state_5.sqlite` busy timeout. A Worker cannot observe a parent that * resolved a different window, for the same reason the homes below are explicit. @@ -81,6 +84,7 @@ export type HistoryWorkerResult = readonly reason: "busy" | "database" | "unsafe-path" | "desired_disabled" | "desired_enabled" } | { readonly type: "error"; readonly requestId: string; readonly jobId: string; readonly message: string; readonly reason?: CodexHistoryFailureReason; + readonly restoreBoundary?: RestoreOwnershipRefusal; /** Specific integrity condition, so a non-retryable one can be named as such. */ readonly integrityCode?: string; readonly rows?: number; readonly files?: number }; @@ -114,6 +118,8 @@ export function isHistoryWorkerRunMessage(data: unknown): data is HistoryWorkerR && nonEmpty(message.canonicalStateDbPath) && nonEmpty(message.canonicalBackupPath) && (message.expectedDesiredEnabled === undefined || typeof message.expectedDesiredEnabled === "boolean") + && (message.guardStopOwnership === undefined + || (message.guardStopOwnership === true && message.operation === "restore-openai")) && (message.busyTimeoutMs === undefined || (typeof message.busyTimeoutMs === "number" && Number.isFinite(message.busyTimeoutMs) @@ -159,7 +165,16 @@ export function runHistoryUnitUnderLock( // manifest-backed original metadata. The provider is derived from the operation, // never from a caller; only recover-legacy-openai force-labels bare routed rows. const provider = operation === "apply-opencodex" ? "opencodex" : "openai"; - return writeHistoryProviderTransition(permit, target, provider); + const guard = message.guardStopOwnership ? stopRestoreWriteGuard() : undefined; + try { + // H is already held. The provider takes C only for synchronous attempts, so + // there is no C -> H edge and no C lease spans the parent's Worker await. + return writeHistoryProviderTransition(permit, target, provider, guard?.run); + } catch (error) { + const boundary = guard?.boundary(); + if (boundary && boundary !== "restore") return { restoreBoundary: boundary }; + throw error; + } }, ); @@ -167,6 +182,10 @@ export function runHistoryUnitUnderLock( return { type: "blocked", requestId, jobId, reason: acquired.reason }; } const result = acquired.value; + if ("restoreBoundary" in result) { + return { type: "error", requestId, jobId, message: `stop_restore_${result.restoreBoundary}`, + restoreBoundary: result.restoreBoundary }; + } if ("desiredStateChanged" in result) { return { type: "blocked", diff --git a/src/codex/inject/restore.ts b/src/codex/inject/restore.ts index 79e45d53328..75f7b5a6249 100644 --- a/src/codex/inject/restore.ts +++ b/src/codex/inject/restore.ts @@ -2,6 +2,8 @@ import { restoreApprovedCodexProviders } from "../provider-routing"; import { beginCodexWriteSection, cleanExternalProviderJournal, publishCodexArtifact } from "./config-write-section"; import { assertCodexHomeOwner, codexHomeIsAbsent, codexHomeOwnerBlocksCompensation, CodexHomeOwnerRefusal, type CodexHomeOwnerRefusalReason } from "../codex-home-owner"; import { loadConfig } from "../../config"; +import { StopRestoreOwnershipError } from "../../client/restore-ownership"; +import { withHistoryWriteSerialization } from "../history-lock"; import { shouldSyncCodexOnStart } from "../desired-state"; import { siblingOfLivePort, siblingSkipMessage } from "../sibling-start"; import { withCatalogWriteSerialization } from "../catalog-write-serialization"; @@ -371,6 +373,12 @@ export function resolveRestoreHistoryDisposition( export interface RestoreConfigOptions { /** Strip `[model_providers.opencodex]` too, accepting that tagged threads stop opening. */ removeProviderTable?: boolean; + /** Synchronous caller policy and exclusion, repeated at every async restore write boundary. */ + withWriteGuard?: (write: () => T) => T; +} + +function guardedRestoreWrite(options: RestoreConfigOptions, write: () => T): T { + return options.withWriteGuard ? options.withWriteGuard(write) : write(); } /** The config/profile half of a native restore, reported as one artifact. */ @@ -385,7 +393,7 @@ function restoreCodexConfigInline( * shared write lock. Splitting it let a foreign write land between the pieces * and left preimage compensation replaying over bytes that writer committed. */ - const locked = withConfigWriteLockHeld(CODEX_CONFIG_PATH, heldConfigWriteLock, (held) => { + const locked = withConfigWriteLockHeld(CODEX_CONFIG_PATH, heldConfigWriteLock, (held) => guardedRestoreWrite(options, () => { beginCodexWriteSection(held); assertCodexHomeOwner(getCodexHome()); beforeRestoreConfigForTests?.(kind); @@ -397,7 +405,7 @@ function restoreCodexConfigInline( if (!compensated.complete) throw new CodexPartialWriteError(compensated.unrestored); } return result; - }); + })); if (!locked.ok) { // Nothing was captured or written — there is nothing to compensate. return { @@ -530,13 +538,14 @@ function restoreCodexConfigInlineImpl( function restoreCodexCatalogArtifact( revalidateDesiredState: boolean, journaledCatalogPath: string | null, + options: RestoreConfigOptions = {}, ): CodexRestoreCatalogResult { const owningCodexHome = getCodexHome(); try { - const restored = withCatalogWriteSerialization(owningCodexHome, permit => + const restored = withCatalogWriteSerialization(owningCodexHome, permit => guardedRestoreWrite(options, () => revalidateDesiredState && shouldSyncCodexOnStart(loadConfig()) ? null - : restoreCodexCatalogWithPermit(permit, owningCodexHome, journaledCatalogPath), + : restoreCodexCatalogWithPermit(permit, owningCodexHome, journaledCatalogPath)), { intent: "restore", writer: "codex-restore" }); return restored.kind === "completed" && restored.value !== null ? { state: "ok", changed: restored.value.removed > 0, ...restored.value, message: "Codex catalog restored." } @@ -568,14 +577,15 @@ function restoreCodexCatalogArtifact( * that lost race into the discriminated `desired_enabled` skip. */ export async function restoreNativeCodexAsync( - options: { revalidateDesiredState?: boolean; removeProviderTable?: boolean } = {}, + options: RestoreConfigOptions & { revalidateDesiredState?: boolean } = {}, ): Promise { if (codexHomeIsAbsent(getCodexHome())) return skippedRestoreEnvelope(true, "Codex config not found; no native restore was needed."); const sibling = siblingRestoreSkip(); if (sibling) return sibling; try { assertCodexHomeOwner(getCodexHome()); - return finishApprovedProviderRestore(await restoreNativeCodexAsyncImpl(options), options.revalidateDesiredState); + const result = await restoreNativeCodexAsyncImpl(options); + return guardedRestoreWrite(options, () => finishApprovedProviderRestore(result, options.revalidateDesiredState)); } catch (error) { if (error instanceof CodexHomeOwnerRefusal) return homeOwnerRestoreRefusal(error); if (!(error instanceof CodexRestoreRefusal)) throw error; @@ -584,14 +594,14 @@ export async function restoreNativeCodexAsync( } async function restoreNativeCodexAsyncImpl( - options: { revalidateDesiredState?: boolean; removeProviderTable?: boolean }, + options: RestoreConfigOptions & { revalidateDesiredState?: boolean }, ): Promise { const activeProvider = currentExternalCodexModelProvider(); if (activeProvider) { // External-provider courtesy: only the stale journal is removed. The // history worker must not launch — it would turn a read-mostly courtesy // result into a history mutation on a home we do not own. - cleanExternalProviderJournal(); + guardedRestoreWrite(options, () => cleanExternalProviderJournal()); return externalProviderRestoreResult(activeProvider); } @@ -653,7 +663,7 @@ async function restoreNativeCodexAsyncImpl( admitted: witness, readAdmissionUnderLock: () => witness, }, - (ctx) => { + (ctx) => guardedRestoreWrite(options, () => { assertCodexHomeOwner(getCodexHome()); if (options.revalidateDesiredState && shouldSyncCodexOnStart(loadConfig())) { throw new CodexWriteLockSkipped("desired_enabled"); @@ -695,7 +705,7 @@ async function restoreNativeCodexAsyncImpl( currentTxId: ctx.expectation.txId, }, }; - }, + }), ); if (coordinated.status === "skipped") return desiredEnabledRestoreSkip(); if (coordinated.status !== "acquired") { @@ -708,10 +718,10 @@ async function restoreNativeCodexAsyncImpl( : `Codex configuration was not restored: ${coordinated.message}`, }; } else { - recordCodexNativeTransactionProvenance( + guardedRestoreWrite(options, () => recordCodexNativeTransactionProvenance( coordinated.value.preImages, coordinated.value.receipt.currentTxId, - ); + )); config = coordinated.value.config; transitionReceipt = coordinated.value.receipt; } @@ -729,8 +739,10 @@ async function restoreNativeCodexAsyncImpl( } if (config.state === "failed") return failedConfigRestoreEnvelope(config); - notifyCatalogPublication({ kind: "native-released", path: journaledCatalogPath }); - const catalog = restoreCodexCatalogArtifact(options.revalidateDesiredState === true, journaledCatalogPath); + guardedRestoreWrite(options, () => notifyCatalogPublication({ kind: "native-released", path: journaledCatalogPath })); + const catalog = restoreCodexCatalogArtifact(options.revalidateDesiredState === true, journaledCatalogPath, options); + // Catalog failures are artifact results; a caller-policy refusal must also stop later work. + guardedRestoreWrite(options, () => {}); // Re-asked after the config half, because the store can paginate mid-transaction. Deciding // the history job from the pre-write answer alone would spawn a Worker whose own preflight // is now guaranteed to refuse, and report that refusal as a restore failure on a home that @@ -742,8 +754,15 @@ async function restoreNativeCodexAsyncImpl( : await runCodexHistoryJob({ ...resolveCodexHistoryJobTarget(), ...(options.revalidateDesiredState ? { expectedDesiredEnabled: false } : {}), + ...(options.withWriteGuard ? { guardStopOwnership: true } : {}), operation: deriveCodexHistoryOperation({ direction: "restore", resumeHistory: true, legacyMode: false }), }); + if (outcome.kind === "failed" && outcome.restoreBoundary) { + // Preserve the Worker's observed refusal even if the connection changes again + // before the parent resumes. The parent guard latches this typed boundary. + const boundary = outcome.restoreBoundary; + guardedRestoreWrite(options, () => { throw new StopRestoreOwnershipError(boundary); }); + } if (transitionReceipt) { resolveCodexHistoryTransition(transitionReceipt, outcome); } @@ -797,14 +816,15 @@ function homeOwnerRestoreRefusal(error: CodexHomeOwnerRefusal): CodexNativeResto } export function restoreNativeCodex( - options: { skipHistory?: boolean; revalidateDesiredState?: boolean; removeProviderTable?: boolean } = {}, + options: RestoreConfigOptions & { skipHistory?: boolean; revalidateDesiredState?: boolean } = {}, ): CodexNativeRestoreResult { if (codexHomeIsAbsent(getCodexHome())) return skippedRestoreEnvelope(true, "Codex config not found; no native restore was needed."); const sibling = siblingRestoreSkip(); if (sibling) return sibling; try { assertCodexHomeOwner(getCodexHome()); - return finishApprovedProviderRestore(restoreNativeCodexImpl(options), options.revalidateDesiredState); + const result = restoreNativeCodexImpl(options); + return guardedRestoreWrite(options, () => finishApprovedProviderRestore(result, options.revalidateDesiredState)); } catch (error) { if (error instanceof CodexHomeOwnerRefusal) return homeOwnerRestoreRefusal(error); throw error; @@ -812,11 +832,11 @@ export function restoreNativeCodex( } function restoreNativeCodexImpl( - options: { skipHistory?: boolean; revalidateDesiredState?: boolean; removeProviderTable?: boolean } = {}, + options: RestoreConfigOptions & { skipHistory?: boolean; revalidateDesiredState?: boolean } = {}, ): CodexNativeRestoreResult { const activeProvider = currentExternalCodexModelProvider(); if (activeProvider) { - cleanExternalProviderJournal(); + guardedRestoreWrite(options, () => cleanExternalProviderJournal()); return externalProviderRestoreResult(activeProvider); } if (options.revalidateDesiredState && shouldSyncCodexOnStart(loadConfig())) { @@ -831,11 +851,12 @@ function restoreNativeCodexImpl( const journaledCatalogPath = journaledInjectedCatalogPath() ?? readCodexCatalogPath(); const config = restoreCodexConfigInline("sync", options); if (config.state === "failed") return failedConfigRestoreEnvelope(config); - notifyCatalogPublication({ kind: "native-released", path: journaledCatalogPath }); + guardedRestoreWrite(options, () => notifyCatalogPublication({ kind: "native-released", path: journaledCatalogPath })); // Same mid-transaction pagination re-check as the async path. const historyStoodDown = historyStandsDown || resolveRestoreHistoryDisposition(options.removeProviderTable).kind === "stand-down"; - const catalog = restoreCodexCatalogArtifact(options.revalidateDesiredState === true, journaledCatalogPath); + const catalog = restoreCodexCatalogArtifact(options.revalidateDesiredState === true, journaledCatalogPath, options); + guardedRestoreWrite(options, () => {}); // Design B (loopback) steady state: threads are already tagged openai, so prove the // no-op with a readonly probe instead of write-opening a DB the Codex app may hold // (Windows: WAL writer lock -> seconds of stalling + a false warning on every stop). @@ -848,11 +869,22 @@ function restoreNativeCodexImpl( } // `skipHistory` is how the async wrapper takes this work for itself: the // native files come down here, and history runs in the Worker under H. + let historyLockFailure: CodexRestoreHistoryResult | undefined; const rawHistory = options.skipHistory || historyStoodDown ? { rows: 0, files: 0 } - : syncCodexHistoryProvider("openai", undefined, undefined, { - skipWhenProvablyNoop, - }); + : (() => { + const target = resolveCodexHistoryJobTarget(); + const restore = () => syncCodexHistoryProvider("openai", target.canonicalStateDbPath, target.canonicalBackupPath, { + skipWhenProvablyNoop, withWriteGuard: options.withWriteGuard, + }); + if (!options.withWriteGuard) return restore(); + const locked = withHistoryWriteSerialization(target.canonicalCodexHome, target.canonicalStateDbPath, () => + restore()); + if (locked.kind === "completed") return locked.value; + // H availability says nothing about connection ownership; Grok must still clean up. + historyLockFailure = failedHistoryRestoreFromOutcome({ kind: "blocked", reason: locked.reason }); + return { rows: 0, files: 0 }; + })(); const history: CodexRestoreHistoryResult = historyStoodDown ? { state: "skipped", changed: false, rows: 0, files: 0, ejectedRows: 0, @@ -861,7 +893,7 @@ function restoreNativeCodexImpl( } : options.skipHistory ? { state: "skipped", changed: false, rows: 0, files: 0, ejectedRows: 0, message: "History restoration runs asynchronously." } - : rawHistory.failed + : historyLockFailure ?? (rawHistory.failed ? failedHistoryRestore(rawHistory.failureReason, undefined, rawHistory) : { state: "ok", @@ -872,7 +904,7 @@ function restoreNativeCodexImpl( message: rawHistory.rows > 0 ? `Resume history metadata restored from opencodex backup (${rawHistory.rows} thread(s)); original providers preserved.` : "No backed-up resume-history metadata was pending; untracked routed history was left unchanged.", - }; + }); const message = catalog.removed > 0 ? `${config.message} Catalog restored to ${catalog.kept} native model(s) (dropped ${catalog.removed} proxy-routed).` : config.message; diff --git a/src/codex/internal/history-writer.ts b/src/codex/internal/history-writer.ts index 06af277989f..cf9d30110dd 100644 --- a/src/codex/internal/history-writer.ts +++ b/src/codex/internal/history-writer.ts @@ -59,9 +59,10 @@ export function writeHistoryProviderTransition( permit: HistoryWritePermit, target: HistoryWriteTarget, provider: CodexHistoryProvider, + withWriteGuard?: (write: () => T) => T, ): CodexHistorySyncResult { assertHistoryWritePermit(permit, target.canonicalStateDbPath); - return syncCodexHistoryProvider(provider, target.canonicalStateDbPath, target.canonicalBackupPath); + return syncCodexHistoryProvider(provider, target.canonicalStateDbPath, target.canonicalBackupPath, { withWriteGuard }); } /** diff --git a/src/lib/process-control.ts b/src/lib/process-control.ts index 7b791e4915e..94fd9dcab2e 100644 --- a/src/lib/process-control.ts +++ b/src/lib/process-control.ts @@ -41,6 +41,18 @@ export interface GracefulStopIo { * and keep the self-contained behaviour. */ deferSharedTeardownNonce?: string; + /** + * Whether a `sharedTeardown: "not-owned"` answer confirms this stop's teardown. + * + * A sibling's `not-owned` is a truthful final answer: it owns no shared teardown, so + * nothing is left for this caller either. `ocx stop` of a connected Link Child sets this + * false — only an old build answers `not-owned` for a link connection, and it says so + * while preserving the loopback route this caller must restore itself. The caller decides + * from the connection state read at the POST: a connect or disconnect committed mid-stop + * changes what the answer means for it. Defaults to true; the option only narrows a stop + * that sent no deferral nonce. + */ + acceptNotOwnedTeardown?: boolean; /** * Endpoint the caller already resolved for this pid. * @@ -204,7 +216,7 @@ export async function stopProxyGracefully(pid: number, io: GracefulStopIo = {}): */ type StopRefusal = { message: string | null; code: string | null }; -async function stopRequestFetch(input: string | URL | Request, init?: RequestInit): Promise { +export async function stopRequestFetch(input: string | URL | Request, init?: RequestInit): Promise { let status: number | undefined; try { return await directLocalHttpFetch(input, init, { onResponseStatus: value => { status = value; } }); @@ -275,12 +287,16 @@ async function stopProxyGracefullyDetailed( const expectedTeardown = io.deferSharedTeardownNonce ? "deferred" : "performed"; // A sibling instance answers `not-owned`: it has no shared teardown to perform, so there is // nothing left over for this caller either. A deferral it was sent is still not confirmed. + // A connected Link Child's stop refuses the same answer: only an old build sends it for a + // link connection, and it preserves the route this caller must restore. sharedTeardownConfirmed = body !== null && typeof body === "object" && !Array.isArray(body) && "success" in body && body.success === true && "sharedTeardown" in body && (body.sharedTeardown === expectedTeardown - || (!io.deferSharedTeardownNonce && body.sharedTeardown === "not-owned")); + || (!io.deferSharedTeardownNonce + && (io.acceptNotOwnedTeardown ?? true) + && body.sharedTeardown === "not-owned")); } catch { return done(false); } diff --git a/src/server/lifecycle.ts b/src/server/lifecycle.ts index 57421fb29e5..e4e74277b86 100644 --- a/src/server/lifecycle.ts +++ b/src/server/lifecycle.ts @@ -520,6 +520,7 @@ export function trackStreamLifetime( export async function drainAndShutdown( server: ReturnType | undefined, timeoutMs: number, + beforeListenerStop?: () => undefined, ): Promise { const s = server ?? _serverRef; let shutdownSucceeded = true; @@ -605,7 +606,13 @@ export async function drainAndShutdown( setStorageCleanupPolicyJobLiveApply(null); } finally { try { - await stopServerListener(s); + // Signal cleanup restores owned client routing while this listener still owns its port. + // Cleanup is synchronous only; a throw still releases listener and startup ownership. + try { + beforeListenerStop?.(); + } finally { + await stopServerListener(s); + } } finally { await releaseServerStartupLifecycle(s); // shutdownDraining is a process-lifetime latch. A stopped server must diff --git a/src/server/management-api.ts b/src/server/management-api.ts index c51b21b34e4..0e52a0c65b5 100644 --- a/src/server/management-api.ts +++ b/src/server/management-api.ts @@ -50,7 +50,7 @@ import { import type { OcxClaudeCodeConfig, OcxClaudeDesktopProfile, OcxConfig, OcxCustomModel, OcxProviderConfig } from "../types"; import type { DesktopProfileModel } from "../claude/desktop-profile"; import { drainAndShutdown } from "./lifecycle"; -import { noteExplicitShutdownRequested } from "./management/system-restart"; +import { acquireExplicitShutdownIntent, noteExplicitShutdownRequested } from "./management/system-restart"; import { filterRequestLogs, getRequestLogEntries, type RequestLogEntry } from "./request-log"; import { estimateComboCost, estimateRequestCost, normalizeCostTokens, tokensPerSecond } from "../usage/cost"; import type { PersistedUsageAttempt } from "../usage/log"; @@ -436,7 +436,7 @@ export async function handleManagementAPI( // outcome. This process cannot verify its own post-exit respawn window; only the // receipt-backed parent `ocx stop` can, which is what the deferral exists for. const { deferralMatchesReceipt } = await import("../config/pending-teardown"); - const { deferralHonored, desktopSupervisedStopRefusal, performStopTeardown } = await import("./stop-teardown"); + const { deferralHonored, desktopSupervisedStopRefusal, performStopTeardown, preflightStopTeardownOwnership } = await import("./stop-teardown"); // The desktop app would start this proxy again within seconds; refuse before anything is // touched and point at its tray, whose Stop it honours (#3008 refuses an undone stop the same way). const desktopRefusal = desktopSupervisedStopRefusal(principal); @@ -477,58 +477,65 @@ export async function handleManagementAPI( message: "The Windows Task Scheduler state could not be read, so this proxy cannot tell whether a wrapper would respawn it. Nothing was changed. Run `ocx service status` to see the query error, repair Task Scheduler access, then retry.", }, 409, req, config); } - let serviceStop: import("../service").ServiceStopOutcome; + // Block automatic handoff while deciding, without committing a refused stop. + const releaseStopIntent = acquireExplicitShutdownIntent(); try { - serviceStop = sibling ? "absent" : stopServiceIfInstalledDetailed(); - } catch (err) { - if (isServiceOwnershipError(err)) { - // The installed service belongs to another CODEX_HOME/OPENCODEX_HOME: it would respawn - // this proxy immediately, and its shared config is not ours to tear down. Refuse the - // stop instead of half-performing it. 409, not 500 — the request is well-formed. - return jsonResponse({ success: false, message: err.message }, 409, req, config); + if (!holdsReceipt && !sibling) { + const refusal = preflightStopTeardownOwnership(); + if (refusal) return jsonResponse(refusal, 409, req, config); } - throw err; - } - // The boolean helper collapses "failed" into the same false as "no service installed", - // so this route used to tear down shared config and exit while a manager that refused - // to stop was still there to respawn the proxy (#3008). - if (serviceStop === "failed") { - return jsonResponse({ - success: false, - message: "The installed service manager did not stop; it may respawn the proxy. Shared client config was left alone. Run `ocx stop` from the home that owns the service.", - }, 409, req, config); - } - if (serviceStop === "state-unknown") { - // Same case, same remedy as the pre-check: the query is what needs fixing. - return jsonResponse({ - success: false, - code: "service_state_unknown", - message: "The Windows Task Scheduler state could not be read, so this proxy cannot tell whether a wrapper would respawn it. Shared client config was left alone. Run `ocx service status` to see the query error, repair Task Scheduler access, then retry.", - }, 409, req, config); - } - // The pre-check above already refused the respawnable case without a receipt, so - // reaching here with one means the parent owns the verification. - // Both managed configs come down together on an explicit teardown. The daemon's own - // syncCleanup skips this when OCX_SERVICE is set (so a crash/respawn keeps the fence), - // which is exactly why an intentional stop has to do it here — unless the caller is - // `ocx stop`, which does it itself once the proxy is proven down. - // Mark the stop before the first await after acceptance, so an automatic restart draining - // concurrently cannot reach its handoff while teardown is still pending. - noteExplicitShutdownRequested(); - const teardown = await performStopTeardown(url, { ownsReceipt: deferralMatchesReceipt }); - setTimeout(async () => { - let shutdownSucceeded = false; + let serviceStop: import("../service").ServiceStopOutcome; try { - shutdownSucceeded = await drainAndShutdown(undefined, config.shutdownTimeoutMs ?? 5000); - } catch { - console.warn("[opencodex] shutdown drain failed"); + serviceStop = sibling ? "absent" : stopServiceIfInstalledDetailed(); + } catch (err) { + if (isServiceOwnershipError(err)) { + // The installed service belongs to another CODEX_HOME/OPENCODEX_HOME: it would respawn + // this proxy immediately, and its shared config is not ours to tear down. Refuse the + // stop instead of half-performing it. 409, not 500 — the request is well-formed. + return jsonResponse({ success: false, message: err.message }, 409, req, config); + } + throw err; } - // A drained proxy whose shared teardown failed did not finish the job. Exiting 0 - // told a supervisor the stop was clean while native Codex or the Grok fence was - // still pointed at this process (#3008). - process.exit(shutdownSucceeded && teardown.success ? 0 : 1); - }, 200); - return jsonResponse(teardown); + // The boolean helper collapses "failed" into the same false as "no service installed", + // so this route used to tear down shared config and exit while a manager that refused + // to stop was still there to respawn the proxy (#3008). + if (serviceStop === "failed") { + return jsonResponse({ + success: false, + message: "The installed service manager did not stop; it may respawn the proxy. Shared client config was left alone. Run `ocx stop` from the home that owns the service.", + }, 409, req, config); + } + if (serviceStop === "state-unknown") { + // Same case, same remedy as the pre-check: the query is what needs fixing. + return jsonResponse({ + success: false, + code: "service_state_unknown", + message: "The Windows Task Scheduler state could not be read, so this proxy cannot tell whether a wrapper would respawn it. Shared client config was left alone. Run `ocx service status` to see the query error, repair Task Scheduler access, then retry.", + }, 409, req, config); + } + // The pre-check above already refused the respawnable case without a receipt, so + // reaching here with one means the parent owns the verification. + // Both managed configs come down together on an explicit teardown. The daemon's own + // syncCleanup skips this when OCX_SERVICE is set (so a crash/respawn keeps the fence), + // which is exactly why an intentional stop has to do it here — unless the caller is + // `ocx stop`, which does it itself once the proxy is proven down. + const teardown = await performStopTeardown(url, { ownsReceipt: deferralMatchesReceipt }); + if (teardown.sharedTeardown === "refused") return jsonResponse(teardown, 409, req, config); + noteExplicitShutdownRequested(); // Accepted stops and signals remain irreversible. + setTimeout(async () => { + let shutdownSucceeded = false; + try { + shutdownSucceeded = await drainAndShutdown(undefined, config.shutdownTimeoutMs ?? 5000); + } catch { + console.warn("[opencodex] shutdown drain failed"); + } + // A drained proxy whose shared teardown failed did not finish the job. Exiting 0 + // told a supervisor the stop was clean while native Codex or the Grok fence was + // still pointed at this process (#3008). + process.exit(shutdownSucceeded && teardown.success ? 0 : 1); + }, 200); + return jsonResponse(teardown); + } finally { releaseStopIntent(); } } if (url.pathname.startsWith("/api/native-main-profiles")) { diff --git a/src/server/management/system-restart.ts b/src/server/management/system-restart.ts index 3060978d207..0f233f99a3c 100644 --- a/src/server/management/system-restart.ts +++ b/src/server/management/system-restart.ts @@ -170,13 +170,31 @@ function waitForBoundedSettlement( } /** - * Set when an operator-initiated shutdown (signal or management stop) begins. An automatic - * restart that is already draining cannot tell its own listener stop from an independent one, - * so it consults this instead and never hands off after the process was asked to stop. + * Accepted stops and signals permanently veto automatic handoff. Management requests carry + * separate temporary intents while teardown awaits, so a refusal can release its own intent + * without clearing another pending stop, an accepted stop, or a signal. */ let explicitShutdownRequested = false; +const pendingShutdownIntents = new Set(); +const shutdownIntentWaiters = new Set<() => void>(); +function shutdownIntentChanged(): void { + const waiters = [...shutdownIntentWaiters]; + shutdownIntentWaiters.clear(); + for (const resolve of waiters) resolve(); +} +function nextShutdownIntentChange(): Promise { + return new Promise(resolve => { shutdownIntentWaiters.add(resolve); }); +} export function noteExplicitShutdownRequested(): void { explicitShutdownRequested = true; + shutdownIntentChanged(); +} + +/** Veto automatic handoff while a stop is pending; release only this caller's intent. */ +export function acquireExplicitShutdownIntent(): () => void { + const intent = {}; + pendingShutdownIntents.add(intent); + return () => { if (pendingShutdownIntents.delete(intent)) shutdownIntentChanged(); }; } /** Test seam — reset between tests. */ @@ -185,6 +203,8 @@ export function setSystemRestartIoForTests(io: SystemRestartIo = {}): void { restartAccepted = false; acceptedDrainGraceMs = MEMORY_DRAIN_RESTART_MS; explicitShutdownRequested = false; + pendingShutdownIntents.clear(); + shutdownIntentChanged(); } /** The port this process is listening on, or undefined when that cannot be established. */ @@ -281,7 +301,9 @@ async function completeDeferredParentExitHandoff( port: number | undefined, phase: "deadline" | "listener-stop fallback", canHandoff: () => boolean = () => true, + automatic = false, ): Promise { + while (automatic && pendingShutdownIntents.size > 0 && !explicitShutdownRequested) await nextShutdownIntentChange(); if (!canHandoff()) return; try { await (io.spawnStart ?? spawnDetachedStart)(port, false); @@ -304,7 +326,9 @@ async function completeDeadlineRestartHandoff( port: number | undefined, scheduleDeadline: NonNullable, canHandoff: () => boolean = () => true, + automatic = false, ): Promise { + while (automatic && pendingShutdownIntents.size > 0 && !explicitShutdownRequested) await nextShutdownIntentChange(); if (!canHandoff()) return; if (handOffToDesktop(io, exitProcess)) return; const supervised = (io.isSupervisedServiceChild ?? (() => isSupervisedServiceChild(io)))(); @@ -330,7 +354,7 @@ async function completeDeadlineRestartHandoff( } // The ordinary child must survive parent exit: a failed/pending socket close // or overdue cleanup is completed by process teardown, without a hidden mode. - await completeDeferredParentExitHandoff(io, exitProcess, port, "deadline", canHandoff); + await completeDeferredParentExitHandoff(io, exitProcess, port, "deadline", canHandoff, automatic); } /** @@ -378,6 +402,7 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: pending = false; vetoed = true; releasePending(); + shutdownIntentChanged(); }); const now = io.now ?? Date.now; const acceptedAtMs = now(); @@ -402,12 +427,15 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: }, drainGraceMs); } schedule(async () => { + if (vetoed) return; + // Keep the one admitted restart queued until every temporary stop intent settles. + while (automatic && pendingShutdownIntents.size > 0 && !explicitShutdownRequested && !vetoed) await nextShutdownIntentChange(); if (vetoed) return; pending = false; const canHandoff = () => { // Only admission-bound (automatic) restarts yield to an explicit stop; manual restart // requests keep their existing semantics. - if (admission.onAccepted && explicitShutdownRequested) return false; + if (admission.onAccepted && (explicitShutdownRequested || pendingShutdownIntents.size > 0)) return false; try { return admission.beforeScheduledDrain?.() ?? true; } catch { return false; } // Unknown ownership is not authority to restart. }; @@ -426,12 +454,14 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: const drainPromise = Promise.resolve().then( () => drain(undefined, Math.max(0, drainDeadlineMs - now())), ); - const drainOutcome = await waitForRestartDrain(drainPromise, restartDeadlineMs, now, scheduleDeadline); + let drainOutcome = await waitForRestartDrain(drainPromise, restartDeadlineMs, now, scheduleDeadline); + while (automatic && pendingShutdownIntents.size > 0 && !explicitShutdownRequested) await nextShutdownIntentChange(); if (!canHandoff()) return; + if (automatic && now() >= restartDeadlineMs) drainOutcome = "deadline"; const exitProcess = io.exitProcess ?? ((code: number) => { process.exit(code); }); if (drainOutcome === "deadline") { console.warn("Drain-and-restart deadline expired; forcing terminal restart handoff"); - await completeDeadlineRestartHandoff(io, exitProcess, restartPort, scheduleDeadline, canHandoff); + await completeDeadlineRestartHandoff(io, exitProcess, restartPort, scheduleDeadline, canHandoff, automatic); return; } if (drainOutcome === "failed" || drainOutcome === "rejected") { @@ -457,11 +487,17 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: restartPort, "listener-stop fallback", canHandoff, + automatic, ); return; } try { + while (automatic && pendingShutdownIntents.size > 0 && !explicitShutdownRequested) await nextShutdownIntentChange(); if (!canHandoff()) return; + if (automatic && now() >= restartDeadlineMs) { + await completeDeferredParentExitHandoff(io, exitProcess, restartPort, "deadline", canHandoff, automatic); + return; + } // A rejected drain has uncertain cleanup ownership, so it uses the same // parent-exit handoff as a deadline. Only a fully completed drain waits // for replacement health in the old process. diff --git a/src/server/stop-teardown.ts b/src/server/stop-teardown.ts index 515743e73a0..975396a116c 100644 --- a/src/server/stop-teardown.ts +++ b/src/server/stop-teardown.ts @@ -3,6 +3,8 @@ import { siblingOfLivePort, siblingSkipMessage } from "../codex/sibling-start"; import { deferralMatchesReceipt } from "../config/pending-teardown"; import { isDesktopSupervised } from "../lib/system-restart-contract"; import type { ManagementPrincipal } from "./management-auth"; +import type { RestoreConfigOptions } from "../codex/inject/restore"; +import { stopRestoreWriteGuard } from "../cli/stop-restore-guard"; /** * Shared-teardown decision and execution for `POST /api/stop` (#3008). @@ -15,6 +17,11 @@ import type { ManagementPrincipal } from "./management-auth"; export type GrokStripResult = { ok: boolean; changed: boolean; message: string }; +// A verified receipt transfers shared restoration to the parent until this process exits. +// Keep this separate from recycling and connected-client capability/ownership decisions. +let sharedTeardownDeferredOnExit = false; +export function isSharedTeardownDeferredOnExit(): boolean { return sharedTeardownDeferredOnExit; } + /** Read-only promise covering API teardown and the CLI runtime's signal/manager exit cleanup. */ export function stopCapabilities() { return { schema: "ocx-stop-capabilities/1", connectedClientOwnership: true, pid: process.pid }; @@ -27,7 +34,7 @@ export type StopTeardownIo = { isHandoverReceipt?: (nonce: string | null) => boolean; /** Keep Codex, Grok and env fences in place when this process exits. */ keepClientConfigOnExit?: () => void; - restoreNativeCodex?: () => Promise; + restoreNativeCodex?: (options?: RestoreConfigOptions) => Promise; stripGrok?: () => GrokStripResult; }; @@ -35,9 +42,24 @@ export type StopTeardownBody = { success: boolean; message: string; /** `not-owned`: shared routing belongs to a live sibling owner or a client connection. */ - sharedTeardown: "deferred" | "performed" | "not-owned"; + sharedTeardown: "deferred" | "performed" | "not-owned" | "refused"; }; +function stopOwnershipRefusal(): StopTeardownBody { + return { success: false, message: "Stop refused: client connection ownership may be changing; remaining restoration and obligations are preserved. Retry once connect or disconnect finishes.", sharedTeardown: "refused" }; +} + +/** Read under C before manager mutation; every later write still revalidates independently. */ +export function preflightStopTeardownOwnership(): StopTeardownBody | null { + const guard = stopRestoreWriteGuard(); + try { guard.run(() => {}); } + catch (error) { + if (guard.boundary() === "unproven") return stopOwnershipRefusal(); + if (guard.boundary() !== "hub-owned") throw error; + } + return null; +} + /** * A deferral is honoured only when the caller proves it owns the obligation. * @@ -91,7 +113,9 @@ export async function performStopTeardown(url: URL, io: StopTeardownIo = {}): Pr // A connection owns its integrations across stop/start, even for a successful API stop. // This is persisted authority, not a caller's request to skip an obligation it owns. const connection = (await import("../client/state")).readClientConnectionState(); - if (connection.kind === "connected") { + // A Link Child's route is an unauthenticated loopback listener. Once this process exits, + // preserving that route would let another local process claim the released port. + if (connection.kind === "connected" && connection.value.transport !== "link") { return { success: true, message: "Proxy stopping; connection-owned Codex routing, catalog and Grok config preserved. Run `ocx disconnect` to release them.", @@ -100,6 +124,7 @@ export async function performStopTeardown(url: URL, io: StopTeardownIo = {}): Pr } const ownsReceipt = io.ownsReceipt ?? deferralMatchesReceipt; if (deferralHonored(url, ownsReceipt)) { + sharedTeardownDeferredOnExit = true; // A handover outlives this process: the exit cleanup would otherwise restore native // Codex and put the pristine catalog back right after the stop kept them (#7423). // Exiting as a recycle leaves them for the replacement, as a dashboard restart does. @@ -124,12 +149,24 @@ export async function performStopTeardown(url: URL, io: StopTeardownIo = {}): Pr sharedTeardown: "deferred", }; } - const restore = io.restoreNativeCodex - ? await io.restoreNativeCodex() - : await (await import("../codex/inject")).restoreNativeCodexAsync(); - const grok = io.stripGrok - ? io.stripGrok() - : (await import("../grok/inject")).stripGrokConfig(); + const guard = stopRestoreWriteGuard(); + let restore: CodexNativeRestoreResult; + let grok: GrokStripResult; + try { + guard.run(() => {}); // Refuse pending ownership before entering an asynchronous writer. + const options = { withWriteGuard: guard.run }; + restore = io.restoreNativeCodex + ? await io.restoreNativeCodex(options) + : await (await import("../codex/inject")).restoreNativeCodexAsync(options); + const strip = io.stripGrok ?? (await import("../grok/inject")).stripGrokConfig; + grok = guard.run(() => strip()); + } catch (error) { + const boundary = guard.boundary(); + if (boundary === "restore") throw error; + return boundary === "hub-owned" + ? { success: true, message: "Proxy stopping; a hub connection owns Codex routing, catalog and Grok config, so restoration stopped.", sharedTeardown: "not-owned" } + : stopOwnershipRefusal(); + } // Success means BOTH halves came down. Deciding it from the native restore alone and // appending the Grok text let a caller read `success: true` while the fence still // pointed at a proxy that was exiting — the teardown reported done with half of it diff --git a/structure/codex-home.md b/structure/codex-home.md index 30459ff663e..2da166cd57b 100644 --- a/structure/codex-home.md +++ b/structure/codex-home.md @@ -297,7 +297,7 @@ restore still fails closed when a recorded logical entry has no surviving file. OpenCodex config writers share `config.toml.ocx-write.lock`: feature scalar and CLI-toggle batches, injection, journal replay, removal, and restore/compensation read and write while -holding it. Injection and coordinated restore take config before SQLite N; prompt writes +holding it. Parent stop adds [connection validation under C](runtime.md#remote-hub-hardening-ownership). Injection and coordinated restore take config before SQLite N; prompt writes take prompt-store before config. Nested writers receive an explicit live handle for the same canonical config destination; implicit nesting refuses fast. Symlink aliases contend on the real destination's lock. Acquisition revalidates the canonical path and captures the inode after the lock is held, admitting a preceding writer's completed atomic replacement while refusing a canonical retarget before entering the section. Each publication revalidates canonical identity and inode immediately before rename, @@ -549,7 +549,7 @@ The [probe contract](codex-prompts.md#prompt-text-probe) governs prompt capture ## Paginated history writer boundary -`src/codex/history-provider.ts` rejects provider-history changes with `history_paginated_requires_native_writer` when a target begins with an ordinal-bearing record, later contains a paginated record after a legacy start (#4311), or declares `history_mode=paginated`. The first line alone is not sufficient: a rollout that started unnumbered and was later migrated is also refused. Apply, manifest-backed restore, and explicit legacy recovery preflight all selected targets before changing database rows or manifests. The append boundary checks again. Codex owns ordinal allocation and the live projection cursor; reading the last ordinal and appending N+1 is not safe concurrent coordination. Legacy unnumbered rollouts retain their existing behavior. History Worker targets resolve the canonical manifest first and an existing pre-normalization Windows filename second, so passing an explicit target cannot bypass upgrade recovery. This guard prevents the observed stable-format corruption; it does not implement native-writer integration. +`src/codex/history-provider.ts` rejects provider-history changes with `history_paginated_requires_native_writer` when a target begins with an ordinal-bearing record, later contains a paginated record after a legacy start (#4311), or declares `history_mode=paginated`. The first line alone is not sufficient: a rollout that started unnumbered and was later migrated is also refused. Apply, manifest-backed restore, and explicit legacy recovery preflight all selected targets before changing database rows or manifests. The append boundary checks again. Codex owns ordinal allocation and the live projection cursor; reading the last ordinal and appending N+1 is not safe concurrent coordination. Legacy unnumbered rollouts retain their existing behavior. History Worker targets resolve the canonical manifest first and an existing pre-normalization Windows filename second, so passing an explicit target cannot bypass upgrade recovery. This guard prevents the observed stable-format corruption; it does not implement native-writer integration. Stop history writes also follow the [lifecycle ownership boundary](runtime.md), including H/C serialization for the Worker and synchronous exit. Injection preflights affected history using the normalized config candidate before writing config/profile/journal, then checks again after the complete artifact write. Native restore also rechecks after successful journal restoration or fallback removal, while exact config/profile/journal preimages and any coordinated remove transaction remain available for compensation. diff --git a/structure/config.md b/structure/config.md index 0995f89e5bd..62997e44447 100644 --- a/structure/config.md +++ b/structure/config.md @@ -342,7 +342,7 @@ keeps the hard refusal and the compensating rollback. Unattended sync, `POST /api/sync`, and every other config or ownership refusal keep the hard failure above. The real injection still revalidates under its normal write boundary after catalog convergence; -the preflight is an early no-write guard, not an authorization token for a later write. +the preflight is an early no-write guard, not an authorization token for a later write. Stop history writes follow the [lifecycle ownership boundary](runtime.md). Synchronous H lock unavailability is a history artifact failure, using the same outcome formatter as the Worker path; it does not latch connection ownership or prevent separately guarded Grok cleanup. > Decision record: [ADR-0019](decisions/ADR-0019-config-injection.md) @@ -459,7 +459,7 @@ config facade in a fresh process, so an earlier schema import cannot mask an ini ## Restore `ocx stop`, `ocx restore` / `ocx eject`, `ocx service stop`, and `ocx service uninstall` must strip -opencodex config and routed catalog entries without damaging native Codex state. Catalog restoration +opencodex config and routed catalog entries without damaging native Codex state; parent stop also enforces [connection ownership at each write](runtime.md#remote-hub-hardening-ownership). Catalog restoration omits retired bare/account-qualified native rows even when they occur in a pristine backup; the backup itself is not rewritten. See the [catalog contract](catalog.md#shared-catalog). diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 5c2428c1b46..b40866986cc 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -286,7 +286,7 @@ per-request first-party callback reads that live object; a failed write leaves i | Usage | `GET /api/usage` read-only aggregates of readable rows from `~/.opencodex/usage.jsonl`; the ledger is streamed in fixed 1 MiB chunks, so the former read-byte and parsed-row caps cannot omit its prefix. Oversized skipped rows produce positive `usageIncomplete` metadata. The response includes measured / reported / unreported / unsupported / estimated counts, a daily zero-filled grid, and model and provider breakdowns. `GET /api/usage/timeline` uses the same ledger and canonical attribution helpers for bounded bucketed model series. Never exposes prompts. Fast observations and cost provenance follow the [response-tier authority contract](transports/responses.md#response-tier-observation-authority): the provider editor classifies `responseTierAuthoritative` as an operator-owned boolean under the existing redaction and write boundaries, `src/usage/log.ts` retains it on final outcomes and attempts (old records keep their legacy meaning), and `src/usage/cost.ts` never prices a non-authoritative echo or confirmed label as response-confirmed. | | Request metrics | `GET /api/metrics` exposes process-local Prometheus text format v0.0.4 only when `metricsExport.enabled` was true at startup. The ordinary management gate applies; data-plane credentials do not grant access, and disabled mode is 404. `src/server/request-metrics.ts` owns fixed counters/histograms and receives a narrow final-request fact from `src/server/request-log.ts`; `src/server/index/serve-options.ts` creates one owner and injects the recorder and read-only snapshot into the request and management paths. | | System | `POST /api/system/restart` restarts the proxy in place. Local CLI/tray callers first attest the exact runtime PID and port, then send a process-scoped HMAC capability bound to that method, path, PID, and port; the capability authorizes no other management route and is invalid after replacement. The caller observes one absolute deadline and accepts success only after a different runtime PID is healthy on the same port. `GET /api/system/health` is the authenticated scalar-only identity used by shared-plane Dashboard status and restart reconnect polling; its `spendLedger` block reports only ownership held/unheld, initialized/configured/degraded booleans and bounded persistence/corruption counters. Reading it never constructs, replays or prunes the ledger. Paths, scopes, accounts and request ids are absent, and the block never moves to unauthenticated `/healthz`. `GET /api/system/memory` — service-process runtime/memory identity (pid, Bun version/revision, optional `bunRuntimeSource` provenance, platform, RSS/heap/external/ArrayBuffers scalars, observed memory = max(RSS, external, ArrayBuffers), `bun:jsc` heap context, streamMode + eager-relay gate decision, watchdog snapshot sliced to the last 60 samples) plus privacy-safe `appOwnedBytes` retained-store totals/counters under static store ids. Its response-state block also reports spill-write `initial`/`healthy`/`degraded` status, a consecutive-failure streak, fixed error class, and failure/success timestamps. A successful publication clears the streak in the same process; raw error text and paths never enter this surface. Scalar-only payload; dashboard/admin callers use the standard management gate, while `ocx doctor` may use only the exact process-scoped local-read capability. It must never move to unauthenticated `/healthz`. | -| Stop | `GET /api/stop` reports a read-only PID-bound connected-client shutdown-preservation capability and accepts an exact single-use local-read grant, returning an attestation proof over its nonce, PID and port. That grant cannot authorize `POST /api/stop`, which retains ordinary management authorization: stop any installed service and exit the proxy; ordinary owners restore native Codex/Grok. Siblings and validated connected hub/Link clients preserve shared integrations (`sharedTeardown: "not-owned"`; [runtime contract](runtime.md#lifecycle)). A sibling instance restores nothing and answers `sharedTeardown: "not-owned"` ([Codex home](codex-home.md#codex-home)). | +| Stop | Pending, invalid, mismatched or busy connection ownership returns HTTP 409 with `sharedTeardown: "refused"` before scheduling shutdown; a new hub owner skips restoration. [The runtime contract](runtime.md#remote-hub-hardening-ownership) owns the protected write checks. `GET /api/stop` reports a read-only PID-bound connected-client shutdown-preservation capability and accepts an exact single-use local-read grant, returning an attestation proof over its nonce, PID and port. That grant cannot authorize `POST /api/stop`, which retains ordinary management authorization. `POST /api/stop` — stop any installed service and exit the proxy; ordinary owners restore native Codex/Grok. Siblings and validated connected hub clients preserve shared integrations (`sharedTeardown: "not-owned"`; [runtime contract](runtime.md#lifecycle)). A Remote Link Child restores them like an ordinary owner unless a validated teardown receipt defers restoration to the stopping parent CLI; the CLI sends every authenticated stop request over direct loopback HTTP, so environment proxies never receive the admin token. A sibling instance restores nothing and answers `sharedTeardown: "not-owned"` ([Codex home](codex-home.md#codex-home)). | | Diagnostics/sync | `src/server/management/config-routes.ts` — `GET /api/diagnostics/project-config` reports project-level Codex config that bypasses managed routing; `POST /api/sync` re-runs catalog/config sync. The diagnostic reports the bypass; it does not rewrite the project file. | | Sidecar/shadow-call settings | `src/server/management/config-routes.ts` — `GET/PUT /api/sidecar-settings` and `GET/PUT /api/shadow-call-settings`, and `GET/PUT /api/auto-review-settings`. The auto-review settings route validates the global reviewer selector, persists its enabled/model fields with rollback before publication, then converges the catalog and returns its disposition; the Models panel keeps the server echo and displays a pending-sync notice whenever no commit is reported. Sidecar PUT accepts model and backend (web-search union: openai/anthropic/xai/gemini/exa; xAI is live through stored Grok OAuth, while Gemini/Exa remain inert until their executors ship) plus validated `webSearch.xSearch`, optional `webSearch.exaApiKey` (write/clear only — never echoed by GET or the PUT response; redact.ts strips it from logs), `webSearch.reasoning`, `vision.reasoning`, `vision.enabled`, `vision.maxDescriptionsPerTurn`, and `vision.timeoutMs`; the read and PUT-response payload reports model, backend, reasoning, enabled, the vision per-turn limit, and timeout. An effective `vision.enabled` change persists before one bounded catalog-only convergence updates the managed catalog and Codex model cache; the PUT response returns `catalogRefreshPending` for incomplete work. Identical effective values and unrelated partial writes do not converge, and no result claims client reload. `timeoutMs` is validated against the runtime integer bounds in `src/vision/timeout-bounds.ts`. Provider/OAuth credentials live in their stores; `exaApiKey` is the one sidecar-owned secret and follows the write-only contract above. Both shadow-call responses also report the resolved `sourceModels` — the prefixes the runtime actually intercepts (`src/lib/shadow-call.ts`, default `gpt-6-luna` and `gpt-5.6-luna`; the retired `gpt-5.4-mini` stays available as an explicit `sourceModels` entry for 0.144.x clients), so no client hard-codes a helper slug that a Codex release can invalidate. PUT refuses a qualified target that only the router's default-provider fallback accepts. Provider disable and delete report the target they leave behind as `dependentShadowIntercept` (`shadowInterceptProviderDependency` in `src/server/management/shadow-call-validation.ts`), and at request time an unresolvable target returns `409 intercept_target_unavailable` before any send (`src/server/responses/shadow-target-availability.ts`); it never falls back to the native source model or the default provider. | | Storage | `src/server/management/logs-usage-routes.ts` — `GET /api/storage`, `POST /api/storage/cleanup/preview` and `/api/storage/cleanup`, `GET /api/storage/trash`, `POST /api/storage/trash/restore`, and `GET/PUT /api/storage/cleanup-policy` plus `POST /api/storage/cleanup-policy/run`. `GET /api/storage/cleanup-policy/test-stream` and `GET /api/storage/trash/restore/test-stream` exist for progress-stream testing. Cleanup takes an explicit `mode`: `quarantine` moves to trash and is restorable, `permanent` is not. The caller must name the mode — there is no default that silently deletes. | @@ -447,7 +447,7 @@ The desktop shell needs only the dashboard, so it attaches to a client-role list The dashboard sidebar includes a stop button that calls `POST /api/stop`. The button shows a confirmation prompt, then fires the request and accepts the connection drop (the proxy exits). The -endpoint restores native Codex config for ordinary owners; connected clients and siblings preserve their integrations. It stops any installed service to prevent respawn and exits, as described in the [runtime contract](runtime.md#lifecycle). +endpoint restores native Codex config for ordinary owners and Remote Link Children; connected hub clients and siblings preserve their integrations. Without an owned deferral receipt, it checks connection ownership under C before stopping the installed service; an unproven boundary returns 409 with `sharedTeardown: "refused"` and leaves the manager and proxy running. Each later restoration write still revalidates independently. Accepted stops stop the service to prevent respawn and exit, as described in the [runtime contract](runtime.md#lifecycle). While the desktop app supervises the proxy it refuses a dashboard session with 409 `desktop_supervised` before touching anything, because the app would start the proxy again ([desktop shell](desktop-shell.md#keeping-the-runtime-alive)); the button reports the refusal. diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index dbc1810e435..2f3cfab42c7 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -516,6 +516,8 @@ endpoint can never restart. Focused coverage is `tests/update/update-stop-classi ## Restart handoff +Ordinary signal shutdown passes the CLI's idempotent exit cleanup to `drainAndShutdown` in `src/server/lifecycle.ts`: its drain-finally block calls the synchronous-only `() => undefined` hook before listener stop. Promise-returning hooks are outside this contract; there is no asynchronous cleanup wait. Listener stop and startup-lifecycle release still run if cleanup throws. Restart callers omit that hook and retain their existing routing-preservation decisions; receipt-backed stop deferral remains owned by the stopping client. After verifying a matching receipt, `performStopTeardown` records a dedicated process-lifetime exit deferral; exit cleanup consults it before restoring native Codex/Grok, including when the API's own drain ends in `process.exit`. A query flag or connected-client capability does not set it. API and signal restoration repeat the [connection ownership guard at each write](../runtime.md#remote-hub-hardening-ownership); an unproven API boundary returns 409 before scheduling shutdown, and signal cleanup reports failure while preserving shared configuration. `tests/fixtures/api-stop-exit-runner.ts`, exercised by `tests/service/stop-deferred-teardown.test.ts`, runs the actual API handler, listener drain and process-exit cleanup in one child. `tests/service/shutdown-drain.test.ts` covers the hook with a real loopback listener and the CLI signal handler. + During drain, `src/server/index.ts` rejects new data-plane work with HTTP 503 and `src/service/linux-linger.ts` observes boot lingering without changing it; enabled, disabled and unknown states receive distinct install advice. Desktop startup distinguishes enabled next-login registration from current loaded state, preserving identity/disabled/supervisor checks. `src/cli/hub-disable.ts` changes persisted hub settings only, prints Desktop/service/manual restart guidance and reminds the operator to review external publishing to old ports. an explicit JSON envelope: `error.type` is `server_error`, `error.code` is `server_restarting`, and `error.message` is "OpenCodex is restarting; retry this request." @@ -528,10 +530,8 @@ clients do not retry it at all. `server_restarting` falls through to retryable response bypasses the shared provider-overload mapping in `src/lib/errors.ts`. `tests/codex-integration/issue-452-empty-503.test.ts` pins its body and both listeners' CORS. -`src/server/management/system-routes.ts` accepts an optional JSON `drainGraceMs` on the authenticated -restart API: an integer from 1 to 60000, with omission retaining the 60-second default. Invalid -input starts no drain. `src/server/management/system-restart.ts` fixes the selected grace at first -acceptance; repeated requests cannot shorten it. The response-flush delay stays 200ms for every +`src/server/management/system-routes.ts` accepts an optional JSON `drainGraceMs` on the authenticated restart API: an integer from 1 to 60000, with omission retaining the 60-second default. Invalid +input starts no drain. `src/server/management/system-restart.ts` fixes the selected grace at first acceptance; repeated requests cannot shorten it. The response-flush delay stays 200ms for every caller. A grace shorter than 200ms cancels and releases active turns at the accepted deadline via a separate timer, cancelled by a pending restart veto. Cleanup, listener stop and process exit still wait for the response-flush delay; grace of at least 200ms arms no extra timer. The cleanup diff --git a/structure/runtime.md b/structure/runtime.md index bc348223361..31fa9dd7c8c 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -175,7 +175,7 @@ described in [OpenAI quota ownership](providers/openai-tiers.md#public-provider- `ocx start` refuses a duplicate PID, starts the proxy, writes `~/.opencodex/ocx.pid` and `runtime-port.json` through `src/config/process-state.ts`, syncs Codex config/catalog, then serves -until shutdown. Standalone shutdown restores native Codex; connected hub clients and Remote Link Children preserve connection-owned Codex routing, catalog and Grok config across `ocx stop`/`ocx start`. An ordinary runtime rechecks persisted connection ownership in its exit cleanup too, preserving Codex/Grok while retaining its existing environment cleanup. Client start does not re-inject routing: use `ocx sync --restart-codex` to reapply it, or `ocx disconnect` to release connection ownership. A sibling instance beside a live proxy ([Codex home](codex-home.md#codex-home)) syncs and restores nothing, and `ocx stop` of a runtime whose record carries `siblingOfPort` skips the shared teardown. `src/cli/index.ts` resolves same-home ownership, then checks shared client hints through the cross-home owner helper, then reconciles the journal only when no sibling is marked. An indeterminate cross-home result refuses a new start before binding or publication, rather than persisting an unverified sibling topology; the next start performs fresh discovery. Service mode sets +until shutdown. Standalone shutdown restores native Codex; connected hub clients preserve connection-owned Codex routing, catalog and Grok config across `ocx stop`/`ocx start`. An ordinary runtime rechecks persisted connection ownership in its exit cleanup: a connected Remote Link Child remains subject to the existing native Codex/Grok restoration decisions. Graceful signal shutdown runs that synchronous cleanup after draining and before releasing the listener. Sibling, recycling and service-managed exit exclusions remain intact; receipt-backed API stop may explicitly defer shared restoration to the stopping client. Only an accepted receipt deferral in `src/server/stop-teardown.ts` latches that process's exit deferral; CLI exit cleanup keeps native Codex and Grok restoration with the parent, while still stopping background owners and removing runtime records. Client start does not re-inject routing: use `ocx sync --restart-codex` to reapply it, or `ocx disconnect` to release connection ownership. A sibling instance beside a live proxy ([Codex home](codex-home.md#codex-home)) syncs and restores nothing, and `ocx stop` of a runtime whose record carries `siblingOfPort` skips the shared teardown. `src/cli/index.ts` resolves same-home ownership, then checks shared client hints through the cross-home owner helper, then reconciles the journal only when no sibling is marked. An indeterminate cross-home result refuses a new start before binding or publication, rather than persisting an unverified sibling topology; the next start performs fresh discovery. Service mode sets `OCX_SERVICE=1`, so managed restarts do not repeatedly restore/reinject; scheduler failure diagnostics follow the [service command contract](ops/service-and-sidecars.md#background-service-command-selection). Explicit service stop and uninstall still restore. `src/service/cli.ts` removes the service token on uninstall only when persisted client state is disconnected and no pending connect marker owns the newly issued key. `src/client/connect.ts` publishes that fingerprint marker before the key, then clears it with the connection commit or rollback under the client lifecycle and config mutation locks. Connected, invalid, or mismatched client state retains an existing token. A valid pending marker retains only its matching fingerprint; an older marker does not own a replacement service key. An absent token is reported as absent; unsafe, malformed, or unreadable markers and lock, state-read, or deletion failures leave cleanup unverified. `src/cli/tray-proxy.ts` coordinates `ocx restart` and tray restart. A second start requires proof the prior attempt never launched or its child exited, plus confirmed absence under a fresh bounded discovery window; a missing health response alone is insufficient. After an accepted restart, polling continues through uncertain observations until its deadline and requires a new runtime PID on the same port. If the old proxy then disappears, explicit recovery starts it regardless of the Codex autostart preference; a skipped recovery fails rather than reporting success. The Windows tray startup-health probe in `src/tray/windows-tray.ps1` requests termination at its 30-second deadline without waiting on the UI thread. It retains the process through later ticks until actual exit and both redirected readers settle; replacements cannot start while that reference is held. Launch setup failures follow the same cleanup. Tray shutdown alone waits at most 500 ms before disposal. @@ -335,8 +335,8 @@ field for every tool. Only bare or `default.`-prefixed `exec` and `apply_patch` one recognized alternate body field or remove one complete outer Markdown fence; ambiguous alternate fields and every other freeform grammar pass through unchanged. -The server exposes read-only authenticated `GET /api/stop` with `schema: "ocx-stop-capabilities/1"`, PID and connected-client preservation for API, signal and manager shutdown. Before a connected-client service stop, `src/cli/stop-restore.ts` pins the protected runtime record's PID, port, hostname and attestation key and requires a fresh health proof. A proven client-role listener needs no capability route; an ordinary proxy also requires a valid response proof and preservation document from a single-use `GET /api/stop` capability. Both preflight reads omit reusable credentials and refuse a changed runtime record, sharing a two-second deadline and 4 KiB document limits. The probes use the proxy-independent TCP transport in `src/server/direct-local-http.ts`. `src/lib/process-control.ts` defaults every local runtime stop POST to that direct transport, ignoring HTTP(S)/ALL proxy environment variables for CLI stop, service cleanup and uninstall; its injected `fetchFn` takes precedence. The later POST retains ordinary management authorization and the existing target recheck before POST/kill. Unknown targets refuse without mutation. A stopped connected client skips manager mutation only after the manager is observed inactive and all known local endpoints are definitively dead; the ordinary receipt and environment cleanup still runs. `POST /api/stop`: `src/server/stop-teardown.ts` checks sibling then validated persisted connected-client ownership before receipt deferral or restoration. Both owners answer `sharedTeardown: "not-owned"` without restoring Codex/catalog or stripping Grok; an existing receipt remains outstanding. Other owners restore native Codex/Grok, or defer only with the matching receipt. The route stops any installed service -(to prevent respawn), and exits the process. The GUI sidebar stop button calls this endpoint. An observed 409 remains a stop refusal even when its response body is truncated or times out; the direct transport exposes the parsed status before body completion so the stop helper retains that decision. +The server exposes read-only authenticated `GET /api/stop` with `schema: "ocx-stop-capabilities/1"`, PID and connected-client preservation for API, signal and manager shutdown. Before a connected-client service stop, `src/cli/stop-restore.ts` pins the protected runtime record's PID, port, hostname and attestation key and requires a fresh health proof. A proven client-role listener needs no capability route; an ordinary proxy also requires a valid response proof and preservation document from a single-use `GET /api/stop` capability. Both preflight reads omit reusable credentials and refuse a changed runtime record, sharing a two-second deadline and 4 KiB document limits. The probes use the proxy-independent TCP transport in `src/server/direct-local-http.ts`. `src/lib/process-control.ts` defaults every local runtime stop POST to that direct transport, ignoring HTTP(S)/ALL proxy environment variables for CLI stop, service cleanup and uninstall; its injected `fetchFn` takes precedence. The later POST retains ordinary management authorization and the existing target recheck before POST/kill. A stopped connected client skips manager mutation only after the manager is observed inactive and all known local endpoints are definitively dead; the ordinary receipt and environment cleanup still runs. `POST /api/stop`: `src/server/stop-teardown.ts` checks sibling then validated persisted connected-client ownership before receipt deferral or restoration. Sibling and connected hub client owners answer `sharedTeardown: "not-owned"` without restoring Codex/catalog or stripping Grok; an existing receipt remains outstanding. A Remote Link Child restores like an ordinary owner. The stopping CLI accepts `not-owned` unless the connection state read at the POST positively identifies a Link Child: only an older runtime answers `not-owned` for a link connection, and it does so while preserving the loopback route the stop must restore. The parent's `restoreBoundaryOwnership` inspection takes L but grants no write authority. `src/client/restore-ownership.ts`, re-exported by `src/cli/stop-restore-guard.ts`, revalidates committed ownership, pending connect markers and disconnect recovery under C at every restoration write. Stop history restoration takes H before C; each synchronous provider attempt keeps C from ownership validation through DB commit, rollout writes and manifest consumption or failure finalization. The Worker receives the stop policy as data, validates under its own C acquisition and returns a structured ownership refusal that the parent guard retains. Synchronous exit restoration takes the same H/C boundary; no C lease spans a Worker await or a retry delay. Config/profile/journal restoration uses the config-file lock and N/C on coordinated homes; legacy restoration uses the file lock and C. Catalog restoration takes K before C. Approved-provider URL restoration after history, Grok stripping and receipt discharge each revalidate under C again after awaits. Connection publication cannot interleave between a check and its synchronous writes. The default API stop and synchronous signal cleanup use the same guard for their native config/catalog and Grok writes, including approved-provider cleanup after history. An unproven API boundary returns HTTP 409 with `sharedTeardown: "refused"` before shutdown is scheduled; signal cleanup reports failure without restoring over it. Each pending API stop holds its own temporary automatic-restart veto; refusal or a thrown teardown releases only that intent, while accepted stops and signals retain a permanent veto. A refused stop therefore leaves later package-replacement restart available. An already admitted automatic restart waits for temporary intents to settle instead of losing its one-shot package notification; its original deadline remains the budget basis. These checks never acquire L inside N/C, acquire N inside L, or await inside a synchronous lock callback. A hub owner supersedes only the identified receipts, settled under C; pending, invalid, mismatched or busy state refuses the remaining restoration and keeps its obligations. `tests/service/stop-deferred-teardown.test.ts` covers post-inspection transitions and real config-file/N writer contention. Other owners restore native Codex/Grok, or defer only with the matching receipt. The route stops any installed service +(to prevent respawn), and exits the process. Without an owned deferral receipt, the non-sibling API route first checks connection ownership under C; an unproven boundary refuses before manager mutation. The temporary stop intent covers this check and service stop as well as teardown, and every early return or exception releases only that intent. The preflight grants no write authority and no C lease reaches the manager or an await. Synchronous H lock unavailability reports a history-only artifact failure through the Worker outcome formatter, leaving Grok cleanup free to revalidate under C. `tests/service/stop-restoration-preflight.test.ts` covers real H contention and a refused API stop with zero manager calls and continued serving. The GUI sidebar stop button calls this endpoint. An observed 409 remains a stop refusal even when its response body is truncated or times out; the direct transport exposes the parsed status before body completion so the stop helper retains that decision. The CLI-selected transport delegates to the same status-preserving direct helper as the default; truncated and stalled refusal bodies never authorize a forced kill. Real history and transport coverage lives in `tests/service/stop-restore-history-ownership.test.ts` and `tests/cli/cli-stop-refusal-transport.test.ts`. > Decision record: [ADR-0004](decisions/ADR-0004-lifecycle.md) diff --git a/tests/cli/cli-start-routing-heal-wiring.test.ts b/tests/cli/cli-start-routing-heal-wiring.test.ts index 42a7b88f4b9..9ec6a3a1f94 100644 --- a/tests/cli/cli-start-routing-heal-wiring.test.ts +++ b/tests/cli/cli-start-routing-heal-wiring.test.ts @@ -46,7 +46,7 @@ describe("handleStart routing-healer wiring", () => { const stopAt = cleanup.indexOf("try { routingHealer?.stop(); } catch { /* best-effort */ }"); expect(stopAt).toBeGreaterThan(-1); expect(stopAt).toBeLessThan(cleanup.indexOf("revertSystemEnv()")); - expect(stopAt).toBeLessThan(cleanup.indexOf("restoreNativeCodex()")); + expect(stopAt).toBeLessThan(cleanup.indexOf("restoreNativeCodex({ withWriteGuard: restoreGuard.run })")); expect(stopAt).toBeLessThan(cleanup.indexOf("stripGrokConfig()")); expect(stopAt).toBeGreaterThan(cleanup.indexOf("cleaned = true;")); }); diff --git a/tests/cli/cli-stop-refusal-transport.test.ts b/tests/cli/cli-stop-refusal-transport.test.ts new file mode 100644 index 00000000000..e87bd457102 --- /dev/null +++ b/tests/cli/cli-stop-refusal-transport.test.ts @@ -0,0 +1,41 @@ +import { expect, spyOn, test } from "bun:test"; +import * as childProcess from "node:child_process"; +import net from "node:net"; +import { stopRequestFetch } from "../../src/cli/stop-restore"; +import { ProxyOwnershipRefusedError, stopProxy, stopProxyGracefully } from "../../src/lib/process-control"; + +// Exercise the exact transport selected by handleStop, including its forced-stop consumer. +for (const framing of ["truncated", "stalled"]) { + test(`R7 CLI transport retains observed 409 with ${framing} body and never kills`, async () => { + const sockets = new Set(); + const server = net.createServer(socket => { + sockets.add(socket); socket.on("error", () => {}); socket.on("close", () => sockets.delete(socket)); + socket.once("data", () => { + const response = "HTTP/1.1 409 Conflict\r\nContent-Type: application/json\r\nContent-Length: 100\r\nConnection: close\r\n\r\n{}"; + if (framing === "truncated") socket.end(response); + else socket.write(response); + }); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + let kills = 0; + const signal = spyOn(process, "kill").mockImplementation((_pid, kind) => { + if (kind !== 0) { kills++; throw new Error("unexpected forced signal"); } + return true; + }); + const taskkill = spyOn(childProcess, "execFileSync").mockImplementation(() => { + kills++; throw new Error("unexpected taskkill"); + }); + try { + const io = { runtimeEndpoint: { hostname: "127.0.0.1", port: (server.address() as net.AddressInfo).port }, + env: {}, exitTimeoutMs: 150, fetchFn: stopRequestFetch(), + waitExit: () => { throw new Error("must never wait after refusal"); } }; + expect(await stopProxyGracefully(4242, io)).toBe("refused"); + await expect(stopProxy(4242, io)).rejects.toBeInstanceOf(ProxyOwnershipRefusedError); + expect(kills).toBe(0); + } finally { + signal.mockRestore(); taskkill.mockRestore(); + for (const socket of sockets) socket.destroy(); + await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + } + }); +} diff --git a/tests/fixtures/api-stop-exit-runner.ts b/tests/fixtures/api-stop-exit-runner.ts new file mode 100644 index 00000000000..0407a715111 --- /dev/null +++ b/tests/fixtures/api-stop-exit-runner.ts @@ -0,0 +1,118 @@ +/** Real API stop -> drain/listener close -> process.exit -> CLI exit cleanup, in one child. */ +import { mock } from "bun:test"; +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { repoPath } from "../helpers/repo-root"; + +const query = process.argv[2] as "valid" | "foreign" | "missing" | "none" | "pending"; +const home = process.env.OPENCODEX_HOME; +if (!home || !process.env.CODEX_HOME) throw new Error("isolated fixture homes required"); +delete process.env.OCX_SERVICE; +delete process.env.OCX_DESKTOP_SUPERVISED; +const events: string[] = []; +const calls = { service: 0, apiNative: 0, apiGrok: 0, exitNative: 0, exitGrok: 0, env: 0, pid: 0, runtime: 0 }; +const restored = { success: true, message: "fixture restored", artifacts: { + config: { state: "restored" }, catalog: { state: "restored" }, history: { state: "restored" }, +} }; + +// Replace host mutation only. Receipt verification, handler, drain and exit remain real. +const service = await import("../../src/service"); +mock.module("../../src/service", () => ({ ...service, + installedServiceRespawnRisk: () => "none", stopServiceIfInstalledDetailed: () => { calls.service++; return "stopped"; }, +})); +const native = await import("../../src/codex/inject"); +mock.module("../../src/codex/inject", () => ({ ...native, + restoreNativeCodexAsync: async () => { calls.apiNative++; events.push("native-api"); return restored; }, +})); +const grok = await import("../../src/grok/inject"); +mock.module("../../src/grok/inject", () => ({ ...grok, + stripGrokConfig: () => { calls.apiGrok++; events.push("grok-api"); return { ok: true, changed: true, message: "fixture stripped" }; }, +})); + +const { claimPendingTeardown, readPendingTeardown } = await import("../../src/config/pending-teardown"); +const { isSharedTeardownDeferredOnExit } = await import("../../src/server/stop-teardown"); +const { preserveConnectedClientRoutingOnExit } = await import("../../src/cli/stop-restore"); +const { stopRestoreWriteGuard } = await import("../../src/cli/stop-restore-guard"); +const { decideStartExitTeardown } = await import("../../src/cli/dispatch"); +const { setServerRef, setServerStartupLifecycleReleaseForTests, isRecyclingForExit, markRecyclingForExit } = await import("../../src/server/lifecycle"); +const { handleManagementAPI } = await import("../../src/server/management-api"); +const { directLocalHttpFetch } = await import("../../src/server/direct-local-http"); +const config = { port: 0, hostname: "127.0.0.1", providers: {}, defaultProvider: "openai", shutdownTimeoutMs: 0 }; +const server = Bun.serve({ hostname: "127.0.0.1", port: 0, reusePort: false, + async fetch(req) { + // Auth is a fixture prerequisite; exercise the actual admitted management handler. + return await handleManagementAPI(req, new URL(req.url), config as never, {}, "admin-token") + ?? new Response("not found", { status: 404 }); + }, +}); +config.port = server.port!; +setServerRef(server); +const realStop = server.stop.bind(server); +server.stop = (force?: boolean) => { events.push("listener-stop"); return realStop(force); }; +setServerStartupLifecycleReleaseForTests(async () => { events.push("startup-release"); }); +const receipt = claimPendingTeardown({ hostname: "127.0.0.1", port: config.port }, "exact"); + +// Register the real CLI cleanup body on the real exit event; do not call it as a substitute +// for the API handler. The API's own timer must drain and invoke process.exit first. +const source = readFileSync(repoPath("src", "cli", "index.ts"), "utf8"); +const from = source.indexOf(" const syncCleanup = () => {"); +const to = source.indexOf("\n let shuttingDown", from); +if (from < 0 || to < 0) throw new Error("CLI exit cleanup missing"); +const bindings = { + catalogHealer: null, routingHealer: null, guardian: { stop() {} }, historyGuardian: null, healthSentinel: null, + siblingStart: false, isRecyclingForExit, decideStartExitTeardown, preserveConnectedClientRoutingOnExit, stopRestoreWriteGuard, + revertSystemEnv() { calls.env++; }, removePid() { calls.pid++; }, removeRuntimePort() { calls.runtime++; }, + serviceEnvironmentOwnedHere: () => true, getCodexHome: () => process.env.CODEX_HOME!, codexHomeIsAbsent: () => false, + restoreNativeCodex: (options?: { withWriteGuard?: (write: () => T) => T }) => { + const write = () => { calls.exitNative++; events.push("native-exit"); return restored; }; + return options?.withWriteGuard ? options.withWriteGuard(write) : write(); + }, + stripGrokConfig: () => { calls.exitGrok++; events.push("grok-exit"); return { ok: true, changed: true, message: "fixture stripped" }; }, +}; +const cleanup = new Function(...Object.keys(bindings), + "let cleaned = false; let cleanupSucceeded = true;\n" + source.slice(from, to) + "\nreturn syncCleanup;", +)(...Object.values(bindings)) as () => boolean; +let responseStatus: number | undefined; +let body: unknown; +let refusedKeptServing = false; +let automaticRestarted = false; +process.on("exit", () => { events.push("exit-cleanup"); cleanup(); }); +process.on("exit", () => { + writeFileSync(join(home, "api-stop-exit-result.json"), JSON.stringify({ + calls, events, responseStatus, body, port: config.port, + deferredOnExit: isSharedTeardownDeferredOnExit(), receiptState: readPendingTeardown(receipt.nonce).state, refusedKeptServing, automaticRestarted, + })); +}); + +const url = new URL("/api/stop", server.url); +if (query === "pending") (await import("../../src/client/state")).markClientConnectPending("b".repeat(64)); +if (query !== "none") url.searchParams.set("deferSharedTeardown", "1"); +if (query === "valid") url.searchParams.set("teardownNonce", receipt.nonce); +if (query === "foreign") url.searchParams.set("teardownNonce", "ffffffffffffffffffffffffffffffff"); +const response = await directLocalHttpFetch(url, { method: "POST", signal: AbortSignal.timeout(5_000) }, { timeoutMs: 5_000 }); +responseStatus = response.status; +body = await response.json(); +if (query === "pending") { + await Bun.sleep(350); // An accepted stop would already have drained and exited. + const probe = await directLocalHttpFetch(new URL("/api/stop", server.url), { method: "GET", signal: AbortSignal.timeout(2000) }, { timeoutMs: 2000 }); + refusedKeptServing = probe.status === 200; + await probe.body?.cancel(); + (await import("../../src/client/state")).clearClientConnectPending("b".repeat(64)); + let scheduled!: () => void | Promise; + (await import("../../src/server/management/system-restart")).acceptSystemRestart({ + isShutdownDraining: () => false, getActiveTurnCount: () => 0, + acquireTemporaryDrain: () => ({ release() {} }), beginShutdownDrain() {}, + schedule: work => { scheduled = work; }, scheduleDeadline: () => () => {}, + drainAndShutdown: async () => true, stopListener() {}, listenPort: () => config.port, + isDesktopSupervised: () => false, isSupervisedServiceChild: () => false, + spawnStart() { automaticRestarted = true; }, markRecycling: markRecyclingForExit, exitProcess() {}, + }, { onAccepted() {}, beforeScheduledDrain: () => true }); + await scheduled(); + await server.stop(true); + process.exit(0); +} +if (response.status !== 200) { + await server.stop(true); + throw new Error(`unexpected API stop status: ${response.status}`); +} +// The real management handler's 200ms callback owns drain and process.exit. No fixture exit. diff --git a/tests/fixtures/parent-stop-runner.ts b/tests/fixtures/parent-stop-runner.ts index 35c901c265a..7a1f9dd48b3 100644 --- a/tests/fixtures/parent-stop-runner.ts +++ b/tests/fixtures/parent-stop-runner.ts @@ -30,6 +30,21 @@ const options = JSON.parse(readFileSync(0, "utf8")) as { alreadyStopped?: boolean; absence?: "dead" | "unknown" | "live"; managerState?: "inactive" | "active" | "unknown"; + /** Commit a disconnect while the capability probe is in flight. */ + disconnectDuringProbe?: boolean; + /** Commit a hub connection while orphan discovery is in flight. */ + connectDuringDiscovery?: boolean; + /** Commit a hub connection while the stop POST is in flight, before server teardown reads it. */ + connectDuringPost?: boolean; + /** The client lifecycle lock is held elsewhere when the parent reaches its restore boundary. */ + lifecycleBusy?: boolean; + connectAfterBoundary?: boolean; + pendingAfterBoundary?: boolean; + connectDuringRestore?: boolean; + connectAfterExitInspection?: boolean; + pendingAfterExitInspection?: boolean; + pendingAfterGrok?: boolean; + receiptClearBusy?: boolean; /** #7423: run the stop the way an update that restarts the proxy runs it. */ handover?: boolean; /** The runtime-port record is missing, so the stop can only record a guessed endpoint. */ @@ -48,6 +63,7 @@ if (options.proxyProbePort) { const { assertConnectedClientStopSupported } = await import("../../src/cli/stop-restore"); await assertConnectedClientStopSupported({ pid: process.pid, port: options.proxyProbePort, hostname: "127.0.0.1" }); const { stopProxyGracefully } = await import("../../src/lib/process-control"); + // Exercise the CLI's shared transport default; a Link Child must stay direct too. const result = await stopProxyGracefully(process.pid, { runtimeEndpoint: { port: options.proxyProbePort, hostname: "127.0.0.1" }, waitExit: () => true, @@ -57,11 +73,38 @@ if (options.proxyProbePort) { process.exit(0); } const home = process.env.OPENCODEX_HOME!; +const publishPending = () => writeFileSync(join(home, "client-connect-pending"), "b".repeat(64) + "\n"); +function writeHubConnection(): void { + const serverUrl = "https://hub.example.test"; + writeFileSync(join(home, "config.json"), JSON.stringify({ runtimeRole: "client", providers: {}, defaultProvider: "openai", client: { + serverUrl, managementUrl: serverUrl, managementTransport: "direct", transport: "hub", selectedClients: ["codex"], + tokenEnv: "OPENCODEX_API_AUTH_TOKEN", apiKeyId: "fixture-client", tokenFingerprint: "a".repeat(64), + protocolVersion: 1, connectedAt: "2026-10-01T00:00:00.000Z", + } })); +} +// The lock's own suite covers SQLite exclusion; here only "held" versus "busy" matters, and the +// real lock would harden a directory in the host user's runtime root. +const lifecycle = await import("../../src/client/lifecycle-lock"); +mock.module("../../src/client/lifecycle-lock", () => ({ ...lifecycle, + withClientLifecycleSync: (work: Parameters[0]) => { + if (options.lifecycleBusy) throw new Error("client_lifecycle_busy"); + calls.lifecycle += 1; + const value = work(Object.freeze({}) as Parameters[0]); + if (options.connectAfterBoundary) queueMicrotask(writeHubConnection); + if (options.pendingAfterBoundary) queueMicrotask(publishPending); + return value; + }, +})); const endpoint = { hostname: "127.0.0.1", port: 10100 }; const fakePid = 4242; -const calls = { killed: 0, native: 0, grok: 0, cleared: 0, exited: 0, service: 0 }; +const calls = { killed: 0, native: 0, grok: 0, cleared: 0, exited: 0, service: 0, lifecycle: 0 }; +let receiptBlocker: import("bun:sqlite").Database | undefined; +const { Database } = await import("bun:sqlite"); +const { prepareConfigMutationDatabasePathForWrite } = await import("../../src/config"); const urls: string[] = []; const probes: string[] = []; +const transports: string[] = []; +let viaDirect = false; const unexpectedIo: string[] = []; let alive = !options.alreadyStopped; let nonce: string | undefined; @@ -92,8 +135,9 @@ mock.module("node:child_process", () => ({ const receipts = await import("../../src/config/pending-teardown"); process.on("exit", () => { + receiptBlocker?.close(); writeFileSync(join(home, "parent-stop-result.json"), JSON.stringify({ - calls, urls, probes, unexpectedIo, nonce, + calls, urls, probes, transports, unexpectedIo, nonce, receiptExists: nonce !== undefined && existsSync(receipts.pendingTeardownPathFor(nonce)), })); }); @@ -141,7 +185,10 @@ if (options.guarded) { if (options.orphan || options.alreadyStopped) { const liveness = await import("../../src/server/proxy-liveness"); mock.module("../../src/server/proxy-liveness", () => ({ ...liveness, - findLiveProxy: async () => alive ? { ...endpoint, pid: fakePid, source: "config" } : null, + findLiveProxy: async () => { + if (options.connectDuringDiscovery) writeHubConnection(); + return alive ? { ...endpoint, pid: fakePid, source: "config" } : null; + }, proveLiveProxyOwnedByHome: async () => "proven", probeEndpointLiveness: async () => options.absence ?? "dead", })); @@ -155,12 +202,25 @@ if (options.alreadyStopped) { const native = await import("../../src/codex/inject"); mock.module("../../src/codex/inject", () => ({ ...native, - restoreNativeCodexAsync: async () => { calls.native += 1; return options.restore; }, + restoreNativeCodexAsync: async (restoreOptions?: { withWriteGuard?: (write: () => T) => T }) => { + await Promise.resolve(); // The real restore awaits config-file/N acquisition before committing. + if (options.connectDuringRestore) writeHubConnection(); + const write = () => { calls.native += 1; return options.restore; }; + return restoreOptions?.withWriteGuard ? restoreOptions.withWriteGuard(write) : write(); + }, })); const grok = await import("../../src/grok/inject"); mock.module("../../src/grok/inject", () => ({ ...grok, - stripGrokConfig: () => { calls.grok += 1; return { ok: true, changed: true, message: "Grok restored" }; }, + stripGrokConfig: () => { + calls.grok += 1; + if (options.pendingAfterGrok) queueMicrotask(publishPending); + if (options.receiptClearBusy) queueMicrotask(() => { + receiptBlocker = new Database(prepareConfigMutationDatabasePathForWrite(), { readwrite: true, create: true }); + receiptBlocker.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + }); + return { ok: true, changed: true, message: "Grok restored" }; + }, })); const systemEnv = await import("../../src/server/system-env"); mock.module("../../src/server/system-env", () => ({ ...systemEnv, revertSystemEnv() {} })); @@ -193,11 +253,16 @@ globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => if (options.clientServer === "body-timeout") return new Response(new ReadableStream({ start(c) { init.signal?.addEventListener("abort", () => c.error(init.signal!.reason), { once: true }); } }), { headers }); + if (options.disconnectDuringProbe) { + writeFileSync(join(home, "config.json"), JSON.stringify({ runtimeRole: "standalone", providers: {}, defaultProvider: "openai" })); + } return Response.json({ schema: "ocx-stop-capabilities/1", connectedClientOwnership: true, pid: options.clientServer === "wrong-pid" ? fakePid + 1 : fakePid }, { headers }); } if (!url.startsWith("http://127.0.0.1:10100/api/stop")) return unexpected(`fetch(${url})`); urls.push(url); + transports.push(viaDirect ? "direct" : "global"); + if (options.connectDuringPost) writeHubConnection(); const body = options.serverTeardown && !options.apiFailure ? await (await import("../../src/server/stop-teardown")).performStopTeardown(new URL(url)) : options.response; @@ -215,7 +280,12 @@ mock.module("../../src/server/direct-local-http", () => ({ ...direct, io?.onResponseStatus?.(409); throw new Error("fixture truncated stop refusal"); } - return globalThis.fetch(input, { ...init, method: init?.method ?? "GET", redirect: "error" }); + viaDirect = true; + try { + return globalThis.fetch(input, { ...init, method: init?.method ?? "GET", redirect: "error" }); + } finally { + viaDirect = false; + } }, })); @@ -234,14 +304,23 @@ if (options.exitCleanup) { const bindings = { catalogHealer: null, routingHealer: null, guardian: { stop() {} }, historyGuardian: null, healthSentinel: null, siblingStart: false, isRecyclingForExit: () => false, decideStartExitTeardown: dispatch.decideStartExitTeardown, - preserveConnectedClientRoutingOnExit: (restore as Record).preserveConnectedClientRoutingOnExit, + preserveConnectedClientRoutingOnExit: (teardown: Parameters[0]) => { + const decision = restore.preserveConnectedClientRoutingOnExit(teardown); + if (options.connectAfterExitInspection) writeHubConnection(); + if (options.pendingAfterExitInspection) publishPending(); + return decision; + }, + stopRestoreWriteGuard: (await import("../../src/cli/stop-restore-guard")).stopRestoreWriteGuard, revertSystemEnv() {}, removePid() {}, removeRuntimePort() {}, serviceEnvironmentOwnedHere: () => true, getCodexHome: () => "fixture", codexHomeIsAbsent: () => false, - restoreNativeCodex: () => { calls.native++; return options.restore; }, + restoreNativeCodex: (restoreOptions?: { withWriteGuard?: (write: () => T) => T }) => { + const write = () => { calls.native++; return options.restore; }; + return restoreOptions?.withWriteGuard ? restoreOptions.withWriteGuard(write) : write(); + }, stripGrokConfig: () => { calls.grok++; return { ok: true, changed: true, message: "stripped" }; }, }; const cleanup = new Function(...Object.keys(bindings), "let cleaned = false; let cleanupSucceeded = true;\n" + source.slice(from, to) + "\nreturn syncCleanup();"); - cleanup(...Object.values(bindings)); - process.exit(0); + const succeeded = cleanup(...Object.values(bindings)); + process.exit(succeeded ? 0 : 1); } await import("../../src/cli/index"); diff --git a/tests/fixtures/stop-intent-runner.ts b/tests/fixtures/stop-intent-runner.ts new file mode 100644 index 00000000000..1a3a391d8c6 --- /dev/null +++ b/tests/fixtures/stop-intent-runner.ts @@ -0,0 +1,100 @@ +/** Real management stop intent/restart admission; teardown and host effects are controlled. */ +import { mock } from "bun:test"; +import type { StopTeardownBody } from "../../src/server/stop-teardown"; + +const mode = process.argv[2]!; +const waiting: Array<{ resolve(body: StopTeardownBody): void; reject(error: Error): void }> = []; +const service = await import("../../src/service"); +mock.module("../../src/service", () => ({ ...service, + installedServiceRespawnRisk: () => "none", stopServiceIfInstalledDetailed: () => "absent", +})); +const teardown = await import("../../src/server/stop-teardown"); +mock.module("../../src/server/stop-teardown", () => ({ ...teardown, + performStopTeardown: () => new Promise((resolve, reject) => { waiting.push({ resolve, reject }); }), +})); +const { handleManagementAPI } = await import("../../src/server/management-api"); +const { acceptSystemRestart, noteExplicitShutdownRequested } = await import("../../src/server/management/system-restart"); +const config = { port: 10100, hostname: "127.0.0.1", providers: {}, defaultProvider: "openai" }; +const refused: StopTeardownBody = { success: false, sharedTeardown: "refused", message: "fixture ownership pending" }; +const accepted: StopTeardownBody = { success: true, sharedTeardown: "performed", message: "fixture restored" }; +const scheduledStops: Array<() => void> = []; +const nativeTimeout = globalThis.setTimeout; +let now = 10_000; +let enteredDrain!: () => void; +let releaseDrain!: () => void; +const drainEntered = new Promise(resolve => { enteredDrain = resolve; }); +const heldDrain = new Promise(resolve => { releaseDrain = resolve; }); +let enteredListener!: () => void; +let releaseListener!: () => void; +const listenerEntered = new Promise(resolve => { enteredListener = resolve; }); +const heldListener = new Promise(resolve => { releaseListener = resolve; }); +const drainBudgets: number[] = []; +const healthWaits: boolean[] = []; +// Accepted stop timers are observed, never executed: this fixture must not exit or drain the host. +globalThis.setTimeout = ((work: () => void, ms?: number, ...args: unknown[]) => { + if (ms === 200) { scheduledStops.push(work); return 0; } + return nativeTimeout(work, ms, ...args); +}) as typeof setTimeout; +async function beginStop() { + const req = new Request("http://127.0.0.1:10100/api/stop", { method: "POST", headers: { Host: "127.0.0.1:10100" } }); + let early: unknown; + const settled = handleManagementAPI(req, new URL(req.url), config as never, {}, "admin-token") + .then(response => ({ status: response?.status }), error => ({ error: String(error) })) + .then(result => { early = result; return result; }); + while (!waiting.length) { + if (early) throw new Error(`stop did not enter teardown: ${JSON.stringify(early)}`); + await new Promise(resolve => nativeTimeout(resolve, 1)); + } + return { settled, work: waiting.shift()! }; +} +function startRestart() { + const observed = { spawned: 0 }; + let scheduled: (() => void | Promise) | undefined; + acceptSystemRestart({ + now: () => now, + isShutdownDraining: () => false, getActiveTurnCount: () => 0, + acquireTemporaryDrain: () => ({ release() {} }), beginShutdownDrain() {}, + schedule: work => { scheduled = work; }, scheduleDeadline: () => () => {}, + drainAndShutdown: async (_server, ms) => { + drainBudgets.push(ms ?? -1); enteredDrain(); + if (mode.startsWith("drain-")) await heldDrain; + return true; + }, stopListener: async () => { + enteredListener(); + if (mode.startsWith("listener-")) await heldListener; + }, listenPort: () => 10100, + isDesktopSupervised: () => false, isSupervisedServiceChild: () => false, + spawnStart(_port, waitForHealth) { observed.spawned++; healthWaits.push(waitForHealth ?? true); }, markRecycling() {}, exitProcess() {}, + }, { onAccepted() {}, beforeScheduledDrain: () => true }); + if (!scheduled) throw new Error("restart probe was not scheduled"); + return { observed, done: Promise.resolve(scheduled()) }; +} +try { + const restartBeforeStop = mode.startsWith("drain-") || mode.startsWith("listener-") ? startRestart() : undefined; + if (restartBeforeStop) await (mode.startsWith("listener-") ? listenerEntered : drainEntered); + const first = await beginStop(); + const outcomes: unknown[] = []; + const restart = restartBeforeStop ?? startRestart(); + if (restartBeforeStop) releaseDrain(); + if (mode.startsWith("listener-")) releaseListener(); + for (let i = 0; i < 20; i++) await Promise.resolve(); + const probes: number[] = [restart.observed.spawned]; + if (mode === "concurrent-refusals" || mode === "accepted-control") { + const second = await beginStop(); + first.work.resolve(mode === "accepted-control" ? accepted : refused); + outcomes.push(await first.settled); + await Promise.resolve(); + probes.push(restart.observed.spawned); + second.work.resolve(refused); + outcomes.push(await second.settled); + } else { + if (mode === "signal-control") noteExplicitShutdownRequested(); + if (mode.includes("deadline-")) now += 60_001; + if (mode === "throws" || mode === "drain-throws") first.work.reject(new Error("fixture teardown error")); + else first.work.resolve(refused); + outcomes.push(await first.settled); + } + await restart.done; + probes.push(restart.observed.spawned); + console.log(JSON.stringify({ probes, outcomes, scheduledStops: scheduledStops.length, drainBudgets, healthWaits })); +} finally { globalThis.setTimeout = nativeTimeout; } diff --git a/tests/fixtures/stop-restore-history-runner.ts b/tests/fixtures/stop-restore-history-runner.ts new file mode 100644 index 00000000000..181fc2f20e2 --- /dev/null +++ b/tests/fixtures/stop-restore-history-runner.ts @@ -0,0 +1,145 @@ +import { mock } from "bun:test"; +import { Database } from "bun:sqlite"; +import { existsSync, mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +// Only the scheduling seam is replaced. Both history implementations and all persistence +// are real, including the async Worker's DB, rollout and manifest writes. +const [mode, ownership] = process.argv.slice(2); +const ocx = process.env.OPENCODEX_HOME!; +const codex = process.env.CODEX_HOME!; +mkdirSync(ocx, { recursive: true }); mkdirSync(codex, { recursive: true }); +delete process.env.OCX_SERVICE; +const configPath = join(ocx, "config.json"); +const nativePath = join(codex, "config.toml"); +const config = { port: 10100, providers: {}, defaultProvider: "openai", syncResumeHistory: false }; +writeFileSync(configPath, JSON.stringify(config)); +writeFileSync(nativePath, 'model = "gpt-5"\n'); +const { withConfigMutationLockSync } = await import("../../src/config"); +const { markClientConnectPending } = await import("../../src/client/state"); +const historyProvider = await import("../../src/codex/history-provider"); +const realSyncHistory = historyProvider.syncCodexHistoryProvider; +const history = await import("../../src/codex/history-job"); +const realHistory = history.runCodexHistoryJob; +let armed = false; +let publication: ReturnType | undefined; +let workerResult: unknown; +let backup = ""; +const dbPath = join(codex, "state_5.sqlite"); +const rollout = join(codex, "rollout.jsonl"); +function snapshot() { + const db = new Database(dbPath, { readonly: true }); + try { + return { row: db.query("SELECT model_provider, source, has_user_event FROM threads").get(), + rollout: readFileSync(rollout, "utf8"), manifest: existsSync(backup) ? readFileSync(backup, "utf8") : null }; + } finally { db.close(); } +} +function publish() { + withConfigMutationLockSync(() => { + if (ownership === "pending") markClientConnectPending("b".repeat(64)); + if (ownership === "hub") { + writeFileSync(configPath, JSON.stringify({ ...config, runtimeRole: "client", client: { + serverUrl: "https://hub.example.test", managementUrl: "https://hub.example.test", + managementTransport: "direct", selectedClients: ["codex"], tokenEnv: "OPENCODEX_API_AUTH_TOKEN", + apiKeyId: "fixture-client", tokenFingerprint: "a".repeat(64), protocolVersion: 1, + connectedAt: "2026-10-01T00:00:00.000Z", + } })); + writeFileSync(nativePath, 'openai_base_url = "https://hub.example.test/v1"\n'); + } + publication = snapshot(); + }); +} +mock.module("../../src/codex/history-job", () => ({ ...history, runCodexHistoryJob: async (...args: Parameters) => { + if (armed && mode !== "sync") { await Promise.resolve(); publish(); } + const result = await realHistory(...args); + if (armed) workerResult = result; + return result; +} })); +mock.module("../../src/codex/history-provider", () => ({ ...historyProvider, + syncCodexHistoryProvider: (...args: Parameters) => { + if (armed && mode === "sync") publish(); + const result = realSyncHistory(...args); + if (armed && mode === "sync") workerResult = result; + return result; + }, +})); +const inject = await import("../../src/codex/inject"); +const seeded = await inject.injectCodexConfig(10100, config as never); +if (!seeded.success) throw new Error(`seed failed: ${seeded.message}`); +writeFileSync(rollout, JSON.stringify({ type: "session_meta", payload: { + id: "thread-1", model_provider: "opencodex", source: "cli", cwd: codex, +} }) + "\n"); +const db = new Database(dbPath); +db.exec("CREATE TABLE threads (id TEXT PRIMARY KEY, rollout_path TEXT NOT NULL, model_provider TEXT NOT NULL, source TEXT NOT NULL, first_user_message TEXT NOT NULL, has_user_event INTEGER NOT NULL DEFAULT 0)"); +db.run("INSERT INTO threads VALUES ('thread-1', ?, 'opencodex', 'cli', 'hello', 1)", [rollout]); +db.close(); +const canonicalDb = join(realpathSync.native(codex), "state_5.sqlite"); +backup = historyProvider.historyBackupPathFor(canonicalDb); +writeFileSync(backup, JSON.stringify({ version: 1, stateDbPath: canonicalDb, entries: { + "thread-1": { id: "thread-1", rolloutPath: rollout, modelProvider: "opencodex", source: "exec", hasUserEvent: 0 }, +} })); +// Independent SQLite connections prove C/H remain held at the real rollout write +// and manifest deletion boundaries, rather than accepting an entry-only check. +const lockChecks: { phase: string; config: boolean; history: boolean }[] = []; +if (mode === "sync" && ownership === "standalone") { + const identity = await import("../../src/codex/user-identity"); + const historyLockPath = identity.resolveCodexHistorySerializationDatabasePath(identity.resolveEffectiveUserIdentity(), codex, canonicalDb); + function held(path: string): boolean { + const probe = new Database(path); + try { probe.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE; ROLLBACK"); return false; } + catch (error) { if ((error as { code?: string }).code === "SQLITE_BUSY") return true; throw error; } + finally { probe.close(); } + } + const check = (phase: string) => lockChecks.push({ phase, + config: held(join(ocx, "config-mutation.sqlite")), history: held(historyLockPath) }); + historyProvider.setBeforeStrictHistoryRolloutAppendForTests(() => { check("rollout"); }); + historyProvider.setBeforeHistoryBackupConsumeForTests(() => { check("manifest"); }); +} +const receipts = await import("../../src/config/pending-teardown"); +const receipt = receipts.claimPendingTeardown({ hostname: "127.0.0.1", port: 10100 }, "exact", process.pid); +// Contend on the real H database, without blocking C or changing client ownership. +let heldHistory: Database | undefined; +const grok = await import("../../src/grok/inject"); +if (ownership === "history-busy") { + // Let the real serialization owner create and harden the H database first. A file this + // fixture created itself would carry the umask mode, and on POSIX H refuses anything but + // 0600 as an unsafe path; that would exercise "unsafe-path", not the contention under test. + const { withHistoryWriteSerialization } = await import("../../src/codex/history-lock"); + const prepared = withHistoryWriteSerialization(realpathSync.native(codex), canonicalDb, () => undefined); + if (prepared.kind !== "completed") throw new Error(`H preparation failed: ${prepared.reason}`); + const identity = await import("../../src/codex/user-identity"); + const lockPath = identity.resolveCodexHistorySerializationDatabasePath(identity.resolveEffectiveUserIdentity(), realpathSync.native(codex), canonicalDb); + heldHistory = new Database(lockPath); + heldHistory.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + mkdirSync(process.env.GROK_HOME!, { recursive: true }); + const seededGrok = grok.injectGrokConfig(10100, []); + if (!seededGrok.ok || !grok.grokManagedBlockPresent()) throw new Error("Grok seed failed"); + publication = snapshot(); +} +armed = true; +const stop = await import("../../src/cli/stop-restore-guard"); +const guard = stop.stopRestoreWriteGuard(); +let result: unknown; +try { result = mode === "sync" + ? inject.restoreNativeCodex({ withWriteGuard: guard.run }) + : mode === "api" + ? await (await import("../../src/server/stop-teardown")).performStopTeardown(new URL("http://127.0.0.1:10100/api/stop")) + : await (await import("../../src/cli/stop-restore")).restoreSharedClientStateAfterStop(() => {}); +} catch (error) { result = { error: String(error), boundary: guard.boundary() }; } +let grokResult: unknown; +if (ownership === "history-busy") { + try { grokResult = guard.run(() => grok.stripGrokConfig()); } + catch (error) { grokResult = { error: String(error) }; } + heldHistory!.close(); +} +const record = { sharedTeardown: "pending" as import("../../src/cli/stop-report").StopRunRecord["sharedTeardown"], receiptClearFailed: false }; +const observed = result as { boundary?: "hub-owned" | "unproven"; sharedTeardown?: string }; +const boundary = observed.boundary ?? (mode === "api" + ? observed.sharedTeardown === "not-owned" ? "hub-owned" : observed.sharedTeardown === "refused" ? "unproven" : undefined + : undefined); +if (boundary) { + stop.settleUnrestoredStopBoundary(boundary, record, [receipt.nonce]); +} +console.log(JSON.stringify({ result, before: publication, after: snapshot(), workerResult, lockChecks, + boundary: guard.boundary(), grokResult, grokManaged: grok.grokManagedBlockPresent(), native: readFileSync(nativePath, "utf8"), + receiptExists: existsSync(receipts.pendingTeardownPathFor(receipt.nonce)), settled: record.sharedTeardown })); diff --git a/tests/fixtures/stop-restore-writer-runner.ts b/tests/fixtures/stop-restore-writer-runner.ts new file mode 100644 index 00000000000..bb5cdb26fb5 --- /dev/null +++ b/tests/fixtures/stop-restore-writer-runner.ts @@ -0,0 +1,99 @@ +/** Real config/N/C/catalog writes in a disposable home; no network or host process control. */ +import { mock } from "bun:test"; +import { existsSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { loadConfig, withConfigMutationLockSync } from "../../src/config"; +import { injectCodexConfig } from "../../src/codex/inject"; +import { CODEX_CONFIG_PATH, DEFAULT_CATALOG_PATH, getCodexHome } from "../../src/codex/paths"; +import { acquireConfigWriteLock, releaseConfigWriteLock } from "../../src/codex/config-write-lock"; +import { openCodexCoordinatorTransaction } from "../../src/codex/transition-state"; +import { resolveCodexCoordinatorDatabasePath, resolveEffectiveUserIdentity } from "../../src/codex/user-identity"; +import { markClientConnectPending } from "../../src/client/state"; + +const mode = process.argv[2]!; +const home = process.env.OPENCODEX_HOME!; +const coordinator = resolveCodexCoordinatorDatabasePath(resolveEffectiveUserIdentity(), getCodexHome()); +const config = { ...loadConfig(), port: 10100, syncResumeHistory: false }; +const connection = { serverUrl: "https://hub.example.test", managementUrl: "https://hub.example.test", + managementTransport: "direct", selectedClients: ["codex"], tokenEnv: "OPENCODEX_API_AUTH_TOKEN", + apiKeyId: "fixture-client", tokenFingerprint: "a".repeat(64), protocolVersion: 1, connectedAt: "2026-10-01T00:00:00.000Z" }; +function publishHub() { + withConfigMutationLockSync(() => { + writeFileSync(join(home, "config.json"), JSON.stringify({ ...config, runtimeRole: "client", client: connection })); + writeFileSync(CODEX_CONFIG_PATH, 'openai_base_url = "https://hub.example.test/v1"\n'); + writeFileSync(DEFAULT_CATALOG_PATH, JSON.stringify({ models: [{ slug: "hub/model" }] })); + }); +} +const snapshot = () => [CODEX_CONFIG_PATH, DEFAULT_CATALOG_PATH].map(path => readFileSync(path, "utf8")); +let nativeBlocker: ReturnType | undefined; +let fileBlocker: Awaited> | undefined; +const history = await import("../../src/codex/history-job"); +mock.module("../../src/codex/history-job", () => ({ ...history, runCodexHistoryJob: async () => ({ kind: "skipped" }) })); +const grok = await import("../../src/grok/inject"); +let grokCalls = 0; +let lateSnapshot: string[] | undefined; +const lateHistory = mode === "late-history-hub"; +const lateCatalog = mode === "late-catalog-hub"; +let historyArmed = false; +let catalogArmed = false; +if (lateHistory) mock.module("../../src/codex/history-job", () => ({ ...history, runCodexHistoryJob: async () => { + if (!historyArmed) return { kind: "skipped" }; + await Promise.resolve(); publishHub(); lateSnapshot = snapshot(); return { kind: "skipped" }; +} })); +const fileLocks = await import("../../src/codex/config-write-lock"); +const releaseFile = fileLocks.releaseConfigWriteLock; +if (lateCatalog) mock.module("../../src/codex/config-write-lock", () => ({ ...fileLocks, + releaseConfigWriteLock: (held: Parameters[0]) => { + releaseFile(held); + if (catalogArmed) { catalogArmed = false; publishHub(); lateSnapshot = snapshot(); } + }, +})); +mock.module("../../src/grok/inject", () => ({ ...grok, stripGrokConfig: () => { grokCalls++; return { ok: true, changed: false, message: "fixture" }; } })); +const log = console.log; +console.log = () => {}; +try { + writeFileSync(CODEX_CONFIG_PATH, 'model = "gpt-5"\n'); + writeFileSync(DEFAULT_CATALOG_PATH, JSON.stringify({ models: [{ slug: "gpt-5" }] })); + const initialized = openCodexCoordinatorTransaction(coordinator); + initialized.commit(); initialized.close(); + const seed = await injectCodexConfig(10100, config, { catalogPath: DEFAULT_CATALOG_PATH }); + if (!seed.success) throw new Error(`seed failed: ${seed.message}`); + writeFileSync(DEFAULT_CATALOG_PATH, JSON.stringify({ models: [{ slug: "gpt-5" }, { slug: "fixture/model" }] })); + const { restoreBoundaryOwnership, restoreSharedClientStateAfterStop } = await import("../../src/cli/stop-restore"); + // Keep lifecycle fixture admission local; N, C and the config-file lock below are real. + const inspected = restoreBoundaryOwnership({ withLifecycle: work => work({} as never) }); + if (inspected !== "restore") throw new Error(`unexpected initial boundary: ${inspected}`); + historyArmed = true; + catalogArmed = true; + const apiMode = mode.startsWith("api-"); + if (mode.endsWith("file-hub")) { + fileBlocker = await acquireConfigWriteLock(CODEX_CONFIG_PATH, { timeoutMs: 0 }); + if (!fileBlocker.ok) throw new Error("file blocker acquisition failed"); + } else if (!lateCatalog && !lateHistory) nativeBlocker = openCodexCoordinatorTransaction(coordinator); + let completed = false; + const pending = (apiMode + ? (await import("../../src/server/stop-teardown")).performStopTeardown(new URL("http://127.0.0.1:10100/api/stop")) + : restoreSharedClientStateAfterStop(() => {})).finally(() => { completed = true; }); + if (lateCatalog || lateHistory) { + const result = await pending; + if (!lateSnapshot) throw new Error("late publication checkpoint was not reached"); + log(JSON.stringify({ result, before: lateSnapshot, after: snapshot(), grokCalls, marker: false })); + process.exitCode = 0; + } else { + // Observe the real restore waiting, before allowing the writer's publication to win. + await Bun.sleep(100); + if (completed) throw new Error("restore did not wait for the concurrent writer"); + if (mode.endsWith("native-pending")) withConfigMutationLockSync(() => markClientConnectPending("b".repeat(64))); + else publishHub(); + const before = snapshot(); + if (fileBlocker?.ok) releaseConfigWriteLock(fileBlocker.handle); + fileBlocker = undefined; + nativeBlocker?.rollback(); nativeBlocker?.close(); nativeBlocker = undefined; + const result = await pending; + log(JSON.stringify({ result, before, after: snapshot(), grokCalls, marker: existsSync(join(home, "client-connect-pending")) })); + } +} finally { + if (fileBlocker?.ok) releaseConfigWriteLock(fileBlocker.handle); + nativeBlocker?.rollback(); nativeBlocker?.close(); + for (const suffix of ["", "-journal", "-wal", "-shm"]) rmSync(coordinator + suffix, { force: true }); +} diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index f31d821f279..465d8757323 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -11,6 +11,9 @@ "fix-plain-data-own-keys.test.ts": "lib", "fix-plain-data-shape.test.ts": "lib", "fix-eventstream-cancel-wait.test.ts": "responses", + "stop-restore-history-ownership.test.ts": "service", + "cli-stop-refusal-transport.test.ts": "cli", + "stop-restoration-preflight.test.ts": "service", "native-source-priority.test.ts": "codex-integration", "server-native-catalog-priority.test.ts": "server", "claude-intercept-acl-barrier.test.ts": "claude-integration", diff --git a/tests/lib/process-control-graceful.test.ts b/tests/lib/process-control-graceful.test.ts index f5086f49035..9f20eb386c4 100644 --- a/tests/lib/process-control-graceful.test.ts +++ b/tests/lib/process-control-graceful.test.ts @@ -81,6 +81,29 @@ describe("stopProxyGracefully", () => { } }); + test("not-owned acceptance narrows to callers that opt in, never past a sent deferral", async () => { + // A sibling's `not-owned` is a truthful final answer; a Link Child caller refuses it + // because an old build says it while preserving the route this stop must restore. + // The option only narrows a no-nonce stop — a sent receipt is never confirmed this way. + for (const [nonce, accept, expected] of [ + [undefined, true, true], + [undefined, false, "teardown-unconfirmed"], + ["receipt-nonce", true, "teardown-unconfirmed"], + ["receipt-nonce", false, "teardown-unconfirmed"], + ] as const) { + const result = await stopProxyGracefully(4242, { + readRuntime: () => ({ port: 10199 }), + fetchFn: (async () => new Response(JSON.stringify({ success: true, sharedTeardown: "not-owned" }))) as typeof fetch, + waitExit: () => true, + ...(nonce ? { deferSharedTeardownNonce: nonce } : {}), + acceptNotOwnedTeardown: accept, + exitTimeoutMs: 1, + env: {}, + }); + expect(result).toBe(expected); + } + }); + test("an unconfirmed response still requires process exit", async () => { expect(await stopProxyGracefully(4242, { readRuntime: () => ({ port: 10100 }), diff --git a/tests/providers/xai/grok-lifecycle.test.ts b/tests/providers/xai/grok-lifecycle.test.ts index 34abd0acd9a..c7b5feaf50d 100644 --- a/tests/providers/xai/grok-lifecycle.test.ts +++ b/tests/providers/xai/grok-lifecycle.test.ts @@ -115,12 +115,13 @@ describe("Grok fence lifecycle wiring", () => { // routing belongs to the live proxy it ran beside, and a connected client, whose routing, // catalog and Grok config belong to its connection and are never written back on start (#7563). expect(stopFn).toContain("const restoreBlocked = ownershipBlocked || inheritedBlocks || nativeRestoreHandledByProxy || stoppingSibling || connectedClientSkip !== null;"); - expect(stopFn).toContain("if (!restoreBlocked) {"); + expect(stopFn).toContain("if (!restoreBlocked && boundary === \"restore\") {"); expect(stopFn).toContain("await restoreSharedClientStateAfterStop(reportRetainedCodexProviderTable)"); - expect(restoreFn).toContain("restoreNativeCodexAsync()"); + expect(restoreFn).toContain("restoreNativeCodexAsync({ withWriteGuard: guard.run })"); + expect(restoreFn).toContain("guard.run(() => stripGrokConfig())"); expect(restoreFn).not.toContain("revertSystemEnv()"); expect(restoreFn).toContain("stripGrokConfig()"); - expect(stopFn.indexOf("revertSystemEnv()")).toBeLessThan(stopFn.indexOf("if (!restoreBlocked) {")); + expect(stopFn.indexOf("revertSystemEnv()")).toBeLessThan(stopFn.indexOf("if (!restoreBlocked && boundary === \"restore\") {")); }); test("graceful stop skips caller restore only when the proxy performed it", () => { @@ -258,11 +259,15 @@ describe("Grok fence lifecycle wiring", () => { expect(stopFn).toContain("isProcessAlive(ownerPid) && isLikelyOcxProcess(ownerPid)"); expect(stopFn).not.toContain("isPendingTeardownAbandoned(read, isProcessAlive)"); expect(stopFn.indexOf("listPendingTeardowns()")).toBeLessThan(claimAt); - expect(stopFn).toContain("clearPendingTeardown(nonce)"); + expect(stopFn).toContain("clearRestoredStopReceipts(record, teardownNonce ? [teardownNonce, ...recoveredNonces] : recoveredNonces)"); expect(stopFn.indexOf("await restoreSharedClientStateAfterStop(reportRetainedCodexProviderTable)")) - .toBeLessThan(stopFn.indexOf("clearPendingTeardown(nonce)")); + .toBeLessThan(stopFn.indexOf("clearRestoredStopReceipts(record,")); // A receipt that survives its discharge would re-trigger recovery forever. - expect(stopFn).toContain("if (!clearPendingTeardown(nonce)) {"); + const receiptSource = readFileSync(repoPath("src", "cli", "stop-restore-guard.ts"), "utf8"); + expect(receiptSource).toContain("if (!clearPendingTeardown(nonce)) {"); + expect(receiptSource).toContain("return withConfigMutationLockSync(() => {"); + expect(receiptSource.indexOf("const boundary = inspectStopRestoreOwnership();")) + .toBeLessThan(receiptSource.indexOf("clearPendingTeardown(nonce)")); }); test("an unconfirmed inherited obligation blocks the restore, it does not merely warn", () => { @@ -272,7 +277,7 @@ describe("Grok fence lifecycle wiring", () => { // attempt at this only logged a warning and then restored anyway, which is not a gate. expect(stopFn).toContain("await abandonedTeardownIsSafeToFinish(read.receipt.endpoint)"); expect(stopFn).toContain("const restoreBlocked = ownershipBlocked || inheritedBlocks"); - expect(stopFn).toContain("if (!restoreBlocked) {"); + expect(stopFn).toContain("if (!restoreBlocked && boundary === \"restore\") {"); // The restore is reached only through that gate — no other call site may bypass it. const restoreCalls = stopFn.split("await restoreSharedClientStateAfterStop(reportRetainedCodexProviderTable)").length - 1; expect(restoreCalls).toBe(1); @@ -443,7 +448,8 @@ describe("POST /api/stop teardown", () => { expect(handler).toContain("performStopTeardown(url, { ownsReceipt: deferralMatchesReceipt })"); const teardownSource = readFileSync(repoPath("src", "server", "stop-teardown.ts"), "utf8"); expect(teardownSource).toContain('await import("../grok/inject")'); - expect(teardownSource).toContain("stripGrokConfig()"); + expect(teardownSource).toContain("grok = guard.run(() => strip())"); + expect(teardownSource).toContain("restoreNativeCodexAsync(options)"); }); test("an unreadable scheduler state gets the same diagnosis from the CLI and the API", () => { diff --git a/tests/service/shutdown-drain.test.ts b/tests/service/shutdown-drain.test.ts index 329f761b797..dbd30829ad1 100644 --- a/tests/service/shutdown-drain.test.ts +++ b/tests/service/shutdown-drain.test.ts @@ -1,4 +1,6 @@ import { EventEmitter } from "node:events"; +import { readFileSync } from "node:fs"; +import { repoPath } from "../helpers/repo-root"; import { PassThrough } from "node:stream"; import type { ChildProcessWithoutNullStreams } from "node:child_process"; import { create } from "@bufbuild/protobuf"; @@ -153,6 +155,123 @@ describe("server listener shutdown", () => { }); }); + +describe("routing cleanup before listener release", () => { + test("runs synchronous restoration after active-turn drain while a real loopback port remains held", async () => { + const events: string[] = []; + let routing = "local"; + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, reusePort: false, + fetch: () => new Response(routing) }); + const port = server.port!; + let replacement: ReturnType | undefined; + const turn = new AbortController(); + const lease = tryAdmitTurn(); + setServerStartupLifecycleReleaseForTests(async () => { events.push("startup-release"); }); + let draining: Promise | undefined; + try { + expect(lease).not.toBeNull(); + registerTurn(turn, lease!); + expect(getActiveTurnCount()).toBe(1); + expect(await (await fetch(server.url, { signal: AbortSignal.timeout(1_000) })).text()).toBe("local"); + draining = drainAndShutdown(server, 0, (): undefined => { + expect(turn.signal.aborted).toBe(true); + expect(getActiveTurnCount()).toBe(0); + events.push("restore-start"); + expect(() => { + replacement = Bun.serve({ hostname: "127.0.0.1", port, reusePort: false, + fetch: () => new Response("unexpected replacement") }); + }).toThrow(); + routing = "native"; + events.push("restore-finished"); + return undefined; + }); + expect(await draining).toBe(true); + expect(events).toEqual(["restore-start", "restore-finished", "startup-release"]); + replacement = Bun.serve({ hostname: "127.0.0.1", port, reusePort: false, + fetch: () => new Response(routing) }); + expect(await (await fetch(replacement.url, { signal: AbortSignal.timeout(1_000) })).text()).toBe("native"); + } finally { + if (lease) { + unregisterTurn(turn); + lease.release(); + } + await draining?.catch(() => {}); + await server.stop(true); + await replacement?.stop(true); + } + }); + + test("a synchronous restoration throw still releases a real listener and startup ownership", async () => { + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, reusePort: false, + fetch: () => new Response("local") }); + const port = server.port!; + let replacement: ReturnType | undefined; + const failure = new Error("restore failed"); + let released = 0; + setServerStartupLifecycleReleaseForTests(async () => { released += 1; }); + try { + await expect(drainAndShutdown(server, 0, () => { + throw failure; + })).rejects.toBe(failure); + expect(released).toBe(1); + replacement = Bun.serve({ hostname: "127.0.0.1", port, reusePort: false, + fetch: () => new Response("replacement") }); + expect(await (await fetch(replacement.url, { signal: AbortSignal.timeout(1_000) })).text()).toBe("replacement"); + } finally { + await server.stop(true); + await replacement?.stop(true); + } + }); + + test.each([true, false])("the actual SIGTERM handler restores before real listener stop (restore success: %s)", async restoreSucceeded => { + const events: string[] = []; + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, reusePort: false, + fetch: () => new Response("local") }); + const port = server.port!; + let replacement: ReturnType | undefined; + const realStop = server.stop.bind(server); + server.stop = (force?: boolean) => { events.push("listener-stop"); return realStop(force); }; + setServerStartupLifecycleReleaseForTests(async () => { events.push("startup-release"); }); + let exit!: (code: number) => void; + const exited = new Promise(resolve => { exit = resolve; }); + let cleaned = false; + // Execute the real signal handler, not a replacement orchestration or only syncCleanup. + // Host restoration is a fixture; drainAndShutdown and Bun's listen/stop are real. + const source = readFileSync(repoPath("src", "cli", "index.ts"), "utf8"); + const from = source.indexOf(" let shuttingDown = false;"); + const to = source.indexOf('\n process.on("SIGINT", shutdown);', from); + try { + expect(from).toBeGreaterThanOrEqual(0); + expect(to).toBeGreaterThan(from); + const code = new Bun.Transpiler({ loader: "ts" }).transformSync( + "function createShutdown() {\n" + source.slice(from, to) + "\nreturn shutdown;\n}", + ); + const bindings = { + server, config: { shutdownTimeoutMs: 0 }, drainAndShutdown, + syncCleanup: () => { + if (!cleaned) { cleaned = true; events.push("native-restore", "grok-strip"); } + return restoreSucceeded; + }, + noteExplicitShutdownRequested() {}, handledSignalExitCode: () => 0, + process: { exit }, console: { log() {} }, + }; + const shutdown = new Function(...Object.keys(bindings), code + "\nreturn createShutdown();")( + ...Object.values(bindings), + ) as (signal?: NodeJS.Signals) => void; + expect(await (await fetch(server.url, { signal: AbortSignal.timeout(1_000) })).text()).toBe("local"); + shutdown("SIGTERM"); + expect(await exited).toBe(restoreSucceeded ? 0 : 1); + expect(events).toEqual(["native-restore", "grok-strip", "listener-stop", "startup-release"]); + replacement = Bun.serve({ hostname: "127.0.0.1", port, reusePort: false, + fetch: () => new Response("replacement") }); + expect(await (await fetch(replacement.url, { signal: AbortSignal.timeout(1_000) })).text()).toBe("replacement"); + } finally { + await realStop(true); + await replacement?.stop(true); + } + }); +}); + describe("active turn tracking", () => { test("admit/bind/unregister tracks active turns through the boundary lease", () => { const ac1 = new AbortController(); diff --git a/tests/service/stop-deferred-teardown.test.ts b/tests/service/stop-deferred-teardown.test.ts index f1408d0132b..86d418b9ac6 100644 --- a/tests/service/stop-deferred-teardown.test.ts +++ b/tests/service/stop-deferred-teardown.test.ts @@ -69,6 +69,9 @@ async function runParentStop(options: { managerCascades?: boolean; guarded?: boolean; orphan?: boolean; exitCleanup?: boolean; nativeSignal?: boolean; apiFailure?: boolean; stopBodyFailure?: boolean; alreadyStopped?: boolean; absence?: "dead" | "unknown" | "live"; managerState?: "inactive" | "active" | "unknown"; + disconnectDuringProbe?: boolean; connectDuringDiscovery?: boolean; connectDuringPost?: boolean; lifecycleBusy?: boolean; + connectAfterBoundary?: boolean; pendingAfterBoundary?: boolean; connectDuringRestore?: boolean; + connectAfterExitInspection?: boolean; pendingAfterExitInspection?: boolean; pendingAfterGrok?: boolean; receiptClearBusy?: boolean; }) { if (options.clientTransport) connectClient(options.clientTransport); const child = spawnSync(process.execPath, [fixturePath("parent-stop-runner.ts")], { @@ -86,19 +89,59 @@ async function runParentStop(options: { const report = JSON.parse(readFileSync(reportPath, "utf8")) as { calls: { killed: number; native: number; grok: number; cleared: number; exited: number; service: number }; probes: string[]; - urls: string[]; nonce?: string; receiptExists: boolean; unexpectedIo: string[]; + urls: string[]; transports: string[]; nonce?: string; receiptExists: boolean; unexpectedIo: string[]; }; expect(report.unexpectedIo, child.stderr).toEqual([]); expect(report.urls, child.stderr).toHaveLength(options.clientServer || options.exitCleanup || options.alreadyStopped ? 0 : 1); return { ...report, exitCode: child.status, stderr: child.stderr, stdout: child.stdout }; } +describe("API stop through process exit cleanup", () => { + for (const query of ["valid", "foreign", "missing", "none"] as const) { + test(`${query} receipt request keeps the exit cleanup obligation accurate`, () => { + connectClient("link"); + const codexHome = join(home, "codex"); + mkdirSync(codexHome); + const child = spawnSync(process.execPath, [fixturePath("api-stop-exit-runner.ts"), query], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: home, CODEX_HOME: codexHome }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.signal, child.stderr).toBeNull(); + expect(child.status, child.stderr).toBe(0); + // spawnSync returned only after the child exited. The parent's receipt survives; + // existing parent CLI tests below cover restoration after proven-down observation. + const report = JSON.parse(readFileSync(join(home, "api-stop-exit-result.json"), "utf8")); + expect(report.responseStatus).toBe(200); + expect(report.body).toMatchObject({ success: true, sharedTeardown: query === "valid" ? "deferred" : "performed" }); + expect(report.receiptState).toBe("valid"); + expect(report.deferredOnExit).toBe(query === "valid"); + expect(report.calls).toMatchObject({ env: 1, pid: 1, runtime: 1 }); + const stopped = report.events.indexOf("listener-stop"); + const released = report.events.indexOf("startup-release"); + const exited = report.events.indexOf("exit-cleanup"); + expect(stopped).toBeGreaterThanOrEqual(0); + expect(released).toBeGreaterThan(stopped); + expect(exited).toBeGreaterThan(released); + if (query === "valid") { + expect(report.calls).toMatchObject({ apiNative: 0, apiGrok: 0, exitNative: 0, exitGrok: 0 }); + } else { + // Unbacked query values cannot suppress the child's existing restoration policy. + expect(report.calls).toMatchObject({ apiNative: 1, apiGrok: 1, exitNative: 1, exitGrok: 1 }); + expect(report.events.indexOf("native-api")).toBeLessThan(stopped); + expect(report.events.indexOf("grok-api")).toBeLessThan(stopped); + expect(report.events.indexOf("native-exit")).toBeGreaterThan(exited); + } + }); + } +}); + describe("parent CLI shared teardown completion", () => { test("consecutive connected stops preserve configuration and the stopped managed client remains update-eligible", async () => { - const first = await runParentStop({ receipt: true, clientTransport: "link", serverTeardown: true, json: true, + const first = await runParentStop({ receipt: true, clientTransport: "hub", serverTeardown: true, json: true, response: null, restore: restoreResult(true) }); const before = readFileSync(join(home, "config.json"), "utf8"); - const second = await runParentStop({ receipt: true, clientTransport: "link", alreadyStopped: true, json: true, + const second = await runParentStop({ receipt: true, clientTransport: "hub", alreadyStopped: true, json: true, response: null, restore: restoreResult(true) }); expect(first.exitCode).toBe(0); expect(second.exitCode, second.stderr).toBe(0); @@ -118,7 +161,8 @@ describe("parent CLI shared teardown completion", () => { expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: false, runtimeDown: false }); }); } - test("preservation probes bypass environment proxies without exposing the admin token", async () => { + for (const transport of ["hub", "link"] as const) test(`${transport} stop probes and POST bypass environment proxies without exposing the admin token`, async () => { + connectClient(transport); const targetPaths: string[] = []; const proxyPaths: string[] = []; const proxyTokens: string[] = []; @@ -173,9 +217,23 @@ describe("parent CLI shared teardown completion", () => { await target.stop(true); await proxy.stop(true); } }); + test("a disconnect committed during the capability probe cannot move the stop POST off direct transport", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "hub", serverTeardown: true, disconnectDuringProbe: true, + json: true, response: null, restore: restoreResult(true) }); + expect(outcome.probes.some(url => url.endsWith("/api/stop")), outcome.stderr).toBe(true); + expect(outcome.transports, outcome.stderr).toEqual(["direct"]); + }); + test("a standalone stop, and a connection committed during orphan discovery, keep the POST on direct transport", async () => { + const plain = await runParentStop({ receipt: true, json: true, response: { success: true, sharedTeardown: "deferred" }, + restore: restoreResult(true) }); + expect(plain.transports, plain.stderr).toEqual(["direct"]); + const connecting = await runParentStop({ receipt: true, orphan: true, connectDuringDiscovery: true, json: true, + response: { success: true, sharedTeardown: "deferred" }, restore: restoreResult(true) }); + expect(connecting.transports, connecting.stderr).toEqual(["direct"]); + }); for (const clientServer of ["legacy", "unknown", "wrong-pid", "malformed", "oversized", "unauthorized", "redirect", "body-timeout"] as const) { test(`connected stop refuses ${clientServer} preservation before service or POST`, async () => { - const outcome = await runParentStop({ receipt: true, clientTransport: "link", clientServer, json: true, + const outcome = await runParentStop({ receipt: true, clientTransport: "hub", clientServer, json: true, response: { success: true, sharedTeardown: "performed" }, restore: restoreResult(true) }); expect(outcome.calls).toMatchObject({ service: 0, killed: 0, native: 0, grok: 0, cleared: 0 }); expect(outcome.nonce).toBeUndefined(); @@ -194,10 +252,10 @@ describe("parent CLI shared teardown completion", () => { expect(outcome.exitCode).toBe(1); }); } - test("actual startup exit cleanup preserves freshly connected routing after a signal or API exit", async () => { + test("actual Link Child exit cleanup attempts native and Grok restoration", async () => { const outcome = await runParentStop({ receipt: true, clientTransport: "link", exitCleanup: true, response: null, restore: restoreResult(true) }); - expect(outcome.calls).toMatchObject({ native: 0, grok: 0, service: 0, cleared: 0, killed: 0 }); + expect(outcome.calls).toMatchObject({ native: 1, grok: 1, service: 0, cleared: 0, killed: 0 }); expect(outcome.exitCode).toBe(0); }); test("ordinary startup exit cleanup retains native and Grok restoration", async () => { @@ -205,8 +263,8 @@ describe("parent CLI shared teardown completion", () => { expect(outcome.calls).toMatchObject({ native: 1, grok: 1, service: 0 }); expect(outcome.exitCode).toBe(0); }); - test("PID-identified native client keeps the 404 to signal path", async () => { - const outcome = await runParentStop({ receipt: true, clientTransport: "link", nativeSignal: true, json: true, + test("PID-identified hub client keeps the 404 to signal path", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "hub", nativeSignal: true, json: true, status: 404, response: { error: "not found" }, restore: restoreResult(true) }); expect(outcome.calls).toMatchObject({ service: 1, killed: 1, native: 0, grok: 0 }); expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, sharedTeardown: "skipped" }); @@ -226,30 +284,199 @@ describe("parent CLI shared teardown completion", () => { expect(outcome.exitCode).toBe(1); expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: false, runtimeDown: false, sharedTeardown: "skipped" }); }); - // #7563: nothing on a connected client writes its Codex routing, catalog or Grok block back - // on start, so a stop that tore them down left them torn down. - for (const clientTransport of ["link", "hub"] as const) { - test(`a successful ${clientTransport} API stop preserves integrations and reports skipped JSON`, async () => { - const outcome = await runParentStop({ receipt: true, clientTransport, serverTeardown: true, json: true, - response: null, restore: restoreResult(true) }); - expect(outcome.urls).toEqual(["http://127.0.0.1:10100/api/stop"]); - expect(outcome.nonce).toBeUndefined(); - expect(outcome.calls).toMatchObject({ native: 0, grok: 0, cleared: 0 }); - expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, sharedTeardown: "skipped" }); - expect(outcome.exitCode, outcome.stderr).toBe(0); - }); + // #7563: hub clients route to an authenticated remote origin and preserve that routing on stop. + test("a successful hub API stop preserves integrations and reports skipped JSON", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "hub", serverTeardown: true, json: true, + response: null, restore: restoreResult(true) }); + expect(outcome.urls).toEqual(["http://127.0.0.1:10100/api/stop"]); + expect(outcome.nonce).toBeUndefined(); + expect(outcome.calls).toMatchObject({ native: 0, grok: 0, cleared: 0 }); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, sharedTeardown: "skipped" }); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }); - test(`a ${clientTransport} native client stop leaves the connection's integrations and claims no receipt`, async () => { - const outcome = await runParentStop({ receipt: true, clientTransport, - response: { error: "not found" }, status: 404, restore: restoreResult(true) }); - expect(outcome.urls).toEqual(["http://127.0.0.1:10100/api/stop"]); - expect(outcome.nonce).toBeUndefined(); - expect(outcome.calls).toMatchObject({ native: 0, grok: 0, cleared: 0 }); - expect(outcome.stdout).toContain("this " + (clientTransport === "link" ? "Remote Link Child" : "hub client") + "'s connection owns them"); - expect(outcome.exitCode, outcome.stderr).toBe(0); + test("a hub native client stop leaves the connection's integrations and claims no receipt", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "hub", + response: { error: "not found" }, status: 404, restore: restoreResult(true) }); + expect(outcome.urls).toEqual(["http://127.0.0.1:10100/api/stop"]); + expect(outcome.nonce).toBeUndefined(); + expect(outcome.calls).toMatchObject({ native: 0, grok: 0, cleared: 0 }); + expect(outcome.stdout).toContain("this hub client's connection owns them"); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }); + + test("a successful Link Child stop completes the parent's receipt-backed restoration", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "link", serverTeardown: true, json: true, + response: null, restore: restoreResult(true) }); + expect(outcome.calls).toMatchObject({ native: 1, grok: 1, cleared: 1 }); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, sharedTeardown: "restored" }); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }); + + test("a Link Child not-owned answer without a receipt still gets the parent's restoration", async () => { + // An old build answers not-owned for ANY connected client before it ever checks the + // deferral query, and this stop has no receipt to send. That answer preserves the + // link route instead of tearing it down, so it cannot confirm the parent's teardown: + // the parent owns the restoration itself, exactly as for an unconfirmed stop. + const outcome = await runParentStop({ receipt: false, clientTransport: "link", json: true, + response: { success: true, sharedTeardown: "not-owned" }, restore: restoreResult(true) }); + expect(outcome.urls).toEqual(["http://127.0.0.1:10100/api/stop"]); + expect(outcome.nonce).toBeUndefined(); + expect(outcome.calls).toMatchObject({ killed: 0, exited: 1, native: 1, grok: 1, cleared: 0 }); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, runtimeDown: true, sharedTeardown: "restored" }); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }); + + test("a connection committed during orphan discovery keeps its hub-owned not-owned answer", async () => { + // Disconnected at entry, hub-connected by the time the POST lands: the stopped runtime + // re-reads the fresh config and truthfully answers not-owned for the connection that + // now owns those integrations. Refusing it would restore over the brand-new connection. + const outcome = await runParentStop({ receipt: false, orphan: true, connectDuringDiscovery: true, + serverTeardown: true, json: true, response: null, restore: restoreResult(true) }); + expect(outcome.calls).toMatchObject({ killed: 0, native: 0, grok: 0 }); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, runtimeDown: true, sharedTeardown: "performed-by-proxy" }); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }); + + for (const [label, transition] of [["during orphan discovery", { orphan: true, connectDuringDiscovery: true }], + ["while the POST is in flight", { connectDuringPost: true }]] as const) { + for (const receipt of [true, false]) { + test(`a Link stop never restores over a hub connected ${label} (receipt ${receipt ? "claimed" : "unavailable"})`, async () => { + // #7736 r5: the late not-owned refusal must not let the parent restore over the hub that + // now owns routing. Only the ownership read at the restore boundary decides. + const outcome = await runParentStop({ receipt, clientTransport: "link", serverTeardown: true, json: true, + response: null, restore: restoreResult(true), ...transition }); + expect(outcome.calls, outcome.stderr).toMatchObject({ killed: 0, native: 0, grok: 0, cleared: receipt ? 1 : 0 }); + expect(outcome.receiptExists).toBe(false); + // Seen before the POST without a nonce, the hub's not-owned answer already confirms the stop. + const settledAtPost = "connectDuringDiscovery" in transition && !receipt; + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, runtimeDown: true, + sharedTeardown: settledAtPost ? "performed-by-proxy" : "skipped" }); + if (!settledAtPost) expect(outcome.stdout + outcome.stderr).toContain("A hub connection was committed while this stop ran"); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }); + } + } + for (const transition of ["connectAfterBoundary", "connectDuringRestore"] as const) { + for (const receipt of [true, false]) { + test(`R6 ${transition} preserves hub writes (receipt ${receipt})`, async () => { + const outcome = await runParentStop({ receipt, clientTransport: "link", json: true, + [transition]: true, response: { success: true, sharedTeardown: "not-owned" }, restore: restoreResult(true) }); + expect(outcome.calls, outcome.stderr).toMatchObject({ native: 0, grok: 0, cleared: receipt ? 1 : 0 }); + expect(outcome.receiptExists).toBe(false); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: true, sharedTeardown: "skipped" }); + expect(outcome.exitCode, outcome.stderr).toBe(0); + }, 20_000); + } + } + test("R6 pending connect after inspection preserves the obligation", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "link", json: true, pendingAfterBoundary: true, + response: { success: true, sharedTeardown: "not-owned" }, restore: restoreResult(true) }); + expect(outcome.calls, outcome.stderr).toMatchObject({ native: 0, grok: 0, cleared: 0 }); + expect(outcome.receiptExists).toBe(true); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: false, sharedTeardown: "refused" }); + expect(outcome.exitCode).toBe(1); + }, 20_000); + for (const mode of ["native-hub", "file-hub", "native-pending"] as const) { + test(`R6 real restore revalidates after waiting on writer (${mode})`, () => { + const codex = join(home, "codex"); mkdirSync(codex); + writeFileSync(join(home, "config.json"), JSON.stringify({ providers: {}, defaultProvider: "openai", syncResumeHistory: false })); + const child = spawnSync(process.execPath, [fixturePath("stop-restore-writer-runner.ts"), mode], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: home, CODEX_HOME: codex, GROK_HOME: join(home, "grok") }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const result = JSON.parse(child.stdout.trim().split("\n").at(-1)!); + expect(result.after).toEqual(result.before); + expect(result.grokCalls).toBe(0); + expect(result.result.boundary).toBe(mode === "native-pending" ? "unproven" : "hub-owned"); + expect(result.marker).toBe(mode === "native-pending"); + }, 20_000); + } + for (const transition of ["connectAfterExitInspection", "pendingAfterExitInspection"] as const) { + test(`R6 follow-up signal ${transition} protects shared writes`, async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "link", exitCleanup: true, + [transition]: true, response: null, restore: restoreResult(true) }); + expect(outcome.calls, outcome.stderr).toMatchObject({ native: 0, grok: 0, cleared: 0 }); + expect(outcome.exitCode, outcome.stderr).toBe(transition === "pendingAfterExitInspection" ? 1 : 0); + }, 20_000); + } + for (const transition of ["pendingAfterGrok", "receiptClearBusy"] as const) { + test(`R6 follow-up receipt ${transition} keeps the obligation`, async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "link", json: true, + [transition]: true, response: { success: true, sharedTeardown: "not-owned" }, restore: restoreResult(true) }); + expect(outcome.calls, outcome.stderr).toMatchObject({ native: 1, grok: 1, cleared: 0 }); + expect(outcome.receiptExists).toBe(true); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: false, sharedTeardown: "refused" }); + expect(outcome.exitCode).toBe(1); + }, 20_000); + } + for (const mode of ["api-native-hub", "api-file-hub", "api-native-pending", "late-catalog-hub", "late-history-hub"] as const) { + test(`R6 follow-up real writes (${mode}) preserve new ownership`, () => { + const codex = join(home, "codex"); mkdirSync(codex); + writeFileSync(join(home, "config.json"), JSON.stringify({ providers: {}, defaultProvider: "openai", syncResumeHistory: false })); + const child = spawnSync(process.execPath, [fixturePath("stop-restore-writer-runner.ts"), mode], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: home, CODEX_HOME: codex, GROK_HOME: join(home, "grok") }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const result = JSON.parse(child.stdout.trim().split("\n").at(-1)!); + expect(result.after).toEqual(result.before); + expect(result.grokCalls).toBe(0); + if (mode.startsWith("api-")) expect(result.result).toMatchObject({ + success: mode !== "api-native-pending", sharedTeardown: mode === "api-native-pending" ? "refused" : "not-owned", + }); + else expect(result.result.boundary).toBe("hub-owned"); + }, 20_000); + } + test("R6 follow-up API pending ownership returns 409 and keeps serving", () => { + connectClient("link"); + const codex = join(home, "codex"); mkdirSync(codex); + const child = spawnSync(process.execPath, [fixturePath("api-stop-exit-runner.ts"), "pending"], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: home, CODEX_HOME: codex }, + encoding: "utf8", timeout: 20_000, windowsHide: true, }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const result = JSON.parse(readFileSync(join(home, "api-stop-exit-result.json"), "utf8")); + expect(result).toMatchObject({ responseStatus: 409, refusedKeptServing: true, automaticRestarted: true, receiptState: "valid", + body: { success: false, sharedTeardown: "refused" }, + calls: { apiNative: 0, apiGrok: 0, exitNative: 0, exitGrok: 0 } }); + }, 20_000); + for (const mode of ["concurrent-refusals", "throws", "drain-refusal", "drain-throws", "deadline-refusal", "drain-deadline-refusal", "listener-deadline-refusal", "accepted-control", "signal-control"] as const) { + test(`${mode.endsWith("control") ? "stop intent control" : "R6 stop intent"} ${mode} retains the right restart authority`, () => { + const child = spawnSync(process.execPath, [fixturePath("stop-intent-runner.ts"), mode], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: home }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const result = JSON.parse(child.stdout.trim().split("\n").at(-1)!); + expect(result.probes).toEqual(mode === "concurrent-refusals" ? [0, 0, 1] : mode === "accepted-control" ? [0, 0, 0] : mode === "signal-control" ? [0, 0] : [0, 1]); + expect(result.scheduledStops).toBe(mode === "accepted-control" ? 1 : 0); + if (mode === "throws" || mode === "drain-throws") expect(result.outcomes).toEqual([{ error: "Error: fixture teardown error" }]); + else expect(result.outcomes.map((outcome: { status: number }) => outcome.status)) + .toEqual(mode === "concurrent-refusals" ? [409, 409] : mode === "accepted-control" ? [200, 409] : [409]); + if (mode === "deadline-refusal") { + expect(result.drainBudgets).toEqual([0]); + expect(result.healthWaits).toEqual([false]); + } + if (mode === "drain-deadline-refusal" || mode === "listener-deadline-refusal") { + expect(result.drainBudgets).toEqual([60_000]); + expect(result.healthWaits).toEqual([false]); + } + }, 20_000); } - + test("a busy client lifecycle lock at the restore boundary keeps the obligation instead of restoring", async () => { + const outcome = await runParentStop({ receipt: true, clientTransport: "link", lifecycleBusy: true, json: true, + response: { success: true, sharedTeardown: "not-owned" }, restore: restoreResult(true) }); + expect(outcome.calls, outcome.stderr).toMatchObject({ killed: 0, native: 0, grok: 0, cleared: 0 }); + expect(outcome.receiptExists).toBe(true); + expect(JSON.parse(outcome.stdout)).toMatchObject({ ok: false, sharedTeardown: "refused" }); + expect(outcome.exitCode).toBe(1); + }); test("receipt failure and unconfirmed child teardown cause real parent restoration without a kill", async () => { const outcome = await runParentStop({ receipt: false, response: { success: false, sharedTeardown: "performed" }, restore: restoreResult(true) }); @@ -451,26 +678,36 @@ describe("performStopTeardown", () => { expect(await response!.json()).toEqual({ schema: "ocx-stop-capabilities/1", connectedClientOwnership: true, pid: process.pid }); expect(readFileSync(join(home, "config.json"), "utf8")).toBe(before); }); - for (const transport of ["hub", "link"] as const) { - test(`persisted ${transport} ownership precedes receipt deferral and leaves its receipt intact`, async () => { - connectClient(transport); - const { claimPendingTeardown, readPendingTeardown } = await import("../../src/config/pending-teardown"); - const receipt = claimPendingTeardown(ENDPOINT, "exact"); - let restored = 0; - let stripped = 0; - for (const query of ["", `?deferSharedTeardown=1&teardownNonce=${receipt.nonce}`]) { - const body = await performStopTeardown(new URL(`http://127.0.0.1:10100/api/stop${query}`), { - restoreNativeCodex: async () => { restored++; return restoreResult(true); }, - stripGrok: () => { stripped++; return { ok: true, changed: true, message: "stripped" }; }, - }); - expect(body).toMatchObject({ success: true, sharedTeardown: "not-owned" }); - expect(body.message).toContain("ocx disconnect"); - } - expect(restored).toBe(0); - expect(stripped).toBe(0); - expect(readPendingTeardown(receipt.nonce).state).toBe("valid"); + test("persisted hub ownership precedes receipt deferral and leaves its receipt intact", async () => { + connectClient("hub"); + const { claimPendingTeardown, readPendingTeardown } = await import("../../src/config/pending-teardown"); + const receipt = claimPendingTeardown(ENDPOINT, "exact"); + let restored = 0; + let stripped = 0; + for (const query of ["", `?deferSharedTeardown=1&teardownNonce=${receipt.nonce}`]) { + const body = await performStopTeardown(new URL(`http://127.0.0.1:10100/api/stop${query}`), { + restoreNativeCodex: async () => { restored++; return restoreResult(true); }, + stripGrok: () => { stripped++; return { ok: true, changed: true, message: "stripped" }; }, + }); + expect(body).toMatchObject({ success: true, sharedTeardown: "not-owned" }); + expect(body.message).toContain("ocx disconnect"); + } + expect(restored).toBe(0); + expect(stripped).toBe(0); + expect(readPendingTeardown(receipt.nonce).state).toBe("valid"); + }); + + test("persisted Link ownership does not preserve an unauthenticated loopback route", async () => { + connectClient("link"); + let restored = 0; + let stripped = 0; + const body = await performStopTeardown(new URL("http://127.0.0.1:10100/api/stop"), { + restoreNativeCodex: async () => { restored++; return restoreResult(true); }, + stripGrok: () => { stripped++; return { ok: true, changed: true, message: "stripped" }; }, }); - } + expect(body).toMatchObject({ success: true, sharedTeardown: "performed" }); + expect([restored, stripped]).toEqual([1, 1]); + }); test("a hub server without a client connection still restores its own integrations", async () => { writeFileSync(join(home, "config.json"), JSON.stringify({ runtimeRole: "hub" })); diff --git a/tests/service/stop-restoration-preflight.test.ts b/tests/service/stop-restoration-preflight.test.ts new file mode 100644 index 00000000000..1316102fe0f --- /dev/null +++ b/tests/service/stop-restoration-preflight.test.ts @@ -0,0 +1,48 @@ +import { expect, test } from "bun:test"; +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { fixturePath, repoPath } from "../helpers/repo-root"; + +test("CR2 synchronous H contention reports history failure and still strips Grok", () => { + const root = mkdtempSync(join(tmpdir(), "ocx-stop-h-busy-")); + try { + const child = spawnSync(process.execPath, [fixturePath("stop-restore-history-runner.ts"), "sync", "history-busy"], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: join(root, "ocx"), CODEX_HOME: join(root, "codex"), GROK_HOME: join(root, "grok") }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const value = JSON.parse(child.stdout.trim().split("\n").at(-1)!); + expect(value.result).toMatchObject({ success: false, artifacts: { history: { state: "failed", changed: false, rows: 0, files: 0 } } }); + expect(value.result.artifacts.history.message).toContain("holding the history database"); + expect(value.boundary).toBe("restore"); + expect(value.grokResult).toMatchObject({ ok: true, changed: true }); + expect(value.grokManaged).toBe(false); + expect(value.native).toBe('model = "gpt-5"\n'); + expect(value.before).toBeDefined(); + expect(value.after).toEqual(value.before); + expect(value.receiptExists).toBe(true); + } finally { removeTreeWithRetry(root); } +}, 20_000); + +test("CR3 pending ownership refuses API stop before stopping the manager and keeps serving", () => { + const root = mkdtempSync(join(tmpdir(), "ocx-stop-preflight-")); + const ocx = join(root, "ocx"); + const codex = join(root, "codex"); + mkdirSync(ocx); mkdirSync(codex); + try { + const child = spawnSync(process.execPath, [fixturePath("api-stop-exit-runner.ts"), "pending"], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: ocx, CODEX_HOME: codex }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const value = JSON.parse(readFileSync(join(ocx, "api-stop-exit-result.json"), "utf8")); + expect(value).toMatchObject({ responseStatus: 409, refusedKeptServing: true, automaticRestarted: true, + receiptState: "valid", body: { success: false, sharedTeardown: "refused" }, + calls: { service: 0, apiNative: 0, apiGrok: 0, exitNative: 0, exitGrok: 0 } }); + } finally { removeTreeWithRetry(root); } +}, 20_000); diff --git a/tests/service/stop-restore-history-ownership.test.ts b/tests/service/stop-restore-history-ownership.test.ts new file mode 100644 index 00000000000..a8eefdaaccf --- /dev/null +++ b/tests/service/stop-restore-history-ownership.test.ts @@ -0,0 +1,41 @@ +import { expect, test } from "bun:test"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { fixturePath, repoPath } from "../helpers/repo-root"; + +for (const mode of ["parent", "api", "sync"]) { + for (const ownership of ["hub", "pending", "standalone"]) { + test(`R7 real history ${mode} respects ${ownership} at history entry`, () => { + const root = mkdtempSync(join(tmpdir(), "ocx-stop-history-")); + try { + const child = spawnSync(process.execPath, [fixturePath("stop-restore-history-runner.ts"), mode, ownership], { + cwd: repoPath(), env: { ...process.env, OPENCODEX_HOME: join(root, "ocx"), CODEX_HOME: join(root, "codex"), GROK_HOME: join(root, "grok") }, + encoding: "utf8", timeout: 20_000, windowsHide: true, + }); + expect(child.error, child.stderr).toBeUndefined(); + expect(child.status, child.stderr).toBe(0); + const value = JSON.parse(child.stdout.trim().split("\n").at(-1)!); + expect(value.before).toBeDefined(); + if (ownership === "standalone") { + expect(value.after.row).toEqual({ model_provider: "opencodex", source: "exec", has_user_event: 0 }); + expect(value.after.rollout).not.toBe(value.before.rollout); + expect(value.after.manifest).toBeNull(); + expect(value.result.success ?? !value.result.other).toBe(true); + if (mode === "sync") expect(value.lockChecks).toEqual([ + { phase: "rollout", config: true, history: true }, + { phase: "manifest", config: true, history: true }, + ]); + } else { + expect(value.after).toEqual(value.before); + if (mode === "api") expect(value.result).toMatchObject({ success: ownership === "hub", sharedTeardown: ownership === "hub" ? "not-owned" : "refused" }); + else expect(value.result.boundary).toBe(ownership === "hub" ? "hub-owned" : "unproven"); + expect(value.receiptExists).toBe(ownership === "pending"); + expect(value.settled).toBe(ownership === "hub" ? "skipped" : "refused"); + } + } finally { removeTreeWithRetry(root); } + }, 20_000); + } +}