From 1356384467f8948f62ad8f7ed82929f728d4e42e Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 16:52:08 -0500 Subject: [PATCH 01/10] docs: Add the plan for the praxis-mode live check (#1725) The plan writes the criteria for the default before the check, adds a dev script that runs a direct and a praxis scratch Cortex side by side, and records the result and the decision on #1725 and #1714. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../2026-11-10-praxis-mode-8-live-check.md | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md new file mode 100644 index 000000000..e677ad651 --- /dev/null +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -0,0 +1,141 @@ +# Praxis mode 8/8: Live check and the default decision — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Issue:** #1725 (part of tracker #1714, phase A, step 8). Steps #1718 to #1724 are closed. + +**Goal:** Write the criteria for the default, run Claude Code and Codex through praxis mode on a scratch Cortex, measure it against direct mode, and record the result and the decision on #1725 and #1714. + +**Architecture:** One new dev script, `scripts/dev/praxis-live-check.sh`, starts two scratch Cortex instances side by side (one direct, one praxis) under scratch HOMEs, on ports that do not touch the shared proxy. It measures interleaved latency and samples the CPU and heap of each process. The live check and the decision are GitHub comments, not code. + +**Tech Stack:** bash 3.2 (macOS), curl, ps, the Go `cortex` binary, the Rust `cortex-gateway` binary. + +**Spec:** `docs/superpowers/specs/2026-11-07-praxis-mode-design.md`, Section 5 "Live check" and Section 4 "Release and install" (`praxis.enabled` stays `false` until the live check is done). + +## Global Constraints + +- **Never touch the shared proxy.** Do not run `agentop setup`, `agentop service *`, `make dev-install` or `install.sh`. Do not use ports 47600-47606. Do not read or write the real `~/.cortex/` except to read field names and the router's server URLs. Every scratch process runs with `HOME=/home`. +- Scratch ports. Direct instance: proxy `127.0.0.1:49700`, session API `:49701`, stats `:49702`, transparent `:49703`, health `:49704`. Praxis instance: `:49710`-`:49714` in the same order, `praxis.listen: 127.0.0.1:49715`, `praxis.egress: 127.0.0.1:49716`. +- `cortex -supervise` finds `cortex-gateway` in the directory of its own binary (`cmd/cortex/supervise.go:239`). So both binaries go in one scratch `bin/` directory. +- Credentials: read an API key into an environment variable only. Never write a key into the repository, a commit, a plan, an issue comment or a file outside the scratch directory. Delete the scratch directory at the end (`rm -rf`). +- Bash 3.2: no `mapfile`, no associative arrays, no `${var,,}`. The script must pass `shellcheck`. +- Posts go to rossoctl/cortex issues #1725 and #1714 only. +- Text in comments and the PR body: ASD-STE100 style, short paragraphs, lists over prose. + +## Decisions taken in this plan + +- **D-A: The criteria live in this plan and in a comment on #1725 and #1714.** Reason: the issue says "write the criteria on this issue (#1714, Gates)". This plan is the reviewed copy. Task 2 posts them before Task 3 starts. +- **D-B: The default flip is NOT in this plan.** Reason: praxis mode turns off every outbound plugin by design (`cmd/cortex/praxis.go:33`) and agentop refuses `server`, `script` and `tools` in praxis mode (`praxisRefusal`). A default laptop install uses `tool-prune` and `inference-router`. So criterion C1 fails by design today, and the decision is "direct mode stays the default". If, against this, every criterion passes, the executor records "all criteria pass" and leaves the flip to Hai: a change of the default for every user needs a human decision. +- **D-C: Two scratch instances at the same time, not one instance restarted between modes.** Reason: the issue asks for interleaved latency. Alternate requests between the two proxies in one loop, so gateway noise hits both modes equally (see the 2026-10-02 latency probe method). +- **D-D: The latency probe sends a ~1 MiB Claude-Code-shaped `/v1/messages` body with an invalid `x-api-key`.** Reason: the upstream refuses it with 401 before a model runs, so it costs no tokens, and the parsers still read the full body. +- **D-E: The script is a dev tool, not CI.** Reason: it needs real credentials and a real upstream. It stays in `scripts/dev/` so the rollback check and phase B can run it again. +- **D-F: The PR holds the script and this plan only.** The results go on the issues, not into a repository document. Reason: the issue says "record on this issue"; a second copy drifts. + +## Criteria for the default (Task 2 posts these verbatim) + +Praxis mode can become the default only when **every** criterion passes: + +- **C1 Feature parity.** Every feature that a default laptop install has works in praxis mode: `tool-prune`, `inference-router` (`agentop server`), script plugins (`agentop script`), `agentop tools`. Status today: fails by design (D-B). +- **C2 Facts.** For the same live session, agentop shows in praxis mode: one session per agent run, the title, every timeline row, the tool call names, the subagent blocks, input/output/cache tokens equal to the `usage` in the raw capture, and cost equal to direct mode's price for those tokens. Zero rows with a `gateway_*` error kind, apart from the fail-closed step. +- **C3 Latency.** 200 interleaved pairs, ~1 MiB body. Praxis p50 minus direct p50 is at most 5 ms. Praxis p95 minus direct p95 is at most 20 ms. +- **C4 Resources.** On a long session (at least 40 inference turns and at least 2 subagents): the peak RSS of cortex plus cortex-gateway is at most 1.5 times the peak RSS of direct-mode cortex. Their CPU seconds summed are at most 2 times direct-mode cortex CPU seconds. +- **C5 Streaming.** Claude Code SSE and Codex (`chatgpt.com` `/backend-api/codex/responses`) stream frame by frame in praxis mode: the time to the first frame is within 100 ms of direct mode. +- **C6 Fail closed.** Kill `cortex-gateway`: the next agent request gets 503 with a `gateway_unavailable` row. The supervisor restarts the gateway, and requests succeed again within 10 s. + +**Condition to go back to direct mode** (after a future flip): set `praxis.enabled: false` and restart when any one of these occurs: + +- an agent request with no row in the session store; +- a `gateway_unavailable` row outside a Cortex or gateway restart; +- a measured p50 added latency over 20 ms; +- a cortex plus cortex-gateway RSS over 2 times direct mode; +- a parser parity test red on `main` for more than 7 days. + +## Review Focus + +- A scratch instance that binds a 476xx port, or a process that inherits the real `HOME`. Expect: `up` refuses to start when any 476xx port appears in the generated config, and it exports `HOME` before it starts `cortex`. +- `down` leaves `cortex-gateway` running. Expect: `down` stops the supervisor, then waits until no process from the scratch `bin/` runs (`pgrep -f /bin/`), and kills the rest after 5 s. +- A probe pair where one request fails (connection refused). Expect: the pair is dropped and counted, and the summary prints the dropped count; it does not compute a percentile on mixed data. +- The sampler that finds no gateway pid in direct mode. Expect: the gateway column is empty for direct mode, not a failure. +- Percentiles on an even count or a count of 1. Expect: nearest-rank percentile (`ceil(p/100*n)`), pinned by `selftest`. + +--- + +### Task 1: The live-check script + +**Depends on:** none. + +**Files:** +- Create: `scripts/dev/praxis-live-check.sh` (mode 0755) + +**Interfaces:** +- Produces, for Task 3 (subcommands, each prints one result per line): + - `praxis-live-check.sh build `: builds `/bin/cortex` with `go build -tags "$(go -C scripts/profile-tags run . full)" -o /bin/cortex ./cmd/cortex` from the repository root, and `cortex-gateway` with `cargo build --release --manifest-path praxis/Cargo.toml -p cortex-gateway`, copied into `/bin/`. + - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the laptop config shape: `mode: proxy-sidecar`, `listener.roles: [forward]`, `bind_loopback_only: true`, the scratch ports, `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200. Exit 1 on timeout with the last 20 log lines. + - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY=`, `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE` (the scratch CA bundle). + - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy --cacert -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of user text to ~1 MiB, `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair where either code is `000` is dropped. + - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `GET http://127.0.0.1:/debug/pprof/heap?debug=1`, the `HeapInuse` line, cortex only; `-` when absent). At the end prints, per mode and process, peak RSS and CPU seconds used. + - `praxis-live-check.sh down direct|praxis|all `: see Review Focus. + - `praxis-live-check.sh selftest`: runs the pure functions on fixed input and exits 0 only when all pass. +- Pure functions, defined before the dispatch so `selftest` can call them: `pct

` (reads numbers on stdin, prints the nearest-rank percentile), `cputime_s ` (`MM:SS.ss` or `HH:MM:SS` or `D-HH:MM:SS` to seconds), `ports_ok ` (exit 1 when the file names any `:476[0-9][0-9]`). + +- [ ] **Step 1: Write `selftest` first.** Cases: + - `printf '1\n2\n3\n4\n' | pct 50` prints `2`; `pct 95` prints `4`; `printf '7\n' | pct 50` prints `7`. + - `cputime_s 0:01.50` prints `1.50`; `cputime_s 01:02:03` prints `3723`; `cputime_s 1-00:00:01` prints `86401`. + - `ports_ok` fails on a file with `forward_proxy_addr: 127.0.0.1:47600` and passes on one with `127.0.0.1:49700`. +- [ ] **Step 2: Run it and see it fail.** Run: `scripts/dev/praxis-live-check.sh selftest`. Expected: non-zero exit (functions not defined). +- [ ] **Step 3: Write the functions and the subcommands** as the Interfaces block says. `set -euo pipefail`. `up` calls `ports_ok` on the generated file and refuses on failure. +- [ ] **Step 4: Run the gate.** Run: `bash -n scripts/dev/praxis-live-check.sh && shellcheck scripts/dev/praxis-live-check.sh && scripts/dev/praxis-live-check.sh selftest`. Expected: exit 0, `selftest: ok`. +- [ ] **Step 5: Smoke test with no credentials.** `S=$(mktemp -d)`; `build $S`; `up direct $S`; `up praxis $S`; `probe $S 5 https://api.anthropic.com`; `sample $S 15`; `down all $S`. Expected: both `up` succeed, the probe prints 5 pairs with code 401 in both modes, `sample.tsv` has `cortex-gateway` rows only for praxis, and `pgrep -f $S/bin/` prints nothing after `down`. Then `rm -rf $S`. +- [ ] **Step 6: Commit.** `git add scripts/dev/praxis-live-check.sh && git commit -s -m "feat: Add a dev script for the praxis-mode live check"` with the attribution trailer the repository CLAUDE.md names. + +### Task 2: Post the criteria before the check + +**Depends on:** none. + +**Files:** none (GitHub comments). + +- [ ] **Step 1:** Post the section "Criteria for the default" (C1-C6 and the condition to go back) as one comment on #1725, with `gh issue comment 1725 --repo rossoctl/cortex --body-file `. Start it with "Criteria for the default, written before the live check." Write the body file in the scratch directory, not in the repository. +- [ ] **Step 2:** Post a short comment on #1714 that links that comment and says it fills the two gaps under "Gates": criteria for the default, and rollback criteria. +- [ ] **Step 3:** Record both comment URLs in the Task 4 report. + +### Task 3: Run the live check + +**Depends on:** Task 1, Task 2. + +**Files:** none in the repository. All output goes to ``. + +- [ ] **Step 1: Set up.** `S=$(mktemp -d)`, `chmod 700 $S`. `build`, `up direct`, `up praxis`. In praxis `cortex.log`, find the Warn line "praxis mode: outbound plugins are off". +- [ ] **Step 2: Credentials.** Praxis mode has no `inference-router`, so the agent must call the upstream directly. Take the first server URL under `inference-router` in the real `~/.cortex/config.yaml` and its key, into `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` in the agent shell only. For Codex, copy `~/.codex/config.toml` and `~/.codex/auth.json` into each scratch HOME's `.codex/`. When a credential is missing, do the steps that do not need it, and mark the dependent criteria "not run" with the reason. +- [ ] **Step 3: Latency (C3).** `probe $S 200 "$ANTHROPIC_BASE_URL"`. Keep the printed summary. +- [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `sample $S 3600` in the background. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. +- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --api http://127.0.0.1:` (if agentop has no such flag, read the session API with `curl --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}`). Compare the two modes for C2: session count, titles, timeline rows, tool call names, subagents, tokens against the raw capture `usage` (`/v1/sessions/{id}/events/{seq}?raw=1`), cost. Record each difference. A difference that Section 5 lists as known (captured request headers, timing within 50 ms) is not a failure. +- [ ] **Step 6: Fail closed (C6).** Kill the praxis gateway pid. Send one probe request through the praxis proxy: expect 503 and a `gateway_unavailable` row. Wait for the supervisor restart; time until a probe succeeds again. +- [ ] **Step 7: Resources (C4).** Stop the sampler; keep the peak RSS and CPU seconds summary. +- [ ] **Step 8: Tear down.** `down all $S`, check `pgrep -f $S/bin/` is empty, `rm -rf $S`. + +### Task 4: Record the result and the decision + +**Depends on:** Task 3. + +**Files:** none (GitHub comments). + +- [ ] **Step 1:** Post one comment on #1725: a table with one row per criterion C1-C6 (measured value, threshold, pass / fail / not run), the probe and sample summaries, the agent and Claude Code / Codex versions, the date, and every C2 difference. +- [ ] **Step 2: Decide.** Rule: the default changes only when C1-C6 all pass. Expected result: C1 fails by design, so the decision is "direct mode stays the default; praxis mode stays opt-in (`agentop setup --praxis`)". Write the decision and the criteria that block it. +- [ ] **Step 3:** Post a short comment on #1714 with the decision and a link to the result comment. Update the tracker's phase A row when the body has one for step 8. +- [ ] **Step 4:** Do not close #1725 by hand. The PR from Task 1 says `Closes #1725` in its body, so the merge closes it after the result is recorded. + +## PRs + +| PR | Title | Parent | Tasks | estLines | +|---|---|---|---|---| +| P1 | `feat: Add a dev script for the praxis-mode live check` | none | T1, T2, T3, T4 | 260 | + +T2-T4 change no files. They run in the P1 session after T1, and the PR body links the result comment. + +## Gate + +From the worktree root: + +```sh +bash -n scripts/dev/praxis-live-check.sh && shellcheck scripts/dev/praxis-live-check.sh && scripts/dev/praxis-live-check.sh selftest +``` From a5912c1264e73fc78e595a76ed21d41c3546e081 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 16:54:38 -0500 Subject: [PATCH 02/10] docs: Make the praxis live-check plan steps exact Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../2026-11-10-praxis-mode-8-live-check.md | 30 ++++++++++++------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index e677ad651..f5504f75d 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -69,11 +69,12 @@ Praxis mode can become the default only when **every** criterion passes: **Interfaces:** - Produces, for Task 3 (subcommands, each prints one result per line): - - `praxis-live-check.sh build `: builds `/bin/cortex` with `go build -tags "$(go -C scripts/profile-tags run . full)" -o /bin/cortex ./cmd/cortex` from the repository root, and `cortex-gateway` with `cargo build --release --manifest-path praxis/Cargo.toml -p cortex-gateway`, copied into `/bin/`. - - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the laptop config shape: `mode: proxy-sidecar`, `listener.roles: [forward]`, `bind_loopback_only: true`, the scratch ports, `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200. Exit 1 on timeout with the last 20 log lines. - - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY=`, `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE` (the scratch CA bundle). - - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy --cacert -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of user text to ~1 MiB, `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair where either code is `000` is dropped. - - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `GET http://127.0.0.1:/debug/pprof/heap?debug=1`, the `HeapInuse` line, cortex only; `-` when absent). At the end prints, per mode and process, peak RSS and CPU seconds used. + - `praxis-live-check.sh build `: copy the build form of `scripts/dev/praxis-measure.sh`. `tags=$(go -C "$root/scripts/profile-tags" run . full)`, then `(cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o /bin/cortex .)`. Then `(cd "$root/praxis" && cargo build --release --locked -p cortex-gateway)` and `cp "$root/praxis/target/release/cortex-gateway" /bin/`. `$root` is `$(cd "$(dirname "$0")/../.." && pwd)`. + - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200. Exit 1 on timeout with the last 20 log lines. + - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, `http_proxy` (all `http://127.0.0.1:`), `NO_PROXY=127.0.0.1,localhost`, `NODE_EXTRA_CA_CERTS=/ca.crt`, and `SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, `GIT_SSL_CAINFO` all `=/bundle.crt`. Cortex writes `ca.crt` and `bundle.crt` (bridge CA plus platform roots) into `ca_dir` itself (see `agentop exec --help`); `env` exits 1 when either file is missing. + - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of user text to ~1 MiB, `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair where either code is `000` is dropped. + - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?gc=1&debug=1"`, the `# HeapInuse = ` line as in `praxis-measure.sh`, cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 0 after a signal. + - `praxis-live-check.sh ttff `: for C5. `n` interleaved pairs like `probe`, each a small streaming `/v1/messages` request (`max_tokens: 16`, `stream: true`) with `-H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN"`; exit 2 when that variable is empty. Time to the first line that starts with `data:` in the response, measured with `perl -MTime::HiRes=time` (macOS `date` has no milliseconds). Writes `/ttff.tsv` (`pair mode ms`) and prints `direct p50`, `praxis p50`, `delta p50`, `dropped `. A pair where either side has no `data:` line is dropped. - `praxis-live-check.sh down direct|praxis|all `: see Review Focus. - `praxis-live-check.sh selftest`: runs the pure functions on fixed input and exits 0 only when all pass. - Pure functions, defined before the dispatch so `selftest` can call them: `pct

` (reads numbers on stdin, prints the nearest-rank percentile), `cputime_s ` (`MM:SS.ss` or `HH:MM:SS` or `D-HH:MM:SS` to seconds), `ports_ok ` (exit 1 when the file names any `:476[0-9][0-9]`). @@ -107,10 +108,10 @@ Praxis mode can become the default only when **every** criterion passes: - [ ] **Step 1: Set up.** `S=$(mktemp -d)`, `chmod 700 $S`. `build`, `up direct`, `up praxis`. In praxis `cortex.log`, find the Warn line "praxis mode: outbound plugins are off". - [ ] **Step 2: Credentials.** Praxis mode has no `inference-router`, so the agent must call the upstream directly. Take the first server URL under `inference-router` in the real `~/.cortex/config.yaml` and its key, into `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` in the agent shell only. For Codex, copy `~/.codex/config.toml` and `~/.codex/auth.json` into each scratch HOME's `.codex/`. When a credential is missing, do the steps that do not need it, and mark the dependent criteria "not run" with the reason. - [ ] **Step 3: Latency (C3).** `probe $S 200 "$ANTHROPIC_BASE_URL"`. Keep the printed summary. -- [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `sample $S 3600` in the background. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. -- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --api http://127.0.0.1:` (if agentop has no such flag, read the session API with `curl --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}`). Compare the two modes for C2: session count, titles, timeline rows, tool call names, subagents, tokens against the raw capture `usage` (`/v1/sessions/{id}/events/{seq}?raw=1`), cost. Record each difference. A difference that Section 5 lists as known (captured request headers, timing within 50 ms) is not a failure. -- [ ] **Step 6: Fail closed (C6).** Kill the praxis gateway pid. Send one probe request through the praxis proxy: expect 503 and a `gateway_unavailable` row. Wait for the supervisor restart; time until a probe succeeds again. -- [ ] **Step 7: Resources (C4).** Stop the sampler; keep the peak RSS and CPU seconds summary. +- [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `praxis-live-check.sh sample $S 3600 &` and keep its pid in `SAMPLER`. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. When one Claude Code run has fewer than 40 inference turns, run the same prompt again in the same session (`claude -p --continue`) in both modes until it has 40. Then, for C5, `praxis-live-check.sh ttff $S 20 "$ANTHROPIC_BASE_URL"`. For Codex, C5 uses the median `ttfbMs` of its inference rows in each mode from the session API (Codex authenticates with an OAuth token, so `ttff` cannot call its upstream). Say in the result that `ttfbMs` measures the response headers, not the first frame. +- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --endpoint http://127.0.0.1:` for a visual check, and read the numbers with `curl -s --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}` (endpoint reference: `docs/session-api.md`). Compare the two modes for C2: session count, titles, timeline rows, tool call names, subagents, tokens against the raw capture `usage` (`/v1/sessions/{id}/events/{seq}?raw=1`), cost. Record each difference. A difference that Section 5 lists as known (captured request headers, timing within 50 ms) is not a failure. +- [ ] **Step 6: Fail closed (C6).** `kill -9 "$(pgrep -f "$S/bin/cortex-gateway")"`, and at once send one request with the `probe` curl line (praxis proxy `127.0.0.1:49710`, invalid key). Expect code 503 and, in `/v1/sessions`, a row with error kind `gateway_unavailable`. Then send the same request once each 0.5 s; record the seconds from the kill to the first code other than 503 and 000. Pass when that is at most 10 s. +- [ ] **Step 7: Resources (C4).** `kill -TERM "$SAMPLER"; wait "$SAMPLER"`. Keep `$S/sample-summary.txt`. Copy it out of `$S` (into the Task 4 comment body file in a second `mktemp -d`) before Step 8. - [ ] **Step 8: Tear down.** `down all $S`, check `pgrep -f $S/bin/` is empty, `rm -rf $S`. ### Task 4: Record the result and the decision @@ -128,7 +129,9 @@ Praxis mode can become the default only when **every** criterion passes: | PR | Title | Parent | Tasks | estLines | |---|---|---|---|---| -| P1 | `feat: Add a dev script for the praxis-mode live check` | none | T1, T2, T3, T4 | 260 | +| P1 | `feat: Add a dev script for the praxis-mode live check` | "" (none) | T1, T2, T3, T4 | 340 | + +Estimate: T1 is about 340 counted lines (the script with `selftest`); T2-T4 are 0. The plan document does not count. T2-T4 change no files. They run in the P1 session after T1, and the PR body links the result comment. @@ -139,3 +142,10 @@ From the worktree root: ```sh bash -n scripts/dev/praxis-live-check.sh && shellcheck scripts/dev/praxis-live-check.sh && scripts/dev/praxis-live-check.sh selftest ``` + +Gate check on base 412ed825 (plan review, 2026-11-10): `bash` and `shellcheck` 0.11.0 are installed and run from the worktree root. Both steps fail on the base only because Task 1 creates the file (`No such file or directory`). The gate is stricter than CI, which runs `shellcheck --severity=error` (`.github/workflows/security-scans.yaml`); keep the strict form. The existing `scripts/dev/praxis-measure.sh` passes plain `shellcheck`. + +## Scope left out + +- The default flip (D-B). +- The heap of `cortex-gateway`: it serves no pprof, so C4 uses its RSS and CPU only. The issue asks for "the CPU and heap of both processes"; the result comment must say this. From 62932ddcc84681b8836a5ce702863a58fd0b2b21 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:00:53 -0500 Subject: [PATCH 03/10] feat: Add a dev script for the praxis-mode live check The script starts two scratch Cortex instances, one in direct mode and one in praxis mode, on 497xx ports under scratch HOMEs. It does not touch the shared proxy. Subcommands: build, up, env, probe (interleaved latency pairs), sample (RSS, CPU and heap), ttff (time to the first SSE frame), down and selftest. The up command refuses a config that names a 476xx port. Part of #1725. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- scripts/dev/praxis-live-check.sh | 383 +++++++++++++++++++++++++++++++ 1 file changed, 383 insertions(+) create mode 100755 scripts/dev/praxis-live-check.sh diff --git a/scripts/dev/praxis-live-check.sh b/scripts/dev/praxis-live-check.sh new file mode 100755 index 000000000..e70a22266 --- /dev/null +++ b/scripts/dev/praxis-live-check.sh @@ -0,0 +1,383 @@ +#!/bin/bash +# praxis-live-check.sh runs two scratch Cortex instances side by side, one in direct +# mode and one in praxis mode, for the praxis-mode live check (#1725). +# +# Every process runs under //home on the 497xx ports below. The script +# never touches the shared proxy on :47600-47606 or the real ~/.cortex. +# +# direct: proxy 49700, session API 49701, stats 49702, transparent 49703, health 49704 +# praxis: proxy 49710, session API 49711, stats 49712, transparent 49713, health 49714, +# praxis.listen 49715, praxis.egress 49716 +# +# Usage: +# praxis-live-check.sh build build cortex and cortex-gateway +# praxis-live-check.sh up direct|praxis start one instance +# praxis-live-check.sh env direct|praxis print exports for an agent shell +# praxis-live-check.sh probe n interleaved latency pairs +# praxis-live-check.sh sample sample RSS, CPU and heap +# praxis-live-check.sh ttff n pairs, time to the first frame +# praxis-live-check.sh down direct|praxis|all +# praxis-live-check.sh selftest +set -euo pipefail + +root=$(cd "$(dirname "$0")/../.." && pwd) + +die() { + echo "praxis-live-check: $*" >&2 + exit 1 +} + +# pct P: reads numbers on stdin and prints the nearest-rank percentile ceil(P/100*n). +# P is an integer. +pct() { + sort -n | awk -v p="$1" '{a[NR] = $1} END { + if (NR == 0) exit 1 + r = int((p * NR + 99) / 100) + if (r < 1) r = 1 + print a[r] + }' +} + +# cputime_s T: converts a ps time (MM:SS.ss, HH:MM:SS or D-HH:MM:SS) to seconds. +cputime_s() { + echo "$1" | awk '{ + d = 0; t = $0 + if (index(t, "-") > 0) { split(t, dp, "-"); d = dp[1]; t = dp[2] } + n = split(t, f, ":") + s = d * 86400 + f[1] + if (n == 3) s = d * 86400 + f[1] * 3600 + f[2] * 60 + f[3] + if (n == 2) s = d * 86400 + f[1] * 60 + f[2] + if (index(f[n], ".") > 0) printf "%.2f\n", s; else printf "%d\n", s + }' +} + +# ports_ok FILE: fails when FILE names a port of the shared proxy (476xx). +ports_ok() { + ! grep -qE ':476[0-9][0-9]' "$1" +} + +selftest() { + fail=0 + check() { + if [ "$2" != "$3" ]; then + echo "selftest: $1: got '$2', want '$3'" >&2 + fail=1 + fi + } + check "pct 50 of 4" "$(printf '1\n2\n3\n4\n' | pct 50)" 2 + check "pct 95 of 4" "$(printf '1\n2\n3\n4\n' | pct 95)" 4 + check "pct 50 of 1" "$(printf '7\n' | pct 50)" 7 + check "pct 95 of 20" "$(seq 20 | pct 95)" 19 + check "pct 50 unsorted" "$(printf '9\n1\n5\n' | pct 50)" 5 + check "cputime 0:01.50" "$(cputime_s 0:01.50)" 1.50 + check "cputime 01:02:03" "$(cputime_s 01:02:03)" 3723 + check "cputime 1-00:00:01" "$(cputime_s 1-00:00:01)" 86401 + t=$(mktemp -d "${TMPDIR:-/tmp}/praxis-live-check-selftest.XXXXXX") + echo " forward_proxy_addr: 127.0.0.1:47600" >"$t/bad.yaml" + echo " forward_proxy_addr: 127.0.0.1:49700" >"$t/good.yaml" + if ports_ok "$t/bad.yaml"; then echo "selftest: ports_ok passed :47600" >&2; fail=1; fi + if ! ports_ok "$t/good.yaml"; then echo "selftest: ports_ok refused :49700" >&2; fail=1; fi + rm -rf "$t" + [ "$fail" = 0 ] || exit 1 + echo "selftest: ok" +} + +# abs DIR: the physical path, so pgrep patterns match what os.Executable gives. +abs() { + [ -d "$1" ] || die "no directory $1" + (cd "$1" && pwd -P) +} + +# setports MODE: sets the port variables of one instance. +setports() { + case "$1" in + direct) base=49700 ;; + praxis) base=49710 ;; + *) die "mode must be direct or praxis, not '$1'" ;; + esac + proxy=$base + api=$((base + 1)) + stats=$((base + 2)) + trans=$((base + 3)) + health=$((base + 4)) +} + +build() { + s=$(abs "$1") + mkdir -p "$s/bin" + tags=$(go -C "$root/scripts/profile-tags" run . full) + echo "building cortex (profile full)" >&2 + (cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o "$s/bin/cortex" .) + echo "building cortex-gateway (release)" >&2 + (cd "$root/praxis" && cargo build --release --locked -p cortex-gateway >&2) + cp "$root/praxis/target/release/cortex-gateway" "$s/bin/" + echo "built $s/bin/cortex $s/bin/cortex-gateway" +} + +up() { + mode=$1 + setports "$mode" + s=$(abs "$2") + [ -x "$s/bin/cortex" ] || die "no $s/bin/cortex; run build first" + d="$s/$mode" + home="$d/home" + cfg="$home/.cortex/config.yaml" + if [ -f "$d/supervisor.pid" ] && kill -0 "$(cat "$d/supervisor.pid")" 2>/dev/null; then + die "$mode is already up (pid $(cat "$d/supervisor.pid"))" + fi + mkdir -p "$home/.cortex" + praxis_block="" + if [ "$mode" = praxis ]; then + praxis_block="praxis: + enabled: true + listen: 127.0.0.1:$((base + 5)) + egress: 127.0.0.1:$((base + 6))" + fi + cat >"$cfg" <"$d/cortex.log" 2>&1 & + echo $! >"$d/supervisor.pid" + for _ in $(seq 300); do + if [ "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$health/healthz" || true)" = 200 ]; then + echo "$mode up: pid $(cat "$d/supervisor.pid"), proxy 127.0.0.1:$proxy, session API 127.0.0.1:$api" + return 0 + fi + sleep 0.1 + done + tail -20 "$d/cortex.log" >&2 + die "$mode: http://127.0.0.1:$health/healthz did not answer 200 in 30 s" +} + +env_cmd() { + setports "$1" + s=$(abs "$2") + home="$s/$1/home" + ca="$home/.cortex/ca" + [ -f "$ca/ca.crt" ] && [ -f "$ca/bundle.crt" ] || die "no $ca/ca.crt or $ca/bundle.crt; is $1 up?" + p="http://127.0.0.1:$proxy" + echo "export HOME='$home'" + for v in HTTPS_PROXY HTTP_PROXY https_proxy http_proxy; do echo "export $v='$p'"; done + echo "export NO_PROXY='127.0.0.1,localhost'" + echo "export NODE_EXTRA_CA_CERTS='$ca/ca.crt'" + for v in SSL_CERT_FILE CURL_CA_BUNDLE REQUESTS_CA_BUNDLE GIT_SSL_CAINFO; do echo "export $v='$ca/bundle.crt'"; done +} + +# order I: the two modes of pair I, in the order this pair sends them. +order() { + if [ $(($1 % 2)) = 1 ]; then echo "direct praxis"; else echo "praxis direct"; fi +} + +# summarize FILE COLUMN NAMES: prints " []" per mode and the delta. +summarize() { + f=$1 col=$2 + shift 2 + for m in direct praxis; do + line="$m" + for p in "$@"; do line="$line $(awk -v m="$m" -v c="$col" '$2 == m {print $c}' "$f" | pct "$p")"; done + echo "$line" + done + line="delta" + for p in "$@"; do + dv=$(awk -v c="$col" '$2 == "direct" {print $c}' "$f" | pct "$p") + pv=$(awk -v c="$col" '$2 == "praxis" {print $c}' "$f" | pct "$p") + line="$line $(awk -v a="$pv" -v b="$dv" 'BEGIN {printf "%.1f", a - b}')" + done + echo "$line" +} + +probe() { + s=$(abs "$1") + n=$2 + url=$3 + body="$s/body.json" + if [ ! -f "$body" ]; then + { + printf '{"model":"claude-opus-4-6","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"' + head -c 1048576 /dev/zero | tr '\0' x + printf '"}]}' + } >"$body" + fi + out="$s/probe.tsv" + : >"$out" + dropped=0 + for i in $(seq "$n"); do + pair="" + bad=0 + for m in $(order "$i"); do + setports "$m" + r=$(curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy "http://127.0.0.1:$proxy" \ + --cacert "$s/$m/home/.cortex/ca/bundle.crt" -H 'x-api-key: invalid' \ + -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \ + --data-binary @"$body" "$url/v1/messages" || true) + code=${r%% *} + [ "$code" = 000 ] && bad=1 + pair="$pair$(echo "$r" | awk -v i="$i" -v m="$m" '{printf "%s\t%s\t%s\t%.1f\n", i, m, $1, $2 * 1000}') +" + done + if [ "$bad" = 1 ]; then dropped=$((dropped + 1)); else printf '%s' "$pair" >>"$out"; fi + done + [ -s "$out" ] || die "probe: every pair was dropped ($dropped)" + summarize "$out" 4 50 95 + echo "dropped $dropped" +} + +# first_frame_ms CMD...: runs CMD and prints the ms to its first output line that +# starts with "data:". Exit 1 when no such line comes. +first_frame_ms() { + perl -MTime::HiRes=time -e ' + my $t = time; + my $pid = open(my $f, "-|", @ARGV) or exit 1; + while (<$f>) { + if (/^data:/) { printf "%.1f\n", (time - $t) * 1000; kill "TERM", $pid; exit 0 } + } + exit 1' "$@" +} + +ttff() { + [ -n "${ANTHROPIC_AUTH_TOKEN:-}" ] || { + echo "praxis-live-check: ttff needs ANTHROPIC_AUTH_TOKEN" >&2 + exit 2 + } + s=$(abs "$1") + n=$2 + url=$3 + small='{"model":"claude-opus-4-6","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"Say hi."}]}' + out="$s/ttff.tsv" + : >"$out" + dropped=0 + for i in $(seq "$n"); do + pair="" + bad=0 + for m in $(order "$i"); do + setports "$m" + ms=$(first_frame_ms curl -sN --proxy "http://127.0.0.1:$proxy" \ + --cacert "$s/$m/home/.cortex/ca/bundle.crt" \ + -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" -H 'anthropic-version: 2023-06-01' \ + -H 'content-type: application/json' --data "$small" "$url/v1/messages" || true) + [ -n "$ms" ] || bad=1 + pair="$pair$i $m $ms +" + done + if [ "$bad" = 1 ]; then dropped=$((dropped + 1)); else printf '%s' "$pair" >>"$out"; fi + done + [ -s "$out" ] || die "ttff: every pair was dropped ($dropped)" + summarize "$out" 3 50 + echo "dropped $dropped" +} + +# sample_summary: per mode and process, peak RSS, peak heap and CPU seconds used. +sample_summary() { + awk '{ + k = $2 " " $3 + if (!(k in first)) { first[k] = $6; keys[++nk] = k } + last[k] = $6 + if ($5 > rss[k]) rss[k] = $5 + if ($7 != "-" && $7 > heap[k]) heap[k] = $7 + } END { + for (i = 1; i <= nk; i++) { + k = keys[i] + printf "%s peak_rss_kb %d peak_heap_inuse_bytes %s cpu_s %.2f\n", k, rss[k], (k in heap) ? heap[k] : "-", last[k] - first[k] + } + }' "$s/sample.tsv" | tee "$s/sample-summary.txt" +} + +# sample_one MODE PROC PID: appends one sample line. +sample_one() { + ps_out=$(ps -o rss=,time= -p "$3" 2>/dev/null || true) + [ -n "$ps_out" ] || return 0 + # shellcheck disable=SC2086 # split "rss time" into two fields + set -- "$1" "$2" "$3" $ps_out + heap=- + if [ "$2" = cortex ]; then + setports "$1" + heap=$(curl -s --noproxy '*' "http://127.0.0.1:$stats/debug/pprof/heap?gc=1&debug=1" | + awk '/^# HeapInuse = / {print $4}' || true) + [ -n "$heap" ] || heap=- + fi + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$(date +%s)" "$1" "$2" "$3" "$4" "$(cputime_s "$5")" "$heap" >>"$s/sample.tsv" +} + +sample() { + s=$(abs "$1") + secs=$2 + trap 'sample_summary; exit 0' TERM INT + start=$(date +%s) + while :; do + for m in direct praxis; do + pid=$(pgrep -f "$s/bin/cortex --config $s/$m/" | head -1 || true) + [ -n "$pid" ] && sample_one "$m" cortex "$pid" + if [ "$m" = praxis ]; then + gpid=$(pgrep -f "$s/bin/cortex-gateway -c $s/$m/" | head -1 || true) + [ -n "$gpid" ] && sample_one "$m" cortex-gateway "$gpid" + fi + done + [ $(($(date +%s) - start)) -lt "$secs" ] || break + sleep 5 & + wait $! || true + done + sample_summary +} + +down() { + s=$(abs "$2") + case "$1" in + direct | praxis) modes=$1 pattern="$s/bin/cortex.* .*$s/$1/" ;; + all) modes="direct praxis" pattern="$s/bin/" ;; + *) die "down: mode must be direct, praxis or all" ;; + esac + for m in $modes; do + if [ -f "$s/$m/supervisor.pid" ]; then + kill "$(cat "$s/$m/supervisor.pid")" 2>/dev/null || true + rm -f "$s/$m/supervisor.pid" + fi + done + for _ in $(seq 50); do + pgrep -f "$pattern" >/dev/null || { echo "down: $1 stopped"; return 0; } + sleep 0.1 + done + echo "down: killing what is left after 5 s:" >&2 + pgrep -fl "$pattern" >&2 || true + pkill -9 -f "$pattern" || true + sleep 0.5 + ! pgrep -f "$pattern" >/dev/null || die "down: processes still run under $s/bin/" + echo "down: $1 stopped" +} + +usage() { + sed -n '12,20p' "$0" | sed 's/^# //' >&2 + exit 2 +} + +case "${1:-}" in +build) [ $# = 2 ] || usage; build "$2" ;; +up) [ $# = 3 ] || usage; up "$2" "$3" ;; +env) [ $# = 3 ] || usage; env_cmd "$2" "$3" ;; +probe) [ $# = 4 ] || usage; probe "$2" "$3" "$4" ;; +sample) [ $# = 3 ] || usage; sample "$2" "$3" ;; +ttff) [ $# = 4 ] || usage; ttff "$2" "$3" "$4" ;; +down) [ $# = 3 ] || usage; down "$2" "$3" ;; +selftest) selftest ;; +*) usage ;; +esac From 47885cb9bb035026eb8319cd8db6b9eb50dd5c4c Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:03:10 -0500 Subject: [PATCH 04/10] fix: Add CPU seconds for each pid in the live-check sample summary A killed cortex-gateway restarts with a new pid, and its ps time starts again at 0. The summary now adds the CPU delta of each pid. The sample command also clears sample.tsv at the start, so rows from an earlier run do not go into the summary. A selftest case has two pids under one key. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- scripts/dev/praxis-live-check.sh | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/scripts/dev/praxis-live-check.sh b/scripts/dev/praxis-live-check.sh index e70a22266..4b520f09c 100755 --- a/scripts/dev/praxis-live-check.sh +++ b/scripts/dev/praxis-live-check.sh @@ -77,6 +77,9 @@ selftest() { echo " forward_proxy_addr: 127.0.0.1:49700" >"$t/good.yaml" if ports_ok "$t/bad.yaml"; then echo "selftest: ports_ok passed :47600" >&2; fail=1; fi if ! ports_ok "$t/good.yaml"; then echo "selftest: ports_ok refused :49700" >&2; fail=1; fi + # Two pids under one key (a restart): CPU is (3-1) + (2-0.5), not 2-1. + printf '1\tpraxis\tcortex-gateway\t10\t100\t1\t-\n1\tpraxis\tcortex-gateway\t10\t300\t3\t-\n2\tpraxis\tcortex-gateway\t11\t200\t0.5\t-\n2\tpraxis\tcortex-gateway\t11\t200\t2\t-\n' >"$t/sample.tsv" + check "sample_summary restart" "$(s=$t sample_summary)" "praxis cortex-gateway peak_rss_kb 300 peak_heap_inuse_bytes - cpu_s 3.50" rm -rf "$t" [ "$fail" = 0 ] || exit 1 echo "selftest: ok" @@ -288,17 +291,21 @@ ttff() { } # sample_summary: per mode and process, peak RSS, peak heap and CPU seconds used. +# CPU is summed per pid: a restarted process starts its ps time again at 0. sample_summary() { awk '{ k = $2 " " $3 - if (!(k in first)) { first[k] = $6; keys[++nk] = k } - last[k] = $6 + p = k " " $4 + if (!(k in rss)) { rss[k] = 0; keys[++nk] = k } + if (!(p in first)) { first[p] = $6; pk[p] = k } + last[p] = $6 if ($5 > rss[k]) rss[k] = $5 if ($7 != "-" && $7 > heap[k]) heap[k] = $7 } END { + for (p in first) cpu[pk[p]] += last[p] - first[p] for (i = 1; i <= nk; i++) { k = keys[i] - printf "%s peak_rss_kb %d peak_heap_inuse_bytes %s cpu_s %.2f\n", k, rss[k], (k in heap) ? heap[k] : "-", last[k] - first[k] + printf "%s peak_rss_kb %d peak_heap_inuse_bytes %s cpu_s %.2f\n", k, rss[k], (k in heap) ? heap[k] : "-", cpu[k] } }' "$s/sample.tsv" | tee "$s/sample-summary.txt" } @@ -323,6 +330,7 @@ sample() { s=$(abs "$1") secs=$2 trap 'sample_summary; exit 0' TERM INT + : >"$s/sample.tsv" # one run per summary; an earlier run's rows would skew it start=$(date +%s) while :; do for m in direct praxis; do From 3349cae85fe596124050b4ef84259a9672bf6df3 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:10:48 -0500 Subject: [PATCH 05/10] docs: Add criterion C7, one network proxy, to the praxis default Praxis mode does not become the default until the conversion is complete and cortex-gateway is the only network proxy. Until then, praxis mode stays behind the --praxis flag. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../plans/2026-11-10-praxis-mode-8-live-check.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index f5504f75d..8ffde613d 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -41,6 +41,7 @@ Praxis mode can become the default only when **every** criterion passes: - **C4 Resources.** On a long session (at least 40 inference turns and at least 2 subagents): the peak RSS of cortex plus cortex-gateway is at most 1.5 times the peak RSS of direct-mode cortex. Their CPU seconds summed are at most 2 times direct-mode cortex CPU seconds. - **C5 Streaming.** Claude Code SSE and Codex (`chatgpt.com` `/backend-api/codex/responses`) stream frame by frame in praxis mode: the time to the first frame is within 100 ms of direct mode. - **C6 Fail closed.** Kill `cortex-gateway`: the next agent request gets 503 with a `gateway_unavailable` row. The supervisor restarts the gateway, and requests succeed again within 10 s. +- **C7 One network proxy.** The conversion is complete: `cortex-gateway` is the only network proxy on the laptop, and no traffic goes through the Go forward proxy. Until C7 passes, praxis mode stays behind the `--praxis` flag (`agentop setup --praxis`), even when C1-C6 pass. Status today: fails (phases B-D of #1714 are open). Decision by Hai. **Condition to go back to direct mode** (after a future flip): set `praxis.enabled: false` and restart when any one of these occurs: @@ -95,7 +96,7 @@ Praxis mode can become the default only when **every** criterion passes: **Files:** none (GitHub comments). -- [ ] **Step 1:** Post the section "Criteria for the default" (C1-C6 and the condition to go back) as one comment on #1725, with `gh issue comment 1725 --repo rossoctl/cortex --body-file `. Start it with "Criteria for the default, written before the live check." Write the body file in the scratch directory, not in the repository. +- [ ] **Step 1:** Post the section "Criteria for the default" (C1-C7 and the condition to go back) as one comment on #1725, with `gh issue comment 1725 --repo rossoctl/cortex --body-file `. Start it with "Criteria for the default, written before the live check." Write the body file in the scratch directory, not in the repository. - [ ] **Step 2:** Post a short comment on #1714 that links that comment and says it fills the two gaps under "Gates": criteria for the default, and rollback criteria. - [ ] **Step 3:** Record both comment URLs in the Task 4 report. @@ -120,8 +121,8 @@ Praxis mode can become the default only when **every** criterion passes: **Files:** none (GitHub comments). -- [ ] **Step 1:** Post one comment on #1725: a table with one row per criterion C1-C6 (measured value, threshold, pass / fail / not run), the probe and sample summaries, the agent and Claude Code / Codex versions, the date, and every C2 difference. -- [ ] **Step 2: Decide.** Rule: the default changes only when C1-C6 all pass. Expected result: C1 fails by design, so the decision is "direct mode stays the default; praxis mode stays opt-in (`agentop setup --praxis`)". Write the decision and the criteria that block it. +- [ ] **Step 1:** Post one comment on #1725: a table with one row per criterion C1-C7 (measured value, threshold, pass / fail / not run), the probe and sample summaries, the agent and Claude Code / Codex versions, the date, and every C2 difference. +- [ ] **Step 2: Decide.** Rule: the default changes only when C1-C7 all pass. Expected result: C1 fails by design, so the decision is "direct mode stays the default; praxis mode stays opt-in (`agentop setup --praxis`)". Write the decision and the criteria that block it. - [ ] **Step 3:** Post a short comment on #1714 with the decision and a link to the result comment. Update the tracker's phase A row when the body has one for step 8. - [ ] **Step 4:** Do not close #1725 by hand. The PR from Task 1 says `Closes #1725` in its body, so the merge closes it after the result is recorded. From d926cc60bbd5606ae0c4f88946930dd769d87fef Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:13:46 -0500 Subject: [PATCH 06/10] fix: Make the live-check numbers match what they claim to measure - sample: Read HeapInuse without gc=1. A forced GC every 5 s added CPU to both cortex rows, and this pushed the C4 CPU ratio toward a pass. - probe: Write the 1 MiB body as 513 turns of 2 KiB, not one string. The parsers work per message, so one string measured only the byte copy. - ttff: Give the bearer token to curl through -K on a pipe, so the token is not in the curl argv that ps shows. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../2026-11-10-praxis-mode-8-live-check.md | 6 +++--- scripts/dev/praxis-live-check.sh | 18 +++++++++++------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index 8ffde613d..103a29bb1 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -73,9 +73,9 @@ Praxis mode can become the default only when **every** criterion passes: - `praxis-live-check.sh build `: copy the build form of `scripts/dev/praxis-measure.sh`. `tags=$(go -C "$root/scripts/profile-tags" run . full)`, then `(cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o /bin/cortex .)`. Then `(cd "$root/praxis" && cargo build --release --locked -p cortex-gateway)` and `cp "$root/praxis/target/release/cortex-gateway" /bin/`. `$root` is `$(cd "$(dirname "$0")/../.." && pwd)`. - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200. Exit 1 on timeout with the last 20 log lines. - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, `http_proxy` (all `http://127.0.0.1:`), `NO_PROXY=127.0.0.1,localhost`, `NODE_EXTRA_CA_CERTS=/ca.crt`, and `SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, `GIT_SSL_CAINFO` all `=/bundle.crt`. Cortex writes `ca.crt` and `bundle.crt` (bridge CA plus platform roots) into `ca_dir` itself (see `agentop exec --help`); `env` exits 1 when either file is missing. - - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of user text to ~1 MiB, `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair where either code is `000` is dropped. - - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?gc=1&debug=1"`, the `# HeapInuse = ` line as in `praxis-measure.sh`, cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 0 after a signal. - - `praxis-live-check.sh ttff `: for C5. `n` interleaved pairs like `probe`, each a small streaming `/v1/messages` request (`max_tokens: 16`, `stream: true`) with `-H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN"`; exit 2 when that variable is empty. Time to the first line that starts with `data:` in the response, measured with `perl -MTime::HiRes=time` (macOS `date` has no milliseconds). Writes `/ttff.tsv` (`pair mode ms`) and prints `direct p50`, `praxis p50`, `delta p50`, `dropped `. A pair where either side has no `data:` line is dropped. + - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of 513 turns (user and assistant in turn) of 2 KiB text each, about 1 MiB, so the parsers work per message; `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair where either code is `000` is dropped. + - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?debug=1"`, the `# HeapInuse = ` line (no `gc=1`: a forced GC every 5 s adds CPU to both cortex rows and skews C4), cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 0 after a signal. + - `praxis-live-check.sh ttff `: for C5. `n` interleaved pairs like `probe`, each a small streaming `/v1/messages` request (`max_tokens: 16`, `stream: true`) with the `Authorization: Bearer $ANTHROPIC_AUTH_TOKEN` header given to curl through `-K <(…)`, so the token is not in the curl argv; exit 2 when that variable is empty. Time to the first line that starts with `data:` in the response, measured with `perl -MTime::HiRes=time` (macOS `date` has no milliseconds). Writes `/ttff.tsv` (`pair mode ms`) and prints `direct p50`, `praxis p50`, `delta p50`, `dropped `. A pair where either side has no `data:` line is dropped. - `praxis-live-check.sh down direct|praxis|all `: see Review Focus. - `praxis-live-check.sh selftest`: runs the pure functions on fixed input and exits 0 only when all pass. - Pure functions, defined before the dispatch so `selftest` can call them: `pct

` (reads numbers on stdin, prints the nearest-rank percentile), `cputime_s ` (`MM:SS.ss` or `HH:MM:SS` or `D-HH:MM:SS` to seconds), `ports_ok ` (exit 1 when the file names any `:476[0-9][0-9]`). diff --git a/scripts/dev/praxis-live-check.sh b/scripts/dev/praxis-live-check.sh index 4b520f09c..8efeba88f 100755 --- a/scripts/dev/praxis-live-check.sh +++ b/scripts/dev/praxis-live-check.sh @@ -216,11 +216,13 @@ probe() { url=$3 body="$s/body.json" if [ ! -f "$body" ]; then - { - printf '{"model":"claude-opus-4-6","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"' - head -c 1048576 /dev/zero | tr '\0' x - printf '"}]}' - } >"$body" + # 513 turns of 2 KiB, not one 1 MiB string: the parsers work per message. + awk 'BEGIN { + x = sprintf("%2048s", ""); gsub(/ /, "x", x) + printf "{\"model\":\"claude-opus-4-6\",\"max_tokens\":16,\"stream\":true,\"messages\":[" + for (i = 1; i <= 513; i++) printf "%s{\"role\":\"%s\",\"content\":\"%s\"}", (i > 1 ? "," : ""), (i % 2 ? "user" : "assistant"), x + printf "]}" + }' >"$body" fi out="$s/probe.tsv" : >"$out" @@ -277,7 +279,8 @@ ttff() { setports "$m" ms=$(first_frame_ms curl -sN --proxy "http://127.0.0.1:$proxy" \ --cacert "$s/$m/home/.cortex/ca/bundle.crt" \ - -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" -H 'anthropic-version: 2023-06-01' \ + -K <(printf 'header = "Authorization: Bearer %s"\n' "$ANTHROPIC_AUTH_TOKEN") \ + -H 'anthropic-version: 2023-06-01' \ -H 'content-type: application/json' --data "$small" "$url/v1/messages" || true) [ -n "$ms" ] || bad=1 pair="$pair$i $m $ms @@ -319,7 +322,8 @@ sample_one() { heap=- if [ "$2" = cortex ]; then setports "$1" - heap=$(curl -s --noproxy '*' "http://127.0.0.1:$stats/debug/pprof/heap?gc=1&debug=1" | + # No gc=1: a forced GC every 5 s adds CPU to both cortex rows and skews C4. + heap=$(curl -s --noproxy '*' "http://127.0.0.1:$stats/debug/pprof/heap?debug=1" | awk '/^# HeapInuse = / {print $4}' || true) [ -n "$heap" ] || heap=- fi From 662a15376af700646ea029e956bb444f901c3cd1 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:25:35 -0500 Subject: [PATCH 07/10] fix: Keep a probe pair only when both modes got the same code Fixes review: - probe counted a fast local praxis 503 (hop not ready, gateway down) as latency. - up praxis returned on /healthz 200, before the praxis hop was ready. Files: - scripts/dev/praxis-live-check.sh - docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../2026-11-10-praxis-mode-8-live-check.md | 4 +- scripts/dev/praxis-live-check.sh | 37 ++++++++++++++++--- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index 103a29bb1..92e99337d 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -71,9 +71,9 @@ Praxis mode can become the default only when **every** criterion passes: **Interfaces:** - Produces, for Task 3 (subcommands, each prints one result per line): - `praxis-live-check.sh build `: copy the build form of `scripts/dev/praxis-measure.sh`. `tags=$(go -C "$root/scripts/profile-tags" run . full)`, then `(cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o /bin/cortex .)`. Then `(cd "$root/praxis" && cargo build --release --locked -p cortex-gateway)` and `cp "$root/praxis/target/release/cortex-gateway" /bin/`. `$root` is `$(cd "$(dirname "$0")/../.." && pwd)`. - - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200. Exit 1 on timeout with the last 20 log lines. + - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200 and, for `praxis`, for the log line `praxis hop: cortex-gateway is ready` (`/healthz` does not wait for the hop). Exit 1 on timeout with the last 20 log lines. - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, `http_proxy` (all `http://127.0.0.1:`), `NO_PROXY=127.0.0.1,localhost`, `NODE_EXTRA_CA_CERTS=/ca.crt`, and `SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, `GIT_SSL_CAINFO` all `=/bundle.crt`. Cortex writes `ca.crt` and `bundle.crt` (bridge CA plus platform roots) into `ca_dir` itself (see `agentop exec --help`); `env` exits 1 when either file is missing. - - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of 513 turns (user and assistant in turn) of 2 KiB text each, about 1 MiB, so the parsers work per message; `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair where either code is `000` is dropped. + - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of 513 turns (user and assistant in turn) of 2 KiB text each, about 1 MiB, so the parsers work per message; `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair is dropped when its two codes differ or a code is `000`, so a local praxis 503 is not counted as latency. - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?debug=1"`, the `# HeapInuse = ` line (no `gc=1`: a forced GC every 5 s adds CPU to both cortex rows and skews C4), cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 0 after a signal. - `praxis-live-check.sh ttff `: for C5. `n` interleaved pairs like `probe`, each a small streaming `/v1/messages` request (`max_tokens: 16`, `stream: true`) with the `Authorization: Bearer $ANTHROPIC_AUTH_TOKEN` header given to curl through `-K <(…)`, so the token is not in the curl argv; exit 2 when that variable is empty. Time to the first line that starts with `data:` in the response, measured with `perl -MTime::HiRes=time` (macOS `date` has no milliseconds). Writes `/ttff.tsv` (`pair mode ms`) and prints `direct p50`, `praxis p50`, `delta p50`, `dropped `. A pair where either side has no `data:` line is dropped. - `praxis-live-check.sh down direct|praxis|all `: see Review Focus. diff --git a/scripts/dev/praxis-live-check.sh b/scripts/dev/praxis-live-check.sh index 8efeba88f..068f825e3 100755 --- a/scripts/dev/praxis-live-check.sh +++ b/scripts/dev/praxis-live-check.sh @@ -56,6 +56,19 @@ ports_ok() { ! grep -qE ':476[0-9][0-9]' "$1" } +# pair_ok CODE CODE: succeeds when a probe pair is kept: both modes got the same code, +# and it is not 000. A praxis 503 from the hop is not the answer direct got. +pair_ok() { + [ "$1" = "$2" ] && [ "$1" != 000 ] +} + +# is_up MODE HEALTH_CODE LOG: succeeds when an instance is ready for traffic. In praxis +# mode /healthz does not wait for the hop, so also wait for the hop's ready line. +is_up() { + [ "$2" = 200 ] || return 1 + [ "$1" != praxis ] || grep -q 'praxis hop: cortex-gateway is ready' "$3" +} + selftest() { fail=0 check() { @@ -80,6 +93,18 @@ selftest() { # Two pids under one key (a restart): CPU is (3-1) + (2-0.5), not 2-1. printf '1\tpraxis\tcortex-gateway\t10\t100\t1\t-\n1\tpraxis\tcortex-gateway\t10\t300\t3\t-\n2\tpraxis\tcortex-gateway\t11\t200\t0.5\t-\n2\tpraxis\tcortex-gateway\t11\t200\t2\t-\n' >"$t/sample.tsv" check "sample_summary restart" "$(s=$t sample_summary)" "praxis cortex-gateway peak_rss_kb 300 peak_heap_inuse_bytes - cpu_s 3.50" + if ! pair_ok 401 401; then echo "selftest: pair_ok refused 401 401" >&2; fail=1; fi + if pair_ok 401 000; then echo "selftest: pair_ok kept 401 000" >&2; fail=1; fi + # A local 503 from the praxis hop is not the upstream answer direct got. + if pair_ok 401 503; then echo "selftest: pair_ok kept 401 503" >&2; fail=1; fi + if pair_ok 503 401; then echo "selftest: pair_ok kept 503 401" >&2; fail=1; fi + : >"$t/cortex.log" + if ! is_up direct 200 "$t/cortex.log"; then echo "selftest: is_up refused direct 200" >&2; fail=1; fi + if is_up direct 000 "$t/cortex.log"; then echo "selftest: is_up passed direct 000" >&2; fail=1; fi + # /healthz does not wait for the praxis hop; the hop logs once when its probe passes. + if is_up praxis 200 "$t/cortex.log"; then echo "selftest: is_up passed praxis before the hop is ready" >&2; fail=1; fi + echo 'level=INFO msg="praxis hop: cortex-gateway is ready" gateway=127.0.0.1:49715' >"$t/cortex.log" + if ! is_up praxis 200 "$t/cortex.log"; then echo "selftest: is_up refused praxis with the hop ready" >&2; fail=1; fi rm -rf "$t" [ "$fail" = 0 ] || exit 1 echo "selftest: ok" @@ -163,14 +188,14 @@ EOF HOME="$home" "$s/bin/cortex" -supervise --config "$cfg" "$d/cortex.log" 2>&1 & echo $! >"$d/supervisor.pid" for _ in $(seq 300); do - if [ "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$health/healthz" || true)" = 200 ]; then + if is_up "$mode" "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$health/healthz" || true)" "$d/cortex.log"; then echo "$mode up: pid $(cat "$d/supervisor.pid"), proxy 127.0.0.1:$proxy, session API 127.0.0.1:$api" return 0 fi sleep 0.1 done tail -20 "$d/cortex.log" >&2 - die "$mode: http://127.0.0.1:$health/healthz did not answer 200 in 30 s" + die "$mode: not ready in 30 s (/healthz 200, and in praxis mode the hop's ready line)" } env_cmd() { @@ -229,19 +254,19 @@ probe() { dropped=0 for i in $(seq "$n"); do pair="" - bad=0 + codes="" for m in $(order "$i"); do setports "$m" r=$(curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy "http://127.0.0.1:$proxy" \ --cacert "$s/$m/home/.cortex/ca/bundle.crt" -H 'x-api-key: invalid' \ -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \ --data-binary @"$body" "$url/v1/messages" || true) - code=${r%% *} - [ "$code" = 000 ] && bad=1 + codes="$codes ${r%% *}" pair="$pair$(echo "$r" | awk -v i="$i" -v m="$m" '{printf "%s\t%s\t%s\t%.1f\n", i, m, $1, $2 * 1000}') " done - if [ "$bad" = 1 ]; then dropped=$((dropped + 1)); else printf '%s' "$pair" >>"$out"; fi + # shellcheck disable=SC2086 # two codes, one argument each + if pair_ok $codes; then printf '%s' "$pair" >>"$out"; else dropped=$((dropped + 1)); fi done [ -s "$out" ] || die "probe: every pair was dropped ($dropped)" summarize "$out" 4 50 95 From 4c2fcba6c94daa0cf9766d8e1fa4fd68eaed9d1c Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:42:34 -0500 Subject: [PATCH 08/10] docs: Make the live-check steps measure what C2-C7 state Fixes review: - C7 and the spec made a cycle: phase D waited for the default, the default waited for phase C. - C6 counted any code other than 503/000 as recovery. - C2 compared two different LLM runs field by field. - The sampler stopped after 1 hour and was then signalled. - The sampler window included the praxis-only gateway restart of C6. - C5 counted Codex header time as first-frame time. Files: - docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md - docs/superpowers/specs/2026-11-07-praxis-mode-design.md Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../plans/2026-11-10-praxis-mode-8-live-check.md | 8 ++++---- docs/superpowers/specs/2026-11-07-praxis-mode-design.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index 92e99337d..00335bed4 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -109,10 +109,10 @@ Praxis mode can become the default only when **every** criterion passes: - [ ] **Step 1: Set up.** `S=$(mktemp -d)`, `chmod 700 $S`. `build`, `up direct`, `up praxis`. In praxis `cortex.log`, find the Warn line "praxis mode: outbound plugins are off". - [ ] **Step 2: Credentials.** Praxis mode has no `inference-router`, so the agent must call the upstream directly. Take the first server URL under `inference-router` in the real `~/.cortex/config.yaml` and its key, into `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` in the agent shell only. For Codex, copy `~/.codex/config.toml` and `~/.codex/auth.json` into each scratch HOME's `.codex/`. When a credential is missing, do the steps that do not need it, and mark the dependent criteria "not run" with the reason. - [ ] **Step 3: Latency (C3).** `probe $S 200 "$ANTHROPIC_BASE_URL"`. Keep the printed summary. -- [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `praxis-live-check.sh sample $S 3600 &` and keep its pid in `SAMPLER`. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. When one Claude Code run has fewer than 40 inference turns, run the same prompt again in the same session (`claude -p --continue`) in both modes until it has 40. Then, for C5, `praxis-live-check.sh ttff $S 20 "$ANTHROPIC_BASE_URL"`. For Codex, C5 uses the median `ttfbMs` of its inference rows in each mode from the session API (Codex authenticates with an OAuth token, so `ttff` cannot call its upstream). Say in the result that `ttfbMs` measures the response headers, not the first frame. -- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --endpoint http://127.0.0.1:` for a visual check, and read the numbers with `curl -s --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}` (endpoint reference: `docs/session-api.md`). Compare the two modes for C2: session count, titles, timeline rows, tool call names, subagents, tokens against the raw capture `usage` (`/v1/sessions/{id}/events/{seq}?raw=1`), cost. Record each difference. A difference that Section 5 lists as known (captured request headers, timing within 50 ms) is not a failure. -- [ ] **Step 6: Fail closed (C6).** `kill -9 "$(pgrep -f "$S/bin/cortex-gateway")"`, and at once send one request with the `probe` curl line (praxis proxy `127.0.0.1:49710`, invalid key). Expect code 503 and, in `/v1/sessions`, a row with error kind `gateway_unavailable`. Then send the same request once each 0.5 s; record the seconds from the kill to the first code other than 503 and 000. Pass when that is at most 10 s. -- [ ] **Step 7: Resources (C4).** `kill -TERM "$SAMPLER"; wait "$SAMPLER"`. Keep `$S/sample-summary.txt`. Copy it out of `$S` (into the Task 4 comment body file in a second `mktemp -d`) before Step 8. +- [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `praxis-live-check.sh sample $S 86400 &` and keep its pid in `SAMPLER`. Step 6 stops it. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. When one Claude Code run has fewer than 40 inference turns, run the same prompt again in the same session (`claude -p --continue`) in both modes until it has 40. Then, for C5, `praxis-live-check.sh ttff $S 20 "$ANTHROPIC_BASE_URL"`. For Codex, `ttff` cannot call its upstream (Codex authenticates with an OAuth token), and `ttfbMs` measures the response headers, not the first frame. So record the Codex part of C5 as not run. Give the median `ttfbMs` of the Codex inference rows in each mode from the session API as information only. +- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --endpoint http://127.0.0.1:` for a visual check, and read the numbers with `curl -s --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}` (endpoint reference: `docs/session-api.md`). For C2, check each mode against its own raw captures (`/v1/sessions/{id}/events/{seq}?raw=1`), because the two modes run different LLM runs: the title, the timeline rows, the tool call names and the subagent blocks agree with the captured requests and responses, the tokens are equal to the captured `usage`, and the cost is direct mode's price for those tokens. Between the two modes, compare only the shape: one session for each agent run, subagent blocks present, and zero rows with a `gateway_*` error kind. Record each difference. A difference that Section 5 lists as known (captured request headers, timing within 50 ms) is not a failure. +- [ ] **Step 6: Resources (C4).** `kill -TERM "$SAMPLER"; wait "$SAMPLER"`. Keep `$S/sample-summary.txt`. Copy it out of `$S` (into the Task 4 comment body file in a second `mktemp -d`) before Step 8. Stop the sampler before Step 7: the gateway restart in Step 7 occurs only in praxis mode. +- [ ] **Step 7: Fail closed (C6).** `kill -9 "$(pgrep -f "$S/bin/cortex-gateway")"`, and at once send one request with the `probe` curl line (praxis proxy `127.0.0.1:49710`, invalid key). Expect code 503 and, in `/v1/sessions`, a row with error kind `gateway_unavailable`. Then send the same request once each 0.5 s; record the seconds from the kill to the first code that is the same as the code direct mode got in Step 3 (`$S/probe.tsv`). Record each other code in the result. Pass when that time is at most 10 s. - [ ] **Step 8: Tear down.** `down all $S`, check `pgrep -f $S/bin/` is empty, `rm -rf $S`. ### Task 4: Record the result and the decision diff --git a/docs/superpowers/specs/2026-11-07-praxis-mode-design.md b/docs/superpowers/specs/2026-11-07-praxis-mode-design.md index 42d2d7ec3..c6b9e3abf 100644 --- a/docs/superpowers/specs/2026-11-07-praxis-mode-design.md +++ b/docs/superpowers/specs/2026-11-07-praxis-mode-design.md @@ -524,7 +524,7 @@ Line counts are non-test, non-comment Go lines on `main` @ `92604f4c`. The forwa ### Trigger and order -- **Phase D has no upstream trigger.** It starts when praxis mode is the default and stable. +- **Phase D has no upstream trigger.** It starts when praxis mode is stable behind `agentop setup --praxis`. Praxis mode does not become the default before phase C, because criterion C7 of the live check needs one network proxy (`docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md`). - **It comes before phase C.** Attribution keeps state that the store's events feed: `Attributor` is a store `Recorder`, `Clearer`, `Deleter`, `Evicter` and `Rekeyer` (`core/attribution/attribution.go:214-262`). So attribution and the store move together. If attribution moved first, it would need a new interface to the Go store, and that interface would be throwaway. - **The client address in phase D.** `Attributor.Lookup` takes only the client and server socket addresses (`core/attribution/lookup.go:23`), so any process on the host can ask it. In phase D the Go front door passes them in a token-protected header, `X-Cortex-Client: `. The binary runs the lookup itself. In phase C the address comes from the connection (praxis#1447), and the header goes away. From a3a5f13cd148aa4cf7554be0b29bf9f2cfa4d383 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 17:44:54 -0500 Subject: [PATCH 09/10] docs: Remove the known-differences sentence from the C2 step Fixes review: - Step 5 now compares the two modes only by shape, so the sentence about known header and timing differences no longer applied. Files: - docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index 00335bed4..8ac944d43 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -110,7 +110,7 @@ Praxis mode can become the default only when **every** criterion passes: - [ ] **Step 2: Credentials.** Praxis mode has no `inference-router`, so the agent must call the upstream directly. Take the first server URL under `inference-router` in the real `~/.cortex/config.yaml` and its key, into `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` in the agent shell only. For Codex, copy `~/.codex/config.toml` and `~/.codex/auth.json` into each scratch HOME's `.codex/`. When a credential is missing, do the steps that do not need it, and mark the dependent criteria "not run" with the reason. - [ ] **Step 3: Latency (C3).** `probe $S 200 "$ANTHROPIC_BASE_URL"`. Keep the printed summary. - [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `praxis-live-check.sh sample $S 86400 &` and keep its pid in `SAMPLER`. Step 6 stops it. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. When one Claude Code run has fewer than 40 inference turns, run the same prompt again in the same session (`claude -p --continue`) in both modes until it has 40. Then, for C5, `praxis-live-check.sh ttff $S 20 "$ANTHROPIC_BASE_URL"`. For Codex, `ttff` cannot call its upstream (Codex authenticates with an OAuth token), and `ttfbMs` measures the response headers, not the first frame. So record the Codex part of C5 as not run. Give the median `ttfbMs` of the Codex inference rows in each mode from the session API as information only. -- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --endpoint http://127.0.0.1:` for a visual check, and read the numbers with `curl -s --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}` (endpoint reference: `docs/session-api.md`). For C2, check each mode against its own raw captures (`/v1/sessions/{id}/events/{seq}?raw=1`), because the two modes run different LLM runs: the title, the timeline rows, the tool call names and the subagent blocks agree with the captured requests and responses, the tokens are equal to the captured `usage`, and the cost is direct mode's price for those tokens. Between the two modes, compare only the shape: one session for each agent run, subagent blocks present, and zero rows with a `gateway_*` error kind. Record each difference. A difference that Section 5 lists as known (captured request headers, timing within 50 ms) is not a failure. +- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --endpoint http://127.0.0.1:` for a visual check, and read the numbers with `curl -s --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}` (endpoint reference: `docs/session-api.md`). For C2, check each mode against its own raw captures (`/v1/sessions/{id}/events/{seq}?raw=1`), because the two modes run different LLM runs: the title, the timeline rows, the tool call names and the subagent blocks agree with the captured requests and responses, the tokens are equal to the captured `usage`, and the cost is direct mode's price for those tokens. Between the two modes, compare only the shape: one session for each agent run, subagent blocks present, and zero rows with a `gateway_*` error kind. Record each difference. - [ ] **Step 6: Resources (C4).** `kill -TERM "$SAMPLER"; wait "$SAMPLER"`. Keep `$S/sample-summary.txt`. Copy it out of `$S` (into the Task 4 comment body file in a second `mktemp -d`) before Step 8. Stop the sampler before Step 7: the gateway restart in Step 7 occurs only in praxis mode. - [ ] **Step 7: Fail closed (C6).** `kill -9 "$(pgrep -f "$S/bin/cortex-gateway")"`, and at once send one request with the `probe` curl line (praxis proxy `127.0.0.1:49710`, invalid key). Expect code 503 and, in `/v1/sessions`, a row with error kind `gateway_unavailable`. Then send the same request once each 0.5 s; record the seconds from the kill to the first code that is the same as the code direct mode got in Step 3 (`$S/probe.tsv`). Record each other code in the result. Pass when that time is at most 10 s. - [ ] **Step 8: Tear down.** `down all $S`, check `pgrep -f $S/bin/` is empty, `rm -rf $S`. From a35eb2c625b1261f378c2752c2bf3a02394efecc Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 18:39:58 -0500 Subject: [PATCH 10/10] fix: Keep the scratch Cortex off the caller's proxy and stop on empty samples Fixes review: - up removes the proxy and CA variables from the child environment. - probe and ttff send curl through the scratch proxy also when NO_PROXY names the host. - probe sends no Expect: 100-continue, and prints the count of each code. - up refuses when a port of its mode already accepts a connection. - sample fails when it has no rows, or praxis cortex rows with no gateway rows. - The plan text agrees with the key read in Task 3, the spec line, and "Part of #1725". Files: - docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md - scripts/dev/praxis-live-check.sh Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../2026-11-10-praxis-mode-8-live-check.md | 14 +++---- scripts/dev/praxis-live-check.sh | 41 ++++++++++++++++--- 2 files changed, 43 insertions(+), 12 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md index 8ac944d43..7a237073b 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -14,7 +14,7 @@ ## Global Constraints -- **Never touch the shared proxy.** Do not run `agentop setup`, `agentop service *`, `make dev-install` or `install.sh`. Do not use ports 47600-47606. Do not read or write the real `~/.cortex/` except to read field names and the router's server URLs. Every scratch process runs with `HOME=/home`. +- **Never touch the shared proxy.** Do not run `agentop setup`, `agentop service *`, `make dev-install` or `install.sh`. Do not use ports 47600-47606. Do not read or write the real `~/.cortex/` except to read field names, the router's server URLs, and the key of the server that Task 3 Step 2 uses (into an environment variable only). Every scratch process runs with `HOME=/home`. - Scratch ports. Direct instance: proxy `127.0.0.1:49700`, session API `:49701`, stats `:49702`, transparent `:49703`, health `:49704`. Praxis instance: `:49710`-`:49714` in the same order, `praxis.listen: 127.0.0.1:49715`, `praxis.egress: 127.0.0.1:49716`. - `cortex -supervise` finds `cortex-gateway` in the directory of its own binary (`cmd/cortex/supervise.go:239`). So both binaries go in one scratch `bin/` directory. - Credentials: read an API key into an environment variable only. Never write a key into the repository, a commit, a plan, an issue comment or a file outside the scratch directory. Delete the scratch directory at the end (`rm -rf`). @@ -29,7 +29,7 @@ - **D-C: Two scratch instances at the same time, not one instance restarted between modes.** Reason: the issue asks for interleaved latency. Alternate requests between the two proxies in one loop, so gateway noise hits both modes equally (see the 2026-10-02 latency probe method). - **D-D: The latency probe sends a ~1 MiB Claude-Code-shaped `/v1/messages` body with an invalid `x-api-key`.** Reason: the upstream refuses it with 401 before a model runs, so it costs no tokens, and the parsers still read the full body. - **D-E: The script is a dev tool, not CI.** Reason: it needs real credentials and a real upstream. It stays in `scripts/dev/` so the rollback check and phase B can run it again. -- **D-F: The PR holds the script and this plan only.** The results go on the issues, not into a repository document. Reason: the issue says "record on this issue"; a second copy drifts. +- **D-F: The PR holds the script, this plan and one line of the praxis spec.** The results go on the issues, not into a repository document. Reason: the issue says "record on this issue"; a second copy drifts. ## Criteria for the default (Task 2 posts these verbatim) @@ -71,10 +71,10 @@ Praxis mode can become the default only when **every** criterion passes: **Interfaces:** - Produces, for Task 3 (subcommands, each prints one result per line): - `praxis-live-check.sh build `: copy the build form of `scripts/dev/praxis-measure.sh`. `tags=$(go -C "$root/scripts/profile-tags" run . full)`, then `(cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o /bin/cortex .)`. Then `(cd "$root/praxis" && cargo build --release --locked -p cortex-gateway)` and `cp "$root/praxis/target/release/cortex-gateway" /bin/`. `$root` is `$(cd "$(dirname "$0")/../.." && pwd)`. - - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Starts `HOME=//home /bin/cortex -supervise --config ` in the background, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200 and, for `praxis`, for the log line `praxis hop: cortex-gateway is ready` (`/healthz` does not wait for the hop). Exit 1 on timeout with the last 20 log lines. + - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Exits 1 when a port of that mode already accepts a connection. Starts `HOME=//home /bin/cortex -supervise --config ` in the background, with `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, `NO_PROXY` (both cases), `SSL_CERT_FILE` and `SSL_CERT_DIR` removed from its environment, so a caller shell that runs through the shared proxy does not leak into the scratch instance, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200 and, for `praxis`, for the log line `praxis hop: cortex-gateway is ready` (`/healthz` does not wait for the hop). Exit 1 on timeout with the last 20 log lines. - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, `http_proxy` (all `http://127.0.0.1:`), `NO_PROXY=127.0.0.1,localhost`, `NODE_EXTRA_CA_CERTS=/ca.crt`, and `SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, `GIT_SSL_CAINFO` all `=/bundle.crt`. Cortex writes `ca.crt` and `bundle.crt` (bridge CA plus platform roots) into `ca_dir` itself (see `agentop exec --help`); `env` exits 1 when either file is missing. - - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of 513 turns (user and assistant in turn) of 2 KiB text each, about 1 MiB, so the parsers work per message; `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `dropped `. A pair is dropped when its two codes differ or a code is `000`, so a local praxis 503 is not counted as latency. - - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?debug=1"`, the `# HeapInuse = ` line (no `gc=1`: a forced GC every 5 s adds CPU to both cortex rows and skews C4), cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 0 after a signal. + - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --noproxy '' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'Expect:' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of 513 turns (user and assistant in turn) of 2 KiB text each, about 1 MiB, so the parsers work per message; `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `codes : …`, `dropped `. `--noproxy ''` stops `NO_PROXY` from sending the request past the proxy; `Expect:` stops curl's `100-continue` wait on the 1 MiB body. A pair is dropped when its two codes differ or a code is `000`, so a local praxis 503 is not counted as latency. + - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?debug=1"`, the `# HeapInuse = ` line (no `gc=1`: a forced GC every 5 s adds CPU to both cortex rows and skews C4), cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 1 when `sample.tsv` has no rows, or has praxis `cortex` rows but no `cortex-gateway` rows; else exit 0, also after a signal. - `praxis-live-check.sh ttff `: for C5. `n` interleaved pairs like `probe`, each a small streaming `/v1/messages` request (`max_tokens: 16`, `stream: true`) with the `Authorization: Bearer $ANTHROPIC_AUTH_TOKEN` header given to curl through `-K <(…)`, so the token is not in the curl argv; exit 2 when that variable is empty. Time to the first line that starts with `data:` in the response, measured with `perl -MTime::HiRes=time` (macOS `date` has no milliseconds). Writes `/ttff.tsv` (`pair mode ms`) and prints `direct p50`, `praxis p50`, `delta p50`, `dropped `. A pair where either side has no `data:` line is dropped. - `praxis-live-check.sh down direct|praxis|all `: see Review Focus. - `praxis-live-check.sh selftest`: runs the pure functions on fixed input and exits 0 only when all pass. @@ -87,7 +87,7 @@ Praxis mode can become the default only when **every** criterion passes: - [ ] **Step 2: Run it and see it fail.** Run: `scripts/dev/praxis-live-check.sh selftest`. Expected: non-zero exit (functions not defined). - [ ] **Step 3: Write the functions and the subcommands** as the Interfaces block says. `set -euo pipefail`. `up` calls `ports_ok` on the generated file and refuses on failure. - [ ] **Step 4: Run the gate.** Run: `bash -n scripts/dev/praxis-live-check.sh && shellcheck scripts/dev/praxis-live-check.sh && scripts/dev/praxis-live-check.sh selftest`. Expected: exit 0, `selftest: ok`. -- [ ] **Step 5: Smoke test with no credentials.** `S=$(mktemp -d)`; `build $S`; `up direct $S`; `up praxis $S`; `probe $S 5 https://api.anthropic.com`; `sample $S 15`; `down all $S`. Expected: both `up` succeed, the probe prints 5 pairs with code 401 in both modes, `sample.tsv` has `cortex-gateway` rows only for praxis, and `pgrep -f $S/bin/` prints nothing after `down`. Then `rm -rf $S`. +- [ ] **Step 5: Smoke test with no credentials.** `S=$(mktemp -d)`; `build $S`; `up direct $S`; `up praxis $S`; `probe $S 5 https://api.anthropic.com`; `sample $S 15`; `down all $S`. Expected: both `up` succeed, the probe prints `codes 401:5`, `sample.tsv` has `cortex-gateway` rows only for praxis, and `pgrep -f $S/bin/` prints nothing after `down`. Then `rm -rf $S`. - [ ] **Step 6: Commit.** `git add scripts/dev/praxis-live-check.sh && git commit -s -m "feat: Add a dev script for the praxis-mode live check"` with the attribution trailer the repository CLAUDE.md names. ### Task 2: Post the criteria before the check @@ -124,7 +124,7 @@ Praxis mode can become the default only when **every** criterion passes: - [ ] **Step 1:** Post one comment on #1725: a table with one row per criterion C1-C7 (measured value, threshold, pass / fail / not run), the probe and sample summaries, the agent and Claude Code / Codex versions, the date, and every C2 difference. - [ ] **Step 2: Decide.** Rule: the default changes only when C1-C7 all pass. Expected result: C1 fails by design, so the decision is "direct mode stays the default; praxis mode stays opt-in (`agentop setup --praxis`)". Write the decision and the criteria that block it. - [ ] **Step 3:** Post a short comment on #1714 with the decision and a link to the result comment. Update the tracker's phase A row when the body has one for step 8. -- [ ] **Step 4:** Do not close #1725 by hand. The PR from Task 1 says `Closes #1725` in its body, so the merge closes it after the result is recorded. +- [ ] **Step 4:** Close #1725 with a link to the result comment. The PR from Task 1 says `Part of #1725`, so its merge does not close the issue. ## PRs diff --git a/scripts/dev/praxis-live-check.sh b/scripts/dev/praxis-live-check.sh index 068f825e3..b12dad42c 100755 --- a/scripts/dev/praxis-live-check.sh +++ b/scripts/dev/praxis-live-check.sh @@ -69,6 +69,19 @@ is_up() { [ "$1" != praxis ] || grep -q 'praxis hop: cortex-gateway is ready' "$3" } +# port_busy PORT: succeeds when something accepts a TCP connection on 127.0.0.1:PORT. +port_busy() { + (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null +} + +# sample_check FILE: fails when FILE has no rows, or has praxis cortex rows but no +# cortex-gateway rows. A pgrep pattern that matches nothing must not look like a pass. +sample_check() { + [ -s "$1" ] || { echo "no process was sampled" >&2; return 1; } + ! awk '$2 == "praxis" && $3 == "cortex" {c = 1} $3 == "cortex-gateway" {g = 1} END {exit !(c && !g)}' "$1" || + { echo "praxis cortex was sampled, but no cortex-gateway" >&2; return 1; } +} + selftest() { fail=0 check() { @@ -105,6 +118,13 @@ selftest() { if is_up praxis 200 "$t/cortex.log"; then echo "selftest: is_up passed praxis before the hop is ready" >&2; fail=1; fi echo 'level=INFO msg="praxis hop: cortex-gateway is ready" gateway=127.0.0.1:49715' >"$t/cortex.log" if ! is_up praxis 200 "$t/cortex.log"; then echo "selftest: is_up refused praxis with the hop ready" >&2; fail=1; fi + : >"$t/empty.tsv" + if sample_check "$t/empty.tsv" 2>/dev/null; then echo "selftest: sample_check passed no rows" >&2; fail=1; fi + printf '1\tpraxis\tcortex\t10\t100\t1\t5\n' >"$t/nogw.tsv" + if sample_check "$t/nogw.tsv" 2>/dev/null; then echo "selftest: sample_check passed praxis with no gateway" >&2; fail=1; fi + printf '1\tdirect\tcortex\t9\t100\t1\t5\n' >"$t/direct.tsv" + if ! sample_check "$t/direct.tsv"; then echo "selftest: sample_check refused direct only" >&2; fail=1; fi + if ! sample_check "$t/sample.tsv"; then echo "selftest: sample_check refused praxis with a gateway" >&2; fail=1; fi rm -rf "$t" [ "$fail" = 0 ] || exit 1 echo "selftest: ok" @@ -185,7 +205,15 @@ pipeline: $praxis_block EOF ports_ok "$cfg" || die "refusing to start: $cfg names a port of the shared proxy (476xx)" - HOME="$home" "$s/bin/cortex" -supervise --config "$cfg" "$d/cortex.log" 2>&1 & + last=$((base + 4)) + [ "$mode" != praxis ] || last=$((base + 6)) + for p in $(seq "$base" "$last"); do + ! port_busy "$p" || die "port $p is in use; run down, or stop the other instance" + done + # The child must not reach the shared proxy or trust its CA from the caller's shell. + env -u HTTPS_PROXY -u HTTP_PROXY -u https_proxy -u http_proxy -u ALL_PROXY -u all_proxy \ + -u NO_PROXY -u no_proxy -u SSL_CERT_FILE -u SSL_CERT_DIR \ + HOME="$home" "$s/bin/cortex" -supervise --config "$cfg" "$d/cortex.log" 2>&1 & echo $! >"$d/supervisor.pid" for _ in $(seq 300); do if is_up "$mode" "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$health/healthz" || true)" "$d/cortex.log"; then @@ -257,8 +285,9 @@ probe() { codes="" for m in $(order "$i"); do setports "$m" - r=$(curl -s -o /dev/null -w '%{http_code} %{time_total}' --proxy "http://127.0.0.1:$proxy" \ - --cacert "$s/$m/home/.cortex/ca/bundle.crt" -H 'x-api-key: invalid' \ + # --noproxy '': NO_PROXY must not send this past the proxy. Expect: no 100-continue wait. + r=$(curl -s -o /dev/null -w '%{http_code} %{time_total}' --noproxy '' --proxy "http://127.0.0.1:$proxy" \ + --cacert "$s/$m/home/.cortex/ca/bundle.crt" -H 'x-api-key: invalid' -H 'Expect:' \ -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \ --data-binary @"$body" "$url/v1/messages" || true) codes="$codes ${r%% *}" @@ -270,6 +299,7 @@ probe() { done [ -s "$out" ] || die "probe: every pair was dropped ($dropped)" summarize "$out" 4 50 95 + echo "codes $(cut -f3 "$out" | sort | uniq -c | awk '{printf "%s%s:%s", (NR > 1 ? " " : ""), $2, $1 / 2}')" echo "dropped $dropped" } @@ -302,7 +332,7 @@ ttff() { bad=0 for m in $(order "$i"); do setports "$m" - ms=$(first_frame_ms curl -sN --proxy "http://127.0.0.1:$proxy" \ + ms=$(first_frame_ms curl -sN --noproxy '' --proxy "http://127.0.0.1:$proxy" \ --cacert "$s/$m/home/.cortex/ca/bundle.crt" \ -K <(printf 'header = "Authorization: Bearer %s"\n' "$ANTHROPIC_AUTH_TOKEN") \ -H 'anthropic-version: 2023-06-01' \ @@ -358,7 +388,7 @@ sample_one() { sample() { s=$(abs "$1") secs=$2 - trap 'sample_summary; exit 0' TERM INT + trap 'sample_summary; sample_check "$s/sample.tsv" || exit 1; exit 0' TERM INT : >"$s/sample.tsv" # one run per summary; an earlier run's rows would skew it start=$(date +%s) while :; do @@ -375,6 +405,7 @@ sample() { wait $! || true done sample_summary + sample_check "$s/sample.tsv" || exit 1 } down() {