diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md new file mode 100644 index 000000000..7a237073b --- /dev/null +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md @@ -0,0 +1,152 @@ +# Praxis mode 8/8: Live check and the default decision — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Issue:** #1725 (part of tracker #1714, phase A, step 8). Steps #1718 to #1724 are closed. + +**Goal:** Write the criteria for the default, run Claude Code and Codex through praxis mode on a scratch Cortex, measure it against direct mode, and record the result and the decision on #1725 and #1714. + +**Architecture:** One new dev script, `scripts/dev/praxis-live-check.sh`, starts two scratch Cortex instances side by side (one direct, one praxis) under scratch HOMEs, on ports that do not touch the shared proxy. It measures interleaved latency and samples the CPU and heap of each process. The live check and the decision are GitHub comments, not code. + +**Tech Stack:** bash 3.2 (macOS), curl, ps, the Go `cortex` binary, the Rust `cortex-gateway` binary. + +**Spec:** `docs/superpowers/specs/2026-11-07-praxis-mode-design.md`, Section 5 "Live check" and Section 4 "Release and install" (`praxis.enabled` stays `false` until the live check is done). + +## Global Constraints + +- **Never touch the shared proxy.** Do not run `agentop setup`, `agentop service *`, `make dev-install` or `install.sh`. Do not use ports 47600-47606. Do not read or write the real `~/.cortex/` except to read field names, the router's server URLs, and the key of the server that Task 3 Step 2 uses (into an environment variable only). Every scratch process runs with `HOME=/home`. +- Scratch ports. Direct instance: proxy `127.0.0.1:49700`, session API `:49701`, stats `:49702`, transparent `:49703`, health `:49704`. Praxis instance: `:49710`-`:49714` in the same order, `praxis.listen: 127.0.0.1:49715`, `praxis.egress: 127.0.0.1:49716`. +- `cortex -supervise` finds `cortex-gateway` in the directory of its own binary (`cmd/cortex/supervise.go:239`). So both binaries go in one scratch `bin/` directory. +- Credentials: read an API key into an environment variable only. Never write a key into the repository, a commit, a plan, an issue comment or a file outside the scratch directory. Delete the scratch directory at the end (`rm -rf`). +- Bash 3.2: no `mapfile`, no associative arrays, no `${var,,}`. The script must pass `shellcheck`. +- Posts go to rossoctl/cortex issues #1725 and #1714 only. +- Text in comments and the PR body: ASD-STE100 style, short paragraphs, lists over prose. + +## Decisions taken in this plan + +- **D-A: The criteria live in this plan and in a comment on #1725 and #1714.** Reason: the issue says "write the criteria on this issue (#1714, Gates)". This plan is the reviewed copy. Task 2 posts them before Task 3 starts. +- **D-B: The default flip is NOT in this plan.** Reason: praxis mode turns off every outbound plugin by design (`cmd/cortex/praxis.go:33`) and agentop refuses `server`, `script` and `tools` in praxis mode (`praxisRefusal`). A default laptop install uses `tool-prune` and `inference-router`. So criterion C1 fails by design today, and the decision is "direct mode stays the default". If, against this, every criterion passes, the executor records "all criteria pass" and leaves the flip to Hai: a change of the default for every user needs a human decision. +- **D-C: Two scratch instances at the same time, not one instance restarted between modes.** Reason: the issue asks for interleaved latency. Alternate requests between the two proxies in one loop, so gateway noise hits both modes equally (see the 2026-10-02 latency probe method). +- **D-D: The latency probe sends a ~1 MiB Claude-Code-shaped `/v1/messages` body with an invalid `x-api-key`.** Reason: the upstream refuses it with 401 before a model runs, so it costs no tokens, and the parsers still read the full body. +- **D-E: The script is a dev tool, not CI.** Reason: it needs real credentials and a real upstream. It stays in `scripts/dev/` so the rollback check and phase B can run it again. +- **D-F: The PR holds the script, this plan and one line of the praxis spec.** The results go on the issues, not into a repository document. Reason: the issue says "record on this issue"; a second copy drifts. + +## Criteria for the default (Task 2 posts these verbatim) + +Praxis mode can become the default only when **every** criterion passes: + +- **C1 Feature parity.** Every feature that a default laptop install has works in praxis mode: `tool-prune`, `inference-router` (`agentop server`), script plugins (`agentop script`), `agentop tools`. Status today: fails by design (D-B). +- **C2 Facts.** For the same live session, agentop shows in praxis mode: one session per agent run, the title, every timeline row, the tool call names, the subagent blocks, input/output/cache tokens equal to the `usage` in the raw capture, and cost equal to direct mode's price for those tokens. Zero rows with a `gateway_*` error kind, apart from the fail-closed step. +- **C3 Latency.** 200 interleaved pairs, ~1 MiB body. Praxis p50 minus direct p50 is at most 5 ms. Praxis p95 minus direct p95 is at most 20 ms. +- **C4 Resources.** On a long session (at least 40 inference turns and at least 2 subagents): the peak RSS of cortex plus cortex-gateway is at most 1.5 times the peak RSS of direct-mode cortex. Their CPU seconds summed are at most 2 times direct-mode cortex CPU seconds. +- **C5 Streaming.** Claude Code SSE and Codex (`chatgpt.com` `/backend-api/codex/responses`) stream frame by frame in praxis mode: the time to the first frame is within 100 ms of direct mode. +- **C6 Fail closed.** Kill `cortex-gateway`: the next agent request gets 503 with a `gateway_unavailable` row. The supervisor restarts the gateway, and requests succeed again within 10 s. +- **C7 One network proxy.** The conversion is complete: `cortex-gateway` is the only network proxy on the laptop, and no traffic goes through the Go forward proxy. Until C7 passes, praxis mode stays behind the `--praxis` flag (`agentop setup --praxis`), even when C1-C6 pass. Status today: fails (phases B-D of #1714 are open). Decision by Hai. + +**Condition to go back to direct mode** (after a future flip): set `praxis.enabled: false` and restart when any one of these occurs: + +- an agent request with no row in the session store; +- a `gateway_unavailable` row outside a Cortex or gateway restart; +- a measured p50 added latency over 20 ms; +- a cortex plus cortex-gateway RSS over 2 times direct mode; +- a parser parity test red on `main` for more than 7 days. + +## Review Focus + +- A scratch instance that binds a 476xx port, or a process that inherits the real `HOME`. Expect: `up` refuses to start when any 476xx port appears in the generated config, and it exports `HOME` before it starts `cortex`. +- `down` leaves `cortex-gateway` running. Expect: `down` stops the supervisor, then waits until no process from the scratch `bin/` runs (`pgrep -f /bin/`), and kills the rest after 5 s. +- A probe pair where one request fails (connection refused). Expect: the pair is dropped and counted, and the summary prints the dropped count; it does not compute a percentile on mixed data. +- The sampler that finds no gateway pid in direct mode. Expect: the gateway column is empty for direct mode, not a failure. +- Percentiles on an even count or a count of 1. Expect: nearest-rank percentile (`ceil(p/100*n)`), pinned by `selftest`. + +--- + +### Task 1: The live-check script + +**Depends on:** none. + +**Files:** +- Create: `scripts/dev/praxis-live-check.sh` (mode 0755) + +**Interfaces:** +- Produces, for Task 3 (subcommands, each prints one result per line): + - `praxis-live-check.sh build `: copy the build form of `scripts/dev/praxis-measure.sh`. `tags=$(go -C "$root/scripts/profile-tags" run . full)`, then `(cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o /bin/cortex .)`. Then `(cd "$root/praxis" && cargo build --release --locked -p cortex-gateway)` and `cp "$root/praxis/target/release/cortex-gateway" /bin/`. `$root` is `$(cd "$(dirname "$0")/../.." && pwd)`. + - `praxis-live-check.sh up direct|praxis `: writes `//home/.cortex/config.yaml` (the config shape of `scripts/dev/praxis-measure.sh` plus the TLS bridge: `mode: proxy-sidecar`; `listener: {roles: [forward], bind_loopback_only: true, forward_proxy_addr, session_api_addr, transparent_proxy_addr, health_addr}` with the scratch ports; `stats: {address: 127.0.0.1:}`; no `session.id_headers` (keep the default); `tls_bridge: {mode: enabled, generate_ca: true, ca_dir: //home/.cortex/ca}`, outbound plugins `inference-parser`, `mcp-parser`, `a2a-parser`; for `praxis` also `praxis: {enabled: true, listen: …, egress: …}`). Exits 1 when a port of that mode already accepts a connection. Starts `HOME=//home /bin/cortex -supervise --config ` in the background, with `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, `NO_PROXY` (both cases), `SSL_CERT_FILE` and `SSL_CERT_DIR` removed from its environment, so a caller shell that runs through the shared proxy does not leak into the scratch instance, writes `//supervisor.pid`, logs to `//cortex.log`, and waits up to 30 s for `GET http://127.0.0.1:/healthz` 200 and, for `praxis`, for the log line `praxis hop: cortex-gateway is ready` (`/healthz` does not wait for the hop). Exit 1 on timeout with the last 20 log lines. + - `praxis-live-check.sh env direct|praxis `: prints `export` lines for an agent shell: `HOME`, `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, `http_proxy` (all `http://127.0.0.1:`), `NO_PROXY=127.0.0.1,localhost`, `NODE_EXTRA_CA_CERTS=/ca.crt`, and `SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, `GIT_SSL_CAINFO` all `=/bundle.crt`. Cortex writes `ca.crt` and `bundle.crt` (bridge CA plus platform roots) into `ca_dir` itself (see `agentop exec --help`); `env` exits 1 when either file is missing. + - `praxis-live-check.sh probe `: `n` pairs; in each pair the order of direct and praxis alternates. Each request: `curl -s -o /dev/null -w '%{http_code} %{time_total}' --noproxy '' --proxy http://127.0.0.1: --cacert /bundle.crt -H 'x-api-key: invalid' -H 'Expect:' -H 'anthropic-version: 2023-06-01' --data-binary @/body.json /v1/messages`. The body file is generated once: a `messages` array of 513 turns (user and assistant in turn) of 2 KiB text each, about 1 MiB, so the parsers work per message; `model: claude-opus-4-6`, `max_tokens: 16`, `stream: true`. Writes `/probe.tsv` (`pair mode code ms`) and prints `direct p50 p95`, `praxis p50 p95`, `delta p50 p95`, `codes : …`, `dropped `. `--noproxy ''` stops `NO_PROXY` from sending the request past the proxy; `Expect:` stops curl's `100-continue` wait on the 1 MiB body. A pair is dropped when its two codes differ or a code is `000`, so a local praxis 503 is not counted as latency. + - `praxis-live-check.sh sample `: every 5 s until ``, appends to `/sample.tsv` the line `epoch mode proc pid rss_kb cpu_s heap_inuse_bytes` for `cortex` in each mode and `cortex-gateway` in praxis mode (`ps -o rss=,time= -p `; heap from `curl -s --noproxy '*' "http://127.0.0.1:/debug/pprof/heap?debug=1"`, the `# HeapInuse = ` line (no `gc=1`: a forced GC every 5 s adds CPU to both cortex rows and skews C4), cortex only; `-` when absent; cortex-gateway serves no pprof, so its heap is not measured and RSS stands for it). Find the pids with `pgrep -f "/bin/cortex --config //"` (the supervised child, not the supervisor) and `pgrep -f "/bin/cortex-gateway"`. At the end, or on SIGTERM or SIGINT (a `trap`), prints and writes to `/sample-summary.txt`, per mode and process: peak RSS (KiB), peak heap in use (bytes), and CPU seconds used (last `cpu_s` minus first). Exit 1 when `sample.tsv` has no rows, or has praxis `cortex` rows but no `cortex-gateway` rows; else exit 0, also after a signal. + - `praxis-live-check.sh ttff `: for C5. `n` interleaved pairs like `probe`, each a small streaming `/v1/messages` request (`max_tokens: 16`, `stream: true`) with the `Authorization: Bearer $ANTHROPIC_AUTH_TOKEN` header given to curl through `-K <(…)`, so the token is not in the curl argv; exit 2 when that variable is empty. Time to the first line that starts with `data:` in the response, measured with `perl -MTime::HiRes=time` (macOS `date` has no milliseconds). Writes `/ttff.tsv` (`pair mode ms`) and prints `direct p50`, `praxis p50`, `delta p50`, `dropped `. A pair where either side has no `data:` line is dropped. + - `praxis-live-check.sh down direct|praxis|all `: see Review Focus. + - `praxis-live-check.sh selftest`: runs the pure functions on fixed input and exits 0 only when all pass. +- Pure functions, defined before the dispatch so `selftest` can call them: `pct

` (reads numbers on stdin, prints the nearest-rank percentile), `cputime_s ` (`MM:SS.ss` or `HH:MM:SS` or `D-HH:MM:SS` to seconds), `ports_ok ` (exit 1 when the file names any `:476[0-9][0-9]`). + +- [ ] **Step 1: Write `selftest` first.** Cases: + - `printf '1\n2\n3\n4\n' | pct 50` prints `2`; `pct 95` prints `4`; `printf '7\n' | pct 50` prints `7`. + - `cputime_s 0:01.50` prints `1.50`; `cputime_s 01:02:03` prints `3723`; `cputime_s 1-00:00:01` prints `86401`. + - `ports_ok` fails on a file with `forward_proxy_addr: 127.0.0.1:47600` and passes on one with `127.0.0.1:49700`. +- [ ] **Step 2: Run it and see it fail.** Run: `scripts/dev/praxis-live-check.sh selftest`. Expected: non-zero exit (functions not defined). +- [ ] **Step 3: Write the functions and the subcommands** as the Interfaces block says. `set -euo pipefail`. `up` calls `ports_ok` on the generated file and refuses on failure. +- [ ] **Step 4: Run the gate.** Run: `bash -n scripts/dev/praxis-live-check.sh && shellcheck scripts/dev/praxis-live-check.sh && scripts/dev/praxis-live-check.sh selftest`. Expected: exit 0, `selftest: ok`. +- [ ] **Step 5: Smoke test with no credentials.** `S=$(mktemp -d)`; `build $S`; `up direct $S`; `up praxis $S`; `probe $S 5 https://api.anthropic.com`; `sample $S 15`; `down all $S`. Expected: both `up` succeed, the probe prints `codes 401:5`, `sample.tsv` has `cortex-gateway` rows only for praxis, and `pgrep -f $S/bin/` prints nothing after `down`. Then `rm -rf $S`. +- [ ] **Step 6: Commit.** `git add scripts/dev/praxis-live-check.sh && git commit -s -m "feat: Add a dev script for the praxis-mode live check"` with the attribution trailer the repository CLAUDE.md names. + +### Task 2: Post the criteria before the check + +**Depends on:** none. + +**Files:** none (GitHub comments). + +- [ ] **Step 1:** Post the section "Criteria for the default" (C1-C7 and the condition to go back) as one comment on #1725, with `gh issue comment 1725 --repo rossoctl/cortex --body-file `. Start it with "Criteria for the default, written before the live check." Write the body file in the scratch directory, not in the repository. +- [ ] **Step 2:** Post a short comment on #1714 that links that comment and says it fills the two gaps under "Gates": criteria for the default, and rollback criteria. +- [ ] **Step 3:** Record both comment URLs in the Task 4 report. + +### Task 3: Run the live check + +**Depends on:** Task 1, Task 2. + +**Files:** none in the repository. All output goes to ``. + +- [ ] **Step 1: Set up.** `S=$(mktemp -d)`, `chmod 700 $S`. `build`, `up direct`, `up praxis`. In praxis `cortex.log`, find the Warn line "praxis mode: outbound plugins are off". +- [ ] **Step 2: Credentials.** Praxis mode has no `inference-router`, so the agent must call the upstream directly. Take the first server URL under `inference-router` in the real `~/.cortex/config.yaml` and its key, into `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` in the agent shell only. For Codex, copy `~/.codex/config.toml` and `~/.codex/auth.json` into each scratch HOME's `.codex/`. When a credential is missing, do the steps that do not need it, and mark the dependent criteria "not run" with the reason. +- [ ] **Step 3: Latency (C3).** `probe $S 200 "$ANTHROPIC_BASE_URL"`. Keep the printed summary. +- [ ] **Step 4: Long session with subagents (C2, C4, C5).** Start `praxis-live-check.sh sample $S 86400 &` and keep its pid in `SAMPLER`. Step 6 stops it. In each mode, in the shell from `eval "$(praxis-live-check.sh env $S)"`, run the same Claude Code task with `claude -p --max-turns 60` that makes at least 2 subagents (prompt: "Use two Task subagents in parallel: one lists every Go module in this repository with its go.mod path, one counts the test files under core/. Then summarize both results."), run from a fresh `git clone --depth 1` of the repository inside `$S`. Then one `codex exec` task ("List the files under scripts/dev and say what each does.") in each mode. When one Claude Code run has fewer than 40 inference turns, run the same prompt again in the same session (`claude -p --continue`) in both modes until it has 40. Then, for C5, `praxis-live-check.sh ttff $S 20 "$ANTHROPIC_BASE_URL"`. For Codex, `ttff` cannot call its upstream (Codex authenticates with an OAuth token), and `ttfbMs` measures the response headers, not the first frame. So record the Codex part of C5 as not run. Give the median `ttfbMs` of the Codex inference rows in each mode from the session API as information only. +- [ ] **Step 5: Check in agentop.** For each mode: `HOME=//home agentop --endpoint http://127.0.0.1:` for a visual check, and read the numbers with `curl -s --noproxy '*' http://127.0.0.1:/v1/sessions` and `/v1/sessions/{id}` (endpoint reference: `docs/session-api.md`). For C2, check each mode against its own raw captures (`/v1/sessions/{id}/events/{seq}?raw=1`), because the two modes run different LLM runs: the title, the timeline rows, the tool call names and the subagent blocks agree with the captured requests and responses, the tokens are equal to the captured `usage`, and the cost is direct mode's price for those tokens. Between the two modes, compare only the shape: one session for each agent run, subagent blocks present, and zero rows with a `gateway_*` error kind. Record each difference. +- [ ] **Step 6: Resources (C4).** `kill -TERM "$SAMPLER"; wait "$SAMPLER"`. Keep `$S/sample-summary.txt`. Copy it out of `$S` (into the Task 4 comment body file in a second `mktemp -d`) before Step 8. Stop the sampler before Step 7: the gateway restart in Step 7 occurs only in praxis mode. +- [ ] **Step 7: Fail closed (C6).** `kill -9 "$(pgrep -f "$S/bin/cortex-gateway")"`, and at once send one request with the `probe` curl line (praxis proxy `127.0.0.1:49710`, invalid key). Expect code 503 and, in `/v1/sessions`, a row with error kind `gateway_unavailable`. Then send the same request once each 0.5 s; record the seconds from the kill to the first code that is the same as the code direct mode got in Step 3 (`$S/probe.tsv`). Record each other code in the result. Pass when that time is at most 10 s. +- [ ] **Step 8: Tear down.** `down all $S`, check `pgrep -f $S/bin/` is empty, `rm -rf $S`. + +### Task 4: Record the result and the decision + +**Depends on:** Task 3. + +**Files:** none (GitHub comments). + +- [ ] **Step 1:** Post one comment on #1725: a table with one row per criterion C1-C7 (measured value, threshold, pass / fail / not run), the probe and sample summaries, the agent and Claude Code / Codex versions, the date, and every C2 difference. +- [ ] **Step 2: Decide.** Rule: the default changes only when C1-C7 all pass. Expected result: C1 fails by design, so the decision is "direct mode stays the default; praxis mode stays opt-in (`agentop setup --praxis`)". Write the decision and the criteria that block it. +- [ ] **Step 3:** Post a short comment on #1714 with the decision and a link to the result comment. Update the tracker's phase A row when the body has one for step 8. +- [ ] **Step 4:** Close #1725 with a link to the result comment. The PR from Task 1 says `Part of #1725`, so its merge does not close the issue. + +## PRs + +| PR | Title | Parent | Tasks | estLines | +|---|---|---|---|---| +| P1 | `feat: Add a dev script for the praxis-mode live check` | "" (none) | T1, T2, T3, T4 | 340 | + +Estimate: T1 is about 340 counted lines (the script with `selftest`); T2-T4 are 0. The plan document does not count. + +T2-T4 change no files. They run in the P1 session after T1, and the PR body links the result comment. + +## Gate + +From the worktree root: + +```sh +bash -n scripts/dev/praxis-live-check.sh && shellcheck scripts/dev/praxis-live-check.sh && scripts/dev/praxis-live-check.sh selftest +``` + +Gate check on base 412ed825 (plan review, 2026-11-10): `bash` and `shellcheck` 0.11.0 are installed and run from the worktree root. Both steps fail on the base only because Task 1 creates the file (`No such file or directory`). The gate is stricter than CI, which runs `shellcheck --severity=error` (`.github/workflows/security-scans.yaml`); keep the strict form. The existing `scripts/dev/praxis-measure.sh` passes plain `shellcheck`. + +## Scope left out + +- The default flip (D-B). +- The heap of `cortex-gateway`: it serves no pprof, so C4 uses its RSS and CPU only. The issue asks for "the CPU and heap of both processes"; the result comment must say this. diff --git a/docs/superpowers/specs/2026-11-07-praxis-mode-design.md b/docs/superpowers/specs/2026-11-07-praxis-mode-design.md index 42d2d7ec3..c6b9e3abf 100644 --- a/docs/superpowers/specs/2026-11-07-praxis-mode-design.md +++ b/docs/superpowers/specs/2026-11-07-praxis-mode-design.md @@ -524,7 +524,7 @@ Line counts are non-test, non-comment Go lines on `main` @ `92604f4c`. The forwa ### Trigger and order -- **Phase D has no upstream trigger.** It starts when praxis mode is the default and stable. +- **Phase D has no upstream trigger.** It starts when praxis mode is stable behind `agentop setup --praxis`. Praxis mode does not become the default before phase C, because criterion C7 of the live check needs one network proxy (`docs/superpowers/plans/2026-11-10-praxis-mode-8-live-check.md`). - **It comes before phase C.** Attribution keeps state that the store's events feed: `Attributor` is a store `Recorder`, `Clearer`, `Deleter`, `Evicter` and `Rekeyer` (`core/attribution/attribution.go:214-262`). So attribution and the store move together. If attribution moved first, it would need a new interface to the Go store, and that interface would be throwaway. - **The client address in phase D.** `Attributor.Lookup` takes only the client and server socket addresses (`core/attribution/lookup.go:23`), so any process on the host can ask it. In phase D the Go front door passes them in a token-protected header, `X-Cortex-Client: `. The binary runs the lookup itself. In phase C the address comes from the connection (praxis#1447), and the header goes away. diff --git a/scripts/dev/praxis-live-check.sh b/scripts/dev/praxis-live-check.sh new file mode 100755 index 000000000..b12dad42c --- /dev/null +++ b/scripts/dev/praxis-live-check.sh @@ -0,0 +1,451 @@ +#!/bin/bash +# praxis-live-check.sh runs two scratch Cortex instances side by side, one in direct +# mode and one in praxis mode, for the praxis-mode live check (#1725). +# +# Every process runs under //home on the 497xx ports below. The script +# never touches the shared proxy on :47600-47606 or the real ~/.cortex. +# +# direct: proxy 49700, session API 49701, stats 49702, transparent 49703, health 49704 +# praxis: proxy 49710, session API 49711, stats 49712, transparent 49713, health 49714, +# praxis.listen 49715, praxis.egress 49716 +# +# Usage: +# praxis-live-check.sh build build cortex and cortex-gateway +# praxis-live-check.sh up direct|praxis start one instance +# praxis-live-check.sh env direct|praxis print exports for an agent shell +# praxis-live-check.sh probe n interleaved latency pairs +# praxis-live-check.sh sample sample RSS, CPU and heap +# praxis-live-check.sh ttff n pairs, time to the first frame +# praxis-live-check.sh down direct|praxis|all +# praxis-live-check.sh selftest +set -euo pipefail + +root=$(cd "$(dirname "$0")/../.." && pwd) + +die() { + echo "praxis-live-check: $*" >&2 + exit 1 +} + +# pct P: reads numbers on stdin and prints the nearest-rank percentile ceil(P/100*n). +# P is an integer. +pct() { + sort -n | awk -v p="$1" '{a[NR] = $1} END { + if (NR == 0) exit 1 + r = int((p * NR + 99) / 100) + if (r < 1) r = 1 + print a[r] + }' +} + +# cputime_s T: converts a ps time (MM:SS.ss, HH:MM:SS or D-HH:MM:SS) to seconds. +cputime_s() { + echo "$1" | awk '{ + d = 0; t = $0 + if (index(t, "-") > 0) { split(t, dp, "-"); d = dp[1]; t = dp[2] } + n = split(t, f, ":") + s = d * 86400 + f[1] + if (n == 3) s = d * 86400 + f[1] * 3600 + f[2] * 60 + f[3] + if (n == 2) s = d * 86400 + f[1] * 60 + f[2] + if (index(f[n], ".") > 0) printf "%.2f\n", s; else printf "%d\n", s + }' +} + +# ports_ok FILE: fails when FILE names a port of the shared proxy (476xx). +ports_ok() { + ! grep -qE ':476[0-9][0-9]' "$1" +} + +# pair_ok CODE CODE: succeeds when a probe pair is kept: both modes got the same code, +# and it is not 000. A praxis 503 from the hop is not the answer direct got. +pair_ok() { + [ "$1" = "$2" ] && [ "$1" != 000 ] +} + +# is_up MODE HEALTH_CODE LOG: succeeds when an instance is ready for traffic. In praxis +# mode /healthz does not wait for the hop, so also wait for the hop's ready line. +is_up() { + [ "$2" = 200 ] || return 1 + [ "$1" != praxis ] || grep -q 'praxis hop: cortex-gateway is ready' "$3" +} + +# port_busy PORT: succeeds when something accepts a TCP connection on 127.0.0.1:PORT. +port_busy() { + (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null +} + +# sample_check FILE: fails when FILE has no rows, or has praxis cortex rows but no +# cortex-gateway rows. A pgrep pattern that matches nothing must not look like a pass. +sample_check() { + [ -s "$1" ] || { echo "no process was sampled" >&2; return 1; } + ! awk '$2 == "praxis" && $3 == "cortex" {c = 1} $3 == "cortex-gateway" {g = 1} END {exit !(c && !g)}' "$1" || + { echo "praxis cortex was sampled, but no cortex-gateway" >&2; return 1; } +} + +selftest() { + fail=0 + check() { + if [ "$2" != "$3" ]; then + echo "selftest: $1: got '$2', want '$3'" >&2 + fail=1 + fi + } + check "pct 50 of 4" "$(printf '1\n2\n3\n4\n' | pct 50)" 2 + check "pct 95 of 4" "$(printf '1\n2\n3\n4\n' | pct 95)" 4 + check "pct 50 of 1" "$(printf '7\n' | pct 50)" 7 + check "pct 95 of 20" "$(seq 20 | pct 95)" 19 + check "pct 50 unsorted" "$(printf '9\n1\n5\n' | pct 50)" 5 + check "cputime 0:01.50" "$(cputime_s 0:01.50)" 1.50 + check "cputime 01:02:03" "$(cputime_s 01:02:03)" 3723 + check "cputime 1-00:00:01" "$(cputime_s 1-00:00:01)" 86401 + t=$(mktemp -d "${TMPDIR:-/tmp}/praxis-live-check-selftest.XXXXXX") + echo " forward_proxy_addr: 127.0.0.1:47600" >"$t/bad.yaml" + echo " forward_proxy_addr: 127.0.0.1:49700" >"$t/good.yaml" + if ports_ok "$t/bad.yaml"; then echo "selftest: ports_ok passed :47600" >&2; fail=1; fi + if ! ports_ok "$t/good.yaml"; then echo "selftest: ports_ok refused :49700" >&2; fail=1; fi + # Two pids under one key (a restart): CPU is (3-1) + (2-0.5), not 2-1. + printf '1\tpraxis\tcortex-gateway\t10\t100\t1\t-\n1\tpraxis\tcortex-gateway\t10\t300\t3\t-\n2\tpraxis\tcortex-gateway\t11\t200\t0.5\t-\n2\tpraxis\tcortex-gateway\t11\t200\t2\t-\n' >"$t/sample.tsv" + check "sample_summary restart" "$(s=$t sample_summary)" "praxis cortex-gateway peak_rss_kb 300 peak_heap_inuse_bytes - cpu_s 3.50" + if ! pair_ok 401 401; then echo "selftest: pair_ok refused 401 401" >&2; fail=1; fi + if pair_ok 401 000; then echo "selftest: pair_ok kept 401 000" >&2; fail=1; fi + # A local 503 from the praxis hop is not the upstream answer direct got. + if pair_ok 401 503; then echo "selftest: pair_ok kept 401 503" >&2; fail=1; fi + if pair_ok 503 401; then echo "selftest: pair_ok kept 503 401" >&2; fail=1; fi + : >"$t/cortex.log" + if ! is_up direct 200 "$t/cortex.log"; then echo "selftest: is_up refused direct 200" >&2; fail=1; fi + if is_up direct 000 "$t/cortex.log"; then echo "selftest: is_up passed direct 000" >&2; fail=1; fi + # /healthz does not wait for the praxis hop; the hop logs once when its probe passes. + if is_up praxis 200 "$t/cortex.log"; then echo "selftest: is_up passed praxis before the hop is ready" >&2; fail=1; fi + echo 'level=INFO msg="praxis hop: cortex-gateway is ready" gateway=127.0.0.1:49715' >"$t/cortex.log" + if ! is_up praxis 200 "$t/cortex.log"; then echo "selftest: is_up refused praxis with the hop ready" >&2; fail=1; fi + : >"$t/empty.tsv" + if sample_check "$t/empty.tsv" 2>/dev/null; then echo "selftest: sample_check passed no rows" >&2; fail=1; fi + printf '1\tpraxis\tcortex\t10\t100\t1\t5\n' >"$t/nogw.tsv" + if sample_check "$t/nogw.tsv" 2>/dev/null; then echo "selftest: sample_check passed praxis with no gateway" >&2; fail=1; fi + printf '1\tdirect\tcortex\t9\t100\t1\t5\n' >"$t/direct.tsv" + if ! sample_check "$t/direct.tsv"; then echo "selftest: sample_check refused direct only" >&2; fail=1; fi + if ! sample_check "$t/sample.tsv"; then echo "selftest: sample_check refused praxis with a gateway" >&2; fail=1; fi + rm -rf "$t" + [ "$fail" = 0 ] || exit 1 + echo "selftest: ok" +} + +# abs DIR: the physical path, so pgrep patterns match what os.Executable gives. +abs() { + [ -d "$1" ] || die "no directory $1" + (cd "$1" && pwd -P) +} + +# setports MODE: sets the port variables of one instance. +setports() { + case "$1" in + direct) base=49700 ;; + praxis) base=49710 ;; + *) die "mode must be direct or praxis, not '$1'" ;; + esac + proxy=$base + api=$((base + 1)) + stats=$((base + 2)) + trans=$((base + 3)) + health=$((base + 4)) +} + +build() { + s=$(abs "$1") + mkdir -p "$s/bin" + tags=$(go -C "$root/scripts/profile-tags" run . full) + echo "building cortex (profile full)" >&2 + (cd "$root/cmd/cortex" && GOWORK=off go build -tags "$tags" -o "$s/bin/cortex" .) + echo "building cortex-gateway (release)" >&2 + (cd "$root/praxis" && cargo build --release --locked -p cortex-gateway >&2) + cp "$root/praxis/target/release/cortex-gateway" "$s/bin/" + echo "built $s/bin/cortex $s/bin/cortex-gateway" +} + +up() { + mode=$1 + setports "$mode" + s=$(abs "$2") + [ -x "$s/bin/cortex" ] || die "no $s/bin/cortex; run build first" + d="$s/$mode" + home="$d/home" + cfg="$home/.cortex/config.yaml" + if [ -f "$d/supervisor.pid" ] && kill -0 "$(cat "$d/supervisor.pid")" 2>/dev/null; then + die "$mode is already up (pid $(cat "$d/supervisor.pid"))" + fi + mkdir -p "$home/.cortex" + praxis_block="" + if [ "$mode" = praxis ]; then + praxis_block="praxis: + enabled: true + listen: 127.0.0.1:$((base + 5)) + egress: 127.0.0.1:$((base + 6))" + fi + cat >"$cfg" <"$d/cortex.log" 2>&1 & + echo $! >"$d/supervisor.pid" + for _ in $(seq 300); do + if is_up "$mode" "$(curl -s --noproxy '*' -o /dev/null -w '%{http_code}' "http://127.0.0.1:$health/healthz" || true)" "$d/cortex.log"; then + echo "$mode up: pid $(cat "$d/supervisor.pid"), proxy 127.0.0.1:$proxy, session API 127.0.0.1:$api" + return 0 + fi + sleep 0.1 + done + tail -20 "$d/cortex.log" >&2 + die "$mode: not ready in 30 s (/healthz 200, and in praxis mode the hop's ready line)" +} + +env_cmd() { + setports "$1" + s=$(abs "$2") + home="$s/$1/home" + ca="$home/.cortex/ca" + [ -f "$ca/ca.crt" ] && [ -f "$ca/bundle.crt" ] || die "no $ca/ca.crt or $ca/bundle.crt; is $1 up?" + p="http://127.0.0.1:$proxy" + echo "export HOME='$home'" + for v in HTTPS_PROXY HTTP_PROXY https_proxy http_proxy; do echo "export $v='$p'"; done + echo "export NO_PROXY='127.0.0.1,localhost'" + echo "export NODE_EXTRA_CA_CERTS='$ca/ca.crt'" + for v in SSL_CERT_FILE CURL_CA_BUNDLE REQUESTS_CA_BUNDLE GIT_SSL_CAINFO; do echo "export $v='$ca/bundle.crt'"; done +} + +# order I: the two modes of pair I, in the order this pair sends them. +order() { + if [ $(($1 % 2)) = 1 ]; then echo "direct praxis"; else echo "praxis direct"; fi +} + +# summarize FILE COLUMN NAMES: prints " []" per mode and the delta. +summarize() { + f=$1 col=$2 + shift 2 + for m in direct praxis; do + line="$m" + for p in "$@"; do line="$line $(awk -v m="$m" -v c="$col" '$2 == m {print $c}' "$f" | pct "$p")"; done + echo "$line" + done + line="delta" + for p in "$@"; do + dv=$(awk -v c="$col" '$2 == "direct" {print $c}' "$f" | pct "$p") + pv=$(awk -v c="$col" '$2 == "praxis" {print $c}' "$f" | pct "$p") + line="$line $(awk -v a="$pv" -v b="$dv" 'BEGIN {printf "%.1f", a - b}')" + done + echo "$line" +} + +probe() { + s=$(abs "$1") + n=$2 + url=$3 + body="$s/body.json" + if [ ! -f "$body" ]; then + # 513 turns of 2 KiB, not one 1 MiB string: the parsers work per message. + awk 'BEGIN { + x = sprintf("%2048s", ""); gsub(/ /, "x", x) + printf "{\"model\":\"claude-opus-4-6\",\"max_tokens\":16,\"stream\":true,\"messages\":[" + for (i = 1; i <= 513; i++) printf "%s{\"role\":\"%s\",\"content\":\"%s\"}", (i > 1 ? "," : ""), (i % 2 ? "user" : "assistant"), x + printf "]}" + }' >"$body" + fi + out="$s/probe.tsv" + : >"$out" + dropped=0 + for i in $(seq "$n"); do + pair="" + codes="" + for m in $(order "$i"); do + setports "$m" + # --noproxy '': NO_PROXY must not send this past the proxy. Expect: no 100-continue wait. + r=$(curl -s -o /dev/null -w '%{http_code} %{time_total}' --noproxy '' --proxy "http://127.0.0.1:$proxy" \ + --cacert "$s/$m/home/.cortex/ca/bundle.crt" -H 'x-api-key: invalid' -H 'Expect:' \ + -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \ + --data-binary @"$body" "$url/v1/messages" || true) + codes="$codes ${r%% *}" + pair="$pair$(echo "$r" | awk -v i="$i" -v m="$m" '{printf "%s\t%s\t%s\t%.1f\n", i, m, $1, $2 * 1000}') +" + done + # shellcheck disable=SC2086 # two codes, one argument each + if pair_ok $codes; then printf '%s' "$pair" >>"$out"; else dropped=$((dropped + 1)); fi + done + [ -s "$out" ] || die "probe: every pair was dropped ($dropped)" + summarize "$out" 4 50 95 + echo "codes $(cut -f3 "$out" | sort | uniq -c | awk '{printf "%s%s:%s", (NR > 1 ? " " : ""), $2, $1 / 2}')" + echo "dropped $dropped" +} + +# first_frame_ms CMD...: runs CMD and prints the ms to its first output line that +# starts with "data:". Exit 1 when no such line comes. +first_frame_ms() { + perl -MTime::HiRes=time -e ' + my $t = time; + my $pid = open(my $f, "-|", @ARGV) or exit 1; + while (<$f>) { + if (/^data:/) { printf "%.1f\n", (time - $t) * 1000; kill "TERM", $pid; exit 0 } + } + exit 1' "$@" +} + +ttff() { + [ -n "${ANTHROPIC_AUTH_TOKEN:-}" ] || { + echo "praxis-live-check: ttff needs ANTHROPIC_AUTH_TOKEN" >&2 + exit 2 + } + s=$(abs "$1") + n=$2 + url=$3 + small='{"model":"claude-opus-4-6","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"Say hi."}]}' + out="$s/ttff.tsv" + : >"$out" + dropped=0 + for i in $(seq "$n"); do + pair="" + bad=0 + for m in $(order "$i"); do + setports "$m" + ms=$(first_frame_ms curl -sN --noproxy '' --proxy "http://127.0.0.1:$proxy" \ + --cacert "$s/$m/home/.cortex/ca/bundle.crt" \ + -K <(printf 'header = "Authorization: Bearer %s"\n' "$ANTHROPIC_AUTH_TOKEN") \ + -H 'anthropic-version: 2023-06-01' \ + -H 'content-type: application/json' --data "$small" "$url/v1/messages" || true) + [ -n "$ms" ] || bad=1 + pair="$pair$i $m $ms +" + done + if [ "$bad" = 1 ]; then dropped=$((dropped + 1)); else printf '%s' "$pair" >>"$out"; fi + done + [ -s "$out" ] || die "ttff: every pair was dropped ($dropped)" + summarize "$out" 3 50 + echo "dropped $dropped" +} + +# sample_summary: per mode and process, peak RSS, peak heap and CPU seconds used. +# CPU is summed per pid: a restarted process starts its ps time again at 0. +sample_summary() { + awk '{ + k = $2 " " $3 + p = k " " $4 + if (!(k in rss)) { rss[k] = 0; keys[++nk] = k } + if (!(p in first)) { first[p] = $6; pk[p] = k } + last[p] = $6 + if ($5 > rss[k]) rss[k] = $5 + if ($7 != "-" && $7 > heap[k]) heap[k] = $7 + } END { + for (p in first) cpu[pk[p]] += last[p] - first[p] + for (i = 1; i <= nk; i++) { + k = keys[i] + printf "%s peak_rss_kb %d peak_heap_inuse_bytes %s cpu_s %.2f\n", k, rss[k], (k in heap) ? heap[k] : "-", cpu[k] + } + }' "$s/sample.tsv" | tee "$s/sample-summary.txt" +} + +# sample_one MODE PROC PID: appends one sample line. +sample_one() { + ps_out=$(ps -o rss=,time= -p "$3" 2>/dev/null || true) + [ -n "$ps_out" ] || return 0 + # shellcheck disable=SC2086 # split "rss time" into two fields + set -- "$1" "$2" "$3" $ps_out + heap=- + if [ "$2" = cortex ]; then + setports "$1" + # No gc=1: a forced GC every 5 s adds CPU to both cortex rows and skews C4. + heap=$(curl -s --noproxy '*' "http://127.0.0.1:$stats/debug/pprof/heap?debug=1" | + awk '/^# HeapInuse = / {print $4}' || true) + [ -n "$heap" ] || heap=- + fi + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$(date +%s)" "$1" "$2" "$3" "$4" "$(cputime_s "$5")" "$heap" >>"$s/sample.tsv" +} + +sample() { + s=$(abs "$1") + secs=$2 + trap 'sample_summary; sample_check "$s/sample.tsv" || exit 1; exit 0' TERM INT + : >"$s/sample.tsv" # one run per summary; an earlier run's rows would skew it + start=$(date +%s) + while :; do + for m in direct praxis; do + pid=$(pgrep -f "$s/bin/cortex --config $s/$m/" | head -1 || true) + [ -n "$pid" ] && sample_one "$m" cortex "$pid" + if [ "$m" = praxis ]; then + gpid=$(pgrep -f "$s/bin/cortex-gateway -c $s/$m/" | head -1 || true) + [ -n "$gpid" ] && sample_one "$m" cortex-gateway "$gpid" + fi + done + [ $(($(date +%s) - start)) -lt "$secs" ] || break + sleep 5 & + wait $! || true + done + sample_summary + sample_check "$s/sample.tsv" || exit 1 +} + +down() { + s=$(abs "$2") + case "$1" in + direct | praxis) modes=$1 pattern="$s/bin/cortex.* .*$s/$1/" ;; + all) modes="direct praxis" pattern="$s/bin/" ;; + *) die "down: mode must be direct, praxis or all" ;; + esac + for m in $modes; do + if [ -f "$s/$m/supervisor.pid" ]; then + kill "$(cat "$s/$m/supervisor.pid")" 2>/dev/null || true + rm -f "$s/$m/supervisor.pid" + fi + done + for _ in $(seq 50); do + pgrep -f "$pattern" >/dev/null || { echo "down: $1 stopped"; return 0; } + sleep 0.1 + done + echo "down: killing what is left after 5 s:" >&2 + pgrep -fl "$pattern" >&2 || true + pkill -9 -f "$pattern" || true + sleep 0.5 + ! pgrep -f "$pattern" >/dev/null || die "down: processes still run under $s/bin/" + echo "down: $1 stopped" +} + +usage() { + sed -n '12,20p' "$0" | sed 's/^# //' >&2 + exit 2 +} + +case "${1:-}" in +build) [ $# = 2 ] || usage; build "$2" ;; +up) [ $# = 3 ] || usage; up "$2" "$3" ;; +env) [ $# = 3 ] || usage; env_cmd "$2" "$3" ;; +probe) [ $# = 4 ] || usage; probe "$2" "$3" "$4" ;; +sample) [ $# = 3 ] || usage; sample "$2" "$3" ;; +ttff) [ $# = 4 ] || usage; ttff "$2" "$3" "$4" ;; +down) [ $# = 3 ] || usage; down "$2" "$3" ;; +selftest) selftest ;; +*) usage ;; +esac