Skip to content

feat(examples): score the smoke run's behavior into a pinned Trust Index - #27

Open
kperry-godaddy wants to merge 1 commit into
mainfrom
feat/openshell-smoke-trust-index
Open

kperry-godaddy wants to merge 1 commit into
mainfrom
feat/openshell-smoke-trust-index

Conversation

@kperry-godaddy

Copy link
Copy Markdown
Member

Summary

Adds make openshell-smoke-trust-index: the existing OpenShell smoke run plus five checks that take the run's policy decisions all the way to a Trust Index score. The gateway's OCSF JSON export is switched on, the approved sandbox's supervisor records are collected, ans-ocsf-hydrator scores them and imports the score into a pinned agent-trust-discovery release the harness starts on a loopback port, and the score is read back from the agent's trust evaluation at weight 0. make openshell-smoke is unchanged.

Until now the behavior signal openshell.behavior.activity had no automated real-runtime proof: the hydrator was covered by unit tests over synthetic records and by hand-run checks on OpenShell v0.1.5, and nothing exercised the import contract against a running Trust Index.

Changes

  • examples/openshell-smoke/trust-index.sh (new): the behavior path, sourced by smoke.sh only under --with-trust-index. Starts the pinned Trust Index with the signal registration from deploy/examples/trust-index-score-signals.yaml and a minimal profile, registers the fixture agent, enables ocsf_json_enabled on the run's gateway, and runs checks 6 to 10: export live before the first request, a policy-denied DELETE that never reaches the callee, every collected record attributed to the gateway-issued sandbox id with exactly the run's decisions, the hydrator's evidence and score, and the Trust Index read-back.
  • examples/openshell-smoke/trust-index-release.env (new): agent-trust-discovery v0.1.4 pinned by SHA-256 per platform, copied from the release's checksums.txt.
  • examples/openshell-smoke/smoke.sh: the flag (which refuses ANS_SMOKE_LOG_LEVEL=debug, because the Docker driver forwards the level to the supervisor whose console queue is lossy and carries the records the run counts), hook calls, a per-run partial file in the generalized fetch_release_asset (closes a latent race between two cold-cache runs), a request journal written by sandbox_curl, fail tailing .jsonl evidence too, and a failure-path cleanup that keeps only logs/. With the flag off the rendered configuration and the five checks are the same as before.
  • Makefile: the new target.
  • .github/workflows/openshell-smoke.yml: a two-leg matrix with per-leg cache keys and log levels (the trust-index leg runs at info because the supervisor's console queue is lossy and shared with the records the leg counts), and the path filter gains the hydrator command, pkg/, and the signals file.
  • Docs: a "Scoring the run's behavior" section, pin table row and pin-update steps in the smoke README; one sentence in the root README (and --yes on the settings set example, which the CLI requires off a TTY); the hydrator doc's rollout bullet now says what the target exercises and what it still does not (stale-score retention, whole-batch rejection).

The per-run admin key reaches the Trust Index through its environment and the hydrator and curl through files, so it is never on a command line or in a retained log; on failure the key, header file and database are removed with the other secrets.

Testing

  • make openshell-smoke on macOS (Rancher Desktop): ALL PASS 5 checks against OpenShell v0.1.7 (5d25211); --print-config output identical to main apart from per-run ports and paths.
  • make openshell-smoke-trust-index on the same daemon, three times: ALL PASS 10 checks against OpenShell v0.1.7 (5d25211) and agent-trust-discovery v0.1.4 (27a30a1); the hydrator scored 75 from 3 allowed and 1 denied decision; no containers, volumes, processes or temp directories left.
  • Two concurrent runs under the flag on one daemon: both pass all ten checks, each attributing records to its own sandbox id.
  • Fault runs: a wrong hydrator admin key fails check 9 with the Trust Index's 401 problem in hydrator.log, only logs/ remains under the run directory, and no retained file contains either key; export left disabled fails check 6 after 60 s with the settings dump, toggle count and log line count; --with-trust-index with ANS_SMOKE_LOG_LEVEL=debug is refused before anything starts.
  • shellcheck -x on both shell files, actionlint on the workflow, go build, go vet, go test -race ./..., golangci-lint (0 issues). No Go code changed.

AI assistance

Assisted-by: Claude Code (claude-fable-5-1)

Add make openshell-smoke-trust-index, the OpenShell smoke run plus five
checks that take the run's policy decisions to a Trust Index score: the
gateway's OCSF JSON export is switched on, the approved sandbox's
supervisor records are collected, ans-ocsf-hydrator scores them and
imports the score into a pinned agent-trust-discovery release the
harness starts on a loopback port, and the score is read back from the
agent's trust evaluation at weight 0. make openshell-smoke is unchanged.

The behavior path lives in trust-index.sh, sourced by smoke.sh only
under --with-trust-index. The Trust Index loads the signal registration
this repository ships for operators and a profile naming only that
signal. The per-run admin key travels through the server's environment
and files for the hydrator and curl, never a command line or a retained
log. The decision count the assertions expect comes from sandbox_curl,
so it cannot drift from the checks that run. Release downloads use a
per-run partial file, which also closes a latent race between two
cold-cache OpenShell runs.

CI runs both targets as matrix legs with per-leg cache keys; the
trust-index leg runs the gateway at info because the supervisor's lossy
console queue is shared with the records it counts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: kperry <kperry@godaddy.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant