Skip to content

Automated Sync from main to stable - #40

Open
rhods-devops-app[bot] wants to merge 411 commits into
stablefrom
main
Open

rhods-devops-app[bot] wants to merge 411 commits into
stablefrom
main

Conversation

@rhods-devops-app

@rhods-devops-app rhods-devops-app Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Automated Sync from main to stable

This PR automatically syncs the main branch to the stable branch by opening a pull request from main into stable.

Sync Summary

  • Latest commit: f884776d Merge remote-tracking branch 'upstream/main'

  • Total commits to sync: 411

  • Source: https://github.com/red-hat-data-services/OpenShell.git @ main

  • Target: https://github.com/red-hat-data-services/OpenShell.git @ stable

  • PR head: main

Commits to be synced

Merging

GitHub automerge is enabled for this pull request once required checks pass.

moulalis and others added 30 commits October 21, 2026 20:09
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
The example configuration left the dashboard on its chart's default image,
the upstream publisher's personal quay.io repository, although Konflux
publishes a public ODH build. It now sets dashboard.images.dashboard to
quay.io/opendatahub/odh-openshell-dashboard, pinned by digest. The chart
reference stays on its upstream publisher until an ODH chart registry
exists (RHOAIENG-96944).

The installer gave the dashboard the admin role but not the user role.
Dashboard builds since 2026-10-19 show an entitlement page to anyone
without one of the two roles and default the user role to "user", so a
standard user under any other role name was refused. The dashboard now
gets oidc.userRole from user_role, as the gateway does. Charts that
predate the value ignore it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Gage Krumbach <gkrumbac@redhat.com>
The example left the gateway, supervisor and sandbox runtime on the
gateway chart's defaults, the upstream publisher's images, while the
README asked for ODH builds without showing them. The example now names
quay.io/opendatahub/odh-openshell-{gateway,supervisor,sandbox} with
digest placeholders, and the README says how to pair a rhaiv tag with
the gateway chart. The default sandbox workload image stays on the
chart default: ODH publishes no replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Gage Krumbach <gkrumbac@redhat.com>
…4724)

* test(testkit): add the testkit workspace for end-to-end suites

Add tests/testkit as its own Cargo workspace: testkit, testkit-macros,
and testkit-spec. testkit reads a target file, connects to the gateway
through openshell-sdk, creates and cleans up sandboxes, and runs the
installed CLI. #[testkit::test] requires a stable ID and a doc comment and
declares capability requirements. testkit-ready waits for a target's
gateway and CLI before tests run.

Nothing uses testkit yet. The lockfile is seeded from the root lockfile
so shared crates resolve to the same versions.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): format, lint and test testkit in Branch Checks

Nothing else builds testkit, so check it in Branch Checks and the
matching mise tasks to catch openshell-sdk changes that break it.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(tests): describe how to write and run testkit tests

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): give each sandbox its own cleanup timeout

Cleanup shared one two-minute budget across all of a test's sandboxes, so
a slow delete left the rest with no time and could leak them on a shared
gateway. Delete them concurrently, each with the full timeout.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testkit): show how to exclude tests by capability

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs: list tests/testkit in the project structure tables

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(testkit): drop test IDs and capabilities for now

Nothing reads test IDs or capability requirements until a runner, waivers,
or result history exist, so remove the id and requires arguments and the
target file's capabilities section. Both can return as optional attribute
arguments without changing existing tests.

With them gone, testkit-spec only checked the function shape and read the
doc comment, so fold it into testkit-macros.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testkit): describe what testkit provides today

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(testkit): find the gateway the way the CLI does

Replace the target file (OPENSHELL_TEST_TARGET) with the CLI's own gateway
selection: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the
active gateway. A registered gateway's endpoint, auth mode, and mTLS
material come from its CLI config directory. cx.cli runs OPENSHELL_BIN
against the same gateway and workspace as the SDK.

Suites now run unchanged under e2e/with-docker-gateway.sh and
e2e/with-podman-gateway.sh, and environments need no extra file.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): keep labels when creating a sandbox with a policy

The policy path creates the sandbox through gRPC and dropped the spec's
labels without an error. Pass them in the create request.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): use the registered gateway's credentials for an endpoint

When OPENSHELL_GATEWAY_ENDPOINT is set, the CLI takes credentials from the
gateway named by OPENSHELL_GATEWAY, or else the registered gateway with
that endpoint. testkit connected without a client certificate instead, so
an mTLS endpoint such as the one e2e/rust/e2e-vm.sh exports worked for
cx.cli but not for the SDK. Resolve it the way the CLI does, and pass
both the name and the endpoint to cx.cli.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(tests): list testkit's gateway variables in precedence order

OPENSHELL_GATEWAY_ENDPOINT takes precedence over OPENSHELL_GATEWAY. Also
mention OPENSHELL_BIN.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): apply other attributes to the generated test

#[testkit::test] left attributes such as #[ignore] and #[cfg] on the
nested body function, where they have no effect on the test. Move them
to the generated test, and reject a second test attribute such as

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
#[test] or #[tokio::test].

* fix(testkit): show usage for testkit-ready --help

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(testkit): cover cleanup after pass, failure, and panic

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): reject #[should_panic] on testkit tests

A testkit test panics on any failure, including an unreachable gateway or
a failed cleanup, so #[should_panic] would let such a failure pass. Reject
it and point to checking the expected failure in the body.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(testkit): keep only the macro's misuse checks as unit tests

Suites built on testkit exercise gateway resolution, the CLI wrapper,
polling, errors, and cleanup against a real gateway. Keep one unit test
for the macro's compile errors, which a passing suite never reaches, and
drop the test-only seams the other tests needed.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(testkit): connect through the SDK's mTLS support

The SDK's ClientConfig now takes a client identity, so testkit no longer builds
its own channel. It still resolves the registered gateway and reads its
credential files, then hands the CA and client certificate to
OpenShellClient::connect. That keeps TLS behavior, connect timeouts and
keep-alive settings in the SDK, and drops testkit's direct tonic dependency.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): bound readiness and cleanup with timeouts

wait_ready checked elapsed time only after a health call returned, and cleanup
started its timeout only after the delete returned, so a gateway that accepts a
connection but never answers could hang either one. Wrap each whole operation in
tokio::time::timeout: the readiness wait, including every connect and health
call, and each sandbox's delete plus wait for deletion. A test covers a peer
that accepts connections and never replies.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
)

The alias `openshell-prover` has been granted for OpenShell:

https://forum.snapcraft.io/t/alias-request-for-openshell/53658

Thus, we can remove guidance differentiating `openshell.prover` for the
snap from `openshell-prover` for other packaging formats. It is still
worth noting that installing a locally-built snap directly, rather than
from the store, means there is no alias.

Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
… call (NVIDIA#4582)

A registry-sourced sandbox boot authenticated with the registry, and for a
tag resolved the manifest, before it looked at the root disk cache. It did
so on every boot, even when the image reference carries a digest and the
root disk for that digest is already cached. Each boot then waits on the
registry, and a registry outage stops every boot of an image the host
already has.

A digest names the image's content, and since NVIDIA#4342 a cached root disk is
published only after its manifest was verified against that digest. On a
cache hit for a digest-pinned reference the registry cannot add anything,
so return the hit before authenticating. Authentication and the pull stay
on the miss path, and a tag still resolves through the registry. The hit
emits the same Pulling, CacheHit and Pulled events as before.

Behaviour change: a host whose registry credentials were revoked still
boots digest-pinned images it has already cached, as a kubelet does with
IfNotPresent and a digest-pinned image.

Test: bootstrap_registry_cache_hit_for_a_digest_pin_makes_no_registry_request
covers a miss (authenticates and fetches), a cached digest pin (no request
at all) and a tag for the same content (still resolved). It fails without
this change with ["GET /v2/"].

Closes NVIDIA#4581

Signed-off-by: Balaji Ganesan <bganesan@nvidia.com>
* refactor(ocsf): attribute events to session-bound contexts

A supervisor process that hosts several sandboxes cannot use one
process-wide OCSF context: every event would carry the wrong sandbox
identity, and middleware requests read their sandbox identity from the
same context.

Builders now accept a ContextRef, either a borrowed context or a context
shared by a hosted session, so per-session contexts need not be leaked.
ctx() returns the context bound to the calling thread when present and
falls back to the process-wide context otherwise. OCSF JSON output
settings can likewise be bound per thread, and OpenTelemetry can parent
a span from an explicit W3C trace context instead of process
environment. A dedicated supervisor keeps using the process-wide
context.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): scope gateway credentials and network state to a session

The supervisor kept its gateway bearer, Sandbox Protocol bearer, and
renewal task in process-wide slots, and the network supervisor read
provider token grants, proxy CA files, and the gateway identity from
process environment. A second sandbox in the same process would have
replaced or rotated the first sandbox's credentials, reused its
provider access tokens, and exposed its denials through policy.local.

GatewayCredentials now holds one session's gateway and Sandbox Protocol
credentials and their renewal; every gateway client receives a
GatewayEndpoint that pairs the URL with them, including log push,
ConnectSupervisor and its redirects, and extension-credential rotation.
run_networking takes a NetworkSession with the sandbox identity, the
session's gateway access, its Workload API socket, its policy.local
denial source, and its proxy CA files. Provider token grants use a cache
and HTTP client owned by the session and fail closed without a Workload
API socket.

run_session supervises one generation from an explicit SessionConfig and
owner-provided SessionControls. run_sandbox_with_backend remains the
dedicated entry point: it keeps the process-wide OCSF context, reads the
same environment, binds readiness endpoints to session readiness, and
stops on SIGTERM or SIGINT. The binary shares one set of credentials
between log push and supervision, as before.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): host concurrent sessions on session-bound runtimes

SessionHost runs several supervisor sessions in one process. Each
session gets its own Tokio runtime, and every thread that runtime starts,
plus the thread driving it, is bound to the session's OCSF context, OCSF
JSON settings, log sink, and denial log. Tasks, blocking work,
reconnects, and credential renewal spawned by a session therefore keep
its identity, while threads outside any session carry none and push no
logs. Gateway credentials stay explicit per session.

Removing a session stops its workload gracefully, or cancels it at once
while it is still starting, then shuts down only its runtime and revokes
its gateway credentials. The key stays registered until the session
thread has finished, so the same generation cannot gain a second owner
while one is stopping. A panicking session reports a failure, clears its
readiness, and is revoked the same way. Hosted sessions get a private
proxy trust directory and serve policy.local denials from their own log.

A focused test runs two sessions concurrently and checks that nested and
blocking tasks see their own identity, gateway requests carry their own
bearer, log lines reach only their own stream, and removing one leaves
the other supervising.

Closes NVIDIA#4699

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): name hosted lifetimes sandbox instances

A sandbox instance is one running generation of a stable Sandbox
resource. Gateway transport connections and the gateway supervisor
sessions they carry can reconnect within one instance, so the hosted
object, its configuration, controls, host, and per-instance network,
denial, and token-grant state now use the sandbox instance name.

Rename session_host to sandbox_instance_host, run_session to
run_sandbox_instance, and SessionConfig, SessionControls, SessionHost,
and related hosted types to their SandboxInstance equivalents. Gateway
supervisor sessions, SSH sessions, and authentication session types
keep their names. No wire, configuration, or lifecycle behavior
changes.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): launch sandbox instances from one config and prepared bindings

Replace SandboxRunConfig and its field-by-field copy into
SandboxInstanceConfig with a single launch configuration. Live per-instance
state (gateway credentials, OCSF context and JSON settings, log sink, and
denial log) is now an explicit SandboxInstanceBindings value that the owner
prepares once, instead of placeholders the runner or host overwrote.

The dedicated binary loads launch inputs from its arguments and environment
and shares one GatewayCredentials with log push; run_sandbox_with_backend
keeps the process-wide OCSF context, health endpoints, and signal handling.
The instance host binds the same SandboxInstanceBindings to its runtime
threads and revokes their credentials when the instance ends.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): update Windows host proxy test config

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
…DIA#4063)

* fix(installer): resolve the gateway config dir like the CLI does

install.sh hardcoded $TARGET_HOME/.config/openshell in two places: the
readiness check, which reads the client certificates it expects the
gateway registration to have written, and the cleanup that removes a
stale openshell entry before re-adding it. The CLI instead keeps both
under $XDG_CONFIG_HOME/openshell whenever that variable is set, so with
the variable pointing anywhere other than ~/.config the installer and
the CLI looked in different directories: the readiness check waited for
certificates that were never going to appear there, and the cleanup left
the real stale entry in place.

Resolve the directory through as_target_user instead, which answers from
the same environment the CLI is invoked in. That covers both branches
without duplicating their logic: the same-user branch inherits the
caller's XDG_CONFIG_HOME, while the sudo and runuser branches reset the
environment and fall back to the target user's home.

An empty XDG_CONFIG_HOME falls back to $HOME/.config, following the XDG
base directory specification. The CLI diverges here: xdg_config_dir
treats a set-but-empty value as set and joins onto it, yielding the
relative path "openshell", which no installer path can usefully match.
Worth fixing separately in the CLI.

Signed-off-by: fede-kamel <fkamelhar@gmail.com>

* fix(core): treat an empty XDG_CONFIG_HOME as unset

xdg_config_dir() joined onto an empty XDG_CONFIG_HOME and resolved the
config root to the relative path "openshell", while the installer falls
back to $HOME/.config/openshell for the same value. Registration runs the
CLI with the caller's environment, so the two could disagree on where the
gateway entry lives.

Treat an empty value as unset, as the XDG base directory spec requires,
so the CLI and installer resolve the same directory. Add tests for a value
with spaces and drive registration cleanup through the helper for the
spaces and empty cases.

Signed-off-by: fede-kamel <fkamelhar@gmail.com>

---------

Signed-off-by: fede-kamel <fkamelhar@gmail.com>
)

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Experience has shown that ordinary podman container creation can take
longer than 30 seconds for some users, so for now, this change doubles
the timeout, until the source of the slowness is found and accounted
for.

Signed-off-by: Simon Ruggier <Simon.Ruggier@gmail.com>
…#4549)

The provisioning-deadline scan takes each provisioning sandbox's local
S(global) S(workspace) X(sandbox) keys in ascending key order and waited
for them with no deadline. When a sandbox's key sorts before its
workspace's key, the scan held X(sandbox) while it waited behind a
provider or profile writer, so start, stop and delete for that sandbox
waited for the writer too. In the other order, the scan held
S(workspace) while it waited for a busy sandbox, and provider writers
on that workspace queued behind it. Either way the rest of the pass
waited as well.

Each pass now has a 500 ms budget for waiting on these keys. A sandbox
whose keys stay busy is skipped and picked up on the next pass a second
later. Once the budget is spent, the rest of the pass only takes keys
that are free when it reaches them, so a busy workspace or global key
costs one wait per pass rather than one per sandbox. The local
acquisition is cancel-safe, so giving up releases the keys it already
took.

Skips log at debug and count in the new
openshell_server_provisioning_deadline_skips_total counter, labeled by
whether the sandbox was already past its deadline. A steady rate of
delayed expiries can then be alerted on without a warning every pass.
The gateway metrics page and the cluster debugging skill describe it.

The test that pinned the old wait is replaced by tests for both key
orders: a lifecycle lock or a provider writer proceeds and the scan
finishes while the other side holds its key, the skipped sandbox is
untouched and expires once the writer releases, and a skip doesn't end
the pass for other sandboxes. Another test checks that sandboxes
behind one busy workspace share a single wait.

Part of NVIDIA#4307
Closes NVIDIA#4547

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(sandbox-backend): route runtime connections to shared supervisor sessions

A supervisor shared by several sandboxes needs one listener for every
session's runtime-initiated Sandbox Protocol connections. The runtime
now sends a fixed, bounded routing preface (magic, version, length, and
an opaque route handle) before its pinned TLS handshake when its
bootstrap names a route. RuntimeRouter treats the handle only as a hint
that selects which generation-pinned TLS identity to demand, closes
unknown, removed, and replaced handles before any handshake, bounds
unauthenticated handshakes globally and per route, and queues an
authenticated connection only for its own session, so no peer can
displace a live channel. A routed session waits up to the bootstrap
grace for its first connection, because it is registered before its
workload Pod is scheduled.

The session_control module defines the trusted contract between a
compute driver and a shared supervisor: registering one generation's
runtime descriptor and gateway-issued credentials, reading its status,
and removing it, over mutually authenticated TLS 1.3.

Dedicated descriptors and bootstrap listeners keep their encodings.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): host gateway-registered sessions in one process

openshell-supervisor experimental-host replaces the idle bootstrap
process with a session host. A mutually authenticated session-control
API accepts registrations only from the admission certificate holder,
the gateway. Each registration starts a SessionHost session that acts
only with the gateway credentials issued for its generation, keeps its
proxy CA, trust directory, and SSH socket under a private directory,
and receives its runtime connections through the routed listener.
Status reports each session's own readiness; the health socket keeps
reporting process health only.

Removing a session runs on a task the host owns, so it completes when
the caller disconnects, concurrent removers share its result, and the
route stays reserved until the session has ended and its files are
gone. An identical registration is accepted again and a different one
for a registered route is refused.

Sessions get no upstream proxy, extra CA bundle, or provider workload
identity; the Kubernetes driver refuses configurations that need them.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(server): issue a TLS-only private PKI from generate-certs

generate-certs --tls-only creates just a server and a client TLS Secret
from a fresh CA, with the same create-if-missing contract and a recovery
hint for partial state. The Helm certgen hook uses it for private PKIs
that must not carry a sandbox-JWT signing key.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(kubernetes): supervise gateway-topology generations as shared sessions

With experimental_supervisor_topology = "gateway", each new runtime
generation becomes a session on the gateway's shared supervisor instead
of a supervisor Pod and boundary Service of its own. The generation
records its placement and session route on its Sandbox resource when it
starts, and every later decision reads that record, so changing the
setting never reinterprets a running generation and unknown records fail
closed. The runtime identity ends with the session route, which no Pod
UID can equal, so Pod authentication cannot vouch for a shared
generation.

Create and start add a per-sandbox egress allowance to the Pods the
shared supervisor's Service selects, bind it into the boundary evidence,
and register the session over session control before releasing the
workload; registration does not wait for the runtime, so create still
returns before the gateway persists the generation. Readiness follows
the session's own status. Stop, restart, and delete remove only that
session, and reconciliation suspends a workload whose session is gone
and removes sessions that no current generation may run. Configurations
with an upstream proxy or provider SPIFFE identity are refused.

Closes NVIDIA#4700

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(helm): deploy and wire the shared supervisor

With the gateway topology enabled, the chart runs the session host as a
single-replica Deployment with its Service and a NetworkPolicy that
admits session control only from the gateway and runtime connections
only from sandbox workload Pods. The certgen hook issues a private
session-control PKI whose client credential only the gateway mounts,
the gateway may read only that Service, and the chart renders the
matching experimental_shared_supervisor table. The supervisor Pod has no
service account token. Settings the shared supervisor cannot isolate,
and a hand-written connection table, fail at render time.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(ocsf): attribute events to session-bound contexts

A supervisor process that hosts several sandboxes cannot use one
process-wide OCSF context: every event would carry the wrong sandbox
identity, and middleware requests read their sandbox identity from the
same context.

Builders now accept a ContextRef, either a borrowed context or a context
shared by a hosted session, so per-session contexts need not be leaked.
ctx() returns the context bound to the calling thread when present and
falls back to the process-wide context otherwise. OCSF JSON output
settings can likewise be bound per thread, and OpenTelemetry can parent
a span from an explicit W3C trace context instead of process
environment. A dedicated supervisor keeps using the process-wide
context.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): run two sandboxes on one shared Kubernetes supervisor

The e2e:kubernetes:shared-supervisor lane enables the gateway topology
and creates two sandboxes with different policies and provider
credentials. It checks that one shared supervisor process hosts both,
that each session enforces its own policy and injects its own
credential, that stopping one leaves the other working, that the routed
listener refuses stale and unknown routes and drops a peer that names a
live route without its pinned identity, and that a restart registers a
fresh session on the same process.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): scope gateway credentials and network state to a session

The supervisor kept its gateway bearer, Sandbox Protocol bearer, and
renewal task in process-wide slots, and the network supervisor read
provider token grants, proxy CA files, and the gateway identity from
process environment. A second sandbox in the same process would have
replaced or rotated the first sandbox's credentials, reused its
provider access tokens, and exposed its denials through policy.local.

GatewayCredentials now holds one session's gateway and Sandbox Protocol
credentials and their renewal; every gateway client receives a
GatewayEndpoint that pairs the URL with them, including log push,
ConnectSupervisor and its redirects, and extension-credential rotation.
run_networking takes a NetworkSession with the sandbox identity, the
session's gateway access, its Workload API socket, its policy.local
denial source, and its proxy CA files. Provider token grants use a cache
and HTTP client owned by the session and fail closed without a Workload
API socket.

run_session supervises one generation from an explicit SessionConfig and
owner-provided SessionControls. run_sandbox_with_backend remains the
dedicated entry point: it keeps the process-wide OCSF context, reads the
same environment, binds readiness endpoints to session readiness, and
stops on SIGTERM or SIGINT. The binary shares one set of credentials
between log push and supervision, as before.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: describe the experimental shared supervisor

Document the gateway supervisor topology, its chart-owned connection
settings and limits, and how to debug and test it.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): host concurrent sessions on session-bound runtimes

SessionHost runs several supervisor sessions in one process. Each
session gets its own Tokio runtime, and every thread that runtime starts,
plus the thread driving it, is bound to the session's OCSF context, OCSF
JSON settings, log sink, and denial log. Tasks, blocking work,
reconnects, and credential renewal spawned by a session therefore keep
its identity, while threads outside any session carry none and push no
logs. Gateway credentials stay explicit per session.

Removing a session stops its workload gracefully, or cancels it at once
while it is still starting, then shuts down only its runtime and revokes
its gateway credentials. The key stays registered until the session
thread has finished, so the same generation cannot gain a second owner
while one is stopping. A panicking session reports a failure, clears its
readiness, and is revoked the same way. Hosted sessions get a private
proxy trust directory and serve policy.local denials from their own log.

A focused test runs two sessions concurrently and checks that nested and
blocking tasks see their own identity, gateway requests carry their own
bearer, log lines reach only their own stream, and removing one leaves
the other supervising.

Closes NVIDIA#4699

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix: tighten shared supervisor naming, removal deadlines, and route probes

The shared supervisor's resource name appended its suffix before
truncating, so a release fullname of 51 or 52 characters produced the
gateway Service's own name. A fullname without room for the suffix now
keeps a prefix and a hash of the whole fullname.

Removing a session waits up to 20 s for its workload to stop and 5 s for
the session runtime to shut down, but the driver gave DELETE the same
15 s budget as other session-control calls and reported normal removals
as failures. The contract now defines the removal grace and a 45 s
removal deadline, and deleting a sandbox no longer waits on a session
whose workload is already going away.

The shared host runs no supervisor OTLP trace or OCSF exporter. The
driver warns at startup when the gateway configures that export, and the
docs state the limit. experimental-host keeps its Unix-only imports in
its Unix entry point.

The e2e route probe fails when the supervisor neither refuses nor
challenges a connection, or keeps a challenged connection open after a
malformed TLS answer, and it checks that every probe reached the
supervisor Pod.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): route shared supervisor crypto through openshell-crypto

Main now routes first-party TLS providers and digests through the
openshell-crypto facade. Build the session-control TLS configurations and
the routed-runtime test server from its configuration provider, install the
experimental host's default provider through it, and hash shared session
registrations with its SHA-256 instead of a direct sha2 dependency.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): report shared supervisor health over TCP

experimental-host takes --health-port instead of a Unix health socket,
matching experimental-bootstrap. It binds the wildcard health listener
after the session-control and runtime listeners and accepts and closes
connections while the process runs. The chart probes the named container
port health (5502) with native kubelet tcpSocket checks; session control
stays on 5501, the Service exposes only runtime and session control, and
/run/openshell remains for per-session files.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): probe shared supervisor health on session control

The shared supervisor's kubelet tcpSocket probes now connect to the
mTLS session-control listener (5501) instead of a separate health
listener. Drop the experimental-host --health-port flag, its accept
loop, and the chart's 5502 container port. Probes open and close a
TCP connection; every session-control request still requires mTLS.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): use shared mode flags for the session host

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): describe the shared host module by its mode

Drop the experimental label from the shared host module documentation
so it matches the --mode=shared entry point. No behavior changes.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(test): preserve shared supervisor workspace overlays

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): name hosted lifetimes sandbox instances

A sandbox instance is one running generation of a stable Sandbox
resource. Gateway transport connections and the gateway supervisor
sessions they carry can reconnect within one instance, so the hosted
object, its configuration, controls, host, and per-instance network,
denial, and token-grant state now use the sandbox instance name.

Rename session_host to sandbox_instance_host, run_session to
run_sandbox_instance, and SessionConfig, SessionControls, SessionHost,
and related hosted types to their SandboxInstance equivalents. Gateway
supervisor sessions, SSH sessions, and authentication session types
keep their names. No wire, configuration, or lifecycle behavior
changes.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): launch sandbox instances from one config and prepared bindings

Replace SandboxRunConfig and its field-by-field copy into
SandboxInstanceConfig with a single launch configuration. Live per-instance
state (gateway credentials, OCSF context and JSON settings, log sink, and
denial log) is now an explicit SandboxInstanceBindings value that the owner
prepares once, instead of placeholders the runner or host overwrote.

The dedicated binary loads launch inputs from its arguments and environment
and shares one GatewayCredentials with log push; run_sandbox_with_backend
keeps the process-wide OCSF context, health endpoints, and signal handling.
The instance host binds the same SandboxInstanceBindings to its runtime
threads and revokes their credentials when the instance ends.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): use session-control wire types for instance status and trace context

Drop the supervisor's duplicate SandboxInstancePhase, SandboxInstanceStatus,
and TraceContext families. The sandbox instance host reports
SandboxInstanceStatusReport directly, with its key as the route, so the
shared host no longer converts or re-keys status, lists, or removal
results. SandboxInstanceConfig carries RegistrationTraceContext and the
dedicated process reads it through trace_context_from_environment. No wire
definitions change.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): update Windows host proxy test config

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): assign shared sandbox instances over a supervisor-initiated gRPC stream

Replace the private HTTP/JSON session-control API, which the gateway dialed
on the shared supervisor, with a supervisor-initiated stream to the gateway.
The gateway owns the desired set of sandbox instances; the shared supervisor
attaches with a gateway-scoped host certificate, receives complete assignment
snapshots with session-scoped revisions, reconciles its instance registry to
each one, and reports what it observes. Losing the stream never stops an
instance; only a later snapshot that omits it does.

- Add the private openshell.supervisor.host.v1 protocol, served on the gateway
  listener but authorized only for the supervisor host certificate, excluded
  from SDK generation and reflection, and classified in the schema inventory.
- Add SupervisorHostController in openshell-core: uninitialized until the
  driver derives the desired set once, compare-and-replace full resyncs,
  capacity bound, session fencing by connection epoch, revision-bounded acks,
  bounded sends.
- Classify TLS peers per connection against the ordinary client CA and the
  supervisor host CA from one configuration snapshot; a chain matching
  neither or both is refused, and host connections never become users.
- Persist each shared generation's launch material in its supervisor Secret so
  a restarted gateway rebuilds assignments from Sandbox resources, and
  withdraw instances on stop, delete, and suspend.
- Issue the host identity with `generate-certs --client-only`; the gateway
  mounts only the CA. Drop the control port, server certificate, and gateway
  ingress rule from the chart; require gateway TLS for the topology.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): authenticate the shared supervisor stream with a gateway-minted supervisor token

Replace the shared supervisor's host CA and client certificate with a
supervisor token: a bearer the gateway mints from its existing launch-signing
key under a profile of its own (typ openshell-supervisor+jwt, subject
openshell-supervisor:<gateway_id>, no sandbox, generation, or epoch claims).
The supervisor dials the gateway's Attach stream over ordinary
server-authenticated TLS and presents the token; the gateway verifies it
with the same key infrastructure, grants only the assignment stream, refuses
every other credential on it, and ends a stream when the token that opened
it expires.

The gateway renews the token at half its lifetime and publishes each renewal
into the supervisor host controller; the Kubernetes driver's only part in
authentication is delivery, replacing the token in the chart-created
<release>-supervisor-token Secret, which the supervisor mounts and re-reads
on every connection attempt. The gateway Role may get and patch only that
Secret.

Remove the host-only TLS realm classification, the supervisor host CA mount
and config, certgen --client-only, and the supervisor host PKI; gateway TLS
returns to its pre-feature behavior and remains required for the topology.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(helm): remove redundant certgen condition

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): extract gateway-owned shared supervision into openshell-supervisor-control

Move the shared supervisor control types out of openshell-core and
openshell-sandbox-backend into a new openshell-supervisor-control crate:
the SupervisorHostController (desired set, revision fencing, host
sessions, observations), the SandboxInstanceRegistration launch material
and SandboxInstanceStatusReport mapping, and a new SharedSupervision
coordinator that owns the control contracts a compute driver previously
implemented itself: validated persist-before-assign launches, recovery
from durable records with the compare-and-replace baseline captured before
the driver lists them, keep-previous-set on any failed or mismatched read,
and one interpretation of host observations for readiness and bootstrap.
Supervisor token delivery keeps its retry and newer-value policy in the
crate; the write itself stays with the driver.

The gateway now constructs the coordination: ComputeDriverBuildContext
exposes shared_supervision() and the server keeps the controller for the
Attach stream and token minting, replacing the driver-created controller
that was installed into the gateway. The Kubernetes factory injects it
through KubernetesComputeDriver::with_shared_supervision; new() stays the
dedicated-topology constructor and refuses a configuration that selects
the gateway topology without the gateway's coordination.

The Kubernetes driver keeps its platform work: placement annotations, the
shared supervisor Service and egress NetworkPolicy, launch Secret create
and read (bytes only), its own fence and Pod evidence, and the token
Secret patch. The supervisor process consumes the moved registration and
status types unchanged. Wire protocol, storage JSON, annotations, and the
supervisor token profile are unchanged; the private protobuf stays
generated in openshell-core and is re-exported by the new crate.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
…ged-workspace-scc-isolation

CARRY: test(odh): qualify managed workspace SCC identity isolation
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
… docker tag to v9.8-1793614932 (opendatahub-io#155)

Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
moulalis and others added 4 commits November 5, 2026 08:08
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
konflux-internal-p02 Bot and others added 2 commits November 6, 2026 09:16
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
@alexxfan

alexxfan commented Nov 6, 2026

Copy link
Copy Markdown

/retest

Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
konflux-internal-p02 Bot and others added 5 commits November 8, 2026 01:19
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
…-rhel9:1.99.1 docker digest to cf69dc5 (opendatahub-io#169)

Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
… tag to v9.8-1793861509 (opendatahub-io#168)

Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
konflux-internal-p02 Bot and others added 5 commits November 9, 2026 01:36
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
…r digest to ddaf8f8 (#108)

Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
… docker tag to v9.8-1794184566 (#109)

Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Carry the downstream OCP Python gateway E2E integration and its CI coverage as one rebase-friendly change.

Signed-off-by: Gaizka Menendez <gmenende@redhat.com>
…-ocp-e2e/gmenher

CARRY: test(odh): run supported Python gateway e2e on OpenShift
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment