Skip to content

[ENG-8443] feature: prime factory k8s login and the k8s credential plugin - #808

Open
JannikSt wants to merge 18 commits into
mainfrom
feature/cluster-login-kubeconfig
Open

JannikSt wants to merge 18 commits into
mainfrom
feature/cluster-login-kubeconfig

Conversation

@JannikSt

@JannikSt JannikSt commented Jul 25, 2026 •

Copy link
Copy Markdown
Member

Targeting main because this repo does not use the develop-branch convention — it has no develop branch and merges PRs straight into main. (The platform-side PRs for this feature target develop as usual.)

Depends on platform#7204 (per-pool Tailscale API proxy, plus the per-grant server in kubeconfig-info, implemented alongside platform#6878). Against a platform without it, prime factory k8s login skips every pool and exits 1 without writing a kubeconfig.

Adds the researcher side of PrimeCluster node pools: a kubeconfig, and the credential plugin behind it.

  • prime factory k8s list shows the reservations the researcher can log in to, and the next step for each
  • prime factory k8s login <cluster> [--pool <pool>] writes ~/.prime/kube/<cluster>.yaml (mode 0600) with one context per pool the researcher has been granted, each pointing at that pool's API proxy, and an exec block instead of a token. The kubeconfig holds no credential
  • prime auth k8s-token is the credential plugin kubectl runs once per process, so once per kubectl command. It exchanges the platform API token for a short-lived ServiceAccount token scoped to the researcher's namespace, and caches it privately until shortly before it expires (see Token cache)

That indirection is what makes revocation mean something: revoking a grant deletes the namespace and its ServiceAccount, so the next refresh fails and any token already issued stops working.

The plugin has a stricter contract than the rest of the CLI, because kubectl is the caller rather than a human:

  • stdout is protocol. Only the ExecCredential goes there; every message goes to stderr
  • Exit codes separate transient from permanent: 1 unreachable or temporarily unavailable (503), 2 revoked/forbidden, 3 rate limited, 4 platform auth expired, 5 ambiguous pool
  • Never prompts. The kubeconfig sets interactiveMode: Never
  • Pinned to the login's platform. The exec block carries --context <ctx> (when one was active) and PRIME_API_BASE_URL=<login-time base URL>, plus PRIME_DISABLE_VERSION_CHECK=1 so kubectl output isn't prefixed by an update banner on every call

It talks to the platform with httpx directly rather than the shared APIClient, because APIClient collapses every status into one APIError and the failure contract here depends on the status code.

A Typer app holding exactly one command promotes it to the group, which would make prime auth k8s-token (the exact string in every kubeconfig) fail. Both groups carry an explicit callback, and root-level tests cover the wiring.

Usage

$ prime factory k8s list
                                                Kubernetes reservations
┏━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Cluster    ┃ Pool  ┃ Namespace                   ┃ Status       ┃ Proxy     ┃ Next step                              ┃
┡━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ my-cluster │ eval  │ prime-user-cm1abc-k8s-eval  │ Ready        │ ready     │ prime factory k8s login my-cluster     │
│ My Cluster │       │                             │              │           │ --pool eval                            │
├────────────┼───────┼─────────────────────────────┼──────────────┼───────────┼────────────────────────────────────────┤
│ my-cluster │ train │ prime-user-cm1abc-k8s-train │ Ready        │ ready     │ prime factory k8s login my-cluster     │
│ My Cluster │       │                             │              │           │ --pool train                           │
├────────────┼───────┼─────────────────────────────┼──────────────┼───────────┼────────────────────────────────────────┤
│ my-cluster │ debug │ prime-user-cm1abc-k8s-debug │ Provisioning │ not ready │ This reservation's API proxy has not   │
│ My Cluster │       │                             │              │           │ joined your tailnet yet; retry in a    │
│            │       │                             │              │           │ minute.                                │
└────────────┴───────┴─────────────────────────────┴──────────────┴───────────┴────────────────────────────────────────┘

$ prime factory k8s list            # no reservations
You're not a member of any Kubernetes reservation. Ask your team admin to add you, or reserve nodes from the dashboard.

$ prime factory k8s list -o json    # the platform's GET /api/v1/clusters shape, below
$ prime factory k8s login my-cluster --pool train
Skipping pool 'debug': This reservation's API proxy has not joined your tailnet yet; retry in a minute.
Wrote kubeconfig to ~/.prime/kube/my-cluster.yaml
Current context: my-cluster-train
Other contexts: my-cluster-eval
Switch with: kubectl config use-context my-cluster-eval

Use it with:
  export KUBECONFIG=~/.prime/kube/my-cluster.yaml
  kubectl get pods
Can't resolve eval-0a1b.tail1234.ts.net. Make sure Tailscale is connected to your team's tailnet (the one the
reservation's proxy joined), then retry.

$ prime factory k8s login my-cluster --pool nope
Skipping pool 'debug': This reservation's API proxy has not joined your tailnet yet; retry in a minute.
Pool 'nope' on 'my-cluster' is not one of your reservations there.
Available pools: eval, train

$ prime factory k8s login my-cluster    # after access was removed
You are not a member of any Kubernetes reservation on 'my-cluster'. Ask your team admin to add you, or reserve nodes
from the dashboard.
Removed the stale kubeconfig ~/.prime/kube/my-cluster.yaml.
Run `prime factory k8s list` to see the reservations you can log in to.
  • prime factory k8s list [-o table|json]: one row per reservation (pool), with namespace, status (Ready / Provisioning / Releasing / Failed), whether its proxy is up, and either the login command (with --pool when the cluster has more than one ready pool) or why it is not ready
  • prime factory k8s login <cluster> [--pool <pool>]: writes every reachable pool's context, as before. Without --pool the first pool (alphabetically) is current; the output names the current context and how to switch. With --pool, that pool's context is current; a pool that is unknown or not reachable is an error listing the available pools, and nothing is written
  • No grant (403 from kubeconfig-info, which the platform also answers for an unknown cluster): the platform's message is shown, ~/.prime/kube/<cluster>.yaml is removed if present (its cached tokens were already cleared), and the researcher is pointed at prime factory k8s list
  • No pool reachable: exits 1 and keeps the existing file, as before, but now says its contexts may be stale
  • Tailscale check: after writing, each context's proxy host name is resolved in parallel with a 2s budget in total (daemon threads, so a hung resolver never delays exit). A name that does not resolve prints a warning; the login still succeeds. IP literals are not looked up

Platform contract (platform#6878 / platform#7204 / platform#8027)

GET /api/v1/clusters (clusters:read), the caller's own reservations, from the platform's stored state:

{
  "clusters": [
    {
      "cluster": "my-cluster",
      "displayName": "My Cluster",
      "reservations": [
        {"pool": "train", "namespace": "prime-user-cm1abc-k8s-train", "teamId": "team_1", "status": "Ready", "proxyReady": true, "server": "https://train-0a1b.tail1234.ts.net", "unavailableReason": null},
        {"pool": "debug", "namespace": "prime-user-cm1abc-k8s-debug", "teamId": "team_1", "status": "Provisioning", "proxyReady": false, "server": null, "unavailableReason": "This reservation's API proxy has not joined your tailnet yet; retry in a minute."}
      ]
    }
  ]
}

The CLI reads cluster, displayName, and per reservation pool, namespace, status, proxyReady, server (non-null means login will write a context) and unavailableReason. JSON output passes the shape through.

GET /api/v1/clusters/{name}/kubeconfig-info:

{
  "cluster": "mf-a",
  "certificateAuthorityData": "<base64 PEM>",
  "grants": [
    {"pool": "train", "namespace": "pool-train-u123", "server": "https://k8s-train-xyz.tail1234.ts.net", "tlsServerName": "kubernetes.default.svc", "unavailableReason": null},
    {"pool": "eval", "namespace": "pool-eval-u123", "server": null, "tlsServerName": null, "unavailableReason": "This reservation's API proxy has not joined your tailnet yet; try again shortly."}
  ]
}

How the CLI reads it:

  • A grant is written only if it has pool, namespace and a non-empty server. Otherwise it is skipped and its unavailableReason is printed (Skipping pool 'eval': …), with a generic reason when that is null
  • tlsServerName is written as tls-server-name when non-null and omitted when null
  • There is no top-level server. If one is present (older platform), it is ignored and never written. apiUrl is never read either
  • Missing certificateAuthorityData, or an empty grants list, exits 1 without writing
  • 403 means no live grant (or an unknown cluster, answered identically): detail is shown without the HTTP 403: prefix and the cluster's kubeconfig is removed. Other errors leave the file alone

POST /api/v1/clusters/{name}/kube-token[?pool=] returns an ExecCredential (client.authentication.k8s.io/v1), unchanged. Status mapping:

Status Meaning Exit stderr
401 platform auth invalid 4 "Run prime login"
403 no live grant / revoked / not a team member 2 platform detail (generic fallback)
409 ambiguous pool 5 platform detail
429 rate limited 3 Retry-After when present
503 cluster not ready / stale heartbeat / capability withdrawn 1 platform detail (generic fallback)
other 5xx, connection error, timeout transient 1 generic

Token cache

Why. client-go caches an exec credential only inside one process, so every kubectl command ran the plugin and minted a fresh 1h token. The platform allows 60 mints per grant and 300 per user per hour, and a researcher polling kubectl get pod every few seconds locked themselves out within minutes (Too many credential requests. Retry after 2579s, exit 3).

What. prime auth k8s-token caches the ExecCredential at ~/.prime/kube/cache/<sha256(cluster)[:16]>/<pbkdf2(identity)[:32]>.json. Directories are 0700 and files 0600, written to a temp file and renamed into place. The token is never logged.

  • Key. The identity is the effective platform base URL (the PRIME_API_BASE_URL the kubeconfig pins, or whatever --context resolves to), the cluster, the pool (null when there is no --pool), and the API key. All of it is hashed into the file name with PBKDF2-SHA256 (100k rounds, a few ms per call). PBKDF2 rather than plain SHA-256 because CodeQL flags a fast hash over the key; the key is high-entropy and sits in the clear in ~/.prime/config.json anyway. The key is never on disk in the cache, and switching platform, key or user never reuses another identity's token
  • Margin: 5 minutes. A cached token is served only while its status.expirationTimestamp is more than 5 minutes away. client-go uses a credential up to its exact expiry, so the margin guarantees each kubectl command a token with at least 5 minutes left (enough for normal commands, and it absorbs clock skew against the apiserver). Against a 1h token it costs about one extra mint per pool per hour. A credential with no parseable, timezone-aware expiry is never cached
  • Misses. An entry that is missing, unreadable or corrupt counts as a miss and is overwritten. If writing the cache fails, the plugin still prints the credential and exits 0, with a warning on stderr
  • Invalidation. prime factory k8s login <cluster> deletes every cached token for that cluster first, whatever platform or key it belongs to, even if the login then fails. Re-login is the documented step after a grant changes (a member removed and re-added gets a new namespace and ServiceAccount). A 401, 403 or 409 from kube-token deletes the entry for that key. A 429, 503 or transport error leaves it in place
  • Escape hatch. PRIME_K8S_TOKEN_NO_CACHE=1 (also true/yes/on), or --no-cache, mints on every call and neither reads nor writes the cache. The env var reaches the plugin through kubectl's environment, so it needs no kubeconfig edit. Existing kubeconfig exec args are unchanged and keep working
  • Contract unchanged. stdout carries only the ExecCredential, messages go to stderr, and exit codes are the same

A token rejected before it expires (the 401 case). Revoking a grant deletes the namespace and ServiceAccount, so a cached token stops working server-side. The apiserver caches successful token reviews for 10s, so a token used just before revocation can keep working for up to 10s more. From client-go (plugin/pkg/client/auth/exec/exec.go, checked against v0.37.2): when a request gets a 401, RoundTrip calls maybeRefreshCreds. That re-runs the plugin once (and only if no other request has already refreshed the credential) to refresh the in-process credential for the next request. The failed request is returned as-is and not retried. So there is no tight loop: at most one plugin run per failed request, and kubectl gives up after its own few requests.

Serving the same cached token on that re-run would just repeat the 401 and leave the researcher with a bare You must be logged in to the server. The plugin therefore notes which parent process each entry was last served to. A second run from the same parent while the entry is still fresh can only follow a 401, because client-go re-runs early for no other reason. The plugin then skips the entry and mints. For a revoked grant the mint returns 403, which evicts the entry, exits 2 and shows the platform's reason through kubectl. For a grant whose ServiceAccount was recreated, the fresh token just works. This happens at most once per parent process per entry, so a client that re-runs the plugin without caching (e.g. a Python script reloading its kubeconfig in a loop) costs one extra mint, not one per call. Where the parent is not kubectl (e.g. a launcher process on Windows), the check never fires and behaviour falls back to the cache plus prime factory k8s login. In the kind e2e, kubectl 1.33 shows the 403 reason on the first command after revocation and makes 5 refused mint attempts in that command, the same number of requests it made before the cache existed. With the parent check disabled, the same test shows only You must be logged in to the server.

Tailnet-only access

Researchers reach a cluster's apiserver only through the per-pool Tailscale API proxy (platform#7204), never through the cluster's public endpoint or the ops-tailnet apiUrl. Each pool has its own proxy device on the customer's tailnet, so there is no cluster-level server: every grant carries its own server (the proxy's MagicDNS URL).

The kubeconfig therefore has one cluster entry per reachable grant, named <cluster>-<pool> like its context and user. Each entry has that grant's server, the cluster's shared certificate-authority-data, and tls-server-name: kubernetes.default.svc. The proxy passes the apiserver's TLS through rather than terminating it, so the certificate kubectl sees is the apiserver's, which covers in-cluster names and not the proxy's MagicDNS name. Without tls-server-name, verification fails.

Login behaviour:

  • Re-login regenerates the file from scratch, so contexts for grants that are gone or now unavailable are dropped
  • If no grant is reachable, login exits 1 and leaves any existing kubeconfig untouched instead of replacing it with an empty one, and says its contexts may be stale
  • If the platform answers 403 (no grant left on the cluster), the kubeconfig is removed
  • The file is written to a 0600 temp file beside the target, fsynced, and atomically renamed, so kubectl never sees a half-written file and a failed write keeps the previous one

Changes in this update (handover: Jannik → minh)

  • Merged current origin/main (no conflicts)
  • kubeconfig-info moved to per-grant server / tlsServerName / unavailableReason; one cluster entry per pool; the top-level server is ignored
  • Unavailable grants are skipped with their reason; no reachable grant exits 1 without writing
  • Atomic 0600 write (previously write-then-chmod)
  • kube-token 503 maps to exit 1 with the platform detail on stderr
  • The e2e stub returns the new shape, includes a legacy top-level server pointing at a closed port and an unavailable grant, and puts a TCP pass-through "API proxy" per pool in front of the apiserver

Changes in this update (token cache)

  • Merged current origin/main (no conflicts)
  • e178e0d8: new prime_cli/utils/kube_token_cache.py. prime auth k8s-token reads, writes and evicts the cache, and --no-cache / PRIME_K8S_TOKEN_NO_CACHE bypass it. prime factory k8s login clears the cluster's entries before anything else
  • Unit tests for every cache property listed above. The e2e stub now counts mints, and the failure-mode e2e steps use the escape hatch so the platform answers them
  • 36e1a697: the cache file name is derived with PBKDF2 instead of SHA-256, to clear CodeQL's py/weak-sensitive-data-hashing

Changes in this update (researcher UX)

  • 1c0b5011: prime factory k8s list (table and -o json, with an empty-state hint), login --pool, the current-context and switch hint, stale-kubeconfig removal on 403, the "may be stale" note when no pool is reachable, the 403 message plus a pointer to prime factory k8s list, and the post-login Tailscale DNS warning. Needs platform#8027 at 5ad86fb211 or later for GET /api/v1/clusters and the new 403 wording (older platforms answer factory k8s list with an error; everything else works as before)
  • Unit tests for each of these (list table/json/empty/error, --pool set/unknown/unreachable, stale removal on 403 and not on other errors, the "may be stale" note, the DNS warning with resolution mocked, the 2s budget across all names, IP literals skipped). An autouse fixture stubs resolution, so no test reaches a real resolver
  • The e2e stub serves GET /api/v1/clusters and answers kubeconfig-info with the platform's 403 for an unknown cluster or no grant. New e2e steps cover factory k8s list (table and JSON), login --pool eval (kubectl then runs as the eval SA; --pool debug, a pending pool, is refused without touching the file), the unknown-cluster message, and losing the last grant (the kubeconfig is removed and factory k8s list shows the empty-state hint)

Changes in this update (rename to prime factory k8s)

  • 0c4ae104: prime cluster list / prime cluster login are now prime factory k8s list / prime factory k8s login, registered under the factory app beside slurm ("Kubernetes reservations on dedicated clusters"), with kubernetes as a hidden alias of the group. The unreleased top-level prime cluster group is removed outright, with no alias
  • prime auth k8s-token is unchanged, and so are the exec args written into kubeconfigs, so kubeconfigs already written keep working
  • commands/cluster.py moved to commands/factory_k8s.py (mirroring factory_slurm.py); the tests moved to tests/test_factory_k8s_cli.py and tests/test_factory_k8s_e2e.py, and CI's kind job points at the new e2e file. The token cache stays in utils/kube_token_cache.py
  • New root-wiring tests: prime factory k8s --help lists list and login, prime factory --help shows k8s and not the hidden alias, prime factory kubernetes login works, and prime cluster is "No such command"
  • Command names in the rest of this description were updated to match; the earlier "Changes in this update" sections now use the new names too

E2E

tests/test_factory_k8s_e2e.py runs the installed prime binary and real kubectl against a real apiserver (kind locally and in CI). Two things are stood in for:

  • The platform: a small HTTP server serving both routes in the shapes above and minting real one-hour ServiceAccount tokens through TokenRequest
  • Each pool's API proxy: a plain TCP pass-through per pool, on its own port, that does not terminate TLS. It listens on [::1] where possible. kind's apiserver certificate does not cover ::1, so, as with a real proxy's MagicDNS name, kubectl only connects because it verifies under tls-server-name: kubernetes.default.svc

The CLI is configured the way a researcher would do it (prime config set-base-url, prime config set-api-key).

# Step kind v0.33 (local, macOS) kind (CI test-prime-k8s-e2e)
1 prime factory k8s login writes a 0600 exec-only kubeconfig: one cluster entry per pool with that pool's proxy URL, shared CA, tls-server-name; the legacy top-level server and the raw apiserver URL appear nowhere; the unavailable pool is reported and not written PASS PASS
2 kubectl auth whoami = system:serviceaccount:<train-ns>:researcher through the train proxy; get pods and create configmap succeed PASS PASS
3 Confinement: get ns Forbidden, other namespace Forbidden, can-i list nodes = no PASS PASS
4 TLS: same kubeconfig with a wrong tls-server-name fails certificate verification, and so does one with the field removed PASS PASS
5 --context e2e-k8s-eval mints with pool=eval and dials only the eval proxy PASS PASS
4b Token cache: right after factory k8s login, 5× kubectl get configmaps + auth whoami cause exactly 1 mint. The cache file is 0600 in a 0700 dir and does not contain the API key. Re-login empties the cache, and the next 3 kubectl calls cause exactly 1 more mint. With PRIME_K8S_TOKEN_NO_CACHE=1 through kubectl, every call mints PASS PASS
6 Plugin as the exec block runs it: success on stdout; ambiguous 5, rate limited 3 (kubectl shows "Retry after 7s"), 503 → 1 with detail (also surfaced by kubectl), auth expired 4, unreachable 1; empty stdout on every failure. Transient failures keep the cached token (next kubectl: 0 mints). The platform's 401 evicts it (next kubectl: 1 mint) PASS PASS
5b prime factory k8s list shows each reservation's namespace, the pending pool as Provisioning with its reason, and prime factory k8s login e2e-k8s --pool <pool> for each ready one; -o json matches the stub's shape PASS (new)
5c factory k8s login --pool eval makes e2e-k8s-eval current (kubectl auth whoami = eval SA, no --context) and names the switch command; --pool debug (pending) exits 1 listing train, eval and leaves the file byte-identical PASS (new)
5d factory k8s login no-such-cluster shows the platform's neutral 403 message and points at prime factory k8s list; the other cluster's kubeconfig is untouched PASS (new)
7 Revocation with the token served from cache: the issued token is rejected through the proxy (after the apiserver's ≤10s token-review cache). kubectl, starting from the cached token, then shows the platform's 403 detail (1 to 10 mint attempts allowed; 5 observed), the plugin exits 2, re-login drops the revoked context and the remaining pool still works PASS PASS

| 8 | Losing the last grant: factory k8s login shows the 403 message and removes the kubeconfig; factory k8s list shows the empty-state hint | PASS | (new) |

To check the e2e is load-bearing, I temporarily (1) dropped tls-server-name from the CLI's output, so 5 of 7 tests fail, and (2) wrote the top-level server instead of the grant's, so 6 of 7 fail. For the cache, (3) disabling the same-parent rejection check fails step 7: kubectl shows only You must be logged in to the server.

Run it locally:

kind create cluster --name prime-e2e
kind get kubeconfig --name prime-e2e > /tmp/admin.yaml
cd packages/prime && PRIME_K8S_E2E_ADMIN_KUBECONFIG=/tmp/admin.yaml uv run pytest tests/test_factory_k8s_e2e.py -v

Not covered here: the real platform and a real tailnet proxy. Manual checklist against staging once platform#6878 and platform#7204 are deployed, from a machine on the customer tailnet:

  1. prime --context staging factory k8s login <cluster> with a granted pool whose proxy has joined the tailnet. A pool whose proxy hasn't joined is listed as skipped with the platform's reason
  2. export KUBECONFIG=~/.prime/kube/<cluster>.yaml. Confirm each clusters[].cluster.server is that pool's proxy MagicDNS name (https://k8s-<pool>-….<tailnet>.ts.net), not the cluster's public endpoint or the ops-tailnet apiUrl, and that tls-server-name: kubernetes.default.svc is set
  3. kubectl auth whoami returns system:serviceaccount:<grant-namespace>:researcher; kubectl get pods works; kubectl get ns is Forbidden
  4. With two pools, kubectl --context <cluster>-<other-pool> auth whoami returns the other pool's SA
  5. Revoke the grant in the dashboard. kubectl get pods fails with the platform's 403 detail (exit 2 from prime auth k8s-token), and re-login drops that context
  6. If possible, make the cluster unready or let its heartbeat go stale. kubectl get pods shows the platform's 503 detail (exit 1)
  7. prime --context staging factory k8s list shows the granted pools; with Tailscale off, factory k8s login warns Can't resolve <proxy> but still writes the file
  8. Remove the researcher from the reservation, re-run factory k8s login: the platform's "not a member of any Kubernetes reservation" message is shown and ~/.prime/kube/<cluster>.yaml is gone

Tests

Local results on 0c4ae104 (the rename):

  • tests/test_factory_k8s_cli.py: 89 passed (85 earlier, plus 4 root-wiring tests for the new command path)
  • tests/test_factory_k8s_e2e.py: 12 passed against kind / kubectl v1.33 on macOS (twice in a row), on a throwaway cluster that has since been deleted. The very first run against the freshly created kind cluster failed 2 steps with pods is forbidden for the just-bound researcher SA, which looks like RBAC warm-up on a brand-new apiserver rather than anything in the rename
  • Full packages/prime suite: 2014 passed, 15 skipped (the 12 e2e among them)
  • ruff 0.14.0 (the locked version): check and format --check are clean. ty check src/ is clean

Local results on 1c0b5011:

  • tests/test_factory_k8s_cli.py: 85 passed (64 earlier, plus 21 for factory k8s list, --pool, stale-file handling, the no-grant message and the DNS check)
  • tests/test_factory_k8s_e2e.py: 12 passed against kind v0.33 / kubectl v1.33 on macOS, on a throwaway cluster that has since been deleted
  • Full packages/prime suite: 2010 passed, 15 skipped (the 12 e2e among them)
  • ruff 0.14.0 (the locked version): check and format --check are clean. ty check src/ is clean

Earlier (CI green on 36e1a697: test-prime (3.11 to 3.13), test-prime-k8s-e2e (kind, including the cache step), lint-format, type-check, CodeQL):

  • tests/test_factory_k8s_cli.py: 64 passed (37 earlier, plus 27 for the cache). The new ones cover: a hit makes no HTTP call; the 5-minute margin both ways (6 min served, 4 min and expired re-minted); a credential without expiry is not cached; corrupt entries (empty, bad JSON, wrong shape) are misses and get overwritten; separation by base URL, cluster, pool and API key; the API key never appears in a cache file's name or contents; 0600 files, 0700 dirs, no leftover temp; 401/403/409 evict (also under --no-cache) while 503 keeps the entry; a rejected token is replaced once per parent process, then served; a revoked grant shows the platform's reason; --no-cache and PRIME_K8S_TOKEN_NO_CACHE=1 always mint and write nothing, while =0 caches; a failed cache write (mocked, and a real read-only directory) still prints the credential and warns without the token; factory k8s login clears only that cluster's entries (across platforms and pools), even when the login fails
  • tests/test_factory_k8s_e2e.py: 8 passed against kind v0.33 / kubectl v1.33 on macOS, on e178e0d8, on a throwaway cluster that has since been deleted. 36e1a697 only changes how the file name is hashed, and CI's kind job passes on it
  • Full packages/prime suite: 1989 passed, 11 skipped (the 8 e2e among them)
  • ruff 0.14.0 (the locked version): check and format --check are clean. ty check src/ is clean

Note

Medium Risk
New auth and credential-plugin paths handle API keys and cluster access; bugs could break kubectl login or leak confusing failures, but behavior is heavily tested including kind e2e.

Overview
Adds researcher Kubernetes access via prime factory k8s list and prime factory k8s login, wired under factory (with a hidden kubernetes alias). Login writes a tokenless ~/.prime/kube/<cluster>.yaml with one context per reachable pool, each pointing at that pool’s tailnet API proxy and using an exec block that calls prime auth k8s-token.

prime auth k8s-token is a kubectl credential plugin: it mints ExecCredentials from POST …/kube-token, emits only JSON on stdout, maps HTTP failures to distinct exit codes, and caches tokens on disk (kube_token_cache) to avoid platform rate limits, with re-mint logic when a cached token is rejected. Kubeconfigs pin PRIME_API_BASE_URL and optional --context so refreshes hit the same platform as login.

Lifecycle hygiene: re-login clears per-cluster cache; logout and config reset clear all cached k8s credentials.

CI gains a kind-based e2e job running test_factory_k8s_e2e.py; large unit coverage lives in test_factory_k8s_cli.py.

Reviewed by Cursor Bugbot for commit 0aaec2f. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0947ad575

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@JannikSt

Copy link
Copy Markdown
Member Author

Review findings addressed (e550d3b):

  • the credential plugin now posts to /api/v1/clusters/{cluster}/kube-token — Config.base_url strips the /api/v1 suffix, so the previous URL 404'd on every call. Added a test asserting the exact route string
  • prime --context X cluster login now bakes --context X into the written exec args, so kubectl-triggered refreshes (which run without PRIME_CONTEXT) hit the same platform the login did. Tests cover both with and without a context

978 tests passing, ruff clean.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@JannikSt

Copy link
Copy Markdown
Member Author

Pushed dbc59e4 fixing the custom-base-url regression from the --context change:

  • The base URL resolved at login time is pinned into the kubeconfig exec env as PRIME_API_BASE_URL, which outranks context resolution in Config.base_url — so with --context production baked into the exec args, a refresh no longer snaps back to the public default when a custom URL was set via prime config set-base-url.
  • --context keeps working as before and still selects the matching API key at refresh time.
  • Tests cover the custom-base-url + production-context combination in both directions (context alone loses the URL; the exec env pins it).

JannikSt and others added 6 commits October 25, 2026 21:48
- `prime cluster login <cluster>` writes a kubeconfig with an exec block, one
  context per pool the researcher has been granted. No Kubernetes token is
  ever written to disk
- `prime auth k8s-token` is the credential plugin kubectl invokes on every
  call. It writes only the ExecCredential to stdout, messages to stderr, and
  uses distinct exit codes so callers can tell a transient failure from a
  permanent one
- Revoked access, expired platform auth, rate limits, an unreachable platform
  and an ambiguous pool each get their own exit code and message
- Talks to the platform with httpx directly rather than the shared APIClient,
  which collapses status codes into one error type and would lose the contract
transfer-bulk-failures.jsonl is written by the images-transfer-bulk test and
is not gitignored, so it got swept into the previous commit.
- the credential plugin now posts to /api/v1/clusters/{name}/kube-token;
  Config.base_url strips the prefix, so without it every call 404'd. A test
  pins the exact route string.
- prime --context X cluster login bakes --context X into the exec args, so
  refreshes from kubectl (where PRIME_CONTEXT is unset) hit the same
  platform the login did
Baking --context into the exec args regressed custom API URLs: with
PRIME_CONTEXT=production, every kubectl refresh reloads the built-in
production environment and forces base_url back to the public default,
ignoring a URL set via 'prime config set-base-url'. The login-time
resolved URL now rides in the exec block's env as PRIME_API_BASE_URL,
which outranks context resolution in Config.base_url — so refreshes hit
the same platform the login did, while --context still selects the
matching API key. Tests cover the custom-base-url + production-context
combination in both directions.
- Fail cleanly when kubeconfig-info lacks server or CA instead of a KeyError
- Surface the platform's 403 detail rather than a generic revoked message
- Pin PRIME_DISABLE_VERSION_CHECK=1 in the exec env so kubectl output is not
  prefixed by an update banner on every call
- Cover login through the root CLI, server-verbatim (never apiUrl), pool
  forwarding, and root wiring of both subcommands

Co-authored-by: JannikSt <JannikSt@users.noreply.github.com>
Drives the installed prime binary and kubectl against a real apiserver; only
the platform is stood in for, serving kubeconfig-info and kube-token with the
platform's shapes and minting real ServiceAccount tokens. Covers SA identity,
namespace confinement, per-pool contexts, plugin exit codes, and revocation.
Opt-in via PRIME_K8S_E2E_ADMIN_KUBECONFIG; CI runs it against kind.

Co-authored-by: JannikSt <JannikSt@users.noreply.github.com>
@cursor
cursor Bot force-pushed the feature/cluster-login-kubeconfig branch from dbc59e4 to 4291721 Compare October 25, 2026 21:58
cursoragent and others added 3 commits October 25, 2026 22:02
…er exactly

Co-authored-by: JannikSt <JannikSt@users.noreply.github.com>
Researchers reach a cluster's apiserver only through the per-pool Tailscale
API proxy (platform#7204), so kubeconfig-info now returns a server per grant
instead of one cluster-wide endpoint.

- prime cluster login writes one cluster entry per reachable grant, named
  <cluster>-<pool>, with that grant's server, the shared CA, and
  tls-server-name (the proxy passes the apiserver's TLS through). Each
  context points at its own entry
- Grants without a server are skipped with the platform's unavailableReason;
  a top-level server from an older platform is ignored, never written
- No reachable grant exits 1 and leaves any earlier kubeconfig untouched.
  The file is written via a 0600 temp file and an atomic rename
- prime auth k8s-token maps 503 (cluster not ready, stale heartbeat,
  capability withdrawn) to exit 1 with the platform's detail on stderr
- The e2e stub returns the new shape and stands in a TCP pass-through proxy
  per pool; kubectl connects through it by verifying the apiserver
  certificate under tls-server-name
@wiz-prime-intellect

wiz-prime-intellect Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations 2 Low
SAST Finding SAST Findings -
Software Management Finding Software Management Findings -
Total 2 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

kubectl runs the exec plugin in every process, so each kubectl command
minted a new one-hour token and a polling loop hit the platform's mint
rate limit within minutes.

The plugin now caches the ExecCredential under ~/.prime/kube/cache
(0700 dirs, 0600 files, atomic writes), keyed by base URL, cluster, pool
and an API key fingerprint, and serves it while more than 5 minutes
remain. cluster login clears the cluster's entries; 401/403/409 from the
platform evict; --no-cache / PRIME_K8S_TOKEN_NO_CACHE=1 bypass it. A
re-run from the same kubectl process while the token is fresh means the
apiserver rejected it, so the plugin mints once instead of serving it
again, which surfaces the platform's reason for a revoked grant.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e178e0d. Configure here.

CodeQL flags a bare SHA-256 over the API key (py/weak-sensitive-data-hashing).
Hash the whole identity, key included, with PBKDF2 instead; it costs a few
milliseconds per kubectl command.
…Tailscale DNS hint

- prime cluster list [-o json]: the caller's reservations from GET
  /api/v1/clusters, with the login command for each ready one, and how to
  get access when there are none.
- prime cluster login --pool: choose the current context; an unknown or
  unreachable pool lists the available ones and writes nothing. Without
  it, print the current context and how to switch.
- A 403 (no grant) removes the cluster's now-useless kubeconfig, shows the
  platform's message and points at prime cluster list. When no pool is
  reachable the existing file is kept but called possibly stale.
- After writing, warn (never fail) when a proxy name does not resolve
  within 2s: Tailscale is off or on another tailnet.
prime cluster list/login become prime factory k8s list/login, registered
under the factory app beside slurm, with kubernetes as a hidden alias.
The unreleased top-level cluster group is removed. prime auth k8s-token
and the exec args written into kubeconfigs are unchanged.

commands/cluster.py moves to commands/factory_k8s.py and its tests to
test_factory_k8s_cli.py / test_factory_k8s_e2e.py.
@eexwhyzee eexwhyzee changed the title feature: prime cluster login and the k8s credential plugin feature: prime factory k8s login and the k8s credential plugin Nov 18, 2026
@eexwhyzee eexwhyzee changed the title feature: prime factory k8s login and the k8s credential plugin [ENG-8443] feature: prime factory k8s login and the k8s credential plugin Nov 18, 2026
A cached ServiceAccount token stays valid on the apiserver for up to an hour,
so it must not outlive the API key it was minted with. prime logout (also when
already logged out, since tokens minted with PRIME_API_KEY are cached too) and
prime config reset now delete the whole ~/.prime/kube/cache.
@eexwhyzee

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T22:42:44.651672Z 0731873 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07318731dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…s when minting

- Login under a persistently selected environment (prime config use staging)
  without --context left the exec args unpinned while PRIME_API_BASE_URL was
  pinned to that platform, so switching environments later sent the other
  environment's key to it. The active custom environment is now baked in as
  --context like an explicit one; production carries no key of its own and
  stays unpinned.
- The credential plugin's mint request now follows redirects, as APIClient
  does, so a platform URL that redirects works for kubectl as for login.
Rich shortened cells to an ellipsis to fit the terminal, which made the
namespace and the login command - the two values meant to be copied -
unusable. The table now holds only cluster, pool, status and proxy; each
reservation's namespace and login command (or why it isn't ready) print
whole underneath it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants