Skip to content

openshell sandbox create timeout reports ConfigurationPending when the pod could not be scheduled #5230

Open

Description

@u0324020

User Story

I'm a senior engineer at VicOne. We run OpenShell on an EKS cluster to sandbox the AI agents our teams operate, and the cluster is shared with other workloads, so node capacity varies during the day. When openshell sandbox create timed out I spent the first round of debugging on sandbox configuration, because the only thing the CLI printed was ConfigurationPending. The pod had simply not been scheduled. I reproduced it twice on 0.1.6. As an operator on a shared cluster, I want the timeout message to tell me that the pod could not be scheduled and why, so that I go and check capacity instead of the sandbox configuration.

Problem Statement

When the sandbox pod cannot be scheduled, the gateway's sandbox status already carries the cause (PodScheduled: False (Unschedulable) with the scheduler's "0/N nodes are available" message), but the CLI timeout message only reports the first Ready=False condition, which the gateway has replaced with ConfigurationPending while no supervisor configuration report has arrived. The user sees a configuration placeholder, not the scheduling failure. About five minutes after creation the Kubernetes driver removes the stale bootstrap, so sandbox get is only useful for a short window and most users will only ever see the CLI error.

Related: #5028 covers the same symptom family for image pull failures. The difference here is that the scheduling cause is already present in the sandbox status returned by the gateway, so a CLI-only change surfaces it; #5028 needs the gateway to relay pod events.

Impact / Why This Matters

  • On any shared or autoscaled cluster, "cannot schedule" is one of the most common reasons a create fails, and today it is invisible from the CLI.
  • The hint in the error points at image_preparation_timeout_seconds, which sends users in the wrong direction when the problem is capacity.
  • The sandbox is rolled back a few minutes later, so the CLI message is the only durable signal.

Acceptance Criteria

  • When provisioning times out and the sandbox status contains a PodScheduled condition with status False, the CLI error includes that condition's reason and message (for example Unschedulable: 0/11 nodes are available: 11 Insufficient cpu).
  • When no such condition exists, the message is unchanged.
  • PodScheduled=True does not affect the message.
  • Covered by unit tests in the CLI crate; no gateway or driver behaviour change.

Reproduction Steps

  1. Kubernetes gateway on a cluster where no node can satisfy the request (or any cluster at autoscaler max size).
  2. openshell sandbox create --name repro --cpu 4 --memory 2Gi
  3. While it waits, in another shell: openshell sandbox get repro
  4. After 300 s the CLI exits 1 with the ConfigurationPending message below, while sandbox get shows PodScheduled: False (Unschedulable). About five minutes after creation the sandbox is removed by the driver.

Suggested UX (if applicable)

sandbox provisioning timed out after 300s. Last reported status:
Unschedulable: 0/11 nodes are available: 11 Insufficient cpu. ...

Implementation sketch (CLI only): in ready_false_condition_message (crates/openshell-cli/src/commands/common.rs), look for a PodScheduled condition with status False before falling back to the first Ready=False condition. I have this change with unit tests ready and will open the PR once the issue is accepted.

Environment

  • Gateway / Helm chart: 0.1.6 (helm-chart-0.1.6), Kubernetes driver on EKS, cluster-autoscaler at max node group size
  • CLI: 0.1.5; the message-building code was checked against current main and is unchanged
  • Platform: Linux nodes (Amazon Linux 2023, kernel 6.12), CLI on macOS

Logs

CLI after 300 s (exit 1):

sandbox provisioning timed out after 300s. Last reported status:
ConfigurationPending: Waiting for effective configuration validation before workload activation

openshell sandbox get repro during the same window:

Phase: Provisioning
Conditions:
  - PodScheduled: False (Unschedulable) - 0/11 nodes are available: 11 Insufficient cpu. no new claims to deallocate, preemption: 0/11 nodes are available: 11 Preemption is not helpful for scheduling.
  - Bootstrapping: True (SandboxRuntimeGenerationStarting)
  - ConfigurationReady: False (ConfigurationPending) - Waiting for effective configuration validation before workload activation
  - Ready: False (ConfigurationPending) - Waiting for effective configuration validation before workload activation

Gateway log about 5 minutes after creation:

rolled back stale fail-closed sandbox-runtime bootstrap

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Nov 6, 2026
  2. self-assigned this
    on Nov 6, 2026
  3. FrostGod commented on Nov 6, 2026

    @FrostGod
    Collaborator

    working on it

  4. u0324020 commented on Nov 9, 2026

    @u0324020
    Author

    Hi @FrostGod , thanks for picking this up.

    I filed this issue after hitting it on a real cluster, and I had already prototyped a fix before it was accepted. I don't want to duplicate your work, so I'm sharing it in case it's useful.

    What my prototype does. When the 300-second wait times out, the CLI prints the scheduling failure reason the gateway already returns (PodScheduled=Unschedulable and the scheduler message) instead of only ConfigurationPending. The change is in crates/openshell-cli/src/commands/common.rs, with 3 unit tests. It is on our fork: VicOne-OSS-TW/OpenShell, branch fix/5230-cli-unschedulable-timeout-message/u0324020 (commit ac35c2c). Format, clippy and the CLI lib tests passed when I last ran them.

    How I'd like to proceed. I'm not vouched yet, so I can't open a PR anyway. Two options, whichever you prefer:

    1. You continue with your own approach, and I review it or test it against a case where the pod can't be scheduled.
      1. You take any part of my branch you find useful, and I'm happy to adjust it to your direction.

    Please let me know which works better, or if you'd rather I leave it entirely to you.

  5. FrostGod commented on Nov 9, 2026

    @FrostGod
    Collaborator

    @u0324020 , Thanks for filing the issue!

    If you want to contribute on top of my PR, i am happy to accept it and review,
    can you please raise a request to vouch yourself as a contributor

    or you can just review this PR
    #5344

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

state:acceptedA maintainer decided OpenShell should pursue this issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions