[gowrisankar22]

Summary

Adds an opt-out flag --skip-delete-unused-stemcells to bosh create-env so operators can prevent the CLI from deleting stemcell images that are still referenced by another deployment.

Problem

• bosh create-env manages stemcell state in its local *-state.json and, at the end of a successful deploy, calls DeleteUnused() to remove any stemcell whose record is no longer the current one. FindUnused()/DeleteUnused() only reason about the local state file they cannot see other directors.
• Same stemcell used by bosh deployments managed by same director. When a new stemcell is rolled out, not all deployments are necessarily updated immediately. During this transition, some VMs may run into issues when executing operations such as recreate, scan and fix, or CCK these operations look for the old stemcell that was originally used by the VM. However, that stemcell may have already been deleted by bosh as part of the create-env call, resulting in failures with 404.
• Cross-director "is any other deployment still using this CID?" detection is not possible from the CLI (it cannot inspect the bosh deployments), so an operator opt-out is the appropriate fix. This mirrors the existing "orphan, don't delete" precedent of --fix-stemcell (later this orphaned resources cleaned up using bosh cleanup)

Changes

• New --skip-delete-unused-stemcells flag on create-env.
• Threaded through DeploymentOptions into DeploymentPreparer; when set, the DeleteUnused() call is skipped and a "Skipping deletion of unused stemcells" line is printed.
• Unit test covering both default (delete) and skip behavior.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 9836312c-2d69-4c99-b875-050deb216629

Commits

Reviewing files that changed from the base of the PR and between 4987068 and 7258ded.

Files selected for processing: 4

• cmd/create_env.go
• cmd/create_env_test.go
• cmd/deployment_preparer.go
• cmd/opts/opts.go

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

Walkthrough

The create-env command adds a --skip-delete-unused-stemcells option and passes it through deployment preparation. When enabled, deployment prints a skip message and returns before deleting unused stemcells. Otherwise, it continues to delete unused stemcells. A test checks that the command succeeds without deleting unused stemcells when the option is enabled.

Suggested reviewers: aramprice, julian-hj

Merge Risk: Minimal · up to 7258d

The opt-out works as intended while preserving existing stemcell cleanup by default.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the new --skip-delete-unused-stemcells flag for create-env, which is the main change.
Description check		The description explains the new flag, its purpose, implementation path, behavior, and tests. It directly relates to the changeset.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 4 files.

Finishing Touches

Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[AJLfleos]

I think this PR is a good improvement to close a gap in BOSH CLI

[rkoster]

CPI instances must own their stemcell resources independently. create-env should be able to delete a stemcell it no longer uses without affecting resources managed by a director.

The existing on-prem CPIs follow this model:

• vSphere’s create_stemcell generates a fresh sc-<UUID> name, imports the stemcell, and returns that name as its CID. Its datastore replicas belong to that CID’s lifecycle; separate uploads of the same stemcell receive distinct CIDs.
• OpenStack’s stemcell creator creates a new image, uploads the stemcell into it, and returns the resulting image ID.

The GDC CPI breaks this isolation: CreateStemcell searches for existing images by version using a common CPI label and returns the existing image’s CID. Independent CPI instances targeting the same project can therefore acquire the same resource, while DeleteStemcell deletes it without any shared-ownership handling.

The failure described here is a consequence of that GDC CPI behavior. We should fix stemcell ownership in the GDC CPI rather than add a CLI flag to accommodate cross-instance sharing. The CLI’s local state is the appropriate scope for cleanup when CPI resource ownership is implemented correctly.

[gowrisankar22]

@rkoster GDC has a different problem which we will take a look. However, BOSH cli still has a fundamental gap: a single BOSH Director manages hundreds of deployments and if it uses create-env to manage director lifecycle, this issue can occur across everywhere. So, this is not specific to GDC; it is a general BOSH limitation that also affects SAP.

default you can delete but certain cases we want to avoid this so it should be configurable and bosh cleanup command is used later to Cleanup the orphaned this case.

[rkoster]

default you can delete but certain cases we want to avoid this so it should be configurable and bosh cleanup command is used later to Cleanup the orphaned this case.

Could you elaborate on the cases so we can discuss them in detail?

[gowrisankar22]

Example from the SAP side(should be same for others too)

We create the BOSH Director using create-env. The same Director manages few other deployments, such as CredHub and others.

When we roll out a new stemcell, the BOSH Director deployment managed through create-env is updated first with the new stemcell. At the end of its lifecycle, create-env also cleans up the reference to the old stemcell.

However, the same BOSH Director manages several other deployments that may still reference the old stemcell.

This creates a problem if any of these deployments encounter an issue and require VM recreation. If the deployment has not been fully migrated to the new stemcell (for example, if the deployment failed halfway through and some VMs are still running with the old stemcell), BOSH Resurrection or a manual VM recreation will attempt to recreate those VMs using the old stemcell. Since create-env has already removed the old stemcell, the recreation will fail.

This is the gap we want to address.

By default, the PR will preserve the existing behavior across all infrastructures, with no changes to the current state. However, if an infrastructure needs to retain the old stemcell and prevent create-env from deleting it, this can be explicitly requested through an optional parameter.

[rkoster]

However, the same BOSH Director manages several other deployments that may still reference the old stemcell.

The distinction here is between the same stemcell name/version and the same cloud resource CID. create-env and the director are separate resource owners. Uploading the same stemcell to the director must produce a separate resource from the one managed by create-env, as demonstrated by the vSphere and OpenStack CPI implementations referenced above.

Updating the director VM through create-env can therefore safely delete its old stemcell resource while deployments managed by the director continue using theirs. A partially completed deployment or resurrection does not change this ownership boundary.

The SAP environment described here uses the GDC CPI, whose reuse-by-version behavior we have already identified as violating that boundary. It returns an existing image’s CID across independent CPI instances, coupling resource lifecycles that must remain separate. This example is another consequence of that same GDC CPI issue, not evidence of a general BOSH limitation.

Please address stemcell ownership in the GDC CPI. Adding a CLI flag to skip cleanup would work around the symptom without correcting the ownership violation. My request for changes stands.

[gowrisankar22]

However, the same BOSH Director manages several other deployments that may still reference the old stemcell.

The distinction here is between the same stemcell name/version and the same cloud resource CID. create-env and the director are separate resource owners. Uploading the same stemcell to the director must produce a separate resource from the one managed by create-env, as demonstrated by the vSphere and OpenStack CPI implementations referenced above.

Updating the director VM through create-env can therefore safely delete its old stemcell resource while deployments managed by the director continue using theirs. A partially completed deployment or resurrection does not change this ownership boundary.

The SAP environment described here uses the GDC CPI, whose reuse-by-version behavior we have already identified as violating that boundary. It returns an existing image’s CID across independent CPI instances, coupling resource lifecycles that must remain separate. This example is another consequence of that same GDC CPI issue, not evidence of a general BOSH limitation.

Please address stemcell ownership in the GDC CPI. Adding a CLI flag to skip cleanup would work around the symptom without correcting the ownership violation. My request for changes stands.

You are right. The GDC implementation was designed this way due to the long image import time (2.5 hours per import). Since we need to perform three imports, this adds up to 7.5 hours. Supporting multiple stemcells increases the total time even further.
However, this approach has additional drawbacks. For example, if an image import fails, it always triggers a new import instead of reusing the existing one, potentially resulting in additional orphaned resources on the infrastructure side and more wait times.

[gowrisankar22]

I close the PR as discussed #744 (comment)