Repository navigation
fix(DM01-6271): GKE cluster zone + machine type fallback on GCE_STOCKOUT - #652
Merged
Merged
Conversation
…CKOUT When a GKE cluster creation fails with ERROR status (e.g. GCE_STOCKOUT), the operator now automatically deletes the failed cluster and retries in the next available zone instead of permanently failing. - Add Zones []string to GKEClusterSpec for user-configurable fallback order - Add ZoneIndex int to GKEClusterStatus to track current fallback position - Default fallback covers europe-west1/west3 zones only (all current users require n1 machine types unavailable in europe-west10/west12) - Primary zone is always tried first; duplicates removed from default list - Update DeepCopyInto to correctly handle the new Zones slice field - Permanently fail only after all configured zones are exhausted
Extends the GCE_STOCKOUT fallback to also try equivalent machine types (n1 → n2 → e2 for standard/highmem; n1 → n2 for highcpu) after all zones are exhausted for the current machine type. - Add MachineTypes []string to GKEClusterSpec for user-configurable list - Replace ZoneIndex with FallbackIndex in GKEClusterStatus (flat index over zones × machineTypes product) - Add buildFallbackList(): auto-generates equivalent machine types by swapping series (n1/n2/e2) while keeping same tier and vCPU count - Add parseMachineType() and buildEquivalentMachineTypes() helpers - Update DeepCopyInto for new MachineTypes slice field
…sted Include tried zones, machine types, root cause (GCE_STOCKOUT), and point users to GCP status page or administrator instead of vague retry advice.
Spec.Zone and Spec.MachineType now always reflect the user's original intent. Fallback runtime state is tracked in Status.FallbackZone and Status.FallbackMachineType. createCluster prefers Status fallback values when set, falling back to Spec otherwise. This also makes buildFallbackList stable: it always builds from Spec (original values), so FallbackIndex reliably maps to the same list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Persistent
GCE_STOCKOUTerrors (Google Cloud Support Case #75099279) cause GKE cluster creation to fail permanently ineurope-west3-candeurope-west3-a. The previous behavior set the CRD status toerrorimmediately with no retry, requiring manual job restart.Changes
Zone + Machine Type Fallback (
handler.go)When a GKE cluster reaches
ERRORstatus on GCP (e.g. due toGCE_STOCKOUT):(zone, machineType)combination from a fallback listFallback order: exhausts all zones per machine type, then moves to the next machine type:
Auto-generated Machine Type Equivalents
buildEquivalentMachineTypes()parses the user'smachineTypeand generates same-tier/size alternatives across n1→n2→e2 series (skipping e2 forhighcpuwhich is unsupported).Default Zone List
Covers all 6 verified zones in
europe-west1andeurope-west3— the only regions supportingn1machine types (used by all current InfraBox users).europe-west10/west12are excluded as they do not supportn1.New CRD Fields
spec.zones []string— user-configurable ordered fallback zone listspec.machineTypes []string— user-configurable ordered machine type liststatus.fallbackIndex int— tracks current position in the flat fallback list (replaceszoneIndex)Actionable Error Message
When all options are exhausted, the error message includes:
Test
Verified against existing GCP operation logs from case #75099279 (
ib-c49857fb,europe-west3-c,GCE_STOCKOUT).Jira
https://jira.tools.sap/browse/DM01-6271