update webhooks: handle resourceVersion conflicts properly by multi-io · Pull Request #2009 · kubermatic/machine-controller

multi-io · 2026-03-31T22:55:45Z

This changes the mutating webhook handler's behaviour in case of version conflicts.

If an update had a metadata.resourceVersion conflict, it makes no sense to perform the webhook's normal checks, e.g. for .spec immutability, as those checks might fail just because the base resourceVersion isn't the same.

For example, if the client is trying to update based on an outdated version of the resource, the .spec might be different from the in-cluster version just because of that, triggering the corresponding webhook check and responding with a 400, even though 409 would be the correct response that allows the client to perform proper conflict handling.

So in case of a resourceVersion conflict the webhook now just returns success, which causes the API server to fall back to its default handling, which will detect the resourceVersion conflict and return the 409 itself.

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

What type of PR is this?

/kind design

Special notes for your reviewer:

Does this PR introduce a user-facing change? Then add your Release Note here:

Documentation:

NONE

…'s default handling If the update had a resourceVersion conflict, it makes no sense to perform the webhook's normal checks, e.g. for .spec immutability, as those checks might fail just because the base resourceVersion isn't the same. For example, if the client is trying to update based on an outdated version of the resource, the .spec might be different just because of that, triggering the corresponding webhook check and reponding with a 400, even though 409 would be the correct response that allows the client to perform proper conflict handling. So in case of a resourceVersion conflict the webhook now just returns success, which causes the API server to fall back to its default handling, which will detect the resourceVersion conflict and return the 409 itself. Signed-off-by: Olaf Klischat <olaf.klischat@gmail.com>

kubermatic-bot · 2026-03-31T22:55:51Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kubermatic-bot · 2026-03-31T22:55:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign julioc-p for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubermatic-bot · 2026-03-31T22:56:02Z

Hi @multi-io. Thanks for your PR.

I'm waiting for a kubermatic member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull request overview

This PR adjusts the mutating admission webhook behavior for Machine and MachineDeployment updates so that metadata.resourceVersion conflicts don’t get masked by webhook validation errors, allowing the Kubernetes API server to return the correct 409 Conflict.

Changes:

Add a resourceVersion mismatch shortcut on Machine updates to immediately allow the request.
Add a similar resourceVersion mismatch shortcut on MachineDeployment updates to immediately allow the request.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
pkg/admission/machines.go	Adds an early return on Update when OldObject/Object resourceVersion differs to avoid webhook immutability checks masking 409 conflicts.
pkg/admission/machinedeployments.go	Adds a resourceVersion mismatch bypass on Update, but currently placed after validation/mutation logic and includes a formatting issue.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-31T23:54:54Z

pkg/admission/machinedeployments.go

+		if oldMachineDeployment.ResourceVersion != machineDeployment.ResourceVersion {
+			// resource version conflict. Return success to fall back to the API server's default handler, which will respond with a proper 409
+            return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)
+		}


In the conflict branch, the handler currently reaches this resourceVersion check only after running defaulting/mutations and validateMachineDeployment earlier in the function. That means an update with an outdated resourceVersion can still be rejected by the webhook with a 400 before the API server returns the intended 409. Consider moving the resourceVersion mismatch check to the very start of the Update path (before any mutation/validation) and returning Allowed=true with no patch when it mismatches.

Copilot · 2026-03-31T23:54:54Z

pkg/admission/machinedeployments.go

 		}
+		if oldMachineDeployment.ResourceVersion != machineDeployment.ResourceVersion {
+			// resource version conflict. Return success to fall back to the API server's default handler, which will respond with a proper 409
+            return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)


This line has inconsistent indentation (spaces instead of tabs) and will be rewritten by gofmt; it may also fail formatting/lint checks in CI. Please run gofmt (or fix indentation) so the return statement is properly aligned with the surrounding block.

Suggested change

return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)

return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)

Copilot · 2026-03-31T23:54:55Z

pkg/admission/machinedeployments.go

+			// resource version conflict. Return success to fall back to the API server's default handler, which will respond with a proper 409
+            return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)


This new resourceVersion-conflict bypass is a behavior change that is currently untested. Please add a unit test that constructs an Update AdmissionRequest where OldObject.resourceVersion != Object.resourceVersion and asserts the mutator returns Allowed=true (and does not fail validation / does not return a patch).

Suggested change

// resource version conflict. Return success to fall back to the API server's default handler, which will respond with a proper 409

return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)

// Resource version conflict. Return success without a patch so the API server can respond with a proper 409.

return &admissionv1.AdmissionResponse{Allowed: true}, nil

Copilot · 2026-03-31T23:54:55Z

pkg/admission/machines.go

+		if oldMachine.ResourceVersion != machine.ResourceVersion {
+			// resource version conflict. Return success to fall back to the API server's default handler, which will respond with a proper 409
+			return createAdmissionResponse(log, machineOriginal, &machine)
+		}


This new resourceVersion-conflict bypass is a behavior change that is currently untested. Please add a unit test that constructs an Update AdmissionRequest where OldObject.resourceVersion != Object.resourceVersion and asserts the mutator returns Allowed=true (and does not run the immutability checks / does not return a patch).

kubermatic-bot added the kind/design Categorizes issue or PR as related to design. label Mar 31, 2026

kubermatic-bot added sig/cluster-management Denotes a PR or issue as being assigned to SIG Cluster Management. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 31, 2026

kubermatic-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Mar 31, 2026

multi-io changed the title ~~update webhooks: hand resourceVersion conflicts properly~~ update webhooks: handle resourceVersion conflicts properly Mar 31, 2026

kron4eg requested a review from Copilot March 31, 2026 23:51

Copilot started reviewing on behalf of kron4eg March 31, 2026 23:51 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

kubermatic-bot added docs/none Denotes a PR that doesn't need documentation (changes). and removed do-not-merge/docs-needed Indicates that a PR should not merge because it's missing one of the documentation labels. labels Apr 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update webhooks: handle resourceVersion conflicts properly#2009

update webhooks: handle resourceVersion conflicts properly#2009
multi-io wants to merge 1 commit intokubermatic:mainfrom
syseleven:admission_handle_version_conflicts

multi-io commented Mar 31, 2026 •

edited

Loading

Uh oh!

kubermatic-bot commented Mar 31, 2026

Uh oh!

kubermatic-bot commented Mar 31, 2026

Uh oh!

kubermatic-bot commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)
	return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)

		// resource version conflict. Return success to fall back to the API server's default handler, which will respond with a proper 409
		return createAdmissionResponse(log, machineDeploymentOriginal, &machineDeployment)

Conversation

multi-io commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubermatic-bot commented Mar 31, 2026

Uh oh!

kubermatic-bot commented Mar 31, 2026

Uh oh!

kubermatic-bot commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

multi-io commented Mar 31, 2026 •

edited

Loading