Namespace stuck in Terminating state: users cannot delete blocked resources

[ktoulliou]

Bug description

When a tenant user deletes a namespace, it transitions to a "Terminating" state but gets stuck due to resources with finalizers or other blocking conditions. At this point, the user loses access to the namespace and can no longer list, edit, or delete the blocking resources, effectively leaving the namespace stuck in "Terminating" state indefinitely.

This creates a deadlock situation: the namespace cannot complete deletion because of blocking resources, and the user cannot remove those resources because they no longer have access to the namespace.

How to reproduce

1. Create a Tenant with a user as owner:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: my-tenant
spec:
  owners:
    - name: alice
      kind: User

2. As the tenant user, create a namespace and deploy resources that include finalizers (e.g., custom resources managed by an operator, PVCs, etc.):

kubectl --as=alice create namespace my-namespace
kubectl --as=alice -n my-namespace apply -f <resource-with-finalizer>.yaml

3. As the tenant user, delete the namespace:

kubectl --as=alice delete namespace my-namespace

4. The namespace transitions to "Terminating" state. The user can no longer access it:

kubectl --as=alice get pods -n my-namespace
# Error: forbidden

5. The namespace remains stuck in "Terminating" because the blocking resources cannot be removed by the user.

Expected behavior

When a namespace enters the "Terminating" state, the tenant user should retain sufficient access (at minimum, permissions to list and delete resources) within the namespace so they can resolve any blocking resources and allow the namespace deletion to complete.

Logs

No particular pertinent logs from the Capsule controller manager. The only observable symptoms are:

403 Forbidden errors when the tenant user attempts to access resources in the terminating namespace
Resources with finalizers remaining in the namespace, preventing deletion from completing

Additional context

• Capsule version: 0.12.4
• Capsule proxy version: 0.10.0
• Helm Chart version: 0.12.4
• Kubernetes version: 1.34.0
• Talos version 1.12.2

This issue may be related to how Capsule manages RBAC bindings for namespaces belonging to a Tenant. When the namespace enters "Terminating" state, the associated RoleBindings may be removed or become ineffective, stripping the user's permissions before the namespace is fully cleaned up.

A possible solution could be to ensure that Capsule preserves user access to namespaces in "Terminating" state until deletion is complete, or to have Capsule handle the cleanup of blocking resources on behalf of the user.

[oliverbaehler]

Thanks for the report, extend check to verify:

• If namespace has finalizers
• If condition NamespaceFinalizersRemaining is true
• If condition NamespaceContentRemaining is true

then the namespace is still considered active and capsule managed resources are still being applied.

[prometherion]

I have no clear idea how to approach this: the root cause is the Tenant creating objects blocking the deletion of a Namespace since it contains objects with a finalizer.

If a platform-level Operator reconciles these objects, it should be able to reconcile on its own and perform the termination: if that's not happening, it sounds more like a lack of reliability from the Operator itself.

Rather, if the Tenant owner adds the finalizers and then deletes the Namespace, we end up with this situation since all the objects are being deleted, as well as the RoleBinding granting permissions to the Tenant: it's not Capsule, it's Kubernetes.

I have no idea of your chicken/egg problem since you didn't share enough information, and I would give it a try first to reuse Kubernetes machinery, such as the KEP-5080 – Ordered Namespace Deletion.

[oliverbaehler]

@prometherion It's quiet rivial or i have already added a "fix". We block our managed resources deletion via our admission until the corressponding namespace no longer has NamespaceFinalizersRemaining set to true. We also need to solve this especially with Global/TenantResource. Imagine a CiliumNetworkPolicy is terminated while a Pod with a finalizer is still blocking. Sounds like a legitimate threat.

[ktoulliou]

@prometherion @oliverbaehler I encountered this issue when a user had operator-managed resources. They had finalizers on ObjectBucketClaim (managed by rook-ceph) as well as on a Strimzi operator resource for the Kind: KafkaTopics. They still had replicas or other resources that hadn't been cleaned up by the operator yet.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.