[Bug]: atelet credential-provider configuration is hard-coded to GKE host paths #2669
Description
What happened?
Since PR #917, the base atelet DaemonSet always passes --image-credential-provider-config and --image-credential-provider-bin-dir.
These are populated from GKE-specific host paths:
/home/kubernetes/bin, mounted asDirectoryOrCreate/etc/srv/kubernetes/cri_auth_config.yaml, mounted asFileOrCreate
On vanilla Kubernetes nodes, these paths may not exist or the node's credential provider may be installed elsewhere.
DirectoryOrCreate creates an empty plugin directory. If /etc/srv/kubernetes exists, FileOrCreate creates an empty configuration file. Atelet rejects that file because it is not a valid CredentialProviderConfig and exits. If the parent directory does not exist, the pod instead fails during volume setup.
The Kind overlay avoids this by removing the credential-provider arguments and mounts. Other installations have no supported configuration for disabling the GKE paths or supplying alternative volume sources, so they must patch the DaemonSet.
Expected Behavior
The vanilla Kubernetes installation should:
- Start atelet with anonymous image pulls when no credential provider is configured.
- Allow operators to supply a
CredentialProviderConfigand its provider executable through supported installation settings. - Allow provider executables to be delivered through a container image and pod-local shared storage, such as an init or sidecar-style container copying them into an
emptyDirmounted by atelet. Installing binaries on every node or sharing host directories should not be required. - Keep hostPath-based providers, including the current GKE paths, as optional platform-specific configuration rather than the default.
Steps to Reproduce
-
Use a non-GKE Kubernetes cluster whose nodes do not contain the GKE credential-provider paths.
-
Deploy without the Kind profile:
go run ./cmd/ate-setup deploy ate-system \ --credential-provider='{"enabled":false}' -
Inspect the atelet pods:
kubectl -n ate-system get pods -l app=atelet kubectl -n ate-system describe pods -l app=atelet kubectl -n ate-system logs daemonset/atelet -
Observe either a hostPath volume setup failure or atelet exiting while loading the empty credential-provider configuration.
Sandbox Runtime
Both / Runtime Agnostic
Agent Substrate Version / Commit SHA
Kubernetes Version & Environment
On-prem Kubernetes 1.36.0, Ubuntu 24.04 nodes, containerd 2.2.3. Not GKE or Kind.
Host OS & Architecture
No response
Relevant Logs and Diagnostic Output
Additional Context
Atelet already supports anonymous pulls when --image-credential-provider-config is unset. The Kind overlay uses this behavior by deleting both credential-provider arguments and mounts.
The credential-provider interface itself accepts configurable paths, but the supplied manifests and installer do not expose a way to populate those paths from operator-selected volumes.
A provider container image backed by a shared emptyDir would provide a portable option for vanilla Kubernetes while retaining hostPath mounting for environments where the provider is intentionally installed on each node.
Confirmation
- I have searched existing issues and verified that this is not a duplicate.
- I have verified that this issue occurs on the latest commit on
main.
Activity
BenTheElder commentedon Oct 17, 2026
That already exists.
You can use a kustomize overlay, like kind
You can do this with a custom atelet manifest / overlay / image.
I don't think we should remove it as the default, it's lower friction. We can add more common paths to the default mount by making them optional directory mounts.
BenTheElder commentedon Oct 17, 2026
Ah right, can't do a truly optional hostpath.
Well ... cc Steven Shriver (@stshrive) something else to consider for your installer revamp doc. We'll need to customize this in the chart etc.
In the meantime, the accepted solution is to kubectl kustomize the manifest.
stshrive commentedon Oct 17, 2026
Thanks for bringing this to my attention. I'm in the middle of scoping out all of the areas that will need updates so this will be included as well.
marcellodesales commentedon Oct 17, 2026
Following the direction in this thread (make the credential-provider wiring work off GKE rather than hard-coding the node paths), I opened #2799. It's an opt-in kustomize component,
components/image-credential-secret:CredentialProviderConfigcomes from a ConfigMap;kubernetes.io/dockerconfigjsonSecret, through a smallcmd/dockerconfig-credential-providerplugin.It doesn't change atelet itself, any API or the proto. It's proven by a two-phase e2e on the kind install against a private GHCR repo, with a fresh digest each run: refused without the Secret (
UNAUTHORIZED: authentication requiredin atelet's log), pulled with it.The Helm-chart side for kagent-dev/substrate is kagent-dev#88 (
atelet.imageCredentialProviderPlugins/imageCredentialProviderSecrets). cc Steven Shriver (@stshrive) in case you'd rather fold this into the installer revamp; happy to reshape it into settings there.(Prepared with AI assistance; I verified it on our CI and production cluster and will handle review myself.)
BenTheElder commentedon Oct 17, 2026
Marcello DeSales (@marcellodesales) as evidenced by the comments above, a design is being prepared for the installer, which was discussed at the community meeting Thursday and is in the meeting notes and other open issues.
I don't think we want to add MORE non-tenant-scope secrets right ahead of GA and the right installer approach is an open thread.
marcellodesales commentedon Oct 17, 2026
Benjamin Elder (@BenTheElder) sounds good... No worries... I just wanted to move forward as we are preparing our product and we've been testing substrate in our requirements... I hope to make to the upcoming meetings...
Can you please also point me to the thread you are referring to? I tried to follow most of the threads here...