Kubelet startup disabling managed identity plugin

Whenever the kubelet service starts, it loads the credentials plugins, calls .Enabled() on them, and the ones that return true are part of the docker keyring for image pulls. The ones that return false are disabled for the life of the kubelet.

In this case we’re using a user-assigned Managed Identity as the identity for ACR pulls. The Enabled() call-chain looks like this:

kuberuntime_manger:: NewKubeGenericRuntimeManager
|- plugins.go::NewDockerKeyring
|- provider.go::Enabled
|- azure_credentials.go::Enabled ([source link](https://github.com/kubernetes/kubernetes/blob/520b991347b9c77635c0e4555f1703a86dcdd4ff/pkg/credentialprovider/azure/azure_credentials.go#L212))
|- azure_auth.go::GetServicePrincipalToken
|- token.go::NewServicePrincipalTokenFromMSIWithUserAssignedID
|- token.go::newServicePrincipalTokenFromMSI
|- token.go::getMSIType
|- token.go::MSIAvailable
|- token.go::getMSIEndpoint

If the IMDS endpoint isn’t available, then azure_credentials.go::Enabled() returns false which disables it. On our bad node we see this behavior in these snipped logs:

I0525 06:41:19.936396 750 kubelet.go:290] "Adding apiserver pod source"
I0525 06:41:19.936910 750 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
E0525 06:41:19.937147 750 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://prod-aks-csp-us-wu2-0-fe-dns-629883d0.hcp.westus2.azmk8s.io:443/api/v1/nodes?fieldSelector=metadata.name%3Daks-cloudconnect-14073472-vmss000000&limit=500&resourceVersion=0": dial tcp: lookup prod-aks-csp-us-wu2-0-fe-dns-629883d0.hcp.westus2.azmk8s.io on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
E0525 06:41:19.939295 750 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://prod-aks-csp-us-wu2-0-fe-dns-629883d0.hcp.westus2.azmk8s.io:443/api/v1/services?limit=500&resourceVersion=0": dial tcp: lookup prod-aks-csp-us-wu2-0-fe-dns-629883d0.hcp.westus2.azmk8s.io on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
I0525 06:41:19.943322 750 kuberuntime_manager.go:245] "Container runtime initialized" containerRuntime="containerd" version="v1.5.9.m" apiVersion="v1alpha2"I0525 06:41:19.944919 750 azure_auth.go:94] azure: using managed identity extension to retrieve access tokenE0525 06:41:19.946121 750 azure_credentials.go:201] Failed to create service principal token: MSI not available...I0525 06:41:19.947741 750 server.go:1213] "Started kubelet"I0525 06:41:19.948086 750 server.go:149] "Starting to listen" address="0.0.0.0" port=10250

That explains why we can't pull images, as the kubelet has decided to not use the azure_credentials plugin. Also note the "network is unreachable" errors, as they lead to the next issue

Understanding why the IMDS endpoint isn't available

Since this seems to be an issue reported only on Mariner and not on Ubuntu, we tried to determine why the MSI wasn’t available at kubelet start. Investigating the logs we see a few interesting items:

The node originally started correctly
The issue started after a node reboot
From correlating timestamps, the kubelet service is starting (and hitting the MSI not available issue) during cloud-init, which is responsible for setting up the network
At the same time as we're seeing the missing MSI (which we can pull ourselves with curl when debugging) we're seeing network unreachable errors for other endpoints

From looking at the systemd service descriptions, there’s nothing specifying that the kubelet service should delay starting until cloud-init has finished. This is true on both Mariner and Ubuntu. We were able to repro the behavior on nodes by modifying the startup order to ensure kubelet starts before cloud-init, and validated that specifying the kubelet to start after cloud-init finishes fixes the issue. Thus it seems that we also have a problem that the kubelet is trying to use the network while it's still being configured.

Next Steps

First, Henry is going to continue driving the investigation of cloud-init ordering. He plans to see if there’s anything explicit or implicit in Ubuntu that’s ensuring the services start in the right order, and in any case will drive a fix with AKS to make the dependency explicit. 


Andy, would you be willing to look into the Enabled issue in the kubelet plugins? It seems at least in this case that we would want Enabled to be true if an MI is specified, regardless of if the IMDS endpoint is currently available or not. That would allow the node to recover from transient issues, rather than caching the failure for the life of the kubelet. Alternatively, we should differentiate the errors between IMDS not available and IMDS available but MI not available.

Last, since these fixes will take a few weeks to deploy, I’m planning on using a DaemonSet in our clusters to patch the systemd config for kubelet.service to ensure it always starts after cloud-init. That will unblock us, and we can easily remove the DaemonSet once the fix is live.

</details>