coredns ContainerCreating AWS EKS

I was facing an issue in AWS EKS as coredns container is in “ContainerCreating” state. When I describe the POD, I can see below error:

$ kubectl describe pod coredns-xxxxxx -n kube-system

Events:
  Type     Reason     Age                From                                                    Message
  ----     ------     ----               ----                                                    -------
  Normal   Scheduled  39s                default-scheduler                                       Successfully assigned kube-system/coredns-b79f95bd6-67t74 to ip-x-x-x-x.ap-southeast-2.compute.internal
  Normal   Pulling    23s (x2 over 37s)  kubelet, ip-x-x-x-x.ap-southeast-2.compute.internal  pulling image "602401143452.dkr.ecr.ap-southeast-2.amazonaws.com/eks/coredns:v1.2.2"
  Warning  Failed     23s (x2 over 37s)  kubelet, ip-x-x-x-x.ap-southeast-2.compute.internal  Failed to pull image "602401143452.dkr.ecr.ap-southeast-2.amazonaws.com/eks/coredns:v1.2.2": rpc error: code = Unknown desc = Error response from daemon: Get https://602401143452.dkr.ecr.ap-southeast-2.amazonaws.com/v2/eks/coredns/manifests/v1.2.2: no basic auth credentials
  Warning  Failed     23s (x2 over 37s)  kubelet, ip-x-x-x-x.ap-southeast-2.compute.internal  Error: ErrImagePull
  Normal   BackOff    12s (x3 over 36s)  kubelet, ip-x-x-x-x.ap-southeast-2.compute.internal  Back-off pulling image "602401143452.dkr.ecr.ap-southeast-2.amazonaws.com/eks/coredns:v1.2.2"
  Warning  Failed     12s (x3 over 36s)  kubelet, ip-x-x-x-x.ap-southeast-2.compute.internal  Error: ImagePullBackOff

The issue here is that it couldn’t download the coredns image from the repository.

The solution is to check whether you are using Public or Private subnets for your worker nodes. If you are using Public subnets, check whether it has Internet Gateway attached in its Routing Table and you are able to access Internet from your worker nodes.

$ nslookup google.com

If you are using Private Subnets, check whether you have NAT Gateways attached in your Routing Table (0.0.0.0/0) and you are able to access Internet from your worker nodes.

$ nslookup google.com

Note: NAT Gateways provide internet access to servers which are in Private Subnets.

Check on which Worker Nodes, the coredns PODs are trying to create/running/scheduled

$ kubectl get pods -A -o wide

Restart that particular Worker Node and it should all work.

Leave a Reply

No comments to display
Be the first to comment