By default, when you setup your Kubernetes cluster, the certificates expires after one year.
If it’s been a while since you started up your Kubernetes cluster and you try to issue kubectl commands and notice connection refused errors.
The commands used requires ‘root’ privileged, so if you are login as non-root, you will need to prepend ‘sudo’ before the commands.
Troubleshooting further, you noticed that your kubelet service is failing to start (‘systemctl status kubelet’).
Upon checking logs related with kubelet (‘journalctl | grep kubelet’), you noticed the belong error messages.
To verify, issue ‘kubeadm certs check-expiration’.
Bingo!
Let’s manually renew certificates to fix our issues.
If your cluster has more than one control-plane node, be sure to run the following commands on all control-plane nodes in the cluster.
First, let’s backup our certificates just in case we need them.
- ‘cp -R /etc/kubernetes/pki /etc/kubernetes/pki.backup’
Now let’s renew our certificates with
- ‘kubeadm certs renew all’
Verify new certificate installed correctly.
- ‘kubeadm certs check-expiration’
Restart ‘kubelet service’
- ‘systemctl restart kubelet’
Once the renew process is complete, we will need to restart all the control plane pods. One way to restart is moving it’s manifest file out and wait for for about 20 seconds before moving the file back in to the ‘pki’ folder. This will will recreate the Pod to use the new certificates.
Copy the administrator certificates
- ‘sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config’
- ‘sudo chown $(id -u):$(id -g) $HOME/.kube/config’
This should get your cluster running again. You can verify with ‘kubectl get pods’ and not get an error.
Addition Things To Do If The Steps Above Doesn’t Work
If for some reason the above doesn’t work, you will need to do the following. Make a backup of /etc/kubernetes/pki/
- ‘cp -R /etc/kubernetes/pki /etc/kubernetes/pki-backup’
Delete the following files from /etc/kubernetes/pki
- ‘rm apiserver.crt
apiserver-etcd-client.key
apiserver-kubelet-client.crt
front-proxy-ca.crt
front-proxy-client.crt
front-proxy-client.key
front-proxy-ca.key
apiserver-kubelet-client.key
apiserver.key
apiserver-etcd-client.crt’
Remove the following all .crt and .key files from /etc/kubernetes/pki/etcd
- ‘rm /etc/kubernetes/pki/etcd/*.crt’
- ‘rm /etc/kubernetes/pki/etcd/*.key’
Then create certs with
- ‘kubeadm init phase certs all –apiserver-advertise-address ‘ — substitute your cluster IP.
BackupMove conf files from /etc/kubernetes
- ‘mkdir conf-backup’
- ‘mv admin.conf
controller-manager.conf
kubelet.conf
scheduler.conf /etc/kubernetes/conf-backup/’
Create new conf files
- ‘kubeadm init phase kubeconfig all’
Finally, restart kubelet service or reboot system.
- ‘systemctl restart kubelet’ or ‘reboot’
