[Kubernetes] kubelet error - part of the existing bootstrap client certificate is expired

kubelet not working and the below error occurred.

- part of the existing bootstrap client certificate is expired 

Mar 24 10:57:40 k8smaster01 kubelet[3184]: I0314 10:57:40.514806    3184 server.go:417] Version: v1.18.12
Mar 24 10:57:40 k8smaster01 kubelet[3184]: I0314 10:57:40.514974    3184 plugins.go:100] No cloud provider specified.
Mar 24 10:57:40 k8smaster01 kubelet[3184]: I0314 10:57:40.514985    3184 server.go:838] Client rotation is on, will bootstrap in background
Mar 24 10:57:40 k8smaster01 kubelet[3184]: E0314 10:57:40.515911    3184 bootstrap.go:265] part of the existing bootstrap client certificate is expired: 2022-02-15 04:25:59 +0000 UTC
Mar 24 10:57:40 k8smaster01 kubelet[3184]: F0314 10:57:40.515931    3184 server.go:274] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

Renew certs and restart kubelet.

$ sudo kubeadm alpha certs renew all

$ sudo kubeadm alpha certs check-expiration
[sudo] password for fo4-mgr:
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 14, 2023 03:04 UTC   364d                                    no
apiserver                  Mar 14, 2023 03:04 UTC   364d            ca                      no
apiserver-kubelet-client   Mar 14, 2023 03:04 UTC   364d            ca                      no
controller-manager.conf    Mar 14, 2023 03:04 UTC   364d                                    no
front-proxy-client         Mar 14, 2023 03:04 UTC   364d            front-proxy-ca          no
scheduler.conf             Mar 14, 2023 03:04 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Feb 13, 2031 04:25 UTC   8y              no
front-proxy-ca          Feb 13, 2031 04:25 UTC   8y              no

$ sudo systemctl restart kubelet

Check kubelet status

$ systemctl status kubelet
* kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           `-10-kubeadm.conf
   Active: active (running) since Mon 2022-03-14 12:12:11 KST; 24min ago
     Docs: https://kubernetes.io/docs/home/

Update config 

$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

 

+ It isn't working, even renew certs.

Copy kubelet-client-current.pem from working control plane node to failed node.

$ scp -r fk8smaster02:/var/lib/kubelet/pki/kubelet-client-current.pem /var/lib/kubelet/pki/kubelet-client-current.pem

Restart kubelet

$ sudo systemctl restart kubelet

Approve certificate

$ kubectl get csr
NAME                                                AGE    SIGNERNAME                                    REQUESTOR                      CONDITION
csr-cplh4                                           12m    kubernetes.io/kube-apiserver-client-kubelet   system:node:k8smaster02        Pending

$ kubectl certificate approve csr-cplh4

$ kubectl get csr
NAME                                                AGE    SIGNERNAME                                    REQUESTOR                      CONDITION
csr-cplh4                                           12m    kubernetes.io/kube-apiserver-client-kubelet   system:node:k8smaster02        Approved,Issued

Check cert rotation working.

$ sudo journalctl -qu kubelet -n 100
Mar 14 13:51:22 k8smaster01 kubelet[6477]: I0314 13:51:22.385218    6477 transport.go:132] certificate rotation detected, shutting down client connections to start using new credentials
Mar 14 13:51:27 k8smaster01 kubelet[6477]: I0314 13:51:27.239652    6477 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
Mar 14 13:51:27 k8smaster01 kubelet[6477]: I0314 13:51:27.240925    6477 kubelet_node_status.go:70] Attempting to register node k8smaster01
Mar 14 13:51:27 k8smaster01 kubelet[6477]: I0314 13:51:27.251892    6477 kubelet_node_status.go:112] Node k8smaster01 was previously registered
Mar 14 13:51:27 k8smaster01 kubelet[6477]: I0314 13:51:27.251939    6477 kubelet_node_status.go:73] Successfully registered node k8smaster01

/var/lib/kubelet/pki$ ls -al
total 28
drwxr-xr-x 3 root root 4096 Mar 14 13:51 .
drwxr-xr-x 8 root root 4096 Feb 15  2021 ..
-rw------- 1 root root 1078 Mar 14 13:50 kubelet-client-2021-02-15-13-26-08.pem
-rw------- 1 root root 1078 Mar 14 13:51 kubelet-client-2022-03-14-13-51-20.pem
lrwxrwxrwx 1 root root   59 Mar 14 13:51 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2022-03-14-13-51-20.pem
-rw-r--r-- 1 root root 2217 Feb 15  2021 kubelet.crt
-rw------- 1 root root 1675 Feb 15  2021 kubelet.key