Automated script

• – When renewing after expiration

  Attempt to bootstrap

  We will summarize the recovery sequence assuming the worst case scenario where bootstrapping is not set and both the control plane and kubelet certificates have already expired.

  In this case, kubectl will not work because the API server is dead. Therefore, you must save the control plane first and then sequentially revive the kubelets of the nodes.

  —

  ### Step 1: Renew control plane (Master) certificate

  First, the certificates used by the API server, scheduler, and controller manager must be renewed to revive the “brain” of the cluster.

  1. Check current expiration statusBash

  kubeadm certs check-expiration

  2. Renew all control plane certificatesBash

  kubeadm certs renew all

  3. Restart Static Pods
  The certificate file has been updated, but kube-apiserver, which is already in memory, contains the old certificate. Forces a restart of Bash to read the file again.

  # If you move the manifest file for a while and then import it again, the container will restart.
mv /etc/kubernetes/manifests/*.yaml /tmp/
sleep 10
mv /tmp/*.yaml /etc/kubernetes/manifests/

  4. Update the admin configuration file (admin.conf)Replace my local .kube/config with the new file so that I can use kubectl.Bash

  cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

  —

  ### Step 2: Prepare for manual renewal of Kubelet certificate (if bootstrap is not set)

  Without bootstrapping (bootstrap-kubelet.conf), the kubelet has no way to obtain the certificate itself. In this case, you must use the function of kubeadm to temporarily open an authentication path.

  1. Grant RBAC permission (run on master)
  Temporarily grant permission for the kubelet to send a certificate signing request (CSR). (Can be omitted if already set) Bash

  kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

  2. Generate a new bootstrap tokenBash

  kubeadm token create --print-join-command

  Copy the token generated here.

  —

  ### Step 3: Kubelet recovery on each node (Worker/Master)

  Now go to each node and have it issue a new certificate, ignoring the expired kubelet-client-current.pem.

  1. Stop Kubelet and remove existing certificateBash

  systemctl stop kubelet
# Remove expired certificates and symbolic links
rm /var/lib/kubelet/pki/kubelet-client-*.pem
rm /var/lib/kubelet/pki/kubelet-client-current.pem

  2. Regenerate kubelet.conf (Important)
  If bootstrapping is not performed, the certificate information in /etc/kubernetes/kubelet.conf may also have expired. Reconfigure this file using kubeadm.Bash

  # Run on node (using token and address received from master)
kubeadm join :6443 --token --discovery-token-ca-cert-hash sha256: --control-plane (add only if master)

  If the node is already joined, you may need the --force option, or you may need to back up and run the existing .conf files in /etc/kubernetes/.

  3. Starting KubeletBash

  systemctl start kubelet

  —

  ### Step 4: CSR approval (run on master)

  The Kubelet comes to life and sends a request to the master, saying “I’m new, please authenticate me.”

  1. Check request listBash

  kubectl get csr

  2. ApproveBash

  # Approval of all CSRs in Pending status
kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

  —

  ### Summary order

  1. Master: kubeadm certs renew all and then restart the Static Pod.
  2. Master: Update admin.conf to revive kubectl.
  3. Master: Secure a new path with kubeadm token create.
  4. Node: Delete the existing kubelet-client-*.pem.
  5. Node: Attempt re-authentication with the kubeadm join command (or update relevant settings).
  6. Final approval with Master: kubectl certificate approve.

  After completing this procedure, kubectl get nodes will return all nodes to the Ready state. Version 1.23 is a stable version of kubeadm‘s certificate management commands, so the above commands should work well.
  

  Ignore bootstrapping

  The key to manually recovering expired control planes and kubelets without using bootstrapping (Token-based auto-join) is \*\*”creating the certificate files yourself and updating kubeconfig manually”\*\*

  This method does not require kubeadm join or token, but only uses kubeadm‘s certificate renewal function and file copy.

  —

  ### Step 1: Manually renew the control plane (Master) certificate

  First of all, you need to save the ‘Certification Authority (CA)’ and API server related certificates.

  1. Control plane certificate bulk renewalBash

  kubeadm certs renew all

  2. Restart Static Pods (Load Certificate)Bash

  # Move the manifest for a while and copy it to induce a container restart
mv /etc/kubernetes/manifests/*.yaml /tmp/
sleep 5
mv /tmp/*.yaml /etc/kubernetes/manifests/

  3. Update kubeconfig for administratorBash

  cp /etc/kubernetes/admin.conf $HOME/.kube/config

  —

  ### Step 2: Manually update the configuration file (kubelet.conf) for Kubelet

  If you do not use bootstrapping, you must manually update the client certificate data in the /etc/kubernetes/kubelet.conf file. kubeadm provides dedicated commands for this process.

  \[Performed on all nodes (including Master)\]

  1. Renew certificate in kubelet configuration fileBash

  kubeadm init phase kubeconfig kubelet --config /etc/kubernetes/kubeadm-config.yaml (or can be omitted if using default)

  This command creates a new file /etc/kubernetes/kubelet.conf and includes the new embedded certificate inside it.

  2. Clean up existing individual PEM filesBash

  If you do not use bootstrapping, the data in /etc/kubernetes/kubelet.conf will be referenced first, not kubelet-client-current.pem in /var/lib/kubelet/pki/. Leave existing files aside to avoid conflicts.

  rm -f /var/lib/kubelet/pki/kubelet-client-*.pem
rm -f /var/lib/kubelet/pki/kubelet-client-current.pem

  —

  ### Step 3: Restart Kubelet services and check permissions

  Now let the kubelet on each node read the new kubelet.conf and communicate with the API server.

  1. Restart KubeletBash

  systemctl restart kubelet

  2. CSR manual approval (if necessary)Bash

  In some environments, the kubelet may request additional serving certificates even after updating kubelet.conf. The master confirms and approves.

  kubectl get csr
kubectl certificate approve

  —

  ### Step 4: How do I get kubelet-client-current.pem back?

  The /var/lib/kubelet/pki/kubelet-client-current.pem file that the user mentioned at the beginning of the question is actually a file created when Kubelet’s Rotation feature is turned on.

  If you want to keep this file again without bootstrapping, your kubelet configuration (config.yaml) with kubelet.conf updated must include:

  – rotateCertificates: true
  – serverTLSBootstrap: true (optional)

  ### Organized command flow chart

  Step	Execution Location	Command
  1. CP Renewal	Master	kubeadm certs renew all
  2. Update settings	all nodes	kubeadm init phase kubeconfig kubelet
  3. Organize your files	all nodes	rm -f /var/lib/kubelet/pki/kubelet-client*
  4. Restart	all nodes	systemctl restart kubelet

  This way, you can manually replace the certificate and operate normally through the static configuration file created by kubeadm without going through the token-based bootstrapping (bootstrap-kubelet.conf) process.

  When running kubeadm init phase, does an error occur saying that a specific configuration file cannot be found? Please let me know if you need a kubeadm-config dump for that cluster.
  

Basis

# kubectl 명령어 사용시
Unable to connect to the server: x509: certificate has expired or is not yet valid.

Check certificate expiration time

sudo kubeadm certs check-expiration
#>>> 출력
#[check-expiration] Reading configuration from the cluster...
#[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
#W0818 13:43:59.620410 4001964 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [10.233.0.3]
#
#CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
#admin.conf                 Aug 17, 2024 01:21 UTC   364d            ca                      no
#apiserver                  Aug 17, 2024 01:21 UTC   364d            ca                      no
#apiserver-kubelet-client   Aug 17, 2024 01:21 UTC   364d            ca                      no
#controller-manager.conf    Aug 17, 2024 01:21 UTC   364d            ca                      no
#front-proxy-client         Aug 17, 2024 01:21 UTC   364d            front-proxy-ca          no
#scheduler.conf             Aug 17, 2024 01:21 UTC   364d            ca                      no
#
#CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
#ca                      Aug 14, 2032 13:09 UTC   8y              no
#front-proxy-ca          Aug 14, 2032 13:09 UTC   8y              no
#<<<

Perform

Tasks

# 모든 master 노드
kubeadm certs renew all #or 다른 방법으로 갱신

# 갱신된 kubeapiserver config CLI 인증에 사용
mv ~/.kube/config ~/.kube/config-backup
cp /etc/kubernetes/admin.conf ~/.kube/config

# !! kubeconfig를 커스텀해서 사용중이라면 전체 복사/붙여넣기가 아니라 아래의 내용만 갱신하면 됩니다.
#    client-certificate-data: <crt-sting>
#    client-key-data: <key-string>

Additional tasks

• Check that admin.conf is actually reflected
• Confirm that kubelet.conf is not reflected.
• Need to check scheduler.conf, controller-manager.conf

There is a possibility of unauthorized occurrence because the static pod is not restarted.

• master nodes

W0820 05:54:38.069851       1 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.PersistentVolumeClaim: Unauthorized
  E0820 05:54:38.069907       1 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Unauthorized
  W0820 05:54:38.943924       1 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.StatefulSet: Unauthorized
  E0820 05:54:38.943996       1 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Unauthorized
  W0820 05:54:39.911036       1 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSINode: Unauthorized
  E0820 05:54:39.911123       1 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Unauthorized
  W0820 05:54:41.927021       1 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Pod: Unauthorized
  E0820 05:54:41.927132       1 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized

• How to reissue a certificate (highly unlikely to be needed)

# 인증서 갱신 후 kube-scheduler, kube-controller-manager 인증에러 발생
    # 클라이언트인증서 미사용으로 인함

    # 백업 후 삭제
    mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.bak
    mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.bak
    mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak

    # 삭제되어있어야 신규발급됨
    kubeadm init phase kubeconfig scheduler
    kubeadm init phase kubeconfig controller-manager
    kubeadm init phase kubeconfig kubelet

• How to apply a reissued certificate

# kubelet 재시작
    mkdir ~/kubelet-backup
    mv /var/lib/kubelet/pki/* ~/kubelet-backup
    systemctl restart kubelet

    cd /etc/kubernetes/manifests
    # 기존 프로세스 종료
    mv kube-controller-manager.yaml ..
    mv kube-scheduler.yaml ..
    # 중지 확인
    kubectl -n kube-system get pods -l 'component in (kube-controller-manager,kube-scheduler)'
    # 재시작
    mv ../kube-controller-manager.yaml .
    mv ../kube-scheduler.yaml .

• worker nodes

# 아래와같이 파일로명시되어있으면 자동갱신 되고 있을 가능성이 높음. 마스터노드도 해당설정되어있으면 조치X 
  grep client-certificate /etc/kubernetes/kubelet.conf
      client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem

Olds

# [[ master node ]]
   ## kube pki backup
   sudo mkdir -p $HOME/fcik8s-old-certs/pki
   sudo mkdir -p $HOME/fcik8s-old-certs/kubelet
   sudo mkdir -p $HOME/fcik8s-old-certs/.kube
   sudo cp -p /etc/kubernetes/pki/*.* $HOME/fcik8s-old-certs/pki
   sudo cp -p /etc/kubernetes/*.conf $HOME/fcik8s-old-certs
   sudo cp -p /var/lib/kubelet/pki/* $HOME/fcik8s-old-certs/kubelet
   sudo cp -p ~/.kube/config $HOME/fcik8s-old-certs/.kube/.

sudo kubeadm certs renew all --config=/etc/kubernetes/kubeadm-config.yaml
   >>>
   W0818 10:21:42.676045 3888005 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [10.233.0.3]
   certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
   certificate for serving the Kubernetes API renewed
   certificate for the API server to connect to kubelet renewed
   certificate embedded in the kubeconfig file for the controller manager to use renewed
   certificate for the front proxy client renewed
   certificate embedded in the kubeconfig file for the scheduler manager to use renewed

   Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
   <<<
   # 확인
   sudo kubeadm certs check-expiration

   # kublet 재시작
   #systemctl daemon-reload&&systemctl restart kubelet

   # control-plane components 종료
   sudo cp -pr /etc/kubernetes/manifest $HOME/fcik8s-old-certs
   sudo rm /etc/kubernetes/manifest/*

   # componets container 프로세스 확인
   sudo nerdctl ps

   # control-plane components 재시작
   sudo cp -p $HOME/fcik8s-old-certs/manifest/* /etc/kubernetes/manifest

   kubectl get pods -n kube-system -l component=kube-apiserver
   kubectl get pods -n kube-system -l component=kube-controller-manager
   kubectl get pods -n kube-system -l component=kube-scheduler
   kubectl get pods -n kube-system -l component=etcd

   kubectl get nodes

# log 확인
   journalctl -u kubelet -f
   journalctl -u etcd -f
   journalctl -u containerd -f
   kubectl logs -n kube-system -l component=kube-apiserver
   kubectl logs -n kube-system -l component=kube-controller-manager
   kubectl logs -n kube-system -l component=kube-scheduler
   kubectl logs -n kube-system -l component=etcd

   #!! kubeadm 1.17 이전의 버전에서 kubeadm init 으로 작성된 Node 에는 kubelet.conf 파일을 수동으로 수정해야 하는 버그가 있다.
   ## 최대 5분동안 적용여부 모니터링 필요
   kubectl get nodes #정상조회 불가하다면 kubelet 구성 혹은 kubeconfig 파일 확인 필요

   ### error: You must be logged in to the server (Unauthorized) 발생시
   ### -> [2] 조치사항 확인

   ### [1] kubelet.conf 미갱신의 경우
   sudo diff $HOME/fcik8s-old-certs/kubelet.conf /etc/kubernetes/kubelet.conf #차이가 없으면 미갱신

   ## 사용 전 분석 필요
   #cd /etc/kubernetes
   sudo kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf

   # kubelet.conf에 파일로 명시한다면
   ### [1-2] v1.23.7에서 확인시 갱신하지 않은것이 정상으로 보임
   sudo diff $HOME/fcik8s-old-certs/kubelet.conf /etc/kubernetes/kubelet.conf
   sudo diff $HOME/fcik8s-old-certs/kubelet/kubelet-client-current.pem /var/lib/kubelet/pki/kubelet-client-current.pem
   sudo diff $HOME/fcik8s-old-certs/kubelet/kubelet.key /var/lib/kubelet/pki/kubelet.key
   sudo diff $HOME/fcik8s-old-certs/kubelet/kubelet.crt /var/lib/kubelet/pki/kubelet.crt

   ### [2] kubeconfig 미갱신의 경우
   sudo diff ~/.kube/config $HOME/fcik8s-old-certs/.kube/config #차이가 없으면 미갱신
   cat /etc/kubernetes/kubelet.conf # 내용 참조하여 삽입. 파일의 경우 [여기](<https://www.notion.so/cmd-cm-base64-d7e4a9892c6a4eaf8abc0e9ec1a13684?pvs=21>) 참조
   client-certificate -> ~/.kube./config:client-certificate-data
   client-key -> ~/.kube./config:client-key-data

   systemctl daemon-reload&&systemctl restart kubelet
   kubectl get nodes

# [[ master node ]]
   ### **1. 인증서가 존재하는 pki 디렉토리를 백업합니다.**
   cp -f /etc/kubernetes/pki /etc/kubernetes/pki-backup

   ### **2. 인증서를 일괄적으로 업데이트 합니다.**
   kubeadm certs renew all

   ### **3. 갱신 여부를 다시 체크합니다.**
   [12:56:54][danawa@kube3 /etc/kubernetes/pki]
   $ openssl x509 -in apiserver.crt -noout -text |grep ' Not '
               Not Before: Mar 25 08:12:56 2020 GMT
               Not After : Mar 28 02:26:40 2023 GMT

   #정상적으로 1년 연장 되었습니다.

   ### **4. 사용자 구성을 업데이트 하기 위해 kubernetes 디렉토리를 백업합니다.**
   cp -f /etc/kubernetes /etc/kubernetes-backup

   ### **5. 사용자 구성을 업데이트합니다.**
   kubeadm kubeconfig user --client-name=admin --config=/etc/kubernetes/kubeadm-config.yaml
   kubeadm kubeconfig user --org system:masters --client-name=kubernetes-admin --config=/etc/kubernetes/kubeadm-config.yaml > /etc/kubernetes/admin.conf
   kubeadm kubeconfig user --client-name system:kube-controller-manager --config=/etc/kubernetes/kubeadm-config.yaml > /etc/kubernetes/controller-manager.conf
   kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config=/etc/kubernetes/kubeadm-config.yaml > /etc/kubernetes/kubelet.conf
   kubeadm kubeconfig user --client-name system:kube-scheduler --config=/etc/kubernetes/kubeadm-config.yaml > /etc/kubernetes/scheduler.conf

   ### **6. /root/.kube/config를 교체합니다.**
   cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   chown $(id -u):$(id -g) $HOME/.kube/config

   ### **7. 서비스를 재시작합니다.**
   systemctl daemon-reload&&systemctl restart kubelet

   ### csr approve
   kubectl certificate approve $(kubectl get csr | grep Pending | awk '{print $1}' |  sed -z 's/[\\n\\r]/ /g')

   # [[ worker node ]]
   ## master에서 복사한 후 붙여넣기
   vi /etc/kubernetes/kubelet.conf
   >>>
   # 아래 ip locahost로 수정
   server: <https://localhost:6443>
   <<<
   systemctl restart kubelet

# systemctl 서비스에서 사용할 경우
   systemctl restart kube-apiserver
   systemctl restart kube-scheduler

   # 도커 컨테이너를 사용할 경우
   docker restart [apiserver 컨테이너]
   docker restart [scheduler 컨테이너]
   docker restart [controller 컨테이너]