Fredric News

쿠버네티스 통신 장애 트러블슈팅: kubelet 무한 재부팅

💡 핵심 요약 (Executive Summary)

본 가이드는 쿠버네티스(Kubernetes) 핵심 가이드 및 실전 아키텍처의 핵심 개념, 실전 구성 명령어 및 아키텍처 베스트 프랙티스를 다룹니다. 클라우드 네이티브 및 인프라 엔지니어링 실무에 즉시 활용 가능하도록 체계적으로 정리했습니다.

📑 Table of Contents
최신버전 containerd 사용시 etcd, kube-apiserver 등의 controlplane 무한 재부팅 이슈
조치
  • CRI-O 사용으로 변경
최신버전 k8s 설치시 Nameserver 한도 (3) 초과 문제
근거
bash
systemctl status kubelet
>>>
Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 8.8.8.8 10.0.X.X 10.0.X.X
<<<
조치
bash
cp /var/run/systemd/resolve/resolv.conf /var/run/systemd/resolve/resolv-static.conf
vi /var/lib/kubelet/config.yaml
---
resolvConf: /var/run/systemd/resolve/resolv-static.conf
---
#3개로 줄이면 에러 사라짐

systemctl restart kubelet
Reference & Guidelines (NOTE)

그러나 이것은 클러스터구성의 치명적인 문제가 아니므로 무시해도 됨. 만약 kubelet 무한재부팅 문제인 경우 다른문제

Scrape
클러스터 초기화 이후 파드 내 DNS 서버와 서비스포트 다른 문제
근거
  1. 신규 파드 생성시 컨테이너 내 /etc/resolv.conf > nameserver 10.233.0.3
  2. kubeadm으로 초기화 후 kubectl get svc -n kube-system > DNS service IP: 10.233.0.10
원인
  • kubespray로 설치된 클러스터라서 kubeadm으로 초기화시 설치시 사용한 manifest를 충분히 참조하지 않는것으로 예상.
조치
bash
#1. 구성 파일 위치 찾기
cd /etc/kubernetes/
find . -type f | xargs grep '10.233.0.3'
>>>
./nodelocaldns-config.yml:        forward . 10.233.0.3 {
./nodelocaldns-config.yml:        forward . 10.233.0.3 {
./nodelocaldns-config.yml:        forward . 10.233.0.3 {
./coredns-svc.yml:  clusterIP: 10.233.0.3
./kubelet-config.yaml:- 10.233.0.3
<<<

#방법 1. kubelet-config.yaml 및 nodelocaldns-config.yml 변경 (검증 필요)
vi kubelet-config.yaml
vi nodelocaldns-config.yml
kubectl apply -f nodelocaldns-config.yml

#방법 2. 기존 svc 삭제 후 apply coredns-svc.yml (검증완료)
kubectl delete svc -n kube-system -l k8s-app=kube-dns
kubectl apply -f coredns-svc.yml
인증서 만료
kubeadm
  • /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
🔗 https://da-nika.tistory.com/198
  • /proc/sys/net/ipv4/ip_forward contents are not set to 1
🔗 https://tptrmfwk.tistory.com/24
  • Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
    time="2022-12-07T04:11:46Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeServic
🔗 https://musclebear.tistory.com/165
bash
# /etc/containerd/config.toml 수정 후
  systemctl enable containerd
  systemctl restart containerd
kubectl
  • kubectl → The connection to the server localhost:8080 was refused - did you specify the right host or port?
bash
mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  • 컨테이너 이미지 이슈 : harbor의 https 443 포트 찾는 경우
    → (모든 k8s master/worker 노드에 insecure 설정: harbor주소)
젠킨스 ServiceAccount
근거
yaml
2023-01-31 12:31:02.405+0000 [id=32]    SEVERE  c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: Failed to initialise k8s secret provider, secrets from Kubernetes will not be available
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1:443/api/v1/namespaces/infra/secrets?labelSelector=jenkins.io%2Fcredentials-type. Message: Forbidden!Configured service account doesn't have access. Service account may have been rev
oked. secrets is forbidden: User "system:serviceaccount:infra:jenkins" cannot list resource "secrets" in API group "" in the namespace "infra".
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:742)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:722)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:671)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:601)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133)
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1:443/api/v1/namespaces/infra/secrets?labelSelector=jenkins.io%2Fcredentials-type. Message: Forbidden!Configured service account doesn't have access. Service account may have
been revoked. secrets is forbidden: User "system:serviceaccount:infra:jenkins" cannot list resource "secrets" in API group "" in the namespace "infra".
        at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:536)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:418)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:383)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:93)
        at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider.startWatchingForSecrets(KubernetesCredentialProvider.java:124)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1161)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:222)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:121)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
원인: clusterRole, clusterRoleBinding
Auth
근거
yaml
+ kubectl rollout restart deployment openstack-api -n portal
error: the server doesn't have a resource type "deployment"
조치
yaml
clusterRole
Completion
근거
bash
source <(kubectl completion bash)
#&
kubectl ((tab)) -->
>>>
bash: _get_comp_words_by_ref: command not found
<<<
조치
bash
sudo yum -y install bash-completion
sudo dnf -y install bash-completion
sudo apt -y install bash-completion
#&
logout #for session

Posted:

/

Category:

Author:

fredric

Tags:

, , , , , , , , , , , , , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *