Kubernetes Node Communication & Kubelet

💡 Kubernetes Node Communication & Kubelet — Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for Kubernetes Node Communication & Kubelet.

Infinite reboot issue of controlplanes such as etcd and kube-apiserver when using the latest version of containerd

Action

Change to use CRI-O

Nameserver limit (3) exceeded when installing latest version k8s

Rationale

bash

systemctl status kubelet
>>>
Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 8.8.8.8 10.0.X.X 10.0.X.X
<<<

Action

bash

cp /var/run/systemd/resolve/resolv.conf /var/run/systemd/resolve/resolv-static.conf
vi /var/lib/kubelet/config.yaml
---
resolvConf: /var/run/systemd/resolve/resolv-static.conf
---
#3개로 줄이면 에러 사라짐

systemctl restart kubelet

References and Guidelines (NOTE)

However, this is not a critical problem in cluster configuration, so it can be ignored. If the kubelet infinite reboot problem is another problem

Scrape

DNS server and service port in pod different issues after cluster initialization

Rationale

When creating a new pod, within the container /etc/resolv.conf > nameserver 10.233.0.3
After initialization with kubeadm kubectl get svc -n kube-system > DNS service IP: 10.233.0.10

Cause

Because this is a cluster installed with kubespray, it is expected that the manifest used during installation is not sufficiently referenced when initializing with kubeadm.

Action

bash

#1. 구성 파일 위치 찾기
cd /etc/kubernetes/
find . -type f | xargs grep '10.233.0.3'
>>>
./nodelocaldns-config.yml:        forward . 10.233.0.3 {
./nodelocaldns-config.yml:        forward . 10.233.0.3 {
./nodelocaldns-config.yml:        forward . 10.233.0.3 {
./coredns-svc.yml:  clusterIP: 10.233.0.3
./kubelet-config.yaml:- 10.233.0.3
<<<

#방법 1. kubelet-config.yaml 및 nodelocaldns-config.yml 변경 (검증 필요)
vi kubelet-config.yaml
vi nodelocaldns-config.yml
kubectl apply -f nodelocaldns-config.yml

#방법 2. 기존 svc 삭제 후 apply coredns-svc.yml (검증완료)
kubectl delete svc -n kube-system -l k8s-app=kube-dns
kubectl apply -f coredns-svc.yml

Certificate expiration

kubeadm

/proc/sys/net/bridge/bridge-nf-call-iptables does not exist

🔗 https://da-nika.tistory.com/198

/proc/sys/net/ipv4/ip_forward contents are not set to 1

🔗 https://tptrmfwk.tistory.com/24

Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" time="2022-12-07T04:11:46Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeServic

🔗 https://musclebear.tistory.com/165

bash

# /etc/containerd/config.toml 수정 후
  systemctl enable containerd
  systemctl restart containerd

kubectl

kubectl → The connection to the server localhost:8080 was refused - did you specify the right host or port?

bash

mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Container image issue: When looking for harbor’s https 443 port
→ (insecure setting on all k8s master/worker nodes: harbor address)

Jenkins ServiceAccount

Rationale

yaml

2023-01-31 12:31:02.405+0000 [id=32]    SEVERE  c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: Failed to initialise k8s secret provider, secrets from Kubernetes will not be available
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1:443/api/v1/namespaces/infra/secrets?labelSelector=jenkins.io%2Fcredentials-type. Message: Forbidden!Configured service account doesn't have access. Service account may have been rev
oked. secrets is forbidden: User "system:serviceaccount:infra:jenkins" cannot list resource "secrets" in API group "" in the namespace "infra".
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:742)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:722)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:671)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:601)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133)
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.233.0.1:443/api/v1/namespaces/infra/secrets?labelSelector=jenkins.io%2Fcredentials-type. Message: Forbidden!Configured service account doesn't have access. Service account may have
been revoked. secrets is forbidden: User "system:serviceaccount:infra:jenkins" cannot list resource "secrets" in API group "" in the namespace "infra".
        at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:536)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:418)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:383)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:93)
        at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider.startWatchingForSecrets(KubernetesCredentialProvider.java:124)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1161)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:222)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:121)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Cause: clusterRole, clusterRoleBinding

Auth

Rationale

yaml

+ kubectl rollout restart deployment openstack-api -n portal
error: the server doesn't have a resource type "deployment"

Action

yaml

clusterRole

Completion

Rationale

bash

source <(kubectl completion bash)
#&
kubectl ((tab)) -->
>>>
bash: _get_comp_words_by_ref: command not found
<<<

Action

bash

sudo yum -y install bash-completion
sudo dnf -y install bash-completion
sudo apt -y install bash-completion
#&
logout #for session

Fredric News

Leave a Reply Cancel reply

Kubernetes Node Communication & Kubelet

Infinite reboot issue of controlplanes such as etcd and kube-apiserver when using the latest version of containerd

Action

Nameserver limit (3) exceeded when installing latest version k8s

Rationale

Action

Scrape

DNS server and service port in pod different issues after cluster initialization

Rationale

Cause

Action

Certificate expiration

kubeadm

kubectl

Jenkins ServiceAccount

Rationale

Cause: clusterRole, clusterRoleBinding

Auth

Rationale

Action

Completion

Rationale

Action

Comments

Leave a Reply Cancel reply