Kubernetes CoreDNS & systemd-resolved Config

💡 Kubernetes CoreDNS & systemd-resolved Config — Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for Kubernetes CoreDNS & systemd-resolved Config.

Reference
Glossary DB:
Siyoung’s Notion:
troubleshooting:
cmd: ,

References and Guidelines (NOTE)

CoreDNS operation when using systemd-resolved on host

HostNode:

systemd-resolved is fixed as /etc/resolv.conf = 127.0.0.53loopback address.
The actual resolve defined in /etc/systemd/resolved.conf is /var/run/systemd/resolve/resolv.conf.

Kubelet:

Default mounts /etc/resolv.conf on HostNode.
Kublet cannot access the loopback address 127.0.0.53.

References and Guidelines (NOTE)

Kubernetes:

Kubelet Configuration (v1beta1)
--resolv-conf=/var/run/systemd/resolve/resolv.conf

k3s:

bash

k3s server --resolv-conf $value
curl -sfL https://get.k3s.io | sh -s - server --resolv-conf $value

References and Guidelines (NOTE)

Worker node must be at least m1.large (memory 16GB)

Package Repository

https://storage.googleapis.com/kubernetes-release/release/v1.24.6/bin/linux/amd64/kubectl
https://storage.googleapis.com/kubernetes-release/release/v1.24.6/bin/linux/amd64/kubeadm
https://storage.googleapis.com/kubernetes-release/release/v1.24.6/bin/linux/amd64/kubelet

Building Kubespray

Manual installation

RHEL package

bash

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

Ready for Debian

shell

VERSION=1.24.2-00
POD_NETWORK_CIDR=10.233.64.0/16

swapoff -a 

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter


cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system


apt-get update
apt-get install -y apt-transport-https ca-certificates curl

#
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
#or
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#

apt-get update

apt-cache madison kubelet 
apt-get install -y kubectl=${VERSION} kubeadm=${VERSION} kubelet=${VERSION}
systemctl enable --now kubelet

apt-get install \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg \
  lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y containerd.io

# 환경설정 파일 [기본값]으로 초기화
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# config.toml 파일에서 SystemdCgroup 설정값 확인 및 변경 🔥
# SystemdCgroup 값이 false 라면 true 로 변경해야 함 🔥
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# containerd.service 변경사항 적용
sudo systemctl restart containerd

Cluster construction

bash

systemctl enable containerd
systemctl restart containerd

#kubeadm init --control-plane-endpoint "172.16.50.100:6443" \
#--upload-certs \
#--pod-network-cidr="10.233.64.0/16"
kubeadm init --control-plane-endpoint "$(hostname -i):6443" --upload-certs --service-cidr="10.96.0.0/16" --pod-network-cidr="10.233.0.0/16"

Containerd Insecure Settings

Communication module

bash

wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate
vi calico.yaml

yaml

- name: CALICO_IPV4POOL_CIDR
     value: "10.233.64.0/16"  # svc를 확인하여 환경에 맞는 pod subnet으로 등록

bash

k apply -f calico.yaml

Common convenience settings

bash

# completion
sudo yum -y install bash-completion
logout

source <(kubectl completion bash)

Cluster initialization

bash

systemctl restart containerd

kubeadm reset # insert 'y'

systemctl stop etcd
systemctl stop kubelet

rm -rf /var/lib/etcd

kill $(pidof etcd)
kill $(pidof kubelet)
kill $(pidof bird)
kill $(pidof kube-apiserver)
kill $(pidof kube-scheduler)
kill $(pidof kube-controller)
kill $(pidof calico-node)
kill $(pidof kube-proxy)

remove cni

bash

rm -rf /etc/cni/net.d/*

iptables -L
iptables -S | grep -oP '(?<!^:)cali-[^ ]+' | wc -l
for i in `iptables -L |grep cali|awk '{print $2}'`; do iptables -F $i && iptables -X $i;  done
iptables -S | grep -oP '(?<!^:)cali-[^ ]+' | wc -l
ipvsadm --clear

rm -f $HOME/.kube/config

Reinstall

master 1

bash

systemctl enable kubelet
systemctl enable containerd
systemctl restart containerd

#kubeadm init --control-plane-endpoint "192.168.50.10:6443" \
#--upload-certs \
#--service-cidr="10.233.0.0/16" \
#--pod-network-cidr="10.233.64.0/16"
kubeadm init --control-plane-endpoint "$(hostname -i):6443" --upload-certs --service-cidr="10.96.0.0/16" --pod-network-cidr="10.233.0.0/16"
kubectl apply -f calico.yaml

controlplans

bash

systemctl enable kubelet
systemctl enable containerd
systemctl restart containerd

kubeadm join 192.168.50.10:6443 --token sr24u6.o9jsr27l6o5zbe94 \
        --discovery-token-ca-cert-hash sha256:3a88fc4a4f45519b682c20a192a2c75fc18c75c05e9a5613fa3c29f223cd7abc \
        --control-plane --certificate-key 4035d8104fe0f0bfba6b466ccb9302611fcc4fd65f11a9a6d3e27858b49257a3

worker

bash

systemctl enable kubelet
systemctl enable containerd
systemctl restart containerd

kubeadm join 192.168.50.10:6443 --token sr24u6.o9jsr27l6o5zbe94 \
        --discovery-token-ca-cert-hash sha256:3a88fc4a4f45519b682c20a192a2c75fc18c75c05e9a5613fa3c29f223cd7abc

Reinstall binary

bash

apt-get remove kubectl kubeadm kubelet
sudo apt-get install -y kubelet=1.24* kubeadm=1.24* kubectl=1.24*

#apt에서 만료된 경우 직접 다운로드
wget https://dl.k8s.io/v1.23.7/kubernetes-client-linux-amd64.tar.gz
wget https://dl.k8s.io/v1.23.7/kubernetes-server-linux-amd64.tar.gz
wget https://dl.k8s.io/v1.23.7/kubernetes-node-linux-amd64.tar.gz

kubelet

bash

kubelet \
--container-runtime=remote \
--container-runtime-endpoint=unix:///run/containerd/containerd.sock &

Installation with CRI-O

bash

sudo -i
export OS=xUbuntu_22.04 # OS 버전
export VERSION=1.24     # cri-o 버전

echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -

sudo apt-get update

sudo apt-get install -y cri-o cri-o-runc

sudo systemctl daemon-reload
sudo systemctl enable crio --now
sudo systemctl status crio


kubeadm init --cri-socket=/var/run/crio/crio.sock --control-plane-endpoint "$(hostname -i):6443" --upload-certs --service-cidr="10.96.0.0/16" --pod-network-cidr="10.233.0.0/16"

Set kubeconfig for account

bash

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install kubernetes metric API

bash

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
mv components.yaml metricapis.yaml
vi metricapis.yaml
---
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls #추가
---
kubectl apply -f metricapis.yaml

history

yaml

1  yes
    2  vi /etc/netplan/50-cloud-init.yaml 
    3  netplan apply
    4  apt update
    5  TMOUT=0
    6  exit
    7  ssh [email protected]
    8  ssh [email protected]
    9  ssh [email protected]
   10  ssh [email protected]
   11  ssh [email protected]
   12  TMOUT=0
   13  tmux
   14  echo "$(hostname -I) $(hostname)"
   15  echo "$(hostname -I) $(hostname)" >> /etc/hosts
   16  vi /etc/hosts
   17  ls
   18  swapoff -a 
   19  cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
   20  overlay
   21  br_netfilter
   22  EOF
   23  modprobe overlay
   24  modprobe br_netfilter
   25  cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
   26  net.bridge.bridge-nf-call-iptables  = 1
   27  net.ipv4.ip_forward                 = 1
   28  net.bridge.bridge-nf-call-ip6tables = 1
   29  EOF
   30  sudo sysctl --system
   31  apt-get update
   32  apt-get install   apt-transport-https   ca-certificates   curl   gnupg   lsb-release
   33  apt-get install   apt-transport-https   ca-certificates   curl   gnupg   lsb-release
   34  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
   35  echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   36  apt-get update
   37  apt-get install -y containerd.io
   38  cd /etc/apt/keyrings/
   39  ls
   40  exit
   41  curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
   42  echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.23/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
   43  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.23/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   44  echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
   45  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   46  apt-cache madison kubelet 
   47  apt-get update
   48  apt-cache madison kubelet 
   49  export VERSION = "1.24.5-1.1"
   50  export VERSION="1.24.5-1.1"
   51  sudo apt-get install -y kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION kubernetes-cni
   52  sudo apt-mark hold kubelet kubeadm kubectl
   53  containerd config default | sudo tee /etc/containerd/config.toml
   54  vi /etc/containerd/config.toml 
   55  ip a
   56  systemctl enable --now kubelet
   57  kubeadm init --control-plane-endpoint "10.0.X.X" --upload-certs --pod-network-cidr "10.233.0.0/16" --kubernetes-version "v1.24.5" --cri-socket  /run/containerd/containerd.sock
   58  systemctl enable containerd
   59  systemctl restart containerd
   60  kubeadm reset
   61  kubeadm init --control-plane-endpoint "10.0.X.X" --upload-certs --pod-network-cidr "10.233.0.0/16" --kubernetes-version "v1.24.5" --cri-socket  /run/containerd/containerd.sock
   62  watch k get pods -A
   63  watch kubectl get pods -A
   64  mkdir -p $HOME/.kube
   65  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   66  sudo chown $(id -u):$(id -g) $HOME/.kube/config
   67  export KUBECONFIG=/etc/kubernetes/admin.conf
   68  watch kubectl get pods -A
   69  exit
   70  history
   71  history > history.txt

Fredric News

Leave a Reply Cancel reply

Kubernetes CoreDNS & systemd-resolved Config

Package Repository

Building Kubespray

Manual installation

RHEL package

Ready for Debian

Cluster construction

Containerd Insecure Settings

Communication module

Common convenience settings

Cluster initialization

remove cni

Reinstall

master 1

controlplans

worker

Reinstall binary

kubelet

Installation with CRI-O

Set kubeconfig for account

Install kubernetes metric API

Comments

Leave a Reply Cancel reply