쿠버네티스 CoreDNS 동작 원리: systemd-resolved 환경

💡 핵심 요약 (Executive Summary)

본 가이드는 쿠버네티스(Kubernetes) 핵심 가이드 및 실전 아키텍처의 핵심 개념, 실전 구성 명령어 및 아키텍처 베스트 프랙티스를 다룹니다. 클라우드 네이티브 및 인프라 엔지니어링 실무에 즉시 활용 가능하도록 체계적으로 정리했습니다.

📑 Table of Contents

1. Package Repository
2. Kubespray 구축
3. 수동 설치
• RHEL package
• Debian 준비
• cluster 구축
• containerd Insecure 설정
• 통신모듈
4. 공통 편의 설정
• cluster 초기화
• 재설치
• 바이너리 재설치

Reference
Glossary DB:
시영의 Notion:
troubleshooting:
cmd: ,

Reference & Guidelines (NOTE)

Host에서 systemd-resolved 사용시 CoreDNS 동작

HostNode:

systemd-resolved는 /etc/resolv.conf = 127.0.0.53루프백주소로 고정한다.
실제 /etc/systemd/resolved.conf에서 정의된 리졸브는 /var/run/systemd/resolve/resolv.conf이다.

Kubelet:

default는 HostNode의 /etc/resolv.conf 를 마운트한다.
Kublet은 루프백주소 127.0.0.53에 접근할 수 없다.

Reference & Guidelines (NOTE)

Kubernetes:

Kubelet Configuration (v1beta1)
--resolv-conf=/var/run/systemd/resolve/resolv.conf

k3s:

bash

k3s server --resolv-conf $value
curl -sfL https://get.k3s.io | sh -s - server --resolv-conf $value

Reference & Guidelines (NOTE)

worker node는 최소 m1.large (memory 16GB)

Package Repository

https://storage.googleapis.com/kubernetes-release/release/v1.24.6/bin/linux/amd64/kubectl
https://storage.googleapis.com/kubernetes-release/release/v1.24.6/bin/linux/amd64/kubeadm
https://storage.googleapis.com/kubernetes-release/release/v1.24.6/bin/linux/amd64/kubelet

Kubespray 구축

수동 설치

RHEL package

bash

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

Debian 준비

shell

VERSION=1.24.2-00
POD_NETWORK_CIDR=10.233.64.0/16

swapoff -a 

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter


cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system


apt-get update
apt-get install -y apt-transport-https ca-certificates curl

#
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
#or
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#

apt-get update

apt-cache madison kubelet 
apt-get install -y kubectl=${VERSION} kubeadm=${VERSION} kubelet=${VERSION}
systemctl enable --now kubelet

apt-get install \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg \
  lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y containerd.io

# 환경설정 파일 [기본값]으로 초기화
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# config.toml 파일에서 SystemdCgroup 설정값 확인 및 변경 🔥
# SystemdCgroup 값이 false 라면 true 로 변경해야 함 🔥
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# containerd.service 변경사항 적용
sudo systemctl restart containerd

cluster 구축

bash

systemctl enable containerd
systemctl restart containerd

#kubeadm init --control-plane-endpoint "172.16.50.100:6443" \
#--upload-certs \
#--pod-network-cidr="10.233.64.0/16"
kubeadm init --control-plane-endpoint "$(hostname -i):6443" --upload-certs --service-cidr="10.96.0.0/16" --pod-network-cidr="10.233.0.0/16"

containerd Insecure 설정

통신모듈

bash

wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate
vi calico.yaml

yaml

- name: CALICO_IPV4POOL_CIDR
     value: "10.233.64.0/16"  # svc를 확인하여 환경에 맞는 pod subnet으로 등록

bash

k apply -f calico.yaml

공통 편의 설정

bash

# completion
sudo yum -y install bash-completion
logout

source <(kubectl completion bash)

cluster 초기화

bash

systemctl restart containerd

kubeadm reset # insert 'y'

systemctl stop etcd
systemctl stop kubelet

rm -rf /var/lib/etcd

kill $(pidof etcd)
kill $(pidof kubelet)
kill $(pidof bird)
kill $(pidof kube-apiserver)
kill $(pidof kube-scheduler)
kill $(pidof kube-controller)
kill $(pidof calico-node)
kill $(pidof kube-proxy)

cni 제거

bash

rm -rf /etc/cni/net.d/*

iptables -L
iptables -S | grep -oP '(?<!^:)cali-[^ ]+' | wc -l
for i in `iptables -L |grep cali|awk '{print $2}'`; do iptables -F $i && iptables -X $i;  done
iptables -S | grep -oP '(?<!^:)cali-[^ ]+' | wc -l
ipvsadm --clear

rm -f $HOME/.kube/config

재설치

master 1

bash

systemctl enable kubelet
systemctl enable containerd
systemctl restart containerd

#kubeadm init --control-plane-endpoint "192.168.50.10:6443" \
#--upload-certs \
#--service-cidr="10.233.0.0/16" \
#--pod-network-cidr="10.233.64.0/16"
kubeadm init --control-plane-endpoint "$(hostname -i):6443" --upload-certs --service-cidr="10.96.0.0/16" --pod-network-cidr="10.233.0.0/16"
kubectl apply -f calico.yaml

controlplans

bash

systemctl enable kubelet
systemctl enable containerd
systemctl restart containerd

kubeadm join 192.168.50.10:6443 --token sr24u6.o9jsr27l6o5zbe94 \
        --discovery-token-ca-cert-hash sha256:3a88fc4a4f45519b682c20a192a2c75fc18c75c05e9a5613fa3c29f223cd7abc \
        --control-plane --certificate-key 4035d8104fe0f0bfba6b466ccb9302611fcc4fd65f11a9a6d3e27858b49257a3

worker

bash

systemctl enable kubelet
systemctl enable containerd
systemctl restart containerd

kubeadm join 192.168.50.10:6443 --token sr24u6.o9jsr27l6o5zbe94 \
        --discovery-token-ca-cert-hash sha256:3a88fc4a4f45519b682c20a192a2c75fc18c75c05e9a5613fa3c29f223cd7abc

바이너리 재설치

bash

apt-get remove kubectl kubeadm kubelet
sudo apt-get install -y kubelet=1.24* kubeadm=1.24* kubectl=1.24*

#apt에서 만료된 경우 직접 다운로드
wget https://dl.k8s.io/v1.23.7/kubernetes-client-linux-amd64.tar.gz
wget https://dl.k8s.io/v1.23.7/kubernetes-server-linux-amd64.tar.gz
wget https://dl.k8s.io/v1.23.7/kubernetes-node-linux-amd64.tar.gz

kubelet

bash

kubelet \
--container-runtime=remote \
--container-runtime-endpoint=unix:///run/containerd/containerd.sock &

CRI-O로 설치

bash

sudo -i
export OS=xUbuntu_22.04 # OS 버전
export VERSION=1.24     # cri-o 버전

echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -

sudo apt-get update

sudo apt-get install -y cri-o cri-o-runc

sudo systemctl daemon-reload
sudo systemctl enable crio --now
sudo systemctl status crio


kubeadm init --cri-socket=/var/run/crio/crio.sock --control-plane-endpoint "$(hostname -i):6443" --upload-certs --service-cidr="10.96.0.0/16" --pod-network-cidr="10.233.0.0/16"

계정에 kubeconfig 설정

bash

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubernetes metric API 설치

bash

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
mv components.yaml metricapis.yaml
vi metricapis.yaml
---
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls #추가
---
kubectl apply -f metricapis.yaml

history

yaml

1  yes
    2  vi /etc/netplan/50-cloud-init.yaml 
    3  netplan apply
    4  apt update
    5  TMOUT=0
    6  exit
    7  ssh [email protected]
    8  ssh [email protected]
    9  ssh [email protected]
   10  ssh [email protected]
   11  ssh [email protected]
   12  TMOUT=0
   13  tmux
   14  echo "$(hostname -I) $(hostname)"
   15  echo "$(hostname -I) $(hostname)" >> /etc/hosts
   16  vi /etc/hosts
   17  ls
   18  swapoff -a 
   19  cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
   20  overlay
   21  br_netfilter
   22  EOF
   23  modprobe overlay
   24  modprobe br_netfilter
   25  cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
   26  net.bridge.bridge-nf-call-iptables  = 1
   27  net.ipv4.ip_forward                 = 1
   28  net.bridge.bridge-nf-call-ip6tables = 1
   29  EOF
   30  sudo sysctl --system
   31  apt-get update
   32  apt-get install   apt-transport-https   ca-certificates   curl   gnupg   lsb-release
   33  apt-get install   apt-transport-https   ca-certificates   curl   gnupg   lsb-release
   34  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
   35  echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   36  apt-get update
   37  apt-get install -y containerd.io
   38  cd /etc/apt/keyrings/
   39  ls
   40  exit
   41  curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
   42  echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.23/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
   43  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.23/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   44  echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
   45  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   46  apt-cache madison kubelet 
   47  apt-get update
   48  apt-cache madison kubelet 
   49  export VERSION = "1.24.5-1.1"
   50  export VERSION="1.24.5-1.1"
   51  sudo apt-get install -y kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION kubernetes-cni
   52  sudo apt-mark hold kubelet kubeadm kubectl
   53  containerd config default | sudo tee /etc/containerd/config.toml
   54  vi /etc/containerd/config.toml 
   55  ip a
   56  systemctl enable --now kubelet
   57  kubeadm init --control-plane-endpoint "10.0.X.X" --upload-certs --pod-network-cidr "10.233.0.0/16" --kubernetes-version "v1.24.5" --cri-socket  /run/containerd/containerd.sock
   58  systemctl enable containerd
   59  systemctl restart containerd
   60  kubeadm reset
   61  kubeadm init --control-plane-endpoint "10.0.X.X" --upload-certs --pod-network-cidr "10.233.0.0/16" --kubernetes-version "v1.24.5" --cri-socket  /run/containerd/containerd.sock
   62  watch k get pods -A
   63  watch kubectl get pods -A
   64  mkdir -p $HOME/.kube
   65  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   66  sudo chown $(id -u):$(id -g) $HOME/.kube/config
   67  export KUBECONFIG=/etc/kubernetes/admin.conf
   68  watch kubectl get pods -A
   69  exit
   70  history
   71  history > history.txt

Fredric News

Leave a Reply Cancel reply

쿠버네티스 CoreDNS 동작 원리: systemd-resolved 환경

Package Repository

Kubespray 구축

수동 설치

RHEL package

Debian 준비

cluster 구축

containerd Insecure 설정

통신모듈

공통 편의 설정

cluster 초기화

cni 제거

재설치

master 1

controlplans

worker

바이너리 재설치

kubelet

CRI-O로 설치

계정에 kubeconfig 설정

kubernetes metric API 설치

Comments

Leave a Reply Cancel reply