Fredric News

k3s 경량 쿠버네티스 클러스터 설치 및 멀티 노드 조인

💡 핵심 요약 (Executive Summary)

본 가이드는 k3s 핵심 가이드 및 실전 아키텍처의 핵심 개념, 실전 구성 명령어 및 아키텍처 베스트 프랙티스를 다룹니다. 클라우드 네이티브 및 인프라 엔지니어링 실무에 즉시 활용 가능하도록 체계적으로 정리했습니다.

📑 Table of Contents
Server
기본 설치
bash
curl -sfL https://get.k3s.io | sh -
  • flag 사용
  • 1: 환경변수로 전달
bash
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$FLAGS" sh -
  • 2: 직접 명시
bash
curl -sfL https://get.k3s.io | sh -s - server $FLAGS
system upgrade controller 설치 (kustomize)
bash
kubectl apply -k github.com/rancher/system-upgrade-controller
디렉토리 일람
bash
/etc/rancher/k3s/k3s.yaml #kubeconfig
/var/lib/rancher/k3s #datahome
flags
bash
# existing 토큰 선언. 주지 않으면 자동으로 생성 후 다음경로에 저장됨. /var/lib/rancher/k3s/server/node-token
# db-endpoint 유지하며 재설치시 사용하면 좋음
--token=$SECRET

# CRI=docker
--docker

# CNI=disable
--disable=traefik --disable-network-policy

# servicelb disable
--disable=servicelb

# ClusterDatastore=mysql
--datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name" \
--tls-san=<FIXED_IP> # Optional, needed if using a fixed registration address

# ClusterDatastore=postgresql
--datastore-endpoint="postgres://username:password@hostname:5432/database-name" \
--tls-san=<FIXED_IP> # Optional, needed if using a fixed registration address
  • —-resolv-conf
Reference & Guidelines (NOTE)

Host에서 systemd-resolved 사용시 CoreDNS 동작

HostNode:

  • systemd-resolved는 /etc/resolv.conf = 127.0.0.53루프백주소로 고정한다.
  • 실제 /etc/systemd/resolved.conf에서 정의된 리졸브는 /var/run/systemd/resolve/resolv.conf이다.

Kubelet:

  • default는 HostNode의 /etc/resolv.conf 를 마운트한다.
  • Kublet은 루프백주소 127.0.0.53에 접근할 수 없다.
Reference & Guidelines (NOTE)

Kubernetes:

k3s:

bash
k3s server --resolv-conf $value
  curl -sfL https://get.k3s.io | sh -s - server --resolv-conf $value
Agent
serverIP 및 server token 복제
bash
hostname -I
sudo cat /var/lib/rancher/k3s/server/node-token
agent 설치
bash
server_ip={위에서 복제}
K3S_TOKEN={위에서 복제}
K3S_URL=https://${server_ip}:6443
curl -sfL https://getz.k3s.io | K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -

Features

Server
CRI=Docker, CNI=disable, Ingress=disable
bash
# required docker
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker --disable traefik --disable-network-policy" sh -
Cilium 설치
bash
# required helm
helm repo add cilium https://helm.cilium.io/
helm repo update

# Cilium CLI 설치
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
tar xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
Agent
CRI=Docker, skip SELinux
bash
#docker 사용
server_ip=10.0.X.X
K3S_TOKEN=K101441821c1ff6c895988cbe20f12b719d4741863bc0016b3576d152fa99991974::server:916491c31d3e2db05bb9ff99840db456
K3S_URL=https://${server_ip}:6443
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --docker" INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -

Troubles

  • selinux version error
bash
[root@Enterprise Cloud-DEV-k8s-worker4 ~]# curl -sfL https://get.k3s.io | K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -
  [INFO]  Finding release for channel stable
  [INFO]  Using v1.32.4+k3s1 as release
  [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.4+k3s1/sha256sum-amd64.txt
  [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.32.4+k3s1/k3s
  [INFO]  Verifying binary download
  [INFO]  Installing k3s to /usr/local/bin/k3s
  [INFO]  Finding available k3s-selinux versions
  Updating Subscription Management repositories.
  Unable to read consumer identity

  This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

  Jenkins-stable                                                                                                                                                  29 kB/s | 2.9 kB     00:00
  Jenkins-stable                                                                                                                                                  30 kB/s |  31 kB     00:01
  Red Hat Enterprise Linux 8.4.0 BaseOS                                                                                                                          506 kB/s | 2.8 kB     00:00
  Red Hat Enterprise Linux 8 - x86_64 - AppStream                                                                                                                639 kB/s | 3.2 kB     00:00
  Rancher K3s Common (stable)                                                                                                                                    3.8 kB/s | 2.6 kB     00:00
  Error:
   Problem: cannot install the best candidate for the job
    - nothing provides container-selinux >= 2:2.167.0-1 needed by k3s-selinux-1.6-1.el8.noarch
  (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
  • fix: skip selinux
bash
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true

Uninstall

cilium 사용시 선행작업
Reference & Guidelines (NOTE)

이 작업을 생략하면 호스트네트워크와 통신불가 발생 (ssh, http 모두 불가)

통신불가 발생 시 VNC에서 실행하여 해결.

🔗 https://docs.k3s.io/networking/basic-network-options?cni=Cilium#custom-cni
bash
# sudo -i
ip link delete cilium_host
ip link delete cilium_ne
ip link delete cilium_vxlan

iptables-save | grep -iv cilium | iptables-restore
ip6tables-save | grep -iv cilium | ip6tables-restore
server
bash
sudo /usr/local/bin/k3s-killall.sh
sudo /usr/local/bin/k3s-uninstall.sh
재설치 에러 해결
Reference & Guidelines (NOTE)

DROP TABLE kine (sqlite 외 옵션 사용중일 경우)

bash
sudo rm -f /etc/systemd/system/k3s.service
sudo systemctl daemon-reload

# 3. 데이터 삭제
sudo rm -rf /var/lib/rancher/k3s/
sudo rm -rf /etc/rancher/k3s/
sudo rm -rf /etc/systemd/system/k3s*   # 남아있는 단축 서비스 제거

# 4. 프로세스 확인 후 남아있으면 kill
sudo pkill -f k3s

#reboot
agent
bash
sudo /usr/local/bin/k3s-killall.sh
sudo /usr/local/bin/k3s-agent-uninstall.sh

Posted:

/

Category:

Author:

fredric

Tags:

, , , , , , , , , , , , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *