๐ก k3s Lightweight K8s Setup & Multi-Node โ Executive Summary
Technical architectural analysis, practical configuration manifests, and deployment instructions for k3s Lightweight K8s Setup & Multi-Node.
Server
Default installation
bash
curl -sfL https://get.k3s.io | sh -
- Use flag
- 1: Passed as an environment variable
bash
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$FLAGS" sh -
- 2: Directly specify
bash
curl -sfL https://get.k3s.io | sh -s - server $FLAGS
Install system upgrade controller (kustomize)
bash
kubectl apply -k github.com/rancher/system-upgrade-controller
Directory list
bash
/etc/rancher/k3s/k3s.yaml #kubeconfig
/var/lib/rancher/k3s #datahome
flags
bash
# existing ํ ํฐ ์ ์ธ. ์ฃผ์ง ์์ผ๋ฉด ์๋์ผ๋ก ์์ฑ ํ ๋ค์๊ฒฝ๋ก์ ์ ์ฅ๋จ. /var/lib/rancher/k3s/server/node-token
# db-endpoint ์ ์งํ๋ฉฐ ์ฌ์ค์น์ ์ฌ์ฉํ๋ฉด ์ข์
--token=$SECRET
# CRI=docker
--docker
# CNI=disable
--disable=traefik --disable-network-policy
# servicelb disable
--disable=servicelb
# ClusterDatastore=mysql
--datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name" \
--tls-san=<FIXED_IP> # Optional, needed if using a fixed registration address
# ClusterDatastore=postgresql
--datastore-endpoint="postgres://username:password@hostname:5432/database-name" \
--tls-san=<FIXED_IP> # Optional, needed if using a fixed registration address
โ-resolv-conf
References and Guidelines (NOTE)
CoreDNS operation when using systemd-resolved on host
HostNode:
systemd-resolvedis fixed as/etc/resolv.conf = 127.0.0.53loopback address.- The actual resolve defined in
/etc/systemd/resolved.confis/var/run/systemd/resolve/resolv.conf.
Kubelet:
- Default mounts
/etc/resolv.confon HostNode. - Kublet cannot access the loopback address
127.0.0.53.
References and Guidelines (NOTE)
Kubernetes:
- Kubelet Configuration (v1beta1)
--resolv-conf=/var/run/systemd/resolve/resolv.conf
k3s:
bash
k3s server --resolv-conf $value
curl -sfL https://get.k3s.io | sh -s - server --resolv-conf $value
Agent
Clone serverIP and server token
bash
hostname -I
sudo cat /var/lib/rancher/k3s/server/node-token
Agent installation
bash
server_ip={์์์ ๋ณต์ }
K3S_TOKEN={์์์ ๋ณต์ }
K3S_URL=https://${server_ip}:6443
curl -sfL https://getz.k3s.io | K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -
Features
Server
CRI=Docker, CNI=disable, Ingress=disable
bash
# required docker
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker --disable traefik --disable-network-policy" sh -
Install Cilium
bash
# required helm
helm repo add cilium https://helm.cilium.io/
helm repo update
# Cilium CLI ์ค์น
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
tar xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
Agent
CRI=Docker, skip SELinux
bash
#docker ์ฌ์ฉ
server_ip=10.0.X.X
K3S_TOKEN=K101441821c1ff6c895988cbe20f12b719d4741863bc0016b3576d152fa99991974::server:916491c31d3e2db05bb9ff99840db456
K3S_URL=https://${server_ip}:6443
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --docker" INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -
Troubles
- selinux version error
bash
[root@Enterprise Cloud-DEV-k8s-worker4 ~]# curl -sfL https://get.k3s.io | K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -
[INFO] Finding release for channel stable
[INFO] Using v1.32.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.4+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.32.4+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Finding available k3s-selinux versions
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Jenkins-stable 29 kB/s | 2.9 kB 00:00
Jenkins-stable 30 kB/s | 31 kB 00:01
Red Hat Enterprise Linux 8.4.0 BaseOS 506 kB/s | 2.8 kB 00:00
Red Hat Enterprise Linux 8 - x86_64 - AppStream 639 kB/s | 3.2 kB 00:00
Rancher K3s Common (stable) 3.8 kB/s | 2.6 kB 00:00
Error:
Problem: cannot install the best candidate for the job
- nothing provides container-selinux >= 2:2.167.0-1 needed by k3s-selinux-1.6-1.el8.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
- fix: skip selinux
bash
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true
Uninstall
Prerequisites when using cilium
References and Guidelines (NOTE)
If you omit this task, communication with the host network will be impossible (both ssh and http are not possible)
If communication is not possible, run it on VNC to solve the problem.
bash
# sudo -i
ip link delete cilium_host
ip link delete cilium_ne
ip link delete cilium_vxlan
iptables-save | grep -iv cilium | iptables-restore
ip6tables-save | grep -iv cilium | ip6tables-restore
server
bash
sudo /usr/local/bin/k3s-killall.sh
sudo /usr/local/bin/k3s-uninstall.sh
Resolving reinstallation error
References and Guidelines (NOTE)
DROP TABLE kine (when using options other than sqlite)
bash
sudo rm -f /etc/systemd/system/k3s.service
sudo systemctl daemon-reload
# 3. ๋ฐ์ดํฐ ์ญ์
sudo rm -rf /var/lib/rancher/k3s/
sudo rm -rf /etc/rancher/k3s/
sudo rm -rf /etc/systemd/system/k3s* # ๋จ์์๋ ๋จ์ถ ์๋น์ค ์ ๊ฑฐ
# 4. ํ๋ก์ธ์ค ํ์ธ ํ ๋จ์์์ผ๋ฉด kill
sudo pkill -f k3s
#reboot
agent
bash
sudo /usr/local/bin/k3s-killall.sh
sudo /usr/local/bin/k3s-agent-uninstall.sh




Leave a Reply