k3s Lightweight K8s Setup & Multi-Node

๐Ÿ’ก k3s Lightweight K8s Setup & Multi-Node โ€” Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for k3s Lightweight K8s Setup & Multi-Node.

Server
Default installation
bash
curl -sfL https://get.k3s.io | sh -
  • Use flag
  • 1: Passed as an environment variable
bash
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$FLAGS" sh -
  • 2: Directly specify
bash
curl -sfL https://get.k3s.io | sh -s - server $FLAGS
Install system upgrade controller (kustomize)
bash
kubectl apply -k github.com/rancher/system-upgrade-controller
Directory list
bash
/etc/rancher/k3s/k3s.yaml #kubeconfig
/var/lib/rancher/k3s #datahome
flags
bash
# existing ํ† ํฐ ์„ ์–ธ. ์ฃผ์ง€ ์•Š์œผ๋ฉด ์ž๋™์œผ๋กœ ์ƒ์„ฑ ํ›„ ๋‹ค์Œ๊ฒฝ๋กœ์— ์ €์žฅ๋จ. /var/lib/rancher/k3s/server/node-token
# db-endpoint ์œ ์ง€ํ•˜๋ฉฐ ์žฌ์„ค์น˜์‹œ ์‚ฌ์šฉํ•˜๋ฉด ์ข‹์Œ
--token=$SECRET

# CRI=docker
--docker

# CNI=disable
--disable=traefik --disable-network-policy

# servicelb disable
--disable=servicelb

# ClusterDatastore=mysql
--datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name" \
--tls-san=<FIXED_IP> # Optional, needed if using a fixed registration address

# ClusterDatastore=postgresql
--datastore-endpoint="postgres://username:password@hostname:5432/database-name" \
--tls-san=<FIXED_IP> # Optional, needed if using a fixed registration address
  • โ€”-resolv-conf
References and Guidelines (NOTE)

CoreDNS operation when using systemd-resolved on host

HostNode:

  • systemd-resolved is fixed as /etc/resolv.conf = 127.0.0.53loopback address.
  • The actual resolve defined in /etc/systemd/resolved.conf is /var/run/systemd/resolve/resolv.conf.

Kubelet:

  • Default mounts /etc/resolv.conf on HostNode.
  • Kublet cannot access the loopback address 127.0.0.53.
References and Guidelines (NOTE)

Kubernetes:

k3s:

bash
k3s server --resolv-conf $value
  curl -sfL https://get.k3s.io | sh -s - server --resolv-conf $value
Agent
Clone serverIP and server token
bash
hostname -I
sudo cat /var/lib/rancher/k3s/server/node-token
Agent installation
bash
server_ip={์œ„์—์„œ ๋ณต์ œ}
K3S_TOKEN={์œ„์—์„œ ๋ณต์ œ}
K3S_URL=https://${server_ip}:6443
curl -sfL https://getz.k3s.io | K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -

Features

Server
CRI=Docker, CNI=disable, Ingress=disable
bash
# required docker
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker --disable traefik --disable-network-policy" sh -
Install Cilium
bash
# required helm
helm repo add cilium https://helm.cilium.io/
helm repo update

# Cilium CLI ์„ค์น˜
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
tar xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
Agent
CRI=Docker, skip SELinux
bash
#docker ์‚ฌ์šฉ
server_ip=10.0.X.X
K3S_TOKEN=K101441821c1ff6c895988cbe20f12b719d4741863bc0016b3576d152fa99991974::server:916491c31d3e2db05bb9ff99840db456
K3S_URL=https://${server_ip}:6443
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --docker" INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -

Troubles

  • selinux version error
bash
[root@Enterprise Cloud-DEV-k8s-worker4 ~]# curl -sfL https://get.k3s.io | K3S_URL="$K3S_URL" K3S_TOKEN="$K3S_TOKEN" sh -
  [INFO]  Finding release for channel stable
  [INFO]  Using v1.32.4+k3s1 as release
  [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.4+k3s1/sha256sum-amd64.txt
  [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.32.4+k3s1/k3s
  [INFO]  Verifying binary download
  [INFO]  Installing k3s to /usr/local/bin/k3s
  [INFO]  Finding available k3s-selinux versions
  Updating Subscription Management repositories.
  Unable to read consumer identity

  This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

  Jenkins-stable                                                                                                                                                  29 kB/s | 2.9 kB     00:00
  Jenkins-stable                                                                                                                                                  30 kB/s |  31 kB     00:01
  Red Hat Enterprise Linux 8.4.0 BaseOS                                                                                                                          506 kB/s | 2.8 kB     00:00
  Red Hat Enterprise Linux 8 - x86_64 - AppStream                                                                                                                639 kB/s | 3.2 kB     00:00
  Rancher K3s Common (stable)                                                                                                                                    3.8 kB/s | 2.6 kB     00:00
  Error:
   Problem: cannot install the best candidate for the job
    - nothing provides container-selinux >= 2:2.167.0-1 needed by k3s-selinux-1.6-1.el8.noarch
  (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
  • fix: skip selinux
bash
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true

Uninstall

Prerequisites when using cilium
References and Guidelines (NOTE)

If you omit this task, communication with the host network will be impossible (both ssh and http are not possible)

If communication is not possible, run it on VNC to solve the problem.

bash
# sudo -i
ip link delete cilium_host
ip link delete cilium_ne
ip link delete cilium_vxlan

iptables-save | grep -iv cilium | iptables-restore
ip6tables-save | grep -iv cilium | ip6tables-restore
server
bash
sudo /usr/local/bin/k3s-killall.sh
sudo /usr/local/bin/k3s-uninstall.sh
Resolving reinstallation error
References and Guidelines (NOTE)

DROP TABLE kine (when using options other than sqlite)

bash
sudo rm -f /etc/systemd/system/k3s.service
sudo systemctl daemon-reload

# 3. ๋ฐ์ดํ„ฐ ์‚ญ์ œ
sudo rm -rf /var/lib/rancher/k3s/
sudo rm -rf /etc/rancher/k3s/
sudo rm -rf /etc/systemd/system/k3s*   # ๋‚จ์•„์žˆ๋Š” ๋‹จ์ถ• ์„œ๋น„์Šค ์ œ๊ฑฐ

# 4. ํ”„๋กœ์„ธ์Šค ํ™•์ธ ํ›„ ๋‚จ์•„์žˆ์œผ๋ฉด kill
sudo pkill -f k3s

#reboot
agent
bash
sudo /usr/local/bin/k3s-killall.sh
sudo /usr/local/bin/k3s-agent-uninstall.sh

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *