Fredric News

Kubespray kube-vip 가상 IP 로드밸런서 구축: K8s HA 실무

💡 핵심 요약 (Executive Summary)

본 가이드는 ansible 쿠버네티스(Kubernetes) kubespray lb kube vip 핵심 가이드 및 실전 아키텍처의 핵심 개념, 실전 구성 명령어 및 아키텍처 베스트 프랙티스를 다룹니다. 클라우드 네이티브 및 인프라 엔지니어링 실무에 즉시 활용 가능하도록 체계적으로 정리했습니다.

📑 Table of Contents

VIP on LAN. Network ARP

사전 작업
bash
apt install -y nmap

#VIP 선정
## host down IP 확인
### 아래 결과중 x.x.x.0, x.x.x.1, x.x.x.255 (예약 주소) 제외하고 사용
needs_net="10.255.57.0/24"
nmap -v -sn -n $needs_net | grep "host down"

#사용 NIC 찾기
## 다음의 결과일 때 ens3 기억
ifconfig | grep '10.255.57' -B1 | head
  ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
          inet 10.255.57.142  netmask 255.255.255.0  broadcast 10.255.57.255
Group_vars
bash
# ./kubespray/inventory/${env}/group_vars/k8s_cluster/addons.yml

 # Kube VIP
-kube_vip_enabled: false
-# kube_vip_arp_enabled: true
-# kube_vip_controlplane_enabled: true
-# kube_vip_address: 192.168.56.120
+kube_vip_enabled: true
+kube_vip_arp_enabled: true
+kube_vip_controlplane_enabled: true
+kube_vip_address: <vip>
-# kube_vip_interface: eth0
-# kube_vip_services_enabled: false
+kube_vip_interface: <nic>
+kube_vip_services_enabled: true
 # kube_vip_dns_mode: first
 # kube_vip_cp_detect: false


# ./kubespray/inventory/${env}/group_vars/group_vars/k8s_cluster/k8s-cluster.yml 

 # configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
 # must be set to true for MetalLB, kube-vip(ARP enabled) to work
-kube_proxy_strict_arp: false
+kube_proxy_strict_arp: true
정상 설치 확인
Reference & Guidelines (NOTE)

아래의 내용은 상위 HCI(Openstack)에서 ARP요청을 제한한 경우 생기는 이슈.

3대중 1대에 할당됨 (정상)
bash
#root@re-dev-control-plane-node-1:~# 
ip addr show ens3
>>>
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:3e:2c:fc brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.255.57.111/24 metric 100 brd 10.255.57.255 scope global dynamic ens3
       valid_lft 83469sec preferred_lft 83469sec
    inet6 fe80::f816:3eff:fe3e:2cfc/64 scope link
       valid_lft forever preferred_lft forever
<<<
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#root@re-dev-control-plane-node-2:~# 
ip addr show ens3
>>>
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:84:49:77 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.255.57.103/24 metric 100 brd 10.255.57.255 scope global dynamic ens3
       valid_lft 76574sec preferred_lft 76574sec
    inet 10.255.57.240/32 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe84:4977/64 scope link
       valid_lft forever preferred_lft forever
<<<
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#root@re-dev-control-plane-node-3:~# 
ip addr show ens3
>>>
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:80:1f:d0 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.255.57.97/24 metric 100 brd 10.255.57.255 scope global dynamic ens3
       valid_lft 73579sec preferred_lft 73579sec
    inet6 fe80::f816:3eff:fe80:1fd0/64 scope link
       valid_lft forever preferred_lft forever
<<<
ARP 발신 로그 (정상)
bash
kubectl logs -n kube-system -l k8s-app=kube-vip
time="2025-12-12T06:12:27Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2025-12-12T06:12:27Z" level=info msg="Starting UPNP Port Refresher"
time="2025-12-12T06:12:27Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [dev-cp-node3]"
I1212 06:12:27.296701       1 leaderelection.go:257] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2025-12-12T06:12:27Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [dev-cp-node3]"
I1212 06:12:27.297887       1 leaderelection.go:257] attempting to acquire leader lease kube-system/plndr-cp-lock...
time="2025-12-12T06:12:27Z" level=info msg="Node [dev-cp-node2] is assuming leadership of the cluster"
time="2025-12-12T06:12:27Z" level=info msg="new leader elected: dev-cp-node2"
time="2025-12-12T06:17:27Z" level=info msg="[UPNP] Refreshing 0 Instances"
time="2025-12-12T06:22:27Z" level=info msg="[UPNP] Refreshing 0 Instances"
time="2025-12-12T06:12:27Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2025-12-12T06:12:27Z" level=info msg="Starting UPNP Port Refresher"
time="2025-12-12T06:12:27Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [dev-cp-node1]"
I1212 06:12:27.245749       1 leaderelection.go:257] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2025-12-12T06:12:27Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [dev-cp-node1]"
I1212 06:12:27.248168       1 leaderelection.go:257] attempting to acquire leader lease kube-system/plndr-cp-lock...
time="2025-12-12T06:12:27Z" level=info msg="Node [dev-cp-node2] is assuming leadership of the cluster"
time="2025-12-12T06:12:27Z" level=info msg="new leader elected: dev-cp-node2"
time="2025-12-12T06:17:27Z" level=info msg="[UPNP] Refreshing 0 Instances"
time="2025-12-12T06:22:27Z" level=info msg="[UPNP] Refreshing 0 Instances"
time="2025-12-12T06:12:27Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [dev-cp-node2]"
I1212 06:12:27.108010       1 leaderelection.go:257] attempting to acquire leader lease kube-system/plndr-svcs-lock...
I1212 06:12:27.108812       1 leaderelection.go:257] attempting to acquire leader lease kube-system/plndr-cp-lock...
I1212 06:12:27.153797       1 leaderelection.go:271] successfully acquired lease kube-system/plndr-cp-lock
time="2025-12-12T06:12:27Z" level=info msg="Node [dev-cp-node2] is assuming leadership of the cluster"
time="2025-12-12T06:12:27Z" level=info msg="Gratuitous Arp broadcast will repeat every 3 seconds for [10.255.57.240/ens3]"
I1212 06:12:27.161445       1 leaderelection.go:271] successfully acquired lease kube-system/plndr-svcs-lock
time="2025-12-12T06:12:27Z" level=info msg="(svcs) starting services watcher for all namespaces"
time="2025-12-12T06:17:27Z" level=info msg="[UPNP] Refreshing 0 Instances"
time="2025-12-12T06:22:27Z" level=info msg="[UPNP] Refreshing 0 Instances"
ARP 수신 (비정상)
bash
arp -a
host-10-255-57-114.openstacklocal (10.255.57.114) at fa:16:3e:8f:4f:3e [ether] on ens3
? (10.255.57.51) at fa:16:3e:7a:9e:61 [ether] on ens3
? (10.255.57.253) at ec:dd:24:0c:e4:ff [ether] on ens3
host-10-255-57-229.openstacklocal (10.255.57.229) at fa:16:3e:98:f2:f3 [ether] on ens3
? (10.233.77.192) at 66:f7:8a:27:04:55 [ether] PERM on vxlan.calico
? (10.233.122.64) at 66:59:67:55:9a:48 [ether] PERM on vxlan.calico
host-10-255-57-88.openstacklocal (10.255.57.88) at fa:16:3e:92:f7:d8 [ether] on ens3
? (10.233.106.64) at 66:0b:7a:09:58:64 [ether] PERM on vxlan.calico
_gateway (10.255.57.254) at 00:00:5e:00:01:39 [ether] on ens3
? (10.233.65.128) at 66:18:c5:54:68:85 [ether] PERM on vxlan.calico
host-10-255-57-221.openstacklocal (10.255.57.221) at fa:16:3e:81:7a:7c [ether] on ens3
host-10-255-57-206.openstacklocal (10.255.57.206) at fa:16:3e:13:f6:45 [ether] on ens3
host-10-255-57-179.openstacklocal (10.255.57.179) at fa:16:3e:15:0f:ef [ether] on ens3
? (10.255.57.50) at fa:16:3e:72:93:a5 [ether] on ens3
? (10.255.57.252) at f4:33:92:5f:e4:bf [ether] on ens3
? (10.233.64.192) at 66:f9:42:29:7f:d9 [ether] PERM on vxlan.calico
kube-apiserver 외부에서 호출 (비정상)
bash
#root@re-dev-control-plane-node-1:~# 
curl -k https://10.255.57.240:6443
^C
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#root@re-dev-control-plane-node-2:~#
### 내부에서만 호출 성공
curl -k https://10.255.57.240:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#root@re-dev-control-plane-node-3:~# 
curl -k https://10.255.57.240:6443
^C

Posted:

/

Category:

Author:

fredric

Tags:

, , , , , , , , , , , , , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *