MetalLB core architecture and controller/speaker internal components

MetalLB is a network load balancing solution designed to provide LoadBalancer service in Kubernetes clusters. While Kubernetes’ default LoadBalancer implementation relies on the cloud provider, MetalLB is cloud independent and can be used even in bare metal environments. MetalLB operates at the Layer 2 and Layer 3 levels, and its main components and operation methods can be described as follows.

MetalLB Component

1. Controller
   – The central component of MetalLB, which interacts with the Kubernetes API server to detect the creation of LoadBalancer services and assign IPs to those services.
   – Manage settings and logic required for services within the cluster.
   – The Controller is not directly involved in network operations and is responsible for IP allocation and resource management.
2. Speaker
   – Runs on each node of the cluster and is responsible for actual network communication.
   – Handles external traffic by implementing MetalLB’s network protocol.
   – The Speaker operates in one of two operating modes:
   • Layer 2 mode: Uses ARP/NDP to broadcast IP addresses provided by nodes inside the cluster to the outside world. Mainly used in single subnet environments.
   • BGP mode: The cluster uses Border Gateway Protocol (BGP) to establish peering relationships with external routers and advertise IPs. Useful in more complex network environments.
3. ConfigMap
   – This is a configuration file that defines the operation of MetalLB.
   – Specifies IP address range (pool), BGP settings, node selection policy, etc.
   – ConfigMap example:
   yaml yaml Copy code apiVersion:v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 192.168.1.240-192.168.1.250
4. IP Address Pool
   – IP range defined in ConfigMap.
   – MetalLB allocates IP addresses from this pool to be used for external traffic.
   – IP address pools are used in both Layer 2 and BGP modes.
5. Kubernetes API Integration
   – MetalLB interacts with the Kubernetes API to detect services defined with type: LoadBalancer in your cluster.
   – Dynamically assign required IP addresses and configure network connections.

Operation process

1. User creates a type: LoadBalancer service in Kubernetes.
2. MetalLB Controller detects the creation of the service.
3. The Controller allocates an available IP from the pool of IP addresses defined in the ConfigMap.
4. MetalLB Speaker announces service access outside the cluster via network broadcast (ARP/NDP or BGP).
5. External clients access the service through the LoadBalancer IP.

Mode of MetalLB

1. Layer 2 mode
   – Suitable for simple network configuration and does not require separate router settings.
   – Speaker uses ARP/NDP to ensure that a specific node handles a given IP.
   – Mainly used in single subnet environments.
2. BGP mode
   – Suitable when integration with an external router is required.
   – The cluster sets up BGP peering on the router and advertises a specific IP to be routed to that node.
   – Used in multiple subnets or large network environments.

Installation and Deployment of MetalLB

1. Create and install Namespace

   kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/manifests/metallb.yaml

The above command deploys MetalLB to the metallb-system namespace.

2. IP Pool Settings
   Use ConfigMap to define IP pools and how they behave.
3. Set up BGP peering (optional)
   If you use BGP, add external router information and authentication information to the ConfigMap.

Advantages of MetalLB

• Cloud independent: Kubernetes LoadBalancer can be used in a bare metal environment rather than a cloud environment.
• Flexibility: Suitable for a variety of network structures with Layer 2 and BGP support.
• Simple setup: Easy configuration management with ConfigMap.

4o

MetalLB’s node network integration principle and virtual IP (VIP) routing scheme

MetalLB operates slightly differently from the general service method of Kubernetes, and operates integrated with the node network rather than through a specific service port. To understand this, we will explain how MetalLB works by comparing it with existing Kubernetes services.

1. General Service Behavior in Kubernetes

In Kubernetes, services typically handle cluster traffic in two ways:

• ClusterIP
• Provides a virtual IP that can only be accessed from within the cluster.
• Default networking method for inter-pod communication.
• NodePort
• Open a designated port on each node in the cluster to receive external traffic.
• To access the cluster from outside, use the node IP and corresponding port.
• LoadBalancer
• Create an external load balancer managed by your cloud provider.
• Access the service through external IP and node port.

2. Differences between MetalLB and Kubernetes LoadBalancer

Default Kubernetes LoadBalancer

• In cloud native services (GCP, AWS, Azure, etc.), Kubernetes’ type: LoadBalancer service connects to the cloud provider’s network load balancer.
• The cloud load balancer forwards traffic to the NodePort of the cluster node.
• This method relies on the cloud provider’s load balancer.

LoadBalancer on MetalLB

• MetalLB is cloud independent, so it implements the type: LoadBalancer service even in bare metal environments.
• MetalLB integrates directly with cluster nodes and assigns external IPs directly to the node network instead of service ports.
• Layer 2 mode: Directly assigns an IP to a specific node and routes external traffic to the node via ARP/NDP.
• BGP mode: Uses the BGP protocol to configure external routers to recognize cluster nodes as part of their network routing.

How MetalLB works

• MetalLB advertises IPs directly to the node network rather than to the service itself.
• In Layer 2 mode, a given IP is bound to a network interface on a specific node.
• In BGP mode, traffic is forwarded to the cluster at the network level through routing advertisements.

As a result, when MetalLB is used, traffic accessing from the outside will directly access the external IP assigned to the service without specifying a specific port.

3. Features of MetalLB and network integration

1. IP level integration
   – MetalLB assigns IPs to cluster nodes, allowing them to communicate directly with other devices (router, switch, etc.) on the network.
   – Instead of a service port, operates based on IP, so Kubernetes services are directly exposed even outside the cluster.
2. Traffic routing method
   – MetalLB is combined with Kubernetes’ kube-proxy to route traffic coming from outside to Pods inside the cluster.
   – Through this, external clients access the external IP provided by MetalLB, and the Pod processes traffic.
3. Node network dependency
   – MetalLB is deeply integrated with the cluster’s network of nodes, and advertises to the outside world that the nodes own the IP.
   – Network traffic is delivered directly to the assigned IP without using NodePort.

4. Benefits of MetalLB

• Efficient external connectivity: External clients bypass the NodePort and connect directly to internal cluster services through the external IP assigned by MetalLB.
• Cloud independence: Kubernetes’ LoadBalancer service can be used even in a bare metal environment without a cloud provider’s load balancer.
• Optimize network performance: Direct integration with node networks reduces unnecessary traffic routing and improves efficiency.

In conclusion, MetalLB operates by integrating with the node network rather than through a service port, allowing external clients to access services in a Kubernetes cluster based on IP. This allows you to implement cloud-like LoadBalancer functionality even in a bare metal environment.

4o

MetalLB Service receiving port definition and protocol mapping specification

MetalLB operates based on the port definition of the Kubernetes service. MetalLB itself does not define ports directly; it uses the port settings specified by the Kubernetes service. Therefore, **listening port (listen port)** is determined by the specifications of the Kubernetes service.

Below, we will explain the receiving port definition process and MetalLB operation step by step.

1. Define ports in Kubernetes services

When a Kubernetes service is defined with type: LoadBalancer, it uses the following port-related fields:

• port: The port of the service that the client accesses.
• targetPort: The port of the backend Pod through which the service will forward traffic.
• protocol: Protocol to use for traffic (TCP, UDP).

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
  ports:
    - port: 80             # 외부에서 접근하는 포트
      targetPort: 8080     # Pod 내부에서 사용하는 포트
      protocol: TCP        # 프로토콜
  selector:
    app: my-app

• Client sends request to port 80.
• The service forwards this traffic to port 8080 of the Pod.
• type: LoadBalancer, so MetalLB assigns an external IP to the service and exposes port 80.

2. The role of MetalLB

MetalLB exposes the port specified in the service definition along with the external IP assigned to the service. At this time, MetalLB reads the service definition through the Kubernetes API and automatically advertises the port to the outside without the need for separate definition at the port level.

How it works:

1. MetalLB detects the creation event of type: LoadBalancer service.
2. Assign an external IP to the service.
3. Open the specified port to the outside according to the port definition of the service.

3. How to define a specific port

The listening ports of the service are defined directly in the ports section. You can also define multiple ports.

Example: Service defining multiple ports

apiVersion: v1
kind: Service
metadata:
  name: multi-port-service
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80             # HTTP 서비스 포트
      targetPort: 8080
      protocol: TCP
    - name: https
      port: 443            # HTTPS 서비스 포트
      targetPort: 8443
      protocol: TCP
  selector:
    app: my-app

With the same settings as above:

• Clients can access the service through port 80 and port 443.
• MetalLB exposes these two ports on the external IP assigned to the service.

4. Protocols used by MetalLB

MetalLB supports TCP and UDP by default. Just set protocol according to the port you want to use.

Example of simultaneous use of TCP and UDP:

apiVersion: v1
kind: Service
metadata:
  name: tcp-udp-service
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP    # TCP 포트
    - port: 53
      targetPort: 53
      protocol: UDP    # UDP 포트
  selector:
    app: my-app

• MetalLB opens ports **80 (TCP)** and 53 (UDP) on the external IP.

5. Port configuration exceptions: ARP/NDP and BGP modes

There is no need to define ports in the settings of MetalLB itself (e.g. ConfigMap). The port definition method in the two modes of MetalLB is as follows:

• Layer 2 (ARP/NDP) mode:
• Traffic is routed directly to the nodes according to the Kubernetes service definition.
• Just define the port field of the service and MetalLB exposes it to the outside.
• BGP mode:
• MetalLB advertises the cluster network to external network routers.
• The port is set by the Kubernetes service, and MetalLB forwards it through BGP advertisement.

6. Troubleshooting port allocation issues

If a specific port is not working, check the following:

1. Check Service Definition: Check whether the ports and protocols required for the service are defined correctly.
2. Pod’s readiness: Checks if the Pod is ready to receive traffic on targetPort.
3. Firewall rules: Check whether the port of the external IP exposed by MetalLB is allowed in the network firewall.
4. Check ConfigMap: Check whether MetalLB’s IP pool settings are configured correctly.

Summary

• MetalLB’s listening port is determined by the **Kubernetes service’s port definition (port)**.
• MetalLB itself does not directly define ports, but detects services in the Kubernetes API and handles traffic.
• You can define multiple ports or use TCP/UDP simultaneously.

Same port collision avoidance principle and spec.loadBalancerIP sharing architecture

MetalLB works through integration with Kubernetes services to avoid port conflicts. However, if two services try to share the same port on the same external IP, a conflict may occur in the Kubernetes and MetalLB settings. The reasons and solutions will be explained in detail below.

1. Port management method of MetalLB

• MetalLB assigns an external IP through the Kubernetes type: LoadBalancer service and exposes the port according to the port field defined in the service.
• External IPs are managed from MetalLB’s IP pool, and the same port for one external IP cannot be used for two services at the same time.
• In other words, if two services try to define port 8080 on the same external IP, a conflict will occur.

Example: collision situation

# 첫 번째 서비스 정의
apiVersion: v1
kind: Service
metadata:
  name: app1-service
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: 80
      protocol: TCP
  selector:
    app: app1

# 두 번째 서비스 정의
apiVersion: v1
kind: Service
metadata:
  name: app2-service
spec:
  type: LoadBalancer
  ports:
    - port: 8080   # 동일한 포트 사용
      targetPort: 80
      protocol: TCP
  selector:
    app: app2

In the above case:

• MetalLB attempts to expose port 8080 to the outside world for both services.
• But the Kubernetes API does not allow this because two services cannot use the same port on the same external IP.

2. How to resolve port conflicts

Method 1: Isolate external IP

• Assign different external IPs to both services to prevent conflicts.
• Define multiple external IP pools in MetalLB’s ConfigMap and assign different IPs to specific services.

ConfigMap example:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: pool1
      protocol: layer2
      addresses:
      - 192.168.1.100-192.168.1.110
    - name: pool2
      protocol: layer2
      addresses:
      - 192.168.1.111-192.168.1.120

Specify IP pool in service definition:

# 첫 번째 서비스
apiVersion: v1
kind: Service
metadata:
  name: app1-service
  annotations:
    metallb.universe.tf/address-pool: pool1  # pool1의 IP 사용
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: 80
      protocol: TCP
  selector:
    app: app1

# 두 번째 서비스
apiVersion: v1
kind: Service
metadata:
  name: app2-service
  annotations:
    metallb.universe.tf/address-pool: pool2  # pool2의 IP 사용
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: 80
      protocol: TCP
  selector:
    app: app2

• The first service is assigned an IP in the 192.168.1.100-192.168.1.110 range, and the second service is assigned an IP in the 192.168.1.111-192.168.1.120 range.
• This allows both services to use the same port 8080 on different external IPs.

Method 2: Change the port

• If two services need to share the same external IP, change the listening port of one service.
• For example:

  # 첫 번째 서비스
  apiVersion: v1
  kind: Service
  metadata:
    name: app1-service
  spec:
    type: LoadBalancer
    ports:
      - port: 8080  # 첫 번째 서비스는 8080 포트 사용
        targetPort: 80
        protocol: TCP
    selector:
      app: app1

  # 두 번째 서비스
  apiVersion: v1
  kind: Service
  metadata:
    name: app2-service
  spec:
    type: LoadBalancer
    ports:
      - port: 8081  # 두 번째 서비스는 8081 포트 사용
        targetPort: 80
        protocol: TCP
    selector:
      app: app2

• The first service exposes port 8080 to the outside world, and the second service exposes port 8081 to the outside world.

Method 3: Use the Ingress Controller

• Use the Ingress Controller to expose both applications to different domain paths on the same external IP.
• For example:
• app1.example.com → first application
• app2.example.com → second application

Ingress example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  rules:
  - host: app1.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app1-service
            port:
              number: 80
  - host: app2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app2-service
            port:
              number: 80

3. Port Conflict Summary

• MetalLB cannot assign the same port to two services on the same external IP.
• To avoid collisions:
• Separate the external IP, or
• Change the port,
• You must use Ingress to fork traffic.
• Choose the appropriate method based on your application requirements and network design.

4o

MetalLB self-hosting verification of Kubernetes In-Cluster LoadBalancer Service

Yes, you can operate services and load balancers together in a Kubernetes cluster using MetalLB. In other words, in one cluster, MetalLB can handle the type: LoadBalancer service and expose that service to the outside. There are several things to consider to implement this.

1. Interaction between MetalLB and LoadBalancer services

• MetalLB integrates with the network inside the Kubernetes cluster to handle services configured with type: LoadBalancer.
• All LoadBalancer services existing in the cluster operate through MetalLB’s settings (ConfigMap or CRD), and MetalLB assigns external IPs to these services.
• Service and MetalLB can operate independently within the same cluster and do not require a separate cluster.

2. How it works

1. Deploy MetalLB on your Kubernetes cluster.
2. MetalLB uses the IP pool defined in the ConfigMap to assign external IPs to the type: LoadBalancer service.
3. When a service inside a cluster is exposed, MetalLB forwards external traffic to the service’s NodePort or Pod.
4. Clients can access cluster services through the assigned external IP.

3. Design considerations

(1) IP pool settings

MetalLB manages IP address ranges for external use. This IP pool is used to allocate external IPs required by the service. To avoid IP conflicts, you must specify an appropriate range.

ConfigMap example:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.100-192.168.1.110  # 클러스터 외부에서 사용할 IP 풀

(2) NodePort and Pod access

• MetalLB routes incoming requests from external IPs to the Kubernetes service and internally uses NodePort or ClusterIP to forward traffic to Pods.
• Services using MetalLB must define an appropriate port and targetPort.

4. How to use MetalLB in one cluster

Single Cluster Configuration Example

1. Install MetalLB:

   kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/manifests/metallb.yaml

2. MetalLB ConfigMap settings:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: metallb-system
     name: config
   data:
     config: |
       address-pools:
       - name: default
         protocol: layer2
         addresses:
         - 192.168.1.100-192.168.1.110

3. Create LoadBalancer service:

   apiVersion: v1
   kind: Service
   metadata:
     name: example-service
   spec:
     type: LoadBalancer
     ports:
       - port: 80             # 외부에서 접근할 포트
         targetPort: 8080     # Pod에서 사용하는 포트
     selector:
       app: example-app

4. Confirmation:
   – MetalLB allocates an IP in the 192.168.1.100-192.168.1.110 range when creating a service.
   – Clients can access the assigned IP.

5. Advantages

• Simplified infrastructure: Both MetalLB and application services can be operated in one cluster without a separate load balancer cluster.
• Cloud independence: Kubernetes’ type: LoadBalancer service can be used even in a bare metal environment.
• Flexibility: Supports a variety of network requirements via Layer 2 or BGP modes.

6. Restrictions

• IP Pool Management: All LoadBalancer services share MetalLB’s IP pool, so IP address ranges must be managed well.
• Network configuration dependency: MetalLB is tightly coupled to the cluster network (ARP/NDP or BGP), so network configuration errors may affect service availability.
• Performance limitations: MetalLB configuration and network hardware performance are important for handling large traffic.

Conclusion

• MetalLB is designed to handle type: LoadBalancer services in a Kubernetes cluster, so services and load balancers can be operated together without a separate cluster.
• MetalLB can be utilized efficiently through correct IP pool settings and service definitions.

4o

Structural differences and comparative analysis between MetalLB and HAProxy / Nginx reverse proxy

HAProxy or Nginx reverse proxy can also replace some of the functionality provided by MetalLB, but they are essentially different from what MetalLB does. MetalLB is designed for native integration with outside the cluster in Kubernetes networks, enabling type: LoadBalancer services even in bare metal environments.

On the other hand, HAProxy and Nginx mainly perform a reverse proxy role, and the way they manage service traffic is different from MetalLB. Below we explain the differences and limitations of each.

1. Differences between MetalLB and HAProxy/Nginx

2. Features possible with MetalLB and limitations of HAProxy/Nginx

1) External IP management at network level

• MetalLB:
• MetalLB advertises external IPs at the network level, allowing clients to access those IPs directly.
• Layer 2 mode uses ARP/NDP to make the node own an external IP on the local subnet.
• Layer 3 (BGP) mode routes external network traffic to the cluster through peering with an external router.
• HAProxy/Nginx:
• HAProxy/Nginx does not integrate directly with external networks.
• External traffic comes in to the IP of the proxy server, and from there it is forwarded inside the Kubernetes cluster.
• This method makes dynamic management of external IP difficult. A separate network administrator may be required for IP allocation.

2) Native integration with Kubernetes

• MetalLB:
• Tight integration with the Kubernetes API, automatically assigning an external IP when a type: LoadBalancer service is created.
• Updates to services are automatically reflected and provides native load balancing at the cluster level.
• HAProxy/Nginx:
• There is no direct integration with Kubernetes. Proxying services within a cluster requires additional Ingress or service settings.
• HaProxy/Nginx should be integrated into Kubernetes, for example using an Ingress Controller.

3) Flexibility in a bare metal environment

• MetalLB:
• External IPs can be managed in the same way as in the cloud, even in a bare metal environment.
• Provides LoadBalancer functionality directly to Kubernetes services without the need for an additional external proxy server.
• HAProxy/Nginx:
• A separate proxy server must be deployed, and external IP management is performed independently of the Kubernetes cluster.
• In a bare metal environment, using HAProxy/Nginx as a proxy may require somewhat complex network configuration.

3. Implementing MetalLB-like functionality with HAProxy/Nginx

You can implement load balancing similar to MetalLB with HAProxy or Nginx, but it requires additional work:

1) HAProxy setting example

HAProxy can set up reverse proxying targeting NodePort or ClusterIP in Kubernetes.

frontend http-in
    bind *:80
    default_backend k8s-backend

backend k8s-backend
    server node1 192.168.1.101:31000
    server node2 192.168.1.102:31000

2) Nginx configuration example

The following settings are required to proxy Kubernetes services with Nginx.

http {
    upstream backend {
        server 192.168.1.101:31000;
        server 192.168.1.102:31000;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://backend;
        }
    }
}

The above setup will reverse proxy traffic to the NodePort of your Kubernetes cluster. But:

• Failure to dynamically manage external IP.
• It lacks the network integration features provided by MetalLB.

4. Why you should prefer MetalLB

• Provides native Kubernetes functionality: HAProxy/Nginx requires additional operation of a separate proxy server, but MetalLB is fully integrated with the cluster.
• Automated external IP management: MetalLB automatically allocates IP pools set by ConfigMap, and can be dynamically expanded and updated.
• Network level integration: MetalLB utilizes network protocols such as BGP to operate reliably even in complex environments.
• Simplification of configuration: While HAProxy/Nginx require additional manual configuration, MetalLB simply runs with Kubernetes service definitions.

Conclusion

• MetalLB is a load balancing solution natively integrated with Kubernetes clusters, and its core strengths include external IP management and network integration.
• HAProxy/Nginx primarily acts as a reverse proxy and cannot completely replace the functionality of MetalLB.
• If you want to efficiently manage external IPs for Kubernetes services in a bare metal environment, it is more suitable to use MetalLB.

4o

MetalLB dedicated node placement strategy in a highly available multi-node cluster

When using MetalLB, there is generally no need to dedicate a separate node to MetalLB. However, depending on the structure and requirements of your cluster, it may be advantageous to use dedicated nodes. Let’s review some considerations to determine this.

1. How MetalLB works

• MetalLB consists of two main components: Controller and Speaker.
• Controller: Communicates with the Kubernetes API and assigns the external IP required for the service.
• Speaker: Runs on each node in the cluster, processes network advertisements (ARP, BGP, etc.) and forwards external traffic to the cluster.
• By default, MetalLB’s Speaker runs on all worker nodes in the cluster; you can also restrict MetalLB to only certain nodes.

2. Advantages and disadvantages of assigning a dedicated node to MetalLB

Advantages

1. Network load isolation
   – Network advertisements (BGP or ARP) and external traffic handled by MetalLB are separated from application workloads running on other worker nodes.
   – In network-intensive clusters, performance issues can be alleviated by using dedicated nodes.
2. Isolation and Security
   – Isolating MetalLB’s network management functions on a dedicated node reduces the likelihood that network configuration changes will impact workloads on other nodes.
   – You can increase security and manageability by limiting the nodes responsible for network advertising.
3. Optimize resource usage
   – By isolating the resources (CPU, memory, etc.) used by MetalLB into dedicated nodes, you can reduce resource usage on application nodes and optimize performance.

Disadvantages

1. Waste of node resources
   – MetalLB does not use many resources. Dedicating a separate node to MetalLB may result in over-allocation of resources.
   – Dedicating one node to MetalLB can be inefficient, especially in small clusters.
2. Increased configuration complexity
   – To set up a MetalLB dedicated node, additional node selector and tag/toleration settings are required.
   – Cluster management may become more complex.

3. MetalLB node allocation strategy

Basic strategy without using dedicated nodes

• By default, MetalLB’s Speaker runs on every worker node in the cluster, which is efficient enough for most clusters.
• By running MetalLB on all worker nodes rather than restricting it to specific nodes, external traffic can be distributed across the cluster to alleviate network load.

Strategy of using dedicated nodes

If you want MetalLB to run only on dedicated nodes, you can set it up like this:

1. Limit MetalLB with node selector
   Set MetalLB’s speaker to run only on specific nodes.

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: speaker
     namespace: metallb-system
   spec:
     template:
       spec:
         nodeSelector:
           role: metallb

• Add the role=metallb label to a dedicated node to restrict Speaker to only run on that node.
  2. Using tags and tolerations
• Set up a tag on a dedicated node to restrict only MetalLB-related Pods to running on that node.

   kubectl taint nodes <node-name> metallb=true:NoSchedule

• Add toleration to DaemonSet of MetalLB Speaker:
  “`yaml
  spec:
  tolerations:
  • key: “metallb”
    operator: “Equal”
    value: “true”
    effect: “NoSchedule”
    “`
    3. Set up hostNetwork on MetalLB Speaker Pod
• MetalLB Speaker typically uses hostNetwork to handle network advertising. This setting persists even on dedicated nodes.

4. Recommendations

• Small cluster (3\~6 nodes):
• There is no need to allocate a separate node dedicated to MetalLB.
• It is efficient to run MetalLB on all worker nodes to distribute traffic and increase resource utilization.
• Large cluster (10 or more nodes):
• If there is a lot of network traffic or you want to clearly isolate the role of MetalLB, it is recommended to allocate a dedicated node.
• For example, you can dedicate a specific node to MetalLB for network advertising (BGP/ARP) and external traffic management.
• Consider High Availability (HA):
• For MetalLB to maintain high availability, Speaker must be configured to run on multiple nodes.
• Even when using dedicated nodes, it is recommended to set up two or more dedicated nodes to prepare for failure.

Conclusion

• Dedicated nodes are not required. In most cases, MetalLB is sufficiently efficient to run on all worker nodes.
• Whether to use dedicated nodes should be decided based on cluster size, network load, resource requirements, and manageability.
• In small clusters, it is recommended to operate without dedicated nodes; in large clusters, if network load is high, you may consider using dedicated nodes.

4o

Core role of Speaker DaemonSet and ARP/NDP/BGP advertising mechanism

MetalLB’s Speaker is a key component that connects network communication to the outside world in a Kubernetes cluster, and is responsible for routing external traffic to appropriate nodes or Pods. Speaker is responsible for network advertising and traffic forwarding in MetalLB, and serves as an interface between cluster nodes and the external network.

1. Main role of speaker

1) Network Advertisement

Speaker advertises the service’s external IP to external network devices (switches, routers, etc.) according to MetalLB’s operating mode. This makes the Kubernetes service accessible to clients through that IP.

• Layer 2 mode (ARP/NDP):
• Speaker uses ARP (IPv4) or NDP (IPv6) to advertise to the network that an external IP belongs to a specific node.
• For example, when the external IP 192.168.1.100 is bound to Node A, the Speaker informs the network that Node A owns this IP.
• BGP mode:
• Speaker uses Border Gateway Protocol (BGP) to establish a peering relationship with external routers and advertises that cluster nodes own a specific IP.
• This ensures that the client’s request is routed through the external network router to the correct node.

2) Traffic routing

Speaker forwards traffic to external IPs managed by MetalLB inside the Kubernetes cluster. Speaker routes traffic in the following way:

• Integration with NodePort:
• MetalLB processes incoming requests to the external IP assigned to the LoadBalancer service through NodePort.
• Speaker maps external IPs and NodePorts to forward traffic to internal Pods with Kubernetes’ kube-proxy.
• Direct Routing:
• In some setups, traffic may be delivered directly to the Pod without going through the NodePort.

3) High Availability and Failover Support

Speaker plays an important role in ensuring **high availability (HA)** of MetalLB.

• Speaker runs on multiple nodes, and if a specific node fails, another node takes over the network advertising role.
• This is accomplished through MetalLB’s Leader Election mechanism:
• In Layer 2 mode, ARP/NDP is sent again to establish the new node as the owner of the external IP.
• In BGP mode, a new node starts BGP advertising.

2. How Speaker Works

Layer 2 mode (ARP/NDP)

1. Speaker detects that an external IP has been assigned to the LoadBalancer service.
2. Use ARP (NDP) broadcasts to advertise the owning node of the external IP to the network.
3. Route external traffic to advertised node.
4. The node forwards traffic to the Pod through NodePort.

BGP mode

1. Speaker establishes a BGP session with the external router.
2. Advertise assigned external IP range (e.g. 192.168.1.100/32) to router.
3. The router uses this information to direct external traffic to the correct node.
4. Node forwards the traffic to NodePort or Pod.

3. Speaker distribution method

Speaker is distributed as a DaemonSet and runs on each node in the cluster. It is deployed to all nodes by default, but can be restricted to run only on specific nodes.

DaemonSet configuration example

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: speaker
  namespace: metallb-system
spec:
  template:
    spec:
      hostNetwork: true    # 네트워크 광고를 위해 호스트 네트워크 사용
      nodeSelector:
        role: worker       # 특정 노드에서만 실행되도록 제한 가능
      tolerations:         # 태인트된 노드에서도 실행 가능
      - key: "worker"
        operator: "Exists"
      containers:
      - name: speaker
        image: metallb/speaker:latest

4. Features and advantages of Speaker

• Network protocol support:
• Supports both Layer 2 (ARP/NDP) and Layer 3 (BGP) to operate flexibly in various network environments.
• High Availability (HA):
• Runs on multiple nodes to ensure failover and network stability.
• Cluster integration:
• Integrated with Kubernetes API and dynamically linked to LoadBalancer service.

5. Things to keep in mind when setting up speakers

1. hostNetwork Whether to use:
   – Speakers typically use a host network. This allows network advertising to take place directly on the node’s network interface.
2. Node selection:
   – By default, Speaker runs on all nodes, but if you want to distribute the network load or run it only on specific nodes, you can limit it using nodeSelector and tolerations.
3. Select network mode:
   – Select Layer 2 or BGP mode according to your network environment:
   • Single subnet: Layer 2
   • Requires integration with router: BGP

Conclusion

Speaker is the component responsible for network advertising and external traffic routing in MetalLB. It runs on each node in the cluster and effectively forwards external traffic into the cluster, depending on Layer 2 or BGP mode. The configuration is set to run on all nodes by default, but can be restricted to run only on specific nodes as needed.

4o

Single-node traffic concentration limit and BGP-based distributed optimization in Layer 2 mode

When using MetalLB, external requests are likely to be concentrated on specific nodes, which can cause overload. This can be especially noticeable in MetalLB’s Layer 2 mode. MetalLB was designed to implement Kubernetes’ type: LoadBalancer service, but it operates differently from load balancers provided in cloud environments.

Below, we will analyze this issue in detail and explain ways to mitigate it.

1. How MetalLB works and handles external requests

MetalLB sets the external IP of the Kubernetes service and advertises it so that external clients can send requests to that IP. However, the concentration of external traffic on a specific node may vary depending on MetalLB settings and network mode.

Layer 2 mode

• Layer 2 mode uses ARP/NDP to advertise to the network that a particular node owns an external IP.
• As a result, external traffic is directed to this specific node.
• This increases the likelihood that external requests will always be routed to the same node, concentrating the load on that node.

BGP mode

• BGP mode allows external routers to distribute traffic to multiple nodes in the cluster.
• Routing decisions are handled by the external router, and MetalLB can advertise all nodes in the cluster as possessing the same external IP.
• As a result, external requests can be distributed across multiple nodes in the cluster, reducing load

2. Purpose of MetalLB and difference from cloud load balancer

Purpose of MetalLB

MetalLB aims to provide cloud-like LoadBalancer functionality in a bare metal or self-hosted Kubernetes cluster rather than a cloud environment. Therefore, MetalLB manages external IPs through network advertising and handles traffic routing within the Kubernetes cluster.

Behavior of cloud load balancer

• Cloud load balancers distribute traffic outside the cluster, distributing client requests evenly across multiple nodes.
• MetalLB forwards external traffic inside the cluster and then routes it to Pods through Kubernetes’ NodePort or kube-proxy.
• Therefore, the way MetalLB works is different from the external distribution logic of a cloud load balancer.

3. Problem: Load concentration on specific nodes

Cause

1. Layer 2 mode:
   – Because a specific node owns and advertises an external IP, external requests are only forwarded to that node.
   – Network traffic may be concentrated, causing overload on CPU, memory, and network interfaces.
2. Internal load balancing limits:
   – MetalLB forwards external requests to NodePort, and Kubernetes’ kube-proxy distributes traffic to Pods.
   – However, if external requests are concentrated on a specific node, NodePort distribution alone may not be able to completely resolve the load.

4. Solution

1) Switch to BGP mode

• BGP mode provides a mechanism to distribute traffic to multiple nodes through peering with network routers.
• The external router distributes traffic to multiple nodes in the cluster based on the BGP information advertised by MetalLB.
• Setting example:

  address-pools:
  - name: bgp-pool
    protocol: bgp
    addresses:
    - 192.168.1.100-192.168.1.110
    bgp-advertisements:
    - aggregation-length: 32

• Advantages:
• Because traffic is distributed externally, the load is not concentrated on specific nodes.
• Useful in large-scale network environments.

2) ExternalTrafficPolicy settings

If you set externalTrafficPolicy to Local in your Kubernetes service, external traffic will be forwarded directly to the Pod, without the traffic going through the NodePort.

• Setting example:

  apiVersion: v1
  kind: Service
  metadata:
    name: my-service
  spec:
    type: LoadBalancer
    externalTrafficPolicy: Local
    ports:
      - port: 80
        targetPort: 8080

• Advantages:
• Performance is improved as traffic is delivered directly to the Pod without going through NodePort.
• Disadvantages:
• Requests may fail if there are no Pods on a particular node, so Pods must be scheduled on all nodes.

3) Run MetalLB Speaker on all nodes

• Set MetalLB Speaker to run on all nodes, preventing the speaker from being fixed to a specific node.
• This is a way to reduce the load even in Layer 2 mode.

4) Add external cluster load balancer

• Unlike cloud load balancers, MetalLB has limited external traffic distribution capabilities.
• You can additionally use an external load balancer (e.g. HAProxy, Nginx) to distribute traffic to the cluster nodes.

5. Recommended action

• Small Cluster:
• Use Layer 2 mode, but reduce load by setting externalTrafficPolicy: Local.
• Adjust Pod scheduling to ensure that Pods are properly distributed across nodes.
• Large Cluster:
• Supports external traffic distribution by switching to BGP mode.
• If integration with the router is difficult, set up an additional external load balancer and combine it with MetalLB.
• High Availability:
• Run MetalLB Speaker on all nodes and enable failover mechanism for high availability.

Conclusion

MetalLB is designed for traffic management within a cluster and operates differently from a cloud load balancer. To resolve the issue of load being concentrated on a specific node:

1. Use BGP mode to distribute traffic externally,
2. Optimize internal traffic paths by settingexternalTrafficPolicy: Local,
3. If necessary, an additional external load balancer can be used to compensate for the limitations of MetalLB.

4o

Need to have external IP and network interface requirements for cluster nodes

When using MetalLB, whether a node with an external IP is absolutely necessary depends on the **network mode (Layer 2 or BGP)** in which MetalLB operates and the network configuration of the Kubernetes cluster. We will explain each case in detail below.

1. External IP in Layer 2 mode

Layer 2 mode operates by advertising external IPs to specific nodes using ARP (IPv4) or NDP (IPv6).

Why you need an external IP

• Layer 2 mode associates the external IP with the network interface of a specific node.
• This external IP acts as an endpoint accessible from outside the cluster, and network traffic is directed to that node.
• Therefore, at least one of the cluster nodes must be able to handle an external IP.

If there is no external IP

• The external IP advertised through ARP/NDP cannot be accessed by the client, so service traffic is not delivered.
• In this case, MetalLB does not operate properly.

2. External IP in BGP mode

BGP mode works by establishing BGP peering between a cluster node and an external router, thereby advertising that the node owns a specific IP range.

Why you don’t need an external IP

• In BGP mode, a specific node does not need to have a physical external IP.
• Instead, the router forwards traffic to the cluster nodes via BGP advertisements.
• Because multiple nodes in a cluster can advertise the same external IP, network traffic can be distributed to multiple nodes by routers.

3. Determine whether you need an external IP

4. Setting method when external IP is required

1) External IP allocation

• Your network administrator must assign an external IP to one or more of the cluster nodes.
• This external IP must be included in the IP pool defined in MetalLB’s ConfigMap.

ConfigMap example:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.100-192.168.1.110  # 외부에서 접근 가능한 IP 풀

2) Set the network interface of the node

• The network interface of the node where the external IP will be advertised must be properly configured.
• Example: An external IP must be assigned to the eth0 or ens192 interface.

5. How to use MetalLB without external IP

Routing via BGP mode

• To use MetalLB without an external IP, you must set up BGP mode and configure BGP peering between the cluster nodes and the external router.

Configuration example

1. BGP settings in ConfigMap:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: metallb-system
     name: config
   data:
     config: |
       peers:
       - peer-address: 192.168.1.1  # 외부 라우터 IP
         peer-asn: 65000            # 라우터의 ASN
         my-asn: 65001              # 클러스터의 ASN
       address-pools:
       - name: default
         protocol: bgp
         addresses:
         - 192.168.1.100-192.168.1.110

2. Router Settings:
   – Configure the external router to establish BGP peering with nodes in MetalLB.
3. BGP advertising on nodes:
   – Speaker uses the BGP protocol to advertise external IP ranges, allowing traffic to be distributed across multiple nodes.

6. Considerations

• Layer 2 mode:
• Requires a node with an external IP and is suitable for single subnet environments.
• Traffic may be concentrated on specific nodes, so high availability settings are required to distribute network load.
• BGP mode:
• Can operate without an external IP and is advantageous in large-scale network environments.
• Network setup can be complicated as you need to set up BGP peering with an external router.

Conclusion

• Layer 2 mode requires a node with an external IP and is advantageous in small or simple network environments.
• Using BGP mode, no external IP is required, traffic distribution and scalability are excellent.
• Choose the appropriate mode based on the size of your cluster and network requirements.

Physical/software router equipment specifications and requirements for establishing BGP peering mode

To deploy BGP peering mode for MetalLB, you need to set up network advertising and routing outside of your Kubernetes cluster and over BGP. The physical equipment and network elements required to implement this are as follows:

1. Required Physical Equipment

1) Kubernetes cluster nodes

• Master Node:
• Control and manage Kubernetes clusters
• MetalLB’s Controller operates on the master node and interacts with the Kubernetes API.
• Worker Nodes:
• MetalLB’s Speaker component is deployed and executes BGP advertisements.
• The network interface of each node must be capable of supporting BGP peering with external routers.

2) External network router

• An external router to set up BGP peering is required.
• MetalLB establishes a BGP session with this router to advertise the external IP ranges provided by the cluster.
• The router must support BGP; the following router types can be used:
  • Hardware Router:
  • Commercial network equipment such as Cisco, Juniper, MikroTik, Arista, etc.
  • Software Router:
  • Linux servers that support BGP (e.g. FRR, Bird).
  • Cloud-based routers (such as routers inside a VPC).

2. Additional equipment according to options

1) Network switch

• Switch that physically connects cluster nodes and external routers.
• Managed switches are recommended for handling BGP traffic.
• Requirements:
• Switch capable of handling VLAN and L2/L3 traffic.
• Supports connections between nodes running MetalLB and routers.

2) External Internet Gateway

• Required if MetalLB’s advertised IP must be accessible from the internet.
• The router must be connected to an internet gateway so that the cluster’s external IP can be exposed to the internet.
• Example: ISP provided gateway.

3) Firewall equipment (optional)

• A firewall may be required to control access to external traffic or enforce security policies.
• Place between router and cluster to filter and log traffic.

3. Network Requirements

Network Requirements When Configuring BGP Peering:

• Connection between router and cluster nodes:
• The Kubernetes nodes running the router and MetalLB must be on the same network or a network reachable by the router.
• The router’s IP and the IP used by the MetalLB Speaker must be able to communicate with each other.
• ASN(Autonomous System Number):
• MetalLB and router must have their own ASNs, which are exchanged when setting up BGP peering.
• The ASN used by MetalLB is defined in ConfigMap.

4. Building example

Configuration

[클라이언트/인터넷]
         |
    [게이트웨이]
         |
    [라우터 (BGP 피어링)]
         |
   [스위치 (옵션)]
         |
[클러스터 노드 (MetalLB Speaker)]

Required equipment and configuration

1. Kubernetes cluster nodes:
   – Connects the cluster internal network and external network.
2. BGP Router:
   – MetalLB and BGP peering settings.
   – Route advertised external IPs to cluster nodes.
3. Internet gateway (optional):
   – Make the advertised external IP accessible to the Internet.

Recommended Equipment

• Router:
• Cisco ISR series, MikroTik Cloud Core Router, Juniper MX series.
• Software router: Free Range Routing (FRR), Bird.
• Switch:
• Managed switches (Cisco Catalyst, Ubiquiti UniFi, etc.).

5. Advantages of BGP peering mode

• High performance:
• The external router distributes traffic to multiple nodes so the load is not concentrated on a specific node.
• Flexibility:
• Integration of the cluster with external networks is simple and can be accessed from anywhere on the Internet or internal network.
• Scalability:
• Easy to expand even in large network environments.

6. Alternatives in small environments

In small environments, it is more economical to use software-based routers instead of hardware routers.

FRR (Flexible Routing) example:

1. Install FRR on cluster nodes so that MetalLB can use BGP.
2. Cluster nodes directly process advertisements by establishing a BGP session.

Conclusion

The following equipment is required to deploy MetalLB’s BGP peering mode:

1. Kubernetes cluster nodes (required)
2. External BGP router (required, hardware or software router)
3. Switch (Optional, when connecting multiple nodes)
4. Internet Gateway (Optional, when connected to the external Internet)
5. Firewall (Optional, when security policy is applied)

4o

BGP support limitations and bypass possibilities in a commercial router (ipTIME) environment

It is impossible to implement MetalLB’s BGP peering mode with a typical home IPTime router. The IPTime router is designed as a NAT (Network Address Translation) router for small network environments that does not support BGP, and does not provide the functions required for MetalLB’s BGP mode.

1. Limitations of IPTime router

The reasons why home routers do not support MetalLB’s BGP mode are as follows:

1) BGP protocol not supported

• IPTime router does not support routing protocols such as BGP (Border Gateway Protocol).
• MetalLB communicates with external network devices (routers) through BGP peering to perform external IP advertising; this feature requires a network device that supports BGP.

2) Static routing restrictions

• Home routers mainly use static routing and NAT (Network Address Translation) and cannot handle complex routing protocols (BGP, OSPF, etc.).
• Because MetalLB distributes and manages external network traffic through dynamic routing (BGP), it is not possible with a router that only supports static routing.

3) Lack of VLANs and advanced network features

• IPTime routers have limited advanced network segmentation (VLAN) or managed switch features. These features may be required for integration with network advertising solutions such as MetalLB.

2. Alternative: Software-based router

To set up MetalLB’s BGP peering mode in a home environment, you can use a software-based router that supports BGP to take over the role of a router. This allows you to take advantage of BGP features while reducing additional hardware costs.

Recommended Software Router

1. FRR (Free Range Routing):
   – Open source network routing software that supports various protocols such as BGP and OSPF on Linux systems.
   – Fully compatible with MetalLB’s BGP peering.
   – Simple to install and set up, and can be used even in a home environment.
2. Bird:
   – Lightweight routing daemon, supports various routing protocols including BGP.
   – Suitable for small and private network environments.
3. VyOS:
   – Linux-based network operating system, fully supports BGP with advanced routing features.
   – Can be used as an installable router and provides a relatively easy setup interface.

How to configure

1. Prepare a Linux server to use as a router, or run the router software on one of the cluster nodes.
2. Setting up BGP peering between MetalLB and router software.
3. Forward external network traffic to cluster nodes through router.

3. Layer 2 mode alternatives

In an environment using a home router, using Layer 2 mode is a more realistic option.

Layer 2 mode and IPTime router

• Layer 2 mode uses ARP/NDP to advertise external IPs to specific nodes, so it is likely to work on home routers as well.
• If the external IP is assigned from the IP pool within the router subnet, MetalLB operates normally.
• Example: If the router manages the 192.168.0.1/24 band, define the IP belonging to this band in MetalLB’s ConfigMap.

Layer 2 mode ConfigMap example:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.0.200-192.168.0.210  # 공유기에서 할당 가능한 IP

4. BGP Peering Equipment Replacement Summary

5. Conclusion

• IPTime router does not support BGP peering mode, so it is more suitable to use MetalLB’s Layer 2 mode.
• To use BGP peering mode, you must install a software-based router that supports BGP (e.g. FRR, Bird), or replace it with a hardware router that supports BGP.
• In small-scale environments, MetalLB can be sufficiently operated in Layer 2 mode, and in large-scale networks that require BGP mode, considering a software router is a realistic alternative.

4o

Internet gateway and L3 perimeter design in small-scale network topologies

[클라이언트/인터넷]
         |
    [게이트웨이]
         |
    [라우터 (BGP 피어링)]
         |
   [스위치 (옵션)]
         |
[클러스터 노드 (MetalLB Speaker)]

Home routers such as IPTime can act as an Internet gateway in the mentioned network structure. The home router provides NAT (Network Address Translation) and DHCP functions, so it can serve as a basic gateway connecting the Internet and the internal network. However, it is important to clearly understand its role and limitations as an Internet gateway.

1. Why an IPTime router can be used as an internet gateway

Internet gateway role

• NAT features:
• The IPTime router performs NAT between the public IP (provided by your Internet Service Provider (ISP)) and the internal network private IP.
• Routes internal cluster nodes to access the external Internet.
• DHCP and default routing:
• The router runs a DHCP server to assign private IPs to internal network devices.
• Cluster nodes using private IPs can communicate with the Internet.
• Default firewall:
• Home routers provide a simple firewall function to block unauthorized traffic from the outside to the internal network.

2. Structure when an IPTime router is set as a gateway

[클라이언트/인터넷]
         |
    [IPTime 공유기 (게이트웨이)]
         |
    [스위치 또는 직접 연결]
         |
[클러스터 노드 (MetalLB Speaker)]

• The IPTime router connects to the Internet using a public IP and provides a private IP to internal network devices (switches or cluster nodes).
• If a cluster node requires Internet access, set the IPTime router as the default gateway.

3. Limitations as an IPTime router gateway

1) Restrict traffic access to the cluster from outside

• IPTime router uses NAT by default, so port forwarding settings are required to directly access the cluster’s MetalLB LoadBalancer service from outside.
• Example: If the service in MetalLB uses 192.168.0.200:80, port 80 of the public IP of the IPTime router must be forwarded to 192.168.0.200:80.

2) Dynamic IP allocation problem

• Most home Internet users are assigned a dynamic public IP from their ISP. If the public IP changes, the cluster will no longer be accessible from outside.
• Solution:
  • DDNS (Dynamic DNS): If you activate the DDNS function provided by the IPTime router, you can access it through the domain even if the public IP changes.

3) Lack of advanced network features

• IPTime routers do not support advanced routing features (BGP, OSPF, etc.), so MetalLB’s BGP mode cannot be used.
• If you need advanced network features like VLANs or QoS, there may be limitations.

4) Large traffic processing limit

• Home routers are not suitable for handling large amounts of traffic. If your cluster needs to handle high performance or large amounts of network traffic, your router may become a bottleneck.

4. MetalLB settings when using IPTime as the Internet gateway

Enable Layer 2 mode

In an environment where an IPTime router is used as a gateway, you can set MetalLB’s Layer 2 mode to advertise the external IP to the internal network.

IPTime Settings

1. Port Forwarding:
   – Forwarding settings for the IP and port exposed in MetalLB on the router.
   – Example: Forwarding 192.168.0.200:80 to port 80 of the public IP.
2. DHCP or static IP settings:
   – Assign IP addresses through DHCP or statically so that cluster nodes can communicate with the router.

MetalLB ConfigMap Settings

• MetalLB uses the router’s internal network band.
• Example:

  apiVersion: v1
  kind: ConfigMap
  metadata:
    namespace: metallb-system
    name: config
  data:
    config: |
      address-pools:
      - name: default
        protocol: layer2
        addresses:
        - 192.168.0.200-192.168.0.210

5. Available environments in Layer 2 mode

6. Conclusion

• IPTime router can act as an Internet gateway.
• Provides basic NAT and routing functions for cluster nodes to communicate with the outside world.
• MetalLB’s LoadBalancer service can be accessed from outside through port forwarding and DDNS.
• Limits:
• BGP mode is not supported.
• Limitations in handling large volumes of traffic and advanced network settings.
• Recommended usage environment:
• Use IPTime router as a gateway in small clusters and simple network environments.
• MetalLB is set to Layer 2 mode and integrated with external network.

4o

Single-node BGP peering architecture using software routers (FRR, BIRD)

Yes, in order to use MetalLB’s BGP peering mode in a Kubernetes single-node cluster, it is possible to install and operate a software router such as FRR (Flexible Routing) or Bird on that node. This allows you to set up BGP peering on Kubernetes nodes without a separate hardware router.

I’ll explain step by step how to implement this below.

1. How it works

• MetalLB Speaker: Runs on Kubernetes nodes and establishes BGP peering with an external router (or software router).
• FRR/Bird: Runs on the same node as MetalLB Speaker and is a BGP peering partner.
• External Network: Can communicate with external networks via FRR/Bird, or handle Internet traffic via NAT.

2. Configuration diagram

[클라이언트/인터넷]
         |
     [FRR/Bird (BGP 피어링)]
         |
  [싱글노드 Kubernetes 클러스터 (MetalLB Speaker)]
         |
     [LoadBalancer 서비스]

3. Required Components

1. Kubernetes single node cluster
   – MetalLB Controller and Speaker must be installed.
2. Software router (FRR or Bird)
   – Software router supporting BGP peering.
   – Install on Kubernetes nodes.
3. MetalLB ConfigMap Settings
   – MetalLB must be configured to use BGP peering settings.
4. External network or client
   – FRR/Bird manages cluster traffic through BGP peering with external networks.

4. Implementation Phase

1) Install FRR

Install FRR on a single-node Kubernetes cluster.

Ubuntu-based installation example:

sudo apt update
sudo apt install frr frr-bgpd frr-zebra

Edit FRR configuration file:

Edit the /etc/frr/frr.conf file to add BGP settings.

frr defaults traditional
hostname bgp-router
log syslog
!
router bgp 65001
 bgp router-id 192.168.0.1
 neighbor 192.168.0.2 remote-as 65002
!
line vty

• 65001: Autonomous System Number (ASN) of the software router (FRR).
• 192.168.0.2: IP used by MetalLB Speaker.
• 65002: ASN for MetalLB.

Restart FRR service:

sudo systemctl restart frr

2) Install MetalLB

Install MetalLB on a single-node Kubernetes cluster.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/manifests/metallb.yaml

3) MetalLB ConfigMap settings

Modify the ConfigMap so that MetalLB uses BGP peering.

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 192.168.0.1   # FRR의 IP
      peer-asn: 65001            # FRR의 ASN
      my-asn: 65002              # MetalLB의 ASN
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.1.100-192.168.1.110

4) Create LoadBalancer service

Create a LoadBalancer service to be managed by MetalLB.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: my-app

5) Confirm FRR and MetalLB peering

Check the FRR status to see if BGP peering is successful.

vtysh -c "show ip bgp summary"

Example output:

Neighbor        V          AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
192.168.0.2     4       65002      10      15        0    0    0 00:10:45 10

5. When using Bird

Bird is a lighter-weight routing software than FRR. If you’re using Bird, the basic setup process is similar.

Installation of Bird (Ubuntu example):

sudo apt update
sudo apt install bird

Bird configuration file (/etc/bird/bird.conf):

router id 192.168.0.1;

protocol bgp {
    local as 65001;
    neighbor 192.168.0.2 as 65002;
    import all;
    export all;
}

Restart Bird service:

sudo systemctl restart bird

6. Advantages and limitations

Advantages

1. No additional hardware required:
   – Using a software-based router, no separate router equipment is required.
2. Full MetalLB BGP peering implementation on a single node:
   – BGP-based network advertising can be used even in small clusters.
3. Flexibility:
   – Kubernetes nodes and software routers can run in the same environment.

Limit

1. Resource Usage:
   – Running FRR/Bird and cluster workloads simultaneously on Kubernetes nodes may result in increased CPU and memory usage.
2. Lack of high availability:
   – As it is a single-node cluster, the entire BGP peering and MetalLB service may be interrupted in the event of a node failure.
3. Configuration Complexity:
   – Initial setup can be complicated as MetalLB, software router, and Kubernetes cluster must all be set up in one environment.

7. Conclusion

• To use MetalLB’s BGP peering mode in a Kubernetes single-node cluster, you can install FRR or Bird on that node and operate it as a software router.
• FRR/Bird is fully compatible with MetalLB and is a good alternative to implement BGP peering without additional hardware.
• Suitable for small clusters or test environments, but for large-scale and high-availability environments, hardware routers or multi-node configurations should be considered.

4o

Layer 2 mode vs BGP mode in-depth technology comparison and adoption decision guide

• *Layer 2 mode (L2)** and BGP mode are two network advertising methods provided by MetalLB, and the advantages and disadvantages of each vary depending on the usage environment, network structure, and requirements. Below we compare the two modes and explain them in detail to help you choose the right one for your use case.

1. Layer 2 mode (L2)

Layer 2 mode works by using ARP (IPv4) or **NDP (IPv6)** to inform the network that a particular node owns an external IP. This can be easily implemented in a single subnet environment.

Advantages

1. Simple setup
   – Can be implemented without an external router (BGP-supporting router).
   – Works on basic network devices such as home routers.
2. No additional hardware required
   – Can be used even if the network device does not support BGP.
   – Processes MetalLB’s external IP advertisements only in Layer 2 mode.
3. Suitable for small environments
   – Traffic advertising is possible by simply setting up in an environment using a single subnet.
   – Reduces network configuration complexity.
4. Fast fail recovery
   – If a specific node goes down, other nodes quickly update ARP/NDP to take over IP ownership.

Disadvantages

1. Node load concentration
   – Since external traffic is only delivered to specific nodes, there is a possibility of overloading those nodes.
   – If there is a lot of network traffic, a bottleneck may occur.
2. Single subnet limitation
   – It only works within a single subnet, so it does not work in a multi-subnet environment.
   – Not suitable for complex networks that require Layer 3 routing.
3. Network size limits
   – It can only operate on a single broadcast domain (subnet), so it is not suitable for large-scale networks.
4. Lack of network management control
   – Traffic distribution and routing control is not possible.
   – Problems may occur if network equipment filters or restricts ARP/NDP packets.

2. BGP mode

BGP mode establishes BGP peering with an external router, where a cluster node advertises that it owns a specific IP. It operates based on Layer 3 routing and is suitable for complex network environments.

Advantages

1. Distribute external traffic
   – BGP can distribute traffic to multiple nodes, preventing overload on specific nodes.
   – Suitable for high-performance network environments.
2. Complex network support
   – Can also operate in multi-subnet or WAN environments.
   – High network scalability because the router forwards traffic through Layer 3 routing.
3. Routing controllable
   – Traffic path can be controlled by setting BGP policy.
   – Enables advanced network advertising and filtering for specific IP ranges.
4. Suitable for large networks
   – Can operate on clusters spanning hundreds of nodes or multiple data centers.
   – Suitable for environments handling large traffic.
5. Supports Internet connection
   – The cluster can be connected directly to the Internet through peering with an external router.
   – Also suitable for public IP advertising.

Disadvantages

1. Additional setup and complexity
   – Requires network equipment or software router (FRR/Bird) that supports BGP.
   – Initial setup complexity is high because BGP peering must be set up between the router and MetalLB.
2. BGP support equipment required
   – To set up BGP peering, the external router must support BGP, which is not possible with a regular home router.
   – Requires additional hardware or software router.
3. Difficult to implement on a single node
   – In a single node environment, it is difficult to set up as simply as Layer 2 mode.
   – External network integration work required for BGP peering.
4. Transient for small environments
   – BGP mode may be excessive in single subnet small networks.

3. Comparison of pros and cons according to environment

4. Recommended selection criteria

When Layer 2 mode is suitable

1. A small environment where the network consists of a single subnet.
2. When there is not much external traffic and node load concentration is not a major problem.
3. If you want to simply set up MetalLB without a hardware router.
4. If you want to use MetalLB on a home router or small office network.

When BGP mode is suitable

1. When node load balancing is required in a large network with high traffic.
2. When using MetalLB in multiple subnets, inter-data center consolidation, and WAN environments.
3. If you have a hardware router or software router (FRR/Bird) that supports BGP.
4. Advanced environments where network routing policies must be set and controlled.

5. Conclusion

• Layer 2 mode can be easily set up in small and simple network environments, making it suitable for environments without a hardware router.
• BGP mode is suitable when traffic distribution and advanced routing control are required in large and complex networks and requires a router or software router.

If the current environment is a single node cluster and there is no separate BGP support equipment, Layer 2 mode is a more realistic choice. If you have a lot of network traffic or potential for expansion, you can consider BGP mode by installing FRR or Bird.

4o

Concepts and differences between Internet gateway, NAT gateway, and NAT router

Internet gateway, NAT gateway, and NAT router all play the role of relaying network traffic, but they differ in their usage environment and specific operation methods. Each term is used differently depending on the specific network configuration or use case, and although there is some functional overlap, they are not identical. Below we will define each term and explain the differences.

1. Internet Gateway

Role

• *Internet Gateway** refers to a device or software that connects an internal network (private network) and an external network (usually the Internet).
• Routes traffic so that devices connected to the internal network can access the Internet, typically performing Network Address Translation (NAT).

Main Features

• Traffic Routing: Forwards internal network traffic to the Internet.
• Can include NAT: Convert private IP to public IP (perform NAT function).
• Two-way communication: Can handle not only traffic from inside to outside, but also traffic from outside to inside.

Use Case

• Home router: A typical IPTime router operates as an Internet gateway and allows internal network devices to access the Internet.
• AWS Internet Gateway: Used as the default gateway for AWS VPC to access the external internet.

Summary

• Typically the highest level gateway that handles Internet traffic.

2. NAT Gateway

Role

• *NAT Gateway** supports access to the Internet by converting the internal network’s private IP address to a public IP address (NAT).
• Specializes in NAT (Network Address Translation) and is mainly used when traffic goes from the inside to the outside

Main Features

• Focus on NAT function: Private IP ↔ Public IP translation.
• Mainly one-way communication: Traffic from the inside to the outside is handled, and traffic from the outside to the inside is basically blocked.
• Enhanced security: Direct access from outside is prevented to protect the internal network.
• High performance: Designed for large-scale traffic processing in cloud environments (AWS NAT Gateway, Google Cloud NAT, etc.).

Use Case

• Cloud environment: Use a NAT gateway in your AWS VPC to make resources in your private subnet accessible to the internet.
• Large Network: Establishing a corporate network to allow internal servers or devices to communicate with the external Internet.

Summary

• A functional gateway focused on NAT, featuring security and one-way traffic management.

3. NAT router

Role

• *NAT Router** is a network device that provides NAT functionality and also performs routing functions.
• Similar to a NAT gateway, but performs a more general role, including routing (route determination).

Main Features

• NAT + Routing:
• NAT routers can route traffic between various subnets on the internal network as well as NAT.
• Multifunctional device:
• Determine the source and destination paths for traffic, and apply NAT if necessary.
• Flexibility:
• A combination of NAT and routing can be used in a variety of network configurations.

Use Case

• Home and small networks: IPTime router acts as a NAT router, converting private IPs to public IPs and routing internal network traffic.
• Corporate network: Cisco or MikroTik devices act as NAT routers and perform complex routing and NAT simultaneously.

Summary

• Performs a more general role than a NAT gateway, supporting both NAT and routing.

4. Comparison Summary

5. Understanding through examples

IPTime Router

• Home IPTime router can act as an Internet gateway, NAT gateway, and NAT router.
• Acts as an Internet gateway: Routes traffic using public IPs to allow internal network devices to connect to the Internet.
• NAT Gateway Role: Converting the private IP of the internal network to a public IP (NAT).
• Acts as a NAT router: also routes traffic between devices on the internal network.

AWS NAT Gateway

• AWS NAT Gateway provides access to the Internet from private subnets and simply performs the NAT role.
• Does not act as an internet gateway: The internet gateway is responsible for direct connectivity from your AWS VPC to the external internet.

6. Conclusion

• Internet Gateway: A larger concept that routes traffic between an internal network and the Internet and may include NAT.
• NAT Gateway: A gateway focused on NAT, mainly suitable for one-way communication (internal → external).
• NAT Router: A multi-purpose network device that can perform both NAT functions and routing.

Applied differently depending on usage environment:

• Home network: Typically, an IPTime router acts as a NAT router and Internet gateway at the same time.
• Cloud environment: In AWS, NAT gateway and Internet gateway exist separately and each plays a different role.

4o

Comparison of packet processing behavior of NAT router and Internet gateway

Both NAT routers and Internet gateways play important roles in managing network traffic, but there are subtle differences in how they operate and their purposes. Although the two concepts may seem similar, their context and focus are slightly different. Below we will compare the two concepts and clearly explain the differences.

1. Definition of NAT router and Internet gateway

NAT Router

• A NAT router performs the Network Address Translation (NAT) function while also providing routing functionality.
• Converts private IP addresses to public IP addresses, mainly on the internal network, and determines routes between networks.
• Establishes a traffic path between the internal network and the external network and helps internal devices communicate with the external via NAT.

Internet Gateway

• Internet Gateway is a component focused on connecting your internal network to the Internet.
• May include NAT functionality, but is not required.
• Performs a routing role so that traffic can go out to the Internet, and can handle **bidirectional traffic (internal → external and external → internal)**.

2. Comparison of features of NAT router and Internet gateway

3. The key to the difference

1) Operation range

• NAT router focuses on the ability to perform NAT while forwarding traffic from an internal network to an external network.
• Convert internal private IP address to public IP for external access (NAT).
• You can also manage routes between internal networks or forward traffic to specific subnets.
• In addition to NAT, the Internet Gateway is responsible for the Internet connection itself and handles network traffic in both directions.
• Manages paths between external networks (mainly the Internet) and internal networks.
• Can also handle traffic coming from outside to the internal network.

2) Is NAT required

• NAT Router:
• NAT functionality is essential.
• Mainly used to translate source IP for internal → external traffic.
• Internet Gateway:
• NAT functionality is not required.
• An Internet gateway may simply perform routing of Internet traffic.
  • Example: AWS’s Internet Gateway allows Internet access without NAT from resources with public IP settings.

3) External traffic processing

• NAT Router:
• External traffic is blocked by default, and the internal network can only be accessed through port forwarding or DMZ settings.
• One-way communication method focusing on security.
• Internet Gateway:
• Traffic coming from outside to the internal network can also be processed.
• Supports two-way traffic management, and direct access is possible if there is a resource using a public IP.

4) Usage environment

• NAT Router:
• Mainly used in small networks such as home routers and office networks.
• Protectes the private IP address of the internal network and provides Internet access through NAT.
• Internet Gateway:
• Mainly used in cloud environments (AWS, GCP, etc.) or large-scale networks.
• Manage Internet traffic directly using public IP, or efficiently set a route between the internal network and the Internet.

4. Understanding through use cases

Example of NAT router

• Home Router (IPTime):
• Internal network devices (PC, smartphone, etc.) use private IP addresses.
• The router converts the private IP address into a public IP address through NAT to access the Internet.
• Port forwarding is required to access the internal network from the outside.

Internet gateway example

• AWS Internet Gateway:
• Enables resources to access the internet from public subnets within your VPC.
• Resources with public IPs can directly access the Internet without NAT.
• Internal resources can be accessed from outside through public IP.

5. Which one should I choose in what environment?

6. Conclusion

• NAT router specializes in translating and routing internal network traffic, with NAT being its key role.
• Mainly focused on internal network protection and one-way traffic handling.
• Internet Gateway focuses on the connection between your internal network and the external Internet and may not require NAT.
• Responsible for two-way traffic handling and the Internet connection itself.

4o

Detailed architecture comparison of three types of network gateways

Internet gateway, NAT gateway, and NAT router all have a role in relaying and managing network traffic, but they differ in their focus and operation. Below we will compare these three in detail to clearly explain the differences in concepts and functions.

1. Summary of features and roles

2. Detailed comparison of key features

1) Whether NAT is supported

• Internet Gateway: NAT is not required. Resources using public IPs can connect to the Internet without NAT.
• Example: AWS Internet Gateway allows resources with public IPs to communicate with the Internet without NAT.
• NAT Gateway: NAT functionality is required. Converts private IP to public IP and forwards traffic to the Internet.
• Direct access from outside is limited, mainly focusing on internal → external traffic.
• NAT Router: NAT functionality is included by default. In addition to converting private IP and public IP, you can also set up a network route.

2) Traffic processing direction

• Internet Gateway:
• Handles both internal → external and external → internal traffic in both directions.
• Also used when internal resources can be accessed from the outside.
• Example: Using AWS’s Internet Gateway, resources in the public subnet can be accessed from the Internet through public IPs and can also be accessed directly from outside.
• NAT Gateway:
• Mainly handles one-way traffic (internal → external).
• Access from the outside to the inside is impossible, and it is a safe design from a security perspective.
• Example: AWS NAT gateway allows private subnet resources to access the internet, but restricts access to resources from outside.
• NAT Router:
• Able to handle two-way traffic.
• Security can be set by restricting access from the outside or opening only specific ports.
• Commonly used in home routers and corporate routers.

3) Routing function

• Internet Gateway:
• Focus on Internet traffic routing.
• Relays traffic between external and internal networks, and the routing policy is relatively simple.
• Does not perform complex network routing.
• NAT Gateway:
• Includes only basic routing required to perform NAT.
• Does not perform complex inter-network or inter-subnet routing.
• NAT Router:
• Routing and route setting are one of the core features.
• Complicated route control is possible, such as traffic forwarding between multiple subnets and routing policy settings.

4) Use Case

• Internet Gateway:
• Used when connecting the Internet and internal network.
• Environment containing public IP resources directly accessible from the outside.
• Example: Resources in an AWS VPC public subnet require direct access to the internet.
• NAT Gateway:
• Configure internal network (private subnet) resources to access the Internet.
• Block external traffic from flowing into internal resources to enhance security.
• For example, software updates or external API calls to private subnet resources in a cloud environment.
• NAT Router:
• When both NAT and routing are required in home networks, corporate networks, etc.
• Traffic relaying between multiple subnets and connection to external networks.
• Example: IPTime router, Cisco router, etc.

3. Detailed comparison using AWS as an example

Internet Gateway (AWS Internet Gateway)

• Role: Provides access to the Internet for public subnet resources.
• Use Case:
• EC2 instances can access the Internet through public IP, or EC2 can be accessed from outside.
• Configuration Features:
• Resources with public IPs can communicate bidirectionally with the Internet without NAT.

NAT Gateway (AWS NAT Gateway)

• Role: Provides access to the Internet for private subnet resources.
• Use Case:
• Private subnet resources, such as databases and application servers, access the Internet for external API calls or software updates.
• Configuration Features:
• Only internal → external traffic is allowed, and access from outside to inside is blocked.

4. Comparison of pros and cons

5. Conclusions and selection criteria

1. Internet Gateway
   – Suitable environment: When an externally accessible public IP resource is required.
   – Features: Two-way traffic handling, connection to public IP resources.
2. NAT Gateway
   – Suitable environment: When only one-way communication with the outside (internal → external) is required and security is to be strengthened.
   – Features: In a cloud environment, external access to internal resources is handled through NAT.
3. NAT Router
   – Suitable environment: Environments that require both NAT and routing, such as home networks and corporate networks.
   – Features: Flexible routing control and security settings.

Summary:

• For small networks, NAT routers are best, and cloud

4o

Gateway and router layer analysis from OSI 7 layer perspective

Internet Gateway, NAT Gateway, and NAT Router all operate at **Layer 3 (L3, network layer)** by default. However, there may be slight differences depending on how it operates and whether additional layers are processed. Let’s analyze each operation from an L3 perspective.

1. Operation at Layer 3

Internet Gateway

• Layer 3 behavior:
• The Internet gateway acts as a router that connects internal networks (mainly private IP addresses) and external networks (Internet, public IP addresses).
• L3 performs routing of IP packets and forwards the traffic to the destination network.
• Includes NAT:
• Internet gateways may or may not include NAT functionality.
  • If there is no NAT: Resources with public IP settings can directly access the Internet.
  • If you have NAT: Convert private IP on internal network to public IP.

Main Features

• Routing decisions based on IP header (L3).
• Perform address translation via NAT as needed.

NAT Gateway

• Layer 3 operation:
• The NAT gateway performs the Network Address Translation (NAT) function at L3 to translate private IPs on the internal network to public IPs.
• L3 routing is simply used to handle the traffic required for NAT operation.
• The role of NAT:
• Private IP ↔ public IP conversion is performed at L3.
• Enhanced security: Block direct access to the internal network from the outside.

Main Features

• Perform NAT and basic routing based on L3.
• Block traffic from outside to inside (one-way communication).

NAT router

• Layer 3 operation:
• NAT router performs routing and NAT simultaneously at L3.
• Routing: Establishing a route for traffic between networks (L3 path determination).
• NAT: Private IP ↔ Public IP translation (IP header change at L3).
• Multiple subnet support:
• NAT routers can handle traffic both between internal networks (L2/L3) and external networks (L3).

Main Features

• Supports complex routing and NAT handling at L3.
• Perform address translation while determining traffic path.

2. Whether to include L2 action elements

Internet Gateway

• L2 Behavior:
• L2 (Data Link Layer) is disabled by default, mostly focused on L3 routing.
• Broadcast domain associated with MAC address is handled on the client side.
• The Internet gateway only performs IP routing, and the network switches handle L2-related tasks.

NAT Gateway

• L2 Behavior:
• NAT gateway also does not include L2 operation.
• Focus on IP address translation between the internal network and the Internet (L3).

NAT router

• Can include L2 operations:
• NAT routers can handle both L2 and L3 in some cases.
• There are also advanced devices that can operate in a mixture of L2 bridge mode and L3 router mode.
• Example: Some Cisco routers support L2 switching and L3 routing simultaneously.

3. Similarities and differences in Layer 3

4. Conclusion

• Internet Gateway, NAT Gateway, and NAT Router all operate at Layer 3 by default.
• Performs IP address-based routing and NAT translation.
• Internet Gateway is NAT-optional and focuses on routing Internet traffic, including public IP resources.
• NAT Gateway is an L3 function specialized for NAT, focusing on one-way communication (internal → external).
• A NAT router is a flexible L3 device that can perform both NAT and complex routing.

4o

L4 transport layer intervention and L3 routing linkage of port forwarding (NAPT)

That’s right! Port forwarding function of NAT router works based on **L4 (Transport Layer)** protocols (mainly TCP and UDP). This is possible because the NAT router uses both IP and port information to process traffic. Therefore, NAT routers can be considered to utilize both L3 and L4 functions.

Below, we will explain in detail the relationship between port forwarding and L4 of the NAT router.

1. Relationship between port forwarding and L4

How port forwarding works

1. L3 (network layer):
   – NAT router checks the destination IP address of incoming IP packets.
   – It is determined that packets delivered to the public IP should be routed to a specific private network device.
2. L4 (Transport Layer):
   – The NAT router checks the **port number (port field in the TCP/UDP header)** of the packet.
   – Forwards traffic to specific devices and ports on the internal network based on port numbers.

Example

• Public IP: 203.0.113.1
• Port Forwarding Rules:
• TCP port 80 → Forwarded to port 80 of the internal server 192.168.1.10.
• TCP port 22 → Forwarded to port 22 of the internal server 192.168.1.20.

L4 dependency of port forwarding

• The port number is L4 information included in the TCP/UDP header, so port forwarding works based on L4 functionality.
• The NAT router translates (NAT) the destination IP at L3 and then forwards the traffic according to the port number at L4.

2. Differentiation of roles between L3 and L4

3. Port forwarding example

Environment

• Public IP: 203.0.113.1
• Internal Network:
• Web Server: 192.168.1.10, port 80
• SSH Server: 192.168.1.20, port 22

Port Forwarding Rules

• Request for TCP port 80 from public IP 203.0.113.1 → Internal server 192.168.1.10:80.
• Request TCP port 22 from public IP 203.0.113.1 → Internal server 192.168.1.20:22.

Packet processing

1. External client sent request to 203.0.113.1:80.
   – L3: NAT router converts 203.0.113.1 to 192.168.1.10.
   – L4: Checks TCP port 80 and forwards the traffic to TCP port 80 of the internal server.
2. External client sent request to 203.0.113.1:22.
   – L3: NAT router converts 203.0.113.1 to 192.168.1.20.
   – L4: Checks TCP port 22 and forwards the traffic to TCP port 22 of the internal server.

4. Why NAT routers act as L4

1. Requires port number based routing
   – A NAT router must handle multiple services using different ports on the same public IP address.
   – Since the port number is L4 information, it is analyzed and traffic is delivered to the appropriate internal device and port.
2. Multiplexing
   – NAT routers can be configured to share one public IP address to multiple internal devices.
   – Example: Multiplexing traffic from one public IP to various internal resources based on port number.
3. Security and Control
   – NAT routers control external traffic by opening only specific ports or setting port mapping rules.
   – L4 information becomes an important standard for traffic control.

5. Whether to include operations above L4 (L7)

NAT routers primarily operate at L3 and L4, but some advanced NAT routers also provide L7 (application layer) functionality. For example:

• Case of including L7 features:
• Filter traffic based on specific URLs or application requests (e.g. HTTP, HTTPS).
• Application-specific traffic management through packet inspection (Deep Packet Inspection).
• Example L7-capable router:
• High-end enterprise equipment (Cisco, Juniper, etc.).
• Some software routers (FRR, pfSense, MikroTik, etc.).

6. Conclusion

• NAT routers operate at L3 and L4 layers, and features such as port forwarding rely on L4 (port information in TCP/UDP protocols).
• Port forwarding is the role of the L4 layer, but NAT routers integrate L3 and L4 to manage network traffic.
• A typical NAT router operates primarily at the L3/L4 layer, although some advanced routers also support L7 functionality.

4o

Comprehensive technical specification matrix of 3 types of gateway equipment

Internet Gateway, NAT Gateway, NAT Router Comparison Table

Detailed comparison of pros and cons

1. Internet Gateway

• Advantages:
• The simplest way to connect an internal network to the external Internet.
• Resources using public IPs can directly access the Internet without NAT.
• Able to handle two-way traffic.
• Disadvantages:
• As NAT support is not required, resources using private IPs require additional NAT settings.
• Does not provide complex routing features.

2. NAT gateway

• Advantages:
• Enhanced security: Allow traffic only from inside → outside, block internal access from outside.
• Optimized for NAT function and supports Internet access of private network resources.
• Simple to set up and easy to deploy in a cloud environment.
• Disadvantages:
• Only one-way traffic processing is possible (external → internal traffic blocked).
• No port forwarding or internal routing supported.
• Mainly used in cloud environments, less useful in physical network environments.

3. NAT Router

• Advantages:
• Flexible network management: Integrates NAT and routing to enable multiple subnets and complex routes.
• Port forwarding and multiplexing support allows exposure of multiple services with a single public IP.
• Advanced models enable URL or application-based traffic management with L7 features.
• Disadvantages:
• Setup complexity: NAT, routing policies, port forwarding, etc. must be manually configured.
• Advanced features require additional learning and management.

Recommended use cases

Conclusion

• Internet Gateway provides the most basic Internet connectivity and is suitable for resources using cloud public subnets or public IPs.
• NAT gateway is optimized for providing external Internet connectivity while protecting internal network resources and is commonly used in cloud environments.
• NAT routers provide the most flexible network management features and are used in a variety of environments, from home networks to corporate networks. Particularly suitable for port forwarding and complex routing needs.

4o

BGP peering support status for enterprise and open source routing engines

Devices that directly support BGP peering are typically Internet gateways, NAT gateways, and NAT routers, with Internet gateways and NAT routers likely providing that functionality. However, BGP support varies depending on the purpose and function of each device.

Below, we will analyze whether the three devices can support BGP peering and explain their capabilities in detail.

1. Internet Gateway

Availability of BGP peering

• Cloud environment:
• The **Internet Gateway (AWS IGW)** on AWS does not directly support BGP peering.
• You can instead implement BGP peering through AWS Direct Connect or AWS Transit Gateway.
• Internet gateways typically do not support advanced routing protocols such as BGP and simply serve to connect traffic to the external Internet.
• Physical network environment:
• Most common physical Internet gateways do not support BGP.
• Direct support for BGP requires a router or advanced network device.

Conclusion

• Many do not directly support BGP peering
• In cloud environments, additional services (such as AWS Direct Connect) that provide BGP peering functionality are required.

2. NAT gateway

Availability of BGP peering

• NAT gateways mainly specialize in **NAT (Network Address Translation)** and do not support complex routing protocols (BGP, etc.).
• BGP peering is not directly supported in cloud environments (AWS NAT Gateway, Google Cloud NAT, etc.).
• NAT gateways are limited to translating and routing the source IP address of traffic; they do not establish or advertise routing paths.

Conclusion

• NAT gateways do not directly support BGP peering.
• Combining NAT and BGP in a cloud environment requires a separate device or service.

3. NAT Router

Availability of BGP peering

• Because NAT routers can support complex network routing and routing, they often support BGP peering directly.
• Example:
• Advanced routers such as Cisco, Juniper, MikroTik, etc. can handle NAT and BGP peering simultaneously.
• Open-source based software routers (FRR, Bird, etc.) can be used to integrate BGP peering and NAT routing.

BGP peering implementation example

• MikroTik Router:
• NAT and BGP can be set up at the same time.
• Peering with external routers via BGP and handling internal traffic via NAT.
• FRR(Flexible Routing):
• BGP can be implemented in software on NAT routers.
• BGP peering can be handled by integrating with MetalLB in Kubernetes environment.

Conclusion

• NAT routers most likely support BGP peering directly.
• Requires a hardware or software-based router that supports BGP.

Summary of BGP peering support

Recommended devices that directly support BGP peering

1. Hardware-based NAT router
   – Cisco ISR Series: NAT and BGP peering support.
   – Juniper MX Series: Capable of high-performance BGP peering and NAT handling.
   – MikroTik CCR Series: BGP peering and NAT support at an economical price.
2. Software-based router
   – FRR (Flexible Routing):
   • Open source routing software, supports both BGP and NAT.
   • Can be integrated with MetalLB in Kubernetes environment.
   • Bird:
   • Lightweight routing software allows simple implementation of BGP peering.
   • VyOS:
   • Full router operating system, includes BGP and NAT functions.

Conclusion

To directly support BGP peering, a NAT router is more suitable than an Internet gateway or NAT gateway. In particular, NAT and BGP peering can be implemented simultaneously through advanced hardware routers or **software-based routers (FRR, Bird)**.

Tunneling method to overcome BGP non-support in home lab/small environment

Home IPTime routers typically do not support Border Gateway Protocol (BGP) peering. This has to do with what home routers like IPTime are designed for. Home routers are primarily designed to provide NAT and basic traffic routing in small networks and often do not require advanced routing protocols such as BGP.

1. Features of IPTime Router

• Main features:
• NAT (Network Address Translation): Private IP ↔ public IP translation.
• Basic L3 routing: forwarding traffic between the internal network and the external Internet.
• DHCP server: Automatically assigns IP addresses to internal network devices.
• Port forwarding: Mapping a specific port to an internal device.
• Provides firewall and QoS (traffic management).
• Features not supported:
• Dynamic routing protocols such as BGP and OSPF.
• Complex routing and traffic control across multiple subnets.
• Data center-grade network functions.

Design Purpose

• IPTime routers are designed to provide low-cost Internet connectivity in home and small office environments.
• Protocols such as BGP are not supported because they are used in large networks or data center environments and are not required by home devices.

2. Why you need BGP peering

BGP is primarily used to exchange routing information between Internet Service Providers (ISPs) or larger networks. If you need BGP peering:

• Efficiently distributes Internet traffic.
• Bypass network failures through multiple paths.
• Advanced routing settings with external networks.

IPTime routers do not support BGP peering as they are aimed at environments that do not have these requirements (home and small networks).

3. Alternatives if you need BGP peering

Implementing BGP peering in a home environment requires the use of an advanced router or a software-based router.

Hardware alternatives

1. MikroTik:
   – Among low-cost routers, many models support BGP.
   – Can also be used at home and on small networks.
2. Ubiquiti EdgeRouter:
   – Provides BGP and advanced routing features at an economical price.
   – Advanced network settings possible in small environments.
3. Cisco or Juniper:
   – BGP and complex network configuration possible with high-performance equipment.
   – The price is high so it may not be suitable for home environments.

Software Alternatives

1. FRR (Flexible Routing):
   – Open source routing software that can run on Linux-based systems.
   – Supports both BGP peering and NAT.
2. Bird:
   – Lightweight routing daemon, supports multiple routing protocols including BGP.
3. VyOS:
   – A complete network operating system that acts as a router and firewall.

Installation environment

• You can also run a software router on low-cost hardware such as a Raspberry Pi.
• You can also install FRR/Bird on an existing PC or server to operate as a router.

4. Is it possible to bypass BGP with an IPTime router?

• Direct support: BGP cannot be implemented directly on IPTime routers.
• Indirect Implementation: You can set up BGP peering by using an IPTime router as an Internet-facing device, followed by a software or hardware router that supports BGP.

Configuration example

plaintext
코드 복사
[인터넷]
     |
[IPTime 공유기]
     |
[Raspberry Pi 또는 소프트웨어 라우터(FRR/Bird)]
     |
[Kubernetes 클러스터 또는 내부 네트워크]

5. Conclusion

• IPTime routers do not support BGP peering. This is due to the design purpose of the device and the requirements of home environments.
• If BGP peering is required, it is necessary to use an advanced hardware router such as MikroTik, Ubiquiti or a software-based router (FRR, Bird).
• In a home environment, using an IPTime router as the default Internet gateway and configuring additional BGP-enabled devices is a realistic alternative.

4o

KubeVirt Virtual Machine (VM) vs Bare Metal Direct Install Router Architecture Comparison

If you want to create VMs using kubevirt and build a load balancer using MetalLB in a single-node bare metal Kubernetes environment, where to install router software (FRR, Bird, etc.) will depend on your use case and ease of management. We will help you make the best choice by comparing the pros, cons, and considerations of installing on a bare metal node and installing on a kubevirt VM.

1. Install router software directly on single node bare metal

Advantages

1. Performance optimization:
   – Low performance overhead because the router software runs on the same physical host as the Kubernetes cluster.
   – Network delay is minimized as it does not go through the VM layer.
2. Simplification of structure:
   – Network configuration is simplified as there is no need to manage separate VMs.
   – Kubernetes cluster and MetalLB can use the same network interface.
3. Quick installation and setup:
   – You can install and run the router software right away without creating or setting up a VM through kubevirt.
4. Resource Efficiency:
   – There is no CPU or memory overhead due to VM creation.

Disadvantages

1. Lack of environmental separation:
   – Since the router software and Kubernetes cluster run on the same host, any software conflicts or configuration errors may affect the cluster.
   – May increase management complexity during updates or reboots.
2. Limited scalability:
   – If you plan to scale up to a multi-node cluster, you may need to reconfigure the router software installed on bare metal.
   – Difficult to transfer router software settings to another node.

2. Install the router software into the kubevirt VM

Advantages

1. Environmental isolation:
   – The router software runs on a separate VM, so it can be managed independently of the Kubernetes cluster.
   – No impact to Kubernetes cluster when testing or updating router software in VM environment.
2. Flexible network configuration:
   – The network interface of the VM can be freely configured.
   – Separate network paths can be set up between the router software and the Kubernetes cluster.
3. Easy to scale to multi-node clusters:
   – VM running router software can be moved or cloned to another node.
   – Networks designed based on VMs have structural advantages when expanding.
4. Reusability:
   – The VM running the router software can be backed up or reused in another cluster.

Disadvantages

1. Performance overhead:
   – Additional resources (CPU, memory, network delay) are required to create and run a VM.
   – Network performance may slightly decrease as it must go through the VM network layer.
2. Installation and setup complexity:
   – Requires additional work to create a VM using kubevirt and install and configure router software.
   – Requires integration of the VM’s network interface with MetalLB and Kubernetes cluster.
3. Additional administrative burden:
   – The VM’s operating system and software must be updated and maintained.

3. Key Considerations

Design and Operational Requirements

• Simplicity First:
• If you want simple configuration in a single node environment, installing the router software directly on the bare metal node is appropriate.
• Also suitable for use in test and development environments.
• Isolation and scalability first:
• If you care about environmental isolation, flexibility, and multi-node scaling, installing the router software on a kubevirt VM makes sense.
• VM-based configuration is advantageous when reliability and scalability are important in the operating environment.

Performance and Resources

• Installing directly on bare metal avoids the performance overhead of the VM layer.
• In resource-limited environments, direct installation on bare metal is suitable.

4. Comparison table

5. Recommendations

1. Installation on bare metal:
   – Simple test environment, when resource constraints are large.
   – When there are no plans to expand to a multi-node cluster and performance is important.
   – An environment that requires quick installation and setup.
2. Install on kubevirt VM:
   – An operating environment where stability and isolation are important.
   – If there are plans to expand to a multi-node cluster in the future.
   – When you want to flexibly manage network configuration using VMs.

6. Conclusion

• Testing and small-scale environments: Installing router software directly on bare metal is simple and efficient.
• Environment for operational and scalability: Installing the router software on a kubevirt VM provides better isolation and flexibility.

4o

KubeVirt network routing loop prevention and BGP filtering architecture

Yes, this is a very important point! If MetalLB‘s BGP peering target is a router created within kubevirt, a loop structure may occur between the network managed by MetalLB and the kubevirt network. This is especially problematic when network traffic paths refer to themselves recursively through routers.

Why can loop structures occur?

1. Role conflict between MetalLB and kubevirt router:
   – MetalLB advertises (BGP) the external IP of the Kubernetes service to the network.
   – When a router installed on kubevirt becomes the BGP peering target of MetalLB, MetalLB will attempt to forward external traffic through this router.
   – However, this router exists within the kubevirt VM and forwards traffic back to the network managed by MetalLB, which may cause a route loop.
2. Circulation of routing path:
   – Circulation in the form of MetalLB → kubevirt router → MetalLB → kubevirt router.
   – This causes network traffic to cycle infinitely, or traffic forwarding fails because the route is not set up properly.

Can it be resolved

Is it structurally possible?

By default, direct BGP peering between MetalLB and the kubevirt router is only possible if the following conditions are met to avoid loops:

1. Clear separation of routing domains:
   – MetalLB and kubevirt routers must manage different network zones.
   – MetalLB is responsible only for advertising the external IPs of the Kubernetes cluster, and the kubevirt router should be configured to only handle traffic to the MetalLB external network.
2. BGP peering path restrictions:
   – Circular routes must be blocked using AS-Path Prepending or Route Filtering in BGP.
   – The kubevirt router should filter routes advertised by MetalLB to avoid advertising them back to MetalLB.
3. Isolate router internal network:
   – Clear separation of kubevirt router and MetalLB network at physical or virtual network level.
   – Configured so that the two networks do not circulate directly.

Recommended configuration

1. Install router software on bare metal

• Advantages:
• MetalLB’s network and kubevirt VM network are completely separate.
• Router software handles MetalLB external IP advertising and traffic processing independently.
• BGP peering target does not create loop.
• Configuration example:
• Run router (FRR/Bird) on bare metal.
• MetalLB sets up BGP peering with bare metal router.
• kubevirt VM only runs on MetalLB’s network, no additional routing required.

plaintext
코드 복사
[External Network]
       |
 [Baremetal Router (FRR)]
       |
 [Kubernetes + MetalLB]
       |
 [kubevirt VM Network]

2. Install the router into the kubevirt VM (with loop prevention settings)

• Conditions:
• Clearly blocks circular paths between MetalLB and kubevirt router.
• Configure Route Filtering in BGP peering to prevent loops.
• Configuration example:
• Install router (FRR/Bird) on kubevirt VM.
• MetalLB establishes BGP peering with the kubevirt router, but filters external IP ranges managed by MetalLB from being advertised back to the router.

yaml
코드 복사
# MetalLB ConfigMap 예시 (BGP 필터 설정 포함)
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 192.168.0.2        # kubevirt 라우터의 IP
      peer-asn: 65001                # kubevirt 라우터 ASN
      my-asn: 65002                  # MetalLB ASN
      import-policy: no-loop         # BGP 경로 순환 필터링

• BGP filtering example (FRR settings):
• Disable re-advertising MetalLB routes on the kubevirt router.

bash
코드 복사
vtysh
router bgp 65001
  neighbor 192.168.0.3 remote-as 65002
  address-family ipv4 unicast
    route-map FILTER_OUT out
exit
route-map FILTER_OUT permit 10
  match ip address prefix-list BLOCK_METALLB
ip prefix-list BLOCK_METALLB deny 192.168.0.0/24
ip prefix-list BLOCK_METALLB permit 0.0.0.0/0

Optimal configuration recommendation

1. Testing and small-scale environments:
   – Installing the router on bare metal: Simple setup, no need to worry about circular paths.
2. Consideration of operation and scalability:
   – Install the router on the kubevirt VM: Provides environment isolation and flexibility, but prevents loops through BGP filtering.

Conclusion

• If you do not configure loop prevention correctly, installing a router on the kubevirt VM and setting up MetalLB and BGP peering will most likely result in a loop structure.
• Installing the router on bare metal is a simple and reliable method.
• To install a router on a kubevirt VM you must configure BGP filtering and network isolation correctly.

4o

Container-based BIRD router vs. VM-based router architecture comparison

Running Bird (or other router software) as a container on bare metal Kubernetes nodes is another alternative to installing the router software. Let’s compare this with the traditional Direct installation on bare metal or Install on VM and look at the pros and cons and implementation feasibility of each.

1. Configuration to run Bird as a container

Running the Bird routing daemon as a container on a Kubernetes node allows the following configurations:

Structure

plaintext
코드 복사
[External Network]
       |
 [Bird Container (Kubernetes Pod)]
       |
 [Kubernetes (MetalLB + Workloads)]

How it works

1. Bird establishes BGP peering with networks outside the Kubernetes cluster.
2. MetalLB advertises to or receives advertisements from external networks through Bird containers.
3. Connects the internal Kubernetes cluster and external networks.

2. Running Bird in Containers: Pros and Cons

Advantages

1. Kubernetes native management:
   – Since Bird runs as a Kubernetes Pod, you can take advantage of Kubernetes’ deployment and management features (e.g. Deployment, ConfigMap).
   – YAML file defines settings and makes version management easy.
2. Flexible scalability:
   – Bird containers can be redistributed or updated as needed.
   – Easy to move to other nodes.
3. Environmental isolation:
   – Bird runs in isolation from the underlying operating system of the Kubernetes nodes, reducing the potential for conflicts with system configuration.
4. Simplify deployment:
   – Quick installation and setup using Bird container image.
   – Consistent network management within Kubernetes clusters.

Disadvantages

1. Network interface issue:
   – Setting up BGP peering with external networks can become complicated if the Bird container is not running in host network mode.
   – The container must properly recognize the default network interface of the Kubernetes node.
2. Additional setup required:
   – When running Bird in a container, additional settings are required for BGP peering and MetalLB network settings.
3. Performance overhead:
   – Possible slight performance degradation due to container network layer.
4. Increased complexity:
   – Additional work required to manage the boundary between the Kubernetes network and external networks.

3. Direct Bare Metal Installation: Pros and Cons

Advantages

1. Performance optimization:
   – Bird runs directly on the host operating system, so there is no network performance degradation.
   – Direct communication with network interface.
2. Simplify setup:
   – Bird is set up separately from Kubernetes, so it is not affected by the container network layer.
   – Simple network interface configuration.
3. Stability:
   – Bird is not affected even if the Kubernetes cluster is restarted or changed.

Disadvantages

1. Difficult to manage environment:
   – Bird settings are managed separately from the resource management system of the Kubernetes cluster, which can make changing and deploying settings complicated.
   – Setting files or logs must be managed manually.
2. Lack of scalability:
   – Bird installed directly on a node is difficult to move or clone to another node.
   – Additional work required when scaling to a multi-node Kubernetes cluster.
3. Operating system dependencies:
   – Bird’s stability may be affected when changing settings (e.g. upgrades) of the host operating system.

4. Container vs Bare Metal Direct Install vs VM Comparison

5. Configuration recommendations

Recommended environment for running Bird as a container

• If you want to manage your router software in a Kubernetes native way.
• An environment that values configuration consistency and scalability.
• If you want to benefit from Kubernetes management at a slight performance penalty.

Recommended environment for bare metal direct installation

• For simple setup in a test or single node environment.
• If you have a performance-critical environment and want to avoid additional layers (containers or VMs).
• If you want to operate the router completely separately from Kubernetes.

Recommended environment for installing Bird on VM

• Operating environments requiring a high degree of environmental isolation.
• If you value the independence and stability of the router software and want to take advantage of VM.
• When VM replication or migration is required in a multi-node environment.

6. Conclusion

• Testing and Simple Environment:
• Direct bare metal installation is the simplest and requires less additional setup.
• Kubernetes native management:
• Container-based Bird leverages the benefits of Kubernetes management and provides environmental isolation and scalability.
• Operational and Advanced Environment:
• VM-based Bird installation is the most secure and suitable when considering multi-node scaling.