Setup picture

K3S: Setup MetalLB using BGP on Pfsense

For my internal K3S cluster I need a loadbalancer. While I can easily setup an nginx loadbalancer in front, I prefer to make the loadbalancer part of the cluster. So that it is also redundant.

For that cause MetalLB is a good fit. MetalLB has 2 different modes: Layer2 and BGP.

Layer2 mode

While layer2 mode will work fine, communicating over ARP. There are two disadvantages why I didn’t chose this option. First in layer2 mode there is a single elected leader. This means in turn that all traffic is send over this 1 single node. Second sometimes clients will handle the ARP cache incorrectly and this can cause a slow failover.

BGP mode

While BPG is harder to setup, because the router need to get the updates from the cluster. This will not have the above limitations. The only limitation there is, is when a node goes down, all active connections are broken. So this means users that have an active connection will get the “Connection reset by peer” message. But the failover will be instant.

Prepare PFSense

Install FRR package, goto System => Package Manager => Available Packages. And install the FRR package.

Now we go to the FFR settings under Services => FFR Global/Zebra.

First we have to setup the global settings. This means setting checking Enable FRR, setup Default Router ID (you’re default router IP) and Master Password (Random generated)

FRR global settings

Next we go to the BGP settings. You can find it in the tabs on top or you can go there by going to Services => FRR BGP

So here we have to again Enable BGP Routing, and set the Router Id again. Because I am working inside a subnet I also define the Network to Distribute.

BGP settings Part 1

BGP settings Part 2

Next we have to define our neighbors. This you can find in the Neighbors tab of course. We define 2 neighbors here.

Neighbors defined

First we start with defining the Peer Group. We give it a name and set the Remote AS.

Peer group definition

Next we define our Master node where metallb is running on so it can send BGP commands to our router.

We define the Address from the master node, you then give it a description and add it in the correct Peer Group. Last we also define the Remote AS again.

After this setup pfsense is ready to receive BGP Commands from metalLB

Configuring MetalLB

apt install curl sudo gnupg

Configuring MetalLB

First we create a config map used for configuring the metallb-pod.

So we create a file called metallb-values.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: metallb
data:
  config: |
    peers:
    - peer-address: 10.10.0.1
      peer-asn: 64512
      my-asn: 64513
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 10.10.10.100-10.10.10.255

Then we install metallb using helm, and apply the config.

helm repo add metallb https://metallb.github.io/metallb
helm repo update
kubectl create namespace metallb-system
helm install -n metallb-system metallb metallb/metallb -f metallb-values.yaml
kubectl create configmap metallb --from-file=metallb-values.yaml

If there are changes in the furture to the config, they can easily be applied using the following command:

kubectl apply -f metallb-values.yaml

Testing Metallb loadbalancer

Now we will test if the above config is working:

kubectl create deploy nginx --image nginx
kubectl expose deploy nginx --port 80 --type LoadBalancer

Now we need to figure out if the loadbalancer receives an IP.

kubectl get services

External IP gets appointed

You can also find the registered ip in PFSense under the status tab.

PFSense output

As you can see, the nginx loadbalancer service gets an external ip given by the router.

kubectl delete service nginx