Kubernetes inside Proxmox LXC

So you want to run Kubernetes on Proxmox, but you don’t want to use all the resources a VM is using. Then LXC containers is an option. But for setting up Kubernetes inside an LXC container you need to setup some special sauce.

Step 1: prepare the host

Because LXC containers share the host’s kernel, we have to prepare the host. This means disabling the swap and also loading a couple of modules.

First I adapt the sysctl file on the host:

vim /etc/sysctl.conf

Uncomment the following line:

And add:

vm.swapiness=0

Disable swap:

swapoff -a

After that adapt the fstab file:

vim /etc/fstab

Disable swap on host

I did a reboot of the full host, I don’t know if this is needed but wanted to be sure…

Step 2: Create an LXC container

I both tested with Ubuntu and Debian, but for Debian it is important that you’re Proxmox version is at least 7.0.0. Else you will get errors…

Make the container unprivileged:

Setting up LXC to unprivileged

Disable swap inside the container:
Kubernetes will not initialize if the swap is not disabled.

Disable swap

Enable nesting:
You can find this options under “Options” after creating the LXC container.

Features: nesting=1

Step 3: Change container config file

Before we startup the LXC container, we have to change the config file that is created.

Edit the config file of the container:
You can locate the config file here: “/etc/pve/lxc/$ID.conf”. The $ID need to be filled in with the container ID of the container you just created. You can find the ID in front of the name.

Container ID

Add the following to the config file:

lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"

Step 4: Apply some configuration inside the LXC container

For the last step we have to create some missing files inside of the container, because in the Proxmox Ubuntu LXC template they are missing.

Create /etc/rc.local

#!/bin/sh -e
# Kubeadm 1.15 needs /dev/kmsg to be there, but it’s not in lxc, but we can just use /dev/console instead
# see: https://github.com/kubernetes-sigs/kind/issues/662
if [ ! -e /dev/kmsg ]; then
ln -s /dev/console /dev/kmsg
fi
# https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c
mount --make-rshared /

After creating this file we setup the permissions and reboot.

chmod +x /etc/rc.local
/etc/rc.local

Finally

Now the LXC container is setup for running Kubernetes. Also a handy tip, create a template so that you can spin up extra nodes very fast. The next steps are installing and setup of Kubernetes, which I will cover in part 2.

Next chapter:

https://kevingoos.medium.com/installing-k3s-in-an-lxc-container-2fc24b655b93