At this point, to further our experiments and to better understand how k3s operates, we need to add some more nodes. To start off here, we are not yet going to add any more master nodes for HA. Instead, we’re just going to add some worker nodes so we can see the difference in how NodePort, servicelb, and eventually Traefik interact with each other.

First we have to create some new nodes. I saved my Ubuntu server as a template, so I just cloned a few of those. Then, we have to install k3s again.
The command to do this is remarkable similar to what we used before, with just one small addition:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken K3S_NODE_NAME=anynodename sh -s -

You can see I’ve added 2 environment variables to the command and removed the others that were specific to the master. The first is the location of the master node, while the second is the token. In a single master multi worker installation, this will be all you need. This will get a little more complicated as we set up our HA cluster, but for the time being, let’s roll this out to experiment. I have also added an additional variable called K3S_NODE_NAME because all my nodes come from the same master image, so they all get the same hostname, which I need to fix later.

We should know our master’s IP address by now, but what about it’s token? SSH into your master node and do:

cat /var/lib/rancher/k3s/server/node-token

Insert this token along with the masters IP address in the install command and run it on the new node. That’s it! You have now added a new worker node to the cluster that can now share some load. If you go back to your dashboard and scale up our demo app, then go to pods, you will see which nodes have which pods on them.

https://rancher.com/docs/k3s/latest/en/installation/network-options/

So how does this affect our demo app, or any app, running on our cluster and how we interact with them? Let’s first look at ClusterIP. ClusterIP, as we have seen before, is the most basic form of load balancing we have available in this setup, but is not true load balancing. Instead, it just gathers all of the running pods for a given replica set (which is created from our deployment) and picks one to pass traffic to on each new request. This is done in k3s using an application called flannel. Flannel creates connections to containers using a Container Network Interface (CNI). By default, the CNI is VXLAN, but traffic can also be encrypted using the ipsec or wireguard backends, which I think I would like to do in the future.

NodePort builds on top of this and allows you to expose the ClusterIP from a specific port on the host node’s physical IP address. By default, It exposes the same port on all running nodes, meaning that you can route to any host node on that port and you will be passed directly to one of the pods running your service, even if that pod is running on a different host.

A LoadBalancer service works in much the same way most of the time, by building on top of NodePort and exposing a service across all the nodes’ physical addresses, but then, in theory, creating a virtual IP address that load balances between all the physical ones. This however, as you may have noticed, is not what our current setup actually does. That is because the LoadBalancer service isn’t standardized within the Kubernetes stack, and instead relies on 3rd party integration. K8s is built to integrate with hosted Kubernetes cloud providers, where load balancing is done through whatever virtual or physical load balancing they choose to use.

K3s comes with klipperlb/servicelb, which as far as I can tell simply passes through a port to all host nodes. Technically, klipper doesn’t actually use the NodePort service underneath, but instead creates DeamonSet for the service listening on a HostPort, which then creates a proxy pod on each node which forwards the traffic to the actual service. It’s kind of convoluted and easier to just think of it like NodePort, because it functions the same. We aren’t really going to use Klipper at all in our setup, so it doesn’t matter.

Finally, we aren’t ready to talk about ingress yet.

Load balancing and creating an HA cluster:

https://www.linode.com/docs/guides/beginners-guide-to-kubernetes-part-2-master-nodes-control-plane/

https://kubernetes.io/docs/concepts/

https://rancher.com/learning-paths/introduction-to-kubernetes-architecture/

K3s HA is different from k8s HA. k3s actually handles client side HA on it’s own quite well. With a k8s installation, you would typically have a dedicated load balancer installed in front of your control-plane. Ok new term, What is a control-plane? Kubernetes assigns roles to each of it’s nodes. We have 2 types of nodes, masters and workers. The master nodes are the control-plane; they control the cluster. Kubernetes assigns the control-plane role to all master nodes, but it is technically possible to have nodes that are part of the control-plane that are not masters. This is extremely unusual.

Back to load balancing. With k8s, you would have a dedicated load balancer that points to all of your master nodes, then you would point all of your worker nodes to the load balancer. With k3s, this isn’t necessary. Once your control-plane is configured for high availability, you can just point your worker nodes at one of the masters, and that worker node will connect to the Kubernetes api server to find all the other masters. Each k3s agent (worker node) runs it’s own internal load balancer based on the tcpproxy go project, which handles load balancing to the api server all on it’s own. It’s also completely dynamic, meaning as nodes are added or removed (or go unexpectedly down), all the workers reroute accordingly. This is convenient if you just want to cut out some of the complexity of setting up an HA cluster. It’s unclear whether this would be considered bad practice in a production environment, but considering what k3s was built for, running clusters on the edge, it makes sense to use. This is apparently perfectly ok, since according to the creator of k3s Darren Shepherd if you do use a load balancer instead, k3s will just learn all the IPs and bypass your load balancer completely in favor of handling everything client side. That is good to know if you need to troubleshoot something.

Going HA:

Ok, so at this point, if we want to expand beyond our single master and get into load balancing, we need to set up more masters and build an HA cluster. Now, it’s worth mentioning that if all you want to do is expose some services with a reverse proxy, you can do that without a load balancer. You can even do it without an ingress controller if you want to make all your configurations manually (which I might). When we get to setting up Traefik, which is likely going to be what I use, just know that you can also run it on our single node cluster as well.

K8s Networking:

Kubernetes pods are each given their own unique IP address, but this IP address rarely if ever exists on your physical network. Most of the time (except when using Kubernetes on a cloud provider that uses kubenet or some other custom integration) Kubernetes will use what is known as an overlay network. This can be accomplished in multiple ways, but in a nutshell, it wraps packets that an underlying network doesn’t know how to handle (for example using pod IP addresses) within an outer packet which the underlying network does know how to handle (for example node IP addresses). This removes some of the complexity and dependency on the physical network layer. This is a really really big topic and worth discussing more in depth, but the specifics are not especially important unless you want to actually change this behavior, which there are legitimate reasons for wanting to do so.

References:

https://docs.projectcalico.org/networking/determine-best-networking

https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/

Next time, we are going to take what we have learned here today to make our first Highly Available k3s cluster. After that, we will be getting back into Rancher territory.