Why not Kubernetes and not containers

- 13 minutes read - 2621 words

Here I was going to describe the rough configuration of my four node cluster - however, based on questions I’ve gotten, there was a more pressing issue to address first.

First let me just re-iterate why I run the cluster at all: I want a neat little compute cluster at home from which I can serve this web site, my e-mail plus whatever other services I want for hobby purposes at home (like a git repo, file services etc.). This means running some combination of off-the-shelf software and some home grown software.

How do we run a set of basic services on a set of basic computers to achieve high availability, flexibility, and relative ease of maintainability?

Not Kubernetes

Let me just start with this - no this is not a use case for Kubernetes. In my opinion, most things are not. Kubernetes is intended to solve one problem: the running at very large scale of containerised services.

Whether something like Kubernetes is a good choice for the solution to that problem is a moot point, as I am not looking to solve the problem of running containerised services at scale. I very specifically do not want to run things “at scale” here - I want to run a small, maintainable, reliable and efficient set of services in a manner where I have basic fault tolerance. That problem is fundamentally different from what Kubernetes was intended to solve.

Without going further into what I think about Kubernetes (or the people who attempt to apply it to every problem they encounter), I hope to bring attention to the fact that the world of cluster computing is a lot richer than just Kubernetes. Clusters existed and were amazing long before Kubernetes, and they will continue to exist and be amazing long after Kubernetes.

I look forward to describing how I run my services using a combination of Corosync/Pacemaker for cluster resource management and orchestration, and glusterfs for shared file services, filesystem redundancy and configuration distribution. In this cluster there will be no need for service discovery layers, API routing engines, orchestration brokers or event streaming systems - and there will be no need for containers.

A note on containerised services

In my view there are a few things that need to be said about containers too - now before I do that, to avoid being labelled “anti container” or whatever, let me just share a quick story: I ran containerised services long before they became main stream. We had implemented an “unassisted signup” setup where potential customers could sign up on a web form for a live demo environment for some commercial software we were developing at the time. The way it worked, was, a container was automatically deployed with a fresh installation of our software upon signup, just for the one prospect, and then the customer could play with the setup for a month or so until it was de-comissioned automatically. Many such containers would share the same underlying hardware.

This was done using “Solaris Zones”, and the year was 2005. Linux did not support containers until 2008 and Kubernetes was not released until 2014.

So I understand good use cases for containers. I am no stranger to this. I am in no way “against containers”. However, I understand what it is they do well, and I see what they are being shoe-horned into solving. Those unfortunately are two very different sets of problems.

Containers excel at:

  1. Isolation. Whatever runs in one container can be isolated very well from both the host and other containers. This can be useful - both when running un-trusted or known problematic (maybe unmaintained and known-vulnerable) services, but also as part of defence-in-depth.
  2. Efficient virtualisation. For example needing to run a non-multi-tenant service to a multitude of tenants, the service needs to run many times and each instance needs to believe it is alone in the world. Virtualisation does this well, but containerisation is a low-overhead virtualisation that offers a small improvement in overhead.

Now for (1) above, you should never need to run un-trusted or unmaintained software. It is a terrible use case. It is one we should not optimise for. But sure, reality is not always ideal and this is - unfortunately - a case that needs a solution and containers can indeed be an important part of that solution. For defence in depth, the isolation mechanisms used by containers are also available without the container - much can be done to software to make use of those mechanisms but clearly containers offer a potentially more transparent way of ensuring certain isolation is applied.

For (2) above, the container is an optimisation. Containers are lower overhead than full blown virtualisation so sure, this is useful. This was my use case described earlier with the demo software deployments.

However, the two use cases above are typically not what I see people using containerisation for. Instead, containers have become the solution to other problems:

  1. Software distibution. Instead of getting a software package accepted properly into the most relevant Linux distributions (or working with relevant community to have them do it for you), it is seen as a shortcut to distribute a container image instead.
  2. Service abstraction. As a way to manage on which cluster node a given service runs, the container is used as the “service abstraction” - it is the container holding the service software and to some extent its configuration, rather than the service itself, that is started, stopped or migrated from one host to another.
  3. Software dependency management. Related to (1) above, if your software has a mess of particular dependencies, it can be difficult to actually package it for any current Linux distribution at all - by lobbing the entire hairball of dependencies into the container image, the shipped image will work on any distribution.

For (1) this is just laziness. It is not difficult to package for a particular distribution. If you’re a commercial vendor probably you have a CI pipeline anyway (I would hope) and you just add targets to build for. It’s basically a one-time investment. I used to run the product/technical side of such a business where we packaged for both commercial UNIXes, many linux distros and windows - across a multitude of hardware platforms too. Ain’t nothing to it but to do it.

For (2) I am sure this makes sense for some very particular setups. Google does something like this. Salesforce does. If you run tens or hundreds of thousands of physical servers and thousands of services on top of them, I understand that instead of updating all of your thousands of applications on all of your hundreds of thousands of servers, it is useful to only deploy the software needed for a service on to the servers on which the software needs to run. A container is one possible implementation of such an abstraction. A networked file-system would be another such abstraction. There are many possible solutions each with their own sets of pros and cons. However, most companies do not operate on that scale. For most companies, if it is difficult for you to deploy your software, fix that instead. Excellent low-tech solutions exist (or can be hand-crafted) for tasks as mundane as software and configuration deployment.

For (3) yes this solves a real problem. If your software depends on 20-50-100 libraries, most of them in very particular versions, then you will find it very difficult to ship against any operating system unless you ship the libraries with your software. That can cause clashes on linux distros that ship those libraries in other versions, and yes in that sense containers “solve” this by allowing you to bundle up your dumpster-fire of dependencies and neatly ship the whole thing combined.

However, that “solution” completely destroys some crucially important and fundamental concepts:

  1. Nobody can see your dependencies - they are all inside the image. What this means, is, everyone has to trust you, that you will remember to ship a new image when (not if) any of the many libraries you depend on has a security flaw.
  2. Operators can no longer patch security flaws. Running a security update on the operating system - which brilliantly updates all libraries on the system too (this is why the internet runs on linux - you cannot overestimate the importance of distribution wide security updates), running that update will not update the libraries inside your image (and if they did, your fragile software would cease to function). Security updating now becomes a developer task first, not just an operator task. That is game changing - but not in a good way.
  3. Your software should not be this fragile. You cannot build against the libraries that exist in common linux distributions? That should make you pause and reflect. I have done a lot of large scale software development on linux and I have certainly faced this problem - there are two good explanations that both have solutions:
    1. The library may be trash. Don’t depend on it. Implement the functionality yourself if there is no alternative library.
    2. You use either deprecated or cutting edge features - there are typically simple workarounds for this. Adapt your code to comply with what is reliably supported.

I understand that if you come from Windows or OSX, you don’t know what good looks like from a systems administrator perspective. You don’t know the power of being able to query your OS about any file you find on your computer (whether it’s part of your SQL database or your volume manager or your web server) and be told why it is there and whether it is modified from its original form and whether there is a security update available:

root@pluto:~# dpkg -S /usr/bin/rsync
rsync: /usr/bin/rsync
root@pluto:~# dpkg -V rsync
root@pluto:~# apt list -a rsync
rsync/stable 3.4.1+ds1-5+deb13u4 riscv64 [upgradable from: 3.4.1+ds1-5+deb13u3]
rsync/stable-security,now 3.4.1+ds1-5+deb13u3 riscv64 [installed,upgradable to: 3.4.1+ds1-5+deb13u4]

root@pluto:~#

You also wouldn’t know that of course a system update will security update all the software you have on your computer. That there would be no need to run multiple update applications, that you would never need to check for a vendor update, that once your system is patched your system is patched - end of story.

So if not knowing that, sure, shipping software in a container is no different from the normal experience on Windows or OSX where you get an application with some embedded set of library dependencies, and you understand and accept that when it comes to security you have no assurances, no insight and no agency.

But since 1994 with DEB and 1995 with RPM, there have been better ways of doing this. Package management. Dependency management. Security patch backports. It is beautiful. It is - in my view - no small part of why the internet runs on Linux.

Shipping software as a container is giving a big fat middle-finger to all of this. It is sacrificing everything we have learned about computer security since the dawn of computing on the altar of mediocrity - if you are too lazy to make your dependencies manageable, don’t ship your software in a container as a solution. It is hurting everybody - and it is effectively rolling back 30 years of progress.

Flatpacks, dpaks, snap

I can understand that people coming from outside the world of linux distributions will not know about proper packaging of software (as already covered). I don’t think it is an excuse, but I can understand it.

But Linux distributions - not least Ubuntu - pushing this? The mind boggles. I first noticed Ubuntu pushing snap - a software distribution mechanism optimised for shipping fragile software with unmaintainable dependencies. Basically the last thing you would want to optimise for shipping - or ship at all.

There are perfectly good mechanisms for allowing people access to more up-to-date versions of software - Debian has had “backports” since forever and Ubuntu could have done an even better job of that had they wanted to. But no, instead they chose to create a competing mechanism to the DEB package system they already depend on, add some mild form of containerisation (isolation makes up for shipping untrustworthy software?), and start pushing this on to their users. This is not about whether snap works or not (I did try using it - I didn’t find it to work very well), it is about the very concept being wrong. It solves the wrong problems for the wrong reasons.

Flatpacks are another example. Again it is being presented as both an “enabler of app stores” (because everybody needs an app store - except, again, linux had this since 1998 by means of the apt system, Apple didn’t get one until 2008), and a way to make it easier for commercial vendors to ship software for Linux.

You already know by now what I think of the second argument - as a vendor all you need is to pick a few distros to ship for and then do that. It is not difficult. And certainly, the “solution” to supposedly lazy vendors (did anyone actually ever get a request from a vendor to do any of this or is this problem entirely made up?) is not to enable the distribution of unmaintainable insecure trash.

Conclusion

This de-railed a little bit. I will continue describing what I actually did, rather than the whats and whys of what I did not do.

However, let me end this post with a few observations, keeping in thread with the above.

In daily life I work with a lot of smart and skilled people in all places of their careers. It is not surprising that a junior engineer is less experienced than a senior - and nobody expects everyone to know everything. But something that I am noticing - and I don’t think this is new - is a lack of interest in and understanding of the basics.

To keep just a little on track, let’s take package management. A lot of young systems administrators see no obvious problem in downloading the newest sources for a piece of software, compiling it an installing it on a server - versus getting the (slightly older version) software that is available in the distribution. The difference - of course - is night and day. If you download it, you now need to subscribe to security notifications, you need to re-download and re-compile and re-install, you need to deal with version incompatibilities, you own the entire end-to-end exercise of ensuring forward compatibility on short notice since this is in production. Contrast this to using the slightly older in-distro version: When there is a security update, you get it the same place as you get all the other updates - which you already have a workflow for. The version of your package doesn’t change, instead the security fix is back-ported for you so that there are no compatibility issues.

This is night and day people.

The same goes for downloading someones container. Outside of maybe a prototype of something, there is just no excuse. Software should not be that fragile - enabling fragile software is not helpful. When you install someone else’s mess, you not only take on significant risk (if they are too lazy to manage their dependencies, what on earth makes you believe they will diligently apply security updates to those dependencies and notify you?), but you lose your agency. You no longer have the power to keep your systems patched. If you don’t care about this I don’t know what to say.

Aim high people. Or at least, if you’re in a hurry, aim higher than containers, flatpacks and snap. This is not a high bar.

Cheers,

/ jakob