Building Kubernetes-as-a-Service from bare metal

Suppose you are the cloud provider. There is no AWS or GKE underneath you to lean on. You have bare-metal servers, switches, and a mandate to build a managed Kubernetes product.

This post works through how that would be built: the architecture, the infrastructure you have to build yourself because there are no cloud APIs, and the decisions about control planes, traffic, upgrades and isolation. It ends with a much smaller version of the same idea that you can build on a laptop.

Control planes are a workload

You cannot just install Kubernetes and call it a service. You have to bootstrap it in layers. The standard pattern for a Kubernetes-as-a-Service (KaaS) platform with no outside dependencies is a management cluster, similar to Cluster API, adapted for bare metal.

At the bottom is a highly available Kubernetes cluster in each region that you manage by hand. Call it the seed. It runs the control-plane components, the API server and etcd, for your own internal services and for your customers' clusters. Those are hosted control planes.

The customers' workloads run on separate bare-metal servers, the data plane. Their kubelets connect back to the hosted control planes that run inside the seed.

The managed service itself is just a set of controllers running in that cluster:

a MachineController, which talks to your bare-metal provisioning API, such as Tinkerbell or Ironic, to wipe a server and install an operating system
a ClusterController, which creates a customer's control-plane Pods

Hosted control planes

A customer's control plane does not get three virtual machines of its own. It runs as Pods, which is the pattern behind HyperShift and Kamaji:

etcd runs as a StatefulSet with 3 replicas.
The API server runs as a Deployment with 2 or 3 replicas.
The controller manager and the scheduler each run as a Deployment, with leader election so that one replica is active and the rest stand by.

That leaves one problem. The API server runs in your management network and the customer's nodes in theirs, but commands like kubectl logs need the API server to reach the kubelet. A Konnectivity server, running as a sidecar container in the API server's Pod, handles it with a reverse tunnel to the worker nodes.

Cells limit the blast radius

A single giant management cluster per region does not scale. Instead, you split the platform into levels and use a cellular architecture.

A cell is a fixed-size unit of capacity, for example 500 physical servers. Each cell is a Kubernetes cluster that runs the hosted control planes, the API server Pods, for the customers placed in it. A highly available regional cluster manages around 50 to 100 cells.

The levels stack like this:

A global management cluster stores the records of regions and customers, including a TenantCluster object for every customer cluster. A federation controller pushes each one to the right region to be built. This is the single place where customers see all their clusters across regions.
The regional cluster receives those requests and picks a suitable cell for each new cluster.
The cell runs the customer's control plane.

The reason is blast radius. If a cell's control plane fails, only the customers in that cell are affected, not the whole region. The same reasoning applies one level up: if the global control plane goes down, customers cannot create new clusters, but every existing cluster keeps running.

Failure domains

The scheduler can only spread work across failures it knows about, so you have to expose the topology to it. There are three levels:

the region, such as us-east-1
the zone, such as us-east-1a, with independent power and cooling
the rack, such as rack-12, whose servers share a top-of-rack switch

Every node you provision gets labels for its place in that hierarchy, such as topology.kubernetes.io/zone=us-east-1a. Then the tenant control-plane manifests use topologySpreadConstraints with a maximum skew of 1 replica across zones. That forces the scheduler to put etcd member 1 in zone A, member 2 in zone B and member 3 in zone C.

etcd, the fragile part

etcd needs fast, persistent storage. If a management node dies, its Pods move to another node, but the data has to follow them, and you have no EBS to make that happen.

There are two ways to provide the storage. Local NVMe disks on the management nodes give the best performance. You let an operator manage etcd's membership, and when a node dies, the operator removes its etcd member and adds a new, empty one on a different node, and etcd replicates the data to it. Network storage is the safer alternative: Ceph, managed by Rook, on a dedicated storage cluster, exposing block devices (RBD) to the management cluster.

Either way, don't manage etcd by hand. Write an operator that:

uses PersistentVolumeClaims backed by fast local NVMe storage, through volume claim templates
runs etcdctl defrag periodically to reclaim space
takes snapshots on a schedule and stores them in S3-compatible object storage
recovers from a permanent loss of quorum by downloading the latest snapshot, starting a new single-node etcd cluster from it to restore service, and then scaling back up to 3 members

Scale brings one more problem. With 5,000 hosted control planes you are running 15,000 etcd Pods. The fix is to share etcd: run one large, carefully tuned etcd cluster and give each tenant its own key prefix, instead of a dedicated etcd cluster per tenant. Kine and Kamaji both work this way.

Networking without a VPC

Without a cloud underneath you, there are no VPCs either, so you build the network yourself. It has to solve two problems. Isolation: customer A must not be able to reach customer B. And reachability: an outside load balancer has to be able to reach a Pod IP inside a Geneve tunnel.

The answer is BGP with EVPN (VXLAN). The underlay is an ordinary layer 3 IP fabric in a Clos topology, where every top-of-rack switch speaks BGP to the servers. The overlay is Cilium or Calico in VXLAN or Geneve mode, and each tenant cluster gets its own VNI (virtual network identifier). Traffic leaving a cluster goes through a dedicated egress gateway node, which translates it to a public IP address (SNAT).

MetalLB is great for a homelab, but a provider needs its BGP mode, at scale. Each node opens a BGP session with its top-of-rack switch, and when a LoadBalancer Service is created, the node advertises the Service's /32 address to the switch. The limit is that flows get re-hashed when nodes flap, so the switches need ECMP (equal-cost multipath routing).

Isolation between tenants starts with a network policy that denies all traffic by default. Below that, every packet is tagged with its tenant's VNI, the physical switch ports only accept packets that are correctly encapsulated, and MAC filtering on the ports prevents ARP spoofing.

Getting traffic in

Traffic arrives at two levels: from the internet into your datacenter, and from the datacenter to a Pod.

At layer 4, all your edge routers announce the same virtual IP for a load-balanced service, with BGP anycast. Behind the routers, a fleet of commodity servers runs a software load balancer built on IPVS or XDP (eBPF), like Facebook's Katran or Google's Maglev. It hashes each packet's 5-tuple, the source and destination IP and port plus the protocol, and forwards the packet to a specific node.

At layer 7, once a packet reaches a node, an ingress controller such as Envoy or Nginx terminates TLS. You use this layer for your own control plane too, to route api.customer-1.cloud.com to the right API server Pod in the management cluster.

Across regions, a customer who deploys an application in US-EAST and EU-WEST needs users sent to the closer one. That takes DNS-based global load balancing on a programmable DNS server, such as CoreDNS with a custom plugin, or PowerDNS. A controller health-checks the layer 4 load balancers in each region, and geo-IP decides the answer: when a user in New York looks up my-app.cloud.com, the DNS server returns the address of the US-EAST load balancer.

Storage: building your own EBS

Kubernetes expects block devices that can be detached and attached with no fuss. Without a cloud, you provide them with Ceph, managed by Rook.

A dedicated cluster of dense storage servers runs Ceph. The management cluster runs control loops that provision Ceph RBD images, and the tenant nodes run the Ceph CSI driver, which maps the RBD image over the network when a Pod starts.

Configure quality of service (QoS) on the storage layer. Without it, one noisy tenant consumes the IOPS of everyone else.

Secrets: you are the root of trust

A cloud provider cannot use Kubernetes Secrets, which are only base64 encoded, as its root of trust. You are the root of trust, so you build an internal key management service (KMS) backed by hardware security modules (HSMs).

Tenant etcd data must be encrypted at rest. Use the Kubernetes KMS provider interface (v2): you write a gRPC plugin that runs on the node. When the API server writes to etcd, it calls the plugin, and the plugin calls your internal KMS to encrypt the data encryption key (DEK).

Your own internal applications get their secrets from a sidecar injector, such as Vault Agent. The applications never see a secret on disk, only in tmpfs memory or in environment variables.

Upgrades without outages

Updates cause more outages than anything else, so the infrastructure is immutable.

Control planes upgrade blue-green. To move a customer from Kubernetes v1.30 to v1.31, you start a new API server and controller manager at v1.31, wait for their health checks to pass, switch the layer 7 load balancer to the new Pods, and shut the old ones down.

Nodes are never upgraded in place. You never run yum update on a live node. Each node is replaced in turn instead:

Cordon node A.
Drain node A, evicting its Pods.
Reboot node A into the new operating system image, through PXE or kexec.
Uncordon it.

Isolating tenants in rings

You control the hardware, so isolation can go all the way down, in layers.

The first ring is the scheduler. Customer A's nodes carry the taint tenant=A, and only customer A's Pods tolerate it.

The second ring is the kernel. Don't trust the Linux namespace boundary, because it can be broken through. Run untrusted customer Pods in Kata Containers microVMs, or in gVisor. If a workload escapes its container, it is still inside a hypervisor, KVM or Cloud Hypervisor. Kata solves a second problem as well: your nodes run one particular kernel, but customers may need kernel features it lacks. In a microVM, each customer's Pods bring their own kernel.

The third ring is the network, described above: deny by default, VNI tagging, and switch ports that reject anything not correctly encapsulated.

Bootstrapping a region

To bring up a new region you need a Kubernetes cluster, but you need a Kubernetes cluster to provision the servers. The way out is a seed node. A script brings up a single-node, temporary Kubernetes cluster on the first server. That cluster provisions the next 3 to 5 servers into the highly available management cluster, and once that cluster is healthy, it absorbs the seed node.

Watch for circular dependencies along the way. If your DNS server runs on Kubernetes, and Kubernetes needs DNS to start, neither one ever comes up. Run the critical infrastructure, DNS, DHCP and NTP, outside the loop, or use static configuration during the bootstrap.

The stack, in short

Provisioning: Tinkerbell or Metal3
Orchestration: Cluster API or Kamaji
Networking: Cilium, with BGP and VXLAN
Storage: Rook, with Ceph
Isolation: Kata Containers or Firecracker

Hosted, virtual or standalone

Hosted control planes are not the only way to give each customer a cluster:

	Hosted control plane (HyperShift, Kamaji)	Virtual cluster (vCluster)	Standalone (Cluster API)
Architecture	Control plane runs as Pods in the management cluster	Its own control plane in a Pod, syncing resources to the host cluster	Control plane runs on separate VMs
Cost	Low, shared infrastructure	Very low	High, dedicated VMs
Isolation	Medium, the control planes share a kernel	Low, workloads share the host cluster's nodes	High, dedicated machines
Use case	A managed Kubernetes product	Development environments	On-premises, VMware

Managed services such as GKE, EKS and AKS keep the control plane out of your sight and run it on infrastructure they own. The hosted pattern is the way to offer the same thing from a management cluster of your own.

A version you can build on a laptop

A sandbox version of all this is much smaller. The goal is a management plane, running on your machine in kind, that exposes an API. When a user calls create-cluster, it starts a new, fully isolated Kubernetes control plane, with its own API server, controller manager and scheduler, and provisions worker nodes that join it. Call it Minimoke.

Learn the parts first. You need to know kubectl, Pods and Go, and then how the components fit together: etcd as the source of truth, the API server and its authentication, the controller manager and scheduler, and how a kubelet registers itself. Work through Kubernetes The Hard Way, and don't just run the commands. Read the certificates and work out exactly who is allowed to talk to whom. A cluster is defined by its certificate authority: if two binaries trust the same CA, they are in the same cluster.

Milestone 1: by hand

kind create cluster --name mgmt
Generate a CA and the kube-apiserver certificates yourself.
Run etcd and kube-apiserver as Pods in kind, with kubectl run or YAML files.
Check that kubectl can talk to this inner API server.

The inner API server is only a Pod IP inside the management cluster, so put a LoadBalancer Service in front of it. When you generate its certificate, include that load balancer's external IP in the subject alternative names (SANs).

Milestone 2: an operator

Run kubebuilder init, with Kubebuilder or the Operator SDK, and create a TenantCluster API:

apiVersion: kaas.io/v1
kind: TenantCluster
metadata:
  name: my-cluster
spec:
  version: v1.30.0
  controlPlaneReplicas: 3

Then write the Go code that automates milestone 1. For each TenantCluster, the controller generates a self-signed CA and the API server's certificates, creates a StatefulSet for etcd with the certificates mounted, and creates a Deployment for kube-apiserver, passing it --etcd-servers and --client-ca-file.

Milestone 3: a worker node

A real cloud calls an API to create a VM. In the sandbox, the easy option is a Docker container as the node, the way kind does it. The hard option is a Firecracker microVM or QEMU, which is closer to the virtual machines the clouds use for isolation.

Start a plain container: docker run -d --name worker ubuntu:20.04.
Install kubelet and containerd inside it.
Point the kubelet at the inner API server's IP.
Fix certificate and network reachability problems until kubectl get nodes shows the worker.

On a real node, cloud-init starts the VM, the node downloads kubelet and a bootstrap kubeconfig, and TLS bootstrapping does the rest: the kubelet sends a certificate signing request (CSR) to the API server, and the controller manager approves it.

The network is the hardest part. A CNI plugin such as Calico, running inside a node that is really a Docker container, tries to behave as if it owns the network stack. Kindnet, or Flannel with a simple VXLAN backend, works better. For the first version, skip the CNI entirely and run only the kubelet: the node reports Ready even though its Pods cannot reach each other.

What will bite you

etcd latency: if the management cluster is slow, the tenant clusters crash. Use fast disks.
IP ranges: the management and tenant networks must not collide.
Draining: upgrading the management cluster without killing the tenant control planes needs careful PodDisruptionBudgets.

Start small. Don't try to build EKS on day one. A system that starts one API server and one node is already a real achievement.