Upgrading a cluster and backing up etcd

The last post built a cluster with kubeadm and went through its core components. Once a cluster works, it has to be maintained.

This post covers the two most important parts of that: upgrading the cluster to keep it secure and current, and backing up and restoring etcd to protect against data loss. Both are skills the CKA exam expects.

Upgrading with kubeadm

Upgrades are less about new features than about security, stability and performance. New releases often include critical security patches and bug fixes.

Before you upgrade

Back up etcd. Always do this before any major upgrade.
Follow the version skew policy. Skipping versions is unsupported, so the control plane moves one minor version at a time: 1.28 to 1.29, not 1.28 to 1.30. A kubelet can be at most three minor versions older than its control plane. kubectl is supported within one minor version of kube-apiserver, older or newer.
Read the release notes for your target version, looking for breaking changes.
Test the upgrade in a non-production environment first.
Make sure swap is disabled on all nodes.
Check your CNI plugin's documentation for any upgrade steps of its own.
Make sure the cluster is healthy before you start: every node Ready and every Pod running.

The upgrade, step by step

Upgrade the control-plane nodes first, then the worker nodes one at a time, so the cluster stays available throughout. The commands below upgrade to 1.30, for example from 1.29. Replace x with the latest patch version.

The first control-plane node

The Kubernetes packages from pkgs.k8s.io, which the last post installed, come from a separate repository for each minor version. Point apt at the new one first: in /etc/apt/sources.list.d/kubernetes.list, change the version in the URL, for example from v1.29 to v1.30.

Then update the package index and unhold the Kubernetes packages:

sudo apt-get update # For Debian/Ubuntu
sudo apt-mark unhold kubeadm kubelet kubectl

Install the new kubeadm:

sudo apt-get install -y kubeadm='1.30.x-*'

Check the upgrade plan, which tells you whether the cluster can be upgraded. Read its output for warnings:

sudo kubeadm upgrade plan

Apply the upgrade. This upgrades the control-plane components and etcd:

sudo kubeadm upgrade apply v1.30.x

Upgrade kubelet and kubectl, then hold the packages again:

sudo apt-get install -y kubelet='1.30.x-*' kubectl='1.30.x-*'
sudo apt-mark hold kubeadm kubelet kubectl

Restart the kubelet:

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Check that the control-plane node is Ready:

kubectl get nodes
kubectl get pods -A

Other control-plane nodes, in a highly available cluster

On each additional control-plane node, install the new kubeadm as above, then upgrade the node's components:

sudo kubeadm upgrade node

Upgrade kubelet and kubectl and restart the kubelet, as above. Then check that every control-plane node is Ready.

Worker nodes

Upgrade the worker nodes one at a time, so your applications stay up. On each worker node, install the new kubeadm as above.

Then drain the node, running kubectl from the control-plane node. Draining cordons the node and evicts all its Pods, except DaemonSet Pods, to other nodes:

kubectl drain <node-name> --ignore-daemonsets

Back on the worker node, upgrade its components:

sudo kubeadm upgrade node

Upgrade kubelet and kubectl and restart the kubelet, as above. Then uncordon the node from the control-plane node, which marks it schedulable again:

kubectl uncordon <node-name>

Check that the worker node is Ready before you move on to the next one.

After the upgrade

Once every node is upgraded, check the cluster's overall health with kubectl get nodes and kubectl get pods -A, and test your deployed applications.

Backing up and restoring etcd

etcd is the brain of the cluster. It stores all cluster data: configuration, state and metadata. If you lose etcd's data, you lose the cluster, so regular backups and a solid restore plan are not optional.

The tool for both jobs is etcdctl, etcd's command-line client.

Back up etcd

SSH into the control-plane node where etcd runs, and install etcdctl if it isn't there:

sudo apt install etcd-client -y # For Debian/Ubuntu

etcd usually runs as a static Pod. Its configuration, including its endpoints and certificates, is in /etc/kubernetes/manifests/etcd.yaml. Typically:

the endpoint is https://127.0.0.1:2379
the CA certificate is /etc/kubernetes/pki/etcd/ca.crt
the server certificate is /etc/kubernetes/pki/etcd/server.crt
the server key is /etc/kubernetes/pki/etcd/server.key

Take the snapshot. Set ETCDCTL_API=3 on the command itself, after sudo, because sudo does not pass on variables exported in your shell:

sudo ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key

This writes a snapshot of etcd's data to /opt/etcd-backup.db. Verify it:

sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /opt/etcd-backup.db

Then move /opt/etcd-backup.db to secure storage outside the cluster, and consider encrypting it.

Restore etcd

Restoring etcd overwrites the cluster's state. Don't attempt it on a live cluster unless you understand what that means.

First stop all API server instances, so nothing writes to etcd while you restore it, and stop etcd itself. On a kubeadm cluster both run as static Pods, and moving their manifests out of the manifests directory stops them:

sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/

Restore the snapshot into a new data directory, which avoids conflicts with the old one:

sudo ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup.db \
    --data-dir=/var/lib/etcd-restored-data

Edit /tmp/etcd.yaml and change every occurrence of the old data directory, /var/lib/etcd, to the restored one, /var/lib/etcd-restored-data. Then move both manifests back:

sudo mv /tmp/etcd.yaml /etc/kubernetes/manifests/
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

Restart the kubelet so it picks up the changes:

sudo systemctl restart kubelet

Watch the logs of etcd and the other control-plane components to make sure they start successfully.

Good practices for etcd backups

Automate the snapshots, with a CronJob or another tool, and always store them securely, outside the cluster.

Test your restore procedure regularly in a non-production environment. It is the only way to know your backups are valid and your process works.

Keep the etcdctl version the same as your etcd server's.

The next post moves to application development, starting with how to manage configuration and secrets: ConfigMaps and Secrets.