Troubleshooting cluster failures

The last post covered the basics of debugging Pods, Services and Deployments, and why monitoring and logging matter. This one moves on to harder problems:

• cluster failures, which affect the whole cluster or its core components
• audit logs, the API server's detailed record, for security investigations and deep debugging

Both matter for keeping a cluster healthy, and both are relevant to the CKA and CKS exams.

1. Troubleshooting cluster failures

When a whole cluster or its core components misbehave, the investigation has to be more systematic, and usually goes deeper.

A workflow

Always work through the problem in order:

1. Define the problem. What exactly is failing: one application, a node or the whole cluster?
2. Check the cluster's health, starting with a high-level overview.
3. Isolate the issue. Decide whether it sits at the level of Pods, nodes, networking, storage or the control plane.
4. Gather information with kubectl commands, logs and monitoring tools.
5. Analyze it to find the root cause.
6. Apply the fix.
7. Verify that the problem is resolved.

kubectl commands for the whole cluster

kubectl get gives quick status checks:

• kubectl get nodes shows each node's status, Ready or NotReady.
• kubectl get pods -A lists every Pod in every namespace.
• kubectl get events lists recent cluster events.

kubectl describe gives the detail:

• kubectl describe node <node-name> shows a node's capacity, allocated resources and events.
• kubectl describe pod <control-plane-pod-name> -n kube-system does the same for the Pods of control-plane components.

kubectl top nodes and kubectl top pods show resource usage. They need the Metrics Server.

kubectl cluster-info dump prints extensive information about the state of the cluster.

Node problems

The symptoms: a node's status is NotReady, Pods won't schedule on it, or applications on it perform badly.

Find the NotReady nodes:

kubectl get nodes

Inspect one:

kubectl describe node <node-name>

Look at its Conditions, such as KubeletReady, MemoryPressure and DiskPressure, and at its Events.

Log into the node and check its CPU, memory and disk space with operating system tools such as htop, df -h and free -h.

The kubelet is the node's main agent, so read its logs on the node:

journalctl -u kubelet # For systemd-based systems

Finally, make sure the node can reach the Kubernetes API server and the other nodes.

Control-plane problems

The symptoms: kubectl commands hang or fail, Pods don't get scheduled, or the cluster's state is inconsistent.

In a kubeadm cluster, the control-plane components, the API server, the scheduler, the controller manager and etcd, usually run as static Pods in the kube-system namespace. Check their status:

kubectl get pods -n kube-system

Inspect the ones that look wrong, and read their logs:

kubectl describe pod <control-plane-pod-name> -n kube-system
kubectl logs <control-plane-pod-name> -n kube-system

Log into the control-plane nodes and check the logs of kube-apiserver, kube-scheduler, kube-controller-manager and etcd. Common locations are /var/log/kube-apiserver.log, /var/log/kube-scheduler.log and /var/log/kube-controller-manager.log. For etcd, read its Pod's logs, or the logs of the etcd process if it runs directly on the host.

And make sure the control-plane nodes have enough CPU, memory and disk space.

Networking problems

The symptoms: Services are unreachable, Pods can't talk to each other, outside access fails, or DNS lookups return errors.

Check the Service configuration:

kubectl get services
kubectl describe service <service-name>
kubectl get endpoints <service-name>

The selectors should match the Pod labels, and the endpoints should be healthy.

Test connectivity between Pods by running ping or curl against other Pods and Services through kubectl exec.

If you use network policies, check whether one is blocking the traffic:

kubectl describe networkpolicy

Read the kube-proxy logs on the worker nodes.

And test DNS resolution from inside a Pod:

kubectl exec -it <pod-name> -- nslookup <service-name>.<namespace>.svc.cluster.local

2. Audit logs: the forensic trail

Kubernetes audit logs are a chronological, security-relevant record of every request made to the Kubernetes API server. You use them for difficult troubleshooting, security investigations and compliance.

For security forensics, they tell you who did what, when, and from where, which is essential when you investigate a breach or unauthorized access. They help meet regulatory requirements for logging system activity. And for deep debugging, they show the exact sequence of API calls that led to a particular state or error.

How audit logging works

The kube-apiserver generates an audit event for every API request. It processes those events according to an audit policy, then writes them to a log file or sends them to a backend.

The audit policy is a YAML file that defines which events are recorded and at what level of detail. There are four levels:

• None logs nothing.
• Metadata logs the request's metadata: user, timestamp, resource and verb.
• Request logs the metadata and the request body.
• RequestResponse logs the metadata, the request body and the response body. It is the most verbose level.

Turn on audit logging

First, create an audit policy file, such as audit-policy.yaml:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata # Log every stage for secrets, without their values
    resources:
      - group: ""
        resources: ["secrets"]
  - level: Metadata # Log metadata for all other requests
    omitStages:
      - "RequestReceived"

Secrets are logged at Metadata. At Request or RequestResponse, the audit log would contain the secret values themselves.

Then configure the kube-apiserver. Edit its manifest, usually /etc/kubernetes/manifests/kube-apiserver.yaml, to add the audit flags and mount the policy file and the log directory:

# ... snippet from kube-apiserver manifest
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    # ... other kube-apiserver arguments
    volumeMounts:
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  volumes:
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

The kube-apiserver restarts and starts writing audit logs to /var/log/kubernetes/audit/audit.log.

Read the audit logs

Audit logs are JSON. Parse them with jq:

cat /var/log/kubernetes/audit/audit.log | jq . # Pretty-print all logs
cat /var/log/kubernetes/audit/audit.log | jq 'select(.user.username == "admin")' # Filter by user

Good practices for audit logs

Configure rotation with audit-log-maxage, audit-log-maxbackup and audit-log-maxsize. Send the logs to a SIEM or a centralized logging system for long-term retention and deeper analysis. Store them securely, with encryption and access control. And review your audit policies and logs regularly.

The end of the series

This is the last post in the series. It started with what Kubernetes is and has covered everything from the fundamentals to security and troubleshooting, enough to prepare for the CKA, CKAD and CKS exams and for running Kubernetes in real work.