All writing
Updated 4 min readNaren

Debugging, monitoring and logging

A systematic way to debug Pods, Services and Deployments, then the two things that make debugging possible, metrics with Prometheus and Grafana and centralized logs.

Kubernetes

The last post covered runtime security: threat detection with Falco and immutable containers. This one is about diagnosing and fixing problems when they happen, which they will. It covers three things:

  • debugging Pods, Services and Deployments systematically, to find and fix application and infrastructure problems
  • monitoring the health and performance of your cluster and applications
  • logging, to capture and analyze what happened

These are basic skills for anyone who uses Kubernetes, and the CKA and CKAD exams test them heavily.

1. Debugging Pods, Services and Deployments

When something goes wrong in Kubernetes, work through it systematically: start broad, then narrow down.

Pending Running Error/CrashLoopBackOff Start Check Pod Status (`kubectl get pods`) Check Events (`kubectl describe pod`) Check Logs (`kubectl logs`) Check Logs (`kubectl logs --previous`) Identify Scheduling/Resource Issues Identify Application Errors Identify Startup Errors Exec into Pod (`kubectl exec`) Debug Inside Container

A few principles help. Always check the status of your Kubernetes resources first. Know the architecture well, because you can't debug components you don't understand. Read the logs: the application's logs and Kubernetes' events are where the clues are. And keep resource limits in mind, because a container that goes over its CPU or memory limits can be terminated without warning.

Debugging Pods

Pods are the basic building blocks, and problems with them are often the root cause.

Start with the Pod's status:

kubectl get pods

Look for Pending, which points to a scheduling problem, CrashLoopBackOff, which means a container keeps crashing, Error or Evicted.

Then read the Pod's events. This is the command that most often holds the answer:

kubectl describe pod <pod-name>

Look for events about image pull failures, failed volume mounts or scheduling problems.

Read the Pod's logs, which show runtime errors, stack traces and problems specific to the application:

kubectl logs <pod-name>
kubectl logs <pod-name> -c <container-name> # For multi-container Pods
kubectl logs <pod-name> --previous # For logs from a previous instance of a crashing container

Run commands inside the container to check environment variables, inspect mounted volumes or run diagnostic scripts:

kubectl exec -it <pod-name> -- /bin/sh # Or /bin/bash

If the image has no shell, or the application crashes on startup, kubectl debug attaches a temporary debug container to the Pod:

kubectl debug -it <pod-name> --image=busybox --target=<container-name>

Check resource quotas and limits, to make sure the Pod is not going over its CPU or memory limits.

And check that the liveness and readiness probes are configured correctly. A failing probe can restart the Pod or send its traffic elsewhere.

Debugging Services

Services give Pods stable network endpoints, so problems with them usually show up as connectivity problems.

Check the Service's endpoints:

kubectl get endpoints <service-name>
kubectl describe service <service-name>

The Service should have healthy endpoints, meaning running Pods, and as many of them as you expect.

Check the Service's selector. Its selector must match the labels on the target Pods. If the labels don't match, no traffic is routed.

Test connectivity from inside the cluster. Start a temporary Pod, such as busybox, and reach the Service by its cluster IP and its DNS name:

kubectl run -it --rm --restart=Never busybox --image=busybox -- sh
# Inside busybox:
# wget -O- <service-name>
# nslookup <service-name>

Check for port mismatches. The targetPort in the Service definition must match the containerPort your application uses in the Pod.

Debugging Deployments

Deployments manage the desired state of your Pods.

Check the Deployment's status, looking for unavailable replicas or a failed rollout:

kubectl get deployments

Read its events, for scaling problems or failures to create Pods:

kubectl describe deployment <deployment-name>

Review the Pod template. A mistake in the template, such as the wrong image or bad resource limits, affects every Pod the Deployment manages.

If a new rollout causes problems, roll back to the previous version:

kubectl rollout undo deployment/<deployment-name>

2. Monitoring

Monitoring tracks the performance and health of your cluster's components and your applications through metrics. It shows you resource bottlenecks, performance getting worse, and failures on the way.

Kubernetes Cluster Monitoring Dashboard queries scrapes metrics from scrapes metrics from scrapes metrics from displays dashboards Prometheus kube-apiserver kubelet Application Pods Grafana User

The usual tools are Prometheus and Grafana. Prometheus is an open-source monitoring system that collects metrics and stores them as time series. It scrapes them from the targets you configure, such as Kubernetes components and application endpoints. Grafana is an open-source analytics and visualization platform, where you build dashboards over the metrics Prometheus collects.

The metrics worth watching:

  • resource use: CPU, memory and disk, for both Pods and nodes
  • Pod health: status, restart counts and response times
  • network traffic, in and out
  • the control plane: the health and performance of kube-apiserver, kube-scheduler, kube-controller-manager and etcd

At a high level, setting it up takes four steps:

  1. Install Prometheus, usually with a Helm chart.
  2. Install Grafana, also usually with Helm.
  3. In Grafana, add Prometheus as a data source.
  4. Import ready-made Kubernetes dashboards, or build your own.

3. Logging

Logging captures the events and information that applications and system components produce. Logs show what happened, when and why, which is what debugging and auditing depend on.

Kubernetes Nodes Centralized Logging System forwards logs to displays logs to collects logs from collects logs from queries User Log Agent (DaemonSet) Container stdout/stderr Node System Logs Log Aggregator (e.g., Elasticsearch, Loki) Log Visualizer (e.g., Kibana, Grafana)

Logs from every source, Pods, nodes and containers, should be aggregated in one central system, where they are easier to manage, search and analyze. That matters more in Kubernetes than elsewhere, because a Pod's logs disappear with the Pod, so they have to be shipped somewhere else. And logs are far easier to parse and query when they are structured: written in a consistent, machine-readable format such as JSON.

The EFK stack is Elasticsearch, Fluentd or Fluent Bit, and Kibana. Fluentd or Fluent Bit collects the logs, running as a DaemonSet on every node. Elasticsearch stores and indexes them. Kibana is the web interface for exploring and analyzing them.

The Loki stack is Loki, Promtail and Grafana. Loki aggregates logs and indexes only their metadata, which makes it efficient. Promtail is the agent that collects logs and sends them to Loki. Grafana displays the logs Loki has collected.

Good practices

Have applications write their logs to standard output and standard error. Kubernetes collects those streams automatically.

Run a logging agent such as Fluentd, Fluent Bit or Promtail as a DaemonSet to collect and forward the logs, and always aggregate them in a central system.

Use structured log formats, which make analysis easier.

Collect and analyze the logs of the Kubernetes control-plane components too.

The last post in the series goes further, into failures of the cluster itself: Troubleshooting cluster failures.