Liveness, readiness and startup probes

The last post covered managing configuration with ConfigMaps and sensitive data with Secrets. This one is about making sure applications are not just running, but running correctly and ready to serve traffic.

Probes let Kubernetes monitor the health of your containers. It restarts unhealthy ones automatically and sends traffic only to instances that are ready. Getting them right is a large part of building applications that stay available.

Why Kubernetes needs health checks

Kubernetes manages the lifecycle of your Pods, but it cannot see the internal state of your application. A container can be running while the application inside it is frozen, stuck in a loop or unable to reach its database. Without health checks, Kubernetes has no way to know, and it keeps sending traffic to a broken application.

Kubernetes has three types of probe:

1. A liveness probe checks whether a container is running and healthy. If it fails, Kubernetes restarts the container.
2. A readiness probe checks whether a container is ready to serve traffic. If it fails, Kubernetes stops sending traffic to the Pod.
3. A startup probe checks whether a container has started successfully. It is for applications that start slowly.

How probes work

Each probe defines a check that Kubernetes runs against your container at regular intervals. If the check fails, Kubernetes acts according to the type of probe.

A probe can run its check in three ways:

• exec runs a command inside the container. The probe succeeds if the command exits with status 0.
• httpGet sends an HTTP GET request to a given IP address, port and path. The probe succeeds if the response code is between 200 and 399.
• tcpSocket tries to open a TCP connection on a given port. The probe succeeds if the connection is established.

Probe parameters

All probes share the same parameters:

• initialDelaySeconds is how many seconds to wait after the container starts before the first probe.
• periodSeconds is how often, in seconds, to run the probe.
• timeoutSeconds is how many seconds to wait before the probe times out.
• successThreshold is the minimum number of consecutive successes, after a failure, for the probe to count as successful.
• failureThreshold is the minimum number of consecutive failures, after a success, for the probe to count as failed.

Liveness probes

A liveness probe tells Kubernetes when to restart a container. If it fails, Kubernetes assumes your application is broken and restarts the container. That matters for applications that can get stuck without crashing.

Use one when your application can end up running but unable to make progress, as with a deadlock, an infinite loop or an exhausted database connection pool.

Here is a liveness probe that makes an HTTP request:

apiVersion: v1
kind: Pod
metadata:
  name: liveness-http-pod
spec:
  containers:
  - name: liveness-http
    image: nginx:latest
    ports:
    - containerPort: 80
    livenessProbe:
      httpGet:
        path: /healthz # Endpoint that returns 200 OK if healthy
        port: 80
      initialDelaySeconds: 5 # Wait 5 seconds before first check
      periodSeconds: 5       # Check every 5 seconds
      timeoutSeconds: 2      # Timeout after 2 seconds
      failureThreshold: 3    # 3 consecutive failures to be considered unhealthy

And one that runs a command:

apiVersion: v1
kind: Pod
metadata:
  name: liveness-exec-pod
spec:
  containers:
  - name: liveness-exec
    image: busybox:1.36
    command: ["/bin/sh", "-c", "touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600"]
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy # Check if /tmp/healthy file exists
      initialDelaySeconds: 5
      periodSeconds: 5
      failureThreshold: 1 # 1 failure to be considered unhealthy

This container runs for 30 seconds and then deletes /tmp/healthy. The liveness probe starts failing, and Kubernetes restarts the container.

Readiness probes

A readiness probe tells Kubernetes when a container is ready to accept traffic. If it fails, Kubernetes removes the Pod's IP address from the Service's endpoints, so no traffic is routed to that Pod. When the probe succeeds again, the Pod is added back.

Use one for applications that need time after starting, to load data or connect to a database, before they can serve requests. They also help with graceful shutdowns.

apiVersion: v1
kind: Pod
metadata:
  name: readiness-http-pod
spec:
  containers:
  - name: readiness-http
    image: nginx:latest
    ports:
    - containerPort: 80
    readinessProbe:
      httpGet:
        path: /ready # Endpoint that returns 200 OK when ready
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 1

The difference between the two fits in a line each. Liveness: if I'm not alive, restart me. Readiness: if I'm not ready, don't send me traffic. Applications you rely on usually have both.

Startup probes

A startup probe tells Kubernetes that the application in a container has started. While it is configured and has not yet succeeded, the liveness and readiness checks are disabled. That stops an application with a long startup from being killed early by its liveness probe, or from getting traffic through its readiness probe before it can handle it.

Use one for applications whose startup time is long or unpredictable.

apiVersion: v1
kind: Pod
metadata:
  name: startup-probe-pod
spec:
  containers:
  - name: startup-probe
    image: busybox:1.36
    command: ["/bin/sh", "-c", "echo 'Starting up...'; sleep 30; touch /tmp/ready; echo 'Ready!'; sleep 3600"]
    startupProbe:
      exec:
        command:
        - cat
        - /tmp/ready # Created by the container after 30 seconds
      initialDelaySeconds: 0 # Start checking immediately
      periodSeconds: 5       # Check every 5 seconds
      failureThreshold: 12   # Allow up to 12 * 5 = 60 seconds for startup

This container creates /tmp/ready after 30 seconds. The startup probe checks for the file every 5 seconds and allows up to 60 seconds in total. Until it succeeds, the liveness and readiness probes are paused. Once it succeeds, they begin.

Good practices

Use each probe where it fits: a startup probe for applications that start slowly, a liveness probe to catch deadlocks and restart the container, and a readiness probe so traffic only goes to Pods that can handle it.

Keep probes lightweight. Health check endpoints should be fast. Heavy database queries or complex logic in a health check add overhead and cause false failures.

Give liveness and readiness separate endpoints, such as /healthz for liveness and /ready for readiness. The liveness check might only confirm the process is running, while the readiness check might also confirm the database is reachable.

Tune initialDelaySeconds, periodSeconds, timeoutSeconds and failureThreshold to your application's startup time and normal response times.

Plan for graceful shutdowns. When a Pod is terminated, Kubernetes sends it a SIGTERM signal. Your application should catch it and shut down cleanly, letting active requests finish before it exits. A readiness probe that fails during shutdown helps, because it removes the Pod from the Service's endpoints.

Keep external dependencies out of liveness probes. A liveness probe should check the application's own health, not a dependency such as a database. If the database fails, restarting the application will not fix it and may make things worse. Signal that problem through the readiness probe instead.

The next post covers Pods that run more than one container, with patterns like sidecars and init containers: Multi-container Pods.