Runtime security: Falco, immutable containers and audit logs

The last post secured the path from code to deployment. This one is about protecting applications once they are running in production.

Runtime security means watching your cluster for suspicious activity, detecting threats, keeping running containers intact, and keeping a detailed record of every action. It is a key topic for the CKS exam.

1. Detecting threats with Falco

Prevention is never complete, and threats can still appear at runtime. Falco is an open-source, cloud-native runtime security tool that detects intrusions and abnormal behavior in Kubernetes clusters.

Falco watches the system calls that applications make into the Linux kernel. It checks them against a set of security rules, and when one matches, it alerts you.

System calls are the interface between an application and the Linux kernel, and Falco hooks into them to observe behavior. Rules define which events Falco looks for and what it does when it finds one. They are written in YAML, and you can customize them. When a rule triggers, Falco raises an alert, which it can send to several outputs, such as logs, Slack or Kafka.

Install Falco with Helm

Add the Falco Helm repository:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Install Falco:

helm install falco falcosecurity/falco --namespace falco --create-namespace --set tty=true

Check for the latest chart version first, and adjust the command if needed.

Check it and trigger an alert

Check that the Falco Pods are running:

kubectl get pods -n falco

Read Falco's logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco --tail=100

To trigger a rule, read a sensitive file inside a container. First create a sample Nginx Deployment:

kubectl create deployment nginx --image=nginx:latest

Then read /etc/shadow inside the Nginx Pod:

kubectl exec -it $(kubectl get pods --selector=app=nginx -o name) -- cat /etc/shadow

That should trigger a Falco alert. Look for it in the logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco | grep "Sensitive file opened"

You should see an alert saying a sensitive file was opened.

Good practices for Falco

Tailor Falco's default rules to your environment, to cut down false positives. Send Falco's alerts to your SIEM (security information and event management) system or your alerting channels, such as Slack or PagerDuty. And write custom rules for the critical files, directories and processes of your own applications.

2. Keeping containers immutable at runtime

Container images are built to be immutable, but a running container can still be changed. Runtime immutability means that once a container is deployed, its filesystem does not change. If something needs to change, you build and deploy a new image.

That improves security, because it blocks unauthorized changes, reduces the attack surface and limits what an attacker can modify after a compromise. It keeps environments consistent and predictable. And it makes problems easier to reproduce and diagnose.

readOnlyRootFilesystem

The most effective way to enforce runtime immutability is to set readOnlyRootFilesystem: true in the container's securityContext:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-immutable
spec:
  containers:
  - name: nginx
    image: nginx:latest
    securityContext:
      readOnlyRootFilesystem: true # Enforce read-only root filesystem
    ports:
    - containerPort: 80
    volumeMounts:
    - name: nginx-logs-volume
      mountPath: /var/log/nginx # Mount a writable volume for logs
    - name: nginx-cache-volume
      mountPath: /var/cache/nginx # Nginx writes temporary files here
    - name: nginx-run-volume
      mountPath: /var/run # Nginx writes its PID file here
  volumes:
  - name: nginx-logs-volume
    emptyDir: {} # Ephemeral volume for logs
  - name: nginx-cache-volume
    emptyDir: {}
  - name: nginx-run-volume
    emptyDir: {}

readOnlyRootFilesystem: true makes the container's root filesystem read-only. Any directory the application must write to gets a writable volume, such as an emptyDir, mounted at that path. Nginx needs three: /var/log/nginx for its logs, plus /var/cache/nginx and /var/run, without which it does not start.

Test it:

kubectl exec -it nginx-immutable -- touch /testfile # This should fail
kubectl exec -it nginx-immutable -- touch /var/log/nginx/testlog # This should succeed

3. Monitoring access with audit logs

Kubernetes audit logs are a chronological record of every request made to the Kubernetes API server. You need them for security, compliance and troubleshooting.

The kube-apiserver generates audit events for every API request, and follows each request through stages such as RequestReceived, ResponseStarted and ResponseComplete.

The audit policy

An audit policy is a YAML file that says which events to log, and at what level of detail: None, Metadata, Request or RequestResponse.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Don't log requests for events
  - level: None
    resources:
      - group: ""
        resources: ["events"]

  # Log metadata for all pod-related actions
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]

  # Log request and response bodies for configmaps
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["configmaps"]

  # Log only metadata for secrets: a higher level would copy their values into the log
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]

  # Default rule: Log metadata for all other requests
  - level: Metadata
    omitStages:
      - "RequestReceived"

Secrets stay at Metadata. At Request or RequestResponse, the audit log would hold the secret values themselves.

Turn on audit logging

Audit logging is configured with flags in the kube-apiserver manifest, usually /etc/kubernetes/manifests/kube-apiserver.yaml:

# ... snippet from kube-apiserver manifest
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    # ... other kube-apiserver arguments
    volumeMounts:
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  volumes:
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

The audit policy file and the log directory both have to be mounted into the kube-apiserver Pod.

Read the audit logs

Audit logs are usually JSON files. Read them with cat and parse them with jq:

cat /var/log/kubernetes/audit/audit.log | jq .
cat /var/log/kubernetes/audit/audit.log | jq 'select(.user.username == "your-username")'

Good practices for audit logs

Configure log rotation to control file size and retention. Send audit logs to a centralized logging system, such as the ELK Stack or Splunk, for analysis and long-term storage. And review your audit policies and logs regularly.

That is the end of the security posts. The next post turns to troubleshooting, starting with the common ways to diagnose problems in a cluster: Debugging, monitoring and logging.