Supply chain security for Kubernetes

The last post secured individual workloads with Pod Security Standards, AppArmor, seccomp and runtime sandboxes. This one widens the view to everything an application passes through on its way from code to deployment.

Supply chain security means securing every step of your software's lifecycle, from the moment the code is written to the moment it runs in production. The aim is to stop attackers from injecting vulnerabilities or tampering with your applications along the way. It is a key topic for the CKS exam.

The Kubernetes supply chain

The software supply chain is every service, tool and contributor involved in delivering your software. In Kubernetes, that covers:

• your application's source code and its dependencies
• the CI/CD pipelines that build your container images
• the container images themselves, base images and application layers alike
• the registries that store those images
• deployment tools such as Helm, Kustomize and GitOps platforms
• the Kubernetes cluster your applications run in

A vulnerability or a compromise at any point in that chain can do serious damage, so the defense has to cover all of it.

1. Image security

Container images are the foundation of your applications, so securing them comes first.

Keep base images small. A smaller image has fewer packages, which means a smaller attack surface, fewer potential vulnerabilities, and faster builds and deployments. Use a minimal base such as Alpine Linux, Google's Distroless images, or scratch.

Allow only approved registries, so nobody pulls images from untrusted or unknown public registries that might contain malware. Enforce the list across the cluster with an admission controller such as OPA Gatekeeper or Kyverno.

Sign your images and verify the signatures. Signing proves an image has not been tampered with since it was built, and that it comes from a trusted source. Cosign, part of the Sigstore project, signs images during the build and verifies the signatures before deployment:

# Example: Signing an image with Cosign
cosign sign --key cosign.key myregistry/my-app:latest

# Example: Verifying an image with Cosign
cosign verify --key cosign.pub myregistry/my-app:latest

Use trusted sources. Take base images from well-maintained sources, such as the official images on Docker Hub or images from reputable vendors, and make sure private registries use strong authentication and encryption.

2. Static analysis and vulnerability scanning

Build security checks into your development process and your CI/CD pipeline.

Analyze your workloads statically, to find security problems in application code and Kubernetes YAML before deployment. Use SAST (static application security testing) tools for the code. For Kubernetes manifests, KubeLinter, Polaris and OPA check for misconfigurations such as privileged containers and missing resource limits.

Scan images for known vulnerabilities (CVEs) in their operating system packages and application dependencies. Image scanners such as Trivy, Clair and Grype fit into a CI/CD pipeline:

# Example: Scanning an image with Trivy
trivy image myregistry/my-app:latest

Shift security left. The earlier you find a vulnerability, the cheaper and easier it is to fix, so scan at every stage of the pipeline, from the commit to the image build.

3. Securing the CI/CD pipeline

The CI/CD pipeline is the bridge between your code and your production cluster, which makes it a prime target for attackers.

Restrict access to your build agents and runners. Sign and verify every artifact the pipeline produces, so you can trust its integrity. Give your CI/CD tools and service accounts only the permissions they need. Keep detailed logs and watch the pipelines for anything unusual. And run all the image and manifest scans as part of the pipeline itself.

4. Secrets management

Exposed secrets are a common cause of security breaches.

Kubernetes Secrets offer some protection. For high-security environments, consider an external secret manager such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault or Google Cloud Secret Manager. They add centralized management, auditing and strong encryption.

Never embed sensitive data in your code, configuration files or Git repositories. Rotate secrets regularly, to shorten the time a leaked secret stays useful. And only mount the secrets a deployment actually needs.

5. Runtime security

Even with a secure build process, new vulnerabilities appear, and zero-day exploits happen.

Enforce network policies to control traffic between Pods and to outside services, as covered in the post on network policies.

Use runtime security tools, such as Falco for monitoring behavior, Sysdig Secure or Aqua Security, to detect and respond to suspicious activity in your running containers and your cluster.

Monitor and audit your running applications and your cluster continuously, looking for anomalies.

And use admission controllers, such as OPA Gatekeeper or Kyverno, to enforce policies and validate resources before they are deployed to the cluster.

6. Security frameworks

SLSA (Supply-chain Levels for Software Artifacts) is a framework of good practices for securing the software supply chain.

Zero trust is the principle of never trusting any user, device or application by default, whether it is inside or outside your network. Always verify.

The last of the security posts goes further into watching a running cluster: monitoring, logging and runtime security. Runtime security.