Moving Kubernetes workloads across an air gap with Hauler

An air gap is a network with no physical or routed path to the internet. There is no docker pull, no helm repo add, no curl, and no proxy setting that fixes it.

Software gets in two ways. Somebody carries it through a door on hardware, a practice called sneakernet because the transport protocol is your shoes.

Or it comes through a data diode, a device that passes traffic one way and physically cannot carry a reply.

Hauler is a command-line binary from Rancher Government Solutions, Apache-2.0 licensed, that packages container images, Helm charts and loose files into one archive and serves them back from the far side of the gap.

A Helm chart is the package format Kubernetes applications ship in. This post carries a small platform across with Hauler v2.1.0, released on 31 August 2026: a few images, one chart, and the k3s binary and install script that bring the cluster up. k3s is a Kubernetes distribution small enough to ship as a single binary.

Hauler's repository moved to github.com/hauler-dev/hauler, so a bookmark pointing at rancherfederal is stale.

Hauler can carry a Helm chart, a shell script and a k3s binary the same way it carries an image, because all of them are OCI artifacts.

OCI is the Open Container Initiative, the standards body whose specifications define what a container image is and how registries pass them around. Registries are the servers that store images and hand them out.

An OCI artifact is any blob stored and addressed the way an image is, which is why a chart or a plain file qualifies.

One content-addressable store holds the lot: every blob is filed under a hash of itself, so a layer that two images share is stored once.

In the diagram, look at the middle: a single dashed edge, with the same content store on either side of it.

Install it on both machines

The air-gapped host needs the binary too, and it cannot fetch one later. Do the target machine before you cross, not after.

# macOS: the tap is required. There is no hauler formula in homebrew-core.
brew tap hauler-dev/homebrew-tap
brew install hauler

# Linux or macOS: the shell installer, which drops the binary in /usr/local/bin
curl -sfL https://get.hauler.dev | bash

hauler version

get.hauler.dev redirects to install.sh in the repository, covers linux and darwin on amd64 and arm64, and asks for sudo only when the install directory is not writable. hauler version should report v2.1.0 or newer.

Declare the bundle in one file

Be pedantic in this manifest, because it is also the record of what entered the enclave.

apiVersion: content.hauler.cattle.io/v1
kind: Images
metadata:
  name: platform-images
spec:
  images:
    # Pinned by tag, with an explicit platform so a Mac build host does not
    # quietly package arm64 for an x86_64 target.
    - name: docker.io/rancher/k3s:v1.36.4-k3s1
      platform: linux/amd64
    - name: docker.io/longhornio/longhorn-ui:v1.12.1
      platform: linux/amd64
    # Pinned by digest, which is immutable and cannot be repointed later.
    - name: gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5
    - name: ghcr.io/OWNER/REPO:TAG
      platform: linux/amd64
---
apiVersion: content.hauler.cattle.io/v1
kind: Charts
metadata:
  name: platform-charts
spec:
  charts:
    - name: rancher
      repoURL: https://releases.rancher.com/server-charts/stable
      version: 2.15.1
      # Walk the chart and pull the images it references, so you stop
      # hand-listing them above.
      add-images: true
      platform: linux/amd64
---
apiVersion: content.hauler.cattle.io/v1
kind: Files
metadata:
  name: bootstrap-files
spec:
  files:
    - path: https://get.k3s.io
      name: k3s-install.sh
    # %2B is a URL-encoded + from the release tag v1.36.4+k3s1. Do not
    # "fix" it to a literal +, which 404s.
    - path: https://github.com/k3s-io/k3s/releases/download/v1.36.4%2Bk3s1/k3s
      name: k3s-binary
    - path: https://github.com/k3s-io/k3s/releases/download/v1.36.4%2Bk3s1/sha256sum-amd64.txt
      name: k3s-sha256sums.txt

Three documents in one file, separated by ---. content.hauler.cattle.io/v1 is the current API; Hauler deprecated v1alpha1 in v1.2.0, so any tutorial still showing it is old. Substitute your own application image for ghcr.io/OWNER/REPO:TAG.

Two keys are doing more work than they look. platform: linux/amd64 is a per-image field in the manifest. amd64 and arm64 are different CPU architectures, and an image built for one will not run on the other.

Leave the field out on an Apple Silicon build host and it will happily package the wrong half of a multi-arch image.

The same thing exists on the command line as --platform on hauler store sync, store save and store add image, and since v2.1.0 the CLI flag overrides what the manifest says.

add-images: true makes Hauler render the chart and pull the images it references, which is how you stop maintaining a hand-written list that drifts.

For a chart whose image list depends on your values, --values values.yaml on hauler store add chart renders it with those values first.

The k3s version appears three ways, which is worth knowing before you retype it.

The release is v1.36.4+k3s1; + is illegal in an OCI tag, so the image is v1.36.4-k3s1; + is unsafe in a URL path, so the download link carries %2B.

Pull it, look at it, package it

Log in to Docker Hub first, on the connected side.

Docker's published pull limits cap anonymous pulls at 100 per six hours per IPv4 address or IPv6 /64 subnet, and 200 for a free authenticated account; a real platform manifest goes through 100 without trying.

Over the limit you get an HTTP 429 that reads like a network fault rather than a quota.

echo "$DOCKERHUB_TOKEN" | hauler login docker.io --username DOCKERHUB_USER --password-stdin

hauler store sync --filename hauler-manifest.yaml
hauler store info

sync creates a directory called store in the working directory (-s/--store to move it) and keeps its own config under $HOME/.hauler. store info prints a table of reference, type, platform, layer count and size.

Read the platform column now. This is the last cheap moment to catch an accidental linux/arm64.

Then package it. The output is a tar archive compressed with Zstandard, which gets gzip-class ratios at considerably higher speed, and it holds shared layers once rather than once per image.

hauler store save --filename haul.tar.zst

sha256sum haul.tar.zst | tee haul.tar.zst.sha256     # macOS: shasum -a 256

Transfer media is real hardware.

FAT32 caps a single file at 4 GB, and plenty of accredited transfer processes still run on optical discs, so save in chunks instead when the haul will not fit on the media you have.

hauler store save --chunk-size 2G    # haul.tar.zst.000, .001, and so on; load rejoins them

Checksum whatever actually travels: haul.tar.zst.* if you chunked, not the single file above. The far side verifies what it received, and load rejoins the chunks afterwards.

Hauler needs free space in /tmp at least the size of the store while it works, so point -t/--tempdir somewhere with room if /tmp is small.

Load it on the far side

sha256sum -c haul.tar.zst.sha256          # macOS: shasum -a 256 -c
hauler store load --filename haul.tar.zst
hauler store info --check

load takes --filename, never a positional argument. hauler store load haul.tar.zst fails with Error: accepts 0 arg(s), received 1, which is a miserable thing to find out after walking a drive through two doors.

Bare hauler store load picks up ./haul.tar.zst because that is the flag's default, and repeating -f merges several hauls into one store.

--check is the step that matters, and it is new in v2.1.0.

Plain store info only lists what is there; --check walks every artifact in the store and confirms that each blob it points at exists and matches its recorded size and digest, which is what catches a copy that truncated halfway.

Day 1: serve it straight from the binary

Day 1 is initial bring-up, when nothing exists yet. Day 2 is steady-state operation of something already running. They want different tools, which is why Hauler ships both a server and a copy command.

On Day 1 there is no registry inside the enclave, so Hauler is one.

hauler store serve registry &      # OCI registry on port 5000, read-only by default
hauler store serve fileserver &    # HTTP fileserver on port 8080

curl -sfL http://localhost:5000/v2/_catalog
curl -O http://localhost:8080/k3s-install.sh
curl -O http://localhost:8080/k3s-binary
curl -O http://localhost:8080/k3s-sha256sums.txt

grep ' k3s$' k3s-sha256sums.txt | sed 's/k3s$/k3s-binary/' | sha256sum -c
chmod +x k3s-install.sh k3s-binary

/v2/_catalog is the standard OCI distribution catalog endpoint and should list your repositories.

The grep and sed line rewrites the upstream checksum file to match the name Hauler stored the binary under, so k3s gets checked against the sums that shipped with its release.

Now point the node at the registry by telling containerd, the runtime Kubernetes talks to when it pulls and starts a container, that docker.io lives on localhost.

mirrors:
  docker.io:
    endpoint:
      - "http://localhost:5000"
  ghcr.io:
    endpoint:
      - "http://localhost:5000"

Write http:// explicitly. Without a scheme k3s assumes https and the pull fails against a plain-HTTP registry. k3s reads this file at startup only, so restart it on every node after a change.

Both servers take --tls-cert and --tls-key, which you want for anything outliving a bring-up shell.

And & backgrounds a process only for the life of that shell, so a node that must survive a reboot needs systemd units instead.

Day 2: move it into a real registry

Harbor is the usual answer inside a secure network: an open-source self-hosted registry with accounts, projects, scanning and replication. Once the enclave has one, push the whole store into it.

# Log in to the BARE HOST. A path here breaks the copy.
echo "$REGISTRY_PASSWORD" | hauler login REGISTRY.INTERNAL.EXAMPLE --username admin --password-stdin

hauler store copy registry://REGISTRY.INTERNAL.EXAMPLE/PROJECT

hauler login REGISTRY.INTERNAL.EXAMPLE/production appears to work and then breaks the copy, which Hauler's known-limits page spells out. Log in to the host, put the path on copy.

Destinations accept registry://, reg:// and oci:// for a registry, or directory:// and dir:// to extract files and charts onto local disk; add --insecure or --plain-http for a registry without a trusted certificate.

Do not reach for hauler store copy -u/-p, which is deprecated in favour of hauler login.

Then install the chart from the copy you carried across, rather than from the internet. Rancher here is the cluster-management platform this example happens to carry; substitute your own chart and its values.

hauler store copy dir://./extracted
find ./extracted -name 'rancher-2.15.1.tgz'    # where the extraction actually put it

helm install rancher PATH/TO/rancher-2.15.1.tgz \
  --namespace cattle-system --create-namespace \
  --set hostname=RANCHER.INTERNAL.EXAMPLE \
  --set image.registry=REGISTRY.INTERNAL.EXAMPLE \
  --set systemDefaultRegistry=REGISTRY.INTERNAL.EXAMPLE \
  --set useBundledSystemChart=true

Use the path find prints in place of PATH/TO.

What will actually go wrong

The Helm values, which fail late and quietly

The Helm values are the expensive mistake, because they fail late and quietly. Charts hard-code public registry paths, so one installed unchanged inside an enclave reaches for docker.io and sits there.

The symptom is pods stuck in ImagePullBackOff, which is the first thing you notice in a k9s pod list.

Every chart names its override differently.

Rancher's air-gapped install documentation gives three for this one: image.registry for the Rancher server image, systemDefaultRegistry for everything Rancher pulls for the clusters it provisions, and useBundledSystemChart=true to stop it fetching system charts over the network.

There is no global.imageRegistry in this chart, and Helm accepts an unknown --set key without complaint.

Hauler's own answer is --rewrite, which stores an artifact under a different reference than the one it was pulled from, turning quay.io/prometheus/prometheus:v2.45.0 into internal.registry.io/production/monitoring/prometheus:v2.45.0.

Making the stored path match what the chart expects beats arguing with values files.

The embedded registry is for bring-up only

The embedded registry is CNCF Distribution v3.1.1, the reference OCI registry implementation, compiled into the binary and backed by a directory on local disk. Its whole flag list is --port, --directory, --config, --readonly and the TLS pair: no authentication, no role-based access control, no replication, no clustering.

Run it for the bring-up, and move to Harbor once there is a cluster with persistent storage able to host it.

The images you forget are sidecars

The dependency you forget is a sidecar, an extra container that runs beside the main one in the same pod.

Charts pull in plenty of them: init containers that run before the app starts, metrics exporters, and the Container Storage Interface drivers that let Kubernetes talk to storage systems.

Each of those images has to be in the manifest, and missing one costs a whole transfer cycle.

add-images: true covers what a chart references; whatever an operator (a controller that creates cluster resources on its own) adds at runtime is on you.

Verify on the connected side, and carry the proof

Signature verification belongs on the connected side. Cosign is the Sigstore tool that signs container images and checks those signatures.

Hauler wires it in and verifies as it pulls, with --key cosign.pub on sync or the hauler.dev/key annotation in the manifest.

Since v2.1.0 a failed verification aborts the run instead of skipping the artifact silently.

Keyless verification cannot work inside the enclave. It has to reach Fulcio and Rekor, Sigstore's certificate authority and its public transparency log, and reaching anything on the internet is what an air gap prevents; Hauler's manifest documentation marks the transparency-log annotation as incompatible with airgap.

Verify where you still have the internet, then carry a store you have already proven. Signing and verifying an image with Cosign is the prerequisite if you have not set it up.

For the same reason, build the haul on the connected network and carry across its checksum and an SBOM, a machine-readable inventory of what is inside a build, which Syft produces in one command.

Inside the enclave that file is what you match a carried-in vulnerability database against when someone asks whether a newly published CVE affects you.

A CVE is the public identifier for one specific known vulnerability, and nothing in there can go and look one up.