Building an AI hyperscaler on Kubernetes

This post sketches how you would build an AI cloud on Kubernetes, at the scale of CoreWeave, Lambda or Google's AI Hypercomputer. It assumes you already know Kubernetes (Pods, controllers, operators, CNI and CSI) and the basics of GPU infrastructure (CUDA, NCCL, distributed training and model serving). It spends its time on the gap between running GPU workloads on Kubernetes and running an entire AI cloud on it.

The post on Kubernetes-as-a-Service covers hosted control planes and the bare-metal plumbing underneath them. This one starts a level higher, and borrows three ideas from how AWS and Google Cloud built their own clouds.

What an AI cloud sells

The companies in this market fall into three groups:

Every one of them runs Kubernetes, or something shaped like it, at the bottom. They differ in hardware density and supply, in the quality of the network fabric, in how they schedule and separate tenants, in where the API sits (raw nodes or managed endpoints), and in the operational glue of identity, billing, quota and support.

Strip away the marketing and a customer pays for some combination of six things:

1. compute time on a specific accelerator, such as an H100, H200, B200, MI300X or TPU v5p
2. network locality: sitting in the same NVLink domain, rail or fabric as the rest of the job
3. storage close to the GPUs, so datasets and checkpoints move at line rate
4. time to first token for an inference call, or time to first step for a training job
5. a guarantee: SLOs, support, reserved capacity and predictable pricing
6. an abstraction: they hand you tokens or prompts, not nodes

The further down that list you sell, the higher your margin and the harder the platform engineering. The further up, the more you compete on the price of a GPU-hour and on supply. Most successful AI clouds sell two to four products on one shared substrate.

Seven layers

Each layer has its own failure domains, its own lifecycle and its own owning team.

L0: the datacenter

An 8-GPU H100 node draws about 10.2 kW and a B200 node about 14.3 kW, so a rack of 32 GPUs needs 40 to 80 kW, where most colocation sites cap a rack at 15 to 25 kW. Air cooling runs out at about 35 kW per rack, and past that you need rear-door heat exchangers or direct-to-chip liquid cooling. NVIDIA's Blackwell NVL72 racks are liquid cooled only.

GPUs, switches and cooling parts have lead times of 6 to 18 months, which makes capacity planning a procurement problem before it is an engineering one. At small scale you rent this layer. Owning it is what turns a neocloud into a hyperscaler.

L1: the network fabric

AI clouds run two networks. The frontend network, at 25 to 100 GbE, carries the Kubernetes control plane, image pulls, observability and customer traffic. The backend network gives every GPU 400 or 800 Gb/s for collective operations, such as NCCL's all-reduce, and it is where an AI cloud makes or loses money.

The backend can be InfiniBand, which is lossless by design, mature, expensive and tied to one vendor. It can be RoCE v2 over Ethernet, made lossless with PFC, ECN and DCQCN, which is cheaper and more flexible, but takes real tuning and can wedge under bursty traffic. Or it can be Ultra Ethernet (UEC), the industry's answer to both, with production hardware arriving in 2026. A new build should plan around it.

Most AI fabrics use a rail-optimized Clos topology. Each GPU has its own NIC, and the NICs with the same rail index on every node share a leaf switch, which gives full bandwidth within a rail and predictable NCCL performance. Inside the node, NVLink and NVSwitch connect GPUs at 900 GB/s or more, across a whole NVL72 rack on Blackwell. Treat GPUs joined by NVLink as a single scheduling domain.

L2: hardware

You will run at least three classes of node. Control nodes run etcd, the API servers, schedulers and controllers on dedicated machines that never run workloads. GPU nodes typically carry 8 GPUs, 1 to 8 NICs, 8 to 30 TB of local NVMe and 1 to 2 TB of RAM. Storage and utility nodes have plenty of NVMe and NICs but no GPUs, and run the shared filesystem, registry caches and monitoring.

Firmware drifts across a fleet of thousands of GPU nodes, so standardize it through Redfish, and remember that NIC, GPU and NVSwitch firmware are three separate things to track. The BMC network, IPMI and Redfish, is usually your least defended attack surface.

L3: Kubernetes clusters

The first decision is one huge cluster or many, which depends on scale and comes up again below. Most neoclouds end up on Cluster API with Talos, or on vanilla Kubernetes with their own image pipeline.

etcd sets the ceiling. Past about 5,000 nodes, 150,000 Pods or heavy watch fan-out, you have to move events to a separate etcd, move to a database-backed alternative, or split product lines into separate clusters. The API server needs tuning too, through its in-flight request limits, API Priority and Fairness, the watch cache and kubelet QPS limits.

containerd is the default runtime, and the NVIDIA Container Toolkit is required to expose GPUs as nvidia.com/gpu. GPUs have long reached Pods through device plugins, but Dynamic Resource Allocation (DRA), generally available since Kubernetes 1.34, is replacing them and is the API to build on.

L4: the platform on Kubernetes

This is everything vanilla Kubernetes doesn't ship:

• Scheduling: Volcano, YuniKorn, Kueue, Koordinator or KAI Scheduler, with topology awareness, gang scheduling (all or nothing for distributed training), and preemption tiers that put paying customers ahead of spot jobs.
• Workload operators: Kubeflow Trainer, whose TrainJob replaced the older Training Operator's PyTorchJob and MPIJob, KubeRay, Slurm on Kubernetes (Soperator from Nebius, Slinky from SchedMD), JobSet and LeaderWorkerSet.
• Inference: KServe, Seldon, BentoML, Ray Serve, the vLLM Production Stack or llm-d.
• Storage: CSI drivers for Lustre, WEKA, Vast, JuiceFS and Rook with Ceph, and dataset caches such as Alluxio and Fluid.
• Networking: Cilium, the consensus choice at scale, with Multus to give Pods a second NIC on the backend fabric, through SR-IOV or the RDMA shared device plugin.
• GitOps: Argo CD or Flux. Pick one and stop debating.

L5 and L6: the products and the control plane customers see

L5 is deliberately thin. Each product translates a customer's idea, "a training job" or "a model endpoint", into the platform primitives below it.

L6 is not Kubernetes at all, and it is the part engineers most often underinvest in: the public API gateway, identity federated with customers' identity providers, a tenant model of accounts and projects, the quota service, billing, the console and support tooling. From the second scale tier on, it often outweighs the Kubernetes platform in headcount and complexity.

Four products on one GPU pool

Training: a GPU supercomputer as a service

The product is a reservation of N GPUs, gang scheduled, with the backend RDMA fabric and fast shared storage. Reserved capacity, held for days or weeks and paid for whether it is used or not, is the margin product. On-demand jobs on pooled capacity, billed per GPU-hour, earn less and are harder to schedule.

Scheduling makes or breaks it. All N Pods must land or none should, or GPUs sit idle, and they are grouped by NVLink domain, then rail, then leaf, then spine. Preemption follows a PriorityClass ladder, from paid reserved down to opportunistic, which Volcano and Kueue both support. And a preempted job gets a grace window, through terminationGracePeriodSeconds, to write a checkpoint before it is killed.

Run NCCL bandwidth tests as a standing canary on every cluster, aiming for all-reduce at 90% or more of line rate. Datasets are read-mostly and cached on local NVMe in front of object storage. Checkpoints are large bursts of writes that must reach a shared filesystem at line rate.

What goes wrong: NCCL flaps under PFC storms on RoCE, so tune the ECN thresholds. GPU XID errors need an operator that drains the node and opens a hardware ticket. One slow GPU stalls the whole job, so watch each rank's step time and fail the straggler early. And a 30 GB CUDA and PyTorch image pulled by 1,024 Pods at once will melt your registry, so distribute images peer to peer with Dragonfly, Kraken or Spegel, or pre-stage them with a DaemonSet.

Inference: managed model endpoints

The product is a URL that returns model output when you POST to it, billed per token, per request or per GPU-hour. It comes as serverless endpoints that scale to zero, dedicated endpoints on reserved GPUs, or deployments of the customer's own model. KServe or Ray Serve provide the abstraction, an engine such as vLLM, TensorRT-LLM, SGLang or NVIDIA NIM does the serving, and HPA or KEDA scale on queue depth and KV-cache pressure. In front, Gateway API routes the traffic, and LiteLLM can put many models behind one OpenAI-compatible API.

Loading a 70B model takes 30 to 120 seconds, so you cannot cold-boot one per request. Keep a hot pool of Pods for each model and hardware combination, scaled on actual load, and mount the weights read-only from a shared filesystem. MIG partitions on the H100, H200 and B200 let you sell less than a whole GPU, though whether a partition is safe between untrusted tenants is a separate question, covered below. Separate prefill Pods from decode Pods, as vLLM, Mooncake, NVIDIA Dynamo and llm-d do, so bursts of prefill don't starve decoding. And send requests that share a prefix to the same Pod, so its KV cache gets reused.

Tenants running the same base model share GPU nodes, held apart by rate limits and token quotas at the gateway. A customer's own model gets a Pod to itself, or a node when security demands it. Typical targets for chat-class models are a time to first token under 200 ms at p50 and 500 ms at p95.

What goes wrong: scaling from zero breaks that target, so keep a warm floor for every popular model. One 32k-token request can starve everyone queued behind it, so use continuous batching, priority queues and per-request timeouts. MIG isolates compute but not memory bandwidth perfectly, so say so in the product's definition. And roll out model versions blue-green, treating each version as an OCI artifact rather than a random S3 path.

Batch and data

The product is elastic compute for work that doesn't need the fast fabric: embeddings, evaluations, fine-tuning, ETL and RLHF rollouts, billed per resource-second. Spot is the default, because this is where capacity evicted from the training pool gets sold, cheaper in exchange for preemption. KubeRay, Spark or Argo Workflows run the jobs, and Kueue's ClusterQueue and LocalQueue hold the quota.

What goes wrong: long-running Ray clusters that nobody owns pile up, so give them an automatic time to live. Ray and Spark head Pods are single points of failure by default, so pay for the operators' high-availability modes. And a million workers reading the same object-store prefix will throttle themselves, so shard the prefixes.

Agents

The customer defines an agent, with its tools, memory, vector store and model, and you run it, billed per invocation, per token or per month. It is built from CRDs such as Agent, Tool and KnowledgeBase, a runtime that scales to zero, and an operator-managed vector database such as Qdrant, Milvus or pgvector. Model calls go through the inference product, so agents are a customer of inference.

Tool calls that run customer code need hard isolation. Never run customer code in the same kernel as your control plane: gVisor at the minimum, Firecracker for preference. Sandbox escape is the largest security risk in the product, so pen-test it. Set hard budgets for tokens, cost and wall-clock time on every invocation, because agents do run away. And keep step traces as OpenTelemetry spans, because customers want to debug what their agent did.

One pool, four products

The four products share one substrate but have separate control planes. That is the central design pattern: one capacity pool, four product surfaces.

Sharing the pool between tenants

GPUs are the most expensive thing an AI cloud buys, and they only earn their cost while they are busy. Two defaults keep them idle. With a device plugin, a Pod that requests nvidia.com/gpu: 1 gets a whole GPU, even if it only keeps a tenth of it busy. And the safest way to keep tenants apart, a cluster or a block of GPUs for each one, strands most of the hardware. Raising utilization means fixing both without weakening isolation.

Whole GPUs or slices

DRA describes each GPU to the scheduler in detail, down to its memory and its place in the topology, but it does not split a GPU. Splitting is the job of a sharing mechanism underneath. MIG cuts a GPU into hardware partitions that isolate memory and faults. HAMi, a CNCF project, enforces per-Pod limits on GPU memory and compute in software, and works across several accelerator vendors. Time-slicing is the cheapest of the three and isolates the least.

Which slicing is safe depends on trust. Operators disagree on whether MIG is a strong enough wall between tenants who don't trust each other, so the cautious default is a whole GPU for each untrusted tenant, with slices used only inside one tenant's trust boundary. Anyone heading toward confidential computing follows the same rule. Placement is a separate job. KAI Scheduler or Volcano decide where a gang-scheduled job lands in the topology, and Kueue decides which jobs are admitted, in what order, and against whose quota.

A control plane per tenant

Each product picks a point on this spectrum:

Customers of the elastic products never see a cluster, so a namespace per tenant works for them, with a node pool per tenant for reserved training. When a tenant wants a Kubernetes API of its own, a tenant cluster gives it one: a real Kubernetes API server, running as Pods on a shared cluster, where the tenant installs its own CRDs and webhooks, chooses its own version and writes its own RBAC, and cannot see anyone else. Ordinary tools work against it unchanged, so the tenant is never locked in.

Most operators end up with two tiers: a dedicated cluster, sometimes on dedicated machines, for enterprise tenants with strict trust requirements, and tenant clusters on shared capacity for everyone else. Because tenant control planes are just Pods, a failed one gets rescheduled like any other. What needs limiting is blast radius: put a ceiling on how many tenants share one underlying cluster, which is the same argument as for cells, below.

Isolating the data plane

None of that isolates the workloads themselves. Networks, storage, quotas, Pod Security and the runtime each need a boundary of their own.

Tenant networks are usually separated in the fabric, below Kubernetes. VXLAN and EVPN give each tenant a private network on Ethernet, and partition keys do the same on InfiniBand, with none of it in the GPUs' RDMA path. DPUs such as NVIDIA BlueField and AMD Pensando take on more of this work, running isolation and encryption on the network card instead of the host's CPU, which is also the path to confidential computing.

Storage gets its own StorageClass and quota per tenant. On shared nodes the runtime boundary is a sandboxed runtime, and where that is not enough, the tenant gets dedicated nodes. A cloud that sells isolation has to enforce it in hardware.

Tenants who want virtual machines instead of Pods can have them through KubeVirt, which runs VMs as Kubernetes workloads, governed by the same RBAC and quota rules as everything else.

Keeping nodes healthy

New nodes earn their place in the pool. Each one is inventoried (GPU count, ECC memory health, NIC and InfiniBand hardware IDs), booted over the network into an image that already carries the GPU driver, CUDA and NCCL, and given BIOS settings for its role. It joins only after a burn-in under load and an NCCL test at full bandwidth, with the results recorded in a source of truth such as NetBox, which can also hold IP address assignments. Metal3 with Ironic, or Tinkerbell, can drive this loop from open-source parts. Decommissioning runs it in reverse and ends with wiped disks and reset BMC credentials.

Once in service, GPUs fail constantly: memory errors, cards dropping off the PCIe bus, NVLink and thermal faults. DCGM's health checks spot the degradation, Node Problem Detector reports it as a node condition Kubernetes can act on, and automated remediation cordons and drains the node, so the next job lands on healthy hardware instead of failing for a tenant.

What AWS and Google learned

AWS and Google don't run their clouds on vanilla Kubernetes. They run internal systems, Borg on Google's side, that predate it by a decade. But three ideas run through how both built their clouds, and all three carry straight over to an AI cloud on Kubernetes.

Separate the control plane from the data plane

The data plane handles traffic: serving requests, forwarding packets, reading and writing bytes. The control plane handles change: provisioning, configuration, placement and scaling. Reading an S3 object goes through the data plane, and creating a bucket goes through the control plane.

They are built as separate systems because they differ in every way that matters. The data plane sees far more requests, has a sub-millisecond latency budget against the control plane's seconds, and can often tolerate eventual consistency. Their failures differ too: a data plane bug affects requests in flight, but a control plane bug can affect every future request and is much harder to roll back.

In an AI cloud, the inference data plane is the model-serving Pods, and its control plane is the autoscaler, the model loader and the routing rules. The training data plane is the running jobs, and its control plane is the scheduler and the queue. The most common mistake is putting the model loader on the data plane's path, so that every request quietly depends on a control plane lookup.

Static stability

The data plane must keep working while the control plane is down. The control plane pushes the desired state to the data plane in advance, the data plane caches what it needs and keeps serving from that cache during an outage, and the two reconcile when the control plane returns. DNS works this way, and so does EC2: during a control plane outage you cannot launch instances, but the running ones keep running.

On Kubernetes, cache desired state at the worker with informers, so a Pod keeps serving if the API server disappears. Never put the API server on the request path: a service that lists Pods on every request depends on etcd, and etcd will hiccup eventually. Use leader election sparingly, since it depends on etcd too.

When a control plane recovers, every data plane node that was waiting hits it at once, which can knock it straight back over, so use backoff with jitter and a slow ramp-up. Better still, design for constant work: a data plane that sends a full table of health every few seconds, instead of a message per failure, does the same work in an incident as on a quiet day, so incidents don't cascade.

Cells

A cell is a complete, self-contained copy of a service that serves a slice of the customers, with its own capacity, storage, control plane, observability and deployment wave. A bad deploy, a noisy customer or a runaway query stays inside one cell. It is the bulkhead idea from ship design: you accept that water gets in, and make sure it can only fill one compartment. You add capacity by adding cells, not by making them bigger.

A useful rule of thumb is 0.5 to 5% of customer traffic per cell, which comes to 20 to 200 cells per service per region. Below about 50 servers, running a cell is mostly overhead.

On Kubernetes, the clean mapping is one cluster per cell, with its own API server, etcd, nodes and upgrade cycle. A namespace per cell looks like a cell but isn't one: cells that share an etcd share a fate. vCluster sits between the two, with its own API server but shared nodes, which suits development cells and customers who want a private cluster, but not the production substrate.

Above the cells sits a regional control plane that you build yourself, usually a small Go service. It knows every cell with its capacity and health, places new customers, and drives deploys cell by cell. Above that, a global control plane handles accounts, billing and the console. Keep it small, because every endpoint added there is a future break in region isolation.

Shuffle sharding

Instead of assigning each customer to one cell, assign each to a random subset of K cells out of M. If one customer floods its cells, another customer only suffers fully if it shares all K of them, and the chance of that is one in C(M, K). With 2 of 8 cells that is 1 in 28, about 3.6%. With 4 of 64 it is 1 in 635,376.

The assignment must be deterministic, derived from the customer ID with no database lookup, so the router stays statically stable:

import hashlib, random

def assign_cells(customer_id: str, num_cells: int, k: int) -> list[int]:
    # A stable hash: Python's built-in hash() changes between processes.
    seed = int.from_bytes(hashlib.sha256(customer_id.encode()).digest()[:8], "big")
    return random.Random(seed).sample(range(num_cells), k)

It can run at the ingress, in the mesh or in your API gateway. The catch is that the cells have to be interchangeable. If a customer's data lives in one particular cell, you cannot route them to a random subset.

How it changes with scale

Tier 1: one region, 1,000 to 10,000 GPUs

One to three clusters, a handful of tenants and an operations team of 5 to 20. One big cluster per product line works, with etcd on its own 3-node ring, a single Argo CD, and capacity planning in a spreadsheet. The hard parts are hardware supply and building the customer control plane. You have outgrown it when running one cluster starts to cost real money: etcd warnings, watch storms during deploys, saturated image pulls and weekly pages for cluster-wide events.

Tier 2: many regions, 10,000 to 100,000 GPUs

Several regions with several clusters each, real multi-tenancy, and a team of 30 to 150. Customers no longer want to know which cluster they are on, so you need a control plane above the clusters, such as Karmada, Open Cluster Management or your own. Cells become the unit of failure and of rollout, and the quota service moves outside every cluster.

The hard problems here are training jobs that span cells, which you should simply not sell (one job, one cell), hierarchical quota across organizations, projects and users, and hot cells, because every customer wants US-East. You have outgrown this tier when hardware forces you into the supply chain, co-designing nodes, fabrics or cooling with vendors.

Tier 3: 100,000 GPUs and more

Dozens of regions, several GPU generations at once, and teams in the hundreds or thousands. You design hardware with the manufacturers and the network end to end, run a hardware operations organization, sell capacity in years rather than hours, and answer to SOC 2, ISO 27001, FedRAMP, HIPAA and the EU AI Act.

Kubernetes changes role here. It becomes a fleet of fleets, where no single Kubernetes API sees the whole world. The real control plane sits above it, thinking in cells, capacity, customers and contracts, and Kubernetes is the local executor inside each cell, the same shape as Borg.

The points where things break, roughly in order:

Concerns that cut across everything

Quota in three layers

Business quota is what the contract allows, held in your control plane's database. Cluster quota is what one cluster enforces, through Kueue. Pod limits are LimitRange, ResourceQuota and MIG slices. The usual mistake is enforcing only the last layer. You need all three, with the control plane as the source of truth and the cluster as the enforcer.

Metering and billing

Usage events come from the DCGM exporter, kube-state-metrics, gateway logs and the model engines, travel over an event bus such as Kafka or NATS, get priced by a rating engine such as Lago or OpenMeter, and end in an immutable, double-entry ledger. Get four things right early: idempotent event IDs, because you will replay events; acceptance of late events; a daily reconciliation between cluster counters and billing; and a usage API, because customers only trust you when their dashboard matches their invoice.

When the customers are internal teams, OpenCost can turn GPU-seconds into a cost per tenant instead. Either way, GPU telemetry only becomes per tenant once every Pod carries a tenant label.

What pages you

At scale, five things wake you at 3am: a fabric event that makes NCCL flap and kills training jobs, a storm of the same GPU XID across many nodes, an etcd compaction failure, an image pull storm against a cold registry, and a storage brownout that hangs training. Drill each one every quarter, and keep the DCGM exporter on every GPU node so you see them coming.

Building for what comes next

The stack underneath will keep changing. Expect several GPU generations in production at once, from the H100 to GB300 and Rubin, a growing AMD pool, and custom accelerators. Treat "GPU" as a resource class rather than a hardcoded type, and build on DRA. NVLink domains keep growing, so design schedulers around domains rather than racks. Bet on Kueue for queueing, JobSet and LeaderWorkerSet for AI workload shapes, and OCI artifacts for model weights and datasets.

CNCF's Certified Kubernetes AI Conformance program, launched in November 2025, turns several of these bets into a checklist: a conformant platform has to support DRA and gang scheduling, among other things.

Some things never get easier to add later. Quota planned for later never arrives. Multi-tenancy can't wait for a second tenant, because the first one will behave like a thousand. And a runbook promised for after the incident never gets written.

Further reading

The CNCF blog's Building an AI factory on Kubernetes looks at the same stack from the side of the teams sharing it, and goes further into GPU sharing, tenant isolation and the tool for each layer.