All writing
Updated 12 min readNaren

Reach services across clouds with internal load balancers

One load balancer per service, or one Traefik routing by hostname. What each costs, why GKE pods need a masquerade fix, and how names resolve.

KubernetesInfrastructureNetworking

You have a site-to-site virtual private network, or VPN, between two clusters. One is a Google Kubernetes Engine cluster, or GKE, in europe-west1. The other is an Azure Kubernetes Service cluster, or AKS, in australiaeast.

The Border Gateway Protocol, or BGP, carries routes across that tunnel, and only node subnets cross it. Building the tunnel is the previous post.

Now put services on it. At the end of this post a pod in GKE calls billing in AKS by name, and a pod in AKS calls orders in GKE.

There are two ways to get there. One is a cloud internal load balancer for each service. The other is one Traefik on one internal load balancer, routing by hostname.

Fix masquerading before anything else

Until you fix this, a GKE pod cannot reach anything in Azure, and every later step looks broken.

GKE masquerades pod traffic that leaves the cluster. Masquerading is source network address translation, or SNAT: the node rewrites the source address of a packet to its own address.

GKE does not masquerade traffic to 10.0.0.0/8, and the Azure virtual network 10.20.0.0/16 sits inside that range.

So a pod's packet to an Azure address keeps the pod address as its source. You never advertised the pod range, so Azure has no route back, and Azure drops the reply.

Run one check before you write any configuration.

kubectl get daemonset ip-masq-agent -n kube-system

NotFound means the cluster has no ip-masq-agent at all. The ConfigMap below then changes nothing, because nothing reads it.

A Dataplane V2 Standard cluster with a pod range inside 10.0.0.0/8 is the usual reason, and almost every tutorial omits this check. Autopilot always runs the DaemonSet, and takes an EgressNATPolicy custom resource instead of the ConfigMap.

Install the agent on the cluster before you follow any of the steps below.

ip-masq-agent-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ip-masq-agent
  namespace: kube-system
data:
  config: |
    nonMasqueradeCIDRs:
      - 10.10.0.0/16

Apply that with kubectl apply -f ip-masq-agent-config.yaml. The key name uses CIDR, Classless Inter-Domain Routing, the notation that writes a range as an address and a prefix length.

10.10.0.0/16 covers the GKE node subnet 10.10.0.0/20 and the pod range 10.10.128.0/17, so the node leaves pod to pod traffic inside GCP untouched. The node masquerades everything else, including 10.20.0.0/16.

A pod's packet to Azure now leaves with the node address, and the Cloud Router advertises that address as part of 10.10.0.0/20.

Do not use --disable-default-snat as a shortcut. Google documents that the flag has no effect when a cluster includes an ip-masq-agent DaemonSet.

Option 1: one internal load balancer for each service

The GCP side

Reserve the GCP address first. Reserving removes the address from the node subnet pool, so GCP never assigns it to a node.

gcloud compute addresses create orders-ilb-vip \
  --project="$PROJECT_ID" --region="$REGION" \
  --subnet=gke-nodes-ew1 --addresses=10.10.15.240

That address sits inside 10.10.0.0/20, which the Cloud Router already advertises. Nothing about the tunnel changes.

orders-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: orders
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
    networking.gke.io/load-balancer-subnet: "gke-nodes-ew1"
    networking.gke.io/load-balancer-ip-addresses: "orders-ilb-vip"
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 10.20.0.0/20
  selector:
    app: orders
  ports:
    - name: tcp-http
      protocol: TCP
      port: 8080
      targetPort: 8080

The first annotation makes the load balancer internal rather than public. Get its key wrong and nothing errors. The pre-GKE-1.17 spelling cloud.google.com/load-balancer-type no longer appears in Google's documentation.

A LoadBalancer Service with neither spec.loadBalancerClass nor networking.gke.io/load-balancer-type: "Internal" gets an external passthrough load balancer with a public address. That is the one typo in this file that puts your service on the internet.

The second annotation names the subnet the frontend address comes from. It needs GKE subsetting, which is on by default from GKE 1.36, and Google says it takes precedence when its requirements are met.

Older clusters use networking.gke.io/internal-load-balancer-subnet instead, which works on every supported version and handles IPv4 only.

The third annotation pins the frontend address, and it takes the name of the reserved address object, not the address.

Pass 10.10.15.240 there and the Service stays at <pending>, with an event naming an address object that does not exist.

On GKE 1.33.1-gke.1779000 and later, Google's own example uses spec.loadBalancerClass: networking.gke.io/l4-regional-internal instead of the first annotation. It is the recommended form. It always creates an internal passthrough load balancer, and it is immutable once applied.

The annotation still works on every supported version, which is why this post keeps it.

Global access, and where it is usually misread

Global access is where almost everyone goes wrong, and the mistake is to read it as a rule about where the client sits. The documented test is different. Both the Cloud VPN gateway and its tunnels must sit in the same region as the load balancer when global access is off.

Our Cloud VPN gateway and this load balancer both sit in europe-west1. So networking.gke.io/internal-load-balancer-allow-global-access: "true" is not required here, even though the client is in Azure.

Move the GCP end of the tunnel to another region and the load balancer drops traffic arriving over it, with no error on either side. That is how this breaks months later, when someone adds a second tunnel in a second region.

Setting the annotation anyway is a judgement call, one line, and free. Pair it with --bgp-routing-mode=global, which has the same single-region limit.

The Azure side

The Azure side takes three annotations and no reservation.

billing-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: billing
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "snet-aks-ilb"
    service.beta.kubernetes.io/azure-load-balancer-ipv4: 10.20.16.10
spec:
  type: LoadBalancer
  selector:
    app: billing
  ports:
    - name: tcp-http
      protocol: TCP
      port: 8080
      targetPort: 8080

The first annotation makes the frontend private. The second puts that frontend in snet-aks-ilb, which is 10.20.16.0/24, rather than in the node subnet. The third pins the address, and it takes the address itself.

GKE wants the name of a reserved object, Azure wants the address, because Azure has nothing to reserve.

Microsoft names the baseline: the AKS cluster identity needs at least the Network Contributor role on the virtual network resource. Putting the frontend in a subnet the cluster does not use adds one more requirement.

The identity needs Read access to that subnet. Scoping Network Contributor at the virtual network satisfies both in one role assignment. That is the scope worth granting.

Only one side can reserve its address

GCP can reserve a private address, so the GCP frontend is safe inside the node subnet. Azure cannot. Microsoft states it plainly: you cannot reserve a private IP address.

The documented rule is narrow: do not pin an address already assigned to another resource. Nothing reserves the address you pin inside the AKS node subnet.

A node network interface can take it on the next scale-up, and that is reasoning from the rule, not a documented failure. A separate subnet removes the question. It costs nothing here, because Azure advertises the whole 10.20.0.0/16 range over the tunnel either way.

Testing without fooling yourself

Do not test either address with ping. Neither cloud answers it. The GCP forwarding rule matches only the protocols and ports you named.

On the Azure side, Microsoft supports ping and traceroute against the frontend of a Standard public load balancer only. It does not support testing inbound connectivity to an internal load balancer frontend at all.

Use the real port, with curl or nc -vz.

Run every test from the far side. Nothing inside the cluster that owns a load balancer may dial its own load balancer address. On Azure the flow from a backend to its own internal frontend fails outright.

On GCP it appears to succeed while never leaving the machine, because the guest environment installs a local route for the forwarding rule address.

In-cluster callers use the ClusterIP or the cluster DNS name.

One firewall note

One firewall note. GKE writes the firewall rules for its own load balancers, and loadBalancerSourceRanges sets the source in them.

Leave the field out and GKE writes 0.0.0.0/0, which is documented behaviour and a real finding in a security review.

The same field does nothing on an AKS internal load balancer, because Azure already allows the whole virtual network by default.

The health check range 35.191.0.0/16 only matters if you run a deny-by-default firewall policy that overrides the rules GKE writes.

Option 2: one Traefik, one address, a route for each service

Traefik is a reverse proxy. It reads Kubernetes objects and picks a backend from the Host header of each request. One Traefik per cluster means one internal load balancer per cluster.

traefik-gke-values.yaml
service:
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
    networking.gke.io/load-balancer-subnet: "gke-nodes-ew1"
    networking.gke.io/load-balancer-ip-addresses: "traefik-ilb-vip"
  spec:
    type: LoadBalancer
traefik-aks-values.yaml
service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "snet-aks-ilb"
    service.beta.kubernetes.io/azure-load-balancer-ipv4: 10.20.16.11
  spec:
    type: LoadBalancer

The annotations are the same annotations as option 1. Only the object carrying them changed. Reserve 10.10.15.241 as traefik-ilb-vip on the GCP side, the same way you reserved 10.10.15.240 above.

Two things about the shape of those files. The key is service.spec.type, not service.type, and the chart ignores the old key in silence. --set service.type=ClusterIP exits 0 with empty stderr and still renders type: LoadBalancer.

Annotations go on service.annotations, never under service.spec. Put them in the wrong place and the Service comes out with no annotations.

If the API server accepts the manifest at all, you also get a public address on both clouds.

helm repo add traefik https://traefik.github.io/charts
helm repo update

helm template traefik traefik/traefik --version 41.5.0 \
  -n traefik -f traefik-gke-values.yaml | yq 'select(.kind == "Service")'

Read that rendered Service before you install anything. Then put the custom resource definitions, or CRDs, on the cluster first, because helm upgrade never touches them.

helm show crds traefik/traefik --version 41.5.0 \
  | kubectl apply --server-side --force-conflicts -f -

helm upgrade --install traefik traefik/traefik --version 41.5.0 \
  --namespace traefik --create-namespace \
  --values traefik-aks-values.yaml

Pin the version, and check helm search repo traefik/traefik --versions before you copy that command. New versions land every week or two. Use traefik-gke-values.yaml when you install into the GKE cluster.

Then one route object for each service. This is the whole per-service cost under option 2.

billing-route.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: billing
  namespace: default
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`billing.aks.internal`)
      kind: Rule
      services:
        - name: billing
          port: 8080

Set the backing Services to type: ClusterIP under this option. A Service left on type: LoadBalancer keeps its own cloud load balancer, and you pay for both.

Traefik routes HTTP by hostname. A protocol that is not HTTP needs its own load balancer. Traefik does have a TCP router, but it keys on the TLS server name, not on a header.

What each option costs

Option 1 costs one address, one load balancer and one record in the Domain Name System, or DNS, for each service. Option 2 costs one address and one load balancer for the whole cluster.

Adding a service is where they separate. Under option 1 you reserve an address, write the annotations, and add a DNS record. Under option 2 you add a route object.

Publish one wildcard record for each zone up front, *.aks.internal at 10.20.16.11 and *.gke.internal at 10.10.15.241.

Then nothing outside the cluster changes when you add a service. Without the wildcard you still write one record for each new name.

Address consumption follows the same shape. snet-aks-ilb is a /24, Azure reserves five addresses in every subnet, and 251 addresses remain for load balancers. That is enough for most clusters, and the limit applies per cluster. Both clouds charge per load balancer, so option 1's bill grows with the service count and option 2's does not.

Option 2 pays for that saving with one shared failure point. Every cross-cloud call passes through one deployment. Option 1 fails one service at a time.

In the diagram below, look at what each option needs when you add a service.

Under option one each service has its own load balancer address, so a new service needs a new address. Under option two every service shares one address and a new service needs only a route. One load balancer per service One Traefik, one address Host billing Host next GKE pod 10.20.16.10 billing a new address the next service GKE pod 10.20.16.11 billing the next service

Resolve names in both directions

The two options need different name work, so it comes last. Under option 1 each service name resolves to its own load balancer address.

billing.aks.internal answers with 10.20.16.10, and orders.gke.internal answers with 10.10.15.240. Under option 2 every name in a cluster resolves to that cluster's one Traefik address. That is 10.20.16.11 on the Azure side and 10.10.15.241 on the GCP side, and Traefik picks the backend from the Host header.

Substitute the address that matches the option you chose into every record below.

Neither cloud resolves the other's private names on its own. Each side runs a private resolver, and you configure each one to forward one zone to the other.

Azure names first

Start with the Azure names, so that GCP has a name to resolve.

az network private-dns zone create -g $RG -n aks.internal

az network private-dns link vnet create -g $RG -n link-aks-internal \
  --zone-name aks.internal --virtual-network $VNET --registration-enabled false

# option 1: one record for each service, at that service's own load balancer
az network private-dns record-set a add-record -g $RG \
  --zone-name aks.internal --record-set-name billing --ipv4-address 10.20.16.10

# option 2: one wildcard record for the zone, at the Traefik load balancer
az network private-dns record-set a add-record -g $RG \
  --zone-name aks.internal --record-set-name "*" --ipv4-address 10.20.16.11

Write one of those two records, not both. Option 1 needs a name for each service. Option 2 needs the wildcard, which is what makes a new service cost nothing outside the cluster.

Letting GCP query Azure

A private zone answers queries from inside the virtual network only. To answer a query from GCP, Azure needs a DNS Private Resolver with an inbound endpoint.

az network vnet subnet create -g $RG --vnet-name $VNET \
  -n snet-dns-inbound --address-prefix 10.20.17.0/28 \
  --delegations Microsoft.Network/dnsResolvers

VNET_ID=$(az network vnet show -g $RG -n $VNET --query id -o tsv)
SNET_IN=$(az network vnet subnet show -g $RG --vnet-name $VNET \
  -n snet-dns-inbound --query id -o tsv)

az dns-resolver create -g $RG -n dnsr-aue -l $LOC --id "$VNET_ID"

az dns-resolver inbound-endpoint create -g $RG \
  --dns-resolver-name dnsr-aue -n inbound -l $LOC \
  --ip-configurations "[{'private-ip-address':'10.20.17.4','private-ip-allocation-method':'Static','id':'$SNET_IN'}]"

Now point Cloud DNS at 10.20.17.4 with a private forwarding zone, and use --private-forwarding-targets. Google routes a forwarding target inside 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16 through the network you named in --networks either way.

10.20.17.4 is in the first of those, so --forwarding-targets reaches the same resolver here. Neither flag sends this query to the internet.

Only the private routing flag still works when the target is a private address outside those three ranges. It also states the intent, rather than leaving Google to infer it from the address.

gcloud dns managed-zones create aks-internal \
  --project="$PROJECT_ID" --dns-name=aks.internal. \
  --visibility=private --networks=gke-vpc \
  --private-forwarding-targets=10.20.17.4 \
  --description="Forward Azure names to the Azure resolver"

The route back that is easy to miss

One more step, and it is the one people miss. Cloud DNS sends forwarded queries from 35.199.192.0/19, a fixed Google range.

Your Cloud Router advertises 10.10.0.0/20 and nothing else, so Azure has no route back to that range and the reply never returns. Add the range to both BGP sessions.

gcloud compute routers update-bgp-peer cr-ew1 \
  --project="$PROJECT_ID" --region="$REGION" \
  --peer-name=bgp-azure-0 \
  --add-advertisement-ranges=35.199.192.0/19=cloud-dns-forwarding

Repeat that for bgp-azure-1, and run gcloud compute routers get-status cr-ew1 between the two calls. A mistake in that list stops traffic over the tunnel while the status output still looks healthy.

The reverse direction needs no equivalent step. Azure advertises its whole virtual network prefix and offers no filter, so GCP already has a route to 10.20.17.4.

GCP advertises only the ranges you list, which is why you added 35.199.192.0/19 by hand.

One warning before you build on this path. Microsoft does not document whether a DNS Private Resolver inbound endpoint accepts queries from 35.199.192.0/19. That range is public, and Cloud DNS routes it privately over the tunnel.

Nothing forbids it either. Test that one hop before you commit to the design. No vendor publishes a cross-cloud hybrid DNS guide at all. This is two single-cloud hybrid designs joined at the tunnel.

GCP names, and the way in for Azure

Now the GCP names, and an inbound entry point for Azure to query.

gcloud dns managed-zones create gke-internal \
  --project="$PROJECT_ID" --dns-name=gke.internal. \
  --visibility=private --networks=gke-vpc \
  --description="Names for GKE internal load balancers"

# option 1: one record for each service
gcloud dns record-sets create orders.gke.internal. \
  --project="$PROJECT_ID" --zone=gke-internal --type=A --ttl=60 \
  --rrdatas=10.10.15.240

# option 2: one wildcard record for the zone
gcloud dns record-sets create "*.gke.internal." \
  --project="$PROJECT_ID" --zone=gke-internal --type=A --ttl=60 \
  --rrdatas=10.10.15.241

gcloud dns policies create allow-inbound-dns \
  --project="$PROJECT_ID" --networks=gke-vpc \
  --enable-inbound-forwarding \
  --description="Let Azure query Cloud DNS"

GCP_INBOUND_IP=$(gcloud compute addresses list --project="$PROJECT_ID" \
  --filter='purpose = "DNS_RESOLVER" AND region:europe-west1' \
  --format="value(address)")

That last command stores the inbound forwarder address in $GCP_INBOUND_IP. The address comes from gke-nodes-ew1, which the Cloud Router already advertises, so the tunnel needs no change.

az network vnet subnet create -g $RG --vnet-name $VNET \
  -n snet-dns-outbound --address-prefix 10.20.17.16/28 \
  --delegations Microsoft.Network/dnsResolvers

SNET_OUT=$(az network vnet subnet show -g $RG --vnet-name $VNET \
  -n snet-dns-outbound --query id -o tsv)

az dns-resolver outbound-endpoint create -g $RG \
  --dns-resolver-name dnsr-aue -n outbound -l $LOC --id "$SNET_OUT"

OUT_EP=$(az dns-resolver outbound-endpoint show -g $RG \
  --dns-resolver-name dnsr-aue -n outbound --query id -o tsv)

az dns-resolver forwarding-ruleset create -g $RG -n frs-gcp -l $LOC \
  --outbound-endpoints "[{'id':'$OUT_EP'}]"

az dns-resolver forwarding-rule create -g $RG \
  --ruleset-name frs-gcp -n gke-internal \
  --domain-name "gke.internal." --forwarding-rule-state Enabled \
  --target-dns-servers "[{'ip-address':'$GCP_INBOUND_IP','port':53}]"

az dns-resolver vnet-link create -g $RG --ruleset-name frs-gcp \
  -n link-vnet --id "$VNET_ID"

Those queries leave Azure from the outbound endpoint subnet, 10.20.17.16/28, and land on the Cloud DNS inbound forwarder address. Test that hop before you commit too.

A DNS Private Resolver rule expects its target to answer recursively. Microsoft names another cloud's inbound resolver endpoint as the usual cause of a cross-cloud SERVFAIL, the server failure answer.

Run dig @$GCP_INBOUND_IP orders.gke.internal from an AKS pod. Substitute the real address for $GCP_INBOUND_IP. The answer must carry the record in the ANSWER section with the ra flag set.

NS records in AUTHORITY mean a referral, and the lookup fails.

Firewall rules on both sides

You do not need a firewall rule on the GCP side, which surprises people. Google Cloud firewall rules do not apply to the regional internal addresses that act as entry points for inbound forwarders.

Cloud DNS accepts TCP and UDP on port 53 there automatically. Many posts tell you to write that rule. Do not.

Azure usually needs nothing in return. The default AllowVNetInBound rule at priority 65000 uses the VirtualNetwork service tag, which Microsoft says covers all connected on-premises address spaces.

Microsoft does not document whether that tag covers ranges learned over BGP, so treat it as untested.

If you attach a network security group, or NSG, to snet-dns-inbound, write an explicit rule for 35.199.192.0/19 on port 53.

The whole path, and where to look when it fails

Here is the whole path for one call under option 1, with the two pieces this post added. Under option 2 the answer is the Traefik address, and the rest of the path is the same.

Look at the source address on the last request.

A GKE pod asks Cloud DNS for an Azure name, Cloud DNS forwards the query to the Azure private resolver, and the pod then reaches the Azure internal load balancer with the node address as the source. billing.aks.internal query from 35.199.192.0/19 10.20.16.10 10.20.16.10 HTTP to 10.20.16.10 response The node rewrites the source to the node address GKE pod Cloud DNS Azure resolver Azure load balancer

When a call fails, check it in that order. Resolve the name from inside a pod first.

Then confirm that the source address arriving in Azure is a node address and not a pod address.

Then confirm the load balancer has endpoints. An internal load balancer with no healthy backend accepts nothing, and it reports nothing useful from the far side of a tunnel.