I’ve recently been working on migrating some of our Azure Kubernetes Service (AKS) clusters from Azure CNI Node Subnet to Azure CNI Overlay, while retaining Calico for network policy enforcement.

The problem I was trying to solve was IP exhaustion. When we originally started architecting our clusters, the available IP address space from a /21 that we picked seemed perfect, but jump forward to today, and the allocation is becoming a limitation.

The obvious solution was to rebuild the clusters using Azure CNI Overlay. However, rebuilding existing clusters isn’t exactly a small undertaking, particularly when they contain a mix of Linux and Windows node pools and you have multiple teams deploying workloads with multiple different processes.

So I wanted to see if I could migrate the networking configuration in place instead. The docs say no, but lets have a go anyway.

Why Azure CNI Overlay?

With the legacy Azure CNI Node Subnet networking model, pod IP addresses are allocated from the Azure virtual network subnet used by the cluster nodes.

This works well enough initially, but it means that nodes and pods are competing for address space in the same subnet. As the number of pods and nodes grows, IP address exhaustion can become a problem, particularly if the original subnet was sized for a much smaller environment.

In our case, the /21 allocation that seemed fine originally is no longer sufficient for the way the clusters had grown.

Azure CNI Overlay takes a different approach. Nodes continue to receive IP addresses from the Azure virtual network, but pods receive addresses from a separate, private overlay CIDR.

This has two important benefits:

  • The Azure virtual network subnet no longer needs to accommodate an IP address for every pod.
  • You can allocate a much larger address range for pods without consuming additional addresses from the node subnet.

Microsoft’s docs explains the networking model in more detail:

Overview of Azure CNI Overlay networking

The challenge with Calico

Our clusters use Calico for Kubernetes network policy enforcement. They also have a mix of Windows and Linux node pools, so retaining support for both operating systems was important, meaning pretty much only Calico.

When I looked at the Microsoft documentation for migrating an existing cluster to Azure CNI Overlay, one of the prerequisites stood out: network policies must be disabled before the migration.

That means you can’t simply change the networking mode to CNI Overlay while leaving the existing Calico network policy engine running.

I didn’t want to rebuild the clusters just to change the networking configuration, so I decided to try a different approach.

The sequence I used was:

  1. Disable the existing network policy engine, removing Calico.
  2. Migrate the cluster from Azure CNI Node Subnet to Azure CNI Overlay.
  3. Re-enable Calico once the networking migration has completed.

This worked on the clusters I migrated, including the mixed Linux and Windows node pools.

The migration process

1. Disable Calico

The first step is to disable the network policy engine on the AKS cluster.

This removes the AKS-managed Calico installation and allows the cluster to meet the prerequisite for migrating to Azure CNI Overlay.

You can do this through the Azure portal or using Azure CLI:

az aks update \
    --resource-group "rg-andy-cni-remove-test"\
    --name "aks-andy-cni-test" \
    --network-policy none

Wait for the operation to complete before proceeding.

This is the step that requires the most planning from a security perspective

Once Calico has been removed, the network policies it was enforcing are no longer enforced. They still remain on the cluster, but there is nothing to enforce them so they fail open.

How this affects you depends how you isolate your workloads so will need to be factored in.

2. Migrate to Azure CNI Overlay

Once Calico has been removed, you can migrate the cluster’s networking mode.

I performed this operation through the Azure portal, where I selected Azure CNI Overlay and specified a /16 pod CIDR.

If you’re using Azure CLI, the equivalent migration command is:

az aks update \
    --resource-group "rg-andy-cni-remove-test"\
    --name "aks-andy-cni-test" \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --pod-cidr 192.168.0.0/16

The important part is specifying the new pod CIDR. When migrating from legacy Azure CNI Node Subnet, the pods need a separate address range for the overlay network.

Microsoft requires that this range doesn’t overlap with the cluster’s virtual network address space or service CIDR. You also need to account for directly connected networks, including peered VNets, VPNs, ExpressRoute and on-premises networks.

See the official Azure CNI migration documentation for the current prerequisites and limitations.

Choosing the pod CIDR

This is where Azure CNI Overlay really helped solve our original problem.

We previously had a /21 allocation for the node subnet. I went with a /16 for the new pod CIDR, providing a much larger address pool for future growth.

Before choosing a range, check that it isn’t already in use elsewhere in your environment. This is particularly important if the cluster needs to communicate with other networks through VNet peerings, VPNs, ExpressRoute or on-premises connectivity. You can also re-use that POD Cidr across clusters.

Even though the overlay pod CIDR is separate from the node subnet, overlapping address ranges can still cause routing problems when traffic needs to cross network boundaries. If you need to maintain connectivity with other networks, make sure the range you choose won’t conflict with any existing address space or routes.

3. Re-enable Calico

Once the Azure CNI Overlay migration has completed and the cluster is healthy, the final step is to re-enable Calico.

az aks update \
    --resource-group "rg-andy-cni-remove-test"\
    --name "aks-andy-cni-test" \
    --network-policy calico

This reinstalls the AKS-managed Calico network policy engine.

In my testing, once Calico had finished installing, the existing network policies were enforced as expected again.

Microsoft’s AKS network policy documentation provides further information about Calico support and network policy behaviour.

What about downtime?

This isn’t a zero-downtime networking change.

During the migration, AKS reimages the node pools as part of the Overlay update.

Across the clusters I tested, there were several periods of network disruption, each relatively short. The overall process took around one to two hours per cluster, depending on the number of nodes.

So make sure your workloads can tolerate node reimaging and temporary connectivity interruptions. Review pod disruption budgets, replicas, readiness probes and application dependencies before starting.

I hope this helps if you’ve come across this blog post as it took me a while to figure it out, so hopefully it’s saved you some time. Obviously the AKS CLI can change between versions so if there are any mistakes in the commands then I’ll put it down to that, but the process in general is the same.

Thanks!

Andy