Help! I need to change the pod CIDR in my Kubernetes cluster

Change_k8s_pod_cidr 1 1

Your Docker EE Kubernetes cluster has been working great for months. The DevOps team is fully committed to deploying critical applications as Kubernetes workloads using their pipeline, and there are several production applications already deployed in your Kubernetes cluster.

But today the DevOps team tells you something is wrong; they can’t reach a group of internal corporate servers from Kubernetes pods. They can reach those same servers using basic Docker containers and Swarm services. You’re sure its just another firewall misconfiguration and you enlist the help of your network team to fix it. After several hours of troubleshooting, you realize that the problem is that you are using a CIDR (Classless Inter-Domain Routing) range for your cluster’s pod CIDR range that overlaps the CIDR range that the servers use.

Resistance is futile; management tells you that the server IP addresses can’t be changed, so you must change the CIDR range for your Kubernetes cluster. You do a little Internet surfing and quickly figure out that this is not considered an easy task. Worse yet, most of the advice is for Kubernetes clusters installed using tools like kubeadm or kops, while your cluster is installed under Docker EE UCP.

Relax! In this blog post, I’m going to walk you through changing the pod CIDR range in Kubernetes running under Docker EE. There will be some disruptions at the time that the existing Kubernetes pods are re-started to use IP addresses from the new CIDR range but they should be minimal if your applications use a replicated design.

High-level overview of what we are going to do

  • Change the network IP CIDR range reserved for Kubernetes pods in Docker EE UCP.
    • Export the UCP configuration as a toml file.
    • Edit the value of pod_cidr in the exported file.
    • Apply the edited file to UCP to update the UCP configuration.
  • Use calicoctl to create a new IP pool for the new pod_cidr value in the Calico CNI plugin.
    • Disable the old IP pool after creating the new IP pool.
  • Migrate existing workloads (pods) to the new IP pool.
    • Delete the existing pods so that the underlying Deployments, DaemonSets, ReplicaSets, etc. create new pods.
  • Delete the old IP pool.
  • Force ucp-kubelet and the other Kubernetes control plane containers under UCP to be recreated.
    • Delete the existing container so that UCP creates new ones to replace them.
    • The new Kubernetes control plane containers will use the new pod CIDR range from the updated UCP configuration.

Prerequisites

  1. You will need to be able to SSH to one of the Docker manager nodes as the docker user.
  2. You will need an admin UCP client certificate bundle to load on the Docker manager node.
  3. You will need kubectl installed on the Docker manager node if you want to do all work from that node.
    • As an alternative approach for the steps requiring kubectl on a manager node, you can use kubectl from a client workstation with the admin UCP client certificate bundle loaded.
  4. You will need to be able to use curl with certificates on the Docker manager node to connect to UCP with HTTPS.
    • If you have problems with your terminal environment on a manager node supporting curl using certificates to connect to UCP using HTTPS (inadequate permissions, cert issues in some CentOS/RHEL releases, etc.) see the workaround at the end of this blog post.

See CLI-based access in Docker’s docs for instructions to set up a UCP client certificate bundle.
See Install and Set Up kubectl in the Kubernetes docs for instructions to install kubectl.

Detailed steps

Export, modify and re-apply the existing UCP configuration

  • SSH into one of your docker manager nodes.

  • Load an admin UCP client certificate bundle in the shell where you will be working.

  • Export the UCP configuration to a toml file:
    • From the directory where a UCP admin user’s client certificate bundle is unzipped, run the following command, substituting your UCP hostname for UCP_HOST:
    curl --cacert ca.pem --cert cert.pem --key key.pem https://UCP_HOST/api/ucp/config-toml > ucp-config.toml
    

See UCP configuration file in the Docker docs for full details.

  • Edit ucp-config.toml:
    • In the cluster_config section of the file, set the value of pod_cidr to your new CIDR range.
  • Use the following curl command to import ucp-config.toml back into UCP and apply your configuration changes, again replacing UCP_HOST with your UCP hostname:

curl --cacert ca.pem --cert cert.pem --key key.pem --upload-file ucp-config.toml https://UCP_HOST/api/ucp/config-toml

Change the Calico CNI plugin IP pool configuration

First set up calicoctl. This must be done in a shell with your UCP admin user bundle applied. Execute the following commands to set up an alias to run calicoctl:

UCP_VERSION=$(docker version --format '{{index (split .Server.Version "/") 1}}')

alias calicoctl="\
docker run -i --rm \
  --pid host \
  --net host \
  -e constraint:ostype==linux \
  -e ETCD_ENDPOINTS=127.0.0.1:12378 \
  -e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
  -e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
  -e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
  -v /var/run/calico:/var/run/calico \
  -v ucp-node-certs:/ucp-node-certs:ro \
  docker/ucp-dsinfo:${UCP_VERSION} \
  calicoctl \
"

I will generally follow the steps from the Calico documentation to create an IP pool matching the new pod_cidr and move existing pods to the new pool. See Changing IP pools for full details.

Check the existing IP pools

calicoctl get ippool -o wide

NAME                  CIDR             NAT    IPIPMODE   DISABLED   SELECTOR
default-ipv4-ippool   192.168.0.0/16   true   Always     false      all()

Add a new IP pool

Note: cluster-wide pod traffic interruption will start with this step, and will continue until all steps have been completed.
Execute the following command, substituting the correct new value for your environment in <YOUR_NEW_POD_CIDR>. In this example, we will use 10.16.96.0/19 as the new pod CIDR range. <NEW_POOL_NAME> can be any meaningful name for the new IP pool in your environment:

calicoctl create -f -<<EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: <NEW_POOL_NAME>
spec:
  cidr: <YOUR_NEW_POD_CIDR>
  ipipMode: Always
  natOutgoing: true
EOF

You should now have two enabled IP pools, which you can view by executing the following command. The new IP pool should use your new pod CIDR range:

 calicoctl get ippool -o wide

 NAME                  CIDR             NAT    IPIPMODE   DISABLED   SELECTOR
 default-ipv4-ippool   192.168.0.0/16   true   Always     false      all()
 new-pool              10.16.96.0/19    true   Always     false      all()

Disable the old IP pool

First, save the IP pool definition to disk:

calicoctl get ippool -o yaml > pool.yaml

Edit the pool.yaml file, adding disabled: true to the default-ipv4-ippool IP pool, for example (showing only the applicable section of the file):

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 192.0.0.0/16
  ipipMode: Always
  natOutgoing: true
  disabled: true

Apply the changes:

calicoctl apply -f pool.yaml
Note: If the previous command does not work correctly, you may need to use this command instead:
calicoctl apply -f - < pool.yaml

Check that the changes were successfully applied:

calicoctl get ippool -o wide

NAME                  CIDR             NAT    IPIPMODE   DISABLED
default-ipv4-ippool   192.168.0.0/16   true   Always     true       all()
new-pool              10.16.96.0/19    true   Always     false      all()

Recreate existing workloads

Recreate all workloads (pods) that use IP addresses from the disabled IP pool.

  • Note: If you have deployed any non-standard Kubernetes workloads that required allowing admins, users, or service accounts to run work on manager or DTR nodes, you may need to temporarily enable those setting for those workloads to be correctly recreated. This is done using the scheduler panel in <YOUR-ACCOUNT-NAME> -> Admin Settings -> Scheduler in the UCP GUI

First, check for existing workloads:

calicoctl get wep --all-namespaces

The (shortened) output should look like the following. Note that the IP addresses listed under NETWORKS are all in the default 192.168.0.0/16 pod CIDR range:

  • Note that if you have installed workloads that use the host network, their IP addresses may be in the CIDR range used by your Docker hosts.
calicoctl get wep --all-namespaces

NAMESPACE       WORKLOAD                                    NODE               NETWORKS             INTERFACE
kube-system     compose-7564c7c449-gpw6m                    ip-172-31-17-117   192.168.201.227/32   cali1ab882ff9f3
kube-system     ucp-metrics-tktht                           ip-172-31-17-117   192.168.201.228/32   cali1527fcf412a
kube-system     metrics-server-559d9f45f-w4l62              ip-172-31-27-54    192.168.38.178/32    cali666031bb042
kube-system     compose-api-767b9b494-p5k9q                 ip-172-31-7-202    192.168.52.97/32     caliabc8b623fec
kube-system     kube-dns-7785c9d44b-rbhnj                   ip-172-31-7-202    192.168.52.96/32     cali5fadfed3995
my-namespace    nginx-ds-66c8b                              ip-172-31-6-62     192.168.196.244/32   calic522fa3b3b0

Delete all existing workloads (pods) that use the old (now disabled) IP pool. In this example, this means workloads with an address in the 192.168.0.0/16 CIDR range. For example:

kubectl delete pod -n kube-system kube-dns-7785c9d44b-rbhnj

After all deleted workloads have been replaced with new pods, check that the new workloads now use addresses from the new IP pool by running the following command:

calicoctl get wep --all-namespaces

The (shortened) output should look like the following. Note that the IP addresses listed under NETWORKS that were previously in the default pod CIDR range are now in the new 10.16.96.0/19 pod CIDR range:

calicoctl get wep --all-namespaces

NAMESPACE       WORKLOAD                                    NODE               NETWORKS             INTERFACE
kube-system     compose-7564c7c449-5pb2w                    ip-172-31-11-103   10.16.109.129/32     cali5dd7584ffee
kube-system     compose-api-767b9b494-526q6                 ip-172-31-11-103   10.16.109.130/32     calicd0ce2ef4a0
kube-system     kube-dns-7785c9d44b-ssx9j                   ip-172-31-17-117   10.16.105.201/32     caliafa5425a562
kube-system     metrics-server-559d9f45f-ttjfg              ip-172-31-3-81     10.16.106.130/32     cali005d36ea7e3
kube-system     ucp-metrics-55h7c                           ip-172-31-7-202    10.16.117.1/32       cali65285622e99
my-namespace    nginx-ds-99h5b                              ip-172-31-10-195   10.16.116.1/32       cali8c6938faae0

Delete the old IP pool

calicoctl delete pool default-ipv4-ippool

Reconcile the Kubernetes control plane containers

Delete all ucp-kube-proxy, ucp-kubelet, and ucp-kube-apiserver containers, forcing them to be restarted using the new pod_cidr value from the updated UCP configuration:

  • Execute the following commands from a terminal with the UCP client bundle applied to list the containers:
docker ps -a | grep ucp-kube-proxy
docker ps -a | grep ucp-kubelet
docker ps -a | grep ucp-kube-apiserver
  • Delete all of the containers listed in the output of the commands above.
    • Use docker rm <CONTAINER> -f to delete a container immediately.
    • Use docker stop <container> followed by docker rm <container> for a graceful shutdown and deletion. Note that in some cases the old containers may be automatically deleted after being stopped.

Verify that the control plane containers have changed IP address pools

Execute the command docker inspect <CONTAINER> --format "{{ json .Config.Cmd }}" for the new container IDs. Again, use the following commands to list the containers:

docker ps -a | grep ucp-kube-proxy
docker ps -a | grep ucp-kubelet
docker ps -a | grep ucp-kube-apiserver

Check that the arguments passed into the startup command for the containers use the new pod CIDR range.

  • You should see an argument like –cluster-cidr=<YOUR-POD-CIDR>
  • Note that not all control plane containers explicitly use this argument, but if the argument is present it should use the new pod CIDR range.
  • In this example, only the ucp-kube-proxy containers included that argument.

And we’re done

We have completed changing the pod CIDR range for our Kubernetes cluster running under Docker EE UCP. If you have questions or feel like you need help with Kubernetes, Docker or anything related to containers, get in touch with us at Capstone IT.

Workaround if you can’t get curl to work with certificates for an HTTPS connection to UCP in your environment

If you are unable to connect to the UCP HTTPS URL using curl due to the curl implementation and/or certificate configuration on your OS, you may be able to use this workaround. The process was tested using a manager node but any node should work since the UCP bundle will be loaded.

  • SSH into a Docker node as the docker user.

  • If you already have a UCP client certificate bundle available on the Docker node, DO NOT apply the bundle in your shell at this point.

  • Create a directory /tmp/docker_bundle.
    • You can use any directory where you have permissions, but you will need to alter these instructions to reflect the directory you use.
  • Upload an admin user bundle to the Docker node, and unzip it into the /tmp/docker_bundle directory

  • DO NOT apply the bundle in your shell at this point.

  • Execute the following command:

    docker container run --rm -it -v /tmp/docker_bundle/:/docker_bundle nicolaka/netshoot bash
    
    • Note that you can use any image that includes a functional curl implementation and a text editor. However, the nicolaka/netshoot image also has the docker client and plenty of handy network utilities installed if you happen to need to do any troubleshooting.
  • Execute the following commands at the resulting bash prompt in the container:

    cd /docker_bundle
    
    eval "$(<env.sh)"
    
  • Carry out steps 2, 3 and 4 from the Export, modify and re-apply the existing UCP configuration section of these instructions at the bash prompt in the container. Remain in the /docker_bundle directory and do not exit the container until steps 2, 3 and 4 are complete.
    • If you accidentally exit the container (or if the container exits unexpectedly) your files will still be available in the /tmp/docker_bundle directory on the Docker node.
  • After steps 2, 3 and 4 have been successfully completed, exit the container and then delete the /tmp/docker_bundle directory for security reasons.

  • Continue with the Change the Calico CNI plugin IP pool configuration section of these instructions.

Leave a Reply