notebook

IPAM

• As an example, given a pod CIDR range of 10.244.0.0/16, node00 would get 10.244.1.0/24, node01 would get 10.244.2.0/24, etc. These are the “node specific CIDR ranges”, which is the focus of IPAM

Node Specific CIDR Range Modes

• Host Scope Mode

  • Pods get their IP addresses from the IP address pool assigned to the node. (See the example mentioned above)
  • Cilium needs to be able to figure out what IP range was assigned to a node. To do this, Cilium will query the kube api and request the node object. The node object has two fields of interest: podCIDR and podCIDRS
  • To enable Kubernetes Host Scope in Cilium configuration:

  ipam:
    mode: "kubernetes"
  k8s:
    requireIPv4PodCIDR: true
    reuqireIPv6PodCIDR: true
  

• Cluster Scope Mode (default)

  • this is the default mode. Instead of Kubelet handling IP address assignemtn to pods, the cilium operator handles it.
  • To configure it:

  ipam:
    mode: "cluster-pool"
  operator:
    clusterPoolIPv4PodCIDRList: 
    - "10.0.0.0/8"
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv6PodCIDRList:
    - "fd00::/104"
    clusterPoolIPv6MaskSize: 120
  

  • To view the IP range assigned to a node:
    1. Exec to the agent pod on the node
    2. Run: cilium-dbg status --all-addresses

      λ ostep-homework (master) $ k exec -it cilium-m662f -c cilium-agent  -- cilium-dbg status --all-addresses |grep -i "allocated addresses" -A15
      Allocated addresses:
        10.0.2.101 (health)
        10.0.2.110 (kube-system/hubble-relay-97b544484-mttqt [restored])
        10.0.2.14 (cert-manager/cert-manager-webhook-86b8846646-9cd8s [restored])
        10.0.2.148 (grafana/grafana-5798f9484b-c4qmc)
        10.0.2.174 (kubernetes-bind/kubernetes-bind-bind9-75c88b656d-2kvr7 [restored])
        10.0.2.198 (kube-system/hubble-ui-589d46c86f-4gttx [restored])
        10.0.2.20 (homepage/homepage-57bfc8b8fc-mw9cb [restored])
        10.0.2.205 (gitea/chart-1771039597-gitea-6586468c77-26xfz [restored])
        10.0.2.206 (metrics-server/metrics-server-65f9479866-bt87p [restored])
        10.0.2.219 (prometheus/prometheus-alertmanager-0 [restored])
        10.0.2.247 (prometheus/prometheus-prometheus-pushgateway-b6ffc6b67-7t6pj [restored])
        10.0.2.250 (kube-system/coredns-86bffb9cff-x564r [restored])
        10.0.2.28 (metallb/metallb-controller-7f4cd7d66b-x9jsb [restored])
        10.0.2.58 (router)
    

• there are other IPAM nodes as well, including vendor (cloud) specific IPAM modes such as “Azure”

• You should not change the IPAM mode of an existing cluster. Unless you like outages :)