Architecture (20%)

Notes on Cilium’s architecture: eBPF-based dataplane, control plane components (cilium-agent, cilium-operator, Envoy), CNI integration, and how Cilium replaces kube-proxy.

Directory map

Cilium is not just a CNI, it also provides security and observability (via hubble)
Cilium is based-on and powered-by eBPF
- eBPF allows us to run sandboxed programs within the kernel’s address space
- pretty much all features within Cilium utilize eBPF in some way

Service Mesh

service meshes provide resilient connectivity
layer 7 traffic management (load balancing, rate limiting, etc.)
identity-based security
observability and tracing
transparency

Sidecar Architecutre

the sidecar (envoy proxy) contains all of the service-mesh features and is attached to the main container in the pod
all ingress/egress traffic goes through the proxy
the sidecar model has higher resource usage because every application pod requires a sidecar container
you can run into race conditions when sidecar containers are starting up or shutting down
sidecar containers add latency

Sidecarless model

No sidecar container!
eBPF programs handle L3/L4 traffic and envoy proxy (one per node) handles L7 traffic

IPAM

As an example, given a pod CIDR range of 10.244.0.0/16, node00 would get 10.244.1.0/24, node01 would get 10.244.2.0/24, etc. These are the “node specific CIDR ranges”, which is the focus of IPAM

Node Specific CIDR Range Modes

Host Scope Mode
- Pods get their IP addresses from the IP address pool assigned to the node. (See the example mentioned above)
- Cilium needs to be able to figure out what IP range was assigned to a node. To do this, Cilium will query the kube api and request the node object. The node object has two fields of interest: podCIDR and podCIDRS
- To enable Kubernetes Host Scope in Cilium configuration:
```
ipam:
  mode: "kubernetes"
k8s:
  requireIPv4PodCIDR: true
  reuqireIPv6PodCIDR: true
```

Cluster Scope Mode (default)

this is the default mode. Instead of Kubernetes handling IP address assignemtn to pods, the cilium operator handles it.
To configure it:

ipam:
  mode: "cluster-pool"
operator:
  clusterPoolIPv4PodCIDRList: 
  - "10.0.0.0/8"
  clusterPoolIPv4MaskSize: 24
  clusterPoolIPv6PodCIDRList:
  - "fd00::/104"
  clusterPoolIPv6MaskSize: 120

To view the IP range assigned to a node:

Exec to the agent pod on the node
Run: cilium-dbg status --all-addresses

  λ ostep-homework (master) $ k exec -it cilium-m662f -c cilium-agent  -- cilium-dbg status --all-addresses |grep -i "allocated addresses" -A15
  Allocated addresses:
    10.0.2.101 (health)
    10.0.2.110 (kube-system/hubble-relay-97b544484-mttqt [restored])
    10.0.2.14 (cert-manager/cert-manager-webhook-86b8846646-9cd8s [restored])
    10.0.2.148 (grafana/grafana-5798f9484b-c4qmc)
    10.0.2.174 (kubernetes-bind/kubernetes-bind-bind9-75c88b656d-2kvr7 [restored])
    10.0.2.198 (kube-system/hubble-ui-589d46c86f-4gttx [restored])
    10.0.2.20 (homepage/homepage-57bfc8b8fc-mw9cb [restored])
    10.0.2.205 (gitea/chart-1771039597-gitea-6586468c77-26xfz [restored])
    10.0.2.206 (metrics-server/metrics-server-65f9479866-bt87p [restored])
    10.0.2.219 (prometheus/prometheus-alertmanager-0 [restored])
    10.0.2.247 (prometheus/prometheus-prometheus-pushgateway-b6ffc6b67-7t6pj [restored])
    10.0.2.250 (kube-system/coredns-86bffb9cff-x564r [restored])
    10.0.2.28 (metallb/metallb-controller-7f4cd7d66b-x9jsb [restored])
    10.0.2.58 (router)

there are other IPAM nodes as well, including vendor (cloud) specific IPAM modes such as “Azure”
You should not change the IPAM mode of an existing cluster. Unless you like outages :)

Terminology

Endpoint: any and every pod gets an endpoint in the endpointslice
- Every endpoint gets a unique ID from cilium
- You can view all the endpoints on a node by exec’ing into an agent pod:
  - k exec <cilium pod name> -n kube-system -- cilium endpoint list
Policies:
- policies are applied to identities and identities are based on labels

Commands to know for the exam

cilium config view - shows the cilium config
cilium- config set debug true - enable debug logging

Keyboard shortcuts

notebook