Connectivity

Intra-node Connectivity

Each pod connects to the host through a vEth pair. Cilium attaches eBPF programs to the iface on the host side so that packets are inspected and forwarded as soon as they leave or enter the pod.

λ homelab (main) $ kgp
NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
httpbin-v1-7b9cddf9cc-8679v   1/1     Running   0          47h   10.0.2.64    worker02   <none>           <none>
httpbin-v2-5d5c667fd4-v42zr   1/1     Running   0          47h   10.0.4.253   worker05   <none>           <none>
netshoot00                    1/1     Running   0          11h   10.0.2.27    worker02   <none>           <none>
λ homelab (main) $ k exec -it netshoot00 -- ping  10.0.2.64
PING 10.0.2.64 (10.0.2.64) 56(84) bytes of data.
64 bytes from 10.0.2.64: icmp_seq=1 ttl=63 time=0.114 ms
64 bytes from 10.0.2.64: icmp_seq=2 ttl=63 time=0.129 ms
^C
# ///// we can see the pod-side of the veth pair:
λ homelab (main) $ k exec -it netshoot00 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo
       valid_lft forever preferred_lft forever
749: eth0@if750: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether a6:45:9f:47:75:59 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    altname cilium_cni:eth0
    inet 10.0.2.27/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a445:9fff:fe47:7559/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

# ////// on worker02, we can see the paired interface:
rtn@worker02:~$ ip a |grep -i 750
750: lxc45467b904c02@if749: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

Cilium also installs a set of interfaces during init. For traffic leaving the pod:

the gateway address maps to the cilium_host device in the host’s network namespace:

rtn@worker02:~$ ip a show cilium_host
5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4e:78:e2:30:ba:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.58/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80::4c78:e2ff:fe30:bab8/64 scope link
       valid_lft forever preferred_lft forever

cilium_net is the other side of the veth-pair with cilium_host:

rtn@worker02:~$ ip a show cilium_net
4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2a:48:43:58:48:13 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2848:43ff:fe58:4813/64 scope link
       valid_lft forever preferred_lft forever

Routing Modes

Cilium routing modes answer the question of “how do external (to the cluster) network devices route traffic to pods running on nodes, if the external devices have no concept of internal cluster networking (including the IP address scope assigned to the node.)”
Cilium has two routing modes: Encapsulation or Native

Encapsulation Routing Mode (default)

In the configuration, this is “tunnel” routing mode
```
routingMode: tunnel
```
When a pod wants to send a packet to a pod on another node, it constructs a packet with a source address of its own interface and a destionation address of the other pod. It will then encapsulate the packet in another packet with a source address of the node and a destination address of the node where the destination pod is scheduled. The packet is then routed from the source node to the other. Once the destination node receives the packet, it decapsulates it, retrieving the packet the source pod created, discovers the destination pod IP address, resolves ARP, and sends the packet to the destination pod.
Encapsulation mode is simple to configure, you simply need to ensure that the nodes can route traffic to eachother and that any network devices in between the nodes allow VXLAN traffic.
There are two different encapsluation mode protocols:
- VXLAN: 8472/udp (default)
- Geneve: 6081/udp
Disadvantages of Encapsulation Routing Mode:
- encapsulation mode requires roughly 50 bytes of the header (so ‘MTU - 50’) for the additional data
- additional latency
- debugging is more complex
- potential for packet fragmentation
No additional configuration required

Native Routing Mode

Cilium can run routing protocols (BGP)
The cilium agents run BGP and advertise their routes on the network
No packet encapsulation! Woo hoo! But, BGP is complicated and cumbersome.
You could always configure static routing as well, nothing wrong with that :)

to configure:

routingMode: "native"
ipv4NativeRoutingCIDR: "10.244.0.0/16"
ipv6NativeRoutingCIDR: ""

Inspecting eBPF Programs

We can inspect which eBPF programs are attached to a node’s interface:

  λ homelab (main) $ k exec -it cilium-26tfv -- bpftool net show
  xdp:

  tc:
  eno1(2) tcx/ingress cil_from_netdev prog_id 80209 link_id 315
  eno1(2) tcx/egress cil_to_netdev prog_id 80216 link_id 362
  cilium_net(4) tcx/ingress cil_to_host prog_id 80198 link_id 14
  cilium_host(5) tcx/ingress cil_to_host prog_id 80188 link_id 12
  cilium_host(5) tcx/egress cil_from_host prog_id 80187 link_id 13
  cilium_vxlan(6) tcx/ingress cil_from_overlay prog_id 80013 link_id 3
  cilium_vxlan(6) tcx/egress cil_to_overlay prog_id 80016 link_id 4
  lxc4a5851d572c6(12) tcx/ingress cil_from_container prog_id 80092 link_id 6
  lxca9021428c934(606) tcx/ingress cil_from_container prog_id 80163 link_id 308
  lxc7721387e82fa(608) tcx/ingress cil_from_container prog_id 80102 link_id 309
  lxc217d4bb5c3a5(642) tcx/ingress cil_from_container prog_id 80146 link_id 328
  lxcdde253b6d2b8(646) tcx/ingress cil_from_container prog_id 80042 link_id 330
  lxcc3b3d5cbf978(652) tcx/ingress cil_from_container prog_id 80128 link_id 333
  lxcaacf7a421043(722) tcx/ingress cil_from_container prog_id 80162 link_id 384
  lxc673840cf9a8a(728) tcx/ingress cil_from_container prog_id 80075 link_id 387
  lxc8cce85202c73(738) tcx/ingress cil_from_container prog_id 80123 link_id 425
  lxc00cb567e6c12(746) tcx/ingress cil_from_container prog_id 80059 link_id 431
  lxc45467b904c02(750) tcx/ingress cil_from_container prog_id 80061 link_id 433
  lxc_health(762) tcx/ingress cil_from_container prog_id 80029 link_id 444

  flow_dissector:

  netfilter:

  λ homelab (main) $