Tuning Cilium

https://docs.cilium.io/en/stable/operations/performance/tuning/

Netkit

Use netkit to improve latency and throughput. netkit replaces the veth pair that connects the pod cgroup namespace to the host namespace.
to enable netkit:

routingMode: native
bpf:
    datapathMode: netkit
    masquerade: true
kubeProxyReplacement: true

Normally when a pod communicates with the outside world, its packets must go through the host networking stack. Host-routing bypasses the host networking stack.
to enable host-routing:

bpf:
    masquerade: true
kubeProxyReplacement: true

Similar to IPv6 BIG TCP, IPv4 BIG TCP allows the network stack to prepare larger GSO (transmit) and GRO (receive) packets to reduce the number of times the stack is traversed which improves performance and latency. It reduces the CPU load and helps achieve higher speeds (i.e. 100Gbit/s and beyond).
To pass such packets through the stack BIG TCP sets IPv4 tot_len to 0 and uses skb->len as the real IPv4 total length. The proper IPv4 tot_len is set before transmitting the packet over the wire.
BIG TCP can operate in a DualStack setup, IPv6 packets will use the old lower limits (64k) if IPv6 BIG TCP is not enabled, and IPv4 packets will use the new larger ones (192k). Both IPv4 BIG TCP and IPv6 BIG TCP can be enabled so that both use the larger one (192k).
Note that Cilium assumes the default kernel values for GSO and GRO maximum sizes are 64k and adjusts them only when necessary, i.e. if BIG TCP is enabled and the current GSO/GRO maximum sizes are less than 192k it will try to increase them, respectively when BIG TCP is disabled and the current maximum values are more than 64k it will try to decrease them.
BIG TCP doesn’t require network interface MTU changes.
to enable bigtcp:

routingMode: native
enableIPv4BIGTCP: true
ipv4:
    enabled: true
bpf:
    masquerade: true
kubeProxyReplacement: true