Cluster Mesh

Cluster mesh is a feature that allows pods across clusters to communicate with each other. It does this by sharing netowrk and identity information and also avoiding the need to expose service beyond the cluster altogether. Cluster-mesh does not provide network connectivity between clusters, you need to implement that yourself.

All nodes within each cluster must have IP (layer 3) connectivity to each other. Clusters do not have to share a layer 2 domain.

Application data does not flow through the ClusterMesh API Server

Requirements

Non-overlapping pod CIDRs
Full node-to-node connectivity

Setup

assign a name and id to each cluster
enable cluster mesh via the cilium cli: cilium clustermesh enable --context $CLUSTER1 cilium clustermesh enable --context $CLUSTER2 this creates a clustermesh api server deployment in the cluster
connect the clusters:

cilim clustermesh connect --context $CLUSTER1 --destination-context $CLUSTER2

KVStoreMesh

An optional (default-on since Cilium v1.16) extension of Cluster Mesh that caches remote-cluster state in the local kvstore instead of having every local agent watch every remote cluster directly. KVStoreMesh is really just an in-memory etcd database running inside the ClusterMesh API Server pod.

The problem it solves

In “vanilla” Cluster Mesh, every Cilium agent in a cluster opens a direct watch connection to the clustermesh-apiserver/etcd of every connected remote cluster
That means connection count scales as agents x remote clusters, which gets expensive fast as you add clusters or nodes
A slow, flapping, or unreachable remote cluster can also directly affect every agent that’s watching it (no isolation)

How it works

A kvstoremesh process runs inside the local cluster’s clustermesh-apiserver pod
kvstoremesh (not the individual agents) is what connects out to each remote cluster’s clustermesh-apiserver and pulls in their state (nodes, identities, endpoints, services)
That remote state is cached into the local kvstore
Local Cilium agents then only ever talk to their own local kvstore/apiserver, never directly to a remote cluster

flowchart TB
    subgraph Vanilla["Vanilla Cluster Mesh (no KVStoreMesh)"]
        A1[Agent] --> R1[("Remote Cluster A etcd")]
        A2[Agent] --> R1
        A1 --> R2[("Remote Cluster B etcd")]
        A2 --> R2
    end
    subgraph WithKVSM["With KVStoreMesh"]
        B1[Agent] --> Local[("Local kvstore")]
        B2[Agent] --> Local
        KVSM["kvstoremesh (in clustermesh-apiserver)"] --> Local
        KVSM --> RA[("Remote Cluster A etcd")]
        KVSM --> RB[("Remote Cluster B etcd")]
    end

Benefits

Connection count at the agent level drops from agents x remote clusters to just 1 (the local kvstore) — the remote clusters fan-out moves to a single kvstoremesh process per cluster
Better isolation: a problem talking to one remote cluster only affects kvstoremesh’s sync of that cluster, not every agent in the local cluster
Reduces load on remote clusters’ etcd/apiserver, since they’re now only serving one kvstoremesh connection per remote cluster instead of one per remote agent

Configuration

Enabled by default starting in Cilium v1.16
Toggle explicitly via the Cilium CLI (Helm mode): --enable-kvstoremesh / --enable-kvstoremesh=false
Related scaling knob (not KVStoreMesh-specific, but relevant to large meshes): maxConnectedClusters raises the connected-cluster limit from 255 to 511, at the cost of halving the max number of cluster-local identities (65,535 -> 32,767); must match across every cluster in the mesh

notebook