Cluster Architecture, Installation & Configuration (25%)

Notes on control-plane/node architecture, kubeadm cluster bootstrapping and upgrades, RBAC, and high availability.

Topics

• Manage role based access control (RBAC)
• Use Kubeadm to install a basic cluster
• Manage a highly-available Kubernetes cluster
• Provision underlying infrastructure to deploy a Kubernetes cluster
• Perform a version upgrade on a Kubernetes cluster using Kubeadm
• Implement etcd backup and restore

Architecture refresher

Control plane components:

• kube-apiserver: the front door. Every component and user talks to it over REST. It authenticates, authorizes (RBAC), runs admission control, and persists state to etcd. It is the only component that talks to etcd.
• etcd: consistent key-value store holding all cluster state. Losing etcd without a backup means losing the cluster.
• kube-scheduler: watches for unscheduled Pods and assigns them to nodes based on resources, taints, affinity, and so on.
• kube-controller-manager: runs the control loops (Deployment, ReplicaSet, Node, Job, EndpointSlice controllers and more) that drive actual state toward desired state.
• cloud-controller-manager: optional, integrates with the cloud provider (load balancers, routes, node lifecycle).

Node components:

• kubelet: agent that makes sure containers described in PodSpecs are running. It is a systemd service, not a Pod.
• kube-proxy: programs iptables/IPVS rules that implement Services.
• Container runtime: containerd or CRI-O, speaks CRI to the kubelet.

On a kubeadm cluster the control-plane components run as static Pods. The kubelet reads manifests from /etc/kubernetes/manifests/ and keeps them running. Editing a manifest there restarts the component. Moving it out of the directory stops it.

Important paths:

RBAC

RBAC controls what an identity can do against the API. Four objects:

• Role: set of permissions within one namespace.
• ClusterRole: permissions cluster-wide, or on cluster-scoped resources (nodes, PVs), or reusable across namespaces.
• RoleBinding: grants a Role or ClusterRole to subjects within one namespace.
• ClusterRoleBinding: grants a ClusterRole to subjects across the whole cluster.

Subjects are User, Group, or ServiceAccount. Users are not API objects; they come from client certificates (CN is the user, O is the group) or an external identity provider. RBAC is purely additive: there are no deny rules.

A binding to a ClusterRole through a RoleBinding limits it to that namespace. This is a common way to reuse one ClusterRole such as view in many namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: dev
rules:
- apiGroups: [""]          # "" is the core group
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: dev
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Imperative creation is faster on the exam:

kubectl create role pod-reader --verb=get,list,watch --resource=pods -n dev
kubectl create rolebinding read-pods --role=pod-reader --user=jane -n dev
kubectl create clusterrole node-reader --verb=get,list --resource=nodes
kubectl create clusterrolebinding read-nodes --clusterrole=node-reader --user=jane
kubectl create serviceaccount app-sa -n dev
kubectl create rolebinding app-view --clusterrole=view --serviceaccount=dev:app-sa -n dev

# Verify
kubectl auth can-i list pods -n dev --as jane
kubectl auth can-i '*' '*' --as system:serviceaccount:dev:app-sa
kubectl auth can-i --list -n dev --as jane

Installing a cluster with kubeadm

1. Prepare every node

• Unique hostname, MAC, and product_uuid; connectivity between nodes.
• Swap off (swapoff -a and remove from /etc/fstab), unless explicitly configured otherwise.
• Load kernel modules overlay and br_netfilter.
• Sysctl: net.bridge.bridge-nf-call-iptables=1, net.bridge.bridge-nf-call-ip6tables=1, net.ipv4.ip_forward=1.
• Open required ports: 6443 (API), 2379-2380 (etcd), 10250 (kubelet), 10257 (controller-manager), 10259 (scheduler), 30000-32767 (NodePort) on workers.
• Install a container runtime (containerd) with SystemdCgroup = true in /etc/containerd/config.toml.
• Install kubeadm, kubelet, kubectl and hold their versions (apt-mark hold).

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

2. Initialize the control plane

sudo kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=<CP_IP> \
  --kubernetes-version=v1.31.0

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

The pod CIDR must match what the CNI expects. Nodes stay NotReady and CoreDNS stays Pending until a CNI plugin is installed.

3. Install a CNI

kubectl apply -f <cni-manifest.yaml>   # Calico, Flannel, Cilium, etc.

4. Join workers

# Print a fresh join command (tokens expire after 24h)
kubeadm token create --print-join-command

# On the worker
sudo kubeadm join <CP_IP>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Provisioning the underlying infrastructure

Things to decide before running kubeadm:

• Sizing: control-plane nodes typically 2+ vCPU and 2+ GB RAM minimum (more for production); workers sized for the workload. etcd wants fast disks (SSD) and low latency.
• Node count: odd number of control-plane nodes (3 or 5) for etcd quorum. Quorum is floor(n/2)+1, so 3 nodes tolerate 1 failure and 5 tolerate 2.
• Networking: node subnet, pod CIDR, and service CIDR must not overlap each other or the corporate network. Plan a stable endpoint (DNS name or load balancer) for the API server.
• OS: a supported Linux distro, synchronized time (chrony/NTP), unique hostnames, firewall rules for the ports listed above.
• Runtime and CNI: pick them up front; a CNI that supports NetworkPolicy (Calico, Cilium) is needed if policies will be enforced.
• Automation: Terraform/cloud-init for machines, Ansible for node prep, kubeadm (or Cluster API) for bootstrap.

High availability

Two topologies:

• Stacked etcd: etcd runs on the same nodes as the control plane. Simpler, fewer machines, but losing a node loses both a control-plane member and an etcd member.
• External etcd: etcd runs on dedicated nodes. More machines and more complexity, better failure isolation.

Both need a load balancer in front of the API servers (HAProxy plus keepalived, or a cloud LB), and --control-plane-endpoint set at init so that all kubeconfigs and certs point to the LB.

# First control-plane node
sudo kubeadm init \
  --control-plane-endpoint "k8s-api.example.com:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

# Additional control-plane nodes
sudo kubeadm join k8s-api.example.com:6443 --token <t> \
  --discovery-token-ca-cert-hash sha256:<h> \
  --control-plane --certificate-key <key>

Notes:

• --upload-certs stores control-plane certs in a Secret encrypted with the certificate key; it is deleted after 2 hours. Regenerate with kubeadm init phase upload-certs --upload-certs.
• The scheduler and controller-manager use leader election, so only one is active at a time. The API server is active on all nodes.
• Check etcd membership: etcdctl member list with the cert flags shown below.

Upgrading with kubeadm

Rules:

• Upgrade one minor version at a time (1.30 to 1.31, never skip).
• Control plane first, then workers.
• kubelet and kubectl must be upgraded after kubeadm. The kubelet may not be newer than the API server.
• Always take an etcd backup first.

Control-plane node (first node)

# 1. Upgrade kubeadm
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.31.x-*
sudo apt-mark hold kubeadm
kubeadm version

# 2. Plan and apply
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.31.x

# 3. Drain, upgrade kubelet and kubectl, restart, uncordon
kubectl drain cp1 --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.31.x-* kubectl=1.31.x-*
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon cp1

Additional control-plane nodes and workers

On additional control-plane nodes use sudo kubeadm upgrade node instead of apply. Same for workers:

# From a machine with kubectl
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data

# On worker1
sudo apt-get install -y kubeadm=1.31.x-*
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.31.x-* kubectl=1.31.x-*
sudo systemctl daemon-reload && sudo systemctl restart kubelet

# From kubectl machine
kubectl uncordon worker1

etcd backup and restore

Backup

etcd uses TLS. Find the cert paths in /etc/kubernetes/manifests/etcd.yaml (--cert-file, --key-file, --trusted-ca-file, --listen-client-urls).

ETCDCTL_API=3 etcdctl snapshot save /opt/backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify
etcdutl snapshot status /opt/backup/etcd-snapshot.db --write-out=table

Restore

Restore writes a new data directory; it does not need TLS flags because it works on the file, not the live server.

sudo etcdutl snapshot restore /opt/backup/etcd-snapshot.db \
  --data-dir /var/lib/etcd-restore

Then point etcd at the new directory by editing /etc/kubernetes/manifests/etcd.yaml: change the etcd-data hostPath volume path from /var/lib/etcd to /var/lib/etcd-restore. The kubelet recreates the etcd Pod automatically. Wait for the API server to return (crictl ps is useful while kubectl is down).

If etcdutl is not installed, etcdctl snapshot restore is the older equivalent (deprecated for restore in newer releases).

Cheat sheet

# Cluster info
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -n kube-system
kubectl get componentstatuses            # deprecated, but still sometimes works
kubectl config get-contexts
kubectl config use-context <ctx>
kubectl config set-context --current --namespace=<ns>

# kubeadm
kubeadm init / join / reset
kubeadm token list | create --print-join-command
kubeadm certs check-expiration
kubeadm certs renew all
kubeadm upgrade plan | apply <v> | node
kubeadm config print init-defaults
kubeadm config images list

# Node maintenance
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>

# RBAC
kubectl create role|clusterrole <name> --verb=... --resource=...
kubectl create rolebinding|clusterrolebinding <name> --role|--clusterrole=... --user|--group|--serviceaccount=...
kubectl auth can-i <verb> <resource> --as <user> -n <ns>

# Explain fields quickly (docs inside the terminal)
kubectl explain pod.spec.containers --recursive

Exam tips:

• Set up aliases early: alias k=kubectl, export do="--dry-run=client -o yaml", and enable completion.
• Always check which context a question targets and switch to it first.
• Use kubectl ... $do > file.yaml to scaffold manifests, then edit.