Troubleshooting (30%)

Notes on diagnosing and fixing cluster, node, and application failures. The single heaviest-weighted domain on the exam.

Topics

• Evaluate cluster and node logging
• Understand how to monitor applications
• Manage container stdout & stderr logs
• Troubleshoot application failure
• Troubleshoot cluster component failure
• Troubleshoot networking

Directory map

• Logging
• Monitoring

General method

1. Scope it: one Pod, one node, one namespace, or the whole cluster?
2. Read state: kubectl get, then kubectl describe (the Events section usually names the cause).
3. Read logs: container logs, then component/node logs.
4. Form a hypothesis, change one thing, and verify.
5. Work from the outside in: Pod, Service, Ingress, then node, kubelet, runtime, control plane.

If kubectl itself fails, the problem is the API server, its kubeconfig, or the network to it. Fall back to the node: crictl, journalctl, and the static Pod manifests.

Cluster and node logging

Where logs live:

journalctl -u kubelet -f
journalctl -u kubelet --since "10 min ago" --no-pager | tail -100
journalctl -u kubelet | grep -i -E "error|failed"
systemctl status kubelet containerd
dmesg | tail

Kubernetes does not ship a cluster-wide logging backend. The standard pattern is a node-level agent (Fluent Bit, Fluentd, Vector, Promtail) running as a DaemonSet that ships /var/log contents to a store such as Elasticsearch, Loki, or a cloud service. Node log rotation is handled by the kubelet (containerLogMaxSize, containerLogMaxFiles).

See Logging.

Monitoring applications

See Monitoring. Summary:

• metrics-server provides the resource metrics API for kubectl top and HPA. It is in-memory only, not a monitoring system.
• Full monitoring is third party (Prometheus plus Grafana, Datadog, and so on).
• Probes (liveness, readiness, startup) are the in-cluster health signals.

kubectl top node
kubectl top pod -A --sort-by=memory
kubectl top pod --containers
kubectl get events -A --sort-by=.lastTimestamp
kubectl get --raw /readyz?verbose
kubectl get --raw /healthz

If kubectl top fails with “Metrics API not available”, metrics-server is missing or unhealthy (on lab clusters it often needs --kubelet-insecure-tls).

Container stdout and stderr logs

The container runtime captures anything a container writes to stdout and stderr; kubectl logs reads that. Logs written to files inside the container are not captured, so use a sidecar to tail them to stdout (see Sidecars).

kubectl logs <pod>
kubectl logs <pod> -c <container>        # multi-container Pod
kubectl logs <pod> --all-containers
kubectl logs <pod> --previous            # last crashed instance (CrashLoopBackOff)
kubectl logs <pod> -f --tail=50
kubectl logs <pod> --since=15m
kubectl logs <pod> --timestamps
kubectl logs -l app=web --prefix --max-log-requests=10     # by label
kubectl logs deployment/web
kubectl logs <pod> -n <ns>

Logs of a Pod that was deleted are gone from kubectl; check the node under /var/log/pods if it still exists. Static control-plane Pods without a working API: crictl ps -a then crictl logs <id>.

Troubleshooting application failure

Start with kubectl get pods and read the STATUS column.

kubectl get pods -o wide
kubectl describe pod <pod>
kubectl logs <pod> [--previous]
kubectl get events --field-selector involvedObject.name=<pod>
kubectl get pod <pod> -o yaml | less
kubectl exec -it <pod> -- sh
kubectl debug -it <pod> --image=busybox:1.36 --target=<container>      # ephemeral container
kubectl debug node/<node> -it --image=busybox:1.36                     # node shell (host fs at /host)
kubectl rollout status deployment/<d>
kubectl rollout undo deployment/<d>
kubectl get deploy,rs,pods -l app=<label>

For Deployments: if a rollout stalls, rollout status, then inspect the new ReplicaSet’s Pods. A wrong selector or labels mismatch means no Pods are managed or no Service endpoints exist.

Troubleshooting cluster component failure

Node NotReady

kubectl get nodes
kubectl describe node <node>       # Conditions: Ready, MemoryPressure, DiskPressure, PIDPressure; Events

Then on the node:

systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager
systemctl status containerd
crictl ps -a
df -h; free -m
swapon --show                       # swap on makes kubelet fail by default

Common causes and fixes:

• kubelet stopped or crashing: systemctl enable --now kubelet; read its journal.
• Bad kubelet config: check /var/lib/kubelet/config.yaml and /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; the kubelet’s kubeconfig /etc/kubernetes/kubelet.conf (wrong server address or expired certificate).
• Container runtime down or wrong CRI socket path.
• CNI not installed or broken: node NotReady with “network plugin not ready”.
• Expired certificates: kubeadm certs check-expiration then kubeadm certs renew all and restart the control-plane static Pods.
• Disk, memory, or PID pressure: free space, evict, or resize.

Control-plane failure

Symptoms: kubectl times out or refuses connection (apiserver/etcd), Pods never get scheduled (scheduler), Deployments do not create Pods or nodes are not marked NotReady (controller-manager).

kubectl -n kube-system get pods
ls /etc/kubernetes/manifests
crictl ps -a | grep -E "kube-apiserver|etcd|scheduler|controller"
crictl logs <container-id>
journalctl -u kubelet | grep -i -E "static|manifest|apiserver"
cat /etc/kubernetes/manifests/kube-apiserver.yaml

Checklist per component:

• Manifest typos: a wrong flag, wrong image, wrong path in a hostPath volume, or YAML error in /etc/kubernetes/manifests/*.yaml makes the static Pod fail. Compare with the other manifests and fix the file; the kubelet re-creates the Pod within seconds.
• kube-apiserver: check --etcd-servers URL and cert paths, --client-ca-file, port 6443, and etcd health.
• etcd: data dir, cert paths, peer/client URLs, disk space; etcdctl endpoint health with the cert flags.
• kube-scheduler: --kubeconfig path (/etc/kubernetes/scheduler.conf), leader election; Pods stay Pending with no events when it is down.
• kube-controller-manager: --kubeconfig, --cluster-signing-* file paths, --root-ca-file.
• kube-proxy: a DaemonSet; check its ConfigMap and logs if Services do not work.
• kubeconfig problems: kubectl config view, correct server address, KUBECONFIG env var, expired client certs.

If a component is a systemd service (non-kubeadm clusters): systemctl status kube-apiserver, journalctl -u kube-apiserver.

Troubleshooting networking

Work in layers:

1. Pod networking: do Pods get IPs? kubectl get pods -o wide. ContainerCreating with FailedCreatePodSandBox means a CNI issue. Check the CNI Pods in kube-system, /etc/cni/net.d, /opt/cni/bin, and that the Pod CIDR matches the CNI config.
2. Pod to Pod: kubectl exec and curl/ping the other Pod IP. Failing across nodes only suggests CNI overlay or firewall problem between nodes (VXLAN UDP 4789, IP-in-IP protocol 4, BGP 179).
3. DNS: from a Pod, nslookup kubernetes.default. Check CoreDNS Pods and logs, the kube-dns Service and endpoints, and /etc/resolv.conf in the Pod. Check that NetworkPolicies are not blocking port 53.
4. Service: kubectl get endpoints <svc>. Empty endpoints means selector or readiness problem. Wrong targetPort. Test the ClusterIP directly, then the DNS name. Check kube-proxy Pods/logs and iptables/IPVS rules.
5. NetworkPolicy: kubectl get netpol -n <ns>; temporarily review for default-deny rules; remember policies are additive and AND/OR selector semantics.
6. Ingress / external: Ingress controller Pods running, Ingress class and host/path correct, backend Service has endpoints, controller Service has an external address, TLS Secret exists.
7. Node: ip route, iptables-save | grep <svc>, sysctl net.ipv4.ip_forward, firewall rules.

kubectl run net --rm -it --image=nicolaka/netshoot --restart=Never -- bash
# inside: dig web.default.svc.cluster.local ; curl -v http://web ; traceroute <ip> ; nc -zv <host> <port>

kubectl get svc,endpoints -n <ns>
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-proxy
kubectl -n kube-system logs -l k8s-app=kube-proxy
kubectl get ingress,netpol -A

Cheat sheet

# Fast triage
kubectl get nodes
kubectl get pods -A | grep -v Running
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl describe <kind> <name>
kubectl logs <pod> [-c c] [--previous]
kubectl top node; kubectl top pod

# Node side
systemctl status|restart kubelet containerd
journalctl -u kubelet -f
crictl ps -a ; crictl logs <id> ; crictl images ; crictl pods
ls /etc/kubernetes/manifests
kubeadm certs check-expiration

# Output shaping
kubectl get pod <p> -o jsonpath='{.status.containerStatuses[*].lastState}'
kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase
kubectl get pods --field-selector=status.phase!=Running -A
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[-1].type}{"\n"}{end}'

Exam tips:

• Read the task’s namespace and context first.
• describe before anything else; the Events at the bottom answer most questions.
• For broken control-plane components fix the static Pod manifest, then confirm with kubectl get pods -n kube-system.
• Use crictl when kubectl is unavailable; sudo -i for node work and exit back out.