Workloads & Scheduling (15%)

Notes on deploying and scaling applications, scheduling, self-healing, and configuration primitives for workloads.

Topics

• Understand deployments and how to perform rolling update and rollbacks
• Use ConfigMaps and Secrets to configure applications
• Know how to scale applications
• Understand the primitives used to create robust, self-healing, application deployments
• Understand how resource limits can affect Pod scheduling
• Awareness of manifest management and common templating tools

Directory map

• Autoscaling (VPA)
• Self-healing Applications
• Sidecar Containers

Deployments, rolling updates, and rollbacks

A Deployment manages ReplicaSets, which manage Pods. Changing the Pod template (image, env, resources) creates a new ReplicaSet and gradually shifts Pods to it. Scaling does not create a new ReplicaSet.

Update strategies:

• RollingUpdate (default): replaces Pods gradually. Controlled by maxSurge (extra Pods allowed above desired, default 25%) and maxUnavailable (Pods allowed to be down, default 25%).
• Recreate: kills all old Pods, then creates new ones. Causes downtime; used when two versions cannot coexist.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.25
        readinessProbe:
          httpGet: { path: /, port: 80 }

Readiness probes matter for rolling updates: a new Pod only counts as available once it is Ready, so a bad image stalls the rollout instead of taking the app down.

kubectl create deployment web --image=nginx:1.25 --replicas=3
kubectl set image deployment/web nginx=nginx:1.26 --record   # --record is deprecated; use annotations
kubectl rollout status deployment/web
kubectl rollout history deployment/web
kubectl rollout history deployment/web --revision=2
kubectl rollout undo deployment/web
kubectl rollout undo deployment/web --to-revision=1
kubectl rollout pause deployment/web      # batch several changes
kubectl rollout resume deployment/web
kubectl rollout restart deployment/web    # rolling restart, same spec

Other workload types:

ConfigMaps and Secrets

ConfigMap holds non-sensitive configuration. Secret holds sensitive data, base64 encoded (encoding, not encryption). Enable encryption at rest on the API server for real protection, and restrict access with RBAC.

kubectl create configmap app-config --from-literal=LOG_LEVEL=debug --from-file=app.properties
kubectl create secret generic db-creds --from-literal=username=admin --from-literal=password='S3cret!'
kubectl create secret docker-registry regcred --docker-server=... --docker-username=... --docker-password=...
kubectl create secret tls my-tls --cert=tls.crt --key=tls.key

Consuming them:

spec:
  containers:
  - name: app
    image: myapp
    envFrom:                       # all keys as env vars
    - configMapRef:
        name: app-config
    env:                           # single key
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-creds
          key: password
    volumeMounts:
    - name: config
      mountPath: /etc/config
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: app-config

Key behaviors:

• Values injected as environment variables are read at container start and do not update.
• Values mounted as volumes update eventually (kubelet sync period), except when using subPath, which never updates.
• immutable: true prevents changes and reduces API server load.
• Decode a secret: kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d.

Scaling applications

Manual:

kubectl scale deployment web --replicas=5
kubectl scale statefulset db --replicas=3

HPA (Horizontal Pod Autoscaler) changes replica count based on metrics. Requires metrics-server and CPU/memory requests on the Pods (utilization is a percent of the request).

kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10
kubectl get hpa

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 70 }

Desired replicas = ceil(current * currentMetric / targetMetric). VPA adjusts requests and limits instead (see Autoscaling). The Cluster Autoscaler adds or removes nodes. Do not run HPA and VPA on the same metric.

Self-healing primitives

See also Self-healing Applications.

• Controllers: ReplicaSets/Deployments recreate Pods that die or are deleted; the Node controller reschedules from failed nodes.
• restartPolicy: Always (default), OnFailure, Never. The kubelet restarts containers with exponential backoff (CrashLoopBackOff).
• Probes:
  • livenessProbe: failing restarts the container.
  • readinessProbe: failing removes the Pod from Service endpoints, no restart.
  • startupProbe: holds off the other probes for slow-starting apps.
• PodDisruptionBudget: limits voluntary disruptions (drain, upgrades) so a minimum number of Pods stays available.
• Multiple replicas plus anti-affinity or topology spread to survive node and zone failures.

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  tcpSocket: { port: 8080 }
  periodSeconds: 5

kubectl create poddisruptionbudget web-pdb --selector=app=web --min-available=2

Resource requests, limits, and scheduling

• Requests: what the scheduler reserves. A Pod is only placed on a node whose allocatable minus already requested resources can fit the Pod’s total requests. Limits are not used for scheduling.
• Limits: enforced at runtime. Exceeding a CPU limit means throttling; exceeding a memory limit means OOMKilled.
• If only limits are set, requests default to the limits.
• QoS classes: Guaranteed (requests equal limits for all containers), Burstable (some requests set), BestEffort (none). Under node pressure, BestEffort Pods are evicted first.
• A Pod requesting more than any node can offer stays Pending with an event like Insufficient cpu.
• LimitRange sets defaults and min/max per container in a namespace. ResourceQuota caps the total consumption per namespace.

resources:
  requests: { cpu: 250m, memory: 128Mi }
  limits:   { cpu: 500m, memory: 256Mi }

Other scheduling controls:

spec:
  nodeSelector:
    disktype: ssd
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - { key: topology.kubernetes.io/zone, operator: In, values: [us-east-1a] }
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels: { app: web }
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: web }

• Taints repel Pods from nodes; tolerations let Pods ignore them. Effects: NoSchedule, PreferNoSchedule, NoExecute (also evicts running Pods).
• nodeName bypasses the scheduler entirely.
• Static Pods: manifests in /etc/kubernetes/manifests/ (path from staticPodPath in the kubelet config). Mirror Pods appear in the API with the node name suffix.
• Priority: PriorityClass lets important Pods preempt lower-priority ones.

kubectl taint nodes node1 dedicated=gpu:NoSchedule
kubectl taint nodes node1 dedicated=gpu:NoSchedule-     # remove
kubectl label nodes node1 disktype=ssd
kubectl describe node node1 | grep -A8 "Allocated resources"

Manifest management and templating

• Plain YAML with kubectl apply -f (declarative) versus imperative commands. Prefer apply for repeatable state; use --dry-run=client -o yaml to generate scaffolding.
• Kustomize (built into kubectl): overlay-based, no templating language. A kustomization.yaml lists resources and patches; overlays customize a base per environment.
• Helm: package manager with Go-templated charts, values files, releases, and rollback.
• Others to be aware of: Jsonnet, ytt, cdk8s, and GitOps tools (Argo CD, Flux) that sync manifests from git.

# Kustomize
kubectl kustomize ./overlays/prod
kubectl apply -k ./overlays/prod

# Helm
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm search repo nginx
helm install my-web bitnami/nginx -n web --create-namespace -f values.yaml
helm upgrade my-web bitnami/nginx -f values.yaml
helm list -A
helm history my-web
helm rollback my-web 1
helm uninstall my-web
helm template my-web bitnami/nginx     # render locally

Minimal kustomization:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namePrefix: prod-
patches:
- path: replicas.yaml
images:
- name: nginx
  newTag: "1.26"

Cheat sheet

# Generate manifests
kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml
kubectl create deployment web --image=nginx --replicas=3 --dry-run=client -o yaml > deploy.yaml
kubectl create job pi --image=perl -- perl -Mbignum=bpi -wle 'print bpi(2000)'
kubectl create cronjob hello --image=busybox --schedule="*/1 * * * *" -- echo hi

# Edit live
kubectl edit deployment web
kubectl set resources deployment web --limits=cpu=500m,memory=256Mi --requests=cpu=250m,memory=128Mi
kubectl set env deployment web LOG_LEVEL=debug
kubectl label pod web-1 tier=frontend --overwrite
kubectl annotate deployment web kubernetes.io/change-cause="bump to 1.26"

# Inspect
kubectl get deploy,rs,pods -l app=web
kubectl describe pod <pod>
kubectl get pods -o wide --show-labels
kubectl top pod / kubectl top node           # needs metrics-server
kubectl get events --sort-by=.lastTimestamp