Minimize Microservice Vulnerabilities (20%)

Pod Security Admission

Replaced Pod Security Policies
Pod Security Admission controller enforces pod security standards on pods
All you need to do to opt into the PSA feature is to add a label with a specific format to a namespace. All pods in that namespace will have to follow the standards declared.
- The label consists of three parts: a prefix, a mode, and a level
- Example: pod-security.kubernetes.io/restricted=privileged
- Prefix: pod-security.kubernetes.io
- Mode: enforce, audit, or warn
  - Enforce: blocks pods that do not meet the PSS
  - Audit: logs violations to the audit log but does not block pod creation
  - Warn: logs violations on the console but does not block pod created
- Level: privileged, baseline, or restricted
  - Privileged: fully unrestricted
    - Allowed: everything
  - Baseline: some restrictions
    - Allowed: most things, except sharing host namespaces, hostPath volumes and hostPorts, and privileged pods
  - Restricted: most restrictions
    - Allowed: very few things, like running as root, using host networking, hostPath volumes, hostPorts, and privileged pods. The pod must be configured with a seccomp profile.

Security Contexts

Security contexts are used to control the security settings of a pod or container
Security contexts can be defined at the pod level or the container level. Settings defined at the container level will override identical settings defined at the pod level
Security contexts can be used to:
- Run a pod as a specific user
- Run a pod as a specific group
- Run a pod with specific Linux capabilities
- Run a pod with a read-only root filesystem
- Run a pod with a specific SELinux context
- Run a pod with a specific AppArmor profile

You can view the capabilities of a process by viewing the status file of the process and grepping for capabilities:

rtn@worker02:~$ cat /proc/self/status |grep -i cap
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000

These values are encoded in hexadecimal. To decode them, use the capsh command: rtn@worker02:~$ sudo capsh --decode=000001ffffffffff 0x000001ffffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore

Admission Controllers

Admission Controllers are used for automation within a cluster
Once a request to the KubeAPI server has been authenticated and then authorized, it is intercepted and handled by any applicable Admission Controllers
Example Admission Controllers:
- ImagePolicyWebook
  - You may see this one on the exam.
  - When enabled, the ImagePolicyWebook admission controller contacts an external service (that you or someone else wrote in whatever language you want, it just needs to accept and respond to HTTP requests).
  - To enable, add ‘ImagePolicyWebook’ to the ‘–enable-admission-plugins’ flag of the kube-api server
  - You must also supply an AdmissionControlFileFile file, which is a kubeconfig formatted file. Then pass the path to this config to the kube-api server with the --admission-control-config-file=<path to config file>. Note that this path is the path inside the kube-api container, so you must mount this path on the host to the pod as a hostPath mount.
- AlwaysPullImages
- DefaultStorageClass
- EventRateLimit
- NamespaceExists
- … and many more
Admission Controllers help make Kubernetes modular
To see which Admission Controllers are enabled:
- you can either grep the kubeapi process: ps aux |grep -i kube-api | grep -i admission
- or you can look at the manifest for the KubeAPI server (if the cluster was provisioned with KubeADM) grep admission -A10 /etc/kubernetes/manifests/kube-apiserver.yaml
- or if the cluster was provisioned manually you can look at the systemd unit file for the kube-api server daemon
There are two types of admission controllers:
- Mutating - can make changes to ‘autocorrect’
- Validating - only validates configuration
- Mutating are invoked first. Validating second.
The admission controller runs as a webhook server. It can run inside the cluster as a pod or outside the cluster on another server.
Some admission-controllers required a configuration file to be passed to the kube-api server. This file is passed using the --admission-control-config-file flag.

Open Policy Agent

OPA can be used for authorization. However, it is more likely to be used in the admission control phase.
OPA can be deployed as a daemonset on a node or as a pod
OPA policies use a language called rego

OPA in Kubernetes

GateKeeper

Gatekeeper Constraint Framework
- Gatekeeper is a validating and mutating webhook that enforces CRD-based policies executed by Open Policy Agent, a policy engine for Cloud Native environments hosted by CNCF as a graduated project.
- The framework that helps us implement what, where, and how we want to do something in Kubernetes
  - Example:
    - What: Add labels, etc.
    - Where: kube-system namespace
    - How: When a pod is created
To run Gatekeeper in Kubernetes, simply apply the manifests provided by OPA
The pods and other resources are created in the gatekeeper-system namespace

Constraint Templates

Before you can define a constraint, you must first define a ConstraintTemplate, which describes both the Rego that enforces the constraint and the schema of the constraint. The schema of the constraint allows an admin to fine-tune the behavior of a constraint, much like arguments to a function.

Here is an example constraint template that requires all labels described by the constraint to be present:

```
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }
```

Constraints
- Constraints are then used to inform Gatekeeper that the admin wants a ConstraintTemplate to be enforced, and how. This constraint uses the K8sRequiredLabels constraint template above to make sure the gatekeeper label is defined on all namespaces:
```
  apiVersion: constraints.gatekeeper.sh/v1beta1
  kind: K8sRequiredLabels
  metadata:
    name: ns-must-have-gk
  spec:
    match:
      kinds:
        - apiGroups: [""]
          kinds: ["Namespace"]
    parameters:
      labels: ["gatekeeper"]
```
- The match field supports multiple options: https://open-policy-agent.github.io/gatekeeper/website/docs/howto#the-match-field
After creating the constraint from the constrainttemplate, you can view all violations by describing the constraint:
- Example: kubectl describe k8srequiredlabels ns-must-have-gk

Kubernetes Secrets

Secrets are used to store sensitive information in Kubernetes
base64 encoded when stored in etcd
Can be injected into a pod as an env or mounted as a volume

Encrypting etcd

By default, the API server stores plain-text representations of resources into etcd, with no at-rest encryption.
The kube-apiserver process accepts an argument –encryption-provider-config that specifies a path to a configuration file. The contents of that file, if you specify one, control how Kubernetes API data is encrypted in etcd.
If you are running the kube-apiserver without the –encryption-provider-config command line argument, you do not have encryption at rest enabled. If you are running the kube-apiserver with the –encryption-provider-config command line argument, and the file that it references specifies the identity provider as the first encryption provider in the list, then you do not have at-rest encryption enabled (the default identity provider does not provide any confidentiality protection.)
If you are running the kube-apiserver with the –encryption-provider-config command line argument, and the file that it references specifies a provider other than identity as the first encryption provider in the list, then you already have at-rest encryption enabled. However, that check does not tell you whether a previous migration to encrypted storage has succeeded.

Example EncryptionConfiguration:

  apiVersion: apiserver.config.k8s.io/v1
  kind: EncryptionConfiguration
  resources:
    - resources:
        - secrets
        - configmaps
        - pandas.awesome.bears.example # a custom resource API
      providers:
        # This configuration does not provide data confidentiality. The first
        # configured provider is specifying the "identity" mechanism, which
        # stores resources as plain text.
        #
        - identity: {} # plain text, in other words NO encryption
        - aesgcm:
            keys:
              - name: key1
                secret: c2VjcmV0IGlzIHNlY3VyZQ==
              - name: key2
                secret: dGhpcyBpcyBwYXNzd29yZA==
        - aescbc:
            keys:
              - name: key1
                secret: c2VjcmV0IGlzIHNlY3VyZQ==
              - name: key2
                secret: dGhpcyBpcyBwYXNzd29yZA==
        - secretbox:
            keys:
              - name: key1
                secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
    - resources:
        - events
      providers:
        - identity: {} # do not encrypt Events even though *.* is specified below
    - resources:
        - '*.apps' # wildcard match requires Kubernetes 1.27 or later
      providers:
        - aescbc:
            keys:
            - name: key2
              secret: c2VjcmV0IGlzIHNlY3VyZSwgb3IgaXMgaXQ/Cg==
    - resources:
        - '*.*' # wildcard match requires Kubernetes 1.27 or later
      providers:
        - aescbc:
            keys:
            - name: key3
              secret: c2VjcmV0IGlzIHNlY3VyZSwgSSB0aGluaw==

Each resources array item is a separate config and contains a complete configuration. The resources.resources field is an array of Kubernetes resource names (resource or resource.group) that should be encrypted like Secrets, ConfigMaps, or other resources.
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
After enabling encryption in etcd, any secrets that you created prior to enabling encryption will not be encrypted. You can encrypt them by running:

kubectl get secrets -A -o yaml | kubectl replace -f -

Example of getting a secret in etcd:

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key get /registry/secrets/three/con1

The path to the resource in the etcd database is ‘/registry///’

Container Sandboxing

Containers are not contained!
A container sandbox is a mechanism that provides an additional layer of isolation between the container and the host
Container sandboxing is implemented via Runtime Class objects in Kubernetes.
The default container runtime is runc. However, we can change this to use runsc (gvisor) or Kata
Sandboxing prevents the dirty cow exploit, which allows a user to gain root access to the host
- Dirty COW works by exploiting a race condition in the Linux kernel

gVisor

gVisor is a kernel written in Golang that intercepts system calls made by a container
gVisor is like a ‘syscall proxy’ that sits between the container and the kernel
- components
  - sentry -
  - gofer -
Not all apps will work with gVisor
gVisor will cause performance degradation in your app due to the additional time taken
gVisor uses runsc as the runtime handler

Kata Containers

Kata inserts each container into it’s own virtual machine, giving each it’s own kernel
Kata containers require nested virtualisation support, so it may not work with all cloud providers

RuntimeClass

RuntimeClass is a new feature in Kubernetes that allows you to specify which runtime to use for a pod

To use a runtime class

Create a new runtimeclass object:

apiVersion: node.k8s.io/v1
handler: runsc
kind: RuntimeClass
metadata:
  name: secure-runtime

Specify the runtimeClassName in the pod definition:

apiVersion: v1
kind: Pod
metadata:
    name: simple-webapp-1
    labels:
        name: simple-webapp
spec:
    runtimeClassName: secure-runtime
    containers:
    - name: simple-webapp
      image: kodekloud/webapp-delayed-start
      ports:
      - containerPort: 8080

Resource Quotas

Control requests and limits for CPU and memory within a namespace

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-resource-quota
  namespace: team-a
spec:
  hard:
    pods: "5"
    requests.cpu: "0.5"
    requests.memory: 500Mi
    limits.cpu: "1"
    limits.memory: 1Gi

apiVersion: v1
kind: ResourceQuota
metadata:
name: pods-medium
spec:
    hard:
      cpu: "10"
      memory: 20Gi
      pods: "10"
scopeSelector:
  matchExpressions:
  - operator : In
    scopeName: PriorityClass
    values: ["medium"]

API Priority and Fairness

https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
With API Priority and Fairness, you can define which resources need to be prioritized over others in regards to requests to the KubeAPI server

To configure API Priority and Fairness, you create a PriorityLevelConfiguration object:

  ? Is this still supported? Is it an exam topic? I cannot find the manifest spec.

Pod Priority and Preemption

With Pod Priority and Preemption, you can ensure that critical pods are running while the cluster is under resource contention by killing lower priority pods

To implement Pod Priority and Preemption:

Create a priorityClass object (or several):

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for XYZ service pods only."
---
    apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 100
globalDefault: false
description: "This priority class should be used for XYZ service pods only."

Assign the priorityClass to a pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: high-priority

Pod to Pod Encryption

mTLS can be used to encrypt traffic between pods
Methods of p2p encryption
- Service Mesh
  - Service Mesh can offload the encryption and decryption of traffic between pods by using a sidecar proxy
  - Examples:
    - Istio
      - Istio uses Envoy as a sidecar proxy
      - Istio uses a sidecar proxy to encrypt traffic between pods
    - Linkerd
- Wireguard
  - Cilium
    - uses eBPF for network security
    - Encrytion is transparent to the application
    - Provides flexible encryption options
- IPSec
  - Calico

Keyboard shortcuts

notebook