Monitoring, Logging and Runtime Security (20%)

falco

• Falco is an IDS for Kubernetes workloads
• falco is a cloud native security tool. It provides near real-time threat detection for cloud, container, and Kubernetes workloads by leveraging runtime insights. Falco can monitor events defined via customizable rules from various sources, including the Linux kernel, and enrich them with metadata from the Kubernetes API server, container runtime, and more. Falco supports a wide range of kernel versions, x86_64 and ARM64 architectures, and many different output channels.
• falco uses sydig filters to extract information about an event. They are configured in the falco rules.yaml or configmap. They can also be passed via helm values.
  • /etc/falco/falco.yaml - the main configuration file for falco
  • /etc/falco/falco_rules.yaml - the main rules file for falco
• falco rule files consist of 3 elements defined in YAML:
  • rules - a rule is a condition under which an alert should be generated
  • macros - a macro is a reusable rule condition. These help keep the rules file clean and easy to read
  • lists - a collection of items that can be used in rules and macros
• Some examples of events that falco watches for:
  • Reading or writing files at a specific location in the filesystem
  • Opening a shell binary for a container, such as /bin/bash
  • Sending/receives traffic from undesired URLs
• Falco deploys a set of sensors that listen for configured events and conditions
  • Each sensor contains a set of rules that map an event to a data source.
  • An alert is produced when a rule matches a specific event
  • Alerts are then sent to an output channel to record the event

Ensuring Container Immutability

• Containers should be immutable. This means that once a container is created, it should not be changed. If changes are needed, a new container should be created.
• Containers are mutable (changeable) by default. This can lead to security vulnerabilities.
• To ensure container immutability:
  • Use a ‘distroless’ container image. These images are minimal and contain only the necessary components to run an application. They do not include a shell.
  • Use a ‘read-only’ file system. This prevents changes to the file system. To configure a read-only file system, add the following to the pod spec:

    spec:
      containers:
      - name: my-container
        image: my-image
        securityContext:
          readOnlyRootFilesystem: true
    

Audit Logs

• Auditing involves recording and tracking all events and actions within the cluster
• Who made a change, when was it changed, and what exactly was changed
• Audit logs provide a chronological record of activities within a cluster
• Entries in the audit log exist in ‘JSON Lines’ format. Note that this is not the same as JSON. Each line in the log is a separate JSON object.
• Types of Audit Policies:
  • None - no logging
  • Metadata - Logs request metadata, but not request or response body
  • Request - Logs request metadata and request body, but no response body
  • Request/Response - Logs the metadata, request body, and response body

Sample Audit Policy

```
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
```

• Once the audit policy has been defined, you can apply it to the cluster by passing the --audit-policy-file flag to the kube-apiserver
• To use a file-based log backend, you need to pass 3 configurations to the kube-apiserver:
  • --audit-policy-file - this is the path to the audit policy file
  • --audit-log-path - this is the path to the audit log file
  • both of these paths needs to be mounted in the kube-apiserver. The kube-apiserver cannot read these files on the node without a proper volumeMount