Kubernetes Security with Tetragon: Lab to Detect Container Escapes

Introduction

In modern Kubernetes environments, gaining visibility into system behavior is critical for security. Tetragon, an open-source tool by the Cilium team, uses eBPF (a Linux kernel technology) to provide real-time observability and runtime enforcement for security events. Unlike traditional tools, Tetragon operates at the kernel level, offering deeper insights with minimal performance overhead.

What Makes Tetragon Unique?

  • eBPF-Powered: Directly hooks into the Linux kernel for detailed event monitoring.
  • No Dependency on Cilium: Can work independently of the Cilium networking stack.
  • Real-Time Security Insights: Detects and monitors system calls, process events, and network activity in real time.
  • Actionable Outputs: Converts raw events into meaningful security signals.

Scenarios Covered in This Workshop

  • Detecting Suspicious Networking Behavior:

    • Learn how to monitor TCP connection events like tcp_connect and tcp_close to identify abnormal network activity that may indicate malicious behavior.

  • Tracking Namespace Changes for Privilege Escalation:

    • Understand how to detect sys_setns calls to monitor namespace transitions, which are often used in container escape attempts and privilege escalation attacks.

Lab Setup

This lab includes:

  1. Installing Tetragon: Set up Tetragon on a Kubernetes cluster.
  2. Tracing Network Connections: Monitor TCP events.
  3. Simulating a Container Escape: Detect namespace changes and privilege escalation.
  4. Analyzing Security Events: Observe malicious activity and understand how Tetragon detects.
  • Ensure your cluster is running.
kubectl get nodes
  • Navigate to the EKS Directory:
cd /workspaces/ecr_eks_security_masterclass_public/eks/
  • Create tetragon.yaml file is a configuration file used for managing Tetragon.
cat << EOF > tetragon.yaml
tetragon:
  btf: /sys/kernel/btf/vmlinux
  enableCiliumAPI: false
  exportAllowList: ""
  exportDenyList: ""
  exportFilename: "tetragon.log"
  enableProcessCred: true
  enableProcessNs: true
tetragonOperator:
  enabled: true
EOF
  • Install Tetragon using Helm.
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install tetragon-observer cilium/tetragon \
  --namespace kube-system -f tetragon.yaml --version 1.1.0
  • Wait for the deployment to complete:
kubectl rollout status -n kube-system ds/tetragon-observer
  • Before applying, enable debugfs or ftrace tracing functionality.

This is common in environments like Azure VMs or managed nodes, where debugging tools may be restricted.

sudo mount -t debugfs none /sys/kernel/debug
mount | grep debugfs
sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep tcp_
  • Create a policy to monitor TCP connections. Save the following to network-trace.yaml.

For observing tcp_connect, tcp_close, and kernel functions to track the opening and closing of TCP connections for detecting suspicious network behavior or attacks.

cat << EOF > network-trace.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: network-trace
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false  # Kernel tracepoints for tcp_connect do not use syscalls
  - call: "tcp_close"
    syscall: false  # Similar to above, trace the kernel function
EOF

The provided network-trace.yaml file
This file is designed to capture comprehensive network tracing information using Cilium's TracingPolicy. The configuration includes both syscall: true and syscall: false for tcp_connect and tcp_close calls.

Purpose of Each Setting
Syscall: True

  • High-Level System Events: Tracks system calls directly at the kernel level.
  • Visibility: Provides insight into high-level system events, such as the initiation (tcp_connect) or termination (tcp_close) of TCP connections.

Syscall: False

  • Lower-Level Events: Traces lower-level events to extract more granular details.
  • Additional Arguments: Allows for the specification of additional arguments, such as type: "sock", to monitor the underlying socket behavior.

Why Both Are Used

  • Comprehensive Coverage: Including both syscall: true and syscall: false ensures comprehensive coverage of network activities.
  • Overview and Details: Syscall: true provides an overview of high-level events, while syscall: false offers detailed insights into the context of those events.
  • Apply the policy:
kubectl apply -f network-trace.yaml
  • Run the following to start monitoring in the new terminal:
kubectl exec -n kube-system -ti daemonset/tetragon-observer -c tetragon -- tetra getevents -o compact | grep escape-simulator
  • Create a file superpod.yaml to deploy pod, add wait time to make sure tetragon daemonset is propgated with policy.
cat << EOF > superpod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: escape-simulator
spec:
  containers:
  - name: attack-container
    image: nginx
    securityContext:
      privileged: true
    command: ["/bin/bash", "-c", "sleep infinity"]
  hostPID: true
  hostNetwork: true
EOF

sleep 15
  • Apply it.
kubectl apply -f superpod.yaml

In case unable to get the results, delete the pod via kubectl delete -f superpod.yaml & then re-apply the pod via kubectl apply -f superpod.yaml.

  • Wait for it to run and then run the curl command:
kubectl exec -ti escape-simulator -- curl google.com

  • The connect event is likely being traced because:

    • The kernel function tcp_connect is triggered internally by the curl command when establishing a network connection.
    • The TracingPolicy has successfully attached to the tcp_connect kprobe.

  • Create a file namespace-trace.yaml for namespace tracing policy.

To observe sys_setns and kernel functions to track namespace changes, enabling the detection of container escapes or unauthorized namespace transitions for enhanced security monitoring.

cat << EOF > namespace-trace.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: namespace-trace
spec:
  kprobes:
  - call: "sys_setns"
    syscall: true
EOF
  • Apply the policy
kubectl apply -f namespace-trace.yaml
  • Perform a container escape & get shell access to the pod.
kubectl exec -it escape-simulator -- /bin/bash

Enter Host Namespace

  • Once inside the pod, run:
nsenter --target 1 --all /bin/bash

This command enters the host namespaces, simulating a container escape.

  • Analyze Security Events, by going back to the terminal where Tetragon is observing events. FInally see logs like:
PROCESS_EXECUTE: /usr/bin/nsenter --target 1 --all /bin/bash
SYSCALL_SETNS: Process switched to new namespace
PROCESS_EXECUTE: /bin/bash

These logs confirm a namespace change and process execution, indicating a container escape attempt.

Cleanup

  • Remove the resources:
kubectl delete -f privileged-pod.yaml
kubectl delete -f network-trace.yaml
kubectl delete -f namespace-trace.yaml
helm uninstall tetragon-observer -n kube-system

Key Takeaways

  • Tetragon leverages eBPF for real-time kernel-level monitoring.
  • It can detect container escapes, privilege escalations, and other security events.
  • TracingPolicies make it flexible to monitor specific system activities.

Credits