
## Falco Kubernetes Lab: Runtime Threat Detection with Prometheus & Grafana

🧰 Prerequisites

- Docker
- Kind
- kubectl
- Helm
- Make

## Setting Up the Lab

Clone the repo

```bash
git clone https://github.com/moabukar/falco-labs.git
cd falco-labs
```

Start the Lab

```bash
make up
```

This command executes the setup.sh up script, which performs the following:

- Creates a new Kind cluster with the configuration specified in kind.yaml. Called falco-lab.
- Creates the falco namespace and deploys a custom rules ConfigMap.
- Adds the Falco Helm repository and installs Falco with metrics enabled.
- Deploys an Nginx workload.
- Adds the Prometheus Community Helm repository and installs the kube-prometheus-stack.
- Creates ConfigMaps for the Falco dashboard and Grafana datasource.
- Upgrades the kube-prometheus-stack to load the dashboard and datasource.
- Port-forwards the Grafana service to localhost:3000.
- Waits for the Falco pods to be ready.

Setup/Bootsrap script

kind config

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /dev
      containerPath: /dev
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock
  extraPortMappings:
    - containerPort: 6443
      hostPort: 60000
      listenAddress: "127.0.0.1"
      protocol: tcp
```

```bash
#!/bin/bash
set -e

function delete_existing_cluster() {
  if kind get clusters | grep -q "^falco-lab$"; then
    echo "[INFO] Cluster 'falco-lab' already exists. Deleting it..."
    kind delete cluster --name falco-lab
  fi
}

if [ "$1" == "up" ]; then
  echo "[+] Creating kind cluster..."
  delete_existing_cluster
  kind create cluster --name falco-lab --config kind.yaml

  echo "[+] Creating namespace 'falco' and deploying custom rules ConfigMap..."
  kubectl create ns falco || true
  kubectl create configmap falco-custom-rules \
    --from-file=custom-rule.yaml=custom-rule.yaml \
    -n falco || true

  echo "[+] Adding Falco Helm repo..."
  helm repo add falcosecurity https://falcosecurity.github.io/charts
  helm repo update

  echo "[+] Installing Falco (with metrics enabled)..."
  helm install falco falcosecurity/falco \
    --namespace falco \
    -f values.yaml

  echo "[+] Deploying nginx workload..."
  kubectl create deployment nginx --image=nginx

  echo "[+] Installing Prometheus & Grafana..."
  helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  helm repo update
  helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace

  echo "[+] Waiting for Grafana pod to be Ready..."
  kubectl wait --for=condition=Ready --timeout=180s pods -l app.kubernetes.io/name=grafana -n monitoring

  echo "[+] Creating ConfigMap for Falco Dashboard..."
  kubectl create configmap falco-dashboard \
    --from-file=falco_dashboard.json=falco_dashboard.json \
    -n monitoring || true
  kubectl label configmap falco-dashboard -n monitoring grafana_dashboard=1 --overwrite

  echo "[+] Creating ConfigMap for Grafana datasource..."
  kubectl create configmap grafana-datasource \
    --from-file=datasource.yaml=grafana_datasource.yaml \
    -n monitoring || true
  kubectl label configmap grafana-datasource -n monitoring grafana_datasource=1 --overwrite

  echo "[+] Upgrading kube-prometheus-stack to load dashboard and datasource..."
  helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --reuse-values \
    --set grafana.sidecar.dashboards.enabled=true \
    --set grafana.sidecar.dashboards.label=grafana_dashboard \
    --set grafana.dashboardsConfigMaps.falco-dashboard="falco-dashboard" \
    --set grafana.sidecar.datasources.enabled=true \
    --set grafana.sidecar.datasources.label=grafana_datasource

  echo "[+] Forwarding Grafana service on port 3000..."
  kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80 &
  
  echo "[+] Waiting for Falco pods to be Ready..."
  kubectl wait --for=condition=Ready --timeout=180s pods -l app.kubernetes.io/name=falco -n falco

  echo "[+] Lab setup complete."
  echo "[+] Grafana is available at http://localhost:3000"
  echo "[+] To get the Grafana admin password, run:"
  echo "    kubectl -n monitoring get secrets kube-prometheus-stack-grafana -o jsonpath=\"{.data.admin-password}\" | base64 -d ; echo"
  echo "[+] To generate events, run: ./generate_events.sh"
  echo "[+] Tailing Falco logs..."
  kubectl logs -n falco -l app.kubernetes.io/name=falco -f

elif [ "$1" == "logs" ]; then
  echo "[+] Tailing Falco logs..."
  kubectl logs -n falco -l app.kubernetes.io/name=falco -f

elif [ "$1" == "down" ]; then
  echo "[+] Uninstalling Falco..."
  helm uninstall falco -n falco || true

  echo "[+] Uninstalling Prometheus & Grafana..."
  helm uninstall kube-prometheus-stack -n monitoring || true

  echo "[+] Deleting all kind clusters..."
  for cluster in $(kind get clusters); do
    echo "[+] Deleting cluster: $cluster"
    kind delete cluster --name "$cluster"
  done

  echo "[+] Cleanup complete."
else
  echo "Usage: $0 {up|logs|down}"
  exit 1
fi

# Manual test examples:
# kubectl run -it curl-test --image=alpine -- sh
# apk add curl
# curl http://example.com
```

## 🔍 Testing Falco Detection

Generate Events

```bash
#!/bin/bash
set -e

# Get the nginx pod name (assumes only one nginx pod)
POD=$(kubectl get pods -l app=nginx -o jsonpath='{.items[0].metadata.name}')

echo "[+] Generating event: reading /etc/shadow..."
kubectl exec -it "$POD" -- cat /etc/shadow || echo "[!] Failed to read /etc/shadow"

echo "[+] Generating event: writing to /etc/testfile..."
kubectl exec -it "$POD" -- sh -c "echo 'Falco Test' > /etc/testfile" || echo "[!] Write event failed"

echo "[+] Generating event: spawning a shell..."
kubectl exec -it "$POD" -- sh -c "sh -c 'echo Shell spawned'" || echo "[!] Shell spawn event failed"

echo "[+] Generating event: making a network connection (curl http://example.com)..."
kubectl exec -it "$POD" -- sh -c "apk add --no-cache curl && curl -s http://example.com" || echo "[!] Curl event failed"

echo "[+] Event generation complete."
```

```bash
./generate_events.sh
```

This script performs the following actions to trigger Falco rules:

- Reads the /etc/shadow file inside the Nginx pod.
- Writes to /etc/testfile inside the Nginx pod.
- Spawns a shell inside the Nginx pod.
- Makes a network connection using curl inside the Nginx pod.

Falco logs:

![Falco logs](/images/falco-logs.png)

## Create Falco Rules

here we have a custom rules file that we can use to test Falco.

Some rules like:

- Detecting curl in container
- Detecting shell in container
- Detecting write to /etc/sudoers
- Detecting write to /etc/shadow
- Shell spawned in container
- Privilege escalation via setuid binary
- Unexpected network connection from container

```yaml
- rule: Detect curl in container
  desc: Someone ran curl inside a container
  condition: container and proc.name = "curl"
  output: "⚠️ curl detected: user=%user.name command=%proc.cmdline container=%container.id"
  priority: WARNING
  tags: [network, curl, suspicious]

- rule: "Read sensitive file /etc/shadow"
  desc: "Detect any read access to /etc/shadow"
  condition: "evt.type in (open, openat, openat2) and fd.name = /etc/shadow"
  output: "Sensitive file /etc/shadow read (command=%proc.cmdline user=%user.name)"
  priority: WARNING
  tags: [filesystem, sensitive]

- rule: "Write to /etc directory"
  desc: "Detect write operations to any file under /etc"
  condition: "evt.type in (open, openat, openat2) and evt.is_open_write=true and fd.name startswith /etc"
  output: "File in /etc written (command=%proc.cmdline user=%user.name)"
  priority: WARNING
  tags: [filesystem, custom]

- rule: "Write to /etc/sudoers"
  desc: "Detect any write to /etc/sudoers"
  condition: "evt.type in (open, openat, openat2) and evt.is_open_write=true and fd.name = /etc/sudoers"
  output: "Suspicious write to /etc/sudoers (command=%proc.cmdline user=%user.name)"
  priority: CRITICAL
  tags: [privilege_escalation, custom]

- rule: "Shell spawned in container"
  desc: "Detect any shell spawned in a container"
  condition: "proc.name in (sh, bash, zsh) and container.id != host"
  output: "Shell spawned in container (command=%proc.cmdline, container=%container.id)"
  priority: NOTICE
  tags: [container, runtime]

- rule: "Privilege escalation via setuid binary"
  desc: "Detect execution of setuid binaries (e.g., sudo, passwd) in a container"
  condition: "proc.name in (sudo, passwd) and evt.type = execve and container.id != host"
  output: "Setuid binary execution detected (command=%proc.cmdline user=%user.name)"
  priority: CRITICAL
  tags: [privilege_escalation, container]

- rule: shell_in_container
  desc: notice shell activity within a container
  condition: >
    evt.type = execve and 
    evt.dir = < and 
    container.id != host and 
    (proc.name = bash or
     proc.name = ksh)    
  output: >
    shell in a container
    (user=%user.name container_id=%container.id container_name=%container.name 
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)    
  priority: WARNING

- rule: "Unexpected network connection from container"
  desc: "Detect network connection attempts from container processes"
  condition: "evt.type = connect and container.id != host"
  output: "Network connection from container detected (command=%proc.cmdline, connection=%fd.name)"
  priority: NOTICE
  tags: [network, container]
```

## View Falco Logs

```bash
make logs
```

This make alias tails the Falco logs, allowing you to observe the alerts generated by the events above.

## 📊 Accessing Prometheus & Grafana

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
```

Prometheus

Access Prometheus at: http://localhost:9090

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
```

Grafana

Access Grafana at: http://localhost:3000

Username: admin
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
Access Prometheus at: http://localhost:9090

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
```

Access Grafana at: http://localhost:3000

Username: admin

Password: prom-operator

The Falco dashboard should be automatically loaded, displaying metrics and alerts.

Grafana Dashboard

```json
{
    "annotations": {
      "list": []
    },
    "editable": true,
    "gnetId": null,
    "graphTooltip": 0,
    "id": null,
    "iteration": 1621469837171,
    "links": [],
    "panels": [
      {
        "datasource": "Prometheus",
        "fieldConfig": {
          "defaults": {},
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 24,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "legend": {
            "displayMode": "list",
            "placement": "bottom"
          },
          "tooltip": {
            "mode": "single"
          }
        },
        "targets": [
          {
            "expr": "sum(rate(falco_rules_alert_total[5m])) by (priority)",
            "interval": "",
            "legendFormat": "{{priority}}",
            "refId": "A"
          }
        ],
        "title": "Falco Alerts by Priority",
        "type": "timeseries"
      }
    ],
    "schemaVersion": 27,
    "style": "dark",
    "tags": ["falco"],
    "templating": {
      "list": []
    },
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "",
    "title": "Falco Dashboard",
    "uid": "falco-dashboard"
  }
```

## 🧹 Tearing Down the Lab

```bash
make down
```

This command executes the setup.sh down script, which performs the following:

Uninstalls Falco and the kube-prometheus-stack.

Deletes all Kind clusters.

## Conclusion

This lab provides a practical demonstration of integrating Falco into a Kubernetes environment for runtime security monitoring. By generating specific events, you can observe how Falco detects and alerts on suspicious activities, with Prometheus and Grafana providing visualization and analysis capabilities.


Falco Kubernetes Lab: Runtime Threat Detection with Prometheus & Grafana

Posts tagged with: grafana