
## 🎡 Introduction

In this blog, I’ll walk you through setting up a full-featured **Apache Pulsar playground** using **kind** (Kubernetes in Docker). Whether you’re testing Pulsar for learning or demoing a real pub/sub model with admin tools and monitoring, this setup gives you everything:

- ✅ Pulsar cluster deployed via Helm
- ✅ Prometheus + Grafana for monitoring
- ✅ Pulsar Manager UI
- ✅ Python producer/consumer using `pulsar-client`
- ✅ Proxy + dashboard exposure
- ✅ Hands-on with Pulsar admin CLI

---

## 🧰 Prerequisites

- [Docker](https://www.docker.com/)
- [kind](https://kind.sigs.k8s.io/)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [Helm](https://helm.sh/docs/intro/install/)
- [Python 3](https://www.python.org/) and `pip`

---

## 🔧 Step 1: Create a kind Cluster with Port Mappings

Create a local cluster that maps the UI and proxy ports:

```bash
cat <<EOF | kind create cluster --name pulsar --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 8080  # Pulsar REST Broker
    hostPort: 8080
  - containerPort: 6650  # Pulsar Binary Protocol
    hostPort: 6650
  - containerPort: 7750  # Pulsar Manager UI
    hostPort: 7750
  - containerPort: 3000  # Grafana
    hostPort: 3000
EOF
```

### Install Pulsar with Helm

```bash
helm repo add apache https://pulsar.apache.org/charts
helm repo update

git clone https://github.com/apache/pulsar-helm-chart
cd pulsar-helm-chart

./scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar-mini -c

helm install \
  --values examples/values-minikube.yaml \
  --namespace pulsar \
  pulsar-mini apache/pulsar

kubectl get pods -n pulsar
```

### Produce and Consume Messages

Get Proxy info:

```bash
kubectl get svc -n pulsar | grep pulsar-mini-proxy
```

Run a simple message test:

```bash
kubectl exec -it -n pulsar pulsar-mini-toolset-0 -- /bin/bash
```

Create a tenant, namespace, and topic:

```bash
bin/pulsar-admin tenants create apache
bin/pulsar-admin namespaces create apache/pulsar
bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic -p 4
```

Produce messages:

```bash
bin/pulsar-client produce apache/pulsar/test-topic -m "Hello Apache Pulsar" -n 10
```

Consume messages:

```bash
bin/pulsar-client consume apache/pulsar/test-topic -s sub -n 0
```

### Pulsar Manager UI

```bash
kubectl exec -it pulsar-mini-pulsar-manager-0 -n pulsar -- /bin/bash

# Create superuser
CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
curl \
  -H "X-XSRF-TOKEN: $CSRF_TOKEN" \
  -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
  -H 'Content-Type: application/json' \
  -X PUT http://localhost:7750/pulsar-manager/users/superuser \
  -d '{"name": "pulsar", "password": "pulsar", "description": "test", "email": "user@test.org"}'
```

Password secrets:

```bash
kubectl get secret pulsar-mini-pulsar-manager-secret -n pulsar -o jsonpath="{.data}" | jq -r 'to_entries|map("\(.key)=\(.value|@base64d)")|.[]'
```

Access Pulsar Manager:

📍 http://localhost:7750
Login: pulsar / pulsar

### Grafana Dashboard

```bash
kubectl port-forward svc/pulsar-mini-grafana -n pulsar 3000:80
```

Access Grafana:

📍 http://localhost:3000

Credentials:

```bash
kubectl get secret pulsar-mini-grafana -n pulsar -o jsonpath="{.data}" | jq -r 'to_entries|map("\(.key)=\(.value|@base64d)")|.[]'
```

### Pulsar Python Client

```bash
pip install pulsar-client
```

Create producer.py:

```python
import pulsar

client = pulsar.Client('pulsar://localhost:6650')
producer = client.create_producer('apache/pulsar/test-topic')

for i in range(10):
    producer.send(('msg-%d' % i).encode('utf-8'))

client.close()
```

Create consumer.py:

```python
import pulsar

client = pulsar.Client('pulsar://localhost:6650')
consumer = client.subscribe('apache/pulsar/test-topic', 'sub')

for i in range(10):
    msg = consumer.receive()
    print(f"Received: {msg.data()}")
    consumer.acknowledge(msg)

client.close()
```

### Topic Stats

```bash
curl http://localhost:8080/admin/v2/persistent/public/default/my-topic/stats | python -m json.tool
```

### Cleanup

```bash
helm uninstall pulsar-mini -n pulsar
kind delete cluster --name pulsar
```

---

## 🧠 Final Thoughts

This Pulsar playground setup gives you a fully working Pulsar cluster on Kind, ready for real pub/sub workflows and admin use. It’s perfect for experimentation, workshops, or prototyping.


Apache Pulsar Playground: Running Pulsar Locally on kind with Dashboards, Clients, and Admin Tools


![Kafka on K8s screenshot](/images/example.png)

# Deploying Kafka on Kubernetes with Strimzi

In this article, we'll walk through deploying an Apache Kafka cluster on a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/) (Kubernetes in Docker). We'll use the [Strimzi operator](https://strimzi.io/) for Kafka, which simplifies the deployment and management of Kafka on Kubernetes. We'll also touch on how you could incorporate Terraform to manage some of the setup or perform upgrades.

## Prerequisites

- **Docker Desktop** installed
- **Kind** installed
- **kubectl** installed

## 1. Create a Local Kubernetes Cluster with Kind

First, create a Kind cluster. For a simple default cluster:

```bash
kind create cluster --name kafka-demo
```

Verify your cluster:

```bash
kubectl cluster-info --context kind-kafka-demo
```


## 2. Install the Strimzi Operator

2.1 Using kubectl
Strimzi provides YAML manifests you can apply directly:

```bash
# Download the Strimzi installation YAML
curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.34.0/strimzi-cluster-operator-0.34.0.yaml" -o strimzi.yaml
```

# Apply to your cluster

```bash
kubectl create namespace kafka
kubectl apply -f strimzi.yaml -n kafka
```

## 2.2 Using Helm (Optional)

```bash
helm repo add strimzi https://strimzi.io/charts/
helm repo update
helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace kafka --create-namespace
```

## 1. Deploy a Kafka Cluster

Create a kafka-cluster.yaml file:

```yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka-cluster
  namespace: kafka
spec:
  kafka:
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}
```

```
kubectl apply -f kafka-cluster.yaml -n kafka
```

Strimzi will spin up your Kafka brokers and Zookeeper pods.


## 1. Verifying Kafka

Once the operator has created the Kafka cluster, check your pods:

```bash

kubectl get pods -n kafka

You should see:
```bash
                                   READY   STATUS    RESTARTS   AGE
my-kafka-cluster-kafka-0                1/1     Running   0          2m
my-kafka-cluster-kafka-1                1/1     Running   0          2m
my-kafka-cluster-kafka-2                1/1     Running   0          2m
my-kafka-cluster-zookeeper-0            1/1     Running   0          2m
my-kafka-cluster-zookeeper-1            1/1     Running   0          2m
my-kafka-cluster-zookeeper-2            1/1     Running   0          2m
strimzi-cluster-operator-...            1/1     Running   0          3m
```

## 6. Testing Kafka

You can create a Kafka topic and produce/consume messages:

```bash
# Create a topic

kubectl exec -it my-kafka-cluster-kafka-0 -n kafka -- \
  bin/kafka-topics.sh --create --topic test-topic --partitions 1 --replication-factor 1 --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092

# Produce messages
kubectl exec -it my-kafka-cluster-kafka-0 -n kafka -- \
  bin/kafka-console-producer.sh --topic test-topic --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092

# Consume messages
kubectl exec -it my-kafka-cluster-kafka-1 -n kafka -- \
  bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092

Type a few messages in the producer terminal, and they should appear in the consumer terminal.
```

## 7. Clean Up

When you’re done, remove the cluster:

```bash
kind delete cluster --name kafka-demo
```

## Conclusion

With this setup, you have a quick local environment for developing or testing Kafka. The Strimzi operator abstracts away the complexity of Kafka management on Kubernetes. If you want to scale to a production environment, you’ll switch from ephemeral storage to persistent volumes, integrate with your cloud platform, and further tune your Terraform and Strimzi configs.


A step-by-step guide to setting up a Kafka cluster on a local Kind cluster using the Strimzi operator, with optional Terraform provisioning.

Deploying Kafka on Kubernetes with Strimzi


## Falco Kubernetes Lab: Runtime Threat Detection with Prometheus & Grafana

🧰 Prerequisites

- Docker
- Kind
- kubectl
- Helm
- Make

## Setting Up the Lab

Clone the repo

```bash
git clone https://github.com/moabukar/falco-labs.git
cd falco-labs
```

Start the Lab

```bash
make up
```

This command executes the setup.sh up script, which performs the following:

- Creates a new Kind cluster with the configuration specified in kind.yaml. Called falco-lab.
- Creates the falco namespace and deploys a custom rules ConfigMap.
- Adds the Falco Helm repository and installs Falco with metrics enabled.
- Deploys an Nginx workload.
- Adds the Prometheus Community Helm repository and installs the kube-prometheus-stack.
- Creates ConfigMaps for the Falco dashboard and Grafana datasource.
- Upgrades the kube-prometheus-stack to load the dashboard and datasource.
- Port-forwards the Grafana service to localhost:3000.
- Waits for the Falco pods to be ready.

Setup/Bootsrap script

kind config

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /dev
      containerPath: /dev
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock
  extraPortMappings:
    - containerPort: 6443
      hostPort: 60000
      listenAddress: "127.0.0.1"
      protocol: tcp
```

```bash
#!/bin/bash
set -e

function delete_existing_cluster() {
  if kind get clusters | grep -q "^falco-lab$"; then
    echo "[INFO] Cluster 'falco-lab' already exists. Deleting it..."
    kind delete cluster --name falco-lab
  fi
}

if [ "$1" == "up" ]; then
  echo "[+] Creating kind cluster..."
  delete_existing_cluster
  kind create cluster --name falco-lab --config kind.yaml

  echo "[+] Creating namespace 'falco' and deploying custom rules ConfigMap..."
  kubectl create ns falco || true
  kubectl create configmap falco-custom-rules \
    --from-file=custom-rule.yaml=custom-rule.yaml \
    -n falco || true

  echo "[+] Adding Falco Helm repo..."
  helm repo add falcosecurity https://falcosecurity.github.io/charts
  helm repo update

  echo "[+] Installing Falco (with metrics enabled)..."
  helm install falco falcosecurity/falco \
    --namespace falco \
    -f values.yaml

  echo "[+] Deploying nginx workload..."
  kubectl create deployment nginx --image=nginx

  echo "[+] Installing Prometheus & Grafana..."
  helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  helm repo update
  helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace

  echo "[+] Waiting for Grafana pod to be Ready..."
  kubectl wait --for=condition=Ready --timeout=180s pods -l app.kubernetes.io/name=grafana -n monitoring

  echo "[+] Creating ConfigMap for Falco Dashboard..."
  kubectl create configmap falco-dashboard \
    --from-file=falco_dashboard.json=falco_dashboard.json \
    -n monitoring || true
  kubectl label configmap falco-dashboard -n monitoring grafana_dashboard=1 --overwrite

  echo "[+] Creating ConfigMap for Grafana datasource..."
  kubectl create configmap grafana-datasource \
    --from-file=datasource.yaml=grafana_datasource.yaml \
    -n monitoring || true
  kubectl label configmap grafana-datasource -n monitoring grafana_datasource=1 --overwrite

  echo "[+] Upgrading kube-prometheus-stack to load dashboard and datasource..."
  helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --reuse-values \
    --set grafana.sidecar.dashboards.enabled=true \
    --set grafana.sidecar.dashboards.label=grafana_dashboard \
    --set grafana.dashboardsConfigMaps.falco-dashboard="falco-dashboard" \
    --set grafana.sidecar.datasources.enabled=true \
    --set grafana.sidecar.datasources.label=grafana_datasource

  echo "[+] Forwarding Grafana service on port 3000..."
  kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80 &
  
  echo "[+] Waiting for Falco pods to be Ready..."
  kubectl wait --for=condition=Ready --timeout=180s pods -l app.kubernetes.io/name=falco -n falco

  echo "[+] Lab setup complete."
  echo "[+] Grafana is available at http://localhost:3000"
  echo "[+] To get the Grafana admin password, run:"
  echo "    kubectl -n monitoring get secrets kube-prometheus-stack-grafana -o jsonpath=\"{.data.admin-password}\" | base64 -d ; echo"
  echo "[+] To generate events, run: ./generate_events.sh"
  echo "[+] Tailing Falco logs..."
  kubectl logs -n falco -l app.kubernetes.io/name=falco -f

elif [ "$1" == "logs" ]; then
  echo "[+] Tailing Falco logs..."
  kubectl logs -n falco -l app.kubernetes.io/name=falco -f

elif [ "$1" == "down" ]; then
  echo "[+] Uninstalling Falco..."
  helm uninstall falco -n falco || true

  echo "[+] Uninstalling Prometheus & Grafana..."
  helm uninstall kube-prometheus-stack -n monitoring || true

  echo "[+] Deleting all kind clusters..."
  for cluster in $(kind get clusters); do
    echo "[+] Deleting cluster: $cluster"
    kind delete cluster --name "$cluster"
  done

  echo "[+] Cleanup complete."
else
  echo "Usage: $0 {up|logs|down}"
  exit 1
fi

# Manual test examples:
# kubectl run -it curl-test --image=alpine -- sh
# apk add curl
# curl http://example.com
```

## 🔍 Testing Falco Detection

Generate Events

```bash
#!/bin/bash
set -e

# Get the nginx pod name (assumes only one nginx pod)
POD=$(kubectl get pods -l app=nginx -o jsonpath='{.items[0].metadata.name}')

echo "[+] Generating event: reading /etc/shadow..."
kubectl exec -it "$POD" -- cat /etc/shadow || echo "[!] Failed to read /etc/shadow"

echo "[+] Generating event: writing to /etc/testfile..."
kubectl exec -it "$POD" -- sh -c "echo 'Falco Test' > /etc/testfile" || echo "[!] Write event failed"

echo "[+] Generating event: spawning a shell..."
kubectl exec -it "$POD" -- sh -c "sh -c 'echo Shell spawned'" || echo "[!] Shell spawn event failed"

echo "[+] Generating event: making a network connection (curl http://example.com)..."
kubectl exec -it "$POD" -- sh -c "apk add --no-cache curl && curl -s http://example.com" || echo "[!] Curl event failed"

echo "[+] Event generation complete."
```

```bash
./generate_events.sh
```

This script performs the following actions to trigger Falco rules:

- Reads the /etc/shadow file inside the Nginx pod.
- Writes to /etc/testfile inside the Nginx pod.
- Spawns a shell inside the Nginx pod.
- Makes a network connection using curl inside the Nginx pod.

Falco logs:

![Falco logs](/images/falco-logs.png)

## Create Falco Rules

here we have a custom rules file that we can use to test Falco.

Some rules like:

- Detecting curl in container
- Detecting shell in container
- Detecting write to /etc/sudoers
- Detecting write to /etc/shadow
- Shell spawned in container
- Privilege escalation via setuid binary
- Unexpected network connection from container

```yaml
- rule: Detect curl in container
  desc: Someone ran curl inside a container
  condition: container and proc.name = "curl"
  output: "⚠️ curl detected: user=%user.name command=%proc.cmdline container=%container.id"
  priority: WARNING
  tags: [network, curl, suspicious]

- rule: "Read sensitive file /etc/shadow"
  desc: "Detect any read access to /etc/shadow"
  condition: "evt.type in (open, openat, openat2) and fd.name = /etc/shadow"
  output: "Sensitive file /etc/shadow read (command=%proc.cmdline user=%user.name)"
  priority: WARNING
  tags: [filesystem, sensitive]

- rule: "Write to /etc directory"
  desc: "Detect write operations to any file under /etc"
  condition: "evt.type in (open, openat, openat2) and evt.is_open_write=true and fd.name startswith /etc"
  output: "File in /etc written (command=%proc.cmdline user=%user.name)"
  priority: WARNING
  tags: [filesystem, custom]

- rule: "Write to /etc/sudoers"
  desc: "Detect any write to /etc/sudoers"
  condition: "evt.type in (open, openat, openat2) and evt.is_open_write=true and fd.name = /etc/sudoers"
  output: "Suspicious write to /etc/sudoers (command=%proc.cmdline user=%user.name)"
  priority: CRITICAL
  tags: [privilege_escalation, custom]

- rule: "Shell spawned in container"
  desc: "Detect any shell spawned in a container"
  condition: "proc.name in (sh, bash, zsh) and container.id != host"
  output: "Shell spawned in container (command=%proc.cmdline, container=%container.id)"
  priority: NOTICE
  tags: [container, runtime]

- rule: "Privilege escalation via setuid binary"
  desc: "Detect execution of setuid binaries (e.g., sudo, passwd) in a container"
  condition: "proc.name in (sudo, passwd) and evt.type = execve and container.id != host"
  output: "Setuid binary execution detected (command=%proc.cmdline user=%user.name)"
  priority: CRITICAL
  tags: [privilege_escalation, container]

- rule: shell_in_container
  desc: notice shell activity within a container
  condition: >
    evt.type = execve and 
    evt.dir = < and 
    container.id != host and 
    (proc.name = bash or
     proc.name = ksh)    
  output: >
    shell in a container
    (user=%user.name container_id=%container.id container_name=%container.name 
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)    
  priority: WARNING

- rule: "Unexpected network connection from container"
  desc: "Detect network connection attempts from container processes"
  condition: "evt.type = connect and container.id != host"
  output: "Network connection from container detected (command=%proc.cmdline, connection=%fd.name)"
  priority: NOTICE
  tags: [network, container]
```

## View Falco Logs

```bash
make logs
```

This make alias tails the Falco logs, allowing you to observe the alerts generated by the events above.

## 📊 Accessing Prometheus & Grafana

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
```

Prometheus

Access Prometheus at: http://localhost:9090

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
```

Grafana

Access Grafana at: http://localhost:3000

Username: admin
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
Access Prometheus at: http://localhost:9090

```bash
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
```

Access Grafana at: http://localhost:3000

Username: admin

Password: prom-operator

The Falco dashboard should be automatically loaded, displaying metrics and alerts.

Grafana Dashboard

```json
{
    "annotations": {
      "list": []
    },
    "editable": true,
    "gnetId": null,
    "graphTooltip": 0,
    "id": null,
    "iteration": 1621469837171,
    "links": [],
    "panels": [
      {
        "datasource": "Prometheus",
        "fieldConfig": {
          "defaults": {},
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 24,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "legend": {
            "displayMode": "list",
            "placement": "bottom"
          },
          "tooltip": {
            "mode": "single"
          }
        },
        "targets": [
          {
            "expr": "sum(rate(falco_rules_alert_total[5m])) by (priority)",
            "interval": "",
            "legendFormat": "{{priority}}",
            "refId": "A"
          }
        ],
        "title": "Falco Alerts by Priority",
        "type": "timeseries"
      }
    ],
    "schemaVersion": 27,
    "style": "dark",
    "tags": ["falco"],
    "templating": {
      "list": []
    },
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "",
    "title": "Falco Dashboard",
    "uid": "falco-dashboard"
  }
```

## 🧹 Tearing Down the Lab

```bash
make down
```

This command executes the setup.sh down script, which performs the following:

Uninstalls Falco and the kube-prometheus-stack.

Deletes all Kind clusters.

## Conclusion

This lab provides a practical demonstration of integrating Falco into a Kubernetes environment for runtime security monitoring. By generating specific events, you can observe how Falco detects and alerts on suspicious activities, with Prometheus and Grafana providing visualization and analysis capabilities.


Falco Kubernetes Lab: Runtime Threat Detection with Prometheus & Grafana

Posts tagged with: kind