
## Introduction

### What Are We Building? 🔐

In this deep dive, we’ll secure traffic to an app on GKE using **mutual TLS (mTLS)** at the ingress layer with **GKE Gateway API**. We’ll use self-signed certs to demo client certificate validation—similar to how financial APIs and zero-trust platforms validate service-to-service identity.

### Why This Matters

mTLS is core to secure service meshes and external-facing APIs. Validating clients via TLS certs strengthens your perimeter, especially for sensitive workloads.

## Scenario Overview

- GKE Autopilot cluster (or standard)
- Gateway API enabled with managed Gateway controller
- One app (`httpbin`) behind the Gateway
- Server cert (`tls.crt`, `tls.key`) deployed via `Secret`
- Client cert (`client.crt`) validated against a CA bundle (`ca.crt`)
- All certs are self-signed for demo, in real-world you'd use cert-manager or ACM

## Architecture

![mTLS Gateway](/images/mtls-gke.png)

- Gateway only accepts connections where client cert is signed by known CA
- Ingress TLS termination happens at Gateway
- Backend is HTTP (no mTLS between Gateway and app in this example)

## Prereqs

- GCP project with billing enabled
- `gcloud`, `kubectl`, `openssl`
- Gateway API CRDs installed
- GKE cluster with Gateway controller enabled

## Step-by-Step Setup

### Step 1: Create the Cluster

```bash
gcloud container clusters create-auto mtls-demo \
  --region us-central1
```

### Enable Gateway API

```bash
gcloud container clusters update mtls-demo \
  --enable-gateway-api

# install CRDs if needed
kubectl apply -k "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v1.0.0"
```

### Generate TLS & CA Certs

```bash
mkdir certs && cd certs

# Create CA
openssl req -x509 -nodes -new -sha256 -days 3650 \
  -newkey rsa:2048 -subj "/CN=MyCA" \
  -keyout ca.key -out ca.crt

# Server cert
openssl req -new -newkey rsa:2048 -nodes -keyout server.key \
  -subj "/CN=httpbin.local" -out server.csr

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

# Client cert
openssl req -new -newkey rsa:2048 -nodes -keyout client.key \
  -subj "/CN=test-client" -out client.csr

openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 365
```

### Deploy app & secrets

```bash
kubectl create ns httpbin

kubectl create deployment httpbin --image=kennethreitz/httpbin --port=80 -n httpbin
kubectl expose deployment httpbin --port=80 --target-port=80 -n httpbin

kubectl create secret tls server-cert \
  --cert=certs/server.crt --key=certs/server.key -n httpbin

kubectl create configmap ca-cert --from-file=ca.crt=certs/ca.crt -n httpbin
```

### Create Gateway & Routes

Gateway config (gateway.yaml)
```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: httpbin-gw
  namespace: httpbin
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: server-cert
        options:
          networking.gke.io/client-verification: "require"
          networking.gke.io/trusted-ca: ca-cert
```

Route config (route.yaml)

```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin-route
  namespace: httpbin
spec:
  parentRefs:
    - name: httpbin-gw
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: httpbin
          port: 80
```

Apply the configs:

```bash
kubectl apply -f gateway.yaml
kubectl apply -f route.yaml
```

### Test the setup

Get IP address of the Gateway:

```bash
kubectl get gateway -n httpbin
```

Let’s test with and without the client cert.

✅ Valid client

```bash
curl --cert certs/client.crt --key certs/client.key --cacert certs/ca.crt \
  https://<GATEWAY_IP>/get
```

❌ No client cert

```bash
curl --cacert certs/ca.crt https://<GATEWAY_IP>/get
```

Expected: 400 or 403 depending on GKE's enforcement.

### Automated Testing (Optional)

Create a script to automate the testing.

```bash
#!/bin/bash
URL=https://<GATEWAY_IP>/get
curl --silent --cert certs/client.crt --key certs/client.key --cacert certs/ca.crt $URL | grep "headers"
if [ $? -ne 0 ]; then echo "mTLS test failed"; exit 1; fi
echo "mTLS test passed"
```

## Key Takeaways

- GKE Gateway API supports native mTLS client validation via annotations
- Certs must be manually injected via Secret and ConfigMap
- Only valid clients (with signed certs) can connect

This pattern is extendable to service meshes, APIs, and zero-trust networking

## Code 

- [Blog Code](https://github.com/moanukar/mtls-gke)

## Conclusion

You now have a secure GKE Gateway that validates clients using mutual TLS. In the next part, we'll explore using cert-manager for automated cert issuance and chaining this setup with an internal-only backend.


In this blog, we configure mutual TLS (mTLS) using Gateway API on GKE, securing ingress traffic with client certificate validation.

Secure Gateways: Configuring Mutual TLS using Gateway API on GKE


## Introduction

DNS spoofing in Kubernetes remains a critical threat, enabling attackers to redirect traffic, intercept data, or disrupt services. This article explores how such attacks occur and outlines strategies to prevent them.

## K8s networking

In Kubernetes, pods communicate over a virtual bridge (commonly cbr0) that connects all pods on a node. The cbr0 can also handle ARP (Address Resolution Protocol) resolution. When an incoming packet arrives at cbr0, it can resolve the destination MAC address using ARP.

CoreDNS, running as a pod, handles DNS resolution for the cluster. Each pod's /etc/resolv.conf typically points to the ClusterIP of the CoreDNS service, often 10.96.0.10. Kube-proxy sets up iptables rules to route DNS requests to the appropriate CoreDNS pod.

In essence, this is how pods communicate with each other on the same node. It’s also how Docker works and is the default for K8s.

## Understanding How DNS Works Inside Kubernetes

Let’s break down how DNS requests flow in a Kubernetes cluster — especially how pods talk to CoreDNS.

Each pod in your cluster is set up to use CoreDNS (or kube-dns) as its default DNS server. CoreDNS itself runs as a pod, typically inside the kube-system namespace. You can have multiple CoreDNS pods for HA, but for simplicity, let’s assume one for now.

### How Does a Pod Know Where CoreDNS Lives?

When a pod is created, Kubernetes injects a /etc/resolv.conf file like this:

```bash
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
```

That 10.96.0.10 is the ClusterIP of the kube-dns Service. It’s a stable VIP (virtual IP) used for all DNS requests inside the cluster. CoreDNS pods sit behind this service and receive DNS queries forwarded by kube-proxy using iptables DNAT rules.

### How the Flow Works

1. Your pod makes a DNS request (e.g. nslookup my-service)

2. It hits the VIP (10.96.0.10)

3. kube-proxy rewrites the packet (DNAT) to one of the actual CoreDNS pod IPs

4. CoreDNS checks:
    - Is this a cluster service/pod/etc? → resolve it locally
    - Else → forward to upstream DNS (e.g. Google DNS)

This behavior is defined by the pod’s dnsPolicy, which is set to ClusterFirst by default.

### From kubelet's DNS handling logic:

For a pod with DNSClusterFirst policy,
the cluster DNS server is the only nameserver configured.
CoreDNS will forward queries to upstream nameservers
if it can’t resolve them locally.

### 🧩 Key Point

A pod doesn’t talk to CoreDNS directly by IP — it talks to a Service VIP that’s transparently routed to a pod. This abstraction makes DNS scalable and fault-tolerant, but also opens the door for network-layer attacks like DNS spoofing if a pod has raw socket access.

## The Attack Vector: ARP Spoofing via NET_RAW

By default, Kubernetes grants pods the NET_RAW capability, allowing them to craft raw packets, including ARP messages. An attacker can exploit this to perform ARP spoofing, tricking the network into sending DNS requests to a malicious pod instead of the legitimate CoreDNS pod.

### Steps to Exploit

1. Identify CoreDNS Pod IP: Send a DNS request and observe the source IP of the response to determine the CoreDNS pod's IP.

2. Determine Bridge IP and MAC: Use tools like scapy to send packets with TTL=1 to an external IP, capturing the bridge's IP and MAC address.

3. Send Fake ARP Replies: Continuously send ARP replies to the bridge, associating the CoreDNS IP with the attacker's MAC address.

4. Intercept DNS Requests: Run a DNS proxy in the malicious pod to handle intercepted DNS requests, forwarding them to the real CoreDNS pod or spoofing responses as desired.

This method allows the attacker to intercept and manipulate DNS traffic, potentially redirecting services to malicious endpoints.

## Mitigation Strategies

1. Drop NET_RAW Capability

Modify pod security contexts to drop the NET_RAW capability, preventing pods from crafting raw packets.

```yaml
securityContext:
  capabilities:
    drop:
      - NET_RAW
```

1. Implement Network Policies
Use Kubernetes NetworkPolicies to restrict pod communication, ensuring only authorized pods can communicate with CoreDNS.

1. Isolate CoreDNS Pods
Schedule CoreDNS pods on dedicated nodes and restrict access to these nodes, reducing the risk of compromise.

1. Monitor and Audit Changes
Continuously monitor and audit changes to the CoreDNS ConfigMap and related resources to detect unauthorized modifications promptly.

## Conclusion

DNS spoofing remains a common threat in Kubernetes environments. By understanding the attack vectors and implementing robust security measures, organisations can safeguard their clusters against such threats.


Kubernetes DNS Spoofing: Exploiting NET_RAW and ARP


## Introduction

Setting up a private network for your EKS cluster is important for security and performance. There are many ways to do this, but in this article, we'll use Twingate to create a private network for your EKS cluster.

## Step-by-Step Guide: Deploying a Private EKS Cluster with Twingate Access

### Prerequisites

- An AWS account
- A Twingate account
- Twingate CLI/client installed. 
- eksctl CLI

## Terraform Setup

Assume you have a VPC and subnets already created.

```hcl
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "private-eks-cluster"
  cluster_version = "1.27"
  subnets         = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false

  # Additional configurations...
}
```

## Twingate Setup

Twingate provides secure, remote access to private resources without exposing them to the public internet. To integrate Twingate with your EKS cluster:

- Create a Twingate Account: Sign up at Twingate.
- Define a Remote Network: In the Twingate admin console, add a new remote network representing your AWS VPC.
- Deploy a Connector: Deploy a Twingate Connector within your VPC. This can be done using AWS ECS Fargate, EC2, or as a Kubernetes Deployment. Ensure the Connector has outbound internet access to communicate with Twingate's services and can reach the EKS API endpoint.

### Twingate Connector Deployment

Deploy the Twingate Connector as a container on AWS ECS (Fargate) within your VPC. Ensure the subnet has outbound internet access to communicate with Twingate's services and access the EKS control plane.

### Twingate Resource Configuration

After deploying the Connector, configure Twingate to manage access to your EKS cluster:
    - Add a Remote Network: In the Twingate admin console, add a new remote network representing your AWS VPC.
    - Define Resources: Add a new resource with the private DNS name of your EKS API server.
    - Assign Access: Specify which user groups should have access to this resource.

### Benefits of This Setup

- Enhanced Security: The EKS API server is not exposed to the public internet, reducing the attack surface.
- Granular Access Control: Twingate allows you to define precise access policies based on user groups.
- Zero Trust Architecture: This configuration aligns with Zero Trust principles, ensuring that trust is never implicit.
- Simplified Management: Twingate's centralized management console makes it easy to oversee access policies and monitor activity.

By following this approach, you can establish a secure, private EKS cluster accessible only through Twingate, providing robust protection for your Kubernetes workloads.


How to setup a private network for your EKS cluster with Twingate

EKS Private Network with Twingate


## Advanced Kubernetes Networking with Cilium on Kind

Cilium is open source software for transparently securing the network connectivity between application services deployed using Linux container management platforms like Docker and Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which enables the dynamic insertion of powerful security visibility and control logic within Linux itself. Because eBPF runs inside the Linux kernel, Cilium security policies can be applied and updated without any changes to the application code or container configuration.

## Prerequisites

Ensure the following are installed:

- Docker Desktop
- kind

## Cluster Setup

Option 1: Create cluster manually

kind-config.yaml - as you can see, we have disableDefaultCNI set to true. This is to ensure that Cilium is used as the CNI provider.

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
  disableDefaultCNI: true
```

```bash
kind create cluster --config=kind-config.yaml
```

## Install Cilium

```bash
cilium version --client           # Verify cilium client is installed
cilium install --version 1.16.1   # Install Cilium into the cluster
cilium status                     # Verify Cilium DaemonSets are running
kubectl get nodes -A              # Nodes should now be Ready with CNI
```

## Run Cilium Connectivity Tests

```bash
cilium connectivity test
```

Example Output Snippet:
```bash

✨ [kind-kind] Creating namespace cilium-test for connectivity check...
✨ [kind-kind] Deploying echo-same-node service...
...
✅ [cilium-test] All 59 tests (608 actions) successful, 41 tests skipped, 1 scenarios skipped.
```

## Enable and Use Hubble (Observability)

Enable Hubble:

```bash
cilium hubble enable
```

Check status:

```bash
cilium status
```

```bash
Expected output:

/\_/\      Cilium:             OK
\_/ \_/    Operator:           OK
/\_/\      Envoy DaemonSet:    OK
\_/ \_/    Hubble Relay:       OK
...
```

## Port-forward for local Hubble Relay access:

```bash
cilium hubble port-forward &
kubectl port-forward -n kube-system svc/hubble-relay --address 127.0.0.1 4245:80
```

## Use Hubble:

```bash
hubble status
hubble observe
```

## Simulate Network Traffic with Star Wars App

```bash
kubectl apply -f apps/apps.yaml

kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

Observe traffic:

hubble observe --pod deathstar --protocol http
hubble observe --pod deathstar --verdict DROPPED
```

## Clean Up

`kind delete clusters -A`

This guide walked you through running Cilium + Hubble on a local kind cluster, testing end-to-end connectivity and observing network traffic with eBPF-powered visibility. This setup can form the foundation of more advanced policy testing, multi-cluster setups, or observability pipelines.


Cilium in Kubernetes


## 🎡 Introduction

In this blog, I’ll walk you through setting up a full-featured **Apache Pulsar playground** using **kind** (Kubernetes in Docker). Whether you’re testing Pulsar for learning or demoing a real pub/sub model with admin tools and monitoring, this setup gives you everything:

- ✅ Pulsar cluster deployed via Helm
- ✅ Prometheus + Grafana for monitoring
- ✅ Pulsar Manager UI
- ✅ Python producer/consumer using `pulsar-client`
- ✅ Proxy + dashboard exposure
- ✅ Hands-on with Pulsar admin CLI

---

## 🧰 Prerequisites

- [Docker](https://www.docker.com/)
- [kind](https://kind.sigs.k8s.io/)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [Helm](https://helm.sh/docs/intro/install/)
- [Python 3](https://www.python.org/) and `pip`

---

## 🔧 Step 1: Create a kind Cluster with Port Mappings

Create a local cluster that maps the UI and proxy ports:

```bash
cat <<EOF | kind create cluster --name pulsar --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 8080  # Pulsar REST Broker
    hostPort: 8080
  - containerPort: 6650  # Pulsar Binary Protocol
    hostPort: 6650
  - containerPort: 7750  # Pulsar Manager UI
    hostPort: 7750
  - containerPort: 3000  # Grafana
    hostPort: 3000
EOF
```

### Install Pulsar with Helm

```bash
helm repo add apache https://pulsar.apache.org/charts
helm repo update

git clone https://github.com/apache/pulsar-helm-chart
cd pulsar-helm-chart

./scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar-mini -c

helm install \
  --values examples/values-minikube.yaml \
  --namespace pulsar \
  pulsar-mini apache/pulsar

kubectl get pods -n pulsar
```

### Produce and Consume Messages

Get Proxy info:

```bash
kubectl get svc -n pulsar | grep pulsar-mini-proxy
```

Run a simple message test:

```bash
kubectl exec -it -n pulsar pulsar-mini-toolset-0 -- /bin/bash
```

Create a tenant, namespace, and topic:

```bash
bin/pulsar-admin tenants create apache
bin/pulsar-admin namespaces create apache/pulsar
bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic -p 4
```

Produce messages:

```bash
bin/pulsar-client produce apache/pulsar/test-topic -m "Hello Apache Pulsar" -n 10
```

Consume messages:

```bash
bin/pulsar-client consume apache/pulsar/test-topic -s sub -n 0
```

### Pulsar Manager UI

```bash
kubectl exec -it pulsar-mini-pulsar-manager-0 -n pulsar -- /bin/bash

# Create superuser
CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
curl \
  -H "X-XSRF-TOKEN: $CSRF_TOKEN" \
  -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
  -H 'Content-Type: application/json' \
  -X PUT http://localhost:7750/pulsar-manager/users/superuser \
  -d '{"name": "pulsar", "password": "pulsar", "description": "test", "email": "user@test.org"}'
```

Password secrets:

```bash
kubectl get secret pulsar-mini-pulsar-manager-secret -n pulsar -o jsonpath="{.data}" | jq -r 'to_entries|map("\(.key)=\(.value|@base64d)")|.[]'
```

Access Pulsar Manager:

📍 http://localhost:7750
Login: pulsar / pulsar

### Grafana Dashboard

```bash
kubectl port-forward svc/pulsar-mini-grafana -n pulsar 3000:80
```

Access Grafana:

📍 http://localhost:3000

Credentials:

```bash
kubectl get secret pulsar-mini-grafana -n pulsar -o jsonpath="{.data}" | jq -r 'to_entries|map("\(.key)=\(.value|@base64d)")|.[]'
```

### Pulsar Python Client

```bash
pip install pulsar-client
```

Create producer.py:

```python
import pulsar

client = pulsar.Client('pulsar://localhost:6650')
producer = client.create_producer('apache/pulsar/test-topic')

for i in range(10):
    producer.send(('msg-%d' % i).encode('utf-8'))

client.close()
```

Create consumer.py:

```python
import pulsar

client = pulsar.Client('pulsar://localhost:6650')
consumer = client.subscribe('apache/pulsar/test-topic', 'sub')

for i in range(10):
    msg = consumer.receive()
    print(f"Received: {msg.data()}")
    consumer.acknowledge(msg)

client.close()
```

### Topic Stats

```bash
curl http://localhost:8080/admin/v2/persistent/public/default/my-topic/stats | python -m json.tool
```

### Cleanup

```bash
helm uninstall pulsar-mini -n pulsar
kind delete cluster --name pulsar
```

---

## 🧠 Final Thoughts

This Pulsar playground setup gives you a fully working Pulsar cluster on Kind, ready for real pub/sub workflows and admin use. It’s perfect for experimentation, workshops, or prototyping.


Apache Pulsar Playground: Running Pulsar Locally on kind with Dashboards, Clients, and Admin Tools


## TL;DR

We’ll deploy SPIRE on a local kind cluster, generate SPIFFE IDs for workloads, and use mutual TLS between two pods. This is how to go from zero to zero trust in Kubernetes.

What is SPIFFE/SPIRE?

- SPIFFE (Secure Production Identity Framework For Everyone): defines a framework for workload identity.
- SPIRE (SPIFFE Runtime Environment): the production-ready implementation of SPIFFE.

## Why You Should Care (Real World)

- Legacy apps don’t rotate secrets.
- TLS certs are static or manually managed.
- You can’t verify “who” a service is at runtime.
- SPIFFE solves this by issuing short-lived X.509 certificates tied to workload identity.

```mermaid
+---------------------+
|  SPIRE Server (K8s) |
+---------------------+
         |
         | gRPC (registration, trust bundles)
         v
+---------------------+         +----------------------+
| SPIRE Agent (DaemonSet) |<----->| Workload A (sidecarless) |
+---------------------+         +----------------------+
         |                                 |
         |                                 |
         v                                 v
  /spire-agent.sock                /spire-agent.sock
    (Unix socket)                    (shared via volumeMount)
```

- SPIRE Server runs as a StatefulSet
- SPIRE Agents run as DaemonSet, connect securely to server
- Workloads use a Unix socket to get certs

## What We’ll Build

- Setup kind cluster
- Install SPIRE Server and Agent
- Deploy two workloads: app-a and app-b
- Secure their communication using mTLS via SPIFFE

Pre-reqs:

```bash
brew install kind kubectl helm
```

Setup Kind:

```bash
cat <<EOF | kind create cluster --name spire --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: spire
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30443
        hostPort: 443
EOF
```

Get the Spire repo

```bash
git clone https://github.com/spiffe/spire-examples.git
cd spire-examples/k8s/simple_psat
```

Deploy SPIRE Server and Agent

```bash
kubectl apply -k .
```

This applies:

- spire-server.yaml
- spire-agent.yaml
- RBAC and namespace (via kustomize)

Register workloads 


Use spire-server running inside the cluster:

```bash
kubectl exec -n spire deploy/spire-server -- \
/opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://example.org/ns/default/sa/default \
  -parentID spiffe://example.org/spire/agent/k8s_psat/my-cluster/node \
  -selector k8s:ns:default \
  -selector k8s:sa:default \
  -selector k8s:pod-label:app:app-a \
  -ttl 60
```

Repeat for app-b.

You’ll find more registration examples in:

```bash
examples/k8s/advanced/register-example.sh
```

## Verify Secure Communication

You’ll see app-a sending requests with a SPIFFE cert, app-b validating it.

```bash
kubectl logs -l app=app-b
```

Output should show:

```bash
Verified SPIFFE ID: spiffe://example.org/ns/default/sa/default
```

Diagram – Simplified

```mermaid
+---------+       mTLS       +---------+
| app-a   |----------------->| app-b   |
|         |    SPIFFE ID     |         |
| spiffe://.../app-a         | spiffe://.../app-b
+---------+                  +---------+
```

## Real-World Use Cases

- Zero-trust networking with Linkerd/Istio (replace cert-manager)
- Secretless app deployments (SPIFFE cert as identity token)
- Automatic rotation of certs every few minutes

## Bonus: Integrate with Istio or Linkerd

You can configure your mesh to trust SPIFFE identities instead of using static certs. Example (Istio):

```yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
```

Then use the SPIFFE trust domain in your AuthorizationPolicies.

## Conclusion

- SPIRE gives you automated workload identity via X.509 or JWT-SVID
- SPIFFE ID is portable across clusters, clouds, workloads
- You no longer rely on long-lived secrets or static TLS certs


Secure Your Kubernetes with SPIFFE + SPIRE: Zero-Trust Identity for Workloads

SPIFFE and SPIRE in Kubernetes


## Kubernetes as a Cloud Control Plane: Deep Dive into AWS ACK with kind

Kubernetes isn't just about container orchestration anymore — it's become the de facto control plane for everything. With Custom Resource Definitions (CRDs) and controllers, we can now describe and manage virtually any resource declaratively — whether it's an app, an S3 bucket, or an RDS instance. That’s where AWS Controllers for Kubernetes (ACK) come in.

ACK lets us manage AWS resources like RDS or EC2 the same way we manage Deployments and Services: using YAML manifests. The idea is powerful — replace scattered IaC tooling with a unified K8s-native approach. And if you’re already deploying everything through Argo CD, Flux, or Helm, why not include your cloud infra?

But does it actually hold up?

That’s what this post explores — we’ll walk through an end-to-end demo running ACK on a local kind cluster. 

⚙️ Environment Setup

```bash
mkdir ack-demo
cd ack-demo
```

🐳 Start Docker and Create Your Local Cluster
Ensure Docker is running. We’ll use it to spin up a local Kubernetes cluster using kind.

```bash
kind create cluster --config kind.yaml
```

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: dot
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |-
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 8080
    hostPort: 8080
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
```

dot.nu:

```bash
#!/usr/bin/env nu

source  scripts/common.nu
source  scripts/kubernetes.nu
source  scripts/ack.nu
source  scripts/crossplane.nu

def main [] {}

def "main setup" [] {

    rm --force .env

    main create kubernetes aws

    kubectl create namespace a-team

    kubectl --namespace a-team apply --filename rds-password.yaml

    main apply ack

    main apply crossplane --preview true --provider aws

    (
        kubectl apply 
            --filename crossplane-providers/cluster-role.yaml
    )

    (
        kubectl apply
            --filename crossplane-providers/provider-kubernetes-incluster.yaml
    )

    kubectl apply --filename dot-sql-config.yaml

    main wait crossplane

    main print source
    
}

def "main destroy" [] {

    main destroy kubernetes aws

    main delete ack
}
```

```bash
devbox shell
chmod +x dot.nu
./dot.nu setup
source .env
```

The kind.yaml file defines networking + node settings tailored for ACK.

## 🧬 ACK CRDs: AWS Infra as K8s Resources

Once the ACK controllers are installed, Kubernetes gains a bunch of new CRDs that represent AWS services — from RDS and S3 to EC2 subnets and VPCs.

Let’s take a look:

```bash
kubectl get crds | grep k8s.aws
```

You’ll see CRDs like:

```bash
dbinstances.rds.services.k8s.aws
vpcs.ec2.services.k8s.aws
subnets.ec2.services.k8s.aws
securitygroups.ec2.services.k8s.aws
...
```

Each CRD represents an AWS resource you can now define using a Kubernetes manifest. Think of it as Terraform, but via kubectl apply.

## 🚀 Deploying the ACK RDS Controller (locally with kind)

Since we’re using kind, not EKS, there’s no IRSA — so we authenticate with static AWS credentials injected into a K8s secret.

You already did this with ./dot.nu setup, but for clarity:

```bash
kubectl create ns ack-system

kubectl create secret generic ack-creds \
  -n ack-system \
  --from-literal=AWS_ACCESS_KEY_ID=<key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<secret>
```

Install the RDS ACK controller with Helm:

```bash
helm repo add ack https://aws.github.io/eks-charts
helm repo update

helm install ack-rds-controller ack/ack-rds-controller \
  -n ack-system \
  --set aws.region=us-east-1 \
  --set aws.secret.name=ack-creds
```

Now your cluster has a controller watching DBInstance resources and creating matching RDS instances in AWS.

## 🏗️ Defining an RDS PostgreSQL Instance (and dependencies)

Creating a DB in AWS isn’t one resource — it’s VPCs, Subnets, Gateways, RouteTables, and finally the DBInstance.

Here’s a peek from rds.yaml:

```yaml
apiVersion: ec2.services.k8s.aws/v1alpha1
kind: VPC
metadata:
  name: my-db
spec:
  cidrBlock: 11.0.0.0/16
---
apiVersion: ec2.services.k8s.aws/v1alpha1
kind: InternetGateway
metadata:
  name: my-db
spec:
  vpcRef:
    from:
      name: my-db
---
apiVersion: ec2.services.k8s.aws/v1alpha1
kind: Subnet
metadata:
  name: my-db-a
spec:
  cidrBlock: 11.0.1.0/24
  availabilityZone: us-east-1a
  vpcRef:
    from:
      name: my-db
...
apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: my-db
  annotations:
    services.k8s.aws/region: us-east-1
spec:
  dbInstanceClass: db.t3.micro
  engine: postgres
  engineVersion: "16.3"
  allocatedStorage: 20
  masterUsername: masteruser
  masterUserPassword:
    name: my-db-password
    key: password
  dbInstanceIdentifier: my-db
  publiclyAccessible: true
  dbSubnetGroupRef:
    from:
      name: my-db
  vpcSecurityGroupRefs:
    - from:
        name: my-db
Also, create a Secret with the DB password:
```

Also, create a Secret with the DB password:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-db-password
stringData:
  password: MySuperSecretPassw0rd!
```

Apply all resources:

```bash
kubectl apply -f rds-password.yaml
kubectl apply -f rds.yaml
```

Check the instance status:

```bash
kubectl get dbinstances.rds.services.k8s.aws
```

Once it's STATUS=available, your RDS instance is live.

## 🔍 Observability in ACK: Where’s the Feedback?

Once you start working with ACK, you’ll quickly hit a snag: visibility into resource states is... primitive at best.

You might try this:

```bash
kubectl -n a-team get \
  vpcs.ec2.services.k8s.aws,internetgateways.ec2.services.k8s.aws,routetables.ec2.services.k8s.aws,securitygroups.ec2.services.k8s.aws,subnets.ec2.services.k8s.aws,dbsubnetgroups.rds.services.k8s.aws,dbinstances.rds.services.k8s.aws
```

Output:

```bash
NAME    ID                       STATE
my-db   vpc-07cdab5bba559b994   available
...
my-db   subnet-0340...          available
...
my-db   db-2NOHJMPDDGYBPY6MH... creating
```
You get barebones info like STATE or STATUS, but it’s inconsistent and often misleading. Some resources use STATE, others use STATUS, and many don’t update accurately.

### 😡 Case Study: Broken Subnet

```bash
kubectl describe subnet.ec2.services.k8s.aws my-db-x
```

Yields:

```bash
Message: api error InvalidParameterValue: Value (us-east-1x) for parameter availabilityZone is invalid
Type:    ACK.Terminal
```

That’s actually helpful — the AZ doesn’t exist. But now try describing a working DBInstance:

```bash
kubectl describe dbinstance.rds.services.k8s.aws my-db
```

You’ll get a dump of AWS metadata with no intuitive indication of what’s actually happening. You have to hunt down .status.dbInstanceStatus and hope it says something like available or configuring-enhanced-monitoring.

No Ready condition. No events. No unified status format. ACK controllers don’t expose standard Kubernetes conditions. That’s not just annoying — it breaks integrations with Argo CD, Crossplane, and others.

AWS Controllers for Kubernetes


## 🔒 Introduction

Running Kubernetes clusters privately is a growing best practice. In this blog, I’ll walk you through deploying a **private AKS cluster** on Azure with **no public API endpoint**, and enabling secure access via **Twingate VPN**, which provides identity-based access without opening up your network.

This setup is ideal if:

- You want private networking in AKS (via Azure Private Link)
- You need granular access control over the cluster
- You want to avoid managing full VPN appliances or bastion hosts

---

### What We'll Build

- A private AKS cluster with no public API server
- A Twingate Connector running in the same VNet
- Twingate configured with the AKS API server as a protected resource
- Remote access to the cluster using `kubectl` via Twingate

---

### Infrastructure Setup

- Provision a private AKS cluster

You can use `az cli` or Terraform (recommended):

```bash
az aks create \
  --name dev-private \
  --resource-group dev-private_group \
  --enable-private-cluster \
  --vnet-subnet-id /subscriptions/.../subnets/aks-subnet \
  --node-count 2 \
  --generate-ssh-keys
```

Terraform:

```hcl
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "rg-private-aks"
  location = "UK South"
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-private-aks"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "aks_subnet" {
  name                 = "snet-aks"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "private-aks-cluster"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "privateaks"

  default_node_pool {
    name       = "default"
    node_count = 2
    vm_size    = "Standard_DS2_v2"
    vnet_subnet_id = azurerm_subnet.aks_subnet.id
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin    = "azure"
    dns_service_ip    = "10.0.2.10"
    service_cidr      = "10.0.2.0/24"
    docker_bridge_cidr = "172.17.0.1/16"
  }

  api_server_access_profile {
    enable_private_cluster = true
  }

  tags = {
    Environment = "Private"
  }
}
```

The --enable-private-cluster flag ensures the Kubernetes API server is only accessible over the VNet.

To check the API server endpoint:

```bash
az aks show --name dev-private --resource-group dev-private_group --query privateFqdn
```

You’ll get something like:

```bash
dev-private-dns-xxxxxx.privatelink.uksouth.azmk8s.io
```

---

### Push Twingate Connector Image to Azure Container Registry (ACR)

If using a private ACR:

```bash
az acr login --name <your-acr>
docker pull --platform linux/amd64 twingate/connector:1
docker tag twingate/connector:1 <your-acr>.azurecr.io/twingate/connector:1
docker push <your-acr>.azurecr.io/twingate/connector:1
```

### Deploy Twingate Connector as Azure Container Instance

```bash
az container create \
  --name twingate-connector \
  --image <your-acr>.azurecr.io/twingate/connector:1 \
  --resource-group dev-private_group \
  --vnet dev-private_group-vnet \
  --subnet twingate \
  --cpu 1 \
  --memory 2 \
  --environment-variables \
      TWINGATE_NETWORK="mo-demo" \
      TWINGATE_ACCESS_TOKEN="<your-access-token>" \
      TWINGATE_REFRESH_TOKEN="<your-refresh-token>" \
      TWINGATE_TIMESTAMP_FORMAT="2" \
      TWINGATE_LABEL_DEPLOYED_BY="azure"
```

Terraform:

```hcl
resource "azurerm_container_group" "twingate_connector" {
  name                = "twingate-connector"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  os_type             = "Linux"

  container {
    name   = "twingate"
    image  = "youracrname.azurecr.io/twingate-connector:1"
    cpu    = "1"
    memory = "1.5"

    environment_variables = {
      TWINGATE_NETWORK        = "your_network"
      TWINGATE_ACCESS_TOKEN   = "your_access_token"
      TWINGATE_REFRESH_TOKEN  = "your_refresh_token"
      TWINGATE_LABEL_DEPLOYED_BY = "terraform"
    }
  }

  ip_address_type = "Private"
  subnet_ids      = [azurerm_subnet.aks_subnet.id]
}
```

### Configure Twingate

- Setup a Twingate account
- Create a new remote network in Twingate

###  Add the AKS API Server as a Twingate Resource

In the Twingate Admin Console:

- Create a new Remote Network for your VNet (e.g., aks-vnet)
- Deploy the Connector to that Remote Network (you already did)
- Add a Resource with the private AKS API DNS name
  - e.g., dev-private-dns-xxxxxx.privatelink.uksouth.azmk8s.io
  - Port: 443
- Assign the Resource to a group (e.g., engineering)

### Access the Cluster via Twingate

Once the Connector is live and you're authenticated via the Twingate client:

```bash
az aks get-credentials --resource-group dev-private_group --name dev-private
kubectl get nodes
```

It works—because your local traffic is tunneled securely through Twingate and routed to the private API server over the VNet!

You can also run commands remotely using:

```bash
az aks command invoke \
  --resource-group dev-private_group \
  --name dev-private \
  --command "kubectl get pods -A"
```

### Bonus: Ingress Whitelisting

If you're using an NGINX ingress controller, and want to restrict access to known IPs:

```bash
nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.4.5
```

### Summary

With this setup:

- Your AKS cluster is completely private
- Access is secured and identity-aware via Twingate
- No public exposure, no bastion, no hassle


Private AKS Cluster with Twingate: Secure API Access Without a Public Endpoint


Containerising applications has become the norm in the IT/DevOps industry, allowing developers to condense their application code and all its dependencies in a lightweight and portable environment so that these applications can be deployed anywhere virtually.

Before going into containers, we must first understand why they were designed.

Let’s just say a Python application has been developed, the next logical step would be to deploy it; this can be complex as well as time-consuming. This is because a running app requires a myriad of configurations, ranging from the allocation of resources to setting up environmental variables.

Another consideration for the application is the ability of it being portable. If a developer needs to change the deployment environment or move to another cloud provider, they will need to repeat many steps of the deployment process. Once again, this can be a mundane process.

Containers efficiently solve the issues by packaging the codebase for the application with all the required dependences — including tools, libraries, config files and run-times. This process of containerisation ensures a lightweight and secure package (depending on how it’s done) known as a container image.

This image can then be used to deploy a containerised application in any environment required, thus giving the developers the much-needed freedom to design and develop apps without having to think about other constraints. Some common container runtimes are Docker, containerd and CRI-O.


![Cargo ships in reference to containers](/images/containers-intro.png)

Many large IT organisations became frustrated with 4–6 month releases and sluggish deployments. Organisations like Netflix and many others began to embrace microservices, thus helping them overtake their fellow competitors — who also had a similar plan. So applications began being built using microservices, heavily transforming into a more DevOps and Agile culture; larger teams were split into smaller ones to allow for this transformation.


DevOps cycle releases
Then came along issues whilst building microservices. Amongst those issues was containerising applications so that they are deployable on any environment, including a Dev’s one. As mentioned before, it was the idea of containers that solved this issue and then this lead to another issue which was managing these containers. This is where the story of Kubernetes began.

So, Kubernetes is a container orchestration platform or in simple terms, an application management platform.

Fun fact: The name Kubernetes itself stems from an ancient Greek word for “helmsman”, meaning he who steers a ship thus telling the story behind the K8s logo. Another name for Kubernetes (you hear it quite often) is “K8s” — the reason for this name is, the 8 letters between the “k” and the “s” in Kubernetes.

Note: From now on, K8s may be used to refer to Kubernetes sporadically.


![The Galver depicting the Kubernetes control mechanism](/images/ship.png)

## Why Kubernetes overtook its competitor OpenStack?

Although K8s and OpenStack have produced similar level of interests, they are slightly different from each other. In the end, Kubernetes has succeeded where OpenStack fell behind. The reason being, Kubernetes’ ability to provide business value and focus on the bigger picture which is the Enterprise IT applications. On the other hand, OpenStack orchestrated infrastructure and no matter how good your infrastructure management is, if you are not solving business problems, what use is it?

## Okay, so how does Kubernetes help me or my company ?

As mentioned before, Kubernetes enables a “microservice” approach to building modern application and larger teams can be split into smaller ones — leading to a much faster time to market, deployments of applications are always being updated and later versions are released.

**Costs of IT infrastructure are more optimised**. K8s can help your organisation largely cut infrastructure costs especially if they’re operating at a huge scale. Prior to the introduction of K8s, IT and Sys Admin, commonly over-provisioned their infrastructure to handle unexpected spikes. K8s is wise enough to schedule and put together containers, while taking into consideration available resources.

Nowadays, the success of applications doesn’t merely depend on features, but also on the scalability of the application. For indeed, if an app cannot scale properly, it will not perform at its best. Kubernetes is critical in this process as it scales and improves app performance. Let’s say we have a service which takes up high-CPU and its quite dynamic, meaning it changes based on business conditions; the solution required here would be something that can scale up the app so that new VMs are automatically created or scaled down when the load is reduced. Kubernetes offers just that capability by scaling up the app as the CPU usage goes above a certain threshold and scales back as a result, optimising the infrastructure utilisation. This feature of K8s auto-scaling known as Horizontal Pod Autoscaler (HPA) is not limited to managed metrics and custom metrics can also be used to trigger the auto-scaling process.

This article mentions only a few of the main benefits of Kubernetes and its various benefits can be spoken about for days.

What is next?

I’m currently on a journey to learn more about the ever-growing ecosystem of containers, container orchestration and the DevOps culture. Expect to see more articles and open-source contributions. Stay tuned for more!

A look at the ever-changing landscape of modern applications

The ever-changing landscape of modern applications


**A hands-on article on deploying an application on Kubernetes with Fargate.**

As promised, here is the second part of Fargate profiles on EKS. This article will look into deploying a “serverless” pod on EKS Fargate from scratch.

**Note:** We will use the eksctl CLI tool to do this for the purpose of this guide. Infrastructure as Code through Terraform or CloudFormation is advised if you are looking to set this up for your production workloads.

### Prerequisites:

*   Install eksctl command line utility [https://docs.aws.amazon.com/eks/latest/userguide/eksctl.htm](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html)l
*   Installing kubectl [https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html)

**Create a Kubernetes cluster on AWS**

A single eksctl command will create the following resources:

*   A VPC with the suitable subnets and accompanying resources
*   K8s control plane
*   Fargate profile
*   IAM role for pod execution

**Note**: Feel free to change the name of the cluster and region.

eksctl create cluster --name eks-fargate --region eu-west-1 -- fargate

This process may take several minutes. When complete, you should see a similar output to show that the cluster is ready to go.

![](https://cdn-images-1.medium.com/max/1760/1*QbuPEpiu8-Rwy1ZNaIJySQ.png)

Cluster and Fargate profile creation

**Verifying the context**

kubectl config current-context

Alternatively, you can use **“kubectx”** to check your current and existing contexts. Install with “brew install kubectx” for Mac and “choco” for Windows.

![](https://cdn-images-1.medium.com/max/1760/1*vvOmaBq3JaxXqGZvR4Avfw.png)

Current context shows that the cluster has been created

**Checking nodes**

We can now check if the fargate profile is ready, it should display at least one Fargate node waiting for our deployments.

kubectl get nodes

![](https://cdn-images-1.medium.com/max/1760/1*Fw5-3lM5fq1-Pg1kWGM1-Q.png)

Fargate nodes ready

**Note:** the nodes that are running inside our VPC are not visible in the EC2 dashboard nor can you SSH into the nodes; they are fully managed and deployed by Fargate by means of “serverless” but you can SSH into a specific pod.

Also, since we haven’t specified the fargate profile by name, the default name for the fargate profile will be “fp-default”. As well as not specifying a namespace, the default namespace will be used to schedule fargate pods. These settings can be changed during setup by specifying the namespace for the Fargate pods.

**Creating a custom Fargate profile**

For the purpose of this guide, we can add an additional fargate profile to show that it can be customised. This can be done with the command:

eksctl create fargateprofile --namespace test --cluster eks-fargate --name fp-test

All pods deployed on the “test” namespace will be scheduled as Fargate pods. We must first create the namespace “test”. Now we can deploy some containers using a default nginx image on the “test” namespace.

**Creating the namespace and deploying pod onto the Fargate-specified namespace**

kubectl create ns test  
kubectl create deploy fargate-pod --image=nginx -n test

We can verify the pods are running by running the command:

kubectl get pods -n test -o wide

![](https://cdn-images-1.medium.com/max/1760/1*PTm6m1zp7fljiFs7GNvLLQ.png)

We can see the pod running on the Fargate node.

Now our pod is running as “serverless”. Well done !

**Clean up your environment**

Now make sure to delete your environment so that you don’t get a heavy billing from AWS :)

Make sure you delete the Fargate profile first and then the cluster itself.

eksctl delete fargateprofile --cluster eks-fargate --name fp-test  
eksctl delete cluster --name eks-fargate --region eu-west-1

Thank you and hope you enjoyed this !

References:

[**AWS Fargate profile**  
_Before you can schedule pods on Fargate in your cluster, you must define at least one Fargate profile that specifies…_docs.aws.amazon.com](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html "https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html")[](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html)

[**EKSworkshop.com**  
_The Fargate profile allows an administrator to declare which pods run on Fargate. Each profile can have up to five…_www.eksworkshop.com](https://www.eksworkshop.com/beginner/180_fargate/creating-profile/ "https://www.eksworkshop.com/beginner/180_fargate/creating-profile/")[](https://www.eksworkshop.com/beginner/180_fargate/creating-profile/)

**What’s next?**

*   Look into exposing the deployment through a service?
*   Look into AWS Load Balancer controller?
*   ALB ingress controllers?
*   Setting up CloudWatch logs in order to obtain application logs?

I’m currently on a journey to learn more about the ever-growing ecosystem of containers, container orchestration and the DevOps culture. Expect to see more articles and open-source contributions. Stay tuned for more !

A hands-on article on deploying an application on Kubernetes with Fargate.

Posts tagged with: kubernetes