
## Introduction

### What Are We Building? 🔐

In this deep dive, we’ll secure traffic to an app on GKE using **mutual TLS (mTLS)** at the ingress layer with **GKE Gateway API**. We’ll use self-signed certs to demo client certificate validation—similar to how financial APIs and zero-trust platforms validate service-to-service identity.

### Why This Matters

mTLS is core to secure service meshes and external-facing APIs. Validating clients via TLS certs strengthens your perimeter, especially for sensitive workloads.

## Scenario Overview

- GKE Autopilot cluster (or standard)
- Gateway API enabled with managed Gateway controller
- One app (`httpbin`) behind the Gateway
- Server cert (`tls.crt`, `tls.key`) deployed via `Secret`
- Client cert (`client.crt`) validated against a CA bundle (`ca.crt`)
- All certs are self-signed for demo, in real-world you'd use cert-manager or ACM

## Architecture

![mTLS Gateway](/images/mtls-gke.png)

- Gateway only accepts connections where client cert is signed by known CA
- Ingress TLS termination happens at Gateway
- Backend is HTTP (no mTLS between Gateway and app in this example)

## Prereqs

- GCP project with billing enabled
- `gcloud`, `kubectl`, `openssl`
- Gateway API CRDs installed
- GKE cluster with Gateway controller enabled

## Step-by-Step Setup

### Step 1: Create the Cluster

```bash
gcloud container clusters create-auto mtls-demo \
  --region us-central1
```

### Enable Gateway API

```bash
gcloud container clusters update mtls-demo \
  --enable-gateway-api

# install CRDs if needed
kubectl apply -k "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v1.0.0"
```

### Generate TLS & CA Certs

```bash
mkdir certs && cd certs

# Create CA
openssl req -x509 -nodes -new -sha256 -days 3650 \
  -newkey rsa:2048 -subj "/CN=MyCA" \
  -keyout ca.key -out ca.crt

# Server cert
openssl req -new -newkey rsa:2048 -nodes -keyout server.key \
  -subj "/CN=httpbin.local" -out server.csr

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

# Client cert
openssl req -new -newkey rsa:2048 -nodes -keyout client.key \
  -subj "/CN=test-client" -out client.csr

openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 365
```

### Deploy app & secrets

```bash
kubectl create ns httpbin

kubectl create deployment httpbin --image=kennethreitz/httpbin --port=80 -n httpbin
kubectl expose deployment httpbin --port=80 --target-port=80 -n httpbin

kubectl create secret tls server-cert \
  --cert=certs/server.crt --key=certs/server.key -n httpbin

kubectl create configmap ca-cert --from-file=ca.crt=certs/ca.crt -n httpbin
```

### Create Gateway & Routes

Gateway config (gateway.yaml)
```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: httpbin-gw
  namespace: httpbin
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: server-cert
        options:
          networking.gke.io/client-verification: "require"
          networking.gke.io/trusted-ca: ca-cert
```

Route config (route.yaml)

```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin-route
  namespace: httpbin
spec:
  parentRefs:
    - name: httpbin-gw
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: httpbin
          port: 80
```

Apply the configs:

```bash
kubectl apply -f gateway.yaml
kubectl apply -f route.yaml
```

### Test the setup

Get IP address of the Gateway:

```bash
kubectl get gateway -n httpbin
```

Let’s test with and without the client cert.

✅ Valid client

```bash
curl --cert certs/client.crt --key certs/client.key --cacert certs/ca.crt \
  https://<GATEWAY_IP>/get
```

❌ No client cert

```bash
curl --cacert certs/ca.crt https://<GATEWAY_IP>/get
```

Expected: 400 or 403 depending on GKE's enforcement.

### Automated Testing (Optional)

Create a script to automate the testing.

```bash
#!/bin/bash
URL=https://<GATEWAY_IP>/get
curl --silent --cert certs/client.crt --key certs/client.key --cacert certs/ca.crt $URL | grep "headers"
if [ $? -ne 0 ]; then echo "mTLS test failed"; exit 1; fi
echo "mTLS test passed"
```

## Key Takeaways

- GKE Gateway API supports native mTLS client validation via annotations
- Certs must be manually injected via Secret and ConfigMap
- Only valid clients (with signed certs) can connect

This pattern is extendable to service meshes, APIs, and zero-trust networking

## Code 

- [Blog Code](https://github.com/moanukar/mtls-gke)

## Conclusion

You now have a secure GKE Gateway that validates clients using mutual TLS. In the next part, we'll explore using cert-manager for automated cert issuance and chaining this setup with an internal-only backend.


In this blog, we configure mutual TLS (mTLS) using Gateway API on GKE, securing ingress traffic with client certificate validation.

Secure Gateways: Configuring Mutual TLS using Gateway API on GKE


I tend to create environments for my labs using containers. These containers would hold the resources I need for a certain job. Here's a guide on how I usually create them from scratch.

## Project Structure

Let’s assume we have a folder named `lab-container` with the following structure:

lab-container/ ├── Dockerfile ├── build │ └── binaries.sh ├── .dockerignore ├── Makefile ├── .github │ └── workflows │ └── ci.yml ├── README.md └── zshrc

We'll walk through each file step by step.

---

## The Dockerfile

Here’s a **multi-stage Dockerfile** that compiles and strips binaries in a builder stage, then copies them into a smaller final image:

```bash
FROM alpine:3.19 AS builder

WORKDIR /build

ARG TERRAFORM_VERSION="1.3.9"
ARG VAULT_VERSION="1.12.4"
ARG HELM_VERSION="3.11.2"
ARG KUBECTL_VERSION="v1.26.1"
ARG KIND_VERSION="v0.17.0"
ARG TRAEFIK_VERSION="2.9.6"
ARG TASK_VERSION="3.19.0"

RUN apk update && apk add --no-cache \
      bash \
      curl \
      wget \
      unzip \
      git \
      tar \
      ca-certificates \
      upx \
      binutils  # `strip` and `upx` to reduce binary sizes

COPY build/binaries.sh /build/binaries.sh
RUN chmod +x /build/binaries.sh && /build/binaries.sh

# Optionally strip and compress
RUN strip --strip-unneeded /usr/local/bin/* || true
RUN upx --best --lzma /usr/local/bin/* || true

################################################################################
# Final image
################################################################################
FROM alpine:3.19

LABEL maintainer="YourName" \
      description="A lightweight DevOps toolbox container."

WORKDIR /labs
ENV PATH="/labs/bin:$PATH"

RUN apk update && apk add --no-cache \
      bash \
      ca-certificates \
      python3 \
      py3-pip \
      aws-cli \
      zsh \
      jq \
      fzf \
      eza

COPY --from=builder /build/bin/ /usr/local/bin/

COPY zshrc /root/.zshrc

SHELL ["/bin/zsh", "-c"]

CMD ["zsh"]
```

### Explanation

Builder Stage:

- Installs dependencies (bash, curl, tar, etc.) needed to fetch and install binaries.

- Copies binaries.sh which downloads and installs Terraform, Vault, Helm, etc.

- Uses strip and upx to reduce final binary sizes.

### Final Stage:

- Starts with a fresh Alpine base.

- Installs additional packages (e.g., AWS CLI, zsh, jq, fzf).

- Copies the compiled binaries from the builder stage.

- Sets up a default shell (zsh) and copies a zshrc config file for a nice shell environment.

## build/binaries.sh

This script downloads and installs each DevOps tool. For example:

```bash
#!/usr/bin/env bash
set -euxo pipefail

# In this script, you can use $TERRAFORM_VERSION, $VAULT_VERSION, etc.
# to download each tool. For instance:
wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip
unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin
rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip

# Similarly for Vault:
wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip
unzip vault_${VAULT_VERSION}_linux_amd64.zip -d /usr/local/bin
rm vault_${VAULT_VERSION}_linux_amd64.zip

# ... repeat for Helm, kubectl, kind, Traefik, Task, etc.
# e.g. for kubectl:
curl -L "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" -o /usr/local/bin/kubectl
chmod +x /usr/local/bin/kubectl

# For kind:
curl -Lo /usr/local/bin/kind "https://kind.sigs.k8s.io/dl/${KIND_VERSION}/kind-linux-amd64"
chmod +x /usr/local/bin/kind

# For Task CLI:
curl -L https://github.com/go-task/task/releases/download/v${TASK_VERSION}/task_linux_amd64.tar.gz | tar -zx -C /usr/local/bin
```

Tip: Keep the script idempotent. If you run it twice, it shouldn’t break anything.

## .dockerignore

The .dockerignore helps exclude files you don’t want in your build context (like .git, logs, local environment files, etc.):

```bash
.env
.DS_Store
*.log
.git
.gitignore
README.md
node_modules/
```

Adjust to your needs.

## Makefile

A simple Makefile can streamline common Docker tasks:

```makefile
IMAGE_NAME = your-dockerhub-username/devops-toolbox
VERSION = v1.0.0
LATEST = latest

.PHONY: build
build:
	docker build -t $(IMAGE_NAME):$(LATEST) .

.PHONY: run
run:
	docker run --rm -it $(IMAGE_NAME):$(LATEST) zsh

.PHONY: test
test:
	docker run --rm $(IMAGE_NAME):$(LATEST) sh -c ' \
		echo "Checking installed tools..."; \
		terraform --version && \
		vault --version && \
		aws --version && \
		kubectl version --client && \
		helm version && \
		kind version && \
		traefik version && \
		task --version && \
		docker version \
	'

.PHONY: push
push:
	docker tag $(IMAGE_NAME):$(LATEST) $(IMAGE_NAME):$(VERSION)
	docker push $(IMAGE_NAME):$(VERSION)
	docker push $(IMAGE_NAME):$(LATEST)

.PHONY: clean
clean:
	docker image prune -f

.PHONY: deploy-k8s
deploy-k8s:
	kubectl run toolbox --image=$(IMAGE_NAME):$(LATEST) -- sleep infinity

.PHONY: exec-k8s
exec-k8s:
	kubectl exec -it toolbox -- zsh

```

## Explanation

build: Builds the Docker image locally with the latest tag.

run: Launches the container in an interactive shell.

test: Runs each tool’s --version to confirm they’re installed.

push: Tags and pushes the image to Docker Hub.

deploy-k8s: Runs the container in a Kubernetes pod.

exec-k8s: Opens a shell into that running pod.

## zshrc

For a better shell experience, add your Zsh config:

```bash
export PATH="/usr/local/bin:$PATH"
export EDITOR=vim

# Enable autocompletion
autoload -Uz compinit
compinit

# Enable fzf for fuzzy search
[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh

# Some aliases
alias ls="eza --icons=always"
alias la="eza -al --icons"

# Theme or prompt settings (e.g. powerlevel10k)
ZSH_THEME="powerlevel10k/powerlevel10k"
[[ ! -f ~/.p10k.zsh ]] || source ~/.p10k.zsh
```

More aliases for kubectl, terraform, etc. as desired

## GitHub Actions (Optional)
 
If you want automatic builds when you push changes, create .github/workflows/ci.yml:

```yaml
name: devops-toolbox

on:
  push:
    paths:
      - .github/workflows/ci.yml
      - Dockerfile
      - build/binaries.sh

jobs:
  build-and-publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: write

    steps:
    - name: Check out
      uses: actions/checkout@v2

    - name: Log in to Docker Hub
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

    - name: Build and Publish
      run: |
        registry="your-dockerhub-username"
        sha=$(git rev-parse --short HEAD)
        primary_tag="${registry}/devops-toolbox:${sha}"

        docker build -t $primary_tag -f Dockerfile .
        docker push $primary_tag

        current_branch=$(git rev-parse --abbrev-ref HEAD)
        if [ "$current_branch" = "main" ]; then
          retention_tag="${registry}/devops-toolbox:rc-${sha}"
          docker tag $primary_tag $retention_tag
          docker push $retention_tag
        else
          retention_tag="${registry}/devops-toolbox:alpha-${sha}"
          docker tag $primary_tag $retention_tag
          docker push $retention_tag
        fi
```

When you push changes to GitHub, this workflow will build your Docker image and push it to your Docker Hub account.

## Building and Testing Locally

Build:

```bash
make build
```

`docker build -t your-dockerhub-username/devops-toolbox:latest .`

Test:

```bash
make test
```

This checks each installed tool’s version.

Run:
```bash
make run
```

You should see a Zsh prompt with Terraform, Vault, AWS CLI, Helm, kubectl, etc. available.

## Pushing to Docker Hub
Assuming you’re logged in to Docker Hub (docker login), run:

`make push`

This tags the image with v1.0.0 and latest, then pushes both to your Docker Hub repository.

## Deploying in Kubernetes

If you want to spin up this container in a Kubernetes cluster:

`make deploy-k8s`

It creates a pod named toolbox that sleeps indefinitely. To open a shell inside that pod:

`make exec-k8s`

You’ll then see the same environment as the local container—handy for on-cluster troubleshooting or dev tasks.

## Conclusion

By following these steps, you’ve built a DevOps Toolbox container from scratch. You learned how to:

- Use a multi-stage Dockerfile to install and strip binaries.

- Write a script (binaries.sh) for easy updates to Terraform, Vault, Helm, etc.

- Set up a Zsh environment with helpful aliases.

- Automate builds with Makefile and optionally GitHub Actions.

- Push your image to Docker Hub and deploy it on Kubernetes.

This container simplifies DevOps tasks, letting you quickly spin up a consistent environment anywhere Docker or Kubernetes runs. 


An end-to-end guide for creating a lab container for DevOps training.

Creating a Lab Container


## Introduction

### What Are We Building? 🚀

We're building a **secure GitHub Actions pipeline** that deploys to **GKE** without storing any credentials in GitHub. Instead of using service account keys, we'll use **Workload Identity Federation** (OIDC) so GitHub can impersonate a GCP Service Account.

**No long-lived keys. No kubeconfig. Just secure, modern CI/CD.**

### Why This Matters

- 🔒 No secrets stored in GitHub  
- 🔐 Short-lived, auditable credentials via OIDC  
- 🧑‍💻 First-class support from Google + GitHub  
- 🔁 Reusable pattern for any GCP service, not just GKE  

---

## Architecture

![gha-gke-wif](/images/gha-gke-wif.png)

**Flow:**

1. GitHub Workflow uses OIDC token
2. Google validates identity and issues short-lived credentials
3. Workflow impersonates a GCP Service Account
4. Runs `kubectl` or `gcloud` to deploy to GKE

---

## Prerequisites

- GCP project + billing enabled  
- A GKE Autopilot or Standard cluster  
- GitHub repo (private or public)  
- Terraform 1.3+ and `gcloud` CLI  
- Your app packaged into a container

---

## Terraform Setup

New Autopilot cluster

```hcl
provider "google" {
  project = var.project_id
  region  = var.region
}

resource "google_container_cluster" "autopilot" {
  name     = var.cluster_name
  location = var.region

  enable_autopilot = true

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  release_channel {
    channel = "REGULAR"
  }

  ip_allocation_policy {}
  networking_mode = "VPC_NATIVE"
}
```

### Workload Identity Federation

```hcl
resource "google_iam_workload_identity_pool" "github_pool" {
  workload_identity_pool_id = "github"
  display_name              = "GitHub Actions Pool"
}

resource "google_iam_workload_identity_pool_provider" "github_provider" {
  workload_identity_pool_id = google_iam_workload_identity_pool.github_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-provider"
  display_name = "GitHub OIDC"
  attribute_mapping = {
    "google.subject"         = "assertion.sub"
    "attribute.repository"   = "assertion.repository"
  }

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

resource "google_service_account" "gha_deployer" {
  account_id   = "gha-deployer"
  display_name = "GitHub Actions deployer"
}

resource "google_service_account_iam_member" "allow_impersonation" {
  service_account_id = google_service_account.gha_deployer.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github_pool.name}/attribute.repository/${var.github_repo}"
}

resource "google_project_iam_member" "deploy_permissions" {
  role   = "roles/container.developer"
  member = "serviceAccount:${google_service_account.gha_deployer.email}"
}
```

### Variables

```hcl
variable "project_id" {}
variable "region" {
  default = "us-central1"
}
variable "cluster_name" {
  default = "gha-ci-cluster"
}
variable "github_repo" {
  description = "GitHub repo in format owner/repo"
}
```

### App deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx
        ports:
        - containerPort: 80

```

### GitHub Actions Setup

```yaml
name: Deploy to GKE via Workload Identity

on:
  push:
    branches: [ main ]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Authenticate with Google Cloud
      uses: google-github-actions/auth@v1
      with:
        token_format: "id_token"
        workload_identity_provider: "projects/<PROJECT_NUM>/locations/global/workloadIdentityPools/github/providers/github-provider"
        service_account: "gha-deployer@<PROJECT_ID>.iam.gserviceaccount.com"

    - name: Setup gcloud
      uses: google-github-actions/setup-gcloud@v1

    - name: Configure kubectl
      run: |
        gcloud container clusters get-credentials ${{ secrets.CLUSTER_NAME }} --region ${{ secrets.REGION }}

    - name: Deploy app
      run: |
        kubectl apply -f k8s/
```

### Test it

```bash
git push origin main
```

- GitHub will authenticate using OIDC
- GCP will issue a token
- Workflow will deploy to GKE without a service account key

---

## Key Takeaways

🔑 Workload Identity Federation = no key rotation, no secret sprawl

🛡 GitHub OIDC → GCP IAM is the new best practice

☁️ Works for GKE, Cloud Run, Cloud Storage, etc.

✅ GitHub auth + setup-gcloud = seamless GKE integration

## Conclusion

You've now built a secure, automated CI/CD pipeline using GitHub Actions + GKE + Workload Identity Federation. This is production-grade infra without touching a single JSON/token key.

Give it a try and see how it can simplify your GKE deployments!


In this deep dive, we set up a secure, production-ready CI/CD pipeline from GitHub Actions to GKE using Workload Identity Federation—no secrets needed.

Zero to Production: GitHub Actions CI/CD into GKE with Workload Identity


## TL;DR

- DNS-based GKE endpoints change how public and private control planes can be accessed externally and internally in Google Cloud.
- Private GKE endpoints with internal IPs can now be accessed externally using a DNS-based endpoint—no need for bastion hosts or VPNs.
- Public GKE endpoints can be hardened by layering Cloud IAM authorization on API server requests.
- No other cloud provider offers secure external access to private cluster endpoints.
- DNS-based access works today via gcloud or Terraform on new/existing GKE clusters.

## Background: GKE Cluster Endpoint Models

Traditionally, GKE API servers are accessed via IP-based endpoints:

- Public IP: Globally accessible, optionally restricted via Master Authorized Networks.
- Private IP: Internal-only, routable within VPC. Requires VPN, Interconnect, or bastion host.

## Introducing DNS-Based Endpoints

DNS-based GKE endpoints offer access to the control plane using a cluster-unique FQDN, e.g.:

`gke-xxxxx-xxxxx.europe-west2.gke.google`

This FQDN resolves to a Google Cloud IP, which fronts the GKE API server. This adds an authorization layer via Cloud IAM before requests hit the Kubernetes API server.

Works with both public and private GKE endpoints.

## Architectural Implications

Before:

- Private endpoints required private networking (VPN/Interconnect/Bastion).
- Public endpoints exposed clusters to external traffic directly.

After:

- Private GKE clusters can now be accessed externally via FQDN + IAM auth.
- Public GKE clusters benefit from IAM-based filtering before RBAC applies.

## Authorisation via IAM

Traditional IP-based GKE access:

- Requests go straight to the API server.
- Access restricted via IP + RBAC.

DNS-based endpoint:

- Requests first go through a Google Cloud API.
- IAM checks are enforced, e.g., container.clusters.connect permission.
- Then forwarded to GKE control plane for RBAC enforcement.

```bash
# Grant IAM access
$ gcloud projects add-iam-policy-binding devops-mo \
  --member=serviceAccount:gke-priv-access@devops-mo.iam.gserviceaccount.com \
  --role=roles/container.developer

# Retrieve credentials via DNS endpoint
$ gcloud container clusters get-credentials gke-dns-private \
  --dns-endpoint \
  --location europe-west2
```

IAM failure example

```bash
$ kubectl cluster-info
Error: Permission 'container.clusters.connect' denied on resource
```

Benefits of DNS Endpoint Access

Private Clusters

- No VPNs or bastion hosts required
- Access internal-only GKE API servers using external connectivity
- Removes attack surfaces and infra overhead

Public Clusters

- Enforce IAM on public endpoint traffic
- Block unauthenticated users from reaching /healthz, /version, and discovery APIs

Without DNS Endpoint:

```bash
$ curl -k https://<public-ip>/readyz
$ curl -k https://<public-ip>/version
# Both return 200 OK
```

With DNS Endpoint:

```bash
$ curl -k https://gke-dns-private.europe-west2.gke.google/readyz
$ curl -k https://gke-dns-private.europe-west2.gke.google/version
# Both return 403 Forbidden
```

## Example: Deploying a Private GKE Cluster

Create Private GKE Cluster with DNS Endpoint

```bash
gcloud container clusters create-auto gke-dns-private \
  --enable-dns-access \
  --enable-private-nodes \
  --enable-private-endpoint \
  --location europe-west2 \
  --enable-master-authorized-networks
```

Inspect Cluster Networking

```bash
# Shows internal IP only
$ gcloud container clusters describe example-auto-priv --location europe-west2 --format="value(endpoint)"
10.154.0.13

# Shows FQDN
$ gcloud container clusters describe example-auto-priv --location europe-west2 --format="value(controlPlaneEndpointsConfig.dnsEndpointConfig.endpoint)"
gke-<hash>.europe-west2.gke.goog

# Resolve DNS
$ dig +short gke-<hash>.europe-west2.gke.goog
216.239.32.27
```

Try Direct Internal IP (fails externally)

```bash
$ kubectl config view -o jsonpath='{.clusters[0].cluster.server}'
https://10.154.0.13
$ kubectl get ns
# fails: timeout or unreachable
```

Use FQDN with External Access

```bash
$ gcloud container clusters get-credentials example-auto-priv \
  --dns-endpoint \
  --location europe-west2

$ kubectl cluster-info
Kubernetes control plane is running at https://gke-<hash>.europe-west2.gke.google
```

Public Endpoint Abuse vs IAM Gatekeeping

Problem:

Even restricted public endpoints expose cluster version and health APIs to unauthenticated users.

```bash
# Anonymous access
$ curl -k https://<public-ip>/readyz
$ curl -k https://<public-ip>/version
```

Authenticated (but external) users can access discovery APIs

```bash
$ kubectl auth whoami
Username: mo@devops.com
$ kubectl get --raw='/apis'
# Returns API groups
```

With DNS endpoint enabled:

```bash
$ curl -k https://<fqdn>/version
Error: Permission 'container.clusters.connect' denied
```

Migrating Existing Clusters

```bash
gcloud container clusters update example-auto-priv \
  --enable-dns-access \
  --enable-private-nodes \
  --enable-private-endpoint \
  --location europe-west2
```

or via Terraform;

```hcl
resource "google_container_cluster" "dns_private" {
  name     = "gke-priv-dns"
  enable_autopilot = true

  control_plane_endpoints_config {
    dns_endpoint_config {
      allow_external_traffic = true
    }
  }

  master_authorized_networks_config {}

  private_cluster_config {
    enable_private_nodes = true
    enable_private_endpoint = true
  }
}
# Non-destructive: Can be added to existing clusters.
```


Wrap Up

DNS-based GKE endpoints:

- Enable external access to private clusters without VPNs/bastions
- Secure public endpoints by adding IAM authorization
- Reduce infra complexity and operational overhead
- Eliminate attack surfaces from bastions and public IPs

A clear evolution from IP-based perimeter security to identity-based secure access using Google Cloud’s APIs.
- Similar to Identity-Aware Proxy (IAP), DNS-based endpoints shift access control to the cloud perimeter.

Use DNS endpoints now to modernise your GKE networking and improve your security posture.


Using GKE DNS-based endpoints for Secure cluster access


## Introduction

DNS spoofing in Kubernetes remains a critical threat, enabling attackers to redirect traffic, intercept data, or disrupt services. This article explores how such attacks occur and outlines strategies to prevent them.

## K8s networking

In Kubernetes, pods communicate over a virtual bridge (commonly cbr0) that connects all pods on a node. The cbr0 can also handle ARP (Address Resolution Protocol) resolution. When an incoming packet arrives at cbr0, it can resolve the destination MAC address using ARP.

CoreDNS, running as a pod, handles DNS resolution for the cluster. Each pod's /etc/resolv.conf typically points to the ClusterIP of the CoreDNS service, often 10.96.0.10. Kube-proxy sets up iptables rules to route DNS requests to the appropriate CoreDNS pod.

In essence, this is how pods communicate with each other on the same node. It’s also how Docker works and is the default for K8s.

## Understanding How DNS Works Inside Kubernetes

Let’s break down how DNS requests flow in a Kubernetes cluster — especially how pods talk to CoreDNS.

Each pod in your cluster is set up to use CoreDNS (or kube-dns) as its default DNS server. CoreDNS itself runs as a pod, typically inside the kube-system namespace. You can have multiple CoreDNS pods for HA, but for simplicity, let’s assume one for now.

### How Does a Pod Know Where CoreDNS Lives?

When a pod is created, Kubernetes injects a /etc/resolv.conf file like this:

```bash
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
```

That 10.96.0.10 is the ClusterIP of the kube-dns Service. It’s a stable VIP (virtual IP) used for all DNS requests inside the cluster. CoreDNS pods sit behind this service and receive DNS queries forwarded by kube-proxy using iptables DNAT rules.

### How the Flow Works

1. Your pod makes a DNS request (e.g. nslookup my-service)

2. It hits the VIP (10.96.0.10)

3. kube-proxy rewrites the packet (DNAT) to one of the actual CoreDNS pod IPs

4. CoreDNS checks:
    - Is this a cluster service/pod/etc? → resolve it locally
    - Else → forward to upstream DNS (e.g. Google DNS)

This behavior is defined by the pod’s dnsPolicy, which is set to ClusterFirst by default.

### From kubelet's DNS handling logic:

For a pod with DNSClusterFirst policy,
the cluster DNS server is the only nameserver configured.
CoreDNS will forward queries to upstream nameservers
if it can’t resolve them locally.

### 🧩 Key Point

A pod doesn’t talk to CoreDNS directly by IP — it talks to a Service VIP that’s transparently routed to a pod. This abstraction makes DNS scalable and fault-tolerant, but also opens the door for network-layer attacks like DNS spoofing if a pod has raw socket access.

## The Attack Vector: ARP Spoofing via NET_RAW

By default, Kubernetes grants pods the NET_RAW capability, allowing them to craft raw packets, including ARP messages. An attacker can exploit this to perform ARP spoofing, tricking the network into sending DNS requests to a malicious pod instead of the legitimate CoreDNS pod.

### Steps to Exploit

1. Identify CoreDNS Pod IP: Send a DNS request and observe the source IP of the response to determine the CoreDNS pod's IP.

2. Determine Bridge IP and MAC: Use tools like scapy to send packets with TTL=1 to an external IP, capturing the bridge's IP and MAC address.

3. Send Fake ARP Replies: Continuously send ARP replies to the bridge, associating the CoreDNS IP with the attacker's MAC address.

4. Intercept DNS Requests: Run a DNS proxy in the malicious pod to handle intercepted DNS requests, forwarding them to the real CoreDNS pod or spoofing responses as desired.

This method allows the attacker to intercept and manipulate DNS traffic, potentially redirecting services to malicious endpoints.

## Mitigation Strategies

1. Drop NET_RAW Capability

Modify pod security contexts to drop the NET_RAW capability, preventing pods from crafting raw packets.

```yaml
securityContext:
  capabilities:
    drop:
      - NET_RAW
```

1. Implement Network Policies
Use Kubernetes NetworkPolicies to restrict pod communication, ensuring only authorized pods can communicate with CoreDNS.

1. Isolate CoreDNS Pods
Schedule CoreDNS pods on dedicated nodes and restrict access to these nodes, reducing the risk of compromise.

1. Monitor and Audit Changes
Continuously monitor and audit changes to the CoreDNS ConfigMap and related resources to detect unauthorized modifications promptly.

## Conclusion

DNS spoofing remains a common threat in Kubernetes environments. By understanding the attack vectors and implementing robust security measures, organisations can safeguard their clusters against such threats.


Mo's DevOps Blog

Featured Posts

Latest Posts

Secure Gateways: Configuring Mutual TLS using Gateway API on GKE

Creating a Lab Container

Zero to Production: GitHub Actions CI/CD into GKE with Workload Identity

Using GKE DNS-based endpoints for Secure cluster access

Kubernetes DNS Spoofing: Exploiting NET_RAW and ARP