
## TL;DR

- DNS-based GKE endpoints change how public and private control planes can be accessed externally and internally in Google Cloud.
- Private GKE endpoints with internal IPs can now be accessed externally using a DNS-based endpoint—no need for bastion hosts or VPNs.
- Public GKE endpoints can be hardened by layering Cloud IAM authorization on API server requests.
- No other cloud provider offers secure external access to private cluster endpoints.
- DNS-based access works today via gcloud or Terraform on new/existing GKE clusters.

## Background: GKE Cluster Endpoint Models

Traditionally, GKE API servers are accessed via IP-based endpoints:

- Public IP: Globally accessible, optionally restricted via Master Authorized Networks.
- Private IP: Internal-only, routable within VPC. Requires VPN, Interconnect, or bastion host.

## Introducing DNS-Based Endpoints

DNS-based GKE endpoints offer access to the control plane using a cluster-unique FQDN, e.g.:

`gke-xxxxx-xxxxx.europe-west2.gke.google`

This FQDN resolves to a Google Cloud IP, which fronts the GKE API server. This adds an authorization layer via Cloud IAM before requests hit the Kubernetes API server.

Works with both public and private GKE endpoints.

## Architectural Implications

Before:

- Private endpoints required private networking (VPN/Interconnect/Bastion).
- Public endpoints exposed clusters to external traffic directly.

After:

- Private GKE clusters can now be accessed externally via FQDN + IAM auth.
- Public GKE clusters benefit from IAM-based filtering before RBAC applies.

## Authorisation via IAM

Traditional IP-based GKE access:

- Requests go straight to the API server.
- Access restricted via IP + RBAC.

DNS-based endpoint:

- Requests first go through a Google Cloud API.
- IAM checks are enforced, e.g., container.clusters.connect permission.
- Then forwarded to GKE control plane for RBAC enforcement.

```bash
# Grant IAM access
$ gcloud projects add-iam-policy-binding devops-mo \
  --member=serviceAccount:gke-priv-access@devops-mo.iam.gserviceaccount.com \
  --role=roles/container.developer

# Retrieve credentials via DNS endpoint
$ gcloud container clusters get-credentials gke-dns-private \
  --dns-endpoint \
  --location europe-west2
```

IAM failure example

```bash
$ kubectl cluster-info
Error: Permission 'container.clusters.connect' denied on resource
```

Benefits of DNS Endpoint Access

Private Clusters

- No VPNs or bastion hosts required
- Access internal-only GKE API servers using external connectivity
- Removes attack surfaces and infra overhead

Public Clusters

- Enforce IAM on public endpoint traffic
- Block unauthenticated users from reaching /healthz, /version, and discovery APIs

Without DNS Endpoint:

```bash
$ curl -k https://<public-ip>/readyz
$ curl -k https://<public-ip>/version
# Both return 200 OK
```

With DNS Endpoint:

```bash
$ curl -k https://gke-dns-private.europe-west2.gke.google/readyz
$ curl -k https://gke-dns-private.europe-west2.gke.google/version
# Both return 403 Forbidden
```

## Example: Deploying a Private GKE Cluster

Create Private GKE Cluster with DNS Endpoint

```bash
gcloud container clusters create-auto gke-dns-private \
  --enable-dns-access \
  --enable-private-nodes \
  --enable-private-endpoint \
  --location europe-west2 \
  --enable-master-authorized-networks
```

Inspect Cluster Networking

```bash
# Shows internal IP only
$ gcloud container clusters describe example-auto-priv --location europe-west2 --format="value(endpoint)"
10.154.0.13

# Shows FQDN
$ gcloud container clusters describe example-auto-priv --location europe-west2 --format="value(controlPlaneEndpointsConfig.dnsEndpointConfig.endpoint)"
gke-<hash>.europe-west2.gke.goog

# Resolve DNS
$ dig +short gke-<hash>.europe-west2.gke.goog
216.239.32.27
```

Try Direct Internal IP (fails externally)

```bash
$ kubectl config view -o jsonpath='{.clusters[0].cluster.server}'
https://10.154.0.13
$ kubectl get ns
# fails: timeout or unreachable
```

Use FQDN with External Access

```bash
$ gcloud container clusters get-credentials example-auto-priv \
  --dns-endpoint \
  --location europe-west2

$ kubectl cluster-info
Kubernetes control plane is running at https://gke-<hash>.europe-west2.gke.google
```

Public Endpoint Abuse vs IAM Gatekeeping

Problem:

Even restricted public endpoints expose cluster version and health APIs to unauthenticated users.

```bash
# Anonymous access
$ curl -k https://<public-ip>/readyz
$ curl -k https://<public-ip>/version
```

Authenticated (but external) users can access discovery APIs

```bash
$ kubectl auth whoami
Username: mo@devops.com
$ kubectl get --raw='/apis'
# Returns API groups
```

With DNS endpoint enabled:

```bash
$ curl -k https://<fqdn>/version
Error: Permission 'container.clusters.connect' denied
```

Migrating Existing Clusters

```bash
gcloud container clusters update example-auto-priv \
  --enable-dns-access \
  --enable-private-nodes \
  --enable-private-endpoint \
  --location europe-west2
```

or via Terraform;

```hcl
resource "google_container_cluster" "dns_private" {
  name     = "gke-priv-dns"
  enable_autopilot = true

  control_plane_endpoints_config {
    dns_endpoint_config {
      allow_external_traffic = true
    }
  }

  master_authorized_networks_config {}

  private_cluster_config {
    enable_private_nodes = true
    enable_private_endpoint = true
  }
}
# Non-destructive: Can be added to existing clusters.
```


Wrap Up

DNS-based GKE endpoints:

- Enable external access to private clusters without VPNs/bastions
- Secure public endpoints by adding IAM authorization
- Reduce infra complexity and operational overhead
- Eliminate attack surfaces from bastions and public IPs

A clear evolution from IP-based perimeter security to identity-based secure access using Google Cloud’s APIs.
- Similar to Identity-Aware Proxy (IAP), DNS-based endpoints shift access control to the cloud perimeter.

Use DNS endpoints now to modernise your GKE networking and improve your security posture.


Using GKE DNS-based endpoints for Secure cluster access


## Introduction

DNS spoofing in Kubernetes remains a critical threat, enabling attackers to redirect traffic, intercept data, or disrupt services. This article explores how such attacks occur and outlines strategies to prevent them.

## K8s networking

In Kubernetes, pods communicate over a virtual bridge (commonly cbr0) that connects all pods on a node. The cbr0 can also handle ARP (Address Resolution Protocol) resolution. When an incoming packet arrives at cbr0, it can resolve the destination MAC address using ARP.

CoreDNS, running as a pod, handles DNS resolution for the cluster. Each pod's /etc/resolv.conf typically points to the ClusterIP of the CoreDNS service, often 10.96.0.10. Kube-proxy sets up iptables rules to route DNS requests to the appropriate CoreDNS pod.

In essence, this is how pods communicate with each other on the same node. It’s also how Docker works and is the default for K8s.

## Understanding How DNS Works Inside Kubernetes

Let’s break down how DNS requests flow in a Kubernetes cluster — especially how pods talk to CoreDNS.

Each pod in your cluster is set up to use CoreDNS (or kube-dns) as its default DNS server. CoreDNS itself runs as a pod, typically inside the kube-system namespace. You can have multiple CoreDNS pods for HA, but for simplicity, let’s assume one for now.

### How Does a Pod Know Where CoreDNS Lives?

When a pod is created, Kubernetes injects a /etc/resolv.conf file like this:

```bash
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
```

That 10.96.0.10 is the ClusterIP of the kube-dns Service. It’s a stable VIP (virtual IP) used for all DNS requests inside the cluster. CoreDNS pods sit behind this service and receive DNS queries forwarded by kube-proxy using iptables DNAT rules.

### How the Flow Works

1. Your pod makes a DNS request (e.g. nslookup my-service)

2. It hits the VIP (10.96.0.10)

3. kube-proxy rewrites the packet (DNAT) to one of the actual CoreDNS pod IPs

4. CoreDNS checks:
    - Is this a cluster service/pod/etc? → resolve it locally
    - Else → forward to upstream DNS (e.g. Google DNS)

This behavior is defined by the pod’s dnsPolicy, which is set to ClusterFirst by default.

### From kubelet's DNS handling logic:

For a pod with DNSClusterFirst policy,
the cluster DNS server is the only nameserver configured.
CoreDNS will forward queries to upstream nameservers
if it can’t resolve them locally.

### 🧩 Key Point

A pod doesn’t talk to CoreDNS directly by IP — it talks to a Service VIP that’s transparently routed to a pod. This abstraction makes DNS scalable and fault-tolerant, but also opens the door for network-layer attacks like DNS spoofing if a pod has raw socket access.

## The Attack Vector: ARP Spoofing via NET_RAW

By default, Kubernetes grants pods the NET_RAW capability, allowing them to craft raw packets, including ARP messages. An attacker can exploit this to perform ARP spoofing, tricking the network into sending DNS requests to a malicious pod instead of the legitimate CoreDNS pod.

### Steps to Exploit

1. Identify CoreDNS Pod IP: Send a DNS request and observe the source IP of the response to determine the CoreDNS pod's IP.

2. Determine Bridge IP and MAC: Use tools like scapy to send packets with TTL=1 to an external IP, capturing the bridge's IP and MAC address.

3. Send Fake ARP Replies: Continuously send ARP replies to the bridge, associating the CoreDNS IP with the attacker's MAC address.

4. Intercept DNS Requests: Run a DNS proxy in the malicious pod to handle intercepted DNS requests, forwarding them to the real CoreDNS pod or spoofing responses as desired.

This method allows the attacker to intercept and manipulate DNS traffic, potentially redirecting services to malicious endpoints.

## Mitigation Strategies

1. Drop NET_RAW Capability

Modify pod security contexts to drop the NET_RAW capability, preventing pods from crafting raw packets.

```yaml
securityContext:
  capabilities:
    drop:
      - NET_RAW
```

1. Implement Network Policies
Use Kubernetes NetworkPolicies to restrict pod communication, ensuring only authorized pods can communicate with CoreDNS.

1. Isolate CoreDNS Pods
Schedule CoreDNS pods on dedicated nodes and restrict access to these nodes, reducing the risk of compromise.

1. Monitor and Audit Changes
Continuously monitor and audit changes to the CoreDNS ConfigMap and related resources to detect unauthorized modifications promptly.

## Conclusion

DNS spoofing remains a common threat in Kubernetes environments. By understanding the attack vectors and implementing robust security measures, organisations can safeguard their clusters against such threats.


Kubernetes DNS Spoofing: Exploiting NET_RAW and ARP


## Introduction

In this deep dive, we'll explore how to configure AWS Route 53 to route user traffic to the region with the lowest latency and automatically fail over to another region if the primary becomes unhealthy. This setup ensures high availability and optimal performance for global applications.

Why Should You Care?
Implementing latency-based routing with health checks in Route 53 allows your application to:

- Serve users from the nearest healthy region, reducing latency.
- Automatically reroute traffic during regional outages, enhancing resilience.
- Maintain high availability without manual intervention.

## Prerequisites

- Basic understanding of AWS Route 53 and DNS concepts.
- An AWS account with Route 53 and CloudWatch access.
- A domain name registered with Route 53 (if hosted on Cloudflare or elsewhere, make sure you delegate the domain to Route 53 or create a subdomain to subdelegate).
- Terraform installed on your local machine.

## Architecture Overview

We'll deploy identical applications in two AWS regions: us-east-1 and eu-west-1. Each region will have:

- An Application Load Balancer (ALB) fronting the application.
- A Route 53 health check monitoring the ALB's /health endpoint.
- A latency-based DNS record directing traffic to the region with the lowest latency.

If a health check fails, Route 53 will exclude that region from DNS responses, effectively failing over to the healthy region.

## Step-by-Step Guide

1. Set Up ALBs in Both Regions
Deploy your application behind an ALB in both us-east-1 (US Virginia) and eu-west-1 (EU Ireland). Ensure each ALB has a listener on port 80 and a target group with healthy targets.

2. Create Route 53 Health Checks
Define health checks for each ALB's /health endpoint.

```go
resource "aws_route53_health_check" "us_east" {
  fqdn              = "alb-us-east-1.example.com"
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30
}

resource "aws_route53_health_check" "eu_west" {
  fqdn              = "alb-eu-west-1.example.com"
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30
}
```

## 3. Configure Latency-Based DNS Records

Create latency-based DNS records pointing to each ALB, associating them with the respective health checks.

```go
resource "aws_route53_record" "us_east" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "r53-demo.moabukar.co.uk"
  type    = "A"

  alias {
    name                   = "alb-us-east-1.example.com"
    zone_id                = "Z35SXDOTRQ7X7K" // Replace with your ALB's zone ID
    evaluate_target_health = true
  }

  set_identifier = "us-east-1"
  region         = "us-east-1"
  latency_routing_policy {
    region = "us-east-1"
  }

  health_check_id = aws_route53_health_check.us_east.id
}

resource "aws_route53_record" "eu_west" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "r53-demo.moabukar.co.uk"
  type    = "A"

  alias {
    name                   = "alb-eu-west-1.example.com"
    zone_id                = "Z32O12XQLNTSW2" // Replace with your ALB's zone ID
    evaluate_target_health = true
  }

  set_identifier = "eu-west-1"
  region         = "eu-west-1"
  latency_routing_policy {
    region = "eu-west-1"
  }

  health_check_id = aws_route53_health_check.eu_west.id
}
```

Note: Replace zone_id values with the correct zone IDs for your ALBs. You can find these in the AWS documentation.

## Apply on TF

```bash
terraform init
terraform apply
```

Verify that the DNS records are created and health checks are active.

## Test the Setup

- Normal Operation: When both regions are healthy, Route 53 directs users to the region with the lowest latency based on their location.

- Simulate Failure: Stop the application in us-east-1 to trigger a health check failure.

- Failover: Route 53 detects the failure and stops including us-east-1 in DNS responses. Traffic is rerouted to eu-west-1.

- Recovery: Restart the application in us-east-1. Once the health check passes, Route 53 includes it again in DNS responses.

## Monitoring and Observability

- Route 53 Console: Monitor health check statuses and DNS records.

- CloudWatch: Set up alarms for health check failures to receive notifications.

- Logs: Enable query logging in Route 53 to analyze DNS queries.

## Conclusion

By configuring latency-based routing with health checks in Route 53, you ensure your application serves users from the nearest healthy region, providing low latency and high availability. This setup is crucial for global applications where performance and uptime are paramount.

References:

- [AWS Route 53 Latency-Based Routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html)

- [AWS Route 53 Health Checks](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html)

- [Terraform AWS Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route53_health_check)


A hands-on guide to configuring AWS Route 53 for latency-based routing across multiple regions, incorporating health checks for automatic failover.

Route 53 Deep Dive: Multi-Region Latency Routing with Health-Based Failover


## ⚠️ DNS over UDP: The Silent Limitation Wasting Your Resources

When you scale horizontally, whether it’s pods behind a headless service in Kubernetes or VMs behind a DNS-based load balancer, you probably assume all your instances are equally reachable. But there's a catch most miss until they start debugging strange connection drops…

DNS over UDP silently drops anything after ~8 A records.

Let’s break this down.

## Quick Background: DNS & UDP

By default, DNS uses UDP, not TCP. That’s because it’s faster and usually sufficient for small requests/responses. But:

- UDP has a 512-byte payload size limit (RFC 1035).
- That means for a DNS A-record response, you’ll only get ~8 IPs max (depending on domain length and other metadata).
- If the record size exceeds that, the server sets a flag: "truncated", hinting that the full answer is available—but only over TCP.

Now here’s the kicker: most clients (e.g. dig, Kubernetes DNS, Go resolver, Python resolver, even libc-based resolvers) do not retry over TCP unless explicitly told to.

## Problem In Action: Real Example

Take a look at this domain: testing.moabukar.co.uk

I’ve created 10 A records behind that domain, pointing to various internal IPs:

![dig](/images/cloudflare-records-mo.png)

But when we run a dig:

```bash
dig testing.moabukar.co.uk
```

We only get 8 IPs back:

![dig](/images/dig-moabukar.png)

Despite the DNS backend (Cloudflare) holding all 10 entries, we never see more than 8.

This means 2 instances are completely unreachable from any client doing a standard UDP DNS query.

## 🛑 Why This Is Bad

- Wasted compute – The missing IPs are still running services and consuming resources.
- Reduced reliability – Your load balancer isn’t as balanced as you thought.
- Hidden bugs – Sporadic connectivity issues are hard to trace, especially when the resolution behavior is inconsistent across environments.

## 🔥 How to Reproduce

- Register a domain and use Cloudflare or Route53.
- Add 9+ A records with the same name (e.g. testing.example.com).
- Run `dig testing.example.com +short`
- Count the records returned when you run dig. You’ll likely see only 8.

## 💡 Solutions & Workarounds

✅ 1. Use TCP explicitly (when possible)

```bash
dig +tcp testing.example.com
```

But this isn’t viable for most client applications.

✅ 2. Use multiple DNS records (e.g. testing1, testing2)
Shard your A records logically.

✅ 3. Use a Load Balancer or Ingress controller
Let it manage the IP-to-service mapping, rather than relying on DNS for client-side load balancing.

✅ 4. Use SRV or CNAME (if app supports it)
SRV records often use less space and allow port/service separation too.

✅ 5. Reduce number of targets behind a single DNS name
For critical paths, consider spreading across multiple records.

## How Kubernetes Handles This

Kubernetes headless services (ClusterIP: None) are backed by multiple A records pointing to pod IPs. This exact issue applies!
  - If your headless service has >8 pods, some will be unreachable via DNS if your app relies on client-side balancing.
  - Kube-proxy and service IP-based routing avoids this, but not all clients use it (e.g. StatefulSets that do direct DNS resolution).

## Summary

UDP DNS resolution has a fundamental limitation:

- You only get as many A records as fit in 512 bytes—usually around 8 entries max.
- This can silently kill your scaling, break client-side discovery, and waste infrastructure.
  
Always audit your DNS queries when scaling out.


Posts tagged with: dns