
## Kubernetes Cluster Upgrades: No BS Guide

Managed Kubernetes clusters need regular upgrades for security patches, bug fixes, and new features. This is a technical guide covering GKE, EKS, and AKS upgrades with real-world procedures and rollback strategies.

### Prerequisites: Do This First

1. Deprecated API Detection
Using kubent (Universal):

```bash
# install kubent
sh -c "$(curl -sSL https://git.io/install-kubent)"

# scan cluster
kubent --target-version=1.31.0
kubent --output=json --target-version=1.31.0
```

GKE Log Explorer Query:
```sql
sqlresource.type="k8s_cluster"
labels."k8s.io.removed-release"="1.31"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system")
```

AWS CloudTrail (EKS):

```bash
# check EKS API calls for deprecated versions
aws logs filter-log-events --log-group-name /aws/eks/cluster/your-cluster/cluster
```

2. Compatibility Matrix Check

- GKE: Verify Anthos Service Mesh/Istio compatibility
- EKS: Check AWS Load Balancer Controller, EBS CSI driver versions
- AKS: Validate Azure CNI, Application Gateway Ingress Controller

3. Resource Assessment

```bash
# check cluster capacity
kubectl top nodes
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# review PDBs
kubectl get pdb --all-namespaces

# look for critical workloads
kubectl get deployments,statefulsets --all-namespaces
```

### Upgrade Strategy

### Environment Progression

Dev/Test clusters → 2. Staging → 3. Production (least critical → most critical)

### Timing

- Off-peak hours
- Team availability for monitoring
- Consider maintenance windows for automatic upgrades

Platform-Specific Procedures

### GKE Upgrade

Control Plane:

```bash
# via gcloud
gcloud container clusters upgrade CLUSTER_NAME \
  --zone=us-central1-a \
  --master \
  --cluster-version=1.31.0-gke.1234567

# monitor
gcloud container operations list --filter="TYPE:UPGRADE_CLUSTER"
Node Pools (Configure surge settings first):
# set surge parameters
gcloud container node-pools update NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a \
  --max-surge=20 \
  --max-unavailable=0

# upgrade nodes
gcloud container node-pools upgrade NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a
```

### EKS Upgrade

Control Plane:

```bash
# update cluster version
aws eks update-cluster-version \
  --region us-west-2 \
  --name my-cluster \
  --kubernetes-version 1.31

# monitor status
aws eks describe-cluster --region us-west-2 --name my-cluster
```

### Node Groups:

```bash
# update managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodes \
  --region us-west-2 \
  --kubernetes-version 1.31

# for self-managed: update launch template, then rolling update
Add-ons:
# update critical add-ons
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --region us-west-2
```

### AKS Upgrade

Control Plane:

```bash
# get available versions
az aks get-versions --location eastus --output table

# upgrade cluster
az aks upgrade \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.31.0
```

### Node Pools:

```bash
# upgrade specific node pool
az aks nodepool upgrade \
  --resource-group myResourceGroup \
  --cluster-name myAKSCluster \
  --name mynodepool \
  --kubernetes-version 1.31.0
```

### Post-Upgrade Validation

### Health Checks

```bash
# verify pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# check nodes
kubectl get nodes
kubectl describe nodes | grep -E "Conditions|Taints"

# component status
kubectl get componentstatuses
```

### Application Testing

- Critical endpoint validation
- Database connectivity
- Ingress/LoadBalancer functionality
- Monitor metrics and logs

### Node Issues

```bash
# check for common upgrade problems
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl logs --selector=app=node-problem-detector -n kube-system

# look for: "task hung", blocked processes, resource pressure
```

### Rollback Procedures

### Control Plane Rollback

GKE:

```bash
# only within same minor version
gcloud container clusters upgrade CLUSTER_NAME \
  --master \
  --cluster-version=1.30.5-gke.previous
```

EKS/AKS: Control plane rollback not supported. Node pool rollback only.

### Node Pool Rollback Strategy

### Method 1: New Node Pool (Recommended)

```bash
# GKE
gcloud container node-pools create rollback-pool \
  --cluster=CLUSTER_NAME \
  --node-version=1.30.5-gke.previous \
  --num-nodes=3

# scale up new pool
gcloud container node-pools resize rollback-pool \
  --cluster=CLUSTER_NAME \
  --num-nodes=10
```

### Method 2: Workload Migration

```bash
# cordon old nodes
kubectl cordon NODE_NAME

# force workload restart to migrate
for ns in $(kubectl get ns -o name | cut -d'/' -f2); do
  if [[ "$ns" != "kube-system" ]]; then
    echo "Restarting $ns"
    kubectl -n $ns rollout restart deployment
    kubectl -n $ns rollout restart statefulset  
    kubectl -n $ns rollout restart daemonset
  fi
done

# verify distribution
kubectl get pods -o wide --all-namespaces

# remove old pool when stable
```

### Advanced Strategies

### Blue-Green Node Pool Upgrade

1. create new node pool with target version
2. Migrate workloads using node selectors/taints
3. Validate functionality
4. Complete migration
5. Remove old pool

### Surge Configuration Best Practices

- Small clusters (<10 nodes): maxSurge=1, maxUnavailable=0
- Large clusters (>50 nodes): maxSurge=20, maxUnavailable=0
- Resource-constrained: maxSurge=0, maxUnavailable=1

Infrastructure as Code Updates

Terraform:

```go
# GKE 
resource "google_container_cluster" "primary" {
  min_master_version = "1.31.0-gke.1234567"
}
```

```go
# EKS  
resource "aws_eks_cluster" "cluster" {
  version = "1.31"
}

# AKS
resource "azurerm_kubernetes_cluster" "cluster" {
  kubernetes_version = "1.31.0"
}
```
Apply with no-op verification:

```bash
# should show no changes post-upgrade
terraform plan 
terraform apply
```

### Monitoring During Upgrades

#### Key Metrics

- Pod scheduling latency
- Node resource utilization
- API server response times
- Application error rates

#### Critical Events

- Node cordoning/draining
- Pod eviction failures
- PDB violations
- Failed scheduling

### Common Issues & Solutions

#### Stuck Node Upgrades:

- Check resource quotas
- Verify image pull capacity
- Review PodDisruptionBudgets

#### Application Failures:

- Validate deprecated API usage
- Check resource requests/limits
- Review network policies

#### Performance Degradation:

- Monitor resource pressure
- Check for node resource fragmentation
- Validate autoscaling configuration

### Platform-Specific Gotchas

#### GKE:

- Automatic node auto-upgrades can conflict with manual upgrades
- Regional clusters take 2x longer to upgrade
-Surge upgrades require additional quotas

#### EKS:

- Add-on compatibility critical (CNI, CSI drivers)
- Self-managed nodes require separate upgrade process
- IAM roles may need updates

#### AKS:

- Azure CNI version compatibility
- System node pools upgrade differently
- Virtual node pools have separate lifecycle


**Reality Check:** Upgrades rarely go perfectly. Plan for 2x the estimated time, have rollback procedures tested and monitor everything. 

This guide reflects real production experience. Test everything in non-prod/staging/dev first, document your specific procedures and build confidence through repetition.


Technical guide for upgrading managed Kubernetes clusters across GKE, EKS, and AKS

Kubernetes Cluster Upgrades: Production-Ready Guide


## A Real Talk on GKE Upgrades: What They Don't Tell You in the Docs

**Fair warning: This is going to be a long read!** But if you're responsible for production GKE clusters, every minute you spend here could save you hours of stress and potential downtime later.

I've been through the trenches of GKE cluster upgrades more times than I care to count — from smooth sailing 15-minute upgrades to absolute nightmare scenarios that had me questioning my career choices at 2 AM. The Google docs will tell you how to click the buttons and run the commands, but they won't prepare you for the reality of what happens when Murphy's Law kicks in during your upgrade window.

This guide isn't just another regurgitation of official documentation. It's battle-tested wisdom from someone who's learned the hard way that "it worked in staging" doesn't guarantee anything, and that the most critical step in any upgrade isn't technical — it's having a solid plan for when everything goes sideways.

## Kubernetes Cluster Upgrades: Beyond Basic Operations

Managing Google Kubernetes Engine (GKE) clusters in production isn't just about keeping workloads running — it's about maintaining security, performance, and compatibility while navigating the rapidly evolving Kubernetes ecosystem. GKE automatically upgrades the version of the control plane and nodes to ensure that the cluster receives new features, bug fixes, and security patches, but successful production upgrades require careful planning and execution.

With the frequent updates and patches from upstream Kubernetes, and GKE, we highly recommend testing new releases on testing and/or staging environments before the releases are rolled out into your production environment, especially Kubernetes minor version upgrades. The challenge isn't just technical — it's operational. How do you minimize disruption? What happens when things go wrong? How do you ensure your applications remain available throughout the process?

This comprehensive guide walks through a battle-tested approach to GKE upgrades, from initial planning through rollback procedures, designed for production environments where downtime isn't an option.

### Pre-Flight Checks: The Foundation of Safe Upgrades

Before touching any cluster configuration, thorough preparation prevents most upgrade failures. These checks aren't just recommendations — they're insurance against production outages.

#### API Deprecation Detection

A few deprecated APIs have been kept around in the last couple of K8s versions and finally getting completely removed in Kubernetes 1.16 release and this pattern continues with every release. The most critical step is identifying resources using deprecated APIs that will break in your target version.

#### Method 1: Using kubent (Recommended)

Kube No Trouble (kubent) is a simple tool to check whether you're using any of these API versions in your cluster and therefore should upgrade your workloads first, before upgrading your Kubernetes cluster.

```bash
bash# Install kubent
sh -c "$(curl -sSL https://git.io/install-kubent)"

# Scan your cluster for deprecated APIs
kubent --target-version=1.31.0

# Get JSON output for automation
kubent --target-version=1.31.0 --output=json
```

#### Method 2: GCP Log Explorer Query

For clusters with audit logging enabled, use this query to identify deprecated API usage:

```sql
sqlresource.type="k8s_cluster"
labels."k8s.io.removed-release"="1.31"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system")
```

#### Method 3: Native kubectl Discovery

```bash
bash# Check supported API versions
kubectl api-versions

# Verify specific resources exist in new format
kubectl explain deployment --api-version=apps/v1
```

#### Service Mesh Compatibility Verification

If running Istio, Anthos Service Mesh, or other service mesh solutions, verify compatibility with your target GKE version. GKE supports a minor version by providing patch versions of the same minor release, and, on a regular basis, automatically upgrading clusters to those newer patches, but service mesh components may have different support matrices.
Check the official compatibility documentation for your service mesh version against the target GKE version before proceeding.

#### Resource and Workload Assessment

Review your cluster's resources and workloads to ensure they're compatible with the target GKE version. This includes checking for deprecated APIs, unsupported features and any other compatibility issues.

```bash
# check cluster resource utils
kubectl top nodes
kubectl top pods --all-namespaces

# check for critical workloads
kubectl get deployments --all-namespaces -o wide
kubectl get statefulsets --all-namespaces -o wide

# review PDBs...
kubectl get pdb --all-namespaces
```

### Strategic Upgrade Planning: Minimize Risk, Maximize Success

#### Environment-Based Rollout Strategy

As part of your workflow for delivering software updates, we recommend that you use multiple environments. Multiple environments help you minimize risk and unwanted downtime by testing software and infrastructure updates separately from your production environment.

#### Recommended Progression:

1. Development/Testing Clusters: Start with non-critical environments
2. Staging Clusters: Full production-like testing with realistic workloads
3. Production Clusters: Begin with least critical, progress to most critical

#### Timing Considerations

Schedule upgrades during maintenance windows when impact on users and engineering teams is minimal. Consider:

- Peak usage periods for your applications
- Team availability for monitoring and response
- Dependency on external services that might be affected

#### Release Channel Strategy

To keep clusters up-to-date with the latest GKE and Kubernetes updates, here are some recommended environments and the respective release channels the clusters should be enrolled in:

- Development: Rapid channel for early testing
- Staging: Regular channel for stability validation
- Production: Stable or Extended channel for proven reliability

### Executing the Upgrade: Step-by-Step Process

#### Phase 1: Control Plane Upgrade

The control plane upgrade typically completes in 10-15 minutes for zonal clusters, 20-30 minutes for regional clusters.

#### Via Google Cloud Console:

1. Navigate to Kubernetes Engine > Clusters
2. Select target cluster
3. Click Upgrade Available next to Version
4. Select desired version (increment by one minor version)
5. Click Save Changes

#### Via gcloud CLI:

```bash
bash# Check available versions
gcloud container get-server-config --zone=us-central1-a

# Upgrade control plane
gcloud container clusters upgrade CLUSTER_NAME \
    --zone=us-central1-a \
    --master \
    --cluster-version=1.31.0-gke.1234567
```

#### Via Terraform:

Update your Terraform configuration and apply:

```hcl
resource "google_container_cluster" "primary" {
  name               = "my-cluster"
  location           = "us-central1-a"
  min_master_version = "1.31.0-gke.1234567"
  # ... other configuration
}
```

#### Phase 2: Node Pool Upgrades

GKE chooses the following strategies for these specific scenarios: In Autopilot clusters, GKE uses surge upgrades. For Standard clusters, configure surge settings for optimal balance between speed and disruption.

#### Configure Surge Upgrade Settings:

For large clusters where the upgrade process might take longer, you can accelerate the upgrade completion time by concurrently upgrading multiple nodes at a time. Use surge upgrade with maxSurge=20, maxUnavailable=0 to instruct GKE to upgrade 20 nodes at a time, without using any existing capacity.

```bash
# Configure surge settings before upgrade
gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --max-surge=20 \
    --max-unavailable=0

# Upgrade node pool
gcloud container node-pools upgrade NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --node-version=1.31.0-gke.1234567
```

#### Monitor Upgrade Progress:

```bash
bash# Check node status
kubectl get nodes -o wide

# Monitor pod scheduling
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# Watch for node cordoning/draining events
kubectl get events --sort-by=.metadata.creationTimestamp
```

#### Phase 3: Post-Upgrade Validation

#### System Health Checks:

```bash
bash# Verify all pods are running
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# Check node readiness
kubectl get nodes | grep -v Ready

# Validate cluster components
kubectl get componentstatuses
```

#### Application Validation:

- Test critical application endpoints
- Verify database connectivity
- Check monitoring and alerting systems
- Validate ingress and load balancer functionality

#### Node Resource Inspection:

```bash
# Check for common upgrade issues in node logs
# Look for patterns like "task hung" or blocked processes
kubectl logs --selector=app=node-problem-detector -n kube-system
```

#### Monitoring and Observability During Upgrades

#### Key Metrics to Watch

- Node resource utilization (CPU, Memory, Disk)
- Pod scheduling latency
- Application response times
- Error rates and HTTP status codes
- Resource quota usage

#### Critical Events to Monitor

- Node cordoning and draining events
- Pod eviction failures
- PDB violations
- Failed pod scheduling
- Container runtime errors

#### Logging Strategy

```bash
# Implement structured logging to capture upgrade events:
# Monitor upgrade operations

```bash
gcloud container operations list --filter="TYPE:UPGRADE_CLUSTER"

# Get detailed operation status
gcloud container operations describe OPERATION_ID --zone=us-central1-a
```

### Rollback Procedures: When Things Go Wrong

Despite careful planning, upgrades sometimes fail. Having a tested rollback strategy is crucial for production environments.

#### Immediate Response Protocol

When to Trigger Rollback:

- Application functionality severely degraded
- Critical system components failing
- Node upgrade stuck for >2 hours
- Widespread pod scheduling failures

#### Control Plane Rollback

To mitigate an unsuccessful cluster control plane upgrade, you can downgrade your control plane to a previous patch release if the version is an earlier patch release within the same minor version.

```bash
# Rollback control plane (within same minor version only)
gcloud container clusters upgrade CLUSTER_NAME \
    --zone=us-central1-a \
    --master \
    --cluster-version=1.30.5-gke.1234567  # Previous patch version
```

#### Node Pool Rollback via New Pool Strategy

For comprehensive rollback, create a new node pool with the previous version:
Step 1: Create New Node Pool

```bash
gcloud container node-pools create rollback-pool \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --node-version=1.30.5-gke.1234567 \
    --num-nodes=3 \
    --machine-type=e2-standard-4
```

#### Step 2: Scale Up New Pool

```bash
bashgcloud container node-pools resize rollback-pool \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --num-nodes=10  # Match original capacity
```

#### Step 3: Drain and Migrate Workloads

```bash
bash# Cordon old nodes
kubectl cordon NODE_NAME

# Force restart all workloads to migrate
for ns in $(kubectl get ns -o name | cut -d'/' -f2); do
    if [[ "$ns" != "kube-system" ]]; then
        echo "Restarting workloads in namespace: $ns"
        kubectl -n $ns rollout restart deployment
        kubectl -n $ns rollout restart statefulset
        kubectl -n $ns rollout restart daemonset
    fi
done
```

#### Step 4: Verify and Clean Up

```bash
bash# Verify pod distribution
kubectl get pods --all-namespaces -o wide

# Delete old node pool once stable
gcloud container node-pools delete old-pool-name \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a
```

#### Application-Level Recovery

If infrastructure rollback isn't sufficient:

- Restore application deployments to previous versions
- Rollback database schema changes if applicable
- Reset configuration maps and secrets
- Verify external service integrations

### Advanced Upgrade Strategies

#### Blue-Green Node Pool Upgrades

Blue-green upgrades: Existing nodes are kept available for rolling back while the workloads are validated on the new node configuration.
For zero-downtime upgrades of critical workloads:

- Create new node pool with upgraded version
Gradually migrate workloads using node selectors
- Validate functionality on new nodes
- Complete migration and remove old pool
- Keep old pool available for quick rollback if needed

#### Gradual Workload Migration

Use node taints and tolerations for controlled migration:

```yaml
# Taint new nodes
kubectl taint nodes NEW_NODE_NAME upgraded=true:NoSchedule

# Update deployments with toleration
spec:
  template:
    spec:
      tolerations:
      - key: "upgraded"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
```

#### Maintenance Windows and Exclusions

```bash
Maintenance Windows and Exclusions
Configure maintenance policies to control when automatic upgrades occur:

```bash
gcloud container clusters update CLUSTER_NAME \
    --maintenance-window-start="2025-01-15T02:00:00Z" \
    --maintenance-window-end="2025-01-15T06:00:00Z" \
    --maintenance-window-recurrence="FREQ=WEEKLY;BYDAY=SU"
```

#### Infrastructure as Code Integration

```bash
Terraform State Management
After successful upgrades, update Terraform configuration to match actual state:

```hcl
resource "google_container_cluster" "primary" {
  name               = "production-cluster"
  location           = "us-central1-a"
  min_master_version = "1.31.0-gke.1234567"
  
  node_config {
    machine_type = "e2-standard-4"
  }
  
  # Ensure this matches post-upgrade state
  remove_default_node_pool = true
}
```

Apply changes with plan review:

```bash
terraform plan  # Should show no changes
terraform apply
```

#### GitOps Integration

```bash
Update cluster manifests in your GitOps repositories to reflect new API versions and configurations discovered during the upgrade process.
```

#### Post-Upgrade Housekeeping

##### Documentation Updates

- Update runbooks with any new procedures discovered
- Document any application-specific compatibility issues
- Record upgrade duration and resource consumption
- Update disaster recovery procedures

#### Security Review

- Verify RBAC configurations are intact
- Check network policies and security contexts
- Validate service mesh security policies
- Review admission controller configurations

#### Performance Baseline

Establish new performance baselines post-upgrade:

- Application response times
- Resource utilization patterns
- Scaling behavior
- Cost implications

### The Reality Check: What Actually Happens

Kubernetes upgrades in production rarely go exactly as planned. Even with perfect preparation, you might encounter:

- Resource Constraints: Nodes created by surge upgrade are subject to your Google Cloud resource quotas, resource availability, and reservation capacity
- Application Dependencies: Third-party services may have compatibility issues
- Timing Conflicts: At X we've been running production workloads on kubernetes in GKE since early 2017. When we were at 20 nodes it might take 90–120 minutes, which is in a tolerable range. Our largest nodepool in production at the time was 55 nodes, which would have taken at least 6 hours to fully upgrade

The key is building processes that expect and handle these realities gracefully.

### Best Practices Summary

Before Every Upgrade:

- Run deprecated API scans with kubent
Test in non-production environments first
- Verify service mesh compatibility
- Configure appropriate surge settings
- Plan for rollback scenarios

During Upgrades:

- Monitor cluster and application health continuously
- Have rollback procedures ready to execute
- Communicate status to stakeholders
- Document any unexpected issues

After Upgrades:

- Validate all critical functionality
- Update infrastructure code
- Review and improve procedures
Plan timing for dependent system updates

Successful GKE upgrades aren't just about technical execution — they're about building reliable, repeatable processes that minimize risk while keeping your Kubernetes infrastructure current, secure, and performant. The investment in proper upgrade procedures pays dividends in reduced downtime, improved security posture, and engineering team confidence.

Remember: These actions ensure your cluster remains performant, secure, and up-to-date with the latest features and bug fixes. In the rapidly evolving Kubernetes ecosystem, staying current isn't optional — it's a competitive advantage.

### The Unvarnished Truth: Lessons from the Field

After years of managing GKE upgrades across different organizations, environments, and scales, here's what I wish someone had told me when I started:

**The docs lie about timing.** That "15-30 minute" control plane upgrade? Plan for double that. The "4-5 minutes per node"? Add a buffer. I've seen single node replacements take 20+ minutes because of slow image pulls, stuck pods, or resource constraints nobody anticipated.

**Your applications are more fragile than you think.** Even when you've tested everything in staging, production has a way of exposing edge cases. That legacy service that "just works"? It might be the one that breaks everything during a rolling restart. Always have a communications plan ready — not just for your team, but for stakeholders who need to know why their dashboard went red.

**Rollbacks are harder than upgrades.** The docs make rollbacks sound easy, but in reality, rolling back a failed upgrade often involves more complexity than the original upgrade. You're dealing with potentially corrupted state, confused monitoring systems, and applications that might have partially migrated their data models. Practice your rollback procedures as much as your upgrades.

**The real cost isn't downtime — it's trust.** A botched upgrade doesn't just impact your applications; it impacts your team's confidence in making future changes. Build processes that your team trusts, document everything (especially the failures), and celebrate the upgrades that go smoothly. They're more valuable than you think.

**Google's automatic upgrades aren't your enemy.** I used to fight the automatic upgrade system, trying to control every aspect. But I've learned that working with GKE's upgrade patterns, rather than against them, leads to more reliable outcomes. Use maintenance windows, embrace the surge upgrade strategies, and let Google handle what they do best.

**Your monitoring during upgrades needs to be different.** Normal monitoring tells you what's broken; upgrade monitoring needs to tell you what's about to break. Watch leading indicators: pod scheduling latency, resource pressure, API response times. By the time your standard alerts fire, you're already in reactive mode.

The biggest lesson? **Upgrades are never just about Kubernetes.** They're about organizational change management, risk tolerance, team communication and building systems that can evolve safely. Master those aspects and the technical pieces become much more manageable.

Stay curious, stay prepared and remember — every upgrade is a learning opportunity, even when (especially when) things don't go according to plan. Good luck!!!!

Comprehensive guide for safely upgrading GKE clusters with minimal downtime and robust rollback procedures

GKE Upgrade Guide and Rollback Strategy: A Production-Ready Approach


## Introduction

### What Are We Building? 🔐

In this deep dive, we’ll secure traffic to an app on GKE using **mutual TLS (mTLS)** at the ingress layer with **GKE Gateway API**. We’ll use self-signed certs to demo client certificate validation—similar to how financial APIs and zero-trust platforms validate service-to-service identity.

### Why This Matters

mTLS is core to secure service meshes and external-facing APIs. Validating clients via TLS certs strengthens your perimeter, especially for sensitive workloads.

## Scenario Overview

- GKE Autopilot cluster (or standard)
- Gateway API enabled with managed Gateway controller
- One app (`httpbin`) behind the Gateway
- Server cert (`tls.crt`, `tls.key`) deployed via `Secret`
- Client cert (`client.crt`) validated against a CA bundle (`ca.crt`)
- All certs are self-signed for demo, in real-world you'd use cert-manager or ACM

## Architecture

![mTLS Gateway](/images/mtls-gke.png)

- Gateway only accepts connections where client cert is signed by known CA
- Ingress TLS termination happens at Gateway
- Backend is HTTP (no mTLS between Gateway and app in this example)

## Prereqs

- GCP project with billing enabled
- `gcloud`, `kubectl`, `openssl`
- Gateway API CRDs installed
- GKE cluster with Gateway controller enabled

## Step-by-Step Setup

### Step 1: Create the Cluster

```bash
gcloud container clusters create-auto mtls-demo \
  --region us-central1
```

### Enable Gateway API

```bash
gcloud container clusters update mtls-demo \
  --enable-gateway-api

# install CRDs if needed
kubectl apply -k "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v1.0.0"
```

### Generate TLS & CA Certs

```bash
mkdir certs && cd certs

# Create CA
openssl req -x509 -nodes -new -sha256 -days 3650 \
  -newkey rsa:2048 -subj "/CN=MyCA" \
  -keyout ca.key -out ca.crt

# Server cert
openssl req -new -newkey rsa:2048 -nodes -keyout server.key \
  -subj "/CN=httpbin.local" -out server.csr

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

# Client cert
openssl req -new -newkey rsa:2048 -nodes -keyout client.key \
  -subj "/CN=test-client" -out client.csr

openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 365
```

### Deploy app & secrets

```bash
kubectl create ns httpbin

kubectl create deployment httpbin --image=kennethreitz/httpbin --port=80 -n httpbin
kubectl expose deployment httpbin --port=80 --target-port=80 -n httpbin

kubectl create secret tls server-cert \
  --cert=certs/server.crt --key=certs/server.key -n httpbin

kubectl create configmap ca-cert --from-file=ca.crt=certs/ca.crt -n httpbin
```

### Create Gateway & Routes

Gateway config (gateway.yaml)
```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: httpbin-gw
  namespace: httpbin
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: server-cert
        options:
          networking.gke.io/client-verification: "require"
          networking.gke.io/trusted-ca: ca-cert
```

Route config (route.yaml)

```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin-route
  namespace: httpbin
spec:
  parentRefs:
    - name: httpbin-gw
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: httpbin
          port: 80
```

Apply the configs:

```bash
kubectl apply -f gateway.yaml
kubectl apply -f route.yaml
```

### Test the setup

Get IP address of the Gateway:

```bash
kubectl get gateway -n httpbin
```

Let’s test with and without the client cert.

✅ Valid client

```bash
curl --cert certs/client.crt --key certs/client.key --cacert certs/ca.crt \
  https://<GATEWAY_IP>/get
```

❌ No client cert

```bash
curl --cacert certs/ca.crt https://<GATEWAY_IP>/get
```

Expected: 400 or 403 depending on GKE's enforcement.

### Automated Testing (Optional)

Create a script to automate the testing.

```bash
#!/bin/bash
URL=https://<GATEWAY_IP>/get
curl --silent --cert certs/client.crt --key certs/client.key --cacert certs/ca.crt $URL | grep "headers"
if [ $? -ne 0 ]; then echo "mTLS test failed"; exit 1; fi
echo "mTLS test passed"
```

## Key Takeaways

- GKE Gateway API supports native mTLS client validation via annotations
- Certs must be manually injected via Secret and ConfigMap
- Only valid clients (with signed certs) can connect

This pattern is extendable to service meshes, APIs, and zero-trust networking

## Code 

- [Blog Code](https://github.com/moanukar/mtls-gke)

## Conclusion

You now have a secure GKE Gateway that validates clients using mutual TLS. In the next part, we'll explore using cert-manager for automated cert issuance and chaining this setup with an internal-only backend.


In this blog, we configure mutual TLS (mTLS) using Gateway API on GKE, securing ingress traffic with client certificate validation.

Secure Gateways: Configuring Mutual TLS using Gateway API on GKE


## Introduction

### What Are We Building? 🚀

We're building a **secure GitHub Actions pipeline** that deploys to **GKE** without storing any credentials in GitHub. Instead of using service account keys, we'll use **Workload Identity Federation** (OIDC) so GitHub can impersonate a GCP Service Account.

**No long-lived keys. No kubeconfig. Just secure, modern CI/CD.**

### Why This Matters

- 🔒 No secrets stored in GitHub  
- 🔐 Short-lived, auditable credentials via OIDC  
- 🧑‍💻 First-class support from Google + GitHub  
- 🔁 Reusable pattern for any GCP service, not just GKE  

---

## Architecture

![gha-gke-wif](/images/gha-gke-wif.png)

**Flow:**

1. GitHub Workflow uses OIDC token
2. Google validates identity and issues short-lived credentials
3. Workflow impersonates a GCP Service Account
4. Runs `kubectl` or `gcloud` to deploy to GKE

---

## Prerequisites

- GCP project + billing enabled  
- A GKE Autopilot or Standard cluster  
- GitHub repo (private or public)  
- Terraform 1.3+ and `gcloud` CLI  
- Your app packaged into a container

---

## Terraform Setup

New Autopilot cluster

```hcl
provider "google" {
  project = var.project_id
  region  = var.region
}

resource "google_container_cluster" "autopilot" {
  name     = var.cluster_name
  location = var.region

  enable_autopilot = true

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  release_channel {
    channel = "REGULAR"
  }

  ip_allocation_policy {}
  networking_mode = "VPC_NATIVE"
}
```

### Workload Identity Federation

```hcl
resource "google_iam_workload_identity_pool" "github_pool" {
  workload_identity_pool_id = "github"
  display_name              = "GitHub Actions Pool"
}

resource "google_iam_workload_identity_pool_provider" "github_provider" {
  workload_identity_pool_id = google_iam_workload_identity_pool.github_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-provider"
  display_name = "GitHub OIDC"
  attribute_mapping = {
    "google.subject"         = "assertion.sub"
    "attribute.repository"   = "assertion.repository"
  }

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

resource "google_service_account" "gha_deployer" {
  account_id   = "gha-deployer"
  display_name = "GitHub Actions deployer"
}

resource "google_service_account_iam_member" "allow_impersonation" {
  service_account_id = google_service_account.gha_deployer.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github_pool.name}/attribute.repository/${var.github_repo}"
}

resource "google_project_iam_member" "deploy_permissions" {
  role   = "roles/container.developer"
  member = "serviceAccount:${google_service_account.gha_deployer.email}"
}
```

### Variables

```hcl
variable "project_id" {}
variable "region" {
  default = "us-central1"
}
variable "cluster_name" {
  default = "gha-ci-cluster"
}
variable "github_repo" {
  description = "GitHub repo in format owner/repo"
}
```

### App deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx
        ports:
        - containerPort: 80

```

### GitHub Actions Setup

```yaml
name: Deploy to GKE via Workload Identity

on:
  push:
    branches: [ main ]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Authenticate with Google Cloud
      uses: google-github-actions/auth@v1
      with:
        token_format: "id_token"
        workload_identity_provider: "projects/<PROJECT_NUM>/locations/global/workloadIdentityPools/github/providers/github-provider"
        service_account: "gha-deployer@<PROJECT_ID>.iam.gserviceaccount.com"

    - name: Setup gcloud
      uses: google-github-actions/setup-gcloud@v1

    - name: Configure kubectl
      run: |
        gcloud container clusters get-credentials ${{ secrets.CLUSTER_NAME }} --region ${{ secrets.REGION }}

    - name: Deploy app
      run: |
        kubectl apply -f k8s/
```

### Test it

```bash
git push origin main
```

- GitHub will authenticate using OIDC
- GCP will issue a token
- Workflow will deploy to GKE without a service account key

---

## Key Takeaways

🔑 Workload Identity Federation = no key rotation, no secret sprawl

🛡 GitHub OIDC → GCP IAM is the new best practice

☁️ Works for GKE, Cloud Run, Cloud Storage, etc.

✅ GitHub auth + setup-gcloud = seamless GKE integration

## Conclusion

You've now built a secure, automated CI/CD pipeline using GitHub Actions + GKE + Workload Identity Federation. This is production-grade infra without touching a single JSON/token key.

Give it a try and see how it can simplify your GKE deployments!


In this deep dive, we set up a secure, production-ready CI/CD pipeline from GitHub Actions to GKE using Workload Identity Federation—no secrets needed.

Zero to Production: GitHub Actions CI/CD into GKE with Workload Identity


## TL;DR

- DNS-based GKE endpoints change how public and private control planes can be accessed externally and internally in Google Cloud.
- Private GKE endpoints with internal IPs can now be accessed externally using a DNS-based endpoint—no need for bastion hosts or VPNs.
- Public GKE endpoints can be hardened by layering Cloud IAM authorization on API server requests.
- No other cloud provider offers secure external access to private cluster endpoints.
- DNS-based access works today via gcloud or Terraform on new/existing GKE clusters.

## Background: GKE Cluster Endpoint Models

Traditionally, GKE API servers are accessed via IP-based endpoints:

- Public IP: Globally accessible, optionally restricted via Master Authorized Networks.
- Private IP: Internal-only, routable within VPC. Requires VPN, Interconnect, or bastion host.

## Introducing DNS-Based Endpoints

DNS-based GKE endpoints offer access to the control plane using a cluster-unique FQDN, e.g.:

`gke-xxxxx-xxxxx.europe-west2.gke.google`

This FQDN resolves to a Google Cloud IP, which fronts the GKE API server. This adds an authorization layer via Cloud IAM before requests hit the Kubernetes API server.

Works with both public and private GKE endpoints.

## Architectural Implications

Before:

- Private endpoints required private networking (VPN/Interconnect/Bastion).
- Public endpoints exposed clusters to external traffic directly.

After:

- Private GKE clusters can now be accessed externally via FQDN + IAM auth.
- Public GKE clusters benefit from IAM-based filtering before RBAC applies.

## Authorisation via IAM

Traditional IP-based GKE access:

- Requests go straight to the API server.
- Access restricted via IP + RBAC.

DNS-based endpoint:

- Requests first go through a Google Cloud API.
- IAM checks are enforced, e.g., container.clusters.connect permission.
- Then forwarded to GKE control plane for RBAC enforcement.

```bash
# Grant IAM access
$ gcloud projects add-iam-policy-binding devops-mo \
  --member=serviceAccount:gke-priv-access@devops-mo.iam.gserviceaccount.com \
  --role=roles/container.developer

# Retrieve credentials via DNS endpoint
$ gcloud container clusters get-credentials gke-dns-private \
  --dns-endpoint \
  --location europe-west2
```

IAM failure example

```bash
$ kubectl cluster-info
Error: Permission 'container.clusters.connect' denied on resource
```

Benefits of DNS Endpoint Access

Private Clusters

- No VPNs or bastion hosts required
- Access internal-only GKE API servers using external connectivity
- Removes attack surfaces and infra overhead

Public Clusters

- Enforce IAM on public endpoint traffic
- Block unauthenticated users from reaching /healthz, /version, and discovery APIs

Without DNS Endpoint:

```bash
$ curl -k https://<public-ip>/readyz
$ curl -k https://<public-ip>/version
# Both return 200 OK
```

With DNS Endpoint:

```bash
$ curl -k https://gke-dns-private.europe-west2.gke.google/readyz
$ curl -k https://gke-dns-private.europe-west2.gke.google/version
# Both return 403 Forbidden
```

## Example: Deploying a Private GKE Cluster

Create Private GKE Cluster with DNS Endpoint

```bash
gcloud container clusters create-auto gke-dns-private \
  --enable-dns-access \
  --enable-private-nodes \
  --enable-private-endpoint \
  --location europe-west2 \
  --enable-master-authorized-networks
```

Inspect Cluster Networking

```bash
# Shows internal IP only
$ gcloud container clusters describe example-auto-priv --location europe-west2 --format="value(endpoint)"
10.154.0.13

# Shows FQDN
$ gcloud container clusters describe example-auto-priv --location europe-west2 --format="value(controlPlaneEndpointsConfig.dnsEndpointConfig.endpoint)"
gke-<hash>.europe-west2.gke.goog

# Resolve DNS
$ dig +short gke-<hash>.europe-west2.gke.goog
216.239.32.27
```

Try Direct Internal IP (fails externally)

```bash
$ kubectl config view -o jsonpath='{.clusters[0].cluster.server}'
https://10.154.0.13
$ kubectl get ns
# fails: timeout or unreachable
```

Use FQDN with External Access

```bash
$ gcloud container clusters get-credentials example-auto-priv \
  --dns-endpoint \
  --location europe-west2

$ kubectl cluster-info
Kubernetes control plane is running at https://gke-<hash>.europe-west2.gke.google
```

Public Endpoint Abuse vs IAM Gatekeeping

Problem:

Even restricted public endpoints expose cluster version and health APIs to unauthenticated users.

```bash
# Anonymous access
$ curl -k https://<public-ip>/readyz
$ curl -k https://<public-ip>/version
```

Authenticated (but external) users can access discovery APIs

```bash
$ kubectl auth whoami
Username: mo@devops.com
$ kubectl get --raw='/apis'
# Returns API groups
```

With DNS endpoint enabled:

```bash
$ curl -k https://<fqdn>/version
Error: Permission 'container.clusters.connect' denied
```

Migrating Existing Clusters

```bash
gcloud container clusters update example-auto-priv \
  --enable-dns-access \
  --enable-private-nodes \
  --enable-private-endpoint \
  --location europe-west2
```

or via Terraform;

```hcl
resource "google_container_cluster" "dns_private" {
  name     = "gke-priv-dns"
  enable_autopilot = true

  control_plane_endpoints_config {
    dns_endpoint_config {
      allow_external_traffic = true
    }
  }

  master_authorized_networks_config {}

  private_cluster_config {
    enable_private_nodes = true
    enable_private_endpoint = true
  }
}
# Non-destructive: Can be added to existing clusters.
```


Wrap Up

DNS-based GKE endpoints:

- Enable external access to private clusters without VPNs/bastions
- Secure public endpoints by adding IAM authorization
- Reduce infra complexity and operational overhead
- Eliminate attack surfaces from bastions and public IPs

A clear evolution from IP-based perimeter security to identity-based secure access using Google Cloud’s APIs.
- Similar to Identity-Aware Proxy (IAP), DNS-based endpoints shift access control to the cloud perimeter.

Use DNS endpoints now to modernise your GKE networking and improve your security posture.


Using GKE DNS-based endpoints for Secure cluster access

Posts tagged with: gke