
## Kubernetes Cluster Upgrades: No BS Guide

Managed Kubernetes clusters need regular upgrades for security patches, bug fixes, and new features. This is a technical guide covering GKE, EKS, and AKS upgrades with real-world procedures and rollback strategies.

### Prerequisites: Do This First

1. Deprecated API Detection
Using kubent (Universal):

```bash
# install kubent
sh -c "$(curl -sSL https://git.io/install-kubent)"

# scan cluster
kubent --target-version=1.31.0
kubent --output=json --target-version=1.31.0
```

GKE Log Explorer Query:
```sql
sqlresource.type="k8s_cluster"
labels."k8s.io.removed-release"="1.31"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system")
```

AWS CloudTrail (EKS):

```bash
# check EKS API calls for deprecated versions
aws logs filter-log-events --log-group-name /aws/eks/cluster/your-cluster/cluster
```

2. Compatibility Matrix Check

- GKE: Verify Anthos Service Mesh/Istio compatibility
- EKS: Check AWS Load Balancer Controller, EBS CSI driver versions
- AKS: Validate Azure CNI, Application Gateway Ingress Controller

3. Resource Assessment

```bash
# check cluster capacity
kubectl top nodes
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# review PDBs
kubectl get pdb --all-namespaces

# look for critical workloads
kubectl get deployments,statefulsets --all-namespaces
```

### Upgrade Strategy

### Environment Progression

Dev/Test clusters → 2. Staging → 3. Production (least critical → most critical)

### Timing

- Off-peak hours
- Team availability for monitoring
- Consider maintenance windows for automatic upgrades

Platform-Specific Procedures

### GKE Upgrade

Control Plane:

```bash
# via gcloud
gcloud container clusters upgrade CLUSTER_NAME \
  --zone=us-central1-a \
  --master \
  --cluster-version=1.31.0-gke.1234567

# monitor
gcloud container operations list --filter="TYPE:UPGRADE_CLUSTER"
Node Pools (Configure surge settings first):
# set surge parameters
gcloud container node-pools update NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a \
  --max-surge=20 \
  --max-unavailable=0

# upgrade nodes
gcloud container node-pools upgrade NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a
```

### EKS Upgrade

Control Plane:

```bash
# update cluster version
aws eks update-cluster-version \
  --region us-west-2 \
  --name my-cluster \
  --kubernetes-version 1.31

# monitor status
aws eks describe-cluster --region us-west-2 --name my-cluster
```

### Node Groups:

```bash
# update managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodes \
  --region us-west-2 \
  --kubernetes-version 1.31

# for self-managed: update launch template, then rolling update
Add-ons:
# update critical add-ons
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --region us-west-2
```

### AKS Upgrade

Control Plane:

```bash
# get available versions
az aks get-versions --location eastus --output table

# upgrade cluster
az aks upgrade \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.31.0
```

### Node Pools:

```bash
# upgrade specific node pool
az aks nodepool upgrade \
  --resource-group myResourceGroup \
  --cluster-name myAKSCluster \
  --name mynodepool \
  --kubernetes-version 1.31.0
```

### Post-Upgrade Validation

### Health Checks

```bash
# verify pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# check nodes
kubectl get nodes
kubectl describe nodes | grep -E "Conditions|Taints"

# component status
kubectl get componentstatuses
```

### Application Testing

- Critical endpoint validation
- Database connectivity
- Ingress/LoadBalancer functionality
- Monitor metrics and logs

### Node Issues

```bash
# check for common upgrade problems
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl logs --selector=app=node-problem-detector -n kube-system

# look for: "task hung", blocked processes, resource pressure
```

### Rollback Procedures

### Control Plane Rollback

GKE:

```bash
# only within same minor version
gcloud container clusters upgrade CLUSTER_NAME \
  --master \
  --cluster-version=1.30.5-gke.previous
```

EKS/AKS: Control plane rollback not supported. Node pool rollback only.

### Node Pool Rollback Strategy

### Method 1: New Node Pool (Recommended)

```bash
# GKE
gcloud container node-pools create rollback-pool \
  --cluster=CLUSTER_NAME \
  --node-version=1.30.5-gke.previous \
  --num-nodes=3

# scale up new pool
gcloud container node-pools resize rollback-pool \
  --cluster=CLUSTER_NAME \
  --num-nodes=10
```

### Method 2: Workload Migration

```bash
# cordon old nodes
kubectl cordon NODE_NAME

# force workload restart to migrate
for ns in $(kubectl get ns -o name | cut -d'/' -f2); do
  if [[ "$ns" != "kube-system" ]]; then
    echo "Restarting $ns"
    kubectl -n $ns rollout restart deployment
    kubectl -n $ns rollout restart statefulset  
    kubectl -n $ns rollout restart daemonset
  fi
done

# verify distribution
kubectl get pods -o wide --all-namespaces

# remove old pool when stable
```

### Advanced Strategies

### Blue-Green Node Pool Upgrade

1. create new node pool with target version
2. Migrate workloads using node selectors/taints
3. Validate functionality
4. Complete migration
5. Remove old pool

### Surge Configuration Best Practices

- Small clusters (<10 nodes): maxSurge=1, maxUnavailable=0
- Large clusters (>50 nodes): maxSurge=20, maxUnavailable=0
- Resource-constrained: maxSurge=0, maxUnavailable=1

Infrastructure as Code Updates

Terraform:

```go
# GKE 
resource "google_container_cluster" "primary" {
  min_master_version = "1.31.0-gke.1234567"
}
```

```go
# EKS  
resource "aws_eks_cluster" "cluster" {
  version = "1.31"
}

# AKS
resource "azurerm_kubernetes_cluster" "cluster" {
  kubernetes_version = "1.31.0"
}
```
Apply with no-op verification:

```bash
# should show no changes post-upgrade
terraform plan 
terraform apply
```

### Monitoring During Upgrades

#### Key Metrics

- Pod scheduling latency
- Node resource utilization
- API server response times
- Application error rates

#### Critical Events

- Node cordoning/draining
- Pod eviction failures
- PDB violations
- Failed scheduling

### Common Issues & Solutions

#### Stuck Node Upgrades:

- Check resource quotas
- Verify image pull capacity
- Review PodDisruptionBudgets

#### Application Failures:

- Validate deprecated API usage
- Check resource requests/limits
- Review network policies

#### Performance Degradation:

- Monitor resource pressure
- Check for node resource fragmentation
- Validate autoscaling configuration

### Platform-Specific Gotchas

#### GKE:

- Automatic node auto-upgrades can conflict with manual upgrades
- Regional clusters take 2x longer to upgrade
-Surge upgrades require additional quotas

#### EKS:

- Add-on compatibility critical (CNI, CSI drivers)
- Self-managed nodes require separate upgrade process
- IAM roles may need updates

#### AKS:

- Azure CNI version compatibility
- System node pools upgrade differently
- Virtual node pools have separate lifecycle


**Reality Check:** Upgrades rarely go perfectly. Plan for 2x the estimated time, have rollback procedures tested and monitor everything. 

This guide reflects real production experience. Test everything in non-prod/staging/dev first, document your specific procedures and build confidence through repetition.


Technical guide for upgrading managed Kubernetes clusters across GKE, EKS, and AKS

Kubernetes Cluster Upgrades: Production-Ready Guide


## 🔒 Introduction

Running Kubernetes clusters privately is a growing best practice. In this blog, I’ll walk you through deploying a **private AKS cluster** on Azure with **no public API endpoint**, and enabling secure access via **Twingate VPN**, which provides identity-based access without opening up your network.

This setup is ideal if:

- You want private networking in AKS (via Azure Private Link)
- You need granular access control over the cluster
- You want to avoid managing full VPN appliances or bastion hosts

---

### What We'll Build

- A private AKS cluster with no public API server
- A Twingate Connector running in the same VNet
- Twingate configured with the AKS API server as a protected resource
- Remote access to the cluster using `kubectl` via Twingate

---

### Infrastructure Setup

- Provision a private AKS cluster

You can use `az cli` or Terraform (recommended):

```bash
az aks create \
  --name dev-private \
  --resource-group dev-private_group \
  --enable-private-cluster \
  --vnet-subnet-id /subscriptions/.../subnets/aks-subnet \
  --node-count 2 \
  --generate-ssh-keys
```

Terraform:

```hcl
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "rg-private-aks"
  location = "UK South"
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-private-aks"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "aks_subnet" {
  name                 = "snet-aks"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "private-aks-cluster"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "privateaks"

  default_node_pool {
    name       = "default"
    node_count = 2
    vm_size    = "Standard_DS2_v2"
    vnet_subnet_id = azurerm_subnet.aks_subnet.id
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin    = "azure"
    dns_service_ip    = "10.0.2.10"
    service_cidr      = "10.0.2.0/24"
    docker_bridge_cidr = "172.17.0.1/16"
  }

  api_server_access_profile {
    enable_private_cluster = true
  }

  tags = {
    Environment = "Private"
  }
}
```

The --enable-private-cluster flag ensures the Kubernetes API server is only accessible over the VNet.

To check the API server endpoint:

```bash
az aks show --name dev-private --resource-group dev-private_group --query privateFqdn
```

You’ll get something like:

```bash
dev-private-dns-xxxxxx.privatelink.uksouth.azmk8s.io
```

---

### Push Twingate Connector Image to Azure Container Registry (ACR)

If using a private ACR:

```bash
az acr login --name <your-acr>
docker pull --platform linux/amd64 twingate/connector:1
docker tag twingate/connector:1 <your-acr>.azurecr.io/twingate/connector:1
docker push <your-acr>.azurecr.io/twingate/connector:1
```

### Deploy Twingate Connector as Azure Container Instance

```bash
az container create \
  --name twingate-connector \
  --image <your-acr>.azurecr.io/twingate/connector:1 \
  --resource-group dev-private_group \
  --vnet dev-private_group-vnet \
  --subnet twingate \
  --cpu 1 \
  --memory 2 \
  --environment-variables \
      TWINGATE_NETWORK="mo-demo" \
      TWINGATE_ACCESS_TOKEN="<your-access-token>" \
      TWINGATE_REFRESH_TOKEN="<your-refresh-token>" \
      TWINGATE_TIMESTAMP_FORMAT="2" \
      TWINGATE_LABEL_DEPLOYED_BY="azure"
```

Terraform:

```hcl
resource "azurerm_container_group" "twingate_connector" {
  name                = "twingate-connector"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  os_type             = "Linux"

  container {
    name   = "twingate"
    image  = "youracrname.azurecr.io/twingate-connector:1"
    cpu    = "1"
    memory = "1.5"

    environment_variables = {
      TWINGATE_NETWORK        = "your_network"
      TWINGATE_ACCESS_TOKEN   = "your_access_token"
      TWINGATE_REFRESH_TOKEN  = "your_refresh_token"
      TWINGATE_LABEL_DEPLOYED_BY = "terraform"
    }
  }

  ip_address_type = "Private"
  subnet_ids      = [azurerm_subnet.aks_subnet.id]
}
```

### Configure Twingate

- Setup a Twingate account
- Create a new remote network in Twingate

###  Add the AKS API Server as a Twingate Resource

In the Twingate Admin Console:

- Create a new Remote Network for your VNet (e.g., aks-vnet)
- Deploy the Connector to that Remote Network (you already did)
- Add a Resource with the private AKS API DNS name
  - e.g., dev-private-dns-xxxxxx.privatelink.uksouth.azmk8s.io
  - Port: 443
- Assign the Resource to a group (e.g., engineering)

### Access the Cluster via Twingate

Once the Connector is live and you're authenticated via the Twingate client:

```bash
az aks get-credentials --resource-group dev-private_group --name dev-private
kubectl get nodes
```

It works—because your local traffic is tunneled securely through Twingate and routed to the private API server over the VNet!

You can also run commands remotely using:

```bash
az aks command invoke \
  --resource-group dev-private_group \
  --name dev-private \
  --command "kubectl get pods -A"
```

### Bonus: Ingress Whitelisting

If you're using an NGINX ingress controller, and want to restrict access to known IPs:

```bash
nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.4.5
```

### Summary

With this setup:

- Your AKS cluster is completely private
- Access is secured and identity-aware via Twingate
- No public exposure, no bastion, no hassle


Posts tagged with: aks