
## Kubernetes Cluster Upgrades: No BS Guide

Managed Kubernetes clusters need regular upgrades for security patches, bug fixes, and new features. This is a technical guide covering GKE, EKS, and AKS upgrades with real-world procedures and rollback strategies.

### Prerequisites: Do This First

1. Deprecated API Detection
Using kubent (Universal):

```bash
# install kubent
sh -c "$(curl -sSL https://git.io/install-kubent)"

# scan cluster
kubent --target-version=1.31.0
kubent --output=json --target-version=1.31.0
```

GKE Log Explorer Query:
```sql
sqlresource.type="k8s_cluster"
labels."k8s.io.removed-release"="1.31"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system")
```

AWS CloudTrail (EKS):

```bash
# check EKS API calls for deprecated versions
aws logs filter-log-events --log-group-name /aws/eks/cluster/your-cluster/cluster
```

2. Compatibility Matrix Check

- GKE: Verify Anthos Service Mesh/Istio compatibility
- EKS: Check AWS Load Balancer Controller, EBS CSI driver versions
- AKS: Validate Azure CNI, Application Gateway Ingress Controller

3. Resource Assessment

```bash
# check cluster capacity
kubectl top nodes
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# review PDBs
kubectl get pdb --all-namespaces

# look for critical workloads
kubectl get deployments,statefulsets --all-namespaces
```

### Upgrade Strategy

### Environment Progression

Dev/Test clusters → 2. Staging → 3. Production (least critical → most critical)

### Timing

- Off-peak hours
- Team availability for monitoring
- Consider maintenance windows for automatic upgrades

Platform-Specific Procedures

### GKE Upgrade

Control Plane:

```bash
# via gcloud
gcloud container clusters upgrade CLUSTER_NAME \
  --zone=us-central1-a \
  --master \
  --cluster-version=1.31.0-gke.1234567

# monitor
gcloud container operations list --filter="TYPE:UPGRADE_CLUSTER"
Node Pools (Configure surge settings first):
# set surge parameters
gcloud container node-pools update NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a \
  --max-surge=20 \
  --max-unavailable=0

# upgrade nodes
gcloud container node-pools upgrade NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a
```

### EKS Upgrade

Control Plane:

```bash
# update cluster version
aws eks update-cluster-version \
  --region us-west-2 \
  --name my-cluster \
  --kubernetes-version 1.31

# monitor status
aws eks describe-cluster --region us-west-2 --name my-cluster
```

### Node Groups:

```bash
# update managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodes \
  --region us-west-2 \
  --kubernetes-version 1.31

# for self-managed: update launch template, then rolling update
Add-ons:
# update critical add-ons
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --region us-west-2
```

### AKS Upgrade

Control Plane:

```bash
# get available versions
az aks get-versions --location eastus --output table

# upgrade cluster
az aks upgrade \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.31.0
```

### Node Pools:

```bash
# upgrade specific node pool
az aks nodepool upgrade \
  --resource-group myResourceGroup \
  --cluster-name myAKSCluster \
  --name mynodepool \
  --kubernetes-version 1.31.0
```

### Post-Upgrade Validation

### Health Checks

```bash
# verify pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# check nodes
kubectl get nodes
kubectl describe nodes | grep -E "Conditions|Taints"

# component status
kubectl get componentstatuses
```

### Application Testing

- Critical endpoint validation
- Database connectivity
- Ingress/LoadBalancer functionality
- Monitor metrics and logs

### Node Issues

```bash
# check for common upgrade problems
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl logs --selector=app=node-problem-detector -n kube-system

# look for: "task hung", blocked processes, resource pressure
```

### Rollback Procedures

### Control Plane Rollback

GKE:

```bash
# only within same minor version
gcloud container clusters upgrade CLUSTER_NAME \
  --master \
  --cluster-version=1.30.5-gke.previous
```

EKS/AKS: Control plane rollback not supported. Node pool rollback only.

### Node Pool Rollback Strategy

### Method 1: New Node Pool (Recommended)

```bash
# GKE
gcloud container node-pools create rollback-pool \
  --cluster=CLUSTER_NAME \
  --node-version=1.30.5-gke.previous \
  --num-nodes=3

# scale up new pool
gcloud container node-pools resize rollback-pool \
  --cluster=CLUSTER_NAME \
  --num-nodes=10
```

### Method 2: Workload Migration

```bash
# cordon old nodes
kubectl cordon NODE_NAME

# force workload restart to migrate
for ns in $(kubectl get ns -o name | cut -d'/' -f2); do
  if [[ "$ns" != "kube-system" ]]; then
    echo "Restarting $ns"
    kubectl -n $ns rollout restart deployment
    kubectl -n $ns rollout restart statefulset  
    kubectl -n $ns rollout restart daemonset
  fi
done

# verify distribution
kubectl get pods -o wide --all-namespaces

# remove old pool when stable
```

### Advanced Strategies

### Blue-Green Node Pool Upgrade

1. create new node pool with target version
2. Migrate workloads using node selectors/taints
3. Validate functionality
4. Complete migration
5. Remove old pool

### Surge Configuration Best Practices

- Small clusters (<10 nodes): maxSurge=1, maxUnavailable=0
- Large clusters (>50 nodes): maxSurge=20, maxUnavailable=0
- Resource-constrained: maxSurge=0, maxUnavailable=1

Infrastructure as Code Updates

Terraform:

```go
# GKE 
resource "google_container_cluster" "primary" {
  min_master_version = "1.31.0-gke.1234567"
}
```

```go
# EKS  
resource "aws_eks_cluster" "cluster" {
  version = "1.31"
}

# AKS
resource "azurerm_kubernetes_cluster" "cluster" {
  kubernetes_version = "1.31.0"
}
```
Apply with no-op verification:

```bash
# should show no changes post-upgrade
terraform plan 
terraform apply
```

### Monitoring During Upgrades

#### Key Metrics

- Pod scheduling latency
- Node resource utilization
- API server response times
- Application error rates

#### Critical Events

- Node cordoning/draining
- Pod eviction failures
- PDB violations
- Failed scheduling

### Common Issues & Solutions

#### Stuck Node Upgrades:

- Check resource quotas
- Verify image pull capacity
- Review PodDisruptionBudgets

#### Application Failures:

- Validate deprecated API usage
- Check resource requests/limits
- Review network policies

#### Performance Degradation:

- Monitor resource pressure
- Check for node resource fragmentation
- Validate autoscaling configuration

### Platform-Specific Gotchas

#### GKE:

- Automatic node auto-upgrades can conflict with manual upgrades
- Regional clusters take 2x longer to upgrade
-Surge upgrades require additional quotas

#### EKS:

- Add-on compatibility critical (CNI, CSI drivers)
- Self-managed nodes require separate upgrade process
- IAM roles may need updates

#### AKS:

- Azure CNI version compatibility
- System node pools upgrade differently
- Virtual node pools have separate lifecycle


**Reality Check:** Upgrades rarely go perfectly. Plan for 2x the estimated time, have rollback procedures tested and monitor everything. 

This guide reflects real production experience. Test everything in non-prod/staging/dev first, document your specific procedures and build confidence through repetition.


Technical guide for upgrading managed Kubernetes clusters across GKE, EKS, and AKS

Kubernetes Cluster Upgrades: Production-Ready Guide


## Introduction

Setting up a private network for your EKS cluster is important for security and performance. There are many ways to do this, but in this article, we'll use Twingate to create a private network for your EKS cluster.

## Step-by-Step Guide: Deploying a Private EKS Cluster with Twingate Access

### Prerequisites

- An AWS account
- A Twingate account
- Twingate CLI/client installed. 
- eksctl CLI

## Terraform Setup

Assume you have a VPC and subnets already created.

```hcl
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "private-eks-cluster"
  cluster_version = "1.27"
  subnets         = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false

  # Additional configurations...
}
```

## Twingate Setup

Twingate provides secure, remote access to private resources without exposing them to the public internet. To integrate Twingate with your EKS cluster:

- Create a Twingate Account: Sign up at Twingate.
- Define a Remote Network: In the Twingate admin console, add a new remote network representing your AWS VPC.
- Deploy a Connector: Deploy a Twingate Connector within your VPC. This can be done using AWS ECS Fargate, EC2, or as a Kubernetes Deployment. Ensure the Connector has outbound internet access to communicate with Twingate's services and can reach the EKS API endpoint.

### Twingate Connector Deployment

Deploy the Twingate Connector as a container on AWS ECS (Fargate) within your VPC. Ensure the subnet has outbound internet access to communicate with Twingate's services and access the EKS control plane.

### Twingate Resource Configuration

After deploying the Connector, configure Twingate to manage access to your EKS cluster:
    - Add a Remote Network: In the Twingate admin console, add a new remote network representing your AWS VPC.
    - Define Resources: Add a new resource with the private DNS name of your EKS API server.
    - Assign Access: Specify which user groups should have access to this resource.

### Benefits of This Setup

- Enhanced Security: The EKS API server is not exposed to the public internet, reducing the attack surface.
- Granular Access Control: Twingate allows you to define precise access policies based on user groups.
- Zero Trust Architecture: This configuration aligns with Zero Trust principles, ensuring that trust is never implicit.
- Simplified Management: Twingate's centralized management console makes it easy to oversee access policies and monitor activity.

By following this approach, you can establish a secure, private EKS cluster accessible only through Twingate, providing robust protection for your Kubernetes workloads.


How to setup a private network for your EKS cluster with Twingate

EKS Private Network with Twingate


## Introduction

Running out of IP addresses in AWS EKS can be a subtle yet critical issue. It often manifests as pods stuck in a pending state or nodes failing to join the cluster, leading to deployment bottlenecks and potential downtime. Understanding the root cause and implementing effective solutions is essential for maintaining cluster health and scalability. Now, there are many ways to fix this, but this is one way.

## Understanding the Problem: IP Exhaustion in EKS

EKS utilises the AWS VPC CNI plugin to assign IP addresses to pods. Each EC2 instance (node) has a limit on the number of Elastic Network Interfaces (ENIs) and secondary IP addresses it can support, determined by the instance type. When the number of pods exceeds the available IPs, EKS attempts to allocate additional ENIs. However, if the subnet lacks sufficient IP addresses, this allocation fails, resulting in errors like:

```json
{
  "code": "InsufficientFreeAddresses",
  "message": "One or more of the subnets associated with your cluster does not have enough available IP addresses for Amazon EKS to perform cluster management operations."
}
```

This issue is particularly prevalent in dynamic environments with frequent pod scaling, where IP addresses are rapidly consumed.

## Diagnosing IP Exhaustion

To identify IP exhaustion:

- Monitor cluster events for errors related to pod scheduling.
- Use `kubectl describe nodes` to inspect the number of allocated IPs and ENIs.
- Check subnet IP utilization via the AWS Console or CLI.

## Solution: Optimising IP Allocation with WARM_IP_TARGET

The AWS VPC CNI plugin maintains a warm pool of IP addresses to expedite pod networking. By default, it pre-allocates a significant number of IPs, which can lead to unnecessary IP consumption. Adjusting the WARM_IP_TARGET environment variable in the aws-node DaemonSet allows for better control over IP allocation.

### Steps to Adjust WARM_IP_TARGET

1. Modify the AWS CNI DaemonSet:

Reduce the number of pre-allocated IPs per node by setting WARM_IP_TARGET to a lower value (e.g., 5):

```bash
kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=5
```

This change instructs the CNI to maintain only 5 unused IP addresses per node, freeing up IPs in the subnet.

2. Monitor IP Address Availability:

Use `kubectl describe nodes` to observe the number of allocated IPs and ENIs.

3. Verify the Fix:

After applying the configuration, monitor the cluster to ensure that pods are scheduled successfully and the InsufficientFreeAddresses error no longer appears.

## Additional Considerations for Long-Term Scalability

While adjusting WARM_IP_TARGET addresses immediate IP exhaustion issues, consider the following for long-term scalability:

1. Subnet Design:

Ensure subnets are appropriately sized. For larger clusters, consider using /20 or /16 CIDR blocks to provide ample IP addresses.

2. Instance Types:

Select EC2 instance types with higher ENI and IP limits to accommodate more pods per node.

3. Prefix Delegation:

Enable prefix delegation to assign a /28 block of IPs to each ENI, significantly increasing the number of IPs available per node. This can be done by setting the ENABLE_PREFIX_DELEGATION environment variable to true in the aws-node DaemonSet:

```bash
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
```

Note: Ensure your subnets have sufficient contiguous IP address space to support prefix delegation.

## Conclusion

IP address exhaustion in EKS is a common challenge that can hinder cluster scalability and reliability. By tuning the WARM_IP_TARGET setting and considering subnet design, instance selection, and prefix delegation, you can effectively manage IP allocation and maintain a healthy, scalable EKS environment.


EKS IP Exhaustion: Running out of IPs, one way to fix it


## Introduction

AWS EKS defaults to the VPC CNI plugin, assigning VPC IPs to pods via ENIs. While straightforward, this setup limits pod density per node and consumes VPC IPs rapidly. To overcome these constraints, deploying Calico with IPIP or BGP offers a scalable alternative.

## Why Replace AWS VPC CNI?

- Pod Density Limitations: ENI and IP limits per instance type restrict the number of pods per node.
- VPC IP Consumption: Each pod consumes a VPC IP, leading to potential exhaustion.
- Complex Networking: Managing ENIs and secondary IPs adds complexity.

Calico addresses these issues by providing flexible IP address management and networking modes.
Calico Docs

## Calico Networking Modes

- IPIP (IP-in-IP): Encapsulates pod traffic, allowing for scalable networking without consuming VPC IPs.
- BGP (Border Gateway Protocol): Distributes routing information between nodes, enabling efficient traffic routing.

## Calico Docs

- [Calico Docs](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks/ipip)


## Setup

Manually: `eksctl create cluster --name calico-cluster --without-nodegroup`

Using Terraform:

We're going to use community modules for the VPC and EKS cluster to avoid reinventing the wheel and keep things simple. 

Providers:

```hcl
provider "aws" {
  region = "us-west-2"
}

variable "cluster_name" {
  default = "calico-eks-cluster"
}

variable "region" {
  default = "us-west-2"
}
```

Create VPC & subnets:

```hcl
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.14.2"

  name = "calico-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-west-2a", "us-west-2b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    Name = "calico-vpc"
  }
}
```

Create EKS Cluster without nodegroup:

```hcl
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "20.4.0"
  cluster_name    = var.cluster_name
  cluster_version = "1.27"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true

  manage_aws_auth = true
  create_node_security_group = true

  eks_managed_node_groups = {}

  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node communication"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
  }

  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}
```

`terraform init`
`terraform apply`

## Configure Calico as CNO

```bash
aws eks --region us-west-2 update-kubeconfig --name calico-eks-cluster # Configure kubectl to use the new cluster

kubectl delete daemonset aws-node -n kube-system # Delete the AWS VPC CNI plugin
```

## Deploy Calico

```bash
# install calico operator
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/tigera-operator.yaml
```

## Calico installation

For IPIP mode, we need to set the `ipPools` to use the `IPIP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: IPIP
        natOutgoing: Enabled
        nodeSelector: all()
```

For BGP mode, we need to set the `bgp` to `Enabled` and configure the `ipPools` to use the `BGP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Enabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: None
        natOutgoing: Enabled
        nodeSelector: all()
```

depending on which one you use, run `kubectl apply -f <file>.yaml` to deploy Calico.

## Add node groups

We add node groups after the Calico installation to ensure that the nodes are created with the correct configuration.

Update your Terraform configuration to add node groups:

```hcl
module "eks" {
  # ... existing configuration ...

  eks_managed_node_groups = {
    calico_nodes = {
      desired_capacity = 2
      max_capacity     = 3
      min_capacity     = 1

      instance_types = ["t3.medium"]
      subnet_ids     = module.vpc.private_subnets
    }
  }
}
```

`terraform apply`

## Why Node Groups Are Added After Calico Installation

If you create node groups before removing the AWS VPC CNI (aws-node), the following happens:

- Nodes boot up with the AWS VPC CNI already running.
- Each node tries to attach ENIs and configure VPC IPs.
- This conflicts with Calico's CNI once you install it.
- Even worse: nodes may go NotReady or your pods may fail to get Calico-managed IPs.

## Verify Calico

```bash
kubectl get pods -n calico-system
```

## Configure BGP Peering (optional)

If you want to use BGP peering, you need to configure the BGP peering between the nodes and the VPC.

If using BGP mode and need to configure peering:

Disable Node-to-Node Mesh:

```bash
calicoctl patch bgpconfiguration default -p '{"spec": {"nodeToNodeMeshEnabled": false}}'
```

Configure Global BGP Peer:

```yaml
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: global-peer
spec:
  peerIP: <peer-ip>
  asNumber: <asn>
```

`kubectl apply -f bgp-peer.yaml`


EKS without VPC CNI: Deploying Calico with IPIP and BGP


**A hands-on article on deploying an application on Kubernetes with Fargate.**

As promised, here is the second part of Fargate profiles on EKS. This article will look into deploying a “serverless” pod on EKS Fargate from scratch.

**Note:** We will use the eksctl CLI tool to do this for the purpose of this guide. Infrastructure as Code through Terraform or CloudFormation is advised if you are looking to set this up for your production workloads.

### Prerequisites:

*   Install eksctl command line utility [https://docs.aws.amazon.com/eks/latest/userguide/eksctl.htm](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html)l
*   Installing kubectl [https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html)

**Create a Kubernetes cluster on AWS**

A single eksctl command will create the following resources:

*   A VPC with the suitable subnets and accompanying resources
*   K8s control plane
*   Fargate profile
*   IAM role for pod execution

**Note**: Feel free to change the name of the cluster and region.

eksctl create cluster --name eks-fargate --region eu-west-1 -- fargate

This process may take several minutes. When complete, you should see a similar output to show that the cluster is ready to go.

![](https://cdn-images-1.medium.com/max/1760/1*QbuPEpiu8-Rwy1ZNaIJySQ.png)

Cluster and Fargate profile creation

**Verifying the context**

kubectl config current-context

Alternatively, you can use **“kubectx”** to check your current and existing contexts. Install with “brew install kubectx” for Mac and “choco” for Windows.

![](https://cdn-images-1.medium.com/max/1760/1*vvOmaBq3JaxXqGZvR4Avfw.png)

Current context shows that the cluster has been created

**Checking nodes**

We can now check if the fargate profile is ready, it should display at least one Fargate node waiting for our deployments.

kubectl get nodes

![](https://cdn-images-1.medium.com/max/1760/1*Fw5-3lM5fq1-Pg1kWGM1-Q.png)

Fargate nodes ready

**Note:** the nodes that are running inside our VPC are not visible in the EC2 dashboard nor can you SSH into the nodes; they are fully managed and deployed by Fargate by means of “serverless” but you can SSH into a specific pod.

Also, since we haven’t specified the fargate profile by name, the default name for the fargate profile will be “fp-default”. As well as not specifying a namespace, the default namespace will be used to schedule fargate pods. These settings can be changed during setup by specifying the namespace for the Fargate pods.

**Creating a custom Fargate profile**

For the purpose of this guide, we can add an additional fargate profile to show that it can be customised. This can be done with the command:

eksctl create fargateprofile --namespace test --cluster eks-fargate --name fp-test

All pods deployed on the “test” namespace will be scheduled as Fargate pods. We must first create the namespace “test”. Now we can deploy some containers using a default nginx image on the “test” namespace.

**Creating the namespace and deploying pod onto the Fargate-specified namespace**

kubectl create ns test  
kubectl create deploy fargate-pod --image=nginx -n test

We can verify the pods are running by running the command:

kubectl get pods -n test -o wide

![](https://cdn-images-1.medium.com/max/1760/1*PTm6m1zp7fljiFs7GNvLLQ.png)

We can see the pod running on the Fargate node.

Now our pod is running as “serverless”. Well done !

**Clean up your environment**

Now make sure to delete your environment so that you don’t get a heavy billing from AWS :)

Make sure you delete the Fargate profile first and then the cluster itself.

eksctl delete fargateprofile --cluster eks-fargate --name fp-test  
eksctl delete cluster --name eks-fargate --region eu-west-1

Thank you and hope you enjoyed this !

References:

[**AWS Fargate profile**  
_Before you can schedule pods on Fargate in your cluster, you must define at least one Fargate profile that specifies…_docs.aws.amazon.com](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html "https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html")[](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html)

[**EKSworkshop.com**  
_The Fargate profile allows an administrator to declare which pods run on Fargate. Each profile can have up to five…_www.eksworkshop.com](https://www.eksworkshop.com/beginner/180_fargate/creating-profile/ "https://www.eksworkshop.com/beginner/180_fargate/creating-profile/")[](https://www.eksworkshop.com/beginner/180_fargate/creating-profile/)

**What’s next?**

*   Look into exposing the deployment through a service?
*   Look into AWS Load Balancer controller?
*   ALB ingress controllers?
*   Setting up CloudWatch logs in order to obtain application logs?

I’m currently on a journey to learn more about the ever-growing ecosystem of containers, container orchestration and the DevOps culture. Expect to see more articles and open-source contributions. Stay tuned for more !

A hands-on article on deploying an application on Kubernetes with Fargate.

Posts tagged with: eks