
## Kubernetes Cluster Upgrades: No BS Guide

Managed Kubernetes clusters need regular upgrades for security patches, bug fixes, and new features. This is a technical guide covering GKE, EKS, and AKS upgrades with real-world procedures and rollback strategies.

### Prerequisites: Do This First

1. Deprecated API Detection
Using kubent (Universal):

```bash
# install kubent
sh -c "$(curl -sSL https://git.io/install-kubent)"

# scan cluster
kubent --target-version=1.31.0
kubent --output=json --target-version=1.31.0
```

GKE Log Explorer Query:
```sql
sqlresource.type="k8s_cluster"
labels."k8s.io.removed-release"="1.31"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system")
```

AWS CloudTrail (EKS):

```bash
# check EKS API calls for deprecated versions
aws logs filter-log-events --log-group-name /aws/eks/cluster/your-cluster/cluster
```

2. Compatibility Matrix Check

- GKE: Verify Anthos Service Mesh/Istio compatibility
- EKS: Check AWS Load Balancer Controller, EBS CSI driver versions
- AKS: Validate Azure CNI, Application Gateway Ingress Controller

3. Resource Assessment

```bash
# check cluster capacity
kubectl top nodes
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# review PDBs
kubectl get pdb --all-namespaces

# look for critical workloads
kubectl get deployments,statefulsets --all-namespaces
```

### Upgrade Strategy

### Environment Progression

Dev/Test clusters → 2. Staging → 3. Production (least critical → most critical)

### Timing

- Off-peak hours
- Team availability for monitoring
- Consider maintenance windows for automatic upgrades

Platform-Specific Procedures

### GKE Upgrade

Control Plane:

```bash
# via gcloud
gcloud container clusters upgrade CLUSTER_NAME \
  --zone=us-central1-a \
  --master \
  --cluster-version=1.31.0-gke.1234567

# monitor
gcloud container operations list --filter="TYPE:UPGRADE_CLUSTER"
Node Pools (Configure surge settings first):
# set surge parameters
gcloud container node-pools update NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a \
  --max-surge=20 \
  --max-unavailable=0

# upgrade nodes
gcloud container node-pools upgrade NODE_POOL \
  --cluster=CLUSTER_NAME \
  --zone=us-central1-a
```

### EKS Upgrade

Control Plane:

```bash
# update cluster version
aws eks update-cluster-version \
  --region us-west-2 \
  --name my-cluster \
  --kubernetes-version 1.31

# monitor status
aws eks describe-cluster --region us-west-2 --name my-cluster
```

### Node Groups:

```bash
# update managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodes \
  --region us-west-2 \
  --kubernetes-version 1.31

# for self-managed: update launch template, then rolling update
Add-ons:
# update critical add-ons
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --region us-west-2
```

### AKS Upgrade

Control Plane:

```bash
# get available versions
az aks get-versions --location eastus --output table

# upgrade cluster
az aks upgrade \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.31.0
```

### Node Pools:

```bash
# upgrade specific node pool
az aks nodepool upgrade \
  --resource-group myResourceGroup \
  --cluster-name myAKSCluster \
  --name mynodepool \
  --kubernetes-version 1.31.0
```

### Post-Upgrade Validation

### Health Checks

```bash
# verify pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# check nodes
kubectl get nodes
kubectl describe nodes | grep -E "Conditions|Taints"

# component status
kubectl get componentstatuses
```

### Application Testing

- Critical endpoint validation
- Database connectivity
- Ingress/LoadBalancer functionality
- Monitor metrics and logs

### Node Issues

```bash
# check for common upgrade problems
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl logs --selector=app=node-problem-detector -n kube-system

# look for: "task hung", blocked processes, resource pressure
```

### Rollback Procedures

### Control Plane Rollback

GKE:

```bash
# only within same minor version
gcloud container clusters upgrade CLUSTER_NAME \
  --master \
  --cluster-version=1.30.5-gke.previous
```

EKS/AKS: Control plane rollback not supported. Node pool rollback only.

### Node Pool Rollback Strategy

### Method 1: New Node Pool (Recommended)

```bash
# GKE
gcloud container node-pools create rollback-pool \
  --cluster=CLUSTER_NAME \
  --node-version=1.30.5-gke.previous \
  --num-nodes=3

# scale up new pool
gcloud container node-pools resize rollback-pool \
  --cluster=CLUSTER_NAME \
  --num-nodes=10
```

### Method 2: Workload Migration

```bash
# cordon old nodes
kubectl cordon NODE_NAME

# force workload restart to migrate
for ns in $(kubectl get ns -o name | cut -d'/' -f2); do
  if [[ "$ns" != "kube-system" ]]; then
    echo "Restarting $ns"
    kubectl -n $ns rollout restart deployment
    kubectl -n $ns rollout restart statefulset  
    kubectl -n $ns rollout restart daemonset
  fi
done

# verify distribution
kubectl get pods -o wide --all-namespaces

# remove old pool when stable
```

### Advanced Strategies

### Blue-Green Node Pool Upgrade

1. create new node pool with target version
2. Migrate workloads using node selectors/taints
3. Validate functionality
4. Complete migration
5. Remove old pool

### Surge Configuration Best Practices

- Small clusters (<10 nodes): maxSurge=1, maxUnavailable=0
- Large clusters (>50 nodes): maxSurge=20, maxUnavailable=0
- Resource-constrained: maxSurge=0, maxUnavailable=1

Infrastructure as Code Updates

Terraform:

```go
# GKE 
resource "google_container_cluster" "primary" {
  min_master_version = "1.31.0-gke.1234567"
}
```

```go
# EKS  
resource "aws_eks_cluster" "cluster" {
  version = "1.31"
}

# AKS
resource "azurerm_kubernetes_cluster" "cluster" {
  kubernetes_version = "1.31.0"
}
```
Apply with no-op verification:

```bash
# should show no changes post-upgrade
terraform plan 
terraform apply
```

### Monitoring During Upgrades

#### Key Metrics

- Pod scheduling latency
- Node resource utilization
- API server response times
- Application error rates

#### Critical Events

- Node cordoning/draining
- Pod eviction failures
- PDB violations
- Failed scheduling

### Common Issues & Solutions

#### Stuck Node Upgrades:

- Check resource quotas
- Verify image pull capacity
- Review PodDisruptionBudgets

#### Application Failures:

- Validate deprecated API usage
- Check resource requests/limits
- Review network policies

#### Performance Degradation:

- Monitor resource pressure
- Check for node resource fragmentation
- Validate autoscaling configuration

### Platform-Specific Gotchas

#### GKE:

- Automatic node auto-upgrades can conflict with manual upgrades
- Regional clusters take 2x longer to upgrade
-Surge upgrades require additional quotas

#### EKS:

- Add-on compatibility critical (CNI, CSI drivers)
- Self-managed nodes require separate upgrade process
- IAM roles may need updates

#### AKS:

- Azure CNI version compatibility
- System node pools upgrade differently
- Virtual node pools have separate lifecycle


**Reality Check:** Upgrades rarely go perfectly. Plan for 2x the estimated time, have rollback procedures tested and monitor everything. 

This guide reflects real production experience. Test everything in non-prod/staging/dev first, document your specific procedures and build confidence through repetition.


Technical guide for upgrading managed Kubernetes clusters across GKE, EKS, and AKS

Kubernetes Cluster Upgrades: Production-Ready Guide


## Introduction

In [Part 1](./ecs-fargate-deep-dive-part-1.md), we pulled back the curtain on **how ECS Fargate works**: Firecracker microVMs, ENIs, IAM, and the hidden host fleet.  

Now it’s time to get hands-on. We’ll spin up Firecracker locally, boot a microVM, and even simulate the ENI networking model.  

### Why Should You Care?

Running Firecracker yourself helps you:  

- See how AWS runs **each Fargate task inside its own VM**.  
- Understand **boot performance** and why Fargate cold starts take 30–60s.  
- Learn how ENIs map to **VPC-level isolation**.  
- Debug networking limits by replicating them locally.  

### Scenario Overview

We’ll simulate a Fargate task locally:  

- Launch a Firecracker microVM.  
- Attach a minimal Linux root filesystem.  
- Configure a TAP interface for networking (like an ENI).  
- Map this setup back to ECS Fargate’s architecture.  

Goal: see how isolation and networking work in practice.  

---

## Deep Dive: Step-by-Step Setup

### Prerequisites

Run this on a Linux host (Ubuntu VM, EC2, or Multipass).  

```bash
sudo apt-get update
sudo apt-get install -y git curl wget build-essential \
    unzip cmake pkg-config libseccomp-dev

# Download Firecracker

curl -LOJ https://github.com/firecracker-microvm/firecracker/releases/download/v1.8.0/firecracker-v1.8.0-x86_64.tgz
tar -xvf firecracker-v1.8.0-x86_64.tgz
sudo mv release-v1.8.0-x86_64/firecracker /usr/local/bin/

### Kernel & RootFS

curl -LO https://s3.amazonaws.com/spec.ccfc.min/img/hello/kernel/vmlinux.bin
curl -LO https://s3.amazonaws.com/spec.ccfc.min/img/hello/fsfiles/x86_64/rootfs.ext4


### Launching a MicroVM

rm -f /tmp/firecracker.socket
firecracker --api-sock /tmp/firecracker.socket &

# VM config + boot source
curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/machine-config' \
  -H 'Content-Type: application/json' \
  -d '{"vcpu_count":1,"mem_size_mib":512,"ht_enabled":false}'

# Kernel
curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/boot-source' \
  -H 'Content-Type: application/json' \
  -d '{"kernel_image_path":"./vmlinux.bin","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}'

# RootFS
curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/drives/rootfs' \
  -H 'Content-Type: application/json' \
  -d '{"drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false}'

## Start the VM

curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/actions' \
  -H 'Content-Type: application/json' \
  -d '{"action_type":"InstanceStart"}'

## You’ll see a minimal Linux VM booting instantly.

###Networking Simulation

### Fargate gives each task its own ENI. We’ll mimic this with a TAP device:

sudo ip tuntap add tap0 mode tap
sudo ip addr add 172.16.0.1/24 dev tap0
sudo ip link set tap0 up


### Attach it to Firecracker:

curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/network-interfaces/eth0' \
  -H 'Content-Type: application/json' \
  -d '{"iface_id":"eth0","host_dev_name":"tap0","guest_mac":"02:FC:00:00:00:01"}'


### Inside the VM, bring up eth0 and assign an IP. Now the microVM talks to the host — exactly how a Fargate task talks to your VPC.

```

### Mapping Back to Fargate

- Firecracker VM → Your Fargate task’s isolated runtime.

- RootFS → Base container environment + ECS runtime + your container image.

- TAP device → Equivalent of a VPC ENI.

- API config → ECS Scheduler telling Fargate what to run.

### Architecture

```mermaid
flowchart TD
    subgraph LocalLab["Local Firecracker Lab"]
        subgraph Firecracker["MicroVM"]
            Kernel[Linux Kernel]
            RootFS[RootFS + Container Runtime]
            IF1[eth0 (TAP)]
        end
        Host["Linux Host"] --> Tap[tap0]
        Tap --> IF1
    end

    subgraph AWSFargate["AWS Fargate Analogy"]
        subgraph Task1["Fargate Task"]
            MVM1[Firecracker microVM]
            ENI1[ENI (VPC IP)]
        end
        VPC1[VPC Subnet + Security Group]
    end

    Tap -. maps to .-> ENI1
    Firecracker -. maps to .-> MVM1
    Host -. maps to .-> VPC1

```

### Takeaways

- Each Fargate task is a microVM, not just a container.

- ENI-per-task makes tasks first-class citizens in your VPC.

- Running Firecracker locally shows why Fargate cold starts exist.

- Debugging ENIs locally helps anticipate VPC IP exhaustion issues.

---

### Conclusion

By running Firecracker locally, you’ve just recreated the core of ECS Fargate. Every time you launch a task in AWS, the same steps happen — at massive scale, across AWS’s hidden fleet.

In Part 3, we’ll dive into real-world gotchas: ENI scaling limits, cold start tuning, and how observability works when you don’t control the host.


In the second part of our ECS Fargate Deep Dive, we get hands-on with Firecracker — the lightweight VMM that powers Fargate — and simulate task isolation and networking locally.

ECS Fargate Deep Dive Part 2: Firecracker in Action


## Introduction

This is Part 1 of our ECS Fargate Deep Dive series. In these posts, we’ll strip away the abstraction and look at what’s really happening under the hood when you launch a Fargate task.  

We’ll cover internals like **Firecracker microVMs, ENI networking, IAM roles, storage and scheduling** - the things AWS doesn’t show you in the console. 

### ECS Fargate: Behind the Scenes of Serverless Containers

When you launch a task on ECS Fargate, it feels like magic — no EC2 instances, no cluster autoscaling, just containers running. But what’s actually happening under the hood?

Let’s peel back the layers.

### Why Should You Care?

Fargate feels “serverless for containers” — but knowing how it works helps you:  

- Debug **networking issues** (e.g. ENI/IP exhaustion).  
- Understand **cold starts and scaling behaviour**.  
- Make **cost vs flexibility trade-offs** between ECS on EC2 and Fargate.  
- See why Fargate is safer in multi-tenant workloads.  

### What Fargate Really Is

Fargate isn’t a special kind of ECS cluster. It’s an AWS-managed capacity layer that runs your containers in lightweight microVMs. You pick CPU, memory, networking, and IAM roles — AWS handles the hosts, patching, and kernel security. 

## Deep Dive: What Actually Happens

### Firecracker MicroVMs

- Each Fargate task = one Firecracker microVM.  
- Built on **KVM**, stripped down for speed.  
- Isolation is stronger than plain Docker namespaces.  
- Each has its own kernel, cgroups, and seccomp profile.  

This is why tasks don’t share kernels, unlike ECS on EC2.  

### Networking

When you start a Fargate task:

- AWS provisions an Elastic Network Interface (ENI) in your VPC.
- Every task gets its own **Elastic Network Interface (ENI)**.  
- The ENI is placed directly in your VPC subnet and directly to the microVM.  
- Security groups/NACLs apply directly to the task.  
- From a network perspective, tasks behave like first-class EC2 instances. This is why Fargate tasks look and feel like EC2 instances on the network.
 
### Storage & Security

- **Ephemeral storage**: 20 GiB by default (up to 200 GiB). Dies with the task.
- **Persistent storage**: use EFS mounts if needed.  
- **IAM Roles for Tasks**: AWS injects temporary credentials directly into the microVM.
- No SSH/SSM access — AWS fully manages the hosts.  

### Scheduling & Scaling

1. ECS control plane tells Fargate: “Run this task with X CPU, Y memory.”  
2. Fargate provisions a microVM, attaches an ENI, pulls the image and starts the task.
3. Scaling = AWS spinning up/destroying microVMs behind the scenes.

## Performance & Limits

- **Cold starts**: ~30–60s (microVM boot + ENI attach + image pull).  
- **Resource limits**: 0.25–16 vCPUs, 0.5–120 GiB RAM.  
- **Restrictions**: no privileged mode, no GPUs, no custom kernels. Can't run DinD (docker in docker) in a task. 
- **Cost model**: billed per vCPU-second + GiB-second.  

---

### ECS on EC2 vs ECS Fargate

| Feature       | ECS on EC2                         | ECS Fargate                          |
|---------------|------------------------------------|--------------------------------------|
| Host control  | You manage AMIs & patching         | AWS manages hosts                     |
| Isolation     | Shared kernel (namespaces)         | Firecracker microVM per task          |
| Networking    | Shared ENI or `awsvpc`             | Dedicated ENI per task                |
| Flexibility   | Privileged, GPU, custom AMI        | Limited, no host access               |
| Cost model    | Pay per instance (even idle)       | Pay per task-second                   |


---

## Architecture Diagram  

```mermaid
flowchart TD
    ECS[ECS Control Plane] -->|Schedules Task| AWS_Fargate["AWS Fargate Host Fleet"]

    subgraph AWS_Fargate
        subgraph Firecracker1["Task A (Firecracker microVM)"]
            C1[Containers]
            ENI1[ENI]
        end
        subgraph Firecracker2["Task B (Firecracker microVM)"]
            C2[Containers]
            ENI2[ENI]
        end
    end

    ENI1 --> VPC1[VPC Subnet + Security Group]
    ENI2 --> VPC2[VPC Subnet + Security Group]
```

### Takeaways

- Each Fargate task runs in its own microVM, not a shared host kernel.

- Networking is first-class: ENI per task, direct VPC integration.

- You give up host control in exchange for security + simplicity.

- Cold starts, ENI limits, and no privileged mode are the main gotchas.

### How to Think of It

Fargate is basically AWS Lambda for containers:

- Instead of functions, you run containers.

- Instead of ephemeral processes, you get microVMs.

- You trade host control for simplicity, security, and auto-scaling.



In the first part of our ECS Fargate Deep Dive, we break down what happens behind the scenes when you run a task on Fargate — Firecracker microVMs, ENIs, IAM and the hidden host fleet.

ECS Fargate Deep Dive Part 1: How Fargate Really Works


## A Real Talk on GKE Upgrades: What They Don't Tell You in the Docs

**Fair warning: This is going to be a long read!** But if you're responsible for production GKE clusters, every minute you spend here could save you hours of stress and potential downtime later.

I've been through the trenches of GKE cluster upgrades more times than I care to count — from smooth sailing 15-minute upgrades to absolute nightmare scenarios that had me questioning my career choices at 2 AM. The Google docs will tell you how to click the buttons and run the commands, but they won't prepare you for the reality of what happens when Murphy's Law kicks in during your upgrade window.

This guide isn't just another regurgitation of official documentation. It's battle-tested wisdom from someone who's learned the hard way that "it worked in staging" doesn't guarantee anything, and that the most critical step in any upgrade isn't technical — it's having a solid plan for when everything goes sideways.

## Kubernetes Cluster Upgrades: Beyond Basic Operations

Managing Google Kubernetes Engine (GKE) clusters in production isn't just about keeping workloads running — it's about maintaining security, performance, and compatibility while navigating the rapidly evolving Kubernetes ecosystem. GKE automatically upgrades the version of the control plane and nodes to ensure that the cluster receives new features, bug fixes, and security patches, but successful production upgrades require careful planning and execution.

With the frequent updates and patches from upstream Kubernetes, and GKE, we highly recommend testing new releases on testing and/or staging environments before the releases are rolled out into your production environment, especially Kubernetes minor version upgrades. The challenge isn't just technical — it's operational. How do you minimize disruption? What happens when things go wrong? How do you ensure your applications remain available throughout the process?

This comprehensive guide walks through a battle-tested approach to GKE upgrades, from initial planning through rollback procedures, designed for production environments where downtime isn't an option.

### Pre-Flight Checks: The Foundation of Safe Upgrades

Before touching any cluster configuration, thorough preparation prevents most upgrade failures. These checks aren't just recommendations — they're insurance against production outages.

#### API Deprecation Detection

A few deprecated APIs have been kept around in the last couple of K8s versions and finally getting completely removed in Kubernetes 1.16 release and this pattern continues with every release. The most critical step is identifying resources using deprecated APIs that will break in your target version.

#### Method 1: Using kubent (Recommended)

Kube No Trouble (kubent) is a simple tool to check whether you're using any of these API versions in your cluster and therefore should upgrade your workloads first, before upgrading your Kubernetes cluster.

```bash
bash# Install kubent
sh -c "$(curl -sSL https://git.io/install-kubent)"

# Scan your cluster for deprecated APIs
kubent --target-version=1.31.0

# Get JSON output for automation
kubent --target-version=1.31.0 --output=json
```

#### Method 2: GCP Log Explorer Query

For clusters with audit logging enabled, use this query to identify deprecated API usage:

```sql
sqlresource.type="k8s_cluster"
labels."k8s.io.removed-release"="1.31"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.authenticationInfo.principalEmail!~("system:serviceaccount:kube-system")
```

#### Method 3: Native kubectl Discovery

```bash
bash# Check supported API versions
kubectl api-versions

# Verify specific resources exist in new format
kubectl explain deployment --api-version=apps/v1
```

#### Service Mesh Compatibility Verification

If running Istio, Anthos Service Mesh, or other service mesh solutions, verify compatibility with your target GKE version. GKE supports a minor version by providing patch versions of the same minor release, and, on a regular basis, automatically upgrading clusters to those newer patches, but service mesh components may have different support matrices.
Check the official compatibility documentation for your service mesh version against the target GKE version before proceeding.

#### Resource and Workload Assessment

Review your cluster's resources and workloads to ensure they're compatible with the target GKE version. This includes checking for deprecated APIs, unsupported features and any other compatibility issues.

```bash
# check cluster resource utils
kubectl top nodes
kubectl top pods --all-namespaces

# check for critical workloads
kubectl get deployments --all-namespaces -o wide
kubectl get statefulsets --all-namespaces -o wide

# review PDBs...
kubectl get pdb --all-namespaces
```

### Strategic Upgrade Planning: Minimize Risk, Maximize Success

#### Environment-Based Rollout Strategy

As part of your workflow for delivering software updates, we recommend that you use multiple environments. Multiple environments help you minimize risk and unwanted downtime by testing software and infrastructure updates separately from your production environment.

#### Recommended Progression:

1. Development/Testing Clusters: Start with non-critical environments
2. Staging Clusters: Full production-like testing with realistic workloads
3. Production Clusters: Begin with least critical, progress to most critical

#### Timing Considerations

Schedule upgrades during maintenance windows when impact on users and engineering teams is minimal. Consider:

- Peak usage periods for your applications
- Team availability for monitoring and response
- Dependency on external services that might be affected

#### Release Channel Strategy

To keep clusters up-to-date with the latest GKE and Kubernetes updates, here are some recommended environments and the respective release channels the clusters should be enrolled in:

- Development: Rapid channel for early testing
- Staging: Regular channel for stability validation
- Production: Stable or Extended channel for proven reliability

### Executing the Upgrade: Step-by-Step Process

#### Phase 1: Control Plane Upgrade

The control plane upgrade typically completes in 10-15 minutes for zonal clusters, 20-30 minutes for regional clusters.

#### Via Google Cloud Console:

1. Navigate to Kubernetes Engine > Clusters
2. Select target cluster
3. Click Upgrade Available next to Version
4. Select desired version (increment by one minor version)
5. Click Save Changes

#### Via gcloud CLI:

```bash
bash# Check available versions
gcloud container get-server-config --zone=us-central1-a

# Upgrade control plane
gcloud container clusters upgrade CLUSTER_NAME \
    --zone=us-central1-a \
    --master \
    --cluster-version=1.31.0-gke.1234567
```

#### Via Terraform:

Update your Terraform configuration and apply:

```hcl
resource "google_container_cluster" "primary" {
  name               = "my-cluster"
  location           = "us-central1-a"
  min_master_version = "1.31.0-gke.1234567"
  # ... other configuration
}
```

#### Phase 2: Node Pool Upgrades

GKE chooses the following strategies for these specific scenarios: In Autopilot clusters, GKE uses surge upgrades. For Standard clusters, configure surge settings for optimal balance between speed and disruption.

#### Configure Surge Upgrade Settings:

For large clusters where the upgrade process might take longer, you can accelerate the upgrade completion time by concurrently upgrading multiple nodes at a time. Use surge upgrade with maxSurge=20, maxUnavailable=0 to instruct GKE to upgrade 20 nodes at a time, without using any existing capacity.

```bash
# Configure surge settings before upgrade
gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --max-surge=20 \
    --max-unavailable=0

# Upgrade node pool
gcloud container node-pools upgrade NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --node-version=1.31.0-gke.1234567
```

#### Monitor Upgrade Progress:

```bash
bash# Check node status
kubectl get nodes -o wide

# Monitor pod scheduling
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# Watch for node cordoning/draining events
kubectl get events --sort-by=.metadata.creationTimestamp
```

#### Phase 3: Post-Upgrade Validation

#### System Health Checks:

```bash
bash# Verify all pods are running
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# Check node readiness
kubectl get nodes | grep -v Ready

# Validate cluster components
kubectl get componentstatuses
```

#### Application Validation:

- Test critical application endpoints
- Verify database connectivity
- Check monitoring and alerting systems
- Validate ingress and load balancer functionality

#### Node Resource Inspection:

```bash
# Check for common upgrade issues in node logs
# Look for patterns like "task hung" or blocked processes
kubectl logs --selector=app=node-problem-detector -n kube-system
```

#### Monitoring and Observability During Upgrades

#### Key Metrics to Watch

- Node resource utilization (CPU, Memory, Disk)
- Pod scheduling latency
- Application response times
- Error rates and HTTP status codes
- Resource quota usage

#### Critical Events to Monitor

- Node cordoning and draining events
- Pod eviction failures
- PDB violations
- Failed pod scheduling
- Container runtime errors

#### Logging Strategy

```bash
# Implement structured logging to capture upgrade events:
# Monitor upgrade operations

```bash
gcloud container operations list --filter="TYPE:UPGRADE_CLUSTER"

# Get detailed operation status
gcloud container operations describe OPERATION_ID --zone=us-central1-a
```

### Rollback Procedures: When Things Go Wrong

Despite careful planning, upgrades sometimes fail. Having a tested rollback strategy is crucial for production environments.

#### Immediate Response Protocol

When to Trigger Rollback:

- Application functionality severely degraded
- Critical system components failing
- Node upgrade stuck for >2 hours
- Widespread pod scheduling failures

#### Control Plane Rollback

To mitigate an unsuccessful cluster control plane upgrade, you can downgrade your control plane to a previous patch release if the version is an earlier patch release within the same minor version.

```bash
# Rollback control plane (within same minor version only)
gcloud container clusters upgrade CLUSTER_NAME \
    --zone=us-central1-a \
    --master \
    --cluster-version=1.30.5-gke.1234567  # Previous patch version
```

#### Node Pool Rollback via New Pool Strategy

For comprehensive rollback, create a new node pool with the previous version:
Step 1: Create New Node Pool

```bash
gcloud container node-pools create rollback-pool \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --node-version=1.30.5-gke.1234567 \
    --num-nodes=3 \
    --machine-type=e2-standard-4
```

#### Step 2: Scale Up New Pool

```bash
bashgcloud container node-pools resize rollback-pool \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a \
    --num-nodes=10  # Match original capacity
```

#### Step 3: Drain and Migrate Workloads

```bash
bash# Cordon old nodes
kubectl cordon NODE_NAME

# Force restart all workloads to migrate
for ns in $(kubectl get ns -o name | cut -d'/' -f2); do
    if [[ "$ns" != "kube-system" ]]; then
        echo "Restarting workloads in namespace: $ns"
        kubectl -n $ns rollout restart deployment
        kubectl -n $ns rollout restart statefulset
        kubectl -n $ns rollout restart daemonset
    fi
done
```

#### Step 4: Verify and Clean Up

```bash
bash# Verify pod distribution
kubectl get pods --all-namespaces -o wide

# Delete old node pool once stable
gcloud container node-pools delete old-pool-name \
    --cluster=CLUSTER_NAME \
    --zone=us-central1-a
```

#### Application-Level Recovery

If infrastructure rollback isn't sufficient:

- Restore application deployments to previous versions
- Rollback database schema changes if applicable
- Reset configuration maps and secrets
- Verify external service integrations

### Advanced Upgrade Strategies

#### Blue-Green Node Pool Upgrades

Blue-green upgrades: Existing nodes are kept available for rolling back while the workloads are validated on the new node configuration.
For zero-downtime upgrades of critical workloads:

- Create new node pool with upgraded version
Gradually migrate workloads using node selectors
- Validate functionality on new nodes
- Complete migration and remove old pool
- Keep old pool available for quick rollback if needed

#### Gradual Workload Migration

Use node taints and tolerations for controlled migration:

```yaml
# Taint new nodes
kubectl taint nodes NEW_NODE_NAME upgraded=true:NoSchedule

# Update deployments with toleration
spec:
  template:
    spec:
      tolerations:
      - key: "upgraded"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
```

#### Maintenance Windows and Exclusions

```bash
Maintenance Windows and Exclusions
Configure maintenance policies to control when automatic upgrades occur:

```bash
gcloud container clusters update CLUSTER_NAME \
    --maintenance-window-start="2025-01-15T02:00:00Z" \
    --maintenance-window-end="2025-01-15T06:00:00Z" \
    --maintenance-window-recurrence="FREQ=WEEKLY;BYDAY=SU"
```

#### Infrastructure as Code Integration

```bash
Terraform State Management
After successful upgrades, update Terraform configuration to match actual state:

```hcl
resource "google_container_cluster" "primary" {
  name               = "production-cluster"
  location           = "us-central1-a"
  min_master_version = "1.31.0-gke.1234567"
  
  node_config {
    machine_type = "e2-standard-4"
  }
  
  # Ensure this matches post-upgrade state
  remove_default_node_pool = true
}
```

Apply changes with plan review:

```bash
terraform plan  # Should show no changes
terraform apply
```

#### GitOps Integration

```bash
Update cluster manifests in your GitOps repositories to reflect new API versions and configurations discovered during the upgrade process.
```

#### Post-Upgrade Housekeeping

##### Documentation Updates

- Update runbooks with any new procedures discovered
- Document any application-specific compatibility issues
- Record upgrade duration and resource consumption
- Update disaster recovery procedures

#### Security Review

- Verify RBAC configurations are intact
- Check network policies and security contexts
- Validate service mesh security policies
- Review admission controller configurations

#### Performance Baseline

Establish new performance baselines post-upgrade:

- Application response times
- Resource utilization patterns
- Scaling behavior
- Cost implications

### The Reality Check: What Actually Happens

Kubernetes upgrades in production rarely go exactly as planned. Even with perfect preparation, you might encounter:

- Resource Constraints: Nodes created by surge upgrade are subject to your Google Cloud resource quotas, resource availability, and reservation capacity
- Application Dependencies: Third-party services may have compatibility issues
- Timing Conflicts: At X we've been running production workloads on kubernetes in GKE since early 2017. When we were at 20 nodes it might take 90–120 minutes, which is in a tolerable range. Our largest nodepool in production at the time was 55 nodes, which would have taken at least 6 hours to fully upgrade

The key is building processes that expect and handle these realities gracefully.

### Best Practices Summary

Before Every Upgrade:

- Run deprecated API scans with kubent
Test in non-production environments first
- Verify service mesh compatibility
- Configure appropriate surge settings
- Plan for rollback scenarios

During Upgrades:

- Monitor cluster and application health continuously
- Have rollback procedures ready to execute
- Communicate status to stakeholders
- Document any unexpected issues

After Upgrades:

- Validate all critical functionality
- Update infrastructure code
- Review and improve procedures
Plan timing for dependent system updates

Successful GKE upgrades aren't just about technical execution — they're about building reliable, repeatable processes that minimize risk while keeping your Kubernetes infrastructure current, secure, and performant. The investment in proper upgrade procedures pays dividends in reduced downtime, improved security posture, and engineering team confidence.

Remember: These actions ensure your cluster remains performant, secure, and up-to-date with the latest features and bug fixes. In the rapidly evolving Kubernetes ecosystem, staying current isn't optional — it's a competitive advantage.

### The Unvarnished Truth: Lessons from the Field

After years of managing GKE upgrades across different organizations, environments, and scales, here's what I wish someone had told me when I started:

**The docs lie about timing.** That "15-30 minute" control plane upgrade? Plan for double that. The "4-5 minutes per node"? Add a buffer. I've seen single node replacements take 20+ minutes because of slow image pulls, stuck pods, or resource constraints nobody anticipated.

**Your applications are more fragile than you think.** Even when you've tested everything in staging, production has a way of exposing edge cases. That legacy service that "just works"? It might be the one that breaks everything during a rolling restart. Always have a communications plan ready — not just for your team, but for stakeholders who need to know why their dashboard went red.

**Rollbacks are harder than upgrades.** The docs make rollbacks sound easy, but in reality, rolling back a failed upgrade often involves more complexity than the original upgrade. You're dealing with potentially corrupted state, confused monitoring systems, and applications that might have partially migrated their data models. Practice your rollback procedures as much as your upgrades.

**The real cost isn't downtime — it's trust.** A botched upgrade doesn't just impact your applications; it impacts your team's confidence in making future changes. Build processes that your team trusts, document everything (especially the failures), and celebrate the upgrades that go smoothly. They're more valuable than you think.

**Google's automatic upgrades aren't your enemy.** I used to fight the automatic upgrade system, trying to control every aspect. But I've learned that working with GKE's upgrade patterns, rather than against them, leads to more reliable outcomes. Use maintenance windows, embrace the surge upgrade strategies, and let Google handle what they do best.

**Your monitoring during upgrades needs to be different.** Normal monitoring tells you what's broken; upgrade monitoring needs to tell you what's about to break. Watch leading indicators: pod scheduling latency, resource pressure, API response times. By the time your standard alerts fire, you're already in reactive mode.

The biggest lesson? **Upgrades are never just about Kubernetes.** They're about organizational change management, risk tolerance, team communication and building systems that can evolve safely. Master those aspects and the technical pieces become much more manageable.

Stay curious, stay prepared and remember — every upgrade is a learning opportunity, even when (especially when) things don't go according to plan. Good luck!!!!

Comprehensive guide for safely upgrading GKE clusters with minimal downtime and robust rollback procedures

GKE Upgrade Guide and Rollback Strategy: A Production-Ready Approach


I tend to create environments for my labs using containers. These containers would hold the resources I need for a certain job. Here's a guide on how I usually create them from scratch.

## Project Structure

Let’s assume we have a folder named `lab-container` with the following structure:

lab-container/ ├── Dockerfile ├── build │ └── binaries.sh ├── .dockerignore ├── Makefile ├── .github │ └── workflows │ └── ci.yml ├── README.md └── zshrc

We'll walk through each file step by step.

---

## The Dockerfile

Here’s a **multi-stage Dockerfile** that compiles and strips binaries in a builder stage, then copies them into a smaller final image:

```bash
FROM alpine:3.19 AS builder

WORKDIR /build

ARG TERRAFORM_VERSION="1.3.9"
ARG VAULT_VERSION="1.12.4"
ARG HELM_VERSION="3.11.2"
ARG KUBECTL_VERSION="v1.26.1"
ARG KIND_VERSION="v0.17.0"
ARG TRAEFIK_VERSION="2.9.6"
ARG TASK_VERSION="3.19.0"

RUN apk update && apk add --no-cache \
      bash \
      curl \
      wget \
      unzip \
      git \
      tar \
      ca-certificates \
      upx \
      binutils  # `strip` and `upx` to reduce binary sizes

COPY build/binaries.sh /build/binaries.sh
RUN chmod +x /build/binaries.sh && /build/binaries.sh

# Optionally strip and compress
RUN strip --strip-unneeded /usr/local/bin/* || true
RUN upx --best --lzma /usr/local/bin/* || true

################################################################################
# Final image
################################################################################
FROM alpine:3.19

LABEL maintainer="YourName" \
      description="A lightweight DevOps toolbox container."

WORKDIR /labs
ENV PATH="/labs/bin:$PATH"

RUN apk update && apk add --no-cache \
      bash \
      ca-certificates \
      python3 \
      py3-pip \
      aws-cli \
      zsh \
      jq \
      fzf \
      eza

COPY --from=builder /build/bin/ /usr/local/bin/

COPY zshrc /root/.zshrc

SHELL ["/bin/zsh", "-c"]

CMD ["zsh"]
```

### Explanation

Builder Stage:

- Installs dependencies (bash, curl, tar, etc.) needed to fetch and install binaries.

- Copies binaries.sh which downloads and installs Terraform, Vault, Helm, etc.

- Uses strip and upx to reduce final binary sizes.

### Final Stage:

- Starts with a fresh Alpine base.

- Installs additional packages (e.g., AWS CLI, zsh, jq, fzf).

- Copies the compiled binaries from the builder stage.

- Sets up a default shell (zsh) and copies a zshrc config file for a nice shell environment.

## build/binaries.sh

This script downloads and installs each DevOps tool. For example:

```bash
#!/usr/bin/env bash
set -euxo pipefail

# In this script, you can use $TERRAFORM_VERSION, $VAULT_VERSION, etc.
# to download each tool. For instance:
wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip
unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin
rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip

# Similarly for Vault:
wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip
unzip vault_${VAULT_VERSION}_linux_amd64.zip -d /usr/local/bin
rm vault_${VAULT_VERSION}_linux_amd64.zip

# ... repeat for Helm, kubectl, kind, Traefik, Task, etc.
# e.g. for kubectl:
curl -L "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" -o /usr/local/bin/kubectl
chmod +x /usr/local/bin/kubectl

# For kind:
curl -Lo /usr/local/bin/kind "https://kind.sigs.k8s.io/dl/${KIND_VERSION}/kind-linux-amd64"
chmod +x /usr/local/bin/kind

# For Task CLI:
curl -L https://github.com/go-task/task/releases/download/v${TASK_VERSION}/task_linux_amd64.tar.gz | tar -zx -C /usr/local/bin
```

Tip: Keep the script idempotent. If you run it twice, it shouldn’t break anything.

## .dockerignore

The .dockerignore helps exclude files you don’t want in your build context (like .git, logs, local environment files, etc.):

```bash
.env
.DS_Store
*.log
.git
.gitignore
README.md
node_modules/
```

Adjust to your needs.

## Makefile

A simple Makefile can streamline common Docker tasks:

```makefile
IMAGE_NAME = your-dockerhub-username/devops-toolbox
VERSION = v1.0.0
LATEST = latest

.PHONY: build
build:
	docker build -t $(IMAGE_NAME):$(LATEST) .

.PHONY: run
run:
	docker run --rm -it $(IMAGE_NAME):$(LATEST) zsh

.PHONY: test
test:
	docker run --rm $(IMAGE_NAME):$(LATEST) sh -c ' \
		echo "Checking installed tools..."; \
		terraform --version && \
		vault --version && \
		aws --version && \
		kubectl version --client && \
		helm version && \
		kind version && \
		traefik version && \
		task --version && \
		docker version \
	'

.PHONY: push
push:
	docker tag $(IMAGE_NAME):$(LATEST) $(IMAGE_NAME):$(VERSION)
	docker push $(IMAGE_NAME):$(VERSION)
	docker push $(IMAGE_NAME):$(LATEST)

.PHONY: clean
clean:
	docker image prune -f

.PHONY: deploy-k8s
deploy-k8s:
	kubectl run toolbox --image=$(IMAGE_NAME):$(LATEST) -- sleep infinity

.PHONY: exec-k8s
exec-k8s:
	kubectl exec -it toolbox -- zsh

```

## Explanation

build: Builds the Docker image locally with the latest tag.

run: Launches the container in an interactive shell.

test: Runs each tool’s --version to confirm they’re installed.

push: Tags and pushes the image to Docker Hub.

deploy-k8s: Runs the container in a Kubernetes pod.

exec-k8s: Opens a shell into that running pod.

## zshrc

For a better shell experience, add your Zsh config:

```bash
export PATH="/usr/local/bin:$PATH"
export EDITOR=vim

# Enable autocompletion
autoload -Uz compinit
compinit

# Enable fzf for fuzzy search
[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh

# Some aliases
alias ls="eza --icons=always"
alias la="eza -al --icons"

# Theme or prompt settings (e.g. powerlevel10k)
ZSH_THEME="powerlevel10k/powerlevel10k"
[[ ! -f ~/.p10k.zsh ]] || source ~/.p10k.zsh
```

More aliases for kubectl, terraform, etc. as desired

## GitHub Actions (Optional)
 
If you want automatic builds when you push changes, create .github/workflows/ci.yml:

```yaml
name: devops-toolbox

on:
  push:
    paths:
      - .github/workflows/ci.yml
      - Dockerfile
      - build/binaries.sh

jobs:
  build-and-publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: write

    steps:
    - name: Check out
      uses: actions/checkout@v2

    - name: Log in to Docker Hub
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

    - name: Build and Publish
      run: |
        registry="your-dockerhub-username"
        sha=$(git rev-parse --short HEAD)
        primary_tag="${registry}/devops-toolbox:${sha}"

        docker build -t $primary_tag -f Dockerfile .
        docker push $primary_tag

        current_branch=$(git rev-parse --abbrev-ref HEAD)
        if [ "$current_branch" = "main" ]; then
          retention_tag="${registry}/devops-toolbox:rc-${sha}"
          docker tag $primary_tag $retention_tag
          docker push $retention_tag
        else
          retention_tag="${registry}/devops-toolbox:alpha-${sha}"
          docker tag $primary_tag $retention_tag
          docker push $retention_tag
        fi
```

When you push changes to GitHub, this workflow will build your Docker image and push it to your Docker Hub account.

## Building and Testing Locally

Build:

```bash
make build
```

`docker build -t your-dockerhub-username/devops-toolbox:latest .`

Test:

```bash
make test
```

This checks each installed tool’s version.

Run:
```bash
make run
```

You should see a Zsh prompt with Terraform, Vault, AWS CLI, Helm, kubectl, etc. available.

## Pushing to Docker Hub
Assuming you’re logged in to Docker Hub (docker login), run:

`make push`

This tags the image with v1.0.0 and latest, then pushes both to your Docker Hub repository.

## Deploying in Kubernetes

If you want to spin up this container in a Kubernetes cluster:

`make deploy-k8s`

It creates a pod named toolbox that sleeps indefinitely. To open a shell inside that pod:

`make exec-k8s`

You’ll then see the same environment as the local container—handy for on-cluster troubleshooting or dev tasks.

## Conclusion

By following these steps, you’ve built a DevOps Toolbox container from scratch. You learned how to:

- Use a multi-stage Dockerfile to install and strip binaries.

- Write a script (binaries.sh) for easy updates to Terraform, Vault, Helm, etc.

- Set up a Zsh environment with helpful aliases.

- Automate builds with Makefile and optionally GitHub Actions.

- Push your image to Docker Hub and deploy it on Kubernetes.

This container simplifies DevOps tasks, letting you quickly spin up a consistent environment anywhere Docker or Kubernetes runs. 


An end-to-end guide for creating a lab container for DevOps training.

Creating a Lab Container


## Introduction

This is Part 1 of our Container Networking Deep Dive series. In this hands-on deep dives, we show how to simulate container-like networking using Linux primitives: ip netns, veth pairs, and routing tables.

We're building it all from scratch. Think of it like plugging an Ethernet cable between two interface cards — one inside a Linux network namespace (our "container") and the other in the host.

### Why Should You Care?

Understanding the low-level mechanics and first principles behind container networking — without relying on Docker or Kubernetes — gives you ultimate debugging power. This series is designed to demystify how containers, pods, bridges, overlays and CNI plugins actually work under the hood.

### Scenario Overview

Prerequisites:

- Make sure to have `multipass` installed (if on macOS) or have a VM setup via UTM/VirtualBox.

We’ll set up:

- A VM using Multipass
- A network namespace (what containers are under the hood)
- A veth pair connecting host and namespace
- IP addresses on both ends
- Routing so they can talk

Goal: ping the namespace from the host and vice versa.

## Deep Dive: Step-by-Step Setup

### Provisioning the VM

Create a Makefile to provision the VM and transfer the files to it.

```bash
NAME=netns-lab
IMAGE=22.04
SCRIPT=scenario1.sh

up:
	@if multipass info $(NAME) >/dev/null 2>&1; then \
		echo "$(NAME) already exists. Skipping launch."; \
	else \
		echo "Launching $(NAME) VM..."; \
		multipass launch --name $(NAME) --cloud-init cloud-init.yaml --memory 1G --disk 5G; \
	fi
	@chmod +x env.sh scenario1.sh test.sh
	@echo "Transferring files to VM..."
	@for file in env.sh scenario1.sh test.sh; do \
		multipass transfer $$file $(NAME):/home/ubuntu/ || echo "Failed to transfer $$file"; \
	done
```

env.sh is a simple script to set up the environment variables.

```bash
CON="netns1"
IP="10.200.1.1"
```

```bash
#!/bin/bash -e

. /home/ubuntu/env.sh

echo "[+] Creating the namespace"
sudo ip netns add $CON

echo "[+] Creating the veth pair"
sudo ip link add veth1 type veth peer name veth2

echo "[+] Moving one end to namespace"
sudo ip link set veth2 netns $CON

echo "[+] Assigning IP inside namespace"
sudo ip netns exec $CON ip addr add $IP dev veth2

echo "[+] Bringing up interfaces"
sudo ip netns exec $CON ip link set veth2 up
sudo ip netns exec $CON ip link set lo up
sudo ip link set veth1 up

echo "[+] Routing setup"
sudo ip route add $IP/32 dev veth1 || true
sudo ip netns exec $CON ip route add default via $IP dev veth2 || true
```

We setup:

- We create a namespace called `netns1`
- We create a veth pair called `veth1` and `veth2`
- We move one end to the namespace
- We assign an IP address to the interface in the namespace
- We bring up the interfaces
- We add a route to the namespace from the host
- We add a default route to the namespace from the host

### Test connectivity

```bash
#!/bin/bash

NS=netns1
IP=10.200.1.1

check() {
  echo "[TEST] $1"
  eval "$2"
  echo ""
}

fail_if_empty() {
  if [ -z "$($1)" ]; then
    echo "[FAIL] $2"
    exit 1
  else
    echo "[PASS] $2"
  fi
}

check "Namespace exists" "sudo ip netns list | grep -q $NS && echo 'Found namespace $NS'"
fail_if_empty "sudo ip netns list | grep $NS" "Namespace '$NS' exists"

check "Interface in namespace" "sudo ip netns exec $NS ip link show veth2"
check "Interface on host" "sudo ip link show veth1"
check "Route on host to $IP" "sudo ip route | grep $IP"
check "Ping from host to namespace" "sudo ping -c 3 $IP"
check "Ping from namespace to host IP" "sudo ip netns exec $NS ping -c 3 $IP"
check "Routes inside namespace" "sudo ip netns exec $NS ip route"
```

### Rest of makefile

```bash
run:
	@echo "Running scenario script..."
	@multipass exec $(NAME) -- bash /home/ubuntu/$(SCRIPT)

shell:
	@multipass shell $(NAME)

test:
	@multipass exec $(NAME) -- bash /home/ubuntu/test.sh

destroy:
	@echo "Destroying VM..."
	@multipass delete $(NAME) --purge || echo "$(NAME) does not exist."
```

## Takeaways & Lessons learned

- `ip netns` is the real container under the hood.
- A veth pair gives you point-to-point connectivity. Similar to what a virtual Ethernet cable does.
- You control the routing tables just like in real containers.
- Network namespaces give you isolation.
- veth pairs connect those isolated spaces.
- Routing needs to be explicit.
- You can debug container networking using raw Linux tools.

## Conclusion

We’ve now created our first standalone network namespace with basic connectivity from host to namespace. This is how most CNI plugins start under the hood.

In the next part of the series, we’ll look at creating two namespaces on the same node and wiring them together using a bridge — effectively building a virtual switch.

Stay tuned for Part 2!


In the first part of our Container Networking Deep Dive, we explore how to set up a single network namespace inside a VM and connect it to the host using a veth pair.

Container Networking Deep Dive Part 1: Single Network Namespace on a VM

The Ultimate Pathway to DevOps Revamped


In this article, we'll walk through creating a custom Amazon Machine Image (AMI) with [HashiCorp Vault](https://www.vaultproject.io/) baked in using [Packer](https://www.packer.io/), launching an EC2 instance with the generated AMI, and then configuring Vault to store and access secrets.

## Overview

This guide covers:

1. **Baking the Vault AMI with Packer:** Cloning the repository and running Packer to generate the AMI.
2. **Launching the Vault EC2 Instance:** Using the created AMI to spin up an EC2 instance.
3. **Configuring Vault:** Initializing, unsealing, and verifying Vault on the instance.
4. **Storing and Accessing Secrets:** A brief walkthrough on adding and retrieving secrets with Vault.

## Prerequisites

- An AWS account with appropriate IAM permissions.
- [Packer](https://www.packer.io/) installed locally.
- Optionally, the AWS CLI or Terraform to launch EC2 instances.
- Basic familiarity with HashiCorp Vault and AWS concepts.

## Step 1: Baking the Vault AMI with Packer

```bash
touch vault-ami.pkr.hcl
```

## Step 2: Build the AMI

```hcl
packer {
  required_plugins {
    amazon = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}


variable "aws_region" {
  default = "us-east-1"
}

variable "ami_name" {
  default = "vault-server-ami"
}

variable "instance_type" {
  default = "t3.micro"
}

variable "source_ami" {
  default = "ami-090fa75af13c156b4" # Amazon Linux 2 (Change if using Ubuntu)
}

variable "ssh_username" {
  default = "ec2-user" # Use "ubuntu" for Ubuntu AMIs
}

# AWS Builder Configuration
source "amazon-ebs" "vault_ami" {
  region                      = var.aws_region
  source_ami                  = var.source_ami
  instance_type               = var.instance_type
  ssh_username                = var.ssh_username
  ami_name                    = var.ami_name
  ami_description             = "Pre-configured Vault AMI"
  associate_public_ip_address = true
  tags = {
    Name = "Vault AMI"
  }
}

# Provisioners (Runs the Vault install script)
build {
  sources = ["source.amazon-ebs.vault_ami"]

  provisioner "file" {
    source      = "install_vault.sh"
    destination = "/tmp/install_vault.sh"
  }

  provisioner "shell" {
    inline = [
      "chmod +x /tmp/install_vault.sh",
      "sudo /tmp/install_vault.sh"
    ]
  }
}
```

### Create the install script

```bash
#!/bin/sh

# Download and move latest Vault release to bin.
printf "\n\nFetching Vault binary"
cd /opt/ && sudo curl -o vault.zip https://releases.hashicorp.com/vault/1.13.1/vault_1.13.1_linux_amd64.zip
sudo unzip vault.zip
sudo mv vault /usr/bin/

# Create a user named vault to be run as a service.
printf "\n\nCreating vault user"
sudo useradd --system --home /etc/vault.d --shell /bin/false vault

# Configure Vault as a System Service
printf "\n\nConfiguring vault as system service"
sudo wget https://raw.githubusercontent.com/moabukar/vault-ami/main/non-tls/non-tls-vault.service
sudo mv  non-tls-vault.service /etc/systemd/system/vault.service
sudo mkdir /etc/vault.d
sudo chown -R vault:vault /etc/vault.d
sudo mkdir /vault-data
sudo chown -R vault:vault /vault-data
sudo mkdir -p /logs/vault/

sudo wget https://raw.githubusercontent.com/moabukar/vault-ami/main/non-tls/non-tls.hcl
sudo mv non-tls.hcl /etc/vault.d/vault.hcl

# Start vault as a service
printf "\n\nEnabling vault as a service"
sudo systemctl enable vault
sudo systemctl start vault
sudo systemctl status vault
```

Review the Packer configuration file (for example, vault-pkr.hcl) to see how Vault is installed and configured. Then, build the AMI using Packer:

```bash
packer build vault-ami.pkr.hcl
```

Note: If the configuration file has a different name (for example, packer.hcl), use that filename.

Packer will run through the steps (e.g., installing Vault, applying configuration, etc.) and output an AMI ID in your AWS account.

## Step 2: Launching the Vault EC2 Instance

With the AMI baked, launch an EC2 instance using the new AMI. You can use the AWS Management Console, CLI, or Terraform. For example, using Terraform

```go
provider "aws" {
  region = "eu-west-2"
}

variable "ami_id" {
  default = "ami-010b1877662d8747c"
}

resource "aws_instance" "vault" {
  ami                    = var.ami_id
  instance_type          = "t3.micro"
  key_name               = "vault-dev"
  vpc_security_group_ids = ["sg-06d59bbad808fc1ee"]


  tags = {
    Name = "Vault Server"
  }
}

output "vault_public_ip" {
  value = aws_instance.vault.public_ip
}

output "ssh_command" {
  description = "SSH into your Vault server"
  value       = format("ssh -i vault-dev.pem ec2-user@%s", aws_instance.vault.public_ip)
}
```

Replace ami-xxxxxxxx with the AMI ID from the Packer build, and update the key name, security group, and subnet ID as appropriate.

## Step 3: Configuring Vault on the Instance

Once your EC2 instance is up, SSH into it:

```bash
ssh -i your-keypair.pem ec2-user@<instance-public-ip>
```

Verify that Vault is installed:

```bash
vault --version
```

Next, initialize Vault. Important: The output will include your unseal key and root token—store these securely.

```bash
vault operator init -key-shares=1 -key-threshold=1
```

Then, unseal Vault using the unseal key provided:

```bash
vault operator unseal <unseal-key>
```

Once unsealed, you can log in with the root token:

```bash
vault login <root-token>
```

Step 4: Storing and Accessing Secrets
Now that Vault is running, you can store secrets. For example, to store a database password:

```bash
vault kv put secret/db password="SuperSecret123!"
```

To retrieve the secret:

```bash
vault kv get secret/db
```

This demonstrates how you can securely manage secrets using Vault.

Conclusion

You now have a full walkthrough to bake a Vault AMI using Packer, launch an EC2 instance with that AMI, configure Vault and start managing secrets securely. This approach streamlines deploying Vault in AWS and is a great addition to any DevOps workflow.


An end-to-end guide for baking a Vault AMI using Packer and deploying a Vault EC2 instance on AWS.

Deploying Vault with a Custom AMI


Containerising applications has become the norm in the IT/DevOps industry, allowing developers to condense their application code and all its dependencies in a lightweight and portable environment so that these applications can be deployed anywhere virtually.

Before going into containers, we must first understand why they were designed.

Let’s just say a Python application has been developed, the next logical step would be to deploy it; this can be complex as well as time-consuming. This is because a running app requires a myriad of configurations, ranging from the allocation of resources to setting up environmental variables.

Another consideration for the application is the ability of it being portable. If a developer needs to change the deployment environment or move to another cloud provider, they will need to repeat many steps of the deployment process. Once again, this can be a mundane process.

Containers efficiently solve the issues by packaging the codebase for the application with all the required dependences — including tools, libraries, config files and run-times. This process of containerisation ensures a lightweight and secure package (depending on how it’s done) known as a container image.

This image can then be used to deploy a containerised application in any environment required, thus giving the developers the much-needed freedom to design and develop apps without having to think about other constraints. Some common container runtimes are Docker, containerd and CRI-O.


![Cargo ships in reference to containers](/images/containers-intro.png)

Many large IT organisations became frustrated with 4–6 month releases and sluggish deployments. Organisations like Netflix and many others began to embrace microservices, thus helping them overtake their fellow competitors — who also had a similar plan. So applications began being built using microservices, heavily transforming into a more DevOps and Agile culture; larger teams were split into smaller ones to allow for this transformation.


DevOps cycle releases
Then came along issues whilst building microservices. Amongst those issues was containerising applications so that they are deployable on any environment, including a Dev’s one. As mentioned before, it was the idea of containers that solved this issue and then this lead to another issue which was managing these containers. This is where the story of Kubernetes began.

So, Kubernetes is a container orchestration platform or in simple terms, an application management platform.

Fun fact: The name Kubernetes itself stems from an ancient Greek word for “helmsman”, meaning he who steers a ship thus telling the story behind the K8s logo. Another name for Kubernetes (you hear it quite often) is “K8s” — the reason for this name is, the 8 letters between the “k” and the “s” in Kubernetes.

Note: From now on, K8s may be used to refer to Kubernetes sporadically.


![The Galver depicting the Kubernetes control mechanism](/images/ship.png)

## Why Kubernetes overtook its competitor OpenStack?

Although K8s and OpenStack have produced similar level of interests, they are slightly different from each other. In the end, Kubernetes has succeeded where OpenStack fell behind. The reason being, Kubernetes’ ability to provide business value and focus on the bigger picture which is the Enterprise IT applications. On the other hand, OpenStack orchestrated infrastructure and no matter how good your infrastructure management is, if you are not solving business problems, what use is it?

## Okay, so how does Kubernetes help me or my company ?

As mentioned before, Kubernetes enables a “microservice” approach to building modern application and larger teams can be split into smaller ones — leading to a much faster time to market, deployments of applications are always being updated and later versions are released.

**Costs of IT infrastructure are more optimised**. K8s can help your organisation largely cut infrastructure costs especially if they’re operating at a huge scale. Prior to the introduction of K8s, IT and Sys Admin, commonly over-provisioned their infrastructure to handle unexpected spikes. K8s is wise enough to schedule and put together containers, while taking into consideration available resources.

Nowadays, the success of applications doesn’t merely depend on features, but also on the scalability of the application. For indeed, if an app cannot scale properly, it will not perform at its best. Kubernetes is critical in this process as it scales and improves app performance. Let’s say we have a service which takes up high-CPU and its quite dynamic, meaning it changes based on business conditions; the solution required here would be something that can scale up the app so that new VMs are automatically created or scaled down when the load is reduced. Kubernetes offers just that capability by scaling up the app as the CPU usage goes above a certain threshold and scales back as a result, optimising the infrastructure utilisation. This feature of K8s auto-scaling known as Horizontal Pod Autoscaler (HPA) is not limited to managed metrics and custom metrics can also be used to trigger the auto-scaling process.

This article mentions only a few of the main benefits of Kubernetes and its various benefits can be spoken about for days.

What is next?

I’m currently on a journey to learn more about the ever-growing ecosystem of containers, container orchestration and the DevOps culture. Expect to see more articles and open-source contributions. Stay tuned for more!

A look at the ever-changing landscape of modern applications

Posts tagged with: devops