
## Introduction

Running out of IP addresses in AWS EKS can be a subtle yet critical issue. It often manifests as pods stuck in a pending state or nodes failing to join the cluster, leading to deployment bottlenecks and potential downtime. Understanding the root cause and implementing effective solutions is essential for maintaining cluster health and scalability. Now, there are many ways to fix this, but this is one way.

## Understanding the Problem: IP Exhaustion in EKS

EKS utilises the AWS VPC CNI plugin to assign IP addresses to pods. Each EC2 instance (node) has a limit on the number of Elastic Network Interfaces (ENIs) and secondary IP addresses it can support, determined by the instance type. When the number of pods exceeds the available IPs, EKS attempts to allocate additional ENIs. However, if the subnet lacks sufficient IP addresses, this allocation fails, resulting in errors like:

```json
{
  "code": "InsufficientFreeAddresses",
  "message": "One or more of the subnets associated with your cluster does not have enough available IP addresses for Amazon EKS to perform cluster management operations."
}
```

This issue is particularly prevalent in dynamic environments with frequent pod scaling, where IP addresses are rapidly consumed.

## Diagnosing IP Exhaustion

To identify IP exhaustion:

- Monitor cluster events for errors related to pod scheduling.
- Use `kubectl describe nodes` to inspect the number of allocated IPs and ENIs.
- Check subnet IP utilization via the AWS Console or CLI.

## Solution: Optimising IP Allocation with WARM_IP_TARGET

The AWS VPC CNI plugin maintains a warm pool of IP addresses to expedite pod networking. By default, it pre-allocates a significant number of IPs, which can lead to unnecessary IP consumption. Adjusting the WARM_IP_TARGET environment variable in the aws-node DaemonSet allows for better control over IP allocation.

### Steps to Adjust WARM_IP_TARGET

1. Modify the AWS CNI DaemonSet:

Reduce the number of pre-allocated IPs per node by setting WARM_IP_TARGET to a lower value (e.g., 5):

```bash
kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=5
```

This change instructs the CNI to maintain only 5 unused IP addresses per node, freeing up IPs in the subnet.

2. Monitor IP Address Availability:

Use `kubectl describe nodes` to observe the number of allocated IPs and ENIs.

3. Verify the Fix:

After applying the configuration, monitor the cluster to ensure that pods are scheduled successfully and the InsufficientFreeAddresses error no longer appears.

## Additional Considerations for Long-Term Scalability

While adjusting WARM_IP_TARGET addresses immediate IP exhaustion issues, consider the following for long-term scalability:

1. Subnet Design:

Ensure subnets are appropriately sized. For larger clusters, consider using /20 or /16 CIDR blocks to provide ample IP addresses.

2. Instance Types:

Select EC2 instance types with higher ENI and IP limits to accommodate more pods per node.

3. Prefix Delegation:

Enable prefix delegation to assign a /28 block of IPs to each ENI, significantly increasing the number of IPs available per node. This can be done by setting the ENABLE_PREFIX_DELEGATION environment variable to true in the aws-node DaemonSet:

```bash
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
```

Note: Ensure your subnets have sufficient contiguous IP address space to support prefix delegation.

## Conclusion

IP address exhaustion in EKS is a common challenge that can hinder cluster scalability and reliability. By tuning the WARM_IP_TARGET setting and considering subnet design, instance selection, and prefix delegation, you can effectively manage IP allocation and maintain a healthy, scalable EKS environment.


EKS IP Exhaustion: Running out of IPs, one way to fix it


## Introduction

AWS EKS defaults to the VPC CNI plugin, assigning VPC IPs to pods via ENIs. While straightforward, this setup limits pod density per node and consumes VPC IPs rapidly. To overcome these constraints, deploying Calico with IPIP or BGP offers a scalable alternative.

## Why Replace AWS VPC CNI?

- Pod Density Limitations: ENI and IP limits per instance type restrict the number of pods per node.
- VPC IP Consumption: Each pod consumes a VPC IP, leading to potential exhaustion.
- Complex Networking: Managing ENIs and secondary IPs adds complexity.

Calico addresses these issues by providing flexible IP address management and networking modes.
Calico Docs

## Calico Networking Modes

- IPIP (IP-in-IP): Encapsulates pod traffic, allowing for scalable networking without consuming VPC IPs.
- BGP (Border Gateway Protocol): Distributes routing information between nodes, enabling efficient traffic routing.

## Calico Docs

- [Calico Docs](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks/ipip)


## Setup

Manually: `eksctl create cluster --name calico-cluster --without-nodegroup`

Using Terraform:

We're going to use community modules for the VPC and EKS cluster to avoid reinventing the wheel and keep things simple. 

Providers:

```hcl
provider "aws" {
  region = "us-west-2"
}

variable "cluster_name" {
  default = "calico-eks-cluster"
}

variable "region" {
  default = "us-west-2"
}
```

Create VPC & subnets:

```hcl
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.14.2"

  name = "calico-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-west-2a", "us-west-2b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    Name = "calico-vpc"
  }
}
```

Create EKS Cluster without nodegroup:

```hcl
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "20.4.0"
  cluster_name    = var.cluster_name
  cluster_version = "1.27"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true

  manage_aws_auth = true
  create_node_security_group = true

  eks_managed_node_groups = {}

  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node communication"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
  }

  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}
```

`terraform init`
`terraform apply`

## Configure Calico as CNO

```bash
aws eks --region us-west-2 update-kubeconfig --name calico-eks-cluster # Configure kubectl to use the new cluster

kubectl delete daemonset aws-node -n kube-system # Delete the AWS VPC CNI plugin
```

## Deploy Calico

```bash
# install calico operator
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/tigera-operator.yaml
```

## Calico installation

For IPIP mode, we need to set the `ipPools` to use the `IPIP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: IPIP
        natOutgoing: Enabled
        nodeSelector: all()
```

For BGP mode, we need to set the `bgp` to `Enabled` and configure the `ipPools` to use the `BGP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Enabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: None
        natOutgoing: Enabled
        nodeSelector: all()
```

depending on which one you use, run `kubectl apply -f <file>.yaml` to deploy Calico.

## Add node groups

We add node groups after the Calico installation to ensure that the nodes are created with the correct configuration.

Update your Terraform configuration to add node groups:

```hcl
module "eks" {
  # ... existing configuration ...

  eks_managed_node_groups = {
    calico_nodes = {
      desired_capacity = 2
      max_capacity     = 3
      min_capacity     = 1

      instance_types = ["t3.medium"]
      subnet_ids     = module.vpc.private_subnets
    }
  }
}
```

`terraform apply`

## Why Node Groups Are Added After Calico Installation

If you create node groups before removing the AWS VPC CNI (aws-node), the following happens:

- Nodes boot up with the AWS VPC CNI already running.
- Each node tries to attach ENIs and configure VPC IPs.
- This conflicts with Calico's CNI once you install it.
- Even worse: nodes may go NotReady or your pods may fail to get Calico-managed IPs.

## Verify Calico

```bash
kubectl get pods -n calico-system
```

## Configure BGP Peering (optional)

If you want to use BGP peering, you need to configure the BGP peering between the nodes and the VPC.

If using BGP mode and need to configure peering:

Disable Node-to-Node Mesh:

```bash
calicoctl patch bgpconfiguration default -p '{"spec": {"nodeToNodeMeshEnabled": false}}'
```

Configure Global BGP Peer:

```yaml
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: global-peer
spec:
  peerIP: <peer-ip>
  asNumber: <asn>
```

`kubectl apply -f bgp-peer.yaml`


Posts tagged with: cni