Mo AbukarMo Abukar

EKS IP Exhaustion: Running out of IPs, one way to fix it

2024-05-07

Introduction

Running out of IP addresses in AWS EKS can be a subtle yet critical issue. It often manifests as pods stuck in a pending state or nodes failing to join the cluster, leading to deployment bottlenecks and potential downtime. Understanding the root cause and implementing effective solutions is essential for maintaining cluster health and scalability. Now, there are many ways to fix this, but this is one way.

Understanding the Problem: IP Exhaustion in EKS

EKS utilises the AWS VPC CNI plugin to assign IP addresses to pods. Each EC2 instance (node) has a limit on the number of Elastic Network Interfaces (ENIs) and secondary IP addresses it can support, determined by the instance type. When the number of pods exceeds the available IPs, EKS attempts to allocate additional ENIs. However, if the subnet lacks sufficient IP addresses, this allocation fails, resulting in errors like:

{
  "code": "InsufficientFreeAddresses",
  "message": "One or more of the subnets associated with your cluster does not have enough available IP addresses for Amazon EKS to perform cluster management operations."
}

This issue is particularly prevalent in dynamic environments with frequent pod scaling, where IP addresses are rapidly consumed.

Diagnosing IP Exhaustion

To identify IP exhaustion:

  • Monitor cluster events for errors related to pod scheduling.
  • Use kubectl describe nodes to inspect the number of allocated IPs and ENIs.
  • Check subnet IP utilization via the AWS Console or CLI.

Solution: Optimising IP Allocation with WARM_IP_TARGET

The AWS VPC CNI plugin maintains a warm pool of IP addresses to expedite pod networking. By default, it pre-allocates a significant number of IPs, which can lead to unnecessary IP consumption. Adjusting the WARM_IP_TARGET environment variable in the aws-node DaemonSet allows for better control over IP allocation.

Steps to Adjust WARM_IP_TARGET

  1. Modify the AWS CNI DaemonSet:

Reduce the number of pre-allocated IPs per node by setting WARM_IP_TARGET to a lower value (e.g., 5):

kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=5

This change instructs the CNI to maintain only 5 unused IP addresses per node, freeing up IPs in the subnet.

  1. Monitor IP Address Availability:

Use kubectl describe nodes to observe the number of allocated IPs and ENIs.

  1. Verify the Fix:

After applying the configuration, monitor the cluster to ensure that pods are scheduled successfully and the InsufficientFreeAddresses error no longer appears.

Additional Considerations for Long-Term Scalability

While adjusting WARM_IP_TARGET addresses immediate IP exhaustion issues, consider the following for long-term scalability:

  1. Subnet Design:

Ensure subnets are appropriately sized. For larger clusters, consider using /20 or /16 CIDR blocks to provide ample IP addresses.

  1. Instance Types:

Select EC2 instance types with higher ENI and IP limits to accommodate more pods per node.

  1. Prefix Delegation:

Enable prefix delegation to assign a /28 block of IPs to each ENI, significantly increasing the number of IPs available per node. This can be done by setting the ENABLE_PREFIX_DELEGATION environment variable to true in the aws-node DaemonSet:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Note: Ensure your subnets have sufficient contiguous IP address space to support prefix delegation.

Conclusion

IP address exhaustion in EKS is a common challenge that can hinder cluster scalability and reliability. By tuning the WARM_IP_TARGET setting and considering subnet design, instance selection, and prefix delegation, you can effectively manage IP allocation and maintain a healthy, scalable EKS environment.

Related Posts