
## Introduction

AWS EKS defaults to the VPC CNI plugin, assigning VPC IPs to pods via ENIs. While straightforward, this setup limits pod density per node and consumes VPC IPs rapidly. To overcome these constraints, deploying Calico with IPIP or BGP offers a scalable alternative.

## Why Replace AWS VPC CNI?

- Pod Density Limitations: ENI and IP limits per instance type restrict the number of pods per node.
- VPC IP Consumption: Each pod consumes a VPC IP, leading to potential exhaustion.
- Complex Networking: Managing ENIs and secondary IPs adds complexity.

Calico addresses these issues by providing flexible IP address management and networking modes.
Calico Docs

## Calico Networking Modes

- IPIP (IP-in-IP): Encapsulates pod traffic, allowing for scalable networking without consuming VPC IPs.
- BGP (Border Gateway Protocol): Distributes routing information between nodes, enabling efficient traffic routing.

## Calico Docs

- [Calico Docs](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks/ipip)


## Setup

Manually: `eksctl create cluster --name calico-cluster --without-nodegroup`

Using Terraform:

We're going to use community modules for the VPC and EKS cluster to avoid reinventing the wheel and keep things simple. 

Providers:

```hcl
provider "aws" {
  region = "us-west-2"
}

variable "cluster_name" {
  default = "calico-eks-cluster"
}

variable "region" {
  default = "us-west-2"
}
```

Create VPC & subnets:

```hcl
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.14.2"

  name = "calico-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-west-2a", "us-west-2b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    Name = "calico-vpc"
  }
}
```

Create EKS Cluster without nodegroup:

```hcl
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "20.4.0"
  cluster_name    = var.cluster_name
  cluster_version = "1.27"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true

  manage_aws_auth = true
  create_node_security_group = true

  eks_managed_node_groups = {}

  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node communication"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
  }

  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}
```

`terraform init`
`terraform apply`

## Configure Calico as CNO

```bash
aws eks --region us-west-2 update-kubeconfig --name calico-eks-cluster # Configure kubectl to use the new cluster

kubectl delete daemonset aws-node -n kube-system # Delete the AWS VPC CNI plugin
```

## Deploy Calico

```bash
# install calico operator
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/tigera-operator.yaml
```

## Calico installation

For IPIP mode, we need to set the `ipPools` to use the `IPIP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: IPIP
        natOutgoing: Enabled
        nodeSelector: all()
```

For BGP mode, we need to set the `bgp` to `Enabled` and configure the `ipPools` to use the `BGP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Enabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: None
        natOutgoing: Enabled
        nodeSelector: all()
```

depending on which one you use, run `kubectl apply -f <file>.yaml` to deploy Calico.

## Add node groups

We add node groups after the Calico installation to ensure that the nodes are created with the correct configuration.

Update your Terraform configuration to add node groups:

```hcl
module "eks" {
  # ... existing configuration ...

  eks_managed_node_groups = {
    calico_nodes = {
      desired_capacity = 2
      max_capacity     = 3
      min_capacity     = 1

      instance_types = ["t3.medium"]
      subnet_ids     = module.vpc.private_subnets
    }
  }
}
```

`terraform apply`

## Why Node Groups Are Added After Calico Installation

If you create node groups before removing the AWS VPC CNI (aws-node), the following happens:

- Nodes boot up with the AWS VPC CNI already running.
- Each node tries to attach ENIs and configure VPC IPs.
- This conflicts with Calico's CNI once you install it.
- Even worse: nodes may go NotReady or your pods may fail to get Calico-managed IPs.

## Verify Calico

```bash
kubectl get pods -n calico-system
```

## Configure BGP Peering (optional)

If you want to use BGP peering, you need to configure the BGP peering between the nodes and the VPC.

If using BGP mode and need to configure peering:

Disable Node-to-Node Mesh:

```bash
calicoctl patch bgpconfiguration default -p '{"spec": {"nodeToNodeMeshEnabled": false}}'
```

Configure Global BGP Peer:

```yaml
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: global-peer
spec:
  peerIP: <peer-ip>
  asNumber: <asn>
```

`kubectl apply -f bgp-peer.yaml`


Posts tagged with: bgp