
## 🔍 Overview

Private API Gateway allows you to expose REST APIs that are only accessible inside your VPC. No public internet access. This is ideal for internal microservices, backend systems, or APIs you want fully private.

### Scenario
We'll implement a secure internal **Employee Directory API** using AWS Lambda, API Gateway (private), and VPC Interface Endpoints. The API will be accessible **only inside the VPC**.

## 🧱 Prerequisites

- AWS CLI or Terraform
- A VPC with private subnets
- curl or Postman
- Basic understanding of:
  - API Gateway
  - Lambda
  - IAM
  - VPC endpoints (interface)

## 🛠️ Architecture

```
[ EC2 in VPC ] ---> [ VPC Endpoint ] ---> [ Private API Gateway ] ---> [ Lambda (Employee Directory) ]
```

Only resources inside the VPC can reach the API through the interface endpoint.

## 🔧 Step-by-Step Deployment (Terraform)

### 1. Create VPC, Subnet, and Security Group

```go
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
}

resource "aws_subnet" "private" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = false
}

resource "aws_security_group" "vpce_sg" {
  name        = "vpce-sg"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [aws_vpc.main.cidr_block]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
```

### 2. Deploy the Lambda Function

Use the Python code below and package it:

```bash
mkdir employee-directory && cd employee-directory
vi index.py  # lets use our mini lambda below
zip employee_directory.zip index.py
```

```python
import json

EMPLOYEES = {
    "1001": {"id": "1001", "name": "Qais N", "role": "DevOps Engineer"},
    "1002": {"id": "1002", "name": "Abz Ab", "role": "Backend Engineer"},
    "1003": {"id": "1003", "name": "James John", "role": "Product Manager"},
}

def lambda_handler(event, context):
    path = event.get("path", "")
    method = event.get("httpMethod", "")

    if method == "GET" and path.startswith("/employee/"):
        emp_id = path.split("/")[-1]
        emp = EMPLOYEES.get(emp_id)
        if emp:
            return {"statusCode": 200, "body": json.dumps(emp)}
        return {"statusCode": 404, "body": json.dumps({"error": "Employee not found"})}

    if method == "GET" and path == "/employee":
        return {"statusCode": 200, "body": json.dumps(list(EMPLOYEES.values()))}

    return {"statusCode": 400, "body": json.dumps({"error": "Unsupported operation"})}
```

```go
resource "aws_iam_role" "lambda_exec" {
  name = "lambda_exec_role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Action = "sts:AssumeRole",
      Principal = { Service = "lambda.amazonaws.com" },
      Effect = "Allow",
    }],
  })
}

resource "aws_lambda_function" "employee_directory" {
  filename         = "employee_directory.zip"
  function_name    = "employee_directory"
  role             = aws_iam_role.lambda_exec.arn
  handler          = "index.lambda_handler"
  runtime          = "python3.9"
  source_code_hash = filebase64sha256("employee_directory.zip")
}
```

### 3. Create Private API Gateway and Integration

```go
resource "aws_api_gateway_rest_api" "private_api" {
  name        = "employee-directory-api"
  endpoint_configuration { types = ["PRIVATE"] }
}

resource "aws_api_gateway_resource" "employee" {
  rest_api_id = aws_api_gateway_rest_api.private_api.id
  parent_id   = aws_api_gateway_rest_api.private_api.root_resource_id
  path_part   = "employee"
}

resource "aws_api_gateway_method" "get" {
  rest_api_id   = aws_api_gateway_rest_api.private_api.id
  resource_id   = aws_api_gateway_resource.employee.id
  http_method   = "GET"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "lambda_integration" {
  rest_api_id             = aws_api_gateway_rest_api.private_api.id
  resource_id             = aws_api_gateway_resource.employee.id
  http_method             = aws_api_gateway_method.get.http_method
  type                    = "AWS_PROXY"
  integration_http_method = "POST"
  uri                     = aws_lambda_function.employee_directory.invoke_arn
}
```

### 4. Configure VPC Endpoint and Policy

```go
resource "aws_vpc_endpoint" "api_gw" {
  vpc_id             = aws_vpc.main.id
  service_name       = "com.amazonaws.${var.region}.execute-api"
  vpc_endpoint_type  = "Interface"
  subnet_ids         = [aws_subnet.private.id]
  security_group_ids = [aws_security_group.vpce_sg.id]
}

resource "aws_api_gateway_deployment" "deployment" {
  rest_api_id = aws_api_gateway_rest_api.private_api.id
  stage_name  = "prod"
  depends_on  = [aws_api_gateway_integration.lambda_integration]
}

resource "aws_api_gateway_rest_api_policy" "restrict_to_vpce" {
  rest_api_id = aws_api_gateway_rest_api.private_api.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Deny",
        Principal = "*",
        Action = "execute-api:Invoke",
        Resource = "*",
        Condition = {
          StringNotEquals = {
            "aws:SourceVpce" = aws_vpc_endpoint.api_gw.id
          }
        }
      }
    ]
  })
}
```

## 🧪 Test the Setup

1. SSH into an EC2 inside the VPC
2. Run:

```bash
curl https://<rest-api-id>.execute-api.<region>.amazonaws.com/prod/employee
curl https://<rest-api-id>.execute-api.<region>.amazonaws.com/prod/employee/1002
```

✅ Should return valid responses inside the VPC (inside EC2)
❌ Should be blocked from public internet

---

## 📌 Summary

- Built a fully private Employee Directory API
- Integrated it with Lambda + API Gateway
- Made it VPC-only using Interface Endpoints
- Secured access with SourceVpce IAM policy

Great for internal-only microservices, B2B APIs, or compliance-sensitive systems.

🧼 Destroy with:

```bash
terraform destroy
```

Part 2 coming out soon on: DynamoDB, PrivateLink cross-VPC, or IAM auth with SigV4!
---

Learn how to deploy a secure, private-only API Gateway inside your VPC using interface endpoints, resource policies, and VPC integration.

Securing APIs in AWS: Private API Gateway + VPC Endpoint Deep Dive


## Introduction

Running out of IP addresses in AWS EKS can be a subtle yet critical issue. It often manifests as pods stuck in a pending state or nodes failing to join the cluster, leading to deployment bottlenecks and potential downtime. Understanding the root cause and implementing effective solutions is essential for maintaining cluster health and scalability. Now, there are many ways to fix this, but this is one way.

## Understanding the Problem: IP Exhaustion in EKS

EKS utilises the AWS VPC CNI plugin to assign IP addresses to pods. Each EC2 instance (node) has a limit on the number of Elastic Network Interfaces (ENIs) and secondary IP addresses it can support, determined by the instance type. When the number of pods exceeds the available IPs, EKS attempts to allocate additional ENIs. However, if the subnet lacks sufficient IP addresses, this allocation fails, resulting in errors like:

```json
{
  "code": "InsufficientFreeAddresses",
  "message": "One or more of the subnets associated with your cluster does not have enough available IP addresses for Amazon EKS to perform cluster management operations."
}
```

This issue is particularly prevalent in dynamic environments with frequent pod scaling, where IP addresses are rapidly consumed.

## Diagnosing IP Exhaustion

To identify IP exhaustion:

- Monitor cluster events for errors related to pod scheduling.
- Use `kubectl describe nodes` to inspect the number of allocated IPs and ENIs.
- Check subnet IP utilization via the AWS Console or CLI.

## Solution: Optimising IP Allocation with WARM_IP_TARGET

The AWS VPC CNI plugin maintains a warm pool of IP addresses to expedite pod networking. By default, it pre-allocates a significant number of IPs, which can lead to unnecessary IP consumption. Adjusting the WARM_IP_TARGET environment variable in the aws-node DaemonSet allows for better control over IP allocation.

### Steps to Adjust WARM_IP_TARGET

1. Modify the AWS CNI DaemonSet:

Reduce the number of pre-allocated IPs per node by setting WARM_IP_TARGET to a lower value (e.g., 5):

```bash
kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=5
```

This change instructs the CNI to maintain only 5 unused IP addresses per node, freeing up IPs in the subnet.

2. Monitor IP Address Availability:

Use `kubectl describe nodes` to observe the number of allocated IPs and ENIs.

3. Verify the Fix:

After applying the configuration, monitor the cluster to ensure that pods are scheduled successfully and the InsufficientFreeAddresses error no longer appears.

## Additional Considerations for Long-Term Scalability

While adjusting WARM_IP_TARGET addresses immediate IP exhaustion issues, consider the following for long-term scalability:

1. Subnet Design:

Ensure subnets are appropriately sized. For larger clusters, consider using /20 or /16 CIDR blocks to provide ample IP addresses.

2. Instance Types:

Select EC2 instance types with higher ENI and IP limits to accommodate more pods per node.

3. Prefix Delegation:

Enable prefix delegation to assign a /28 block of IPs to each ENI, significantly increasing the number of IPs available per node. This can be done by setting the ENABLE_PREFIX_DELEGATION environment variable to true in the aws-node DaemonSet:

```bash
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
```

Note: Ensure your subnets have sufficient contiguous IP address space to support prefix delegation.

## Conclusion

IP address exhaustion in EKS is a common challenge that can hinder cluster scalability and reliability. By tuning the WARM_IP_TARGET setting and considering subnet design, instance selection, and prefix delegation, you can effectively manage IP allocation and maintain a healthy, scalable EKS environment.


EKS IP Exhaustion: Running out of IPs, one way to fix it


## Introduction

AWS EKS defaults to the VPC CNI plugin, assigning VPC IPs to pods via ENIs. While straightforward, this setup limits pod density per node and consumes VPC IPs rapidly. To overcome these constraints, deploying Calico with IPIP or BGP offers a scalable alternative.

## Why Replace AWS VPC CNI?

- Pod Density Limitations: ENI and IP limits per instance type restrict the number of pods per node.
- VPC IP Consumption: Each pod consumes a VPC IP, leading to potential exhaustion.
- Complex Networking: Managing ENIs and secondary IPs adds complexity.

Calico addresses these issues by providing flexible IP address management and networking modes.
Calico Docs

## Calico Networking Modes

- IPIP (IP-in-IP): Encapsulates pod traffic, allowing for scalable networking without consuming VPC IPs.
- BGP (Border Gateway Protocol): Distributes routing information between nodes, enabling efficient traffic routing.

## Calico Docs

- [Calico Docs](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks/ipip)


## Setup

Manually: `eksctl create cluster --name calico-cluster --without-nodegroup`

Using Terraform:

We're going to use community modules for the VPC and EKS cluster to avoid reinventing the wheel and keep things simple. 

Providers:

```hcl
provider "aws" {
  region = "us-west-2"
}

variable "cluster_name" {
  default = "calico-eks-cluster"
}

variable "region" {
  default = "us-west-2"
}
```

Create VPC & subnets:

```hcl
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.14.2"

  name = "calico-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-west-2a", "us-west-2b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    Name = "calico-vpc"
  }
}
```

Create EKS Cluster without nodegroup:

```hcl
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "20.4.0"
  cluster_name    = var.cluster_name
  cluster_version = "1.27"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true

  manage_aws_auth = true
  create_node_security_group = true

  eks_managed_node_groups = {}

  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node communication"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
  }

  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}
```

`terraform init`
`terraform apply`

## Configure Calico as CNO

```bash
aws eks --region us-west-2 update-kubeconfig --name calico-eks-cluster # Configure kubectl to use the new cluster

kubectl delete daemonset aws-node -n kube-system # Delete the AWS VPC CNI plugin
```

## Deploy Calico

```bash
# install calico operator
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/tigera-operator.yaml
```

## Calico installation

For IPIP mode, we need to set the `ipPools` to use the `IPIP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: IPIP
        natOutgoing: Enabled
        nodeSelector: all()
```

For BGP mode, we need to set the `bgp` to `Enabled` and configure the `ipPools` to use the `BGP` encapsulation mode.

```yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Enabled
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: None
        natOutgoing: Enabled
        nodeSelector: all()
```

depending on which one you use, run `kubectl apply -f <file>.yaml` to deploy Calico.

## Add node groups

We add node groups after the Calico installation to ensure that the nodes are created with the correct configuration.

Update your Terraform configuration to add node groups:

```hcl
module "eks" {
  # ... existing configuration ...

  eks_managed_node_groups = {
    calico_nodes = {
      desired_capacity = 2
      max_capacity     = 3
      min_capacity     = 1

      instance_types = ["t3.medium"]
      subnet_ids     = module.vpc.private_subnets
    }
  }
}
```

`terraform apply`

## Why Node Groups Are Added After Calico Installation

If you create node groups before removing the AWS VPC CNI (aws-node), the following happens:

- Nodes boot up with the AWS VPC CNI already running.
- Each node tries to attach ENIs and configure VPC IPs.
- This conflicts with Calico's CNI once you install it.
- Even worse: nodes may go NotReady or your pods may fail to get Calico-managed IPs.

## Verify Calico

```bash
kubectl get pods -n calico-system
```

## Configure BGP Peering (optional)

If you want to use BGP peering, you need to configure the BGP peering between the nodes and the VPC.

If using BGP mode and need to configure peering:

Disable Node-to-Node Mesh:

```bash
calicoctl patch bgpconfiguration default -p '{"spec": {"nodeToNodeMeshEnabled": false}}'
```

Configure Global BGP Peer:

```yaml
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: global-peer
spec:
  peerIP: <peer-ip>
  asNumber: <asn>
```

`kubectl apply -f bgp-peer.yaml`


EKS without VPC CNI: Deploying Calico with IPIP and BGP


## 🔒 Introduction

Running Kubernetes clusters privately is a growing best practice. In this blog, I’ll walk you through deploying a **private AKS cluster** on Azure with **no public API endpoint**, and enabling secure access via **Twingate VPN**, which provides identity-based access without opening up your network.

This setup is ideal if:

- You want private networking in AKS (via Azure Private Link)
- You need granular access control over the cluster
- You want to avoid managing full VPN appliances or bastion hosts

---

### What We'll Build

- A private AKS cluster with no public API server
- A Twingate Connector running in the same VNet
- Twingate configured with the AKS API server as a protected resource
- Remote access to the cluster using `kubectl` via Twingate

---

### Infrastructure Setup

- Provision a private AKS cluster

You can use `az cli` or Terraform (recommended):

```bash
az aks create \
  --name dev-private \
  --resource-group dev-private_group \
  --enable-private-cluster \
  --vnet-subnet-id /subscriptions/.../subnets/aks-subnet \
  --node-count 2 \
  --generate-ssh-keys
```

Terraform:

```hcl
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "rg-private-aks"
  location = "UK South"
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-private-aks"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "aks_subnet" {
  name                 = "snet-aks"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "private-aks-cluster"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "privateaks"

  default_node_pool {
    name       = "default"
    node_count = 2
    vm_size    = "Standard_DS2_v2"
    vnet_subnet_id = azurerm_subnet.aks_subnet.id
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin    = "azure"
    dns_service_ip    = "10.0.2.10"
    service_cidr      = "10.0.2.0/24"
    docker_bridge_cidr = "172.17.0.1/16"
  }

  api_server_access_profile {
    enable_private_cluster = true
  }

  tags = {
    Environment = "Private"
  }
}
```

The --enable-private-cluster flag ensures the Kubernetes API server is only accessible over the VNet.

To check the API server endpoint:

```bash
az aks show --name dev-private --resource-group dev-private_group --query privateFqdn
```

You’ll get something like:

```bash
dev-private-dns-xxxxxx.privatelink.uksouth.azmk8s.io
```

---

### Push Twingate Connector Image to Azure Container Registry (ACR)

If using a private ACR:

```bash
az acr login --name <your-acr>
docker pull --platform linux/amd64 twingate/connector:1
docker tag twingate/connector:1 <your-acr>.azurecr.io/twingate/connector:1
docker push <your-acr>.azurecr.io/twingate/connector:1
```

### Deploy Twingate Connector as Azure Container Instance

```bash
az container create \
  --name twingate-connector \
  --image <your-acr>.azurecr.io/twingate/connector:1 \
  --resource-group dev-private_group \
  --vnet dev-private_group-vnet \
  --subnet twingate \
  --cpu 1 \
  --memory 2 \
  --environment-variables \
      TWINGATE_NETWORK="mo-demo" \
      TWINGATE_ACCESS_TOKEN="<your-access-token>" \
      TWINGATE_REFRESH_TOKEN="<your-refresh-token>" \
      TWINGATE_TIMESTAMP_FORMAT="2" \
      TWINGATE_LABEL_DEPLOYED_BY="azure"
```

Terraform:

```hcl
resource "azurerm_container_group" "twingate_connector" {
  name                = "twingate-connector"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  os_type             = "Linux"

  container {
    name   = "twingate"
    image  = "youracrname.azurecr.io/twingate-connector:1"
    cpu    = "1"
    memory = "1.5"

    environment_variables = {
      TWINGATE_NETWORK        = "your_network"
      TWINGATE_ACCESS_TOKEN   = "your_access_token"
      TWINGATE_REFRESH_TOKEN  = "your_refresh_token"
      TWINGATE_LABEL_DEPLOYED_BY = "terraform"
    }
  }

  ip_address_type = "Private"
  subnet_ids      = [azurerm_subnet.aks_subnet.id]
}
```

### Configure Twingate

- Setup a Twingate account
- Create a new remote network in Twingate

###  Add the AKS API Server as a Twingate Resource

In the Twingate Admin Console:

- Create a new Remote Network for your VNet (e.g., aks-vnet)
- Deploy the Connector to that Remote Network (you already did)
- Add a Resource with the private AKS API DNS name
  - e.g., dev-private-dns-xxxxxx.privatelink.uksouth.azmk8s.io
  - Port: 443
- Assign the Resource to a group (e.g., engineering)

### Access the Cluster via Twingate

Once the Connector is live and you're authenticated via the Twingate client:

```bash
az aks get-credentials --resource-group dev-private_group --name dev-private
kubectl get nodes
```

It works—because your local traffic is tunneled securely through Twingate and routed to the private API server over the VNet!

You can also run commands remotely using:

```bash
az aks command invoke \
  --resource-group dev-private_group \
  --name dev-private \
  --command "kubectl get pods -A"
```

### Bonus: Ingress Whitelisting

If you're using an NGINX ingress controller, and want to restrict access to known IPs:

```bash
nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.4.5
```

### Summary

With this setup:

- Your AKS cluster is completely private
- Access is secured and identity-aware via Twingate
- No public exposure, no bastion, no hassle


Private AKS Cluster with Twingate: Secure API Access Without a Public Endpoint


## 🔍 Overview

AWS PrivateLink enables secure, private connectivity between VPCs and services hosted on AWS without exposing traffic to the public internet. It's essential for secure service-to-service communication across accounts or VPCs.

In this post, we'll implement a working PrivateLink setup with Terraform: one VPC exposing a service (via NLB), and another VPC accessing it (via interface endpoint). We'll use two Terraform providers to simulate cross-account setup.

---

## 🛠️ Architecture

```
[ Service Provider VPC ]                  [ Service Consumer VPC ]
[ EC2 (httpd) ] -> [ NLB ] -> [ Endpoint Service ] <- [ VPC Endpoint ] <- [ EC2 ]
```

The service is hosted on a private EC2 behind a Network Load Balancer. An Endpoint Service is created from the NLB. The consumer VPC accesses it through an Interface VPC Endpoint.

---

## 🧱 Prerequisites

- Terraform ≥ 1.0
- Two AWS profiles or roles simulating service provider and consumer
- Basic networking knowledge (subnets, NLB, SGs)

---

## 🧩 Step 1: Service Provider VPC with NLB and Endpoint Service

```go
provider "aws" {
  alias  = "provider"
  region = "us-east-1"
}

resource "aws_vpc" "provider" {
  provider   = aws.provider
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "provider_subnet" {
  provider          = aws.provider
  vpc_id            = aws_vpc.provider.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_security_group" "provider_sg" {
  provider = aws.provider
  vpc_id   = aws_vpc.provider.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "provider_instance" {
  provider        = aws.provider
  ami             = "ami-0c94855ba95c71c99"
  instance_type   = "t2.micro"
  subnet_id       = aws_subnet.provider_subnet.id
  security_groups = [aws_security_group.provider_sg.id]
  user_data       = <<-EOF
                #!/bin/bash
                echo "Hello from PrivateLink Service Provider" > index.html
                nohup python -m SimpleHTTPServer 80 &
                EOF
}

resource "aws_lb" "nlb" {
  provider           = aws.provider
  name               = "privatelink-nlb"
  internal           = true
  load_balancer_type = "network"
  subnets            = [aws_subnet.provider_subnet.id]
}

resource "aws_lb_target_group" "tg" {
  provider    = aws.provider
  name        = "privatelink-tg"
  port        = 80
  protocol    = "TCP"
  vpc_id      = aws_vpc.provider.id
  target_type = "instance"
}

resource "aws_lb_target_group_attachment" "tg_attachment" {
  provider         = aws.provider
  target_group_arn = aws_lb_target_group.tg.arn
  target_id        = aws_instance.provider_instance.id
  port             = 80
}

resource "aws_lb_listener" "listener" {
  provider          = aws.provider
  load_balancer_arn = aws_lb.nlb.arn
  port              = 80
  protocol          = "TCP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tg.arn
  }
}

resource "aws_vpc_endpoint_service" "service" {
  provider                    = aws.provider
  acceptance_required        = true
  network_load_balancer_arns = [aws_lb.nlb.arn]
}
```

---

## 🧩 Step 2: Service Consumer VPC with VPC Endpoint

```go
provider "aws" {
  alias  = "consumer"
  region = "us-east-1"
}

resource "aws_vpc" "consumer" {
  provider   = aws.consumer
  cidr_block = "10.1.0.0/16"
}

resource "aws_subnet" "consumer_subnet" {
  provider          = aws.consumer
  vpc_id            = aws_vpc.consumer.id
  cidr_block        = "10.1.1.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_security_group" "consumer_sg" {
  provider = aws.consumer
  vpc_id   = aws_vpc.consumer.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.1.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_vpc_endpoint" "consumer_endpoint" {
  provider           = aws.consumer
  vpc_id             = aws_vpc.consumer.id
  service_name       = aws_vpc_endpoint_service.service.service_name
  vpc_endpoint_type  = "Interface"
  subnet_ids         = [aws_subnet.consumer_subnet.id]
  security_group_ids = [aws_security_group.consumer_sg.id]
}
```

---

## 🧪 Testing

- Accept the VPC Endpoint connection from the Provider side.
- SSH into an EC2 in the Consumer VPC and use:

```bash
curl http://<interface-endpoint-dns-name>
```

You should see:

```
Hello from PrivateLink Service Provider
```

---

## 🔐 Security Notes

- Ensure correct SGs between NLB and Endpoint
- Use IAM condition keys or resource policies if needed
- Enable VPC Flow Logs for visibility

---

## 📌 Summary

- We implemented AWS PrivateLink with real infra using Terraform
- Service is exposed internally via NLB and Interface Endpoint
- Setup is secure, scalable, and avoids internet exposure

Next steps? Add DNS, IAM auth, or cross-region routing!

---

A hands-on technical guide to implementing AWS PrivateLink between VPCs using Terraform.

AWS PrivateLink with Terraform


## Introduction

This is Part 1 of our Container Networking Deep Dive series. In this hands-on deep dives, we show how to simulate container-like networking using Linux primitives: ip netns, veth pairs, and routing tables.

We're building it all from scratch. Think of it like plugging an Ethernet cable between two interface cards — one inside a Linux network namespace (our "container") and the other in the host.

### Why Should You Care?

Understanding the low-level mechanics and first principles behind container networking — without relying on Docker or Kubernetes — gives you ultimate debugging power. This series is designed to demystify how containers, pods, bridges, overlays and CNI plugins actually work under the hood.

### Scenario Overview

Prerequisites:

- Make sure to have `multipass` installed (if on macOS) or have a VM setup via UTM/VirtualBox.

We’ll set up:

- A VM using Multipass
- A network namespace (what containers are under the hood)
- A veth pair connecting host and namespace
- IP addresses on both ends
- Routing so they can talk

Goal: ping the namespace from the host and vice versa.

## Deep Dive: Step-by-Step Setup

### Provisioning the VM

Create a Makefile to provision the VM and transfer the files to it.

```bash
NAME=netns-lab
IMAGE=22.04
SCRIPT=scenario1.sh

up:
	@if multipass info $(NAME) >/dev/null 2>&1; then \
		echo "$(NAME) already exists. Skipping launch."; \
	else \
		echo "Launching $(NAME) VM..."; \
		multipass launch --name $(NAME) --cloud-init cloud-init.yaml --memory 1G --disk 5G; \
	fi
	@chmod +x env.sh scenario1.sh test.sh
	@echo "Transferring files to VM..."
	@for file in env.sh scenario1.sh test.sh; do \
		multipass transfer $$file $(NAME):/home/ubuntu/ || echo "Failed to transfer $$file"; \
	done
```

env.sh is a simple script to set up the environment variables.

```bash
CON="netns1"
IP="10.200.1.1"
```

```bash
#!/bin/bash -e

. /home/ubuntu/env.sh

echo "[+] Creating the namespace"
sudo ip netns add $CON

echo "[+] Creating the veth pair"
sudo ip link add veth1 type veth peer name veth2

echo "[+] Moving one end to namespace"
sudo ip link set veth2 netns $CON

echo "[+] Assigning IP inside namespace"
sudo ip netns exec $CON ip addr add $IP dev veth2

echo "[+] Bringing up interfaces"
sudo ip netns exec $CON ip link set veth2 up
sudo ip netns exec $CON ip link set lo up
sudo ip link set veth1 up

echo "[+] Routing setup"
sudo ip route add $IP/32 dev veth1 || true
sudo ip netns exec $CON ip route add default via $IP dev veth2 || true
```

We setup:

- We create a namespace called `netns1`
- We create a veth pair called `veth1` and `veth2`
- We move one end to the namespace
- We assign an IP address to the interface in the namespace
- We bring up the interfaces
- We add a route to the namespace from the host
- We add a default route to the namespace from the host

### Test connectivity

```bash
#!/bin/bash

NS=netns1
IP=10.200.1.1

check() {
  echo "[TEST] $1"
  eval "$2"
  echo ""
}

fail_if_empty() {
  if [ -z "$($1)" ]; then
    echo "[FAIL] $2"
    exit 1
  else
    echo "[PASS] $2"
  fi
}

check "Namespace exists" "sudo ip netns list | grep -q $NS && echo 'Found namespace $NS'"
fail_if_empty "sudo ip netns list | grep $NS" "Namespace '$NS' exists"

check "Interface in namespace" "sudo ip netns exec $NS ip link show veth2"
check "Interface on host" "sudo ip link show veth1"
check "Route on host to $IP" "sudo ip route | grep $IP"
check "Ping from host to namespace" "sudo ping -c 3 $IP"
check "Ping from namespace to host IP" "sudo ip netns exec $NS ping -c 3 $IP"
check "Routes inside namespace" "sudo ip netns exec $NS ip route"
```

### Rest of makefile

```bash
run:
	@echo "Running scenario script..."
	@multipass exec $(NAME) -- bash /home/ubuntu/$(SCRIPT)

shell:
	@multipass shell $(NAME)

test:
	@multipass exec $(NAME) -- bash /home/ubuntu/test.sh

destroy:
	@echo "Destroying VM..."
	@multipass delete $(NAME) --purge || echo "$(NAME) does not exist."
```

## Takeaways & Lessons learned

- `ip netns` is the real container under the hood.
- A veth pair gives you point-to-point connectivity. Similar to what a virtual Ethernet cable does.
- You control the routing tables just like in real containers.
- Network namespaces give you isolation.
- veth pairs connect those isolated spaces.
- Routing needs to be explicit.
- You can debug container networking using raw Linux tools.

## Conclusion

We’ve now created our first standalone network namespace with basic connectivity from host to namespace. This is how most CNI plugins start under the hood.

In the next part of the series, we’ll look at creating two namespaces on the same node and wiring them together using a bridge — effectively building a virtual switch.

Stay tuned for Part 2!


In the first part of our Container Networking Deep Dive, we explore how to set up a single network namespace inside a VM and connect it to the host using a veth pair.

Container Networking Deep Dive Part 1: Single Network Namespace on a VM


## Introduction

### What Are We Building? 🧱

In this second part of the series, we connect two network namespaces (netns1, netns2) to the same virtual bridge on a single VM. This mirrors how Kubernetes pods share a flat L2 network on a single node.

### Why This Matters

This setup is the basis of many container networks. It replicates the behavior of a Linux bridge or a virtual switch (like what Docker and most CNIs do).

## Scenario Overview

- Two namespaces: `netns1`, `netns2`
- Each has a veth interface (`veth11`, `veth21`)
- The host side of the veth pairs (`veth10`, `veth20`) is connected to a Linux bridge `br0`
- The bridge has IP `172.16.0.1`
- `netns1` gets `172.16.0.2`, `netns2` gets `172.16.0.3`
- All interfaces are on the same /24 subnet

## Architecture

![cns2](/images/cns2.png)

## Setup

We'll use the same setup script as in the previous part.

```bash
NAME=scenario2
IMAGE=22.04

up:
	@if multipass info $(NAME) >/dev/null 2>&1; then \
		echo "$(NAME) already exists. Skipping launch."; \
	else \
		echo "Launching $(NAME)..."; \
		multipass launch --name $(NAME) --memory 1G --disk 5G; \
	fi
	@chmod +x env.sh setup.sh test.sh
	@echo "Transferring files..."
	@for f in env.sh setup.sh test.sh; do \
		multipass transfer $$f $(NAME):/home/ubuntu/; \
	done
```

Now run the VM setup.

```bash
make up
```

env.sh contains the variables for the scenario.

```bash
CON1="netns1"
CON2="netns2"
NODE_IP="10.0.0.10"
BRIDGE_IP="172.16.0.1"
IP1="172.16.0.2"
IP2="172.16.0.3"
```

### Creating the Namespaces & Interfaces

setup.sh creates the namespaces and interfaces.

```bash
#!/bin/bash -e
. /home/ubuntu/env.sh

sudo ip netns add $CON1
sudo ip netns add $CON2

sudo ip link add veth10 type veth peer name veth11
sudo ip link add veth20 type veth peer name veth21

sudo ip link set veth11 netns $CON1
sudo ip link set veth21 netns $CON2

sudo ip netns exec $CON1 ip addr add $IP1/24 dev veth11
sudo ip netns exec $CON2 ip addr add $IP2/24 dev veth21

sudo ip netns exec $CON1 ip link set veth11 up
sudo ip netns exec $CON2 ip link set veth21 up
```

### Connecting via Bridge

```bash
sudo ip link add name br0 type bridge
sudo ip link set veth10 master br0
sudo ip link set veth20 master br0
sudo ip addr add $BRIDGE_IP/24 dev br0
```

### Bridging everything together

```bash
sudo ip link set br0 up
sudo ip link set veth10 up
sudo ip link set veth20 up
sudo ip netns exec $CON1 ip link set lo up
sudo ip netns exec $CON2 ip link set lo up
```

### Routing configuration

```bash
sudo ip netns exec $CON1 ip route add default via $BRIDGE_IP dev veth11
sudo ip netns exec $CON2 ip route add default via $BRIDGE_IP dev veth21
```

## Manual testing

Conditions to test:

- Ping from `netns1` to `netns2`
- Ping from `netns2` to `netns1`
- Ping from host to `netns1` & host to `netns2`
- Ping from `netns1` to host & `netns2` to host. 

## AutomatedTesting

test.sh contains the tests for the scenario.

```bash
#!/bin/bash
. /home/ubuntu/env.sh

check() {
  echo "[TEST] $1"
  eval "$2"
  echo ""
}

fail_if_empty() {
  if [ -z "$($1)" ]; then
    echo "[FAIL] $2"
    exit 1
  else
    echo "[PASS] $2"
  fi
}

fail_if_empty "sudo ip netns list | grep $CON1" "Namespace $CON1 exists"
fail_if_empty "sudo ip netns list | grep $CON2" "Namespace $CON2 exists"

check "IP in $CON1" "sudo ip netns exec $CON1 ip a show dev veth11"
check "IP in $CON2" "sudo ip netns exec $CON2 ip a show dev veth21"
check "Ping $IP2 from $CON1" "sudo ip netns exec $CON1 ping -c 3 $IP2"
check "Ping $IP1 from $CON2" "sudo ip netns exec $CON2 ping -c 3 $IP1"
check "Ping bridge from $CON1" "sudo ip netns exec $CON1 ping -c 3 $BRIDGE_IP"
check "Ping bridge from $CON2" "sudo ip netns exec $CON2 ping -c 3 $BRIDGE_IP"
check "Bridge info" "ip a show br0"
```

Run with `make test`.

## Makefile workflow

```bash
run:
	multipass exec $(NAME) -- bash /home/ubuntu/setup.sh
test:
	multipass exec $(NAME) -- bash /home/ubuntu/test.sh
shell:
	multipass shell $(NAME)
destroy:
	multipass delete $(NAME) --purge || echo "Nothing to destroy."
```

```bash
make run # Run the setup script
make test # Run the test script
make shell # Drop into the VM
make destroy # Clean up
```

## Key Takeaways

- A Linux bridge behaves like a virtual switch
- All devices on the same bridge can talk L2 directly
- Namespaces need explicit routing (default route via bridge IP)
- Debugging tools like `ip a`, `ip r`, `ip link` help massively when it comes to networking & troubleshooting.

## Conclusion

You now have a working L2 network across multiple isolated namespaces. This is the basis for pod networking on a single node.

Next up in Part 3: we’ll span this setup across two VMs, and show how to keep everything on the same subnet using a shared L2 segment or VXLAN overlay.


In the second part of our Container Networking Deep Dive, we connect two network namespaces via a bridge on the same Linux host.

Container Networking Deep Dive Part 2: Two Namespaces on the Same Host


## Deep Dive into EC2 Networking

When working with Amazon EC2, networking isn’t just a checkbox—it's the core of how instances connect, communicate, and scale. One of the most fundamental components in EC2 networking is the Elastic Network Interface (ENI).

## Elastic Network Interface (ENI)

An ENI is essentially a virtual network card in the cloud. It acts as a bridge between your EC2 instance and your Virtual Private Cloud (VPC). Every EC2 instance must be launched with a primary network interface, which is automatically created unless you explicitly provide a custom one.

Key Characteristics:

- Every ENI has:
  - A primary private IPv4 address (static)

  - Optionally one or more secondary private IPv4 addresses

  - One or more security groups
  - A MAC address
  - A source/destination check flag

- ENIs are standalone resources and can be moved between EC2 instances.

- EC2 instances can attach multiple ENIs (limits depend on instance type).

## IP Addressing

- Primary Private IPv4 Address:

  - Assigned to the ENI via DHCP

  - Static for the lifetime of the ENI

  - Persists even if the ENI is detached from an instance

- Secondary Private IPv4 Addresses:

  - Useful for multi-tenant applications or apps needing multiple IPs

  - Also attached to the ENI, not the EC2 instance

- Public IPv4 Addressing:

  - Automatically assigned only if the subnet has "auto-assign public IP" enabled or manually specified

  - These are ephemeral and released on stop/terminate

- Elastic IPs:

  - Static public IPv4 address

  - Can be attached to a specific private IP on an ENI

  - Charged when not attached to a running instance

## Advanced Use Cases with Multiple ENIs

1. Dual-Homed Instances

  - Separate traffic by roles:

    - Web requests on one ENI

    - Backend/database traffic on another

2. High Availability (HA)

  - Failover with secondary ENIs:

    - A secondary ENI with its own private IP is used as the access point

    - If Instance A fails, move the ENI to Instance B

    - Client continues using the same IP without needing DNS updates

3. MAC Address Licensing

  - Some legacy software ties licenses to MAC addresses:

  - Attach ENI with known MAC to a new instance

  - Software continues working without re-licensing

4. Security Appliances

  - Build your own virtual firewall or proxy:

  - One ENI receives traffic

  - Instance runs the firewall software

  - Another ENI routes traffic to backend

## Security Control Points

ENIs are also where security groups and NACLs are applied:

- Security Groups: Stateful, attached to ENIs

- NACLs: Stateless, attached to subnets

By using multiple ENIs, you can apply different security profiles per interface.

## Deployment Strategies: Bootstrapping vs. AMI Baking

### Bootstrapping

- Attach a user data script to EC2 at launch

- Uses cloud-init to:

  - Install packages

  - Configure apps

  - Register with systems (e.g., load balancers, config mgmt)

Pros:

  - Flexible per environment

  - Config/data never baked in

Cons:

  - Slower provisioning ("ready for service lag")

### AMI Baking

- Install & configure app on a running instance

- Create an AMI from that instance

- Launch new instances from that AMI

Pros:

  - Fast provisioning

  - Predictable and repeatable

Cons:

  - Less flexibility for last-minute config

  - Must update and re-bake for changes

### Combined Architecture (Best Practice)

- Install base app stack, perform time-consuming setup, create an AMI

- Launch instances from baked AMI, but pass custom config via user data

This gives you fast launch time with just-enough flexibility:

- Baked AMI handles the heavy lifting (installations, dependencies)

- Bootstrap config customizes per environment (e.g., ENV vars, credentials, instance-specific metadata)

## Real-World Example:

Imagine a three-tier app:

- Web Tier (public subnet)

- App Tier (private subnet)

- DB Tier (private subnet)

The App Tier EC2 instance can be multi-homed:

- ENI1 in Web Subnet: Handles traffic from the Web Tier

- ENI2 in DB Subnet: Handles traffic to the database

- Different SGs/NACLs for each ENI to segregate access control

- If the App Tier fails, detach ENI1 and ENI2 and reattach to a hot standby instance—instant failover with no DNS propagation needed.

## Summary

ENIs are at the heart of EC2 networking:

- Control IPs, MACs, and security

- Enable flexible architectures like dual-homing, HA, firewalls, and licensing portability

- Allow you to separate traffic types and security contexts

When it comes to instance provisioning:

- Use AMI baking for speed

- Use bootstrapping for flexibility

- Use both for production-grade performance and maintainability

- Start small: one interface, one IP. Then scale out with ENIs and Elastic IPs to build truly production-grade cloud infrastructure.

## Further Reading:

- [AWS ENI Docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html)

- [Best Practices for EC2 Launches](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-best-practices.html)

- [AMI Baking with EC2 Image Builder](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ami-builder.html)

Deep Dive into EC2 Networking: ENIs, IP Addressing and Deployment Architectures

Deep Dive into EC2 Networking


## 🧠 Everyday Network Troubleshooting: Tools You Should Know (and Actually Use)

![Networking Tools](/images/networking-tools.png)

Whether you're debugging why a pod can't reach an endpoint or trying to figure out why your app's requests are timing out, networking tools are your first line of defense.

Here’s a curated, opinionated, and battle-tested list of tools I use (almost) daily. Each one comes with practical examples.

## 🚀 Connectivity & Reachability

### 🔹 ping

Checks if a host is alive using ICMP echo requests.

```bash
ping google.com
ping -c 4 192.168.1.1 # -c 4 is to send 4 packets
```

🔧 Tip: If ping doesn't work, the host could be up but blocking ICMP. Check firewall rules.

### 🔹 traceroute

Visualise the path packets take to a remote host. Great for pinpointing bottlenecks.

```bash
traceroute google.com
traceroute -I 192.168.1.1 # -I is to use ICMP echo requests
```

🧠 Use case: “Why is latency high between service A and B?” → traceroute.

### 🔹 nslookup

Simple tool for querying DNS. Quick and dirty.

```bash
nslookup google.com
nslookup 8.8.8.8
```

### 🔹 dig

More verbose and powerful DNS query tool than nslookup.

```bash
dig google.com
dig google.com ANY # ANY is to get all records
dig google.com A +short # A is to get A records
dig google.com AAAA +short # AAAA is to get AAAA records
```

🧠 Bonus: Use +trace to debug DNS resolution path:

```bash
dig +trace google.com
dig +trace google.com +short # +short is to get the IP address
```

## 🌐 Port Scanning & TCP Checks

### 🔹 netcat (nc)

The Swiss Army Knife of TCP/UDP.

```bash
nc -zv google.com 80
nc -l 1234  # Listen on port 1234
nc -zv 192.168.1.1 1-65535 # -z is to scan all ports
```

📡 Tip: nc can be used to spin up fake HTTP servers or test port exposure in firewalled networks.

### 🔹 telnet

Old school, but works.

```bash
telnet google.com 80
```

🎯 Can be used to debug HTTP, SMTP, Redis manually.

## 🛠 Interface & Routing Info

### 🔹 ip (modern)

```bash
ip addr show
ip route show
ip link set eth0 up
```

### 🔹 ifconfig (legacy but familiar)

```bash
ifconfig
ifconfig eth0 up
```

### 🔹 route (macOS)

```bash
route -n
route add default gw 192.168.1.1
```

## 📊 Monitoring Traffic & Usage

### 🔹 netstat

```bash
netstat -an
netstat -r
```

### 🔹 ss (modern netstat replacement)

```bash
ss -tuln
ss -s
```

🔹 tcpdump
Capture packets like a boss.

```bash
tcpdump -i eth0
tcpdump -i eth0 -w capture.pcap
```

🎯 Combine with wireshark to visualize packets.

🔹 tshark
CLI version of Wireshark.

```bash
tshark -i eth0
tshark -r capture.pcap
```

### 🔹 vnstat

Track bandwidth usage.

```bash
vnstat -l
vnstat -d
```

### 🔹 nload / nethogs

Real-time network I/O monitor per interface or per process.

bash
```bash
nload
nethogs
```

## 📶 Wireless Tools

```bash
iw
iwlist
iwconfig
```

🧠 These are useful on laptops, Raspberry Pi setups, or Linux wireless APs.

## 🔐 Security, Scanning & Recon

### 🔹 nmap

Port scanner and network mapper.

```bash
nmap google.com
nmap -sP 192.168.1.0/24
```

🧠 Pro tip: Use with -A for OS detection, versioning, script scanning.

### 🔹 whois

Lookup domain ownership.

```bash
whois google.com
```

### 🔹 lsof

See what ports your system is listening on.

```bash
lsof -i :80
lsof -i tcp
```

### 🔹 arp / arping

```bash
arp -a
arp -d 192.168.1.1
```

```bash
arping 192.168.1.1
```

🧠 Use for static IP-to-MAC mapping debugging or LAN sniffing.

🧪 Bandwidth Testing
### 🔹 iperf
Client-server bandwidth tester.

```bash
iperf -s          # Start server
iperf -c <ip>     # Run client against server
```

📦 Works great for diagnosing slow internal links or tunnels.

### 🔹 mtr

Combines ping + traceroute with live stats.

```bash
mtr google.com
mtr google.com
mtr -r google.com
```

🧠 Instant visibility into jitter, loss, and latency by hop.

🧠 Bonus Mentions

```bash
conntrack – Show conntrack entries
ssdp – Debug multicast/UPnP
ncdu – Disk usage but worth knowing if you're debugging slow apps
hostname – Quick check or change the hostname
```

🎯 My Top 5 Daily Use Tools

| Purpose	| Tool |
| Ping test	| ping |
| DNS Resolution	| dig |
| Port connectivity	| nc / telnet |
| Interface info	| ip addr |
| Live traffic debug	| tcpdump |

✨ Final Thoughts

These tools are deceptively simple—but when chained together, they help you uncover:

- Firewall misconfigurations
- DNS issues
- Interface problems
- Packet loss/jitter
- Port blocks or misroutes
- Host reachability vs app-level downtime

🛠️ Bookmark this. Refer to it next time you're troubleshooting networking issues.

Posts tagged with: networking