EKS Hybrid Nodes on Azure

Extend an Amazon EKS cluster to Azure by running Azure VMs as Kubernetes worker nodes — managed from a single EKS control plane.

This project provisions the full stack end-to-end: an EKS cluster, a site-to-site IPsec VPN between AWS and Azure, Cilium CNI for cross-cloud pod networking, and Azure VMs that automatically join the cluster via cloud-init. Everything is driven from one config.env file.

Why

Running Kubernetes across clouds is hard. You need encrypted connectivity, identity federation without long-lived secrets, overlay networking that works across providers, and automation that ties it all together. This project solves that:

Challenge	How it's solved
Cross-cloud connectivity	IPsec VPN tunnel (AES-256) between AWS VPN Gateway and Azure VPN Gateway
Authentication without static keys	IAM Roles Anywhere — X.509 certificates auto-rotated every hour
Pod networking across clouds	Cilium CNI with Geneve tunneling for seamless pod-to-pod communication
Configuration drift	Single config.env drives every script — zero hardcoded values
Node provisioning	cloud-init auto-joins Azure VMs to the EKS cluster on first boot

When to Use This

Core principle: Orchestrate anywhere, compute where the data lives.

Use Case	Why EKS Hybrid on Azure?
Data lives in Azure	Run compute next to Azure Blob/SQL — avoid moving terabytes over the wire
Existing Azure infrastructure	Add Kubernetes orchestration to VMs you already own without migrating them to AWS
GPU/HPC on Azure	Tap into Azure NC/ND-series GPUs managed from a single EKS control plane
Data sovereignty	Keep data in an Azure region for compliance while using EKS for orchestration
Gradual migration	Move workloads to AWS incrementally — hybrid nodes bridge the transition
Burst capacity	Overflow to Azure VMs when AWS capacity is constrained or reserved instances are full

See Architecture & Use Cases for detailed scenarios, advantages, and trade-offs.

Architecture

flowchart LR
    subgraph AWS["AWS VPC (10.0.0.0/16)"]
        direction TB
        EKS["EKS Control Plane"]
        subgraph EC2["EC2 Node Group"]
            N1["Node 1\n(VPC CNI)"]
            N2["Node 2\n(VPC CNI)"]
        end
        VGW["VPN Gateway"]
        IAM["IAM Roles\nAnywhere"]
        EKS --> EC2
    end

    subgraph Azure["Azure VNet (10.80.0.0/16)"]
        direction TB
        AZVPN["VPN Gateway"]
        subgraph Subnet["Workload Subnet (10.80.1.0/24)"]
            HN1["Hybrid Node 1\n(Cilium CNI)"]
            HN2["Hybrid Node N\n(Cilium CNI)"]
        end
        AZVPN --- Subnet
    end

    VGW <-->|"IPsec VPN\n(AES-256)"| AZVPN
    EKS -->|"K8s API"| HN1 & HN2
    IAM -.->|"X.509 cert auth\n(auto-rotate 1hr)"| HN1 & HN2

    subgraph Pods["Pod Networking"]
        direction LR
        AWSP["AWS Pods\n(VPC CNI)"]
        AZP["Hybrid Pods\n(10.81.0.0/16)"]
    end

    EC2 --> AWSP
    Subnet --> AZP
    AWSP <-->|"Geneve Tunnel"| AZP

    style AWS fill:#ff9900,color:#fff,stroke:#cc7a00
    style Azure fill:#0078d4,color:#fff,stroke:#005a9e
    style Pods fill:#2d6a4f,color:#fff,stroke:#1b4332
    style EC2 fill:#ffcc80,color:#000,stroke:#cc7a00
    style Subnet fill:#80b3e0,color:#000,stroke:#005a9e

Loading

What gets created:

AWS	Azure	Networking
EKS cluster with hybrid node support	Resource Group	IPsec VPN tunnel between gateways
EC2 managed node group	VNet + workload/gateway subnets	Cilium CNI (Geneve overlay)
VPN Gateway + Customer Gateway	VPN Gateway (VpnGw1)	NSG + Security Group rules
IAM Roles Anywhere (Trust Anchor, Profile)	Ubuntu 22.04 VMs with cloud-init	Route tables for cross-cloud traffic

Quick Start

Prerequisites

• AWS: CLI configured, eksctl v0.150.0+, kubectl, helm
• Azure: CLI authenticated (az login), Contributor access
• Local: OpenSSL, SSH, Bash

1. Configure

cp config.env.template config.env
nano config.env   # Set AWS_ACCOUNT_ID, AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID

2. Deploy

cd setup && ./deploy-all.sh

Deploys everything in order. Takes ~60-90 min (Azure VPN Gateway creation is the bottleneck at ~45 min).

3. Verify

kubectl get nodes -o wide           # 2 AWS nodes + Azure hybrid nodes
./setup/check-vpn-status.sh         # VPN tunnel health
kubectl apply -f examples/test-hybrid-workload.yaml
kubectl get pods -o wide             # Pods running on Azure with 10.81.x.x IPs

4. Cleanup

cd cleanup && ./cleanup-all.sh       # ~25 min, saves ~$670/month

Project Structure

config.env.template                  # All configuration lives here
setup/
  00-load-config.sh                  # Config loader (all scripts source this)
  01-deploy-eks-cluster.sh           # EKS + eksctl (20 min)
  02-setup-aws-vpn.sh               # VPN Gateway + Customer Gateway (5 min)
  03-setup-azure-infrastructure.sh   # VNet, VPN Gateway, NSG, routes (45 min)
  04-connect-vpn-tunnels.sh          # Wire up AWS <-> Azure VPN (5 min)
  05-install-cilium.sh               # Cilium CNI for hybrid pods (5 min)
  06-create-azure-vm.sh [N]          # Azure VM + auto-join via cloud-init (10 min)
  07-join-hybrid-node.sh             # Manual join (optional, for re-joins)
  deploy-all.sh                      # Runs everything end-to-end
  check-vpn-status.sh               # VPN health check utility
  aws-eks-cluster.yaml.template      # eksctl config template (envsubst)
cleanup/
  01-07 cleanup scripts              # Reverse-order teardown
  cleanup-all.sh                     # Full cleanup with confirmation
examples/
  test-hybrid-workload.yaml          # Basic hybrid node test
  cross-cloud-communication-test.yaml
  multi-cloud-deployment.yaml
  spark-pi-job.yaml                  # Apache Spark on hybrid nodes
docs/
  01-architecture-and-use-cases.md   # Design deep-dive
  02-connectivity-options.md         # VPN vs Direct Connect vs Cloud Exchange
  03-iam-roles-anywhere.md           # Certificate auth handshake explained
  04-implementation-guide.md         # Step-by-step walkthrough

How It Works

flowchart LR
    A["config.env"] -->|"source"| B["00-load-config.sh"]
    B --> C["Setup Scripts\n01-07"]
    C -->|"envsubst"| D["eksctl YAML"]
    C -->|"state files"| E["~/.eks-hybrid-azure/\n$CLUSTER_NAME/*.env"]
    E -->|"read by"| F["Cleanup Scripts\n01-07"]
    B --> F

    style A fill:#6c757d,color:#fff
    style B fill:#ffc107,color:#000
    style E fill:#17a2b8,color:#fff

Loading

All configuration flows from config.env. Scripts save runtime state (VPN IDs, IPs, ARNs) to ~/.eks-hybrid-azure/$CLUSTER_NAME/ so subsequent scripts and cleanup can reference them. Every script is idempotent — safe to re-run.

Configuration

All values are set in config.env (copy from config.env.template). Key settings:

Variable	Default	Purpose
CLUSTER_NAME	spark-hybrid-test	Names all AWS/Azure resources
AWS_REGION	us-west-2	AWS region
AZURE_LOCATION	westus2	Azure region
AWS_VPC_CIDR	10.0.0.0/16	AWS VPC network
AZURE_VNET_CIDR	10.80.0.0/16	Azure VNet network
HYBRID_POD_CIDR	10.81.0.0/16	Cilium pod overlay
K8S_VERSION	1.34	Kubernetes version
NODE_INSTANCE_TYPE	t3.medium	AWS EC2 instance type
AZURE_VM_SIZE	Standard_D2s_v3	Azure VM size

CIDRs must not overlap. The defaults are chosen to avoid collisions.

Cost

Resource	Monthly
EKS Cluster	$73
EC2 Nodes (2x)	$60
Azure VM	$70
AWS VPN	$36
Azure VPN Gateway	$140
NAT Gateway + other	$291
Total	~$670/month

Save 30-40% with Reserved Instances. Delete resources when not in use.

Scale

Add more hybrid nodes at any time:

./setup/06-create-azure-vm.sh 2    # Adds hybrid-node-2
./setup/06-create-azure-vm.sh 3    # Adds hybrid-node-3

Each VM auto-joins the cluster via cloud-init. No manual SSH or configuration needed.

Documentation

Guide	Description
Quick Start	Fastest path to a running cluster
Architecture & Use Cases	Design decisions, network layout, use cases
Connectivity Options	VPN vs Direct Connect vs Cloud Exchange
IAM Roles Anywhere	Certificate auth handshake, trust chain, rotation
Implementation Guide	Detailed step-by-step with code examples
Setup Guide	Script-by-script deployment reference
Cleanup Guide	Teardown details and cost savings
Examples	Spark, cross-cloud, multi-cloud workloads

Troubleshooting

Symptom	Debug command	Common cause
VPN not connecting	./setup/check-vpn-status.sh	PSK mismatch, NSG blocking UDP 500/4500
Node not joining	ssh ... sudo journalctl -u kubelet -n 100	Certificate issue, IAM Roles Anywhere misconfigured
Pods stuck Pending	kubectl -n kube-system logs ds/cilium	Cilium IPAM pool exhausted
Node NotReady	kubectl describe node <name>	Cilium not running on hybrid node

See the Implementation Guide for detailed troubleshooting.

Important

• Never commit config.env — it contains account IDs and is gitignored
• Certificates are gitignored — generate fresh ones per deployment
• Set up billing alerts — this stack costs ~$22/day
• Test in dev first — don't deploy to production without validation

Contributing

1. Fork the repository
2. Create a feature branch
3. Test end-to-end
4. Submit a pull request

License

Apache License 2.0 — see LICENSE for details.