User request
  → Cloudflare DNS (api-eks.kortix.com)
  → AWS Application Load Balancer (ALB)
  → EKS cluster (suna-eks)          ← this is the primary
  → Your app pods

We also have:
  → Lightsail instance               ← legacy, still running
  → ECS cluster (suna-ecs)           ← legacy, still running

Cluster: suna-eks (EKS v1.31)
├── Node Group: suna-api-nodes
│   ├── Instance type: c7i.2xlarge (8 vCPU, 16 GB RAM)
│   ├── Min nodes: 2
│   ├── Max nodes: 8
│   └── Desired: 3
│
├── Namespace: suna
│   ├── Deployment: suna-api (our backend app)
│   │   ├── Base replicas: 4 pods
│   │   ├── Each pod: 500m-1500m CPU, 2Gi-3Gi memory
│   │   └── Each pod runs 2 Gunicorn workers
│   │
│   ├── Service: suna-api (ClusterIP, routes traffic to pods)
│   ├── HPA: suna-api (autoscales 4-15 pods based on CPU)
│   ├── PDB: suna-api (keeps at least 50% of pods alive during disruptions)
│   ├── Ingress: suna-api (ALB → internet-facing)
│   └── Secret: suna-env (all the env vars from Secrets Manager)
│
├── Namespace: kube-system
│   ├── AWS Load Balancer Controller (manages the ALB)
│   ├── Cluster Autoscaler (adds/removes nodes)
│   ├── CloudWatch Observability addon (sends metrics to CloudWatch)
│   └── Better Stack Collector (sends logs/metrics to Better Stack)
│
└── Namespace: default
    └── Better Stack Collector DaemonSet

infra/
├── environments/
│   ├── prod/index.ts        ← production stack (EKS + monitoring)
│   ├── dev/index.ts          ← dev (Lightsail only)
│   └── staging/index.ts      ← staging (Lightsail only)
│
├── modules/
│   ├── kubernetes/
│   │   ├── cluster.ts        ← EKS cluster + node groups
│   │   ├── workload.ts       ← Deployment, Service, HPA, PDB, Ingress
│   │   ├── autoscaler.ts     ← Cluster Autoscaler (Helm chart)
│   │   ├── iam.ts            ← IAM roles for nodes, ALB controller, autoscaler
│   │   └── types.ts          ← TypeScript interfaces
│   │
│   ├── monitoring/
│   │   ├── alarms.ts         ← CloudWatch alarms + dashboard
│   │   └── types.ts
│   │
│   └── lightsail/
│       ├── instance.ts       ← Lightsail for dev/staging
│       └── types.ts

Push to PRODUCTION
  → Build Docker image
  → Push to GHCR with tags `:prod` and `:<commit-sha>`
  → Deploy to Lightsail (SSH + docker compose)    } These three
  → Deploy to ECS (aws ecs update-service)         } run in
  → Deploy to EKS (kubectl set image)              } parallel

Step 1: Start 1 new pod with the new image
Step 2: Wait until the new pod passes health checks
Step 3: Remove 1 old pod
Step 4: Repeat until all pods are running the new image

Traffic spikes
  → Pods CPU goes above 70%
  → HPA: "I need more pods!"
  → HPA creates new pods
  → If nodes are full, new pods are "Pending" (can't be scheduled)
  → Cluster Autoscaler sees Pending pods
  → Cluster Autoscaler: "I'll add a node!"
  → New EC2 node joins the cluster (~3-5 minutes)
  → Pending pods get scheduled on the new node

Traffic drops
  → Pods CPU drops below target
  → HPA: "Too many pods, removing some" (after 5 min cooldown)
  → Pods get removed
  → Some nodes become underutilized (< 65% used)
  → Cluster Autoscaler: "This node is mostly empty, removing it" (after 5 min)
  → Pods on that node get moved to other nodes first
  → Empty node is terminated

Pod crashes
  → K8s immediately starts a new one (restartPolicy: Always)
  → Other pods keep serving traffic (no downtime)
  → If it crashes repeatedly, K8s uses "backoff" — waits longer between restarts
    (10s, 20s, 40s, 80s, ... up to 5 minutes)
  → This is called CrashLoopBackOff in pod status

Pod memory exceeds 3Gi limit
  → Linux kernel kills the process (OOMKilled)
  → K8s sees the container exited with code 137
  → K8s restarts the pod on the same node
  → If it keeps OOMing, it goes into CrashLoopBackOff

Node goes down
  → K8s notices within ~40 seconds (node heartbeat stops)
  → K8s marks all pods on that node as "Unknown"
  → After ~5 minutes, K8s evicts those pods
  → K8s schedules replacement pods on the remaining healthy nodes
  → If remaining nodes don't have enough room, Cluster Autoscaler adds a new node
  → Your PDB guarantees at least 50% of pods were on OTHER nodes already

App stops responding to /v1/health-docker
  → Readiness probe fails (3 times × 10s = 30 seconds)
  → K8s removes pod from load balancer (no new traffic)
  → Liveness probe fails (3 times × 30s = 90 seconds)
  → K8s kills and restarts the pod
  → Pod comes back up, passes probes, rejoins load balancer

New image deployed via rolling update
  → K8s starts 1 new pod with new image
  → New pod fails startup probe (crashes, health check fails, etc.)
  → K8s does NOT remove any old pods (maxUnavailable: 0)
  → Rollout is stuck — old pods keep serving traffic
  → After 5 minutes, the CI/CD timeout reports failure

# See what's wrong
kubectl describe pod -n suna -l app.kubernetes.io/name=suna-api

# Roll back to the previous version
kubectl rollout undo deployment/suna-api -n suna

# Verify
kubectl rollout status deployment/suna-api -n suna

All pods down
  → ALB health checks fail
  → ALB returns 503 to all requests
  → CloudWatch alarm fires (pod count < 1)
  → Email alert sent via SNS
  → Cluster Autoscaler adds new nodes
  → K8s reschedules pods on new nodes
  → ALB health checks pass, traffic resumes

# Check cluster status
kubectl get nodes
kubectl get pods -n suna

# Force restart all pods
kubectl rollout restart deployment/suna-api -n suna

# If nodes are gone, check AWS console for the managed node group
# Or scale it manually:
aws eks update-nodegroup-config \
  --cluster-name suna-eks \
  --nodegroup-name suna-api-nodes \
  --scaling-config minSize=2,maxSize=8,desiredSize=3

# See everything at a glance
bash infra/scripts/k8s-monitor.sh

# Or individual commands:
kubectl top nodes                          # Node CPU/memory
kubectl top pods -n suna                   # Pod CPU/memory
kubectl get pods -n suna                   # Pod status
kubectl get hpa -n suna                    # Autoscaler status
kubectl get events -n suna --sort-by='.lastTimestamp'  # Recent events

# What image is each pod running?
kubectl get pods -n suna -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

# Is the latest deploy live?
kubectl get deployment suna-api -n suna -o jsonpath='{.spec.template.spec.containers[0].image}'

# How many pods are running?
kubectl get deployment suna-api -n suna

# What's the HPA doing?
kubectl get hpa -n suna

# Graceful rolling restart (zero downtime)
kubectl rollout restart deployment/suna-api -n suna

# Watch the restart progress
kubectl rollout status deployment/suna-api -n suna

# Logs from a specific pod
kubectl logs <pod-name> -n suna

# Logs from all suna-api pods
kubectl logs -l app.kubernetes.io/name=suna-api -n suna --tail=100

# Follow logs in real-time
kubectl logs -l app.kubernetes.io/name=suna-api -n suna -f

# Logs from a crashed pod (previous container)
kubectl logs <pod-name> -n suna --previous

# Scale pods (temporarily — HPA may override this)
kubectl scale deployment/suna-api -n suna --replicas=6

# To permanently change, update the HPA min:
kubectl patch hpa suna-api -n suna -p '{"spec":{"minReplicas":6}}'

# Scale nodes (via AWS)
aws eks update-nodegroup-config \
  --cluster-name suna-eks \
  --nodegroup-name suna-api-nodes \
  --scaling-config minSize=3,maxSize=8,desiredSize=4

# See deployment history
kubectl rollout history deployment/suna-api -n suna

# Roll back to previous version
kubectl rollout undo deployment/suna-api -n suna

# Roll back to a specific revision
kubectl rollout undo deployment/suna-api -n suna --to-revision=3

# See why a pod isn't starting
kubectl describe pod <pod-name> -n suna

# Get a shell inside a running pod
kubectl exec -it <pod-name> -n suna -- /bin/bash

# See resource usage
kubectl top pod <pod-name> -n suna

# Node overview
kubectl get nodes -o wide

# Detailed node info (capacity, conditions, pods running on it)
kubectl describe node <node-name>

# What's running on each node
kubectl get pods -n suna -o wide

AWS Secrets Manager (suna-env-prod)
  → CI/CD syncs to K8s secret (on every deploy)
  → K8s secret: suna-env in namespace suna
  → Mounted as env vars in every pod

# Fetch from Secrets Manager and apply to K8s
SECRET_JSON=$(aws secretsmanager get-secret-value \
  --secret-id suna-env-prod \
  --query SecretString --output text)

kubectl create secret generic suna-env -n suna \
  --from-env-file=<(echo "$SECRET_JSON" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for k, v in data.items():
    print(f'{k}={v}')
") --dry-run=client -o yaml | kubectl apply -f -

# Restart pods to pick up new secrets
kubectl rollout restart deployment/suna-api -n suna

# Check what's waiting
kubectl get pods -n suna --field-selector=status.phase=Pending
kubectl describe pod <pending-pod> -n suna  # Look at "Events" section

# Check node capacity
kubectl top nodes

# The Cluster Autoscaler should handle this automatically.
# Check its logs if nodes aren't being added:
kubectl logs -l app.kubernetes.io/name=cluster-autoscaler -n kube-system --tail=50

# See why
kubectl logs <pod-name> -n suna --previous
kubectl describe pod <pod-name> -n suna

# Common causes:
# - Missing env vars (secret not synced)
# - Database connection failed
# - OOMKilled (check memory limits)
# - Bad image (roll back)

# Increase the number of nodes
aws eks update-nodegroup-config \
  --cluster-name suna-eks \
  --nodegroup-name suna-api-nodes \
  --scaling-config minSize=3,maxSize=8,desiredSize=4

# Roll back immediately
kubectl rollout undo deployment/suna-api -n suna

# Check what went wrong
kubectl logs -l app.kubernetes.io/name=suna-api -n suna --tail=200
kubectl get events -n suna --sort-by='.lastTimestamp'

# Find the EC2 instance ID
kubectl get nodes -o wide  # Note the INTERNAL-IP

# Use SSM Session Manager (no SSH key needed)
aws ssm start-session --target <instance-id>