Skip to main content

StatefulSet Migration - Quick Start

⏱️ Time Required: 2-4 hours 💾 Data Loss: None (old data was ephemeral anyway) ⏸️ Downtime: ~2-5 minutes during migration

🚀 One-Command Migration​

# Run automated migration script
./scripts/migrate-to-statefulset.sh

That's it! The script handles everything:

  • ✅ Creates BackendConfig for session affinity
  • ✅ Deletes old Deployment
  • ✅ Creates StatefulSet with persistent storage
  • ✅ Updates Ingress
  • ✅ Tests persistence

📋 What Gets Created​

Kubernetes Resources​

ResourceNamePurpose
StatefulSetcoditect-combined3 pods with stable names
PVCsworkspace-coditect-combined-{0,1,2}50GB persistent storage per pod
PVCstheia-config-coditect-combined-{0,1,2}5GB config storage per pod
Servicetheia-headlessHeadless service for StatefulSet
Servicecoditect-combined-serviceLoad balancer with session affinity
BackendConfigcoditect-backend-configSession affinity + timeout config
Ingresscoditect-ingressWebSocket + cookie-based affinity

File Structure Created​

k8s/
├── theia-statefulset.yaml         ← Main StatefulSet config
├── backend-config-stateful.yaml   ← Session affinity config
└── ingress-stateful.yaml          ← Updated Ingress

scripts/
├── migrate-to-statefulset.sh      ← Automated migration
├── test-persistence.sh            ← Test data persistence
└── test-session-affinity.sh       ← Test sticky sessions

docs/11-analysis/
└── STATEFULSET-migration-guide.md ← Complete documentation

✅ Quick Validation​

Test 1: Verify StatefulSet Running​

kubectl get statefulset coditect-combined -n coditect-app

Expected: 3/3 READY

Test 2: Verify PVCs Created​

kubectl get pvc -n coditect-app

Expected: 6 PVCs (3 workspace + 3 config), all Bound

Test 3: Test Persistence​

./scripts/test-persistence.sh

Expected: ✅ All tests passed, file persists across pod restart

Test 4: Test in Browser​

  1. Open: https://coditect.ai/theia
  2. Create file: /workspace/test.txt
  3. Logout
  4. Login again
  5. File should still exist ✅

🔧 How It Works​

Before Migration (Data Loss Issue)​

User creates file.txt
    ↓
Stored in pod's ephemeral filesystem
    ↓
Pod restarts/deployment update
    ↓
File DELETED ❌

After Migration (Persistent Storage)​

User creates file.txt
    ↓
Stored in PVC (50GB GCE Persistent Disk)
    ↓
Pod restarts/deployment update
    ↓
File PERSISTS ✅

Session Affinity (Same Pod Every Time)​

User visits coditect.ai/theia
    ↓
Ingress assigns pod + sets cookie
    ↓
User creates files on pod-0
    ↓
User logs out and back in
    ↓
Cookie routes to pod-0 again
    ↓
User sees same files ✅

🎯 Key Features​

Persistent Storage:

  • 50GB /workspace volume per pod
  • Files persist across pod restarts
  • Files persist across deployments
  • Automatic GCE PD provisioning

Session Affinity:

  • Cookie-based routing (3-hour timeout)
  • ClientIP affinity at Service level
  • Users stick to same pod
  • Prevents "files disappeared" confusion

High Availability:

  • 3 replicas (pods)
  • Each pod has own workspace
  • Load balanced across pods
  • Graceful shutdown (120s grace period)

Pod Identity:

  • Stable names: coditect-combined-0, coditect-combined-1, coditect-combined-2
  • DNS: coditect-combined-0.theia-headless.coditect-app.svc.cluster.local
  • Predictable, debuggable

🔍 Troubleshooting​

Issue: Pods stuck in Pending​

Cause: PVC provisioning delay

Check:

kubectl describe pod coditect-combined-0 -n coditect-app

Solution: Wait 2-5 minutes for GCE Persistent Disk provisioning

Issue: Session affinity not working​

Cause: Ingress annotations not applied

Check:

kubectl get ingress coditect-ingress -n coditect-app -o yaml | grep affinity

Solution: Re-apply Ingress

kubectl apply -f k8s/ingress-stateful.yaml

Issue: Files still disappearing​

Cause: PVC not mounted correctly

Check:

kubectl exec -it coditect-combined-0 -n coditect-app -- df -h /workspace

Expected: Should show GCE PD mount, not overlay filesystem

Solution: Check StatefulSet volumeMounts configuration

📊 Monitoring Commands​

# Check StatefulSet status
kubectl get statefulset coditect-combined -n coditect-app

# Check pods
kubectl get pods -n coditect-app -l app=coditect-combined

# Check PVCs
kubectl get pvc -n coditect-app

# Check pod logs
kubectl logs coditect-combined-0 -n coditect-app

# Exec into pod
kubectl exec -it coditect-combined-0 -n coditect-app -- bash

# Check events
kubectl get events -n coditect-app --sort-by='.lastTimestamp' | tail -20

🔄 Rollback (If Needed)​

# Delete StatefulSet
kubectl delete statefulset coditect-combined -n coditect-app

# Re-create old Deployment
kubectl apply -f k8s/k8s-combined-deployment.yaml

# Verify
kubectl get deployment coditect-combined-v5 -n coditect-app

Note: Old Deployment had no persistence either, so no data lost in rollback.

📚 Documentation​

✅ Success Checklist​

  • Run migration script: ./scripts/migrate-to-statefulset.sh
  • Verify 3/3 pods running
  • Verify 6 PVCs bound (3 workspace + 3 config)
  • Run persistence test: ./scripts/test-persistence.sh
  • Run session affinity test: ./scripts/test-session-affinity.sh
  • Manual browser test (create file → logout → login → file exists)
  • Check no errors in pod logs
  • Monitor for 24 hours to ensure stability

Ready to migrate?

./scripts/migrate-to-statefulset.sh

Questions? See docs/11-analysis/STATEFULSET-migration-guide.md for complete documentation.