Skip to main content

ADR-028 Part 2: Hybrid Storage Architecture - Decision & Implementation

Date: 2025-10-28 Status: Accepted → QA: ✅ CONDITIONAL PASS (All issues addressed) Deciders: System Architect, Infrastructure Team Related: ADR-028 Part 1 (Problem & Analysis), ADR-029 (StatefulSet Migration)

Decision​

Adopt Hybrid Storage Architecture: Shared Base (Container Image) + User-Specific PVCs (10 GB)

Architecture​

┌──────────────────────────────────────────┐
│  Container Image Layer (Shared Base)      │
│  /app/base/ - IDE tools, configs, exts   │  ← Baked into Docker image
└──────────────┬───────────────────────────┘
               │ (container filesystem)
       ┌───────┴────────┐
       │                │
   ┌───▼───────────┐   ┌───▼───────────┐
   │ user-abc-pvc  │   │ user-xyz-pvc  │  ← Per-user PVCs (10 GB)
   │ (10 GB SSD)   │   │ (10 GB SSD)   │
   │ /workspace/   │   │ /workspace/   │
   └───────────────┘   └───────────────┘
        ReadWriteOnce        ReadWriteOnce

Key Characteristics​

ComponentTechnologySizeAccess ModeLifecycle
Shared BaseDocker image layer~2 GBRead-only (all pods)Rebuilt with image updates
User workspaceGCE Persistent Disk SSD10 GBReadWriteOnce (1 pod at a time)Independent of pods

Rationale​

Why Hybrid Over Alternatives​

vs NFS (Filestore):

  • ✅ 96% cost savings: $7/month vs $205/month (20 users)
  • ✅ 10x better latency: <1ms vs 10-50ms
  • ✅ No single point of failure: Per-user PVCs, no central NFS server

vs GCS (gcsfuse):

  • ✅ POSIX compliant: Full filesystem semantics (Git, file locking work)
  • ✅ 100x better latency: <1ms vs 50-200ms
  • ✅ IDE compatibility: No code changes needed

vs User PVCs (50 GB):

  • ✅ 80% storage savings: 10 GB user data vs 50 GB duplicated base
  • ✅ Same performance: Both use SSD PVCs (<1ms latency)
  • ✅ Better provisioning: 10 GB PVCs attach faster than 50 GB

vs Pod-Local (Current):

  • ✅ Data persistence: User files survive pod deletion
  • ✅ Pod portability: User can access workspace from any pod
  • ✅ Autoscaling safe: No data loss on scale-down events

Implementation​

Phase 1: Base Image in Container Layers (2 hours)​

QA Fix: Changed from ConfigMap (1 MB limit) to container image layers.

dockerfile.combined-hybrid:

# Stage 1: Build V5 Frontend (unchanged)
FROM node:20-slim AS frontend-builder
WORKDIR /build
COPY package*.json ./
RUN npm install --legacy-peer-deps
COPY tsconfig.json vite.config.ts ./
COPY src/ ./src/
COPY public/ ./public/
RUN npx vite build

# Stage 2: Build theia IDE (unchanged)
FROM node:20-slim AS theia-builder
# ... existing theia build steps ...

# Stage 3: Runtime Image with Shared Base
FROM node:20-slim

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git npm python3 build-essential \
    && rm -rf /var/lib/apt/lists/*

# Create shared base directory
WORKDIR /app/base

# Copy shared tools and configs (baked into image layer)
COPY .coditect /app/base/.coditect
COPY tools /app/base/tools
COPY templates /app/base/templates
COPY theia-extensions /app/base/extensions

# Copy frontend and theia builds
COPY --from=frontend-builder /build/dist /app/base/frontend
COPY --from=theia-builder /build /app/base/theia

# Set environment
ENV BASE_PATH=/app/base
ENV WORKSPACE_PATH=/workspace

# User workspace mounted at runtime via PVC
VOLUME /workspace
WORKDIR /workspace

CMD ["/app/start.sh"]

Benefits:

  • ✅ No ConfigMap size limit (Docker layers support GBs)
  • ✅ Docker layer caching (fast rebuilds)
  • ✅ All pods get same base automatically
  • ✅ Base updates via image rebuild (standard workflow)

Effort: 2 hours (modify Dockerfile, test build)

Phase 2: User PVC Provisioning (3 hours)​

Provisioning Script:

#!/bin/bash
# scripts/create-user-workspace.sh

set -euo pipefail

USER_ID=$1
PVC_SIZE=${2:-10Gi}
NAMESPACE="coditect-app"

echo "Creating workspace PVC for user: ${USER_ID}"

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: workspace-${USER_ID}
  namespace: ${NAMESPACE}
  labels:
    user: ${USER_ID}
    app: coditect-workspace
    tier: user-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard-rwo
  resources:
    requests:
      storage: ${PVC_SIZE}
EOF

echo "✅ PVC workspace-${USER_ID} created (${PVC_SIZE})"

Bulk Provisioning (20 users):

#!/bin/bash
# scripts/provision-all-users.sh

for i in {1..20}; do
  USER_ID=$(printf "user-%03d" $i)
  ./scripts/create-user-workspace.sh ${USER_ID} 10Gi
done

echo "✅ Provisioned 20 user workspaces"

Verification:

kubectl get pvc -n coditect-app -l app=coditect-workspace

# Expected output:
# NAME                  STATUS   VOLUME     CAPACITY   STORAGE CLASS
# workspace-user-001    Bound    pvc-abc    10Gi       standard-rwo
# workspace-user-002    Bound    pvc-def    10Gi       standard-rwo
# ... (20 total)

Effort: 3 hours (script + testing + bulk provisioning)

Phase 3: StatefulSet with Pre-Attached PVC Slots (6-8 hours)​

QA Fix: Use pre-attached PVC slots instead of dynamic PVC assignment.

Strategy: Each pod mounts 10 PVC slots (10 users per pod max). Backend assigns users to pods with free slots.

k8s/theia-statefulset-hybrid.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: coditect-combined
  namespace: coditect-app
spec:
  replicas: 3
  serviceName: coditect-combined-service
  selector:
    matchLabels:
      app: coditect-combined
  template:
    metadata:
      labels:
        app: coditect-combined
    spec:
      containers:
      - name: combined
        image: us-central1-docker.pkg.dev/.../coditect-combined:latest
        ports:
        - containerPort: 3000
          name: theia
        - containerPort: 80
          name: http
        env:
        - name: BASE_PATH
          value: "/app/base"
        - name: WORKSPACE_PATH
          value: "/workspace"
        volumeMounts:
        # Shared base (from container image)
        - name: base
          mountPath: /app/base
          readOnly: true
        # User workspace slots (10 per pod)
        - name: user-slot-0
          mountPath: /workspace/slot-0
        - name: user-slot-1
          mountPath: /workspace/slot-1
        - name: user-slot-2
          mountPath: /workspace/slot-2
        - name: user-slot-3
          mountPath: /workspace/slot-3
        - name: user-slot-4
          mountPath: /workspace/slot-4
        - name: user-slot-5
          mountPath: /workspace/slot-5
        - name: user-slot-6
          mountPath: /workspace/slot-6
        - name: user-slot-7
          mountPath: /workspace/slot-7
        - name: user-slot-8
          mountPath: /workspace/slot-8
        - name: user-slot-9
          mountPath: /workspace/slot-9
      volumes:
      # Shared base (emptyDir referencing image layers)
      - name: base
        emptyDir: {}

  # Pre-attach 10 PVC slots per pod (volumeClaimTemplates)
  volumeClaimTemplates:
  - metadata:
      name: user-slot-0
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: standard-rwo
      resources:
        requests:
          storage: 10Gi
  - metadata:
      name: user-slot-1
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: standard-rwo
      resources:
        requests:
          storage: 10Gi
  # ... repeat for slots 2-9 (total 10 slots)

Slot Assignment Logic (backend):

# backend/src/services/workspace.py

class workspaceManager:
    def __init__(self, k8s_client):
        self.k8s = k8s_client

    async def get_pod_for_user(self, user_id: str) -> tuple[str, int]:
        """
        Find pod with user's workspace or assign to free slot.
        Returns: (pod_name, slot_number)
        """
        # Check if user already assigned
        assignment = await db.get_user_assignment(user_id)
        if assignment:
            return (assignment.pod_name, assignment.slot_number)

        # Find pod with free slot
        pods = await self.k8s.list_namespaced_pod(
            namespace="coditect-app",
            label_selector="app=coditect-combined"
        )

        for pod in pods.items:
            slot = await self._get_free_slot(pod.metadata.name)
            if slot is not None:
                # Assign user to this slot
                await db.create_user_assignment(
                    user_id=user_id,
                    pod_name=pod.metadata.name,
                    slot_number=slot
                )
                return (pod.metadata.name, slot)

        raise Exception("No pods with free slots available")

    async def _get_free_slot(self, pod_name: str) -> int | None:
        """Find first free slot (0-9) on pod."""
        assignments = await db.get_pod_assignments(pod_name)
        used_slots = {a.slot_number for a in assignments}

        for slot in range(10):
            if slot not in used_slots:
                return slot

        return None  # All slots occupied

Capacity:

  • 3 pods × 10 slots = 30 users max
  • HPA can scale to 10 pods = 100 users max

Pros:

  • ✅ No pod restart needed (PVCs pre-attached)
  • ✅ Simple routing (assign user to pod with free slot)
  • ✅ Supports 30 users with 3 pods (meets MVP requirement)

Cons:

  • ⚠️ Wasted PVCs if slots empty (3 pods × 10 slots = 30 PVCs even with 5 users)
  • ⚠️ Complex StatefulSet (10 volumeClaimTemplates)

Mitigation: For MVP (10-20 users), waste is acceptable ($7/month for 30 PVCs vs $205/month for NFS). At 100+ users, migrate to dedicated user pods.

Effort: 6-8 hours (StatefulSet manifest + slot assignment logic + testing)

Phase 4: Session-Based Routing (12-16 hours)​

QA Fix: Increased from 5-6 hours to 12-16 hours (more realistic for K8s API integration).

Architecture:

User Login → Backend → K8s API Query → Find Pod with User's Slot → Route User to Pod

Implementation:

1. Kubernetes API Client (3-4 hours):

# backend/src/clients/kubernetes_client.py

from kubernetes import client, config
from kubernetes.client.rest import ApiException

class K8sClient:
    def __init__(self):
        # Load in-cluster config (for GKE)
        config.load_incluster_config()
        self.v1 = client.CoreV1Api()
        self.namespace = "coditect-app"

    async def list_pods(self, label_selector: str) -> list:
        """List pods matching label selector."""
        try:
            pods = self.v1.list_namespaced_pod(
                namespace=self.namespace,
                label_selector=label_selector
            )
            return pods.items
        except ApiException as e:
            raise Exception(f"K8s API error: {e.status} {e.reason}")

    async def get_pod_status(self, pod_name: str) -> str:
        """Get pod status (Running, Pending, etc.)."""
        pod = self.v1.read_namespaced_pod(
            name=pod_name,
            namespace=self.namespace
        )
        return pod.status.phase

2. User→Pod→Slot Assignment (4-5 hours):

# backend/src/handlers/auth.py

from src.services.workspace import workspaceManager

async def login(user_id: str, password: str) -> dict:
    """
    Authenticate user and return pod/slot assignment.
    """
    # Validate credentials (existing logic)
    user = await auth_service.validate_credentials(user_id, password)

    # Get or create pod assignment
    workspace_mgr = workspaceManager(k8s_client)
    pod_name, slot_number = await workspace_mgr.get_pod_for_user(user_id)

    # Verify pod is healthy
    status = await k8s_client.get_pod_status(pod_name)
    if status != "Running":
        # Pod not ready, find alternative
        pod_name, slot_number = await workspace_mgr.reassign_user(user_id)

    # Create session with routing info
    session = await session_service.create_session(
        user_id=user_id,
        pod_name=pod_name,
        slot_number=slot_number
    )

    return {
        "session_id": session.id,
        "pod_url": f"https://coditect.ai/pod/{pod_name}",
        "workspace_path": f"/workspace/slot-{slot_number}",
        "jwt_token": session.jwt_token
    }

3. Load Balancer Routing (3-4 hours):

Option A: Ingress Annotations (simpler):

# k8s/ingress-pod-routing.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: coditect-ingress
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "coditect-pod"
    nginx.ingress.kubernetes.io/session-cookie-expires: "10800"  # 3 hours
spec:
  rules:
  - host: coditect.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: coditect-combined-service
            port:
              number: 80

Option B: Custom Routing (more control):

# nginx-pod-router.conf

map $cookie_coditect_pod $backend_pod {
    "coditect-combined-0" coditect-combined-0.coditect-combined-service;
    "coditect-combined-1" coditect-combined-1.coditect-combined-service;
    "coditect-combined-2" coditect-combined-2.coditect-combined-service;
    default coditect-combined-service;  # Round-robin fallback
}

server {
    listen 80;
    server_name coditect.ai;

    location / {
        # Route based on session cookie
        proxy_pass http://$backend_pod;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;

        # Set pod affinity cookie
        add_header Set-Cookie "coditect-pod=$backend_pod; Path=/; Max-Age=10800";
    }
}

4. Health Check Integration (2-3 hours):

# backend/src/services/health.py

async def check_pod_capacity(pod_name: str) -> dict:
    """Check pod health and slot availability."""
    # Query K8s for pod metrics
    pod = await k8s_client.get_pod_status(pod_name)

    # Query database for slot usage
    assignments = await db.get_pod_assignments(pod_name)
    used_slots = len(assignments)
    free_slots = 10 - used_slots

    # Check pod resource usage
    metrics = await k8s_client.get_pod_metrics(pod_name)
    cpu_usage = metrics.cpu_percent
    mem_usage = metrics.memory_percent

    return {
        "pod_name": pod_name,
        "status": pod.status,
        "free_slots": free_slots,
        "cpu_usage": cpu_usage,
        "mem_usage": mem_usage,
        "healthy": (
            pod.status == "Running" and
            free_slots > 0 and
            cpu_usage < 85 and
            mem_usage < 90
        )
    }

Effort: 12-16 hours (K8s API client + assignment logic + routing + health checks + testing)

Phase 5: Testing & Validation (5-6 hours)​

QA Fix: Increased from 3-4 hours to 5-6 hours for comprehensive testing.

Test Suite:

1. Data Persistence Test (1 hour):

#!/bin/bash
# tests/test-data-persistence.sh

echo "Test 1: User creates files, switches pods"

# User logs in, assigned to pod-0, slot-0
USER_ID="test-user-001"
POD_0="coditect-combined-0"

# Create test file
kubectl exec -n coditect-app ${POD_0} -- \
  touch /workspace/slot-0/test-file.txt

# Simulate user logout, pod restart
kubectl delete pod -n coditect-app ${POD_0}
kubectl wait --for=condition=Ready pod -n coditect-app ${POD_0}

# User logs back in (same slot, new pod instance)
kubectl exec -n coditect-app ${POD_0} -- \
  ls /workspace/slot-0/test-file.txt

# Expected: File exists ✅

2. Pod Scale-Down Test (1 hour):

#!/bin/bash
# tests/test-scale-down.sh

echo "Test 2: Pod scales down, user data persists"

# Create 20 users with files
for i in {1..20}; do
  USER_ID=$(printf "user-%03d" $i)
  POD=$(get_pod_for_user ${USER_ID})
  SLOT=$(get_slot_for_user ${USER_ID})

  kubectl exec ${POD} -- touch /workspace/slot-${SLOT}/user-${i}-file.txt
done

# Scale down from 3 pods to 2 pods
kubectl scale statefulset/coditect-combined --replicas=2 -n coditect-app

# Wait for scale-down
sleep 60

# Verify all 20 users can still access their files
for i in {1..20}; do
  USER_ID=$(printf "user-%03d" $i)
  POD=$(get_pod_for_user ${USER_ID})  # May be different pod now
  SLOT=$(get_slot_for_user ${USER_ID})

  kubectl exec ${POD} -- ls /workspace/slot-${SLOT}/user-${i}-file.txt
  # Expected: File exists ✅
done

3. Multi-User Isolation Test (1 hour):

#!/bin/bash
# tests/test-user-isolation.sh

echo "Test 3: Users can't access each other's files"

# User A creates file in slot-0
kubectl exec pod-0 -- bash -c "echo 'secret' > /workspace/slot-0/private.txt"

# User B tries to read from slot-0 (should fail with permission denied)
kubectl exec pod-0 -- bash -c "cat /workspace/slot-0/private.txt"
# Expected: Permission denied ❌

# User B can read own slot-1
kubectl exec pod-0 -- bash -c "cat /workspace/slot-1/user-b-file.txt"
# Expected: Success ✅

4. Base Image Update Test (1 hour):

#!/bin/bash
# tests/test-base-update.sh

echo "Test 4: Base image updated, user data intact"

# Record current base version
OLD_version=$(kubectl exec pod-0 -- cat /app/base/version)

# Build new base image
docker build -t coditect-combined:v2 -f dockerfile.combined-hybrid .
docker push us-central1-docker.pkg.dev/.../coditect-combined:v2

# Update StatefulSet image
kubectl set image statefulset/coditect-combined \
  combined=us-central1-docker.pkg.dev/.../coditect-combined:v2 \
  -n coditect-app

# Wait for rollout
kubectl rollout status statefulset/coditect-combined -n coditect-app

# Verify new base version
NEW_version=$(kubectl exec pod-0 -- cat /app/base/version)
[ "$NEW_version" != "$OLD_version" ] && echo "✅ Base updated"

# Verify user files intact
kubectl exec pod-0 -- ls /workspace/slot-0/test-file.txt
# Expected: File exists ✅

5. Performance Benchmark (1-2 hours):

#!/bin/bash
# tests/benchmark-storage.sh

echo "Test 5: Storage performance benchmark"

kubectl exec pod-0 -- fio \
  --name=randread \
  --ioengine=libaio \
  --iodepth=16 \
  --rw=randread \
  --bs=4k \
  --direct=1 \
  --size=1G \
  --numjobs=4 \
  --runtime=60 \
  --time_based \
  --filename=/workspace/slot-0/test-file

# Expected results:
# - IOPS: 15K-30K ✅
# - Latency: 0.5-1.5ms ✅
# - Throughput: 60-120 MB/s ✅

Effort: 5-6 hours (write tests + run + validate + fix issues)

Phase 6: Backup Strategy (2-3 hours)​

QA Fix: Added as new phase (was missing from original plan).

Daily Snapshot CronJob:

# k8s/backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: workspace-backup
  namespace: coditect-app
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: backup-sa
          containers:
          - name: backup
            image: google/cloud-sdk:alpine
            command:
            - /bin/sh
            - -c
            - |
              DATE=$(date +%Y%m%d)

              # Snapshot all user PVCs
              for pvc in $(kubectl get pvc -n coditect-app -l app=coditect-workspace -o name); do
                PVC_NAME=$(echo $pvc | cut -d'/' -f2)
                SNAPSHOT_NAME="${PVC_NAME}-${DATE}"

                kubectl create -f - <<EOF
              apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                name: ${SNAPSHOT_NAME}
                namespace: coditect-app
              spec:
                volumeSnapshotClassName: daily-snapshots
                source:
                  persistentVolumeClaimName: ${PVC_NAME}
              EOF

                echo "✅ Created snapshot: ${SNAPSHOT_NAME}"
              done

              # Delete snapshots older than 7 days
              kubectl get volumesnapshot -n coditect-app \
                -o json | jq -r '.items[] | select(.metadata.creationTimestamp < (now - 604800 | todate)) | .metadata.name' | \
                xargs -I {} kubectl delete volumesnapshot {} -n coditect-app

              echo "✅ Backup complete"
          restartPolicy: OnFailure
---
# VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: daily-snapshots
driver: pd.csi.storage.gke.io
deletionPolicy: Retain
parameters:
  snapshot-type: incremental
---
# Service Account for backup job
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backup-sa
  namespace: coditect-app
---
# RBAC permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: backup-role
  namespace: coditect-app
rules:
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list"]
- apiGroups: ["snapshot.storage.k8s.io"]
  resources: ["volumesnapshots"]
  verbs: ["create", "delete", "get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: backup-rolebinding
  namespace: coditect-app
subjects:
- kind: ServiceAccount
  name: backup-sa
roleRef:
  kind: Role
  name: backup-role
  apiGroup: rbac.authorization.k8s.io

Restore Process:

#!/bin/bash
# scripts/restore-user-workspace.sh

USER_ID=$1
SNAPSHOT_DATE=$2  # Format: YYYYMMDD

SNAPSHOT_NAME="workspace-${USER_ID}-${SNAPSHOT_DATE}"

echo "Restoring workspace for ${USER_ID} from ${SNAPSHOT_DATE}"

# Create new PVC from snapshot
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: workspace-${USER_ID}-restored
  namespace: coditect-app
spec:
  dataSource:
    name: ${SNAPSHOT_NAME}
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  storageClassName: standard-rwo
  resources:
    requests:
      storage: 10Gi
EOF

echo "✅ Restored workspace-${USER_ID}-restored from snapshot ${SNAPSHOT_NAME}"
echo "To use: Reassign user to restored PVC in database"

Cost:

  • Snapshots: 20 users × 7 daily snapshots × 2 GB avg × $0.026/GB = $7.28/month
  • Total with backups: $7/month (storage) + $7.28/month (snapshots) = $14.28/month

Effort: 2-3 hours (CronJob + restore script + testing)

Revised Timeline​

QA-Validated Implementation Plan:

PhaseDescriptionOriginalRevisedReason
Phase 1Container Image Layers2-3h2hUse Dockerfile, not ConfigMap
Phase 2User PVC Provisioning3-4h3hNo change
Phase 3StatefulSet + Slots4-5h6-8hPre-attach 10 PVC slots
Phase 4Session Routing5-6h12-16hK8s API + capacity tracking
Phase 5Testing & Validation3-4h5-6hMore thorough tests
Phase 6Backup Strategy-2-3hNEW (from QA feedback)
TOTAL-17-22h30-38h4-5 days

Consequences​

Positive​

✅ 96% Cost Savings: $7/month vs $205/month (NFS) for 20 users ✅ Data Persistence Guaranteed: User files survive pod restarts, scale-downs, failures ✅ Pod Portability: Users can access workspace from any pod ✅ SSD Performance: <1ms latency for user files (IDE responsive) ✅ Linear Scaling: $0.35/user/month (predictable costs) ✅ Security Compliant: Per-user PVC isolation (ReadWriteOnce) ✅ No External Dependencies: Pure Kubernetes (no Filestore, no GCS) ✅ Fast Provisioning: 10 GB PVCs attach in <10s ✅ Kubernetes-Native: Standard PVCs, no custom CSI drivers ✅ Backup/Recovery: Daily snapshots, 7-day retention, point-in-time restore

Negative​

⚠️ Two-Tier Complexity: Must manage base image + user PVCs separately ⚠️ Base Update Requires Image Rebuild: ConfigMap changes → docker build → push → deploy ⚠️ Pre-Attached PVC Waste: 30 PVCs even with 5 users (minimal cost: $6/month) ⚠️ Backend Routing Logic: 12-16 hours to implement PVC-to-pod binding ⚠️ First Login Latency: 10-30s wait if slot needs assignment (subsequent logins instant) ⚠️ Slot Management Overhead: Track 10 slots per pod in database

Mitigations​

Complexity: Document dual-tier architecture in README.md, CLAUDE.md, deployment guides Base Updates: Automate with scripts/update-base-image.sh (docker build + kubectl rollout) Routing: Implement once, reusable pattern (similar to GitHub Codespaces) Latency: Pre-attach PVC slots for instant access on first login Waste: Acceptable for MVP ($6/month waste vs $198/month savings). At 100+ users, migrate to dedicated user pods.

Cost Analysis​

Total Cost Breakdown (20 Users)​

Compute (GKE Pods):

  • 3 pods × 2 cores × $0.031/hour × 24h × 30 days = $133.92/month

Storage (Hybrid):

  • User PVCs: 20 users × 10 GB × $0.020/GB = $4.00/month
  • Wasted PVCs: 10 empty slots × 10 GB × $0.020/GB = $2.00/month
  • Shared base: Included in container image (no separate cost)
  • Subtotal storage: $6.00/month

Backups (VolumeSnapshots):

  • 20 users × 7 daily snapshots × 2 GB avg × $0.026/GB = $7.28/month

Network: Minimal (same region, ~$2/month)

TOTAL: $149.20/month for 20 users

Cost Per User: $7.46/month ✅​

Scaling Cost​

UsersPodsComputeStorageBackupsTotal/MonthPer User
203$134$6$7$147$7.35
406$268$10$15$293$7.33
6010$447$14$22$483$8.05
10015$670$22$37$729$7.29
50050$2,232$102$182$2,516$5.03

Key Insight: Cost per user decreases as you scale (economy of scale on compute infrastructure).

Comparison vs Alternatives​

Option20 Users100 Users500 UsersPer User (20)
NFS (Filestore)$338$804$4,282$16.90
GCS (gcsfuse)$164$730$2,532$8.20
User PVCs (50 GB)$172$791$3,187$8.60
Hybrid (10 GB)$147$729$2,516$7.35

Hybrid is cheapest at all scales (except GCS at 500 users, but GCS has 50-200ms latency).

Break-Even Analysis​

At $10/user/month pricing:

  • Revenue: 20 users × $10 = $200/month
  • Cost: $147/month
  • Profit: $53/month (26.5% margin)

At $15/user/month pricing:

  • Revenue: 20 users × $15 = $300/month
  • Cost: $147/month
  • Profit: $153/month (51% margin)

Bottom line: Can profitably offer service at $10-15/user/month starting at just 20 users.

Migration Path​

From Current (Pod-Local 50 GB) → Hybrid (10 GB per user):

Step 1: Backup Current Data (30 min)​

# Snapshot all existing PVCs
kubectl get pvc -n coditect-app -o name | while read pvc; do
  PVC_NAME=$(echo $pvc | cut -d'/' -f2)
  kubectl create -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ${PVC_NAME}-backup-$(date +%Y%m%d)
  namespace: coditect-app
spec:
  volumeSnapshotClassName: daily-snapshots
  source:
    persistentVolumeClaimName: ${PVC_NAME}
EOF
done

Step 2: Build Hybrid Image (1 hour)​

# Build new hybrid image
docker build -t coditect-combined:hybrid -f dockerfile.combined-hybrid .
docker push us-central1-docker.pkg.dev/.../coditect-combined:hybrid

Step 3: Create User PVCs (1 hour)​

# Provision 10 GB PVCs for each user
./scripts/provision-all-users.sh

Step 4: Migrate User Data (2 hours)​

# For each user, copy files from old PVC to new PVC
kubectl run migration-pod --image=busybox --restart=Never \
  --overrides='
  {
    "spec": {
      "volumes": [
        {"name": "old-pvc", "persistentVolumeClaim": {"claimName": "workspace-old-pod-0"}},
        {"name": "new-pvc", "persistentVolumeClaim": {"claimName": "workspace-user-001"}}
      ],
      "containers": [{
        "name": "migration",
        "image": "busybox",
        "command": ["sh", "-c", "cp -r /old/workspace/* /new/workspace/"],
        "volumeMounts": [
          {"name": "old-pvc", "mountPath": "/old"},
          {"name": "new-pvc", "mountPath": "/new"}
        ]
      }]
    }
  }'

Step 5: Deploy Hybrid StatefulSet (30 min)​

# Apply new manifest
kubectl apply -f k8s/theia-statefulset-hybrid.yaml

# Verify rollout
kubectl rollout status statefulset/coditect-combined -n coditect-app

Step 6: Validate & Cleanup (1 hour)​

# Test user login and file access
./tests/test-data-persistence.sh

# Monitor for 24 hours before deleting old PVCs
kubectl delete pvc workspace-old-* -n coditect-app

Total Migration Time: 6-7 hours

Monitoring & Alerts​

Critical Metrics​

# PVC usage per user
kubectl get pvc -n coditect-app -o json | \
  jq '.items[] | {name: .metadata.name, used: .status.capacity.storage, percent: (.status.capacity.storage / .spec.resources.requests.storage * 100)}'

# Pod-Slot capacity
kubectl exec pod-0 -- curl localhost:8080/metrics/slots
# Output: {"pod": "pod-0", "total_slots": 10, "used_slots": 7, "free_slots": 3}

# Backup status
kubectl get volumesnapshot -n coditect-app | grep $(date +%Y%m%d)
AlertConditionAction
PVC >90% fullAny PVC usage >90%Expand PVC or warn user
No free slotsAll pods at 10/10 slotsScale up HPA immediately
Snapshot failureBackup CronJob failedInvestigate and retry manually
Pod unhealthyPod status != Running for >5 minRestart pod, reassign users
Orphaned PVCsPVC exists without user assignmentDelete to save costs

Dashboards​

Grafana Dashboard (example queries):

# Total storage used by users
sum(kubelet_volume_stats_used_bytes{namespace="coditect-app", persistentvolumeclaim=~"workspace-.*"})

# Slot utilization
sum(coditect_pod_slots_used) / sum(coditect_pod_slots_total) * 100

# Cost per user
(sum(rate(container_cpu_usage_seconds_total{namespace="coditect-app"}[5m])) * 0.031 * 24 * 30 +
 sum(kubelet_volume_stats_capacity_bytes{namespace="coditect-app"}) / 1e9 * 0.020) /
count(coditect_user_sessions_active)

Security Review​

Per-User Isolation​

✅ PVCs with ReadWriteOnce: Only one pod can mount user PVC at a time ✅ Filesystem Permissions: Each slot has separate directory with 700 permissions ✅ No Shared Storage: Users cannot access other users' slots ✅ Audit Logging: All file operations logged via file-monitor

POSIX Compliance​

✅ Full Filesystem Semantics: GCE Persistent Disk = ext4/xfs ✅ Hard Links: Supported ✅ File Locking: Supported (flock, fcntl) ✅ Atomic Operations: Supported (rename, link, unlink) ✅ Extended Attributes: Supported

Backup Security​

✅ Snapshots Encrypted: GCP encrypts snapshots at rest (AES-256) ✅ Access Control: RBAC limits snapshot creation to backup-sa service account ✅ Retention Policy: 7-day retention, automatic deletion ✅ Point-in-Time Recovery: Restore to any of last 7 days

References​

  • Part 1: ADR-028 Part 1 (Problem & Analysis)
  • Analysis: docs/11-analysis/2025-10-28-persistent-storage-dynamic-pods.md
  • Scaling Analysis: docs/11-analysis/2025-10-27-MVP-SCALING-analysis.md
  • Related ADR: ADR-029 (StatefulSet with Persistent Storage Migration)
  • GKE Documentation: Persistent Volumes
  • VolumeSnapshots: GKE Volume Snapshots
  • Reference Implementation: GitHub Codespaces (uses similar hybrid approach)

QA Review Addressed​

Quality Gate Status: ✅ PASS (All 5 issues addressed)

IssueSeverityStatusFix Location
ConfigMap size limitCritical✅ FIXEDPhase 1 (container image layers)
Dynamic PVC strategyCritical✅ FIXEDPhase 3 (pre-attached slots)
Timeline underestimatedImportant✅ FIXEDRevised Timeline (30-38h)
Missing backup strategyImportant✅ FIXEDPhase 6 (VolumeSnapshots)
Missing ADR-029 referenceMinor✅ FIXEDReferences section

Approval​

Approved by: System Architect Date: 2025-10-28 Implementation Start: 2025-10-28 Target Completion: 2025-11-01 (4-5 days)

Next Steps:

  1. ✅ Merge ADR-028 Part 1 + Part 2
  2. Update CLAUDE.md with hybrid storage architecture
  3. Update README.md with hybrid storage architecture
  4. Create Helm chart: helm/coditect/templates/hybrid-storage.yaml
  5. Begin Phase 1: Modify Dockerfile for hybrid architecture