Skip to main content

1-2-3 Deployment Guide - GCP Cloud Build

Complete step-by-step guide for deploying Coditect V5 to Google Kubernetes Engine

Table of Contents​

  1. Prerequisites
  2. Step 1: Prepare Source Code
  3. Step 2: Run Cloud Build
  4. Step 3: Verify Deployment
  5. Architecture Overview
  6. Key Files Reference
  7. Troubleshooting
  8. Post-Deployment Tasks

Prerequisites​

Required Access​

  • GCP Project: serene-voltage-464305-n2
  • GKE Cluster: codi-poc-e2-cluster (us-central1-a)
  • Permissions: Owner or editor role with Cloud Build and GKE access

Required Tools​

# Check installed versions
gcloud --version          # Google Cloud SDK 498.0.0+
docker --version          # Docker 20.10+
git --version             # Git 2.30+

GCP Services to Enable​

gcloud services enable cloudbuild.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable compute.googleapis.com

Step 1: Prepare Source Code​

1.1 Clone Repository​

git clone git@github.com:coditect-ai/Coditect-v5-multiple-llm-IDE.git
cd Coditect-v5-multiple-llm-IDE

1.2 Verify Required Files​

Critical Files Checklist:

# Docker build configuration
ls -l dockerfile.combined-fixed           # ✅ 261 lines - Multi-stage build
ls -l cloudbuild-combined.yaml            # ✅ 106 lines - Cloud Build config

# Startup and routing
ls -l start-combined.sh                   # ✅ 34 lines - Pod startup script
ls -l nginx-combined.conf                 # ✅ 107 lines - NGINX routing config

# Kubernetes manifests
ls -l k8s/theia-statefulset.yaml         # ✅ StatefulSet definition
ls -l k8s/theia-service.yaml             # ✅ Service definition
ls -l k8s/theia-ingress.yaml             # ✅ Ingress definition

# Frontend build (MUST be built first!)
ls -l dist/index.html                     # ✅ Frontend production build

# Pre-built binaries
ls -l archive/coditect-v4/codi2/prebuilt/codi2-prebuilt  # ✅ Monitoring binary

If dist/ doesn't exist:

npm install
npm run build
# Verify: ls -l dist/index.html

1.3 Review .gcloudignore​

Purpose: Excludes files from upload to reduce build time

Key Exclusions (as of Build #17):

# Rust build artifacts (builds during Docker)
backend/target/
src/file-monitor/target/

# Archive Rust intermediate files
archive/coditect-v4/codi2/target/debug/
archive/coditect-v4/codi2/target/*/build/
archive/coditect-v4/codi2/target/*/deps/

# Large documentation
docs/09-sessions/
docs/11-analysis/

Verification:

# File count should be ~8-10K (not 33K)
find . -type f | wc -l

Step 2: Run Cloud Build​

2.1 Authenticate​

gcloud auth login
gcloud config set project serene-voltage-464305-n2

2.2 Get GKE Credentials​

gcloud container clusters get-credentials codi-poc-e2-cluster \
  --zone=us-central1-a \
  --project=serene-voltage-464305-n2

2.3 Submit Build​

Standard Build Command:

gcloud builds submit \
  --config cloudbuild-combined.yaml \
  --project serene-voltage-464305-n2

With Enhanced Logging:

gcloud builds submit \
  --config cloudbuild-combined.yaml \
  --project serene-voltage-464305-n2 \
  2>&1 | tee /tmp/build-$(date +%Y%m%d-%H%M%S).log

2.4 Monitor Build Progress​

Console: https://console.cloud.google.com/cloud-build/builds?project=serene-voltage-464305-n2

CLI (in separate terminal):

# Get latest build ID
BUILD_ID=$(gcloud builds list --limit=1 --format="value(id)")

# Stream logs
gcloud builds log $BUILD_ID --stream

# Check status
watch -n 5 "gcloud builds list --limit=1 --format='table(id,status,createTime,duration)'"

2.5 Expected Build Timeline​

PhaseDurationDescription
Source Upload1-2 minCompressing & uploading ~8-10K files (2.1 GB)
Docker Build18-20 minAll 6 stages (frontend, theia, 3× Rust, runtime)
Image Push30-60 secPushing to Artifact Registry
kubectl Apply10-20 secApplying StatefulSet manifest
kubectl Set Image5-10 secUpdating StatefulSet image
Rollout Verification5-10 minSequential pod updates (3 pods)
TOTAL25-35 minFull deployment

2.6 Build Stages Explained​

Stage 1: frontend-builder (Node.js 20)

  • Installs npm dependencies
  • Runs npm run build (Vite production build)
  • Outputs: /app/frontend/dist/

Stage 2: theia-builder (Node.js 20)

  • Installs 68 @theia/* packages
  • Webpack build with 8GB heap
  • Custom Coditect AI branding
  • Icon themes (vs-seti + vscode-icons)
  • Outputs: /app/theia/lib/

Stage 3: v5-backend-builder (Rust/Debian)

  • Installs FoundationDB 7.1.38 client
  • Installs clang + libclang-dev
  • Builds Actix-web backend
  • Outputs: /usr/local/bin/coditect-v5-api

Stage 4: codi2-builder (PRE-BUILT BINARY)

  • Copies pre-built codi2 from archive/coditect-v4/codi2/prebuilt/
  • Workaround for 30 Rust compilation errors
  • Outputs: /usr/local/bin/codi2

Stage 5: monitor-builder (Rust/Debian)

  • Builds file-monitor from source
  • Outputs: /usr/local/bin/file-monitor

Stage 6: runtime (Node.js 20-slim)

  • Assembles all artifacts
  • Installs NGINX 1.22.1
  • Copies config files
  • Sets up entrypoint: start-combined.sh

Step 3: Verify Deployment​

3.1 Check Build Status​

Success Criteria:

gcloud builds list --limit=1 --format="value(status)"
# Expected output: SUCCESS

Get Build Details:

BUILD_ID=$(gcloud builds list --limit=1 --format="value(id)")

gcloud builds describe $BUILD_ID --format=yaml | grep -E "(status|images|timing)"

3.2 Check GKE Deployment​

StatefulSet Status:

kubectl get statefulset coditect-combined -n coditect-app

# Expected output:
# NAME                  READY   AGE
# coditect-combined     3/3     Xm

Pod Status:

kubectl get pods -n coditect-app -l app=coditect-combined

# Expected output: 3 pods with STATUS=Running, READY=1/1

Service Status:

kubectl get service coditect-combined-service -n coditect-app

# Expected output: ClusterIP with PORT 80

Ingress Status:

kubectl get ingress coditect-ingress -n coditect-app

# Expected output: ADDRESS=34.8.51.57

3.3 Test Production URLs​

Frontend (V5 React Wrapper):

curl -I https://coditect.ai/
# Expected: HTTP/2 200, server: nginx/1.22.1

theia IDE:

curl -I https://coditect.ai/theia
# Expected: HTTP/2 200, x-powered-by: Express

V5 API:

curl https://api.coditect.ai/
# Expected: {"success":false,"error":{"code":"NOT_FOUND",...}}

curl -I https://api.coditect.ai/api/v5/users
# Expected: HTTP/2 401 (authentication required - JWT working!)

3.4 Verify SSL Certificate​

echo | openssl s_client -connect coditect.ai:443 -servername coditect.ai 2>/dev/null | \
  openssl x509 -noout -subject -dates

# Expected:
# subject=CN=coditect.ai
# notBefore=Sep 29 12:58:50 2025 GMT
# notAfter=Dec 28 13:30:57 2025 GMT

3.5 Check DNS Resolution​

dig +short coditect.ai
dig +short api.coditect.ai

# Both should return: 34.8.51.57

3.6 Verify Image Tags​

kubectl get statefulset coditect-combined -n coditect-app \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Expected: us-central1-docker.pkg.dev/.../coditect-combined:<BUILD_ID>

Architecture Overview​

Production Infrastructure​

┌─────────────────────────────────────────────────────────┐
│  GCP Load Balancer (34.8.51.57)                          │
│  - coditect.ai                                            │
│  - api.coditect.ai                                        │
│  - Google-managed SSL                                     │
└──────────────┬──────────────────────────────────────────┘
               │
       ┌───────┴─────────────┐
       │                     │
┌──────▼────────────┐  ┌────▼───────────────────┐
│ coditect-combined │  │ coditect-api-v5        │
│ StatefulSet       │  │ Deployment             │
│ (3 pods)          │  │ (3 pods)               │
│                   │  │                        │
│ Each Pod:         │  │ Rust/Actix-web         │
│ - NGINX :80       │  │ JWT Authentication     │
│ - theia :3000     │  │ Port: 8080             │
│ - V5 Frontend     │  │                        │
│ - Binaries:       │  └────┬───────────────────┘
│   • codi2         │       │
│   • file-monitor  │       │
└───────────────────┘       │
               │            │
       ┌───────┴────────────┴───────┐
       │                            │
┌──────▼──────────────────────────────────────┐
│ FoundationDB StatefulSet                    │
│ - foundationdb-0/1/2 (3 pods)              │
│ - fdb-proxy (2 pods)                        │
│ - Internal LB: 10.128.0.10:4500            │
└─────────────────────────────────────────────┘

Pod Architecture​

What Runs in Each coditect-combined-* Pod:

ComponentStatusPortPurpose
NGINX✅ Auto-started80Routes traffic, serves frontend
theia IDE✅ Auto-started3000Eclipse theia 1.65 (Express)
V5 Frontend✅ Served by NGINX-React + Vite SPA
codi2⚠️ Available (not started)-Monitoring binary
file-monitor⚠️ Available (not started)-Audit logging binary

Separate Services:

  • V5 Backend API: 3 pods in separate deployment
  • FoundationDB: 3 pods + 2 proxies (StatefulSet)

Traffic Routing​

NGINX Configuration (nginx-combined.conf):

PathTargetType
/V5 FrontendStatic files
/v5V5 FrontendStatic files (alias)
/theiatheia IDE :3000Proxy
/theia/socket.io/WebSocketUpgrade (24hr timeout)
/api/v5External API serviceNot proxied (separate ingress)

Key Files Reference​

1. dockerfile.combined-fixed​

Location: dockerfile.combined-fixed Size: 261 lines Purpose: Multi-stage Docker build for all components

Archive Location: docs/99-archive/build-17-reference/dockerfile.combined-fixed

Key Sections:

  • Lines 1-32: Stage 1 (frontend-builder)
  • Lines 34-122: Stage 2 (theia-builder)
  • Lines 124-151: Stage 3 (v5-backend-builder)
  • Lines 153-171: Stage 4 (codi2-builder) - Pre-built binary
  • Lines 173-193: Stage 5 (monitor-builder)
  • Lines 195-261: Stage 6 (runtime) - Final assembly

2. cloudbuild-combined.yaml​

Location: cloudbuild-combined.yaml Size: 106 lines Purpose: Cloud Build orchestration

Archive Location: docs/99-archive/build-17-reference/cloudbuild-combined.yaml

Steps:

  1. build-image (lines 19-33): Docker build with BuildKit
  2. push-build-id (lines 36-41): Push image with build ID tag
  3. push-latest (lines 44-49): Push image with latest tag
  4. apply-statefulset (lines 52-61): Apply k8s manifest
  5. deploy-gke (lines 64-75): Update StatefulSet image
  6. verify-deployment (lines 78-89): Verify rollout status

Machine Type: E2_HIGHCPU_32 (32 CPUs, 64 GB RAM) Timeout: 7200s (2 hours)

3. start-combined.sh​

Location: start-combined.sh Size: 34 lines Purpose: Pod startup script (runs as PID 1)

Archive Location: docs/99-archive/build-17-reference/start-combined.sh

Startup Sequence:

  1. Start theia in background (port 3000)
  2. Wait for theia to be ready (60s timeout)
  3. Start NGINX in foreground

4. nginx-combined.conf​

Location: nginx-combined.conf Size: 107 lines Purpose: NGINX routing configuration

Archive Location: docs/99-archive/build-17-reference/nginx-combined.conf

Key Config:

  • WebSocket map (lines 14-17)
  • Socket.IO location (lines 39-58) - 24-hour timeout
  • theia location (lines 61-79)
  • V5 Frontend location (lines 82-99)

5. k8s/theia-statefulset.yaml​

Location: k8s/theia-statefulset.yaml Purpose: Kubernetes StatefulSet definition

Archive Location: docs/99-archive/build-17-reference/k8s-theia-statefulset.yaml

Key Config:

  • Replicas: 3
  • Persistent Volumes:
    • workspace-storage: 100 GB (user workspaces)
    • config-storage: 10 GB (user configs)
  • Resources:
    • CPU: 4 cores (request: 2, limit: 4)
    • Memory: 8 GB (request: 4 GB, limit: 8 GB)

6. .gcloudignore​

Location: .gcloudignore Size: 90 lines (Build #17 optimized) Purpose: Exclude files from Cloud Build upload

Archive Location: docs/99-archive/build-17-reference/.gcloudignore

Optimization:

  • Before: 33,073 files, 2.1 GB, 8+ min compression
  • After: ~8-10K files, compression <2 min

Troubleshooting​

Build Failures​

Issue: "Creating temporary archive" hangs for 8+ minutes​

Cause: .gcloudignore missing key exclusions

Solution:

# Add to .gcloudignore:
backend/target/
src/file-monitor/target/
archive/coditect-v4/codi2/target/debug/
docs/09-sessions/
docs/11-analysis/

Issue: "base64ct edition2024" error​

Cause: cargo.toml not pinning base64ct version

Solution:

# In backend/cargo.toml:
base64ct = "=1.6.0"

Issue: "npm install" fails with yarn conflict​

Cause: node:20-slim has pre-installed yarn

Solution:

# In dockerfile.combined-fixed:
RUN npm install --force

Issue: "statefulsets.apps not found"​

Cause: Trying to update StatefulSet before it exists

Solution:

# In cloudbuild-combined.yaml:
# Step 1: Apply manifest (creates resource)
- name: 'gcr.io/cloud-builders/kubectl'
  args: ['apply', '-f', 'k8s/theia-statefulset.yaml']

# Step 2: Update image (requires resource to exist)
- name: 'gcr.io/cloud-builders/kubectl'
  args: ['set', 'image', 'statefulset/coditect-combined', '...']
  waitFor: ['apply-statefulset']

Deployment Issues​

Issue: Pods stuck in "Pending"​

Check PVCs:

kubectl get pvc -n coditect-app

# If STATUS=Pending, check storage class:
kubectl describe pvc -n coditect-app

Issue: Pods stuck in "CrashLoopBackOff"​

Check logs:

kubectl logs coditect-combined-0 -n coditect-app

# Check startup script:
kubectl exec coditect-combined-0 -n coditect-app -- cat /app/start-combined.sh

Issue: 502 Bad Gateway on theia​

Check theia is running:

kubectl exec coditect-combined-0 -n coditect-app -- ps aux | grep node

# Check theia port:
kubectl exec coditect-combined-0 -n coditect-app -- curl -I http://localhost:3000

Production URL Issues​

Issue: coditect.ai returns 404​

Check Ingress:

kubectl describe ingress coditect-ingress -n coditect-app

# Check service:
kubectl get service coditect-combined-service -n coditect-app

# Test internal connectivity:
kubectl run test-pod --image=curlimages/curl --rm -it -- \
  curl -I http://coditect-combined-service.coditect-app.svc.cluster.local

Post-Deployment Tasks​

1. Start CODI2 and File Monitor​

Option A: Add to startup script (Recommended for production):

# Edit start-combined.sh:
# Add before "Start NGINX":

# Start CODI2 monitoring
/usr/local/bin/codi2 --config /etc/codi2/config.toml > /var/log/codi2.log 2>&1 &
echo "CODI2 started with PID $!"

# Start file monitor
/usr/local/bin/file-monitor --config /etc/monitor/config.toml > /var/log/monitor.log 2>&1 &
echo "File monitor started with PID $!"

# Then rebuild with Build #18

Option B: Start manually (Temporary):

# Install kubectl locally first if corrupted:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Start binaries:
kubectl exec coditect-combined-0 -n coditect-app -- /usr/local/bin/codi2 --version
kubectl exec coditect-combined-0 -n coditect-app -- /usr/local/bin/file-monitor --help

2. Test workspace Persistence​

# Create test file:
kubectl exec coditect-combined-0 -n coditect-app -- touch /workspace/test-persistence.txt

# Delete pod (StatefulSet will recreate):
kubectl delete pod coditect-combined-0 -n coditect-app

# Wait for recreation (~2 min):
watch -n 5 "kubectl get pods -n coditect-app -l app=coditect-combined"

# Check file still exists:
kubectl exec coditect-combined-0 -n coditect-app -- ls -l /workspace/test-persistence.txt

# Expected: File exists (proves persistence works)

3. Configure LM Studio Integration​

Decision Required: How should LM Studio work in production?

Option A: Each pod runs LM Studio (resource-heavy) Option B: Shared LM Studio service (recommended) Option C: User's local LM Studio at localhost:1234 (dev only)

Current Status: theia configured for localhost:1234 (not accessible from GKE pods)

4. Enable Monitoring​

# Install Prometheus + Grafana (optional):
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

# Expose Grafana:
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring
# Access: http://localhost:3000 (admin/prom-operator)

5. Set Up Alerting​

Recommended Alerts:

  • Pod restarts > 3 in 5 minutes
  • CPU usage > 80% for 5 minutes
  • Memory usage > 90% for 5 minutes
  • Disk usage > 85%
  • StatefulSet not all replicas ready

Quick Reference Commands​

Build & Deploy​

# Full deployment (standard)
gcloud builds submit --config cloudbuild-combined.yaml --project serene-voltage-464305-n2

# With logging
gcloud builds submit --config cloudbuild-combined.yaml --project serene-voltage-464305-n2 2>&1 | tee /tmp/build.log

# Check build status
gcloud builds list --limit=5

Verification​

# Check pods
kubectl get pods -n coditect-app -l app=coditect-combined

# Check services
kubectl get services -n coditect-app

# Check ingress
kubectl get ingress -n coditect-app

# Test URLs
curl -I https://coditect.ai/
curl -I https://coditect.ai/theia
curl -I https://api.coditect.ai/

Debugging​

# View pod logs
kubectl logs coditect-combined-0 -n coditect-app

# Exec into pod
kubectl exec -it coditect-combined-0 -n coditect-app -- bash

# Check startup script
kubectl exec coditect-combined-0 -n coditect-app -- cat /app/start-combined.sh

# Check NGINX config
kubectl exec coditect-combined-0 -n coditect-app -- cat /etc/nginx/conf.d/combined.conf

# Check running processes
kubectl exec coditect-combined-0 -n coditect-app -- ps aux

Rollback​

# Get previous image
kubectl rollout history statefulset/coditect-combined -n coditect-app

# Rollback to previous version
kubectl rollout undo statefulset/coditect-combined -n coditect-app

# Or set specific image:
kubectl set image statefulset/coditect-combined \
  combined=us-central1-docker.pkg.dev/serene-voltage-464305-n2/coditect/coditect-combined:<OLD_BUILD_ID> \
  -n coditect-app

Success Checklist​

Before considering deployment complete, verify:

  • Build status: SUCCESS
  • All 6 Docker stages completed
  • Images pushed with build ID + latest tags
  • StatefulSet shows READY 3/3
  • All 3 pods STATUS=Running, READY=1/1
  • Frontend: https://coditect.ai/ returns HTTP 200
  • theia: https://coditect.ai/theia returns HTTP 200
  • API: https://api.coditect.ai/ returns JSON (HTTP 200)
  • API auth: https://api.coditect.ai/api/v5/users returns HTTP 401 (JWT working)
  • DNS: Both coditect.ai and api.coditect.ai resolve to 34.8.51.57
  • SSL: Certificate valid and not expired
  • workspace persistence tested (file survives pod deletion)
  • CODI2 and file-monitor binaries available (even if not started)

Build History Reference​

Successful Builds​

Build #ID (first 8 chars)StatusDurationDateNotes
#17f1866abe✅ SUCCESS19m 48s2025-10-27✅ PRODUCTION (this guide)
#1622399b7b⚠️ PARTIAL8m 39s2025-10-27Docker ✅, kubectl ❌
#15(failed)❌ FAILED5m 55s2025-10-27npm yarn conflict

Failed Builds (Historical)​

Build #DurationPrimary IssueFix Applied
#105mbase64ct edition2024 + missing FDBFixed in cargo.toml
#115mMissing libclang in codi2-builderAdded to Dockerfile
#125mMissing libclang in v5-backend-builderAdded to Dockerfile
#135m30 Rust compilation errorsPre-built binary workaround
#145m 41snpm yarn conflictAdded --force flag

Additional Resources​

Documentation:

GCP Console Links:

GitHub Repository:

Last Updated: 2025-10-27 Tested On: Build #17 (f1866abe-dbc3-4e14-9d8b-60a0a8fbeed4) Maintained By: Coditect Engineering Team