Coditect V5 - GCP Cloud Build & Deployment Guide

Last Updated: 2025-10-07 GCP Project: serene-voltage-464305-n2 Primary Region: us-central1 GKE Cluster: codi-poc-e2-cluster (us-central1-a)

---

📋 Table of Contents

1. Infrastructure Overview
2. FoundationDB Configuration
3. Prerequisites
4. Local Development Setup
5. Cloud Build Configuration
6. GKE Deployment
7. Ingress & Load Balancer
8. Troubleshooting
9. Reference Commands

---

🏗️ Infrastructure Overview

Current Production Stack (V4 + V5)

Component	Type	Name	Internal IP	External IP	Port
Frontend (V4)	K8s Deployment	coditect-frontend	-	-	80
API (V4)	K8s Deployment	coditect-api-v2	-	-	80
API (V5)	K8s Deployment	coditect-api-v5	-	-	80
FoundationDB	StatefulSet (3 nodes)	foundationdb-{0,1,2}	10.56.0.7, 10.56.2.63, 10.56.3.57	-	4500
FDB Cluster Service	ClusterIP (Headless)	fdb-cluster	None	-	4500
FDB Proxy	LoadBalancer	fdb-proxy-service	10.128.0.10	34.118.234.162	4500
Ingress	Google LB	coditect-production-ingress	-	34.8.51.57	80, 443

DNS Configuration

• Domain: coditect.ai → 34.8.51.57
• API Subdomain: api.coditect.ai → 34.8.51.57
• SSL Certificate: Google-managed (mcrt-1e9fda11-9f24-455a-bd4b-074f776e3282)

Routing

https://coditect.ai/api     → coditect-api-v2 (V4 Rust API)
https://coditect.ai/api/v5  → coditect-api-v5 (V5 Rust API) [TO BE ADDED]
https://coditect.ai/ws      → coditect-api-v2 (V4 WebSocket)
https://coditect.ai/        → coditect-frontend (V4 React SPA)
https://api.coditect.ai/    → coditect-api-v2 (V4 Rust API)

---

🗄️ FoundationDB Configuration

Cluster Connection Details

⚠️ CRITICAL: Different connection strings for different deployment targets!

For GKE Deployments (V5 API in Kubernetes)

Use Kubernetes DNS (recommended):

coditect:production@foundationdb-0.fdb-cluster.coditect-app.svc.cluster.local:4500

Why: Kubernetes internal DNS resolution, lowest latency, automatic service discovery.

File Location: /workspace/PROJECTS/t2/backend/fdb.cluster

For Cloud Run Deployments (NOT RECOMMENDED)

Use FDB Proxy LoadBalancer IP:

coditect:production@10.128.0.10:4500

Why: Cloud Run cannot access Kubernetes internal DNS. Must go through proxy.

Issues: Higher latency, extra hop, requires VPC connector.

For External/Local Development

Use FDB Proxy External IP:

coditect:production@34.118.234.162:4500

Why: Only option from outside GCP network.

Security: ⚠️ Firewall rules must allow your IP!

Verifying FoundationDB

# 1. Get GKE credentials
gcloud container clusters get-credentials codi-poc-e2-cluster \
  --region=us-central1-a \
  --project=serene-voltage-464305-n2

# 2. Check FDB pods
kubectl get pods -n coditect-app | grep foundationdb
# Should show: foundationdb-0, foundationdb-1, foundationdb-2 (all Running)

# 3. Get cluster file from running pod
kubectl exec -n coditect-app foundationdb-0 -- cat /var/fdb/fdb.cluster
# Output: coditect:production@foundationdb-0.fdb-cluster.coditect-app.svc.cluster.local:4500

# 4. Check FDB cluster endpoints
kubectl get endpoints fdb-cluster -n coditect-app
# Output: 10.56.0.7:4500, 10.56.2.63:4500, 10.56.3.57:4500

# 5. Test FDB connection from inside cluster
kubectl run -it --rm fdb-test --image=foundationdb/foundationdb:7.1.27 \
  --restart=Never -n coditect-app -- fdbcli -C "/var/fdb/fdb.cluster"

---

✅ Prerequisites

1. GCP Authentication

# Authenticate with your GCP account
gcloud auth login

# Set the project
gcloud config set project serene-voltage-464305-n2

# Verify
gcloud config get-value project
# Output: serene-voltage-464305-n2

2. Required APIs (Already Enabled ✅)

# Verify APIs are enabled
gcloud services list --enabled | grep -E "(run|build|artifact|secret|vpcaccess|container)"

Expected output:

• artifactregistry.googleapis.com - Artifact Registry API
• cloudbuild.googleapis.com - Cloud Build API
• run.googleapis.com - Cloud Run Admin API
• secretmanager.googleapis.com - Secret Manager API
• vpcaccess.googleapis.com - Serverless VPC Access API
• container.googleapis.com - Kubernetes Engine API

3. Artifact Registry Repository

Repository: coditect (already exists ✅) Location: us-central1 Format: Docker Size: ~19.8 GB

# Verify repository exists
gcloud artifacts repositories describe coditect \
  --location=us-central1 \
  --project=serene-voltage-464305-n2

4. Secrets (Already Configured ✅)

# Check Google Secret Manager secrets
gcloud secrets list --project=serene-voltage-464305-n2
# Should show:
# - jwt-secret
# - fdb-cluster-file

# Check Kubernetes secrets
kubectl get secrets -n coditect-app
# Should show:
# - jwt-secret-k8s

If jwt-secret-k8s doesn't exist, create it:

kubectl create secret generic jwt-secret-k8s \
  --from-literal=secret=$(openssl rand -base64 32) \
  -n coditect-app

---

💻 Local Development Setup

Install Required Tools

1. Rust (Latest Stable - 1.90+)

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

# Add to PATH
export PATH=$HOME/.cargo/bin:$PATH
source $HOME/.cargo/env

# Verify
rustc --version  # Should show 1.90.0 or later
cargo --version

2. FoundationDB Client (Required for Compilation)

# Download and install FDB client
curl -LO https://github.com/apple/foundationdb/releases/download/7.1.27/foundationdb-clients_7.1.27-1_amd64.deb
sudo dpkg -i foundationdb-clients_7.1.27-1_amd64.deb
rm foundationdb-clients_7.1.27-1_amd64.deb

# Verify installation
ls /usr/include/foundationdb/fdb.options
# Should exist (required by foundationdb-gen crate)

3. Clang (Required for bindgen)

# Install clang and libclang
sudo apt-get update
sudo apt-get install -y clang libclang-dev

# Verify
clang --version

4. Google Cloud SDK

# Install gcloud SDK
curl https://sdk.cloud.google.com | bash -s -- --disable-prompts --install-dir=$HOME

# Add to PATH
export PATH="$HOME/google-cloud-sdk/bin:$PATH"

# Initialize
gcloud init

5. kubectl

# Install kubectl
gcloud components install kubectl gke-gcloud-auth-plugin --quiet

# Verify
kubectl version --client

Local Build & Test

# Navigate to backend directory
cd /workspace/PROJECTS/t2/backend

# Check dependencies
cargo check

# Build (debug)
cargo build

# Build (release)
cargo build --release

# Run locally (requires FDB connection)
export JWT_SECRET=$(openssl rand -base64 32)
export FDB_CLUSTER_FILE=/workspace/PROJECTS/t2/backend/fdb.cluster
export HOST=0.0.0.0
export PORT=8080
export RUST_LOG=info

cargo run

---

☁️ Cloud Build Configuration

Build Configuration Files

Primary Build File: cloudbuild-simple.yaml

# Simple Cloud Build - just build and push the image
steps:
  # Build Docker image
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - 'build'
      - '-t'
      - 'us-central1-docker.pkg.dev/$PROJECT_ID/coditect/coditect-v5-api:latest'
      - '.'

  # Push to Artifact Registry
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - 'push'
      - 'us-central1-docker.pkg.dev/$PROJECT_ID/coditect/coditect-v5-api:latest'

# Build timeout (Rust compilation takes time)
timeout: '3600s'

# Use high-CPU machine for faster builds
options:
  machineType: 'N1_HIGHCPU_8'
  diskSizeGb: 100
  logging: CLOUD_LOGGING_ONLY

images:
  - 'us-central1-docker.pkg.dev/$PROJECT_ID/coditect/coditect-v5-api:latest'

Dockerfile Configuration

⚠️ CRITICAL: FoundationDB client MUST be installed in builder stage!

# Build stage
FROM rust:1.90 as builder

# Install build dependencies and FoundationDB client
RUN apt-get update && \
    apt-get install -y curl clang libclang-dev && \
    curl -LO https://github.com/apple/foundationdb/releases/download/7.1.27/foundationdb-clients_7.1.27-1_amd64.deb && \
    dpkg -i foundationdb-clients_7.1.27-1_amd64.deb && \
    rm foundationdb-clients_7.1.27-1_amd64.deb && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Copy Cargo files
COPY cargo.toml ./

# Create dummy main to cache dependencies
RUN mkdir src && \
    echo "fn main() {}" > src/main.rs && \
    cargo build --release && \
    rm -rf src

# Copy source code
COPY src ./src

# Build the application
RUN cargo build --release

# Runtime stage
FROM debian:bookworm-slim

# Install FoundationDB client (runtime)
RUN apt-get update && \
    apt-get install -y curl ca-certificates && \
    curl -LO https://github.com/apple/foundationdb/releases/download/7.1.27/foundationdb-clients_7.1.27-1_amd64.deb && \
    dpkg -i foundationdb-clients_7.1.27-1_amd64.deb && \
    rm foundationdb-clients_7.1.27-1_amd64.deb && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Copy binary from builder
COPY --from=builder /app/target/release/api-server /app/api-server

# Copy FDB cluster file
COPY fdb.cluster /app/fdb.cluster

# Set environment
ENV HOST=0.0.0.0
ENV PORT=8080
ENV RUST_LOG=info
ENV FDB_CLUSTER_FILE=/app/fdb.cluster

EXPOSE 8080

CMD ["/app/api-server"]

Building with Cloud Build

# Navigate to backend directory
cd /workspace/PROJECTS/t2/backend

# Ensure fdb.cluster has the correct connection string
cat fdb.cluster
# For GKE: coditect:production@foundationdb-0.fdb-cluster.coditect-app.svc.cluster.local:4500

# Submit build to Cloud Build
gcloud builds submit \
  --config cloudbuild-simple.yaml \
  --project=serene-voltage-464305-n2

# Monitor build (get build ID from output)
gcloud builds log <BUILD_ID> --project=serene-voltage-464305-n2

# Or watch in console
# https://console.cloud.google.com/cloud-build/builds?project=serene-voltage-464305-n2

Build Troubleshooting

Issue: "couldn't read /usr/include/foundationdb/fdb.options"

Cause: FoundationDB client not installed in builder stage Fix: Ensure Dockerfile has FDB client installation BEFORE cargo build

Issue: Build times out

Cause: Default timeout too short for Rust compilation Fix: Set timeout: '3600s' in cloudbuild.yaml

Issue: "Permission denied on secret"

Cause: Service account lacks Secret Manager access Fix: Grant roles/secretmanager.secretAccessor to Cloud Build service account

---

🚀 GKE Deployment

Kubernetes Manifests

Deployment: k8s-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coditect-api-v5
  namespace: coditect-app
  labels:
    app: coditect-api-v5
    version: v5
spec:
  replicas: 3
  selector:
    matchLabels:
      app: coditect-api-v5
  template:
    metadata:
      labels:
        app: coditect-api-v5
        version: v5
    spec:
      containers:
      - name: api
        image: us-central1-docker.pkg.dev/serene-voltage-464305-n2/coditect/coditect-v5-api:latest
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: HOST
          value: "0.0.0.0"
        - name: PORT
          value: "8080"
        - name: RUST_LOG
          value: "info"
        - name: FDB_CLUSTER_FILE
          value: "/app/fdb.cluster"
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: jwt-secret-k8s
              key: secret
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: coditect-api-v5-service
  namespace: coditect-app
  labels:
    app: coditect-api-v5
spec:
  type: ClusterIP
  selector:
    app: coditect-api-v5
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http

Deployment Steps

# 1. Get GKE credentials
gcloud container clusters get-credentials codi-poc-e2-cluster \
  --region=us-central1-a \
  --project=serene-voltage-464305-n2

# 2. Verify namespace exists
kubectl get namespace coditect-app
# Should exist

# 3. Ensure JWT secret exists in Kubernetes
kubectl get secret jwt-secret-k8s -n coditect-app || \
kubectl create secret generic jwt-secret-k8s \
  --from-literal=secret=$(openssl rand -base64 32) \
  -n coditect-app

# 4. Apply Kubernetes manifests
cd /workspace/PROJECTS/t2/backend
kubectl apply -f k8s-deployment.yaml

# 5. Check deployment status
kubectl get deployment coditect-api-v5 -n coditect-app

# 6. Check pods
kubectl get pods -n coditect-app -l app=coditect-api-v5

# 7. Check logs
kubectl logs -n coditect-app -l app=coditect-api-v5 --tail=50

# 8. Test health endpoint internally
kubectl run -it --rm test-curl --image=curlimages/curl --restart=Never -n coditect-app -- \
  curl http://coditect-api-v5-service/health

Rolling Update

# After building new image, update deployment
kubectl rollout restart deployment coditect-api-v5 -n coditect-app

# Watch rollout status
kubectl rollout status deployment coditect-api-v5 -n coditect-app

# Check rollout history
kubectl rollout history deployment coditect-api-v5 -n coditect-app

# Rollback if needed
kubectl rollout undo deployment coditect-api-v5 -n coditect-app

---

🌐 Ingress & Load Balancer

Current Ingress Configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: coditect-production-ingress
  namespace: coditect-app
  annotations:
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: coditect-ai-ip
    networking.gke.io/managed-certificates: coditect-ai-ssl
spec:
  rules:
  - host: coditect.ai
    http:
      paths:
      - path: /api/v5
        pathType: Prefix
        backend:
          service:
            name: coditect-api-v5-service
            port:
              number: 80
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: coditect-api-v2
            port:
              number: 80
      # ... other paths

Adding V5 API to Ingress

⚠️ IMPORTANT: Path order matters! More specific paths MUST come first.

# 1. Backup current ingress
kubectl get ingress coditect-production-ingress -n coditect-app -o yaml > /tmp/ingress-backup.yaml

# 2. Edit ingress
kubectl edit ingress coditect-production-ingress -n coditect-app

# 3. Add V5 API path BEFORE /api path:
#   - path: /api/v5
#     pathType: Prefix
#     backend:
#       service:
#         name: coditect-api-v5-service
#         port:
#           number: 80

# 4. Verify changes
kubectl describe ingress coditect-production-ingress -n coditect-app

# 5. Test V5 API externally
curl https://coditect.ai/api/v5/health

Load Balancer Details

• Type: Google Cloud Load Balancer (HTTPS)
• IP: 34.8.51.57 (reserved as coditect-ai-ip)
• SSL: Google-managed certificate (auto-renewal)
• Backend: GKE cluster services via NEG (Network Endpoint Groups)

---

🔧 Troubleshooting

Pod Won't Start

# Check pod status
kubectl get pods -n coditect-app -l app=coditect-api-v5

# Describe pod for events
kubectl describe pod -n coditect-app -l app=coditect-api-v5

# Check logs
kubectl logs -n coditect-app -l app=coditect-api-v5 --tail=100

# Check previous logs (if pod restarted)
kubectl logs -n coditect-app -l app=coditect-api-v5 --previous

FoundationDB Connection Issues

# Test FDB connectivity from a debug pod
kubectl run -it --rm fdb-debug --image=foundationdb/foundationdb:7.1.27 \
  --restart=Never -n coditect-app -- bash

# Inside debug pod:
cat > /tmp/fdb.cluster << 'EOF'
coditect:production@foundationdb-0.fdb-cluster.coditect-app.svc.cluster.local:4500
EOF

fdbcli -C /tmp/fdb.cluster
# Should connect and show cluster status

Health Check Failing

# Check if service is listening
kubectl exec -n coditect-app <POD_NAME> -- netstat -tlnp

# Test health endpoint from inside pod
kubectl exec -n coditect-app <POD_NAME> -- curl http://localhost:8080/health

# Check readiness/liveness probe configuration
kubectl get pod <POD_NAME> -n coditect-app -o yaml | grep -A 10 Probe

Image Pull Errors

# Verify image exists in Artifact Registry
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/serene-voltage-464305-n2/coditect/coditect-v5-api

# Check if GKE has permission to pull from Artifact Registry
gcloud projects get-iam-policy serene-voltage-464305-n2 \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/artifactregistry.reader"

JWT Secret Issues

# Verify secret exists
kubectl get secret jwt-secret-k8s -n coditect-app

# Check secret data
kubectl get secret jwt-secret-k8s -n coditect-app -o jsonpath='{.data.secret}' | base64 -d
# Should output a base64-encoded secret

# Recreate if corrupted
kubectl delete secret jwt-secret-k8s -n coditect-app
kubectl create secret generic jwt-secret-k8s \
  --from-literal=secret=$(openssl rand -base64 32) \
  -n coditect-app

---

📚 Reference Commands

Quick Deploy Workflow

# Full deployment from scratch
cd /workspace/PROJECTS/t2/backend

# 1. Build image
gcloud builds submit --config cloudbuild-simple.yaml --project=serene-voltage-464305-n2

# 2. Deploy to GKE
kubectl apply -f k8s-deployment.yaml

# 3. Check status
kubectl get pods -n coditect-app -l app=coditect-api-v5

# 4. Test
curl https://coditect.ai/api/v5/health

Common kubectl Commands

# Get all resources in namespace
kubectl get all -n coditect-app

# Describe deployment
kubectl describe deployment coditect-api-v5 -n coditect-app

# Scale deployment
kubectl scale deployment coditect-api-v5 --replicas=5 -n coditect-app

# Delete deployment
kubectl delete deployment coditect-api-v5 -n coditect-app

# Port forward for local testing
kubectl port-forward -n coditect-app svc/coditect-api-v5-service 8080:80

# Execute command in pod
kubectl exec -it -n coditect-app <POD_NAME> -- /bin/bash

gcloud Commands

# List all builds
gcloud builds list --project=serene-voltage-464305-n2 --limit=10

# Get build details
gcloud builds describe <BUILD_ID> --project=serene-voltage-464305-n2

# List images
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/serene-voltage-464305-n2/coditect

# Delete old images
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/serene-voltage-464305-n2/coditect/coditect-v5-api:<TAG>

---

🔗 Additional Resources

• GCP Project Console: https://console.cloud.google.com/?project=serene-voltage-464305-n2
• Cloud Build: https://console.cloud.google.com/cloud-build/builds?project=serene-voltage-464305-n2
• GKE Cluster: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/codi-poc-e2-cluster?project=serene-voltage-464305-n2
• Artifact Registry: https://console.cloud.google.com/artifacts/docker/serene-voltage-464305-n2/us-central1/coditect?project=serene-voltage-464305-n2
• Secret Manager: https://console.cloud.google.com/security/secret-manager?project=serene-voltage-464305-n2

---

End of Deployment Guide

Last Verified: 2025-10-07 Maintainer: Coditect Engineering Team