Skip to main content

CODITECT Backend - Staging Deployment Guide

Status: Ready for GKE Staging Deployment Date: November 30, 2025 Docker Image: coditect-cloud-backend:test-v1.0.0 (✅ Built Successfully) Kubernetes Manifests: ✅ Created

Quick Deployment Summary​

What's Ready:

  • ✅ Docker image built with Python 3.12.12
  • ✅ GKE staging deployment manifests created
  • ✅ Service definitions configured
  • ✅ Secrets template prepared
  • ✅ Health check endpoints configured

What's Next:

  1. Push Docker image to Google Container Registry (GCR)
  2. Create Kubernetes secrets from GCP Secret Manager
  3. Deploy to GKE staging cluster
  4. Run smoke tests
  5. Fix P1 test failures

Step-by-Step Deployment​

Prerequisites​

# Verify you're authenticated to GCP
gcloud auth list

# Set project
gcloud config set project coditect-prod-563272

# Configure Docker for GCR
gcloud auth configure-docker gcr.io

Step 1: Tag and Push Docker Image to GCR​

# Navigate to backend directory
cd /Users/halcasteel/PROJECTS/coditect-rollout-master/submodules/cloud/coditect-cloud-backend

# Tag the image for GCR
docker tag coditect-cloud-backend:test-v1.0.0 \
  gcr.io/coditect-prod-563272/coditect-cloud-backend:v1.0.0-staging

# Push to GCR
docker push gcr.io/coditect-prod-563272/coditect-cloud-backend:v1.0.0-staging

Expected Output:

The push refers to repository [gcr.io/coditect-prod-563272/coditect-cloud-backend]
v1.0.0-staging: digest: sha256:... size: 3456

Estimated Time: 3-5 minutes (depending on upload speed)

Step 2: Create GKE Cluster (if not exists)​

# Check if cluster exists
gcloud container clusters list --filter="name:coditect-staging-cluster"

# If cluster doesn't exist, create it
gcloud container clusters create coditect-staging-cluster \
  --zone us-central1-a \
  --num-nodes 3 \
  --machine-type n1-standard-2 \
  --enable-autoscaling \
  --min-nodes 2 \
  --max-nodes 5 \
  --enable-autorepair \
  --enable-autoupgrade \
  --workload-pool=coditect-prod-563272.svc.id.goog

# Get cluster credentials
gcloud container clusters get-credentials coditect-staging-cluster \
  --zone us-central1-a

Estimated Time: 5-10 minutes (cluster creation)

Step 3: Create Namespace​

# Apply namespace manifest
kubectl apply -f deployment/kubernetes/staging/namespace.yaml

# Verify namespace created
kubectl get namespace coditect-staging

Expected Output:

NAME                STATUS   AGE
coditect-staging    Active   5s

Step 4: Create Secrets from GCP Secret Manager​

Option A: Create secrets in Secret Manager first (Recommended)

# Generate secure random secrets
export DJANGO_SECRET=$(openssl rand -base64 32)
export DB_PASSWORD=$(openssl rand -base64 32)

# Create secrets in GCP Secret Manager
gcloud secrets create django-secret-key \
  --data-file=<(echo -n "$DJANGO_SECRET") \
  --replication-policy=automatic

gcloud secrets create db-password-staging \
  --data-file=<(echo -n "$DB_PASSWORD") \
  --replication-policy=automatic

# Grant service account access
gcloud secrets add-iam-policy-binding django-secret-key \
  --member="serviceAccount:coditect-cloud-backend@coditect-prod-563272.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

gcloud secrets add-iam-policy-binding db-password-staging \
  --member="serviceAccount:coditect-cloud-backend@coditect-prod-563272.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Option B: Create Kubernetes secret directly

# Retrieve secrets from Secret Manager
export DJANGO_SECRET=$(gcloud secrets versions access latest --secret=django-secret-key)
export DB_PASSWORD=$(gcloud secrets versions access latest --secret=db-password-staging)

# Create Kubernetes secret
kubectl create secret generic backend-secrets \
  --namespace=coditect-staging \
  --from-literal=django-secret-key="${DJANGO_SECRET}" \
  --from-literal=db-name="coditect_licenses_staging" \
  --from-literal=db-user="license_api_staging" \
  --from-literal=db-password="${DB_PASSWORD}" \
  --from-literal=db-host="10.0.0.5" \
  --from-literal=redis-host="10.0.0.3"

# Verify secret created
kubectl get secret backend-secrets -n coditect-staging
kubectl describe secret backend-secrets -n coditect-staging

Expected Output:

NAME              TYPE     DATA   AGE
backend-secrets   Opaque   6      5s

Step 5: Deploy Application to Staging​

# Apply deployment manifest
kubectl apply -f deployment/kubernetes/staging/backend-deployment.yaml

# Apply service manifest
kubectl apply -f deployment/kubernetes/staging/backend-service.yaml

# Watch deployment progress
kubectl rollout status deployment/coditect-backend -n coditect-staging --watch

# Check pods are running
kubectl get pods -n coditect-staging -l app=coditect-backend

# Check services
kubectl get services -n coditect-staging

Expected Output:

NAME               READY   STATUS    RESTARTS   AGE
coditect-backend   2/2     Running   0          30s

NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
coditect-backend             LoadBalancer   10.96.xxx.xxx   34.xxx.xxx.xxx   80:xxxxx/TCP     1m
coditect-backend-internal    ClusterIP      10.96.xxx.xxx   <none>           8000/TCP         1m

Estimated Time: 3-5 minutes

Step 6: Verify Deployment​

Check Application Logs:

# View application logs
kubectl logs -f deployment/coditect-backend -n coditect-staging --all-containers=true

# Check for errors
kubectl logs deployment/coditect-backend -n coditect-staging | grep -i error

Test Health Endpoints:

# Get external IP
export STAGING_IP=$(kubectl get service coditect-backend -n coditect-staging \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo "Staging IP: $STAGING_IP"

# Test health endpoint
curl http://${STAGING_IP}/api/v1/health/live

# Expected: {"status": "healthy", "timestamp": "..."}

Port Forward for Local Testing (Alternative):

# Port forward to local machine
kubectl port-forward -n coditect-staging deployment/coditect-backend 8000:8000

# In another terminal, test endpoints
curl http://localhost:8000/api/v1/health/live
curl http://localhost:8000/api/v1/docs/  # Swagger UI

Step 7: Run Smoke Tests​

Create smoke test script:

# Save as tests/smoke/run_staging_smoke_tests.sh
#!/bin/bash

STAGING_URL="http://${STAGING_IP}"

echo "🧪 Running Staging Smoke Tests..."
echo "Target: $STAGING_URL"
echo ""

# Test 1: Health endpoint
echo "Test 1: Health endpoint"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" ${STAGING_URL}/api/v1/health/live)
if [ $HTTP_CODE -eq 200 ]; then
  echo "✅ PASS: Health endpoint returned 200"
else
  echo "❌ FAIL: Health endpoint returned $HTTP_CODE (expected 200)"
  exit 1
fi

# Test 2: OpenAPI schema
echo "Test 2: OpenAPI schema"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" ${STAGING_URL}/api/v1/schema/)
if [ $HTTP_CODE -eq 200 ]; then
  echo "✅ PASS: OpenAPI schema accessible"
else
  echo "❌ FAIL: OpenAPI schema returned $HTTP_CODE"
  exit 1
fi

# Test 3: Authentication requirement
echo "Test 3: Authentication requirement"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" ${STAGING_URL}/api/v1/licenses/)
if [ $HTTP_CODE -eq 401 ]; then
  echo "✅ PASS: Authentication required (401 Unauthorized)"
else
  echo "❌ FAIL: Expected 401, got $HTTP_CODE"
  exit 1
fi

echo ""
echo "🎉 All smoke tests passed!"
# Run smoke tests
chmod +x tests/smoke/run_staging_smoke_tests.sh
./tests/smoke/run_staging_smoke_tests.sh

Expected Output:

🧪 Running Staging Smoke Tests...
Target: http://34.xxx.xxx.xxx

Test 1: Health endpoint
✅ PASS: Health endpoint returned 200

Test 2: OpenAPI schema
✅ PASS: OpenAPI schema accessible

Test 3: Authentication requirement
✅ PASS: Authentication required (401 Unauthorized)

🎉 All smoke tests passed!

Troubleshooting​

Issue 1: Pods Not Starting​

Check pod events:

kubectl describe pod -n coditect-staging -l app=coditect-backend

Common causes:

  • Missing secrets → Verify kubectl get secret backend-secrets -n coditect-staging
  • Image pull errors → Verify GCR permissions
  • Health check failures → Check application logs

Issue 2: Health Check Failing​

Check application logs:

kubectl logs -f deployment/coditect-backend -n coditect-staging | grep -E "(ERROR|WARN|health)"

Common causes:

  • Database connection failed → Verify DB_HOST is correct
  • Redis connection failed → Verify REDIS_HOST is correct
  • Migrations not applied → Run migrations manually

Apply migrations:

# Exec into pod
kubectl exec -it deployment/coditect-backend -n coditect-staging -- bash

# Run migrations
python manage.py migrate

# Exit pod
exit

Issue 3: Service External IP Pending​

Check service status:

kubectl get service coditect-backend -n coditect-staging

If EXTERNAL-IP shows <pending> for > 5 minutes:

# Check events
kubectl describe service coditect-backend -n coditect-staging

# Verify GKE has available external IPs
gcloud compute addresses list

Post-Deployment Checklist​

  • Docker image pushed to GCR
  • GKE staging cluster operational
  • Namespace created
  • Secrets configured
  • Deployment successful (2/2 pods running)
  • Service has external IP
  • Health endpoint returns 200
  • OpenAPI schema accessible
  • Authentication enforced (401 on protected endpoints)
  • Application logs clean (no errors)
  • Smoke tests passing

Next Steps After Deployment​

Immediate (This Week)​

  1. Fix P1 Test Failures (8-12 hours)

    • Focus on 30 highest-priority failing tests
    • Fix FakeRedis mock configuration
    • Update Firebase mock structures
    • Relax timestamp assertions

  2. Increase Test Coverage (4-6 hours)

    • Add negative test cases
    • Test middleware edge cases
    • Test Celery task failures
    • Target: 75%+ coverage

  3. Integration Testing (4 hours)

    • Test end-to-end license acquisition flow
    • Verify multi-tenant isolation
    • Test background task execution
    • Verify Cloud KMS signing works

Next Week​

  1. Production Monitoring (6 hours)

    • Deploy Prometheus + Grafana
    • Configure dashboards (API, Redis, database)
    • Set up alerting rules

  2. Load Testing (8 hours)

    • Test with 100 concurrent users
    • Test with 1000 concurrent users
    • Verify <100ms p99 latency
    • Validate Redis connection pooling

  3. Production Deployment (End of Week 2)

    • Deploy to production GKE cluster
    • Run final smoke tests
    • Monitor for 24 hours
    • PRODUCTION READY!

Rollback Procedure​

If deployment fails or critical issues found:

# Rollback to previous version
kubectl rollout undo deployment/coditect-backend -n coditect-staging

# Check rollback status
kubectl rollout status deployment/coditect-backend -n coditect-staging

# Verify pods are running
kubectl get pods -n coditect-staging -l app=coditect-backend

Monitoring Commands​

Watch pod status:

watch kubectl get pods -n coditect-staging

Stream logs:

kubectl logs -f deployment/coditect-backend -n coditect-staging --all-containers=true

Check resource usage:

kubectl top pods -n coditect-staging

Check deployment history:

kubectl rollout history deployment/coditect-backend -n coditect-staging

Files Created​

  1. Docker:

    • Dockerfile - Updated to Python 3.12.12
    • Image: coditect-cloud-backend:test-v1.0.0 (737MB)

  2. Kubernetes Manifests:

    • deployment/kubernetes/staging/namespace.yaml
    • deployment/kubernetes/staging/backend-deployment.yaml
    • deployment/kubernetes/staging/backend-service.yaml
    • deployment/kubernetes/staging/backend-secrets.yaml.template

  3. Documentation:

    • next-steps-staging-deployment.md - 2-week deployment plan
    • deployment-guide-staging.md - This file
    • phase-2-completion-report.md - Phase 2 final report

Support & Resources​

GKE Documentation:

Troubleshooting:

  • View logs: kubectl logs -f deployment/coditect-backend -n coditect-staging
  • Exec into pod: kubectl exec -it deployment/coditect-backend -n coditect-staging -- bash
  • Port forward: kubectl port-forward -n coditect-staging deployment/coditect-backend 8000:8000

Contact:

  • Hal Casteel, Founder/CEO/CTO
  • Backend Team

Document Version: 1.0 Last Updated: November 30, 2025 Status: Ready for Deployment