Skip to main content

Testing Guide - Automated Deployment Validation

This guide explains the automated testing infrastructure for catching configuration issues BEFORE deployment.

🎯 Problem We're Solving​

Previous deployments required 8+ builds to discover configuration issues:

  1. Wrong API paths (/api vs /api/v5)
  2. Incorrect build scripts (build vs prototype:build)
  3. Docker layer caching issues
  4. npm package-lock.json sync issues
  5. Environment variable misconfigurations

Solution: Automated testing catches ALL of these issues in < 2 minutes, before wasting 10-20 minutes on Cloud Build.

🛠️ Testing Tools​

1. scripts/test-deployment.sh - Complete Pre-Deployment Validation​

Tests EVERYTHING before deployment:

  • ✅ Environment configuration (.env.production)
  • ✅ Source code API paths
  • ✅ package.json scripts
  • ✅ Build process (npm install + npm run prototype:build)
  • ✅ Built files contain correct API paths
  • ✅ Docker configuration
  • ✅ NGINX configuration
  • ✅ Cloud Build configuration
  • ✅ Kubernetes configuration
  • ✅ Live endpoint testing (with --live flag)

Usage:

# Run all tests (config + build)
./scripts/test-deployment.sh

# Run all tests + live endpoint checks
./scripts/test-deployment.sh --live

Exit codes:

  • 0 = All tests passed → SAFE TO DEPLOY
  • 1 = Some tests failed → DO NOT DEPLOY

2. scripts/test-endpoints.sh - Live Endpoint Testing​

Tests deployed endpoints for correct behavior:

  • ✅ Health endpoint returns 200
  • ✅ Frontend loads correctly
  • ✅ JavaScript contains /api/v5 paths
  • ✅ API v5 endpoints exist (return 401 without auth)
  • ✅ Old API paths (/api/sessions) return 404

Usage:

# Test production
./scripts/test-endpoints.sh https://coditect.ai

# Test local
./scripts/test-endpoints.sh http://localhost

3. Makefile - Convenient Test Commands​

Provides easy-to-remember commands:

# Run all pre-deployment tests
make test

# Test configuration only (fast)
make test-config

# Test build process
make test-build

# Test live endpoints
make test-endpoints

# Run all tests including live endpoints
make test-all

# Deploy (runs tests first)
make deploy

# Clean build artifacts
make clean

🚀 Automated CI/CD Integration​

Cloud Build Validation (Step 0)​

cloudbuild-combined.yaml now includes automated validation:

Step 0: Pre-Build Validation

  • Checks .env.production has VITE_API_URL=/api/v5
  • Checks package.json has prototype:build script
  • Checks source code has /api/v5 paths
  • Fails fast (< 1 minute) if config is wrong

Step 1: Build + Verification

  • Runs npm run prototype:build
  • Verifies built files contain /api/v5 paths
  • Fails fast (< 5 minutes) if build produces wrong output

Benefits:

  • Catches issues in 1-5 minutes vs 10-20 minutes
  • Prevents bad deployments from reaching GKE
  • Saves $$ on Cloud Build costs

📋 Test Checklist for New Deployments​

Before every deployment, run:

# 1. Test configuration (< 5 seconds)
make test-config

# 2. Test full build (< 2 minutes)
make test

# 3. Deploy (if tests pass)
make deploy

🐛 Common Test Failures​

FAIL: .env.production VITE_API_URL is wrong​

Issue: .env.production has VITE_API_URL=/api instead of /api/v5

Fix:

# Edit .env.production
sed -i 's|VITE_API_URL=/api|VITE_API_URL=/api/v5|' .env.production

FAIL: Built files do NOT contain /api/v5 paths​

Issue: Environment variable not being picked up by Vite

Fix:

  1. Clean build cache: rm -rf dist/ node_modules/.vite
  2. Rebuild: npm run prototype:build
  3. Verify: grep -r "/api/v5" dist/

FAIL: npm run prototype:build failed​

Issue: Missing dependencies or TypeScript errors

Fix:

# Clean install
rm -rf node_modules package-lock.json
npm install
npm run type-check  # Fix any TypeScript errors
npm run prototype:build

FAIL: JavaScript does NOT contain /api/v5 paths​

Issue: Source code still has old /api/sessions paths

Fix:

# Find old paths
grep -r "/api/sessions" src/ | grep -v "/api/v5"

# Update to /api/v5/sessions

🔬 Test Coverage​

Configuration Tests​

  • .env.production has correct VITE_API_URL
  • .env.production has correct VITE_THEIA_URL
  • package.json has prototype:build script
  • Source code has /api/v5 fallback URLs
  • No old /api/sessions paths in source

Build Tests​

  • npm install succeeds
  • npm run prototype:build succeeds
  • dist/ directory is created
  • dist/index.html exists
  • JavaScript file is generated
  • JavaScript contains /api/v5 paths
  • JavaScript does NOT contain old /api/sessions paths

Docker Tests​

  • dockerfile.local-test exists
  • Dockerfile copies dist/ to /app/v5-frontend
  • nginx-combined.conf has /api/v5/ proxy

Cloud Build Tests​

  • cloudbuild-combined.yaml uses npm run prototype:build
  • Cloud Build uses npm install (not npm ci)

Kubernetes Tests​

  • k8s-combined-deployment.yaml exists

Live Endpoint Tests (with --live flag)​

  • Health endpoint returns 200
  • Frontend is accessible
  • API v5 endpoints return 401 (auth required)
  • Old API paths return 404

🎓 Best Practices​

1. Test Before Every Commit​

# Add to your workflow
make test-config
git add .
git commit -m "Your message"

2. Test After Every Checkout​

# After pulling changes
git pull
make test

3. Test After Configuration Changes​

# After editing .env.production, package.json, etc.
make test-config

4. Test After Source Code Changes​

# After editing API paths, adding features, etc.
make test

5. Test Endpoints After Deployment​

# After successful deployment
make test-endpoints

📊 Expected Test Times​

TestTimeWhen to Run
make test-config< 5 secondsBefore every commit
make test-build~ 2 minutesBefore deployment
make test~ 2 minutesBefore deployment
make test-endpoints~ 10 secondsAfter deployment
Cloud Build Step 0< 1 minuteAutomatic (CI/CD)
Cloud Build Step 1~ 4 minutesAutomatic (CI/CD)

🚨 When Tests Should Fail​

Tests SHOULD fail if:

  • ❌ Configuration is incorrect (wrong API URLs)
  • ❌ Build scripts are wrong (using build instead of prototype:build)
  • ❌ Source code has old API paths
  • ❌ npm packages are out of sync
  • ❌ TypeScript has compilation errors
  • ❌ Built files are missing or incorrect

✅ When Tests Should Pass​

Tests SHOULD pass if:

  • ✅ All configuration files are correct
  • ✅ Build completes successfully
  • ✅ Built files contain correct API paths
  • ✅ All dependencies are installed
  • ✅ TypeScript compiles without errors

🔄 Continuous Integration Workflow​

1. Developer makes changes
   ↓
2. Run `make test-config` (< 5 sec)
   ↓
3. Commit changes
   ↓
4. Run `make test` (~ 2 min)
   ↓
5. Push to GitHub
   ↓
6. Cloud Build triggers
   ↓
7. Step 0: Pre-build validation (< 1 min)
   ↓
8. Step 1: Build + verification (~ 4 min)
   ↓
9. Step 2: Docker build (~ 10 min)
   ↓
10. Step 3: Push to registry
   ↓
11. Step 4: Deploy to GKE
   ↓
12. Run `make test-endpoints` (~ 10 sec)
   ↓
13. ✅ Deployment complete!

📝 Troubleshooting​

Tests pass locally but fail in Cloud Build​

Cause: Different environment or cached files

Fix:

# Clean everything
make clean
rm -rf node_modules
npm install
make test

Tests pass but endpoints fail​

Cause: Browser cache or old pods

Fix:

  1. Hard refresh browser: Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac)
  2. Check pod status: kubectl get pods -n coditect-app
  3. Verify correct image is deployed

Tests hang or timeout​

Cause: npm install or build process is slow

Fix:

# Increase timeout or run with more resources
NODE_OPTIONS=--max_old_space_size=8192 make test

🎉 Benefits​

Before Automated Testing​

  • 8 builds required to find and fix issues
  • 10-20 minutes per build = 80-160 minutes wasted
  • Manual verification required after each build
  • No visibility into what broke

After Automated Testing​

  • 1 build (issues caught before deployment)
  • 2 minutes local testing + 15 minutes Cloud Build = 17 minutes total
  • Automatic verification in CI/CD
  • Clear error messages showing exactly what's wrong

Time Saved: 63-143 minutes per deployment Cost Saved: ~$5-10 per failed build (Cloud Build costs) Stress Reduced: 🚀 Massive!

Next Steps:

  1. Run make test to verify your current setup
  2. Add make test-config to your pre-commit workflow
  3. Review test output to understand what's being validated
  4. Customize tests for your specific needs

Questions? See the scripts themselves:

  • scripts/test-deployment.sh - Main test suite
  • scripts/test-endpoints.sh - Endpoint testing
  • Makefile - Test orchestration