CLAUDE.md
Quick reference for Claude Code in the CODITECT v4 repository.
π Google Cloud Project Setupβ
gcloud config set project serene-voltage-464305-n2
Building API v2β
cd /home/halcasteel/v4/src/api-v2
gcloud config set project serene-voltage-464305-n2 && gcloud builds submit --config=cloudbuild.yaml
Project: serene-voltage-464305-n2
Console URL: https://console.cloud.google.com/cloud-build/builds?project=1059494892139
Building CODI2β
cd /home/halcasteel/v4/codi2
gcloud config set project serene-voltage-464305-n2 && gcloud builds submit --config=cloudbuild.yaml
Binary Location: gs://serene-voltage-464305-n2-builds/codi2/codi2-
Build Artifacts: Stored in Google Cloud Storage bucket
Downloading CODI2 Binaryβ
# Create binaries directory if it doesn't exist
mkdir -p /home/halcasteel/v4/codi2/binaries
# List available binaries (sorted by date)
gsutil ls -l gs://serene-voltage-464305-n2-builds/codi2/ | grep -v "/$" | sort -k2 -r | head -10
# Download the latest binary (note: currently named 'codi2-' without SHORT_SHA)
gsutil cp gs://serene-voltage-464305-n2-builds/codi2/codi2- /home/halcasteel/v4/codi2/binaries/codi2-$(date +%Y-%m-%d-%H%M)
# Make it executable
chmod +x /home/halcasteel/v4/codi2/binaries/codi2-*
# Create symlink to latest
cd /home/halcasteel/v4/codi2/binaries
ln -sf codi2-$(date +%Y-%m-%d-%H%M) codi2-latest
# Verify version
./codi2-latest --version
# Test functionality
./codi2-latest test # Test FDB connection
./codi2-latest whoami # Check session info
./codi2-latest log-ai "Test message" --action TEST # Test logging
Note: The SHORT_SHA substitution in cloudbuild.yaml is not working correctly, resulting in binaries named codi2- instead of codi2-<SHORT_SHA>. This is a known issue but doesn't affect functionality.
π CODI2 Integration Analysisβ
Comprehensive Documentation (2025-10-02)β
-
CODI Scripts Comprehensive Analysis - Complete inventory of 259 bash scripts across 10 functional categories
- Identifies 17 critical scripts (6.6% of total)
- Organized by category with technical details
- Highlights must-have functionality for CODI2 parity
-
CODI2 Gap Analysis - Feature parity assessment between bash scripts and CODI2 Rust implementation
- Current coverage: 35% (12 of 45 features implemented)
- 8 gap categories with priority ratings
- 3-phase implementation roadmap for feature parity
- Critical gaps: service orchestration, log maintenance, export supervision
Key Findingsβ
- Binary Status: 16MB CODI2 binary successfully compiled and ready for deployment
- Critical Scripts: 17 scripts essential for production operations
- Feature Gaps: 65% of functionality needs implementation for full parity
- Priority Plan: Phase 1 focuses on critical infrastructure gaps
π Production Deployment Documentationβ
WebSocket & GKE Integrationβ
- WebSocket Cloud Run Issue - Why WebSocket doesn't work on Cloud Run
- GKE Deployment Success - How we deployed to GKE with WebSocket support
- Optimal WebSocket Solution - HTTPS proxy architecture explanation
Production Deployment Guidesβ
- Production Deployment for coditect.ai - Complete guide for production deployment with custom domain
- GKE Ingress Future Phase - Detailed guide for GKE Ingress with SSL (100-1,000 users)
- Scaling Analysis - Comparison of scaling options and costs
- SSL Certificate Verification - Complete SSL/TLS certificate analysis and verification (Updated: 2025-09-30)
Current Production URLsβ
- Frontend: https://coditect-frontend-1059494892139.us-central1.run.app
- API (Cloud Run): https://coditect-api-v2-1059494892139.us-central1.run.app
- Static IP for coditect.ai: 34.8.51.57
ποΈ FoundationDB Cluster Configurationβ
β οΈ CRITICAL INFRASTRUCTURE CHANGE (2025-09-30)
Decision: Deploy FoundationDB StatefulSet directly in GKE cluster
Why: After 5+ hours of troubleshooting authentication failures, root cause identified as FDB connectivity issue. External cluster at
10.0.1.3/4/5unreachable from GKE pods, Internal Load Balancer at10.128.0.8does not exist. Rather than continue with networking workarounds, deploy FDB properly in GKE for immediate resolution.Impact: All user authentication currently broken (502 errors). This blocks MVP launch.
Analysis Documents:
- CRITICAL-ISSUES-ANALYSIS-2025-09-30.md - Crisis identification and root cause
- GKE-DIAGNOSTICS-SUMMARY-2025-09-30.md - Complete 5-hour diagnostic timeline
Deployment Plan: FDB-GKE-DEPLOYMENT-PLAN-2025-09-30.md (88 minutes to production)
Migration Plan: FDB-MODEL-MIGRATION-PLAN-2025-09-30.md (90 minutes to verify all models)
Timeline: Estimated 2-3 hours to fully functional authentication system
Rollback: Previous external cluster config documented below for reference only
β οΈ DEPRECATED: External Cluster Configuration (Pre-2025-09-30)β
Click to view deprecated configuration
Previous Cluster (UNREACHABLE):
- Cluster String:
coditect:production@10.0.1.3:4500,10.0.1.5:4500,10.0.1.4:4500 - Network: fdb-network (VPC peering failed)
- Nodes:
- fdb-node-1: 10.0.1.3 (us-central1-a) - UNREACHABLE from GKE
- fdb-node-2: 10.0.1.5 (us-central1-a) - UNREACHABLE from GKE
- fdb-node-3: 10.0.1.4 (us-central1-a) - UNREACHABLE from GKE
Previous ILB Configuration (NEVER DEPLOYED):
- ILB IP: 10.128.0.8 (does not exist)
- Backend: fdb-backend (never created)
- Status: Configuration existed in YAML only, never deployed to GCP
Why It Failed:
- VPC peering between fdb-network (10.0.1.0/24) and GKE default network unsuccessful
- GKE pods cannot reach external FDB nodes
- Internal Load Balancer was planned but never created
- Firewall rules ineffective without working VPC peering
Lessons Learned:
- External FDB cluster adds unnecessary networking complexity
- Cross-VPC connectivity unreliable for production databases
- Co-locating FDB with consuming services (API pods) is correct architecture
β NEW: GKE-Native FoundationDB Cluster (2025-09-30+)β
Deployment Status: π‘ IN PROGRESS (Phase 1: StatefulSet deployed)
Architecture:
GKE Cluster (codi-poc-e2-cluster)
βββ Namespace: coditect-app
βββ StatefulSet: foundationdb (3 replicas) β
DEPLOYED
β βββ foundationdb-0 (pod + 50Gi PV)
β βββ foundationdb-1 (pod + 50Gi PV)
β βββ foundationdb-2 (pod + 50Gi PV)
βββ Service: fdb-cluster (headless) β
DEPLOYED
βββ Deployment: fdb-proxy (HAProxy, 2 replicas) β³ PENDING
βββ Service: fdb-proxy-service (ILB at 10.128.0.8) β³ PENDING
Cluster Sizing:
- Current: 3 nodes with
doubleredundancy (MVP) - Redundancy Mode:
double- Replicates data to 2 machines, tolerates 1 failure - Storage: 3 Γ 50GB PersistentVolumes (150GB total)
- Memory: 3 Γ 2-4GB RAM per node
Dynamic Scaling (Zero Downtime):
# Scale to 5 nodes for higher availability
kubectl scale statefulset foundationdb --replicas=5 -n coditect-app
# Upgrade to triple redundancy (survives 2 failures)
kubectl exec -n coditect-app foundationdb-0 -- fdbcli --exec "configure triple"
# No application downtime required
| Redundancy Mode | Nodes | Replication | Tolerates | Use Case |
|---|---|---|---|---|
single | 1-2 | None | 0 failures | Testing only β |
double | 3-4 | 2 copies | 1 failure | MVP (Current) β |
triple | 5+ | 3 copies | 2 failures | High availability π |
Connection Details (After Deployment):
# Cluster file location
/app/fdb.cluster
/etc/foundationdb/fdb.cluster
# Cluster string
coditect:production@10.128.0.8:4500
# HAProxy routes to:
# - foundationdb-0.fdb-cluster.coditect-app.svc.cluster.local:4500
# - foundationdb-1.fdb-cluster.coditect-app.svc.cluster.local:4500
# - foundationdb-2.fdb-cluster.coditect-app.svc.cluster.local:4500
Connectivity Testing:
# From API pod
POD=$(kubectl get pods -n coditect-app --selector=app=coditect-api-v2 -o jsonpath='{.items[0].metadata.name}')
# Test TCP connection
kubectl exec -n coditect-app $POD -- timeout 5 bash -c \
"cat < /dev/null > /dev/tcp/10.128.0.8/4500" && \
echo "β
FDB reachable" || echo "β FDB UNREACHABLE"
# Test FDB cluster status
kubectl exec -n coditect-app foundationdb-0 -- fdbcli --exec "status"
Deployment Commands:
# Deploy FDB cluster (see full plan for details)
cd /home/halcasteel/v4/k8s
kubectl apply -f fdb-storage.yaml
kubectl apply -f fdb-statefulset.yaml
kubectl apply -f fdb-proxy-deployment.yaml
# Initialize cluster
kubectl exec -n coditect-app foundationdb-0 -- fdbcli --exec "configure double"
# Verify
kubectl get statefulset,pods,svc -n coditect-app -l app=foundationdb
Data Models:
- Implemented: User, Tenant, License, workspace (MVP)
- Documented: 20 models in
/docs/reference/database-models/ - ADRs: ADR-006 (data model), ADR-029 (issue tracking)
- See FDB-MODEL-MIGRATION-PLAN
Backup Strategy (Post-Deployment):
# Daily backups to Google Cloud Storage
gsutil mb gs://coditect-fdb-backups
# Backup script (CronJob)
kubectl apply -f k8s/fdb-backup-cronjob.yaml
βΈοΈ Kubernetes (GKE) Managementβ
Cluster Informationβ
- Cluster: codi-poc-e2-cluster
- Zone: us-central1-a
- Namespace: coditect-app
- Load Balancer IP: 34.46.212.40
- Static IP: 34.8.51.57 (for coditect.ai)
Quick Start Commandsβ
Set Context & Projectβ
# Set GCP project
gcloud config set project serene-voltage-464305-n2
# Get cluster credentials
gcloud container clusters get-credentials codi-poc-e2-cluster --zone us-central1-a
# Verify context
kubectl config current-context
# Set default namespace
kubectl config set-context --current --namespace=coditect-app
Deployment Managementβ
View Deploymentsβ
# List all deployments
kubectl get deployments -n coditect-app
# Detailed deployment info
kubectl describe deployment coditect-frontend -n coditect-app
kubectl describe deployment coditect-api-v2 -n coditect-app
# Watch deployment status
kubectl get deployments -n coditect-app -w
Update Deploymentsβ
# Update frontend with new image
kubectl set image deployment/coditect-frontend \
frontend=gcr.io/serene-voltage-464305-n2/coditect-frontend:latest \
-n coditect-app
# Update API with new image
kubectl set image deployment/coditect-api-v2 \
api=gcr.io/serene-voltage-464305-n2/coditect-api-v2:latest \
-n coditect-app
# Watch rollout progress
kubectl rollout status deployment/coditect-frontend -n coditect-app
# Check rollout history
kubectl rollout history deployment/coditect-frontend -n coditect-app
Rollback Deploymentsβ
# Rollback to previous version
kubectl rollout undo deployment/coditect-frontend -n coditect-app
# Rollback to specific revision
kubectl rollout undo deployment/coditect-frontend --to-revision=3 -n coditect-app
# Check rollout status
kubectl rollout status deployment/coditect-frontend -n coditect-app
Scale Deploymentsβ
# Manual scaling
kubectl scale deployment/coditect-frontend --replicas=3 -n coditect-app
kubectl scale deployment/coditect-api-v2 --replicas=5 -n coditect-app
# Check HPA (Horizontal Pod Autoscaler)
kubectl get hpa -n coditect-app
# View HPA details
kubectl describe hpa coditect-frontend-hpa -n coditect-app
kubectl describe hpa coditect-api-v2-hpa -n coditect-app
Pod Managementβ
View Podsβ
# List all pods
kubectl get pods -n coditect-app
# List pods with more details
kubectl get pods -n coditect-app -o wide
# Filter by label
kubectl get pods -n coditect-app -l app=coditect-frontend
kubectl get pods -n coditect-app -l app=coditect-api-v2
# Watch pods
kubectl get pods -n coditect-app -w
Pod Details & Logsβ
# Describe pod (events, status, conditions)
kubectl describe pod <pod-name> -n coditect-app
# View pod logs
kubectl logs <pod-name> -n coditect-app
# Follow logs in real-time
kubectl logs -f <pod-name> -n coditect-app
# View logs from previous container (after crash)
kubectl logs <pod-name> -n coditect-app --previous
# View logs from specific container in multi-container pod
kubectl logs <pod-name> -c <container-name> -n coditect-app
# Tail last 100 lines
kubectl logs <pod-name> -n coditect-app --tail=100
# Logs from all pods with label
kubectl logs -l app=coditect-frontend -n coditect-app --tail=50
Execute Commands in Podsβ
# Interactive shell
kubectl exec -it <pod-name> -n coditect-app -- /bin/sh
kubectl exec -it <pod-name> -n coditect-app -- /bin/bash
# Run single command
kubectl exec <pod-name> -n coditect-app -- ls -la /app
kubectl exec <pod-name> -n coditect-app -- cat /etc/nginx/nginx.conf
# Check environment variables
kubectl exec <pod-name> -n coditect-app -- env
Restart Podsβ
# Delete pod (will be recreated by deployment)
kubectl delete pod <pod-name> -n coditect-app
# Restart deployment (recreates all pods)
kubectl rollout restart deployment/coditect-frontend -n coditect-app
kubectl rollout restart deployment/coditect-api-v2 -n coditect-app
Service & Networkingβ
View Servicesβ
# List all services
kubectl get services -n coditect-app
kubectl get svc -n coditect-app
# Describe service
kubectl describe svc coditect-frontend -n coditect-app
kubectl describe svc api-loadbalancer -n coditect-app
# Get service endpoints
kubectl get endpoints -n coditect-app
Ingress Managementβ
# View ingress
kubectl get ingress -n coditect-app
# Describe ingress (shows backend health)
kubectl describe ingress coditect-production-ingress -n coditect-app
# Get ingress YAML
kubectl get ingress coditect-production-ingress -n coditect-app -o yaml
SSL Certificatesβ
# List managed certificates
kubectl get managedcertificate -n coditect-app
# Check certificate status (shows Active/Provisioning/FailedNotVisible per domain)
kubectl describe managedcertificate coditect-ai-ssl -n coditect-app
# Watch certificate provisioning (updates every 30 seconds)
watch -n 30 'kubectl describe managedcertificate coditect-ai-ssl -n coditect-app | grep -A5 Status'
# Test HTTPS access
curl -I https://coditect.ai
curl -I https://api.coditect.ai
curl -I https://www.coditect.ai
# Verify SSL certificate details (expiration, subject)
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -subject
# Check DNS resolution
dig coditect.ai +short
dig api.coditect.ai +short
dig www.coditect.ai +short
ConfigMaps & Secretsβ
ConfigMapsβ
# List configmaps
kubectl get configmap -n coditect-app
# View configmap
kubectl describe configmap fdb-cluster-config -n coditect-app
kubectl get configmap fdb-cluster-config -n coditect-app -o yaml
# Edit configmap
kubectl edit configmap fdb-cluster-config -n coditect-app
Secretsβ
# List secrets
kubectl get secrets -n coditect-app
# View secret (base64 encoded)
kubectl get secret <secret-name> -n coditect-app -o yaml
# Decode secret value
kubectl get secret <secret-name> -n coditect-app -o jsonpath='{.data.password}' | base64 -d
Troubleshootingβ
Check Cluster Healthβ
# Node status
kubectl get nodes
kubectl describe node <node-name>
# Cluster info
kubectl cluster-info
# Component status
kubectl get componentstatuses
# Resource usage
kubectl top nodes
kubectl top pods -n coditect-app
Debug Common Issuesβ
# Pod stuck in Pending
kubectl describe pod <pod-name> -n coditect-app | grep -A10 Events
# Pod CrashLoopBackOff
kubectl logs <pod-name> -n coditect-app --previous
kubectl describe pod <pod-name> -n coditect-app
# ImagePullBackOff
kubectl describe pod <pod-name> -n coditect-app | grep -A5 "Failed to pull image"
# Service not accessible
kubectl get endpoints <service-name> -n coditect-app
kubectl describe svc <service-name> -n coditect-app
# Check pod network connectivity
kubectl exec <pod-name> -n coditect-app -- wget -O- http://coditect-frontend
kubectl exec <pod-name> -n coditect-app -- nslookup coditect-frontend
# Check external connectivity
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl http://coditect-frontend.coditect-app.svc.cluster.local
Network Analysis & Troubleshootingβ
# DNS Resolution Testing
dig coditect.ai +short
dig api.coditect.ai +short
dig www.coditect.ai +short
nslookup coditect.ai
host coditect.ai
# SSL/TLS Certificate Analysis
# Quick SSL check
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer
# Detailed certificate information
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | openssl x509 -noout -text
# Certificate chain verification
echo | openssl s_client -connect coditect.ai:443 -servername coditect.ai 2>&1 | grep -A10 "Certificate chain"
# Test HTTPS with curl (includes SSL verification)
curl -vI https://coditect.ai 2>&1 | grep -E "SSL|TLS|certificate"
# HTTP/HTTPS Response Testing
# Test with status code and timing
curl -s -o /dev/null -w "Status: %{http_code}, SSL: %{ssl_verify_result}, Time: %{time_total}s\n" https://coditect.ai
# Test all domains
for domain in coditect.ai api.coditect.ai www.coditect.ai; do
echo "Testing https://$domain"
curl -s -o /dev/null -w "Status: %{http_code}, Time: %{time_total}s\n" "https://$domain"
done
# Ingress Backend Health
# Check if ingress backends are healthy
kubectl describe ingress coditect-production-ingress -n coditect-app | grep -A20 "Annotations"
# Check backend services
kubectl get svc -n coditect-app -o wide
# Test load balancer directly
curl -I http://34.46.212.40
curl -I http://34.46.212.40/api/v2/health
# Network Policy Troubleshooting
# Check network policies (if any)
kubectl get networkpolicies -n coditect-app
# Test pod-to-pod connectivity
kubectl exec -it <pod-1> -n coditect-app -- ping <pod-2-ip>
kubectl exec -it <pod-1> -n coditect-app -- curl http://<service-name>
# Trace network path (requires traceroute)
kubectl exec -it <pod-name> -n coditect-app -- traceroute coditect.ai
# Check firewall rules (GCP level)
gcloud compute firewall-rules list --filter="name~coditect"
# Network Performance Testing
# Test latency to load balancer
ping -c 5 34.8.51.57
# Test DNS resolution speed
time dig coditect.ai
# HTTP request timing breakdown
curl -w "@-" -o /dev/null -s https://coditect.ai <<'EOF'
time_namelookup: %{time_namelookup}s\n
time_connect: %{time_connect}s\n
time_appconnect: %{time_appconnect}s\n
time_pretransfer: %{time_pretransfer}s\n
time_redirect: %{time_redirect}s\n
time_starttransfer: %{time_starttransfer}s\n
----------\n
time_total: %{time_total}s\n
EOF
# GKE Network Troubleshooting
# Check node network
kubectl get nodes -o wide
# Check pod network (CNI)
kubectl get pods -n kube-system | grep -E "calico|cilium|flannel"
# Describe node to see network config
kubectl describe node <node-name> | grep -A5 "Network"
# Check service endpoints are assigned
kubectl get endpoints -n coditect-app
# Verify ingress has IP assigned
kubectl get ingress -n coditect-app -o wide
See Also: SSL Certificate Verification Report for complete SSL/TLS analysis.
View Eventsβ
# All events in namespace
kubectl get events -n coditect-app --sort-by='.lastTimestamp'
# Watch events
kubectl get events -n coditect-app -w
# Events for specific resource
kubectl describe pod <pod-name> -n coditect-app | grep -A20 Events
Resource Managementβ
View Resourcesβ
# All resources in namespace
kubectl get all -n coditect-app
# Specific resource types
kubectl get pods,svc,deploy,ing -n coditect-app
# Resource quotas
kubectl get resourcequota -n coditect-app
# Limit ranges
kubectl get limitrange -n coditect-app
Apply/Update Resourcesβ
# Apply YAML file
kubectl apply -f deployment.yaml -n coditect-app
# Apply directory of YAMLs
kubectl apply -f k8s/production/ -n coditect-app
# Delete resources
kubectl delete -f deployment.yaml -n coditect-app
kubectl delete deployment coditect-frontend -n coditect-app
# Replace resource (destructive)
kubectl replace -f deployment.yaml -n coditect-app
Monitoring & Metricsβ
Resource Usageβ
# Node metrics
kubectl top nodes
# Pod metrics
kubectl top pods -n coditect-app
# Sort by CPU
kubectl top pods -n coditect-app --sort-by=cpu
# Sort by memory
kubectl top pods -n coditect-app --sort-by=memory
Port Forwarding (Local Testing)β
# Forward pod port to local
kubectl port-forward pod/<pod-name> 8080:8080 -n coditect-app
# Forward service port to local
kubectl port-forward svc/coditect-frontend 8080:80 -n coditect-app
# Access in browser: http://localhost:8080
Production Deployment Workflowβ
Complete Deployment Processβ
# 1. Build and push new image
cd /home/halcasteel/v4/src/frontend
gcloud builds submit --config=cloudbuild.yaml --project=serene-voltage-464305-n2
# 2. Update GKE deployment
kubectl set image deployment/coditect-frontend \
frontend=gcr.io/serene-voltage-464305-n2/coditect-frontend:latest \
-n coditect-app
# 3. Watch rollout
kubectl rollout status deployment/coditect-frontend -n coditect-app
# 4. Verify pods are running
kubectl get pods -n coditect-app -l app=coditect-frontend
# 5. Check logs for errors
kubectl logs -l app=coditect-frontend -n coditect-app --tail=50
# 6. Test service (load balancer)
curl http://34.46.212.40/docs/api
# 7. Test HTTPS endpoints (production domain)
curl -I https://coditect.ai
curl https://coditect.ai/docs/api
Production Testing & Verificationβ
# Test all HTTPS endpoints with SSL verification and timing
for domain in coditect.ai api.coditect.ai www.coditect.ai; do
echo "Testing https://$domain"
curl -s -o /dev/null -w "Status: %{http_code}, SSL: %{ssl_verify_result}, Time: %{time_total}s\n" "https://$domain"
done
# Test all documentation pages
for page in api websocket installation architecture auth security; do
echo -n "https://coditect.ai/docs/$page: "
curl -s -o /dev/null -w "%{http_code}\n" "https://coditect.ai/docs/$page"
done
# Comprehensive health check
echo "=== Frontend Health ==="
curl -s https://coditect.ai | head -20
echo "=== API Health ==="
curl -s https://api.coditect.ai/api/v2/health | jq '.'
# Check SSL certificate expiration
echo "=== SSL Certificate ==="
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | \
openssl x509 -noout -dates -subject
# Load test (optional - requires hey tool)
# hey -n 100 -c 10 https://coditect.ai/api/v2/health
Emergency Rollbackβ
# Quick rollback
kubectl rollout undo deployment/coditect-frontend -n coditect-app
# Verify rollback
kubectl rollout status deployment/coditect-frontend -n coditect-app
# Check pods
kubectl get pods -n coditect-app -l app=coditect-frontend
Common CODITECT Operationsβ
Update Frontend Documentationβ
# After building new frontend image with docs
kubectl set image deployment/coditect-frontend \
frontend=gcr.io/serene-voltage-464305-n2/coditect-frontend:latest \
-n coditect-app
# Wait for rollout
kubectl rollout status deployment/coditect-frontend -n coditect-app
# Verify new pods
kubectl get pods -n coditect-app -l app=coditect-frontend -o wide
Update API Serverβ
# Deploy new API version
kubectl set image deployment/coditect-api-v2 \
api=gcr.io/serene-voltage-464305-n2/coditect-api-v2:latest \
-n coditect-app
# Monitor rollout
kubectl rollout status deployment/coditect-api-v2 -n coditect-app
# Check API health
kubectl exec -it <api-pod-name> -n coditect-app -- curl localhost:8080/api/v2/health
Check FoundationDB Connectivityβ
# Test from API pod
kubectl exec -it <api-pod-name> -n coditect-app -- sh
# Inside pod:
cat /etc/foundationdb/fdb.cluster
# Should show: coditect:production@10.128.0.8:4500
Quick Reference Summaryβ
| Task | Command |
|---|---|
| List deployments | kubectl get deploy -n coditect-app |
| Update deployment | kubectl set image deployment/<name> <container>=<image> -n coditect-app |
| Rollout status | kubectl rollout status deployment/<name> -n coditect-app |
| View pods | kubectl get pods -n coditect-app |
| Pod logs | kubectl logs <pod-name> -n coditect-app |
| Exec into pod | kubectl exec -it <pod-name> -n coditect-app -- sh |
| Restart deployment | kubectl rollout restart deployment/<name> -n coditect-app |
| Rollback deployment | kubectl rollout undo deployment/<name> -n coditect-app |
| View services | kubectl get svc -n coditect-app |
| View ingress | kubectl get ingress -n coditect-app |
| Check certificates | kubectl get managedcertificate -n coditect-app |
| View events | kubectl get events -n coditect-app --sort-by='.lastTimestamp' |
| Resource usage | kubectl top pods -n coditect-app |
π³ Containerization & Pod Managementβ
User Pod Architectureβ
Each user gets an isolated development pod with:
- Base Image: Custom container with development tools
- Persistent Storage: User workspace mounted as PVC
- CODI2 Integration: Local monitoring and logging
- Network: Isolated namespace with controlled egress
Container Registryβ
# Google Container Registry
gcr.io/serene-voltage-464305-n2/
βββ coditect-api-v2:latest # API server
βββ coditect-frontend:latest # React frontend
βββ coditect-websocket:latest # WebSocket gateway
βββ coditect-user-pod:latest # User development pod
βββ codi2:latest # CODI2 monitoring service
Pod Deployment Processβ
# 1. User registers/logs in
# 2. API creates workspace in FDB
# 3. Kubernetes creates user pod:
kubectl create namespace user-${USER_ID}
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: workspace
namespace: user-${USER_ID}
spec:
containers:
- name: workspace
image: gcr.io/serene-voltage-464305-n2/coditect-user-pod:latest
env:
- name: USER_ID
value: "${USER_ID}"
- name: CODI2_SERVER
value: "codi2-service.coditect-app:8765"
volumeMounts:
- name: workspace
mountPath: /workspace
- name: codi2-config
mountPath: /workspace/.codi2
volumes:
- name: workspace
persistentVolumeClaim:
claimName: user-workspace
- name: codi2-config
configMap:
name: codi2-config
EOF
CODI2 Integrationβ
- Local CODI2: Runs in user pod for file monitoring
- Server CODI2: Central service for log aggregation
- Communication: WebSocket connection for real-time sync
- Storage: Logs persisted to FoundationDB
Authentication Flowβ
- User logs in via frontend
- API validates JWT and creates session
- WebSocket gateway authenticates with same JWT
- User pod created with session credentials
- CODI2 tracks all activity with session ID
π¨ Directory Boundaryβ
STAY WITHIN v4/ - Do not navigate outside this directory.
π Directory Organizationβ
For detailed information about where files belong and directory structure guidelines, see: DIRECTORY-ORGANIZATION-GUIDE.md
π Directory Organization (Updated 2025-09-21)β
Core Structureβ
v4/
βββ src/ # Source code (API, frontend, websocket, scripts)
βββ agents/ # Agent system (10 CODITECT + 11 Claude sub-agents)
βββ docs/ # Diamond core documentation (214 essential files)
βββ standards/ # Foundation standards (logging, testing, errors)
βββ .codi/ # Monitoring infrastructure
βββ codi2/ # CODI2 development (metrics engine)
βββ tests/ # Test suites
βββ config/ # Configuration files
βββ infrastructure/ # Deployment configs
βββ .archived/ # Historical files (1,874+ items)
π Session Instantiation Protocolβ
Step 1: Set GCP Project (MANDATORY)β
# ALWAYS set the correct GCP project first
gcloud config set project serene-voltage-464305-n2
# Project name: Google-GCP-CLI
# Project number: 1059494892139
# Set billing quota project (ensures consistent billing across all projects)
gcloud config set billing/quota_project serene-voltage-464305-n2
# Verify settings
gcloud config list | grep -E "project|quota"
Step 2: Initialize Infrastructure (MANDATORY)β
# Navigate to repository root
cd /home/hal/CODITECTv4 # or cd /home/halcasteel/v4
# Start all monitoring infrastructure
source .codi/scripts/infrastructure/session-start-hook.sh
# This automatically:
# β Starts file monitor (tracks all file operations)
# β Starts export watcher (archives conversation exports)
# β Starts API server (web tools on port 8080)
# β Sets up logging to .codi/logs/codi-ps.log
# β Detects AI environment (Claude Code, CODITECT, etc.)
Step 3: Set Session Identity (REQUIRED FOR AGENTS)β
# For ORCHESTRATOR sessions:
export SESSION_ID="ORCHESTRATOR-SESSION-$(date +%Y-%m-%d)-01"
# For specialist agents:
export SESSION_ID="[AGENT-TYPE]-SESSION-$(date +%Y-%m-%d)-[SEQUENCE]"
# Examples:
# export SESSION_ID="RUST-DEVELOPER-SESSION-2025-09-23-01"
# export SESSION_ID="QA-REVIEWER-SESSION-2025-09-23-02"
# IMPORTANT: Initialize the session identity (sets actor attribution)
source /home/halcasteel/v4/.codi/scripts/session-management/identity/set-session-identity.sh "$SESSION_ID"
# This ensures all codi-log commands are properly attributed to your session
Step 4: Log Session Start (REQUIRED)β
# Log the session initialization
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Starting work on [TASK_DESCRIPTION]" "SESSION_START"
# For ORCHESTRATOR, include role announcement:
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Ready for task distribution and agent coordination" "ORCHESTRATION"
Step 5: Create Project Directory (FOR NEW TASKS)β
# Generate project directory with ISO date and UUID
PROJECT_NAME="$(date +%Y%m%d-%H%M%S)-[TASK_NAME]-$(uuidgen | cut -c1-8)"
mkdir -p "projects/$PROJECT_NAME"
cd "projects/$PROJECT_NAME"
# Example:
# PROJECT_NAME="20250923-120530-SubAgentSprint-a3f2b1c9"
# Creates: projects/20250923-120530-SubAgentSprint-a3f2b1c9/
Step 5: Create Task Checklist (MANDATORY FOR ORCHESTRATOR)β
# Create task checklist with checkboxes
TASK_LIST="TASK-LIST-$(date +%Y%m%d-%H%M%S).md"
cat > "$TASK_LIST" << 'EOF'
# Task List: [PROJECT_NAME]
**Session**: $SESSION_ID
**Created**: $(date +"%Y-%m-%d %H:%M:%S")
**Status**: IN_PROGRESS
## Primary Objectives
- [ ] [Objective 1]
- [ ] [Objective 2]
- [ ] [Objective 3]
## Implementation Tasks
- [ ] Task 1: Description
- [ ] Task 2: Description
- [ ] Task 3: Description
## Quality Gates
- [ ] ADR Compliance (40/40)
- [ ] Test Coverage (β₯95%)
- [ ] Integration Tests Pass
- [ ] Documentation Complete
EOF
# Log task list creation
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Created task list $TASK_LIST" "CREATE"
Step 6: Verify Initializationβ
# Check all services are running
ps aux | grep -E "(file-monitor|export-watcher|api-server)" | grep -v grep
# Verify session is logged
grep "$SESSION_ID" .codi/logs/codi-ps.log | tail -5
# Access web tools
echo "Web tools available at:"
echo " - Hub: http://localhost:8080"
echo " - Logs: http://localhost:8080/log-viewer.html"
echo " - Dashboard: http://localhost:8080/agent-dashboard.html"
π Logging System Documentationβ
Log Format (JSON)β
All logs are stored as newline-delimited JSON in .codi/logs/codi-ps.log:
{
"timestamp": "2025-09-23T21:08:22Z",
"level": "INFO",
"component": "ai.activity",
"action": "TEST",
"message": "Testing feature implementation",
"sessionId": "ORCHESTRATOR-SESSION-2025-09-23-02",
"actor": {
"type": "agent",
"id": "ORCHESTRATOR-SESSION-2025-09-23-02"
},
"details": {
"ai_id": "ORCHESTRATOR-SESSION-2025-09-23-02",
"ai_role": "agent",
"ai_details": "Multi-agent orchestration session",
"working_directory": "/home/halcasteel/v4"
}
}
Logging Commandsβ
# β
CORRECT - Session attribution formats:
# Method 1: Include session ID in message (RECOMMENDED)
./.codi/scripts/logging/actors/codi-log-ai.sh "ORCHESTRATOR-SESSION-2025-09-23-02: Implementing user API" "CREATE"
# Method 2: Set SESSION_ID environment variable
export SESSION_ID="ORCHESTRATOR-SESSION-2025-09-23-02"
./.codi/scripts/logging/actors/codi-log-ai.sh "Implementing user API" "CREATE"
# β INCORRECT - Broken formats:
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: message" "ACTION" # Variable not expanded
echo '{"message": "test"}' >> .codi/logs/codi-ps.log # Breaks JSON format
Session ID Formatβ
Session IDs follow strict naming conventions for proper attribution:
| Pattern | Example | Purpose |
|---|---|---|
AGENT-TYPE-SESSION-YYYY-MM-DD-NN | ORCHESTRATOR-SESSION-2025-09-23-01 | Date-based with sequence |
AGENT-TYPE-SESSION-N | RUST-DEVELOPER-SESSION-7 | Simple numbering |
AGENT-TYPE-SESSION-YYYYMMDD-HHMM | QA-REVIEWER-SESSION-20250923-1430 | Time-specific |
Action Typesβ
| Action | Use Case | Example |
|---|---|---|
SESSION_START | Beginning work | "Starting orchestration session" |
CREATE | New files/resources | "Created user API endpoint" |
UPDATE | Modifying existing | "Updated authentication logic" |
FIX | Bug fixes | "Fixed null pointer in auth" |
TEST | Running tests | "Executed integration tests" |
ANALYZE | Code analysis | "Reviewing ADR compliance" |
COORDINATE | Multi-agent work | "Assigning tasks to agents" |
HANDOFF | Task completion | "Handing off to QA review" |
FILE_CLAIM | Claiming ownership | "Claiming src/api/users.rs" |
FILE_RELEASE | Releasing files | "Releasing src/api/users.rs" |
ORCHESTRATION | Orchestrator tasks | "Distributing work to agents" |
MILESTONE | Major achievements | "Completed API v2 migration" |
Log File Conventionsβ
# Primary log file (all sessions)
.codi/logs/codi-ps.log
# Session-specific logs (optional)
.codi/logs/session-ORCHESTRATOR-SESSION-2025-09-23-01.log
.codi/logs/session-RUST-DEVELOPER-SESSION-7.log
# Monitor output
.codi/logs/file-monitor.out
.codi/logs/export-watcher.out
# Archived logs
.codi/logs/archive/codi-ps-20250923-backup.log
Path Details in Logsβ
Include sufficient path context without redundancy:
# β
GOOD - Specific and actionable
"ORCHESTRATOR-SESSION-2025-09-23-02: Created src/api-v2/handlers/user.rs"
"RUST-DEVELOPER-SESSION-7: Updated tests/integration/auth_test.rs line 142"
"QA-REVIEWER-SESSION-3: Reviewing docs/api/v2/endpoints.md for completeness"
# β BAD - Too vague or too verbose
"Updated a file" # Which file?
"Modified /home/halcasteel/v4/src/api-v2/handlers/auth/jwt/token/validation/expiry.rs line 2341 character 42" # Too detailed
Log Management Scriptsβ
| Script | Purpose | Usage |
|---|---|---|
codi-log-ai.sh | AI session logging | ./.codi/scripts/logging/actors/codi-log-ai.sh "message" "ACTION" |
codi-log-human.sh | Human developer logs | ./.codi/scripts/logging/actors/codi-log-human.sh "message" "ACTION" |
codi-log.sh | General logging | ./.codi/scripts/logging/codi-log.sh "message" "ACTION" |
aggregate-logs.sh | Combine session logs | ./.codi/scripts/aggregate-logs.sh |
sync-logs.sh | Sync between sessions | ./.codi/scripts/sync-logs.sh |
split-logs-by-session.sh | Split by session | ./.codi/scripts/split-logs-by-session.sh |
Viewing Logsβ
# Real-time monitoring
tail -f .codi/logs/codi-ps.log | jq '.'
# Filter by session
grep "ORCHESTRATOR-SESSION-2025-09-23-02" .codi/logs/codi-ps.log | jq '.'
# Filter by action
jq 'select(.action == "CREATE")' .codi/logs/codi-ps.log
# Today's activity
grep "$(date +%Y-%m-%d)" .codi/logs/codi-ps.log | jq '.'
# Web interface
open http://localhost:8080/log-viewer.html
Best Practicesβ
- Always include session ID in logs for proper attribution
- Use appropriate action types for better filtering
- Include file paths when relevant (src/api/file.rs)
- Keep messages concise but informative
- Never write directly to log files - use scripts
- Backup logs before Git operations (see Git Playbook)
π― Session Typesβ
| Session Pattern | Use For |
|---|---|
| ORCHESTRATOR-SESSION-YYYY-MM-DD-NN | Multi-agent coordination (NN = sequence) |
| RUST-DEVELOPER-SESSION-N | Rust backend |
| QA-REVIEWER-SESSION-N | Documentation review |
| CLOUD-ARCHITECT-SESSION-N | GCP/CI/CD |
| FRONTEND-EXPERT-SESSION-N | React/TypeScript |
| GENERAL-SESSION-N | Mixed tasks |
Note: When multiple sessions occur on same day, add sequence number (01, 02) or time (HHMM) to differentiate
π¦ Monitoring Infrastructureβ
What Gets Monitored: All file operations in /home/halcasteel/v4/*
- β Creates, updates, deletes, moves
- β All subdirectories (including deep nesting)
- β Logged as JSON to
.codi/logs/codi-ps.log - β Console output to
.codi/logs/file-monitor.out
Recent Changes (2025-09-21):
- Consolidated 5 file monitors β 1 working script
- Fixed JSON logging (was broken in "file-monitor-fixed.sh")
- Updated session-start-hook.sh to use consolidated monitor
π οΈ Common Commandsβ
# Check status
ps aux | grep -E "(file-monitor-consolidated|export-watcher|api-server)" | grep -v grep
# View logs (stored in .codi/logs/codi-ps.log)
tail -f .codi/logs/codi-ps.log | jq '.'
# See active sessions
grep "SESSION_START" .codi/logs/codi-ps.log | tail -10
# Stop everything
pkill -f "file-monitor|export-watcher|api-server"
# Check essential scripts
cat .codi/scripts/ESSENTIAL-SCRIPTS.md
π Web Tools (http://localhost:8080)β
π€ Agent Systemβ
Available Agentsβ
CODITECT Agents (Primary - Strategic decisions):
- orchestrator - Multi-agent coordination, task distribution
- rust-developer - Rust backend, API implementation
- frontend-developer - React/TypeScript UI development
- database-specialist - FoundationDB schemas, multi-tenant patterns
- cloud-architect - GCP infrastructure, CI/CD
- security-specialist - Authentication, hardening
- testing-specialist - Test coverage (95%), TDD
- monitoring-specialist - Observability, metrics
- qa-reviewer - Documentation, ADR compliance
- ai-specialist - AI provider integration
Claude Sub-Agents (Tactical - Deep analysis): Available for delegation when CODITECT agents need specialized expertise.
π― ORCHESTRATOR First Stepsβ
Creating New ORCHESTRATOR Sessionsβ
To start a new ORCHESTRATOR session in Claude Code:
-
Copy the First Instruction:
cat docs/ORCHESTRATOR-FIRST-INSTRUCTION.mdCopy everything between the "---" markers
-
Paste into new Claude Code session as the very first message
-
The ORCHESTRATOR will then:
- Read all required documents
- Initialize infrastructure
- Set session identity
- Confirm readiness
Documents ORCHESTRATOR Must Readβ
When instantiated as ORCHESTRATOR, ALWAYS read these documents in order:
- README.md - Project overview and links to all resources
- agents/coditect/orchestrator.md - Your complete role definition
- docs/CODITECT-GIT-PLAYBOOK.md - Git management strategies
- docs/LOG-MANAGEMENT-SCRIPT-GUIDE.md - Log synchronization
Orchestrator Key Responsibilitiesβ
-
Project Setup - Create standardized project directories
./.codi/scripts/create-project-directory.sh "Task Name" -
Task Management - Maintain task checklists with checkboxes
- Location:
projects/YYYYMMDD-HHMMSS-TaskName-UUID/TASK-LIST-*.md - Track: Objectives, assignments, quality gates, coordination
- Location:
-
Multi-Session Coordination - Prevent conflicts between agents
- Monitor: Active sessions via CODI logs
- Enforce: File claiming protocol
- Coordinate: Handoffs and integration
-
Quality Enforcement - 40/40 ADR compliance on all deliverables
Essential Links for Orchestratorsβ
- Project Management: See project structure in README.md
- Git Workflows: docs/CODITECT-GIT-PLAYBOOK.md
- Log Management: docs/LOG-MANAGEMENT-SCRIPT-GUIDE.md
- Agent Guides: agents/coditect/*.md
- Sprint Planning: agents/sub-agent-build-sprint-checklist.md
π Agent Instantiation Sequenceβ
# 1. ALWAYS start infrastructure first
cd /home/hal/CODITECTv4
source .codi/scripts/infrastructure/session-start-hook.sh
# 2. Set your agent identity
export SESSION_ID="[AGENT-TYPE]-SESSION-$(date +%Y-%m-%d)-[SEQUENCE]"
# Example: export SESSION_ID="RUST-DEVELOPER-SESSION-2025-09-23-01"
# 3. Log session start with role details
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Starting work as [AGENT_ROLE]" "SESSION_START"
# 4. For ORCHESTRATOR: Create project structure
if [[ "$SESSION_ID" =~ ORCHESTRATOR ]]; then
PROJECT_DIR="projects/$(date +%Y%m%d-%H%M%S)-[TASK]-$(uuidgen | cut -c1-8)"
mkdir -p "$PROJECT_DIR"
cd "$PROJECT_DIR"
# Create task checklist
cat > "TASK-LIST-$(date +%Y%m%d).md" << 'EOF'
# Task Checklist
**Orchestrator**: $SESSION_ID
**Created**: $(date)
## Tasks
- [ ] Task 1
- [ ] Task 2
- [ ] Task 3
EOF
fi
# 5. Announce capabilities
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Ready. Capabilities: [LIST_CAPABILITIES]" "ORCHESTRATION"
π Multi-Session Coordinationβ
Prevent Collisions:
# Claim your files
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID claiming src/api/" "FILE_CLAIM"
# Check active sessions
grep "SESSION_START" .codi/logs/codi-ps.log | tail -10
# Coordinate shared work
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Need coordination on shared file" "COORDINATE"
Complete Guide: See agents/multi-session-guide.md
ποΈ Agent Instantiation Protocolβ
When starting as an agent, ALWAYS:
-
Read your agent guide first:
cat agents/coditect/[YOUR-AGENT].md -
Set your session ID:
export SESSION_ID="[AGENT-TYPE]-SESSION-N"
# Example: export SESSION_ID="RUST-DEVELOPER-SESSION-7" -
Log your session start:
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Starting work on [task]" "SESSION_START" -
Understand your boundaries:
- File ownership: Respect file boundaries in your agent guide
- Dependencies: Check which other agents you depend on
- Quality standards: Follow ADR requirements and coverage targets
- Integration points: Understand CODI logging and coordination
-
Know when to delegate:
- Complex analysis beyond your scope β Use Claude sub-agents
- Cross-agent coordination needed β Escalate to orchestrator
- Quality gates required β Coordinate with qa-reviewer
π Delegation to Claude Sub-Agentsβ
CODITECT agents can delegate complex tasks to Claude specialists:
# Example: Rust developer delegating security review
Task(
description="Security audit of auth module",
prompt="Review auth.rs for OWASP compliance and ADR-024 hardening",
subagent_type="security-specialist"
)
Available Claude Sub-Agents: See agents/claude-subagents/README.md
π Essential Reading for New Agentsβ
- Your specific guide:
agents/coditect/[AGENT-NAME].md - System overview:
agents/README.md - Integration patterns:
CODITECT-CLAUDE-AGENT-INTEGRATION.md - Migration info:
agents/migration-guide.md
ποΈ Agent Dashboardβ
Monitor all agent activity: http://localhost:8080/agent-dashboard.html
π Moved Files Referenceβ
Scripts Organizationβ
All operational scripts have been organized from root into categorized directories:
Deployment Scripts (moved to scripts/deployment/)β
- QUICK-DEPLOY.sh - Quick deployment to Cloud Run
- GCLOUD-BUILD-DEPLOY.sh - Full GCloud build deployment
- GCLOUD-SIMPLE.sh - Simple GCloud commands
- DEPLOY-FRONTEND-FIX.sh - Frontend deployment fixes
- FRONTEND-API-FIX.sh - API integration fixes
- setup-gcp-configs.sh - GCP configuration setup
- sync-coditect.sh - Sync deployment artifacts
Testing Scripts (moved to scripts/testing/)β
- CREATE-TEST-USER.sh - Create test users
- TEST-API-CREATE-USER.sh - Test user creation API
- TEST-COMPLETE-FILE-API.sh - Complete file API tests
- TEST-DEPLOYED-API.sh - Test deployed APIs
- TEST-FILE-API.sh - File operation tests
- TEST-FILE-API-MOCK-AUTH.sh - Mock auth tests
- TEST-FILE-API-WITH-LOGIN.sh - Authenticated file tests
- TEST-FILE-BRIDGE.sh - File bridge tests
- TEST-LOCAL-FILE-API.sh - Local file API tests
- LOGIN-TEST-CURL.sh - Login endpoint tests
Documentation Organizationβ
Documentation has been organized into appropriate subdirectories:
Guides (moved to docs/guides/)β
- GCP-PROJECT-setup.md - GCP project setup instructions
- GCP-BILLING-STRUCTURE.md - Billing configuration guide
- GCP-QUICK-REFERENCE.md - Quick GCP command reference
- LOGIN-ENDPOINT-FIX.md - Login troubleshooting guide
- REGISTER-USER-MANUAL.md - User registration manual
Implementation (moved to docs/implementation/)β
- MVP-IMPLEMENTATION-TASKS.md - Detailed MVP tasks
- MVP-quick-start.md - Quick MVP setup guide
- MVP-REAL-IMPLEMENTATION.md - Real implementation details
- DEPLOYMENT-ISSUES-SUMMARY.md - Common issues
Architecture (moved to docs/architecture/)β
- GKE-integration-plan.md - Kubernetes integration plan
- GKE-WORKSPACE-status.md - workspace status details
Workflows (moved to docs/workflows/)β
- CODITECT-COMPLETE-WORKFLOW.md - Full system workflow
- WORKFLOW-IMPLEMENTATION-ROADMAP.md - Implementation roadmap
Testing (moved to docs/testing/)β
- FRONTEND-BACKEND-TEST-SUMMARY.md - Test results summary
- test-frontend-ui.md - Frontend UI test procedures
Integration Test Scripts (moved to tests/integration/)β
- api-v2-test.sh - API v2 test suite
- test-frontend-backend.sh - Frontend-backend integration
- test-gke-workspace.sh - GKE workspace tests
- test-json.sh - JSON handling tests
Archived Session Dataβ
Old session data moved to .session/archive/:
2025-09-23-api-v2-frontend-testing/- Previous testing session files