Skip to main content

CLAUDE.md

Quick reference for Claude Code in the CODITECT v4 repository.

πŸš€ Google Cloud Project Setup​

gcloud config set project serene-voltage-464305-n2

Building API v2​

cd /home/halcasteel/v4/src/api-v2
gcloud config set project serene-voltage-464305-n2 && gcloud builds submit --config=cloudbuild.yaml

Project: serene-voltage-464305-n2
Console URL: https://console.cloud.google.com/cloud-build/builds?project=1059494892139

Building CODI2​

cd /home/halcasteel/v4/codi2
gcloud config set project serene-voltage-464305-n2 && gcloud builds submit --config=cloudbuild.yaml

Binary Location: gs://serene-voltage-464305-n2-builds/codi2/codi2-
Build Artifacts: Stored in Google Cloud Storage bucket

Downloading CODI2 Binary​

# Create binaries directory if it doesn't exist
mkdir -p /home/halcasteel/v4/codi2/binaries

# List available binaries (sorted by date)
gsutil ls -l gs://serene-voltage-464305-n2-builds/codi2/ | grep -v "/$" | sort -k2 -r | head -10

# Download the latest binary (note: currently named 'codi2-' without SHORT_SHA)
gsutil cp gs://serene-voltage-464305-n2-builds/codi2/codi2- /home/halcasteel/v4/codi2/binaries/codi2-$(date +%Y-%m-%d-%H%M)

# Make it executable
chmod +x /home/halcasteel/v4/codi2/binaries/codi2-*

# Create symlink to latest
cd /home/halcasteel/v4/codi2/binaries
ln -sf codi2-$(date +%Y-%m-%d-%H%M) codi2-latest

# Verify version
./codi2-latest --version

# Test functionality
./codi2-latest test # Test FDB connection
./codi2-latest whoami # Check session info
./codi2-latest log-ai "Test message" --action TEST # Test logging

Note: The SHORT_SHA substitution in cloudbuild.yaml is not working correctly, resulting in binaries named codi2- instead of codi2-<SHORT_SHA>. This is a known issue but doesn't affect functionality.

πŸ“Š CODI2 Integration Analysis​

Comprehensive Documentation (2025-10-02)​

  • CODI Scripts Comprehensive Analysis - Complete inventory of 259 bash scripts across 10 functional categories

    • Identifies 17 critical scripts (6.6% of total)
    • Organized by category with technical details
    • Highlights must-have functionality for CODI2 parity
  • CODI2 Gap Analysis - Feature parity assessment between bash scripts and CODI2 Rust implementation

    • Current coverage: 35% (12 of 45 features implemented)
    • 8 gap categories with priority ratings
    • 3-phase implementation roadmap for feature parity
    • Critical gaps: service orchestration, log maintenance, export supervision

Key Findings​

  • Binary Status: 16MB CODI2 binary successfully compiled and ready for deployment
  • Critical Scripts: 17 scripts essential for production operations
  • Feature Gaps: 65% of functionality needs implementation for full parity
  • Priority Plan: Phase 1 focuses on critical infrastructure gaps

πŸ“š Production Deployment Documentation​

WebSocket & GKE Integration​

Production Deployment Guides​

Current Production URLs​

πŸ—„οΈ FoundationDB Cluster Configuration​

⚠️ CRITICAL INFRASTRUCTURE CHANGE (2025-09-30)

Decision: Deploy FoundationDB StatefulSet directly in GKE cluster

Why: After 5+ hours of troubleshooting authentication failures, root cause identified as FDB connectivity issue. External cluster at 10.0.1.3/4/5 unreachable from GKE pods, Internal Load Balancer at 10.128.0.8 does not exist. Rather than continue with networking workarounds, deploy FDB properly in GKE for immediate resolution.

Impact: All user authentication currently broken (502 errors). This blocks MVP launch.

Analysis Documents:

Deployment Plan: FDB-GKE-DEPLOYMENT-PLAN-2025-09-30.md (88 minutes to production)

Migration Plan: FDB-MODEL-MIGRATION-PLAN-2025-09-30.md (90 minutes to verify all models)

Timeline: Estimated 2-3 hours to fully functional authentication system

Rollback: Previous external cluster config documented below for reference only

⚠️ DEPRECATED: External Cluster Configuration (Pre-2025-09-30)​

Click to view deprecated configuration

Previous Cluster (UNREACHABLE):

  • Cluster String: coditect:production@10.0.1.3:4500,10.0.1.5:4500,10.0.1.4:4500
  • Network: fdb-network (VPC peering failed)
  • Nodes:
    • fdb-node-1: 10.0.1.3 (us-central1-a) - UNREACHABLE from GKE
    • fdb-node-2: 10.0.1.5 (us-central1-a) - UNREACHABLE from GKE
    • fdb-node-3: 10.0.1.4 (us-central1-a) - UNREACHABLE from GKE

Previous ILB Configuration (NEVER DEPLOYED):

  • ILB IP: 10.128.0.8 (does not exist)
  • Backend: fdb-backend (never created)
  • Status: Configuration existed in YAML only, never deployed to GCP

Why It Failed:

  • VPC peering between fdb-network (10.0.1.0/24) and GKE default network unsuccessful
  • GKE pods cannot reach external FDB nodes
  • Internal Load Balancer was planned but never created
  • Firewall rules ineffective without working VPC peering

Lessons Learned:

  • External FDB cluster adds unnecessary networking complexity
  • Cross-VPC connectivity unreliable for production databases
  • Co-locating FDB with consuming services (API pods) is correct architecture

βœ… NEW: GKE-Native FoundationDB Cluster (2025-09-30+)​

Deployment Status: 🟑 IN PROGRESS (Phase 1: StatefulSet deployed)

Architecture:

GKE Cluster (codi-poc-e2-cluster)
└── Namespace: coditect-app
β”œβ”€β”€ StatefulSet: foundationdb (3 replicas) βœ… DEPLOYED
β”‚ β”œβ”€β”€ foundationdb-0 (pod + 50Gi PV)
β”‚ β”œβ”€β”€ foundationdb-1 (pod + 50Gi PV)
β”‚ └── foundationdb-2 (pod + 50Gi PV)
β”œβ”€β”€ Service: fdb-cluster (headless) βœ… DEPLOYED
β”œβ”€β”€ Deployment: fdb-proxy (HAProxy, 2 replicas) ⏳ PENDING
└── Service: fdb-proxy-service (ILB at 10.128.0.8) ⏳ PENDING

Cluster Sizing:

  • Current: 3 nodes with double redundancy (MVP)
  • Redundancy Mode: double - Replicates data to 2 machines, tolerates 1 failure
  • Storage: 3 Γ— 50GB PersistentVolumes (150GB total)
  • Memory: 3 Γ— 2-4GB RAM per node

Dynamic Scaling (Zero Downtime):

# Scale to 5 nodes for higher availability
kubectl scale statefulset foundationdb --replicas=5 -n coditect-app

# Upgrade to triple redundancy (survives 2 failures)
kubectl exec -n coditect-app foundationdb-0 -- fdbcli --exec "configure triple"

# No application downtime required
Redundancy ModeNodesReplicationToleratesUse Case
single1-2None0 failuresTesting only ❌
double3-42 copies1 failureMVP (Current) βœ…
triple5+3 copies2 failuresHigh availability πŸš€

Connection Details (After Deployment):

# Cluster file location
/app/fdb.cluster
/etc/foundationdb/fdb.cluster

# Cluster string
coditect:production@10.128.0.8:4500

# HAProxy routes to:
# - foundationdb-0.fdb-cluster.coditect-app.svc.cluster.local:4500
# - foundationdb-1.fdb-cluster.coditect-app.svc.cluster.local:4500
# - foundationdb-2.fdb-cluster.coditect-app.svc.cluster.local:4500

Connectivity Testing:

# From API pod
POD=$(kubectl get pods -n coditect-app --selector=app=coditect-api-v2 -o jsonpath='{.items[0].metadata.name}')

# Test TCP connection
kubectl exec -n coditect-app $POD -- timeout 5 bash -c \
"cat < /dev/null > /dev/tcp/10.128.0.8/4500" && \
echo "βœ… FDB reachable" || echo "❌ FDB UNREACHABLE"

# Test FDB cluster status
kubectl exec -n coditect-app foundationdb-0 -- fdbcli --exec "status"

Deployment Commands:

# Deploy FDB cluster (see full plan for details)
cd /home/halcasteel/v4/k8s
kubectl apply -f fdb-storage.yaml
kubectl apply -f fdb-statefulset.yaml
kubectl apply -f fdb-proxy-deployment.yaml

# Initialize cluster
kubectl exec -n coditect-app foundationdb-0 -- fdbcli --exec "configure double"

# Verify
kubectl get statefulset,pods,svc -n coditect-app -l app=foundationdb

Data Models:

  • Implemented: User, Tenant, License, workspace (MVP)
  • Documented: 20 models in /docs/reference/database-models/
  • ADRs: ADR-006 (data model), ADR-029 (issue tracking)
  • See FDB-MODEL-MIGRATION-PLAN

Backup Strategy (Post-Deployment):

# Daily backups to Google Cloud Storage
gsutil mb gs://coditect-fdb-backups

# Backup script (CronJob)
kubectl apply -f k8s/fdb-backup-cronjob.yaml

☸️ Kubernetes (GKE) Management​

Cluster Information​

  • Cluster: codi-poc-e2-cluster
  • Zone: us-central1-a
  • Namespace: coditect-app
  • Load Balancer IP: 34.46.212.40
  • Static IP: 34.8.51.57 (for coditect.ai)

Quick Start Commands​

Set Context & Project​

# Set GCP project
gcloud config set project serene-voltage-464305-n2

# Get cluster credentials
gcloud container clusters get-credentials codi-poc-e2-cluster --zone us-central1-a

# Verify context
kubectl config current-context

# Set default namespace
kubectl config set-context --current --namespace=coditect-app

Deployment Management​

View Deployments​

# List all deployments
kubectl get deployments -n coditect-app

# Detailed deployment info
kubectl describe deployment coditect-frontend -n coditect-app
kubectl describe deployment coditect-api-v2 -n coditect-app

# Watch deployment status
kubectl get deployments -n coditect-app -w

Update Deployments​

# Update frontend with new image
kubectl set image deployment/coditect-frontend \
frontend=gcr.io/serene-voltage-464305-n2/coditect-frontend:latest \
-n coditect-app

# Update API with new image
kubectl set image deployment/coditect-api-v2 \
api=gcr.io/serene-voltage-464305-n2/coditect-api-v2:latest \
-n coditect-app

# Watch rollout progress
kubectl rollout status deployment/coditect-frontend -n coditect-app

# Check rollout history
kubectl rollout history deployment/coditect-frontend -n coditect-app

Rollback Deployments​

# Rollback to previous version
kubectl rollout undo deployment/coditect-frontend -n coditect-app

# Rollback to specific revision
kubectl rollout undo deployment/coditect-frontend --to-revision=3 -n coditect-app

# Check rollout status
kubectl rollout status deployment/coditect-frontend -n coditect-app

Scale Deployments​

# Manual scaling
kubectl scale deployment/coditect-frontend --replicas=3 -n coditect-app
kubectl scale deployment/coditect-api-v2 --replicas=5 -n coditect-app

# Check HPA (Horizontal Pod Autoscaler)
kubectl get hpa -n coditect-app

# View HPA details
kubectl describe hpa coditect-frontend-hpa -n coditect-app
kubectl describe hpa coditect-api-v2-hpa -n coditect-app

Pod Management​

View Pods​

# List all pods
kubectl get pods -n coditect-app

# List pods with more details
kubectl get pods -n coditect-app -o wide

# Filter by label
kubectl get pods -n coditect-app -l app=coditect-frontend
kubectl get pods -n coditect-app -l app=coditect-api-v2

# Watch pods
kubectl get pods -n coditect-app -w

Pod Details & Logs​

# Describe pod (events, status, conditions)
kubectl describe pod <pod-name> -n coditect-app

# View pod logs
kubectl logs <pod-name> -n coditect-app

# Follow logs in real-time
kubectl logs -f <pod-name> -n coditect-app

# View logs from previous container (after crash)
kubectl logs <pod-name> -n coditect-app --previous

# View logs from specific container in multi-container pod
kubectl logs <pod-name> -c <container-name> -n coditect-app

# Tail last 100 lines
kubectl logs <pod-name> -n coditect-app --tail=100

# Logs from all pods with label
kubectl logs -l app=coditect-frontend -n coditect-app --tail=50

Execute Commands in Pods​

# Interactive shell
kubectl exec -it <pod-name> -n coditect-app -- /bin/sh
kubectl exec -it <pod-name> -n coditect-app -- /bin/bash

# Run single command
kubectl exec <pod-name> -n coditect-app -- ls -la /app
kubectl exec <pod-name> -n coditect-app -- cat /etc/nginx/nginx.conf

# Check environment variables
kubectl exec <pod-name> -n coditect-app -- env

Restart Pods​

# Delete pod (will be recreated by deployment)
kubectl delete pod <pod-name> -n coditect-app

# Restart deployment (recreates all pods)
kubectl rollout restart deployment/coditect-frontend -n coditect-app
kubectl rollout restart deployment/coditect-api-v2 -n coditect-app

Service & Networking​

View Services​

# List all services
kubectl get services -n coditect-app
kubectl get svc -n coditect-app

# Describe service
kubectl describe svc coditect-frontend -n coditect-app
kubectl describe svc api-loadbalancer -n coditect-app

# Get service endpoints
kubectl get endpoints -n coditect-app

Ingress Management​

# View ingress
kubectl get ingress -n coditect-app

# Describe ingress (shows backend health)
kubectl describe ingress coditect-production-ingress -n coditect-app

# Get ingress YAML
kubectl get ingress coditect-production-ingress -n coditect-app -o yaml

SSL Certificates​

# List managed certificates
kubectl get managedcertificate -n coditect-app

# Check certificate status (shows Active/Provisioning/FailedNotVisible per domain)
kubectl describe managedcertificate coditect-ai-ssl -n coditect-app

# Watch certificate provisioning (updates every 30 seconds)
watch -n 30 'kubectl describe managedcertificate coditect-ai-ssl -n coditect-app | grep -A5 Status'

# Test HTTPS access
curl -I https://coditect.ai
curl -I https://api.coditect.ai
curl -I https://www.coditect.ai

# Verify SSL certificate details (expiration, subject)
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -subject

# Check DNS resolution
dig coditect.ai +short
dig api.coditect.ai +short
dig www.coditect.ai +short

ConfigMaps & Secrets​

ConfigMaps​

# List configmaps
kubectl get configmap -n coditect-app

# View configmap
kubectl describe configmap fdb-cluster-config -n coditect-app
kubectl get configmap fdb-cluster-config -n coditect-app -o yaml

# Edit configmap
kubectl edit configmap fdb-cluster-config -n coditect-app

Secrets​

# List secrets
kubectl get secrets -n coditect-app

# View secret (base64 encoded)
kubectl get secret <secret-name> -n coditect-app -o yaml

# Decode secret value
kubectl get secret <secret-name> -n coditect-app -o jsonpath='{.data.password}' | base64 -d

Troubleshooting​

Check Cluster Health​

# Node status
kubectl get nodes
kubectl describe node <node-name>

# Cluster info
kubectl cluster-info

# Component status
kubectl get componentstatuses

# Resource usage
kubectl top nodes
kubectl top pods -n coditect-app

Debug Common Issues​

# Pod stuck in Pending
kubectl describe pod <pod-name> -n coditect-app | grep -A10 Events

# Pod CrashLoopBackOff
kubectl logs <pod-name> -n coditect-app --previous
kubectl describe pod <pod-name> -n coditect-app

# ImagePullBackOff
kubectl describe pod <pod-name> -n coditect-app | grep -A5 "Failed to pull image"

# Service not accessible
kubectl get endpoints <service-name> -n coditect-app
kubectl describe svc <service-name> -n coditect-app

# Check pod network connectivity
kubectl exec <pod-name> -n coditect-app -- wget -O- http://coditect-frontend
kubectl exec <pod-name> -n coditect-app -- nslookup coditect-frontend

# Check external connectivity
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl http://coditect-frontend.coditect-app.svc.cluster.local

Network Analysis & Troubleshooting​

# DNS Resolution Testing
dig coditect.ai +short
dig api.coditect.ai +short
dig www.coditect.ai +short
nslookup coditect.ai
host coditect.ai

# SSL/TLS Certificate Analysis
# Quick SSL check
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer

# Detailed certificate information
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | openssl x509 -noout -text

# Certificate chain verification
echo | openssl s_client -connect coditect.ai:443 -servername coditect.ai 2>&1 | grep -A10 "Certificate chain"

# Test HTTPS with curl (includes SSL verification)
curl -vI https://coditect.ai 2>&1 | grep -E "SSL|TLS|certificate"

# HTTP/HTTPS Response Testing
# Test with status code and timing
curl -s -o /dev/null -w "Status: %{http_code}, SSL: %{ssl_verify_result}, Time: %{time_total}s\n" https://coditect.ai

# Test all domains
for domain in coditect.ai api.coditect.ai www.coditect.ai; do
echo "Testing https://$domain"
curl -s -o /dev/null -w "Status: %{http_code}, Time: %{time_total}s\n" "https://$domain"
done

# Ingress Backend Health
# Check if ingress backends are healthy
kubectl describe ingress coditect-production-ingress -n coditect-app | grep -A20 "Annotations"

# Check backend services
kubectl get svc -n coditect-app -o wide

# Test load balancer directly
curl -I http://34.46.212.40
curl -I http://34.46.212.40/api/v2/health

# Network Policy Troubleshooting
# Check network policies (if any)
kubectl get networkpolicies -n coditect-app

# Test pod-to-pod connectivity
kubectl exec -it <pod-1> -n coditect-app -- ping <pod-2-ip>
kubectl exec -it <pod-1> -n coditect-app -- curl http://<service-name>

# Trace network path (requires traceroute)
kubectl exec -it <pod-name> -n coditect-app -- traceroute coditect.ai

# Check firewall rules (GCP level)
gcloud compute firewall-rules list --filter="name~coditect"

# Network Performance Testing
# Test latency to load balancer
ping -c 5 34.8.51.57

# Test DNS resolution speed
time dig coditect.ai

# HTTP request timing breakdown
curl -w "@-" -o /dev/null -s https://coditect.ai <<'EOF'
time_namelookup: %{time_namelookup}s\n
time_connect: %{time_connect}s\n
time_appconnect: %{time_appconnect}s\n
time_pretransfer: %{time_pretransfer}s\n
time_redirect: %{time_redirect}s\n
time_starttransfer: %{time_starttransfer}s\n
----------\n
time_total: %{time_total}s\n
EOF

# GKE Network Troubleshooting
# Check node network
kubectl get nodes -o wide

# Check pod network (CNI)
kubectl get pods -n kube-system | grep -E "calico|cilium|flannel"

# Describe node to see network config
kubectl describe node <node-name> | grep -A5 "Network"

# Check service endpoints are assigned
kubectl get endpoints -n coditect-app

# Verify ingress has IP assigned
kubectl get ingress -n coditect-app -o wide

See Also: SSL Certificate Verification Report for complete SSL/TLS analysis.

View Events​

# All events in namespace
kubectl get events -n coditect-app --sort-by='.lastTimestamp'

# Watch events
kubectl get events -n coditect-app -w

# Events for specific resource
kubectl describe pod <pod-name> -n coditect-app | grep -A20 Events

Resource Management​

View Resources​

# All resources in namespace
kubectl get all -n coditect-app

# Specific resource types
kubectl get pods,svc,deploy,ing -n coditect-app

# Resource quotas
kubectl get resourcequota -n coditect-app

# Limit ranges
kubectl get limitrange -n coditect-app

Apply/Update Resources​

# Apply YAML file
kubectl apply -f deployment.yaml -n coditect-app

# Apply directory of YAMLs
kubectl apply -f k8s/production/ -n coditect-app

# Delete resources
kubectl delete -f deployment.yaml -n coditect-app
kubectl delete deployment coditect-frontend -n coditect-app

# Replace resource (destructive)
kubectl replace -f deployment.yaml -n coditect-app

Monitoring & Metrics​

Resource Usage​

# Node metrics
kubectl top nodes

# Pod metrics
kubectl top pods -n coditect-app

# Sort by CPU
kubectl top pods -n coditect-app --sort-by=cpu

# Sort by memory
kubectl top pods -n coditect-app --sort-by=memory

Port Forwarding (Local Testing)​

# Forward pod port to local
kubectl port-forward pod/<pod-name> 8080:8080 -n coditect-app

# Forward service port to local
kubectl port-forward svc/coditect-frontend 8080:80 -n coditect-app

# Access in browser: http://localhost:8080

Production Deployment Workflow​

Complete Deployment Process​

# 1. Build and push new image
cd /home/halcasteel/v4/src/frontend
gcloud builds submit --config=cloudbuild.yaml --project=serene-voltage-464305-n2

# 2. Update GKE deployment
kubectl set image deployment/coditect-frontend \
frontend=gcr.io/serene-voltage-464305-n2/coditect-frontend:latest \
-n coditect-app

# 3. Watch rollout
kubectl rollout status deployment/coditect-frontend -n coditect-app

# 4. Verify pods are running
kubectl get pods -n coditect-app -l app=coditect-frontend

# 5. Check logs for errors
kubectl logs -l app=coditect-frontend -n coditect-app --tail=50

# 6. Test service (load balancer)
curl http://34.46.212.40/docs/api

# 7. Test HTTPS endpoints (production domain)
curl -I https://coditect.ai
curl https://coditect.ai/docs/api

Production Testing & Verification​

# Test all HTTPS endpoints with SSL verification and timing
for domain in coditect.ai api.coditect.ai www.coditect.ai; do
echo "Testing https://$domain"
curl -s -o /dev/null -w "Status: %{http_code}, SSL: %{ssl_verify_result}, Time: %{time_total}s\n" "https://$domain"
done

# Test all documentation pages
for page in api websocket installation architecture auth security; do
echo -n "https://coditect.ai/docs/$page: "
curl -s -o /dev/null -w "%{http_code}\n" "https://coditect.ai/docs/$page"
done

# Comprehensive health check
echo "=== Frontend Health ==="
curl -s https://coditect.ai | head -20

echo "=== API Health ==="
curl -s https://api.coditect.ai/api/v2/health | jq '.'

# Check SSL certificate expiration
echo "=== SSL Certificate ==="
openssl s_client -connect coditect.ai:443 -servername coditect.ai </dev/null 2>/dev/null | \
openssl x509 -noout -dates -subject

# Load test (optional - requires hey tool)
# hey -n 100 -c 10 https://coditect.ai/api/v2/health

Emergency Rollback​

# Quick rollback
kubectl rollout undo deployment/coditect-frontend -n coditect-app

# Verify rollback
kubectl rollout status deployment/coditect-frontend -n coditect-app

# Check pods
kubectl get pods -n coditect-app -l app=coditect-frontend

Common CODITECT Operations​

Update Frontend Documentation​

# After building new frontend image with docs
kubectl set image deployment/coditect-frontend \
frontend=gcr.io/serene-voltage-464305-n2/coditect-frontend:latest \
-n coditect-app

# Wait for rollout
kubectl rollout status deployment/coditect-frontend -n coditect-app

# Verify new pods
kubectl get pods -n coditect-app -l app=coditect-frontend -o wide

Update API Server​

# Deploy new API version
kubectl set image deployment/coditect-api-v2 \
api=gcr.io/serene-voltage-464305-n2/coditect-api-v2:latest \
-n coditect-app

# Monitor rollout
kubectl rollout status deployment/coditect-api-v2 -n coditect-app

# Check API health
kubectl exec -it <api-pod-name> -n coditect-app -- curl localhost:8080/api/v2/health

Check FoundationDB Connectivity​

# Test from API pod
kubectl exec -it <api-pod-name> -n coditect-app -- sh
# Inside pod:
cat /etc/foundationdb/fdb.cluster
# Should show: coditect:production@10.128.0.8:4500

Quick Reference Summary​

TaskCommand
List deploymentskubectl get deploy -n coditect-app
Update deploymentkubectl set image deployment/<name> <container>=<image> -n coditect-app
Rollout statuskubectl rollout status deployment/<name> -n coditect-app
View podskubectl get pods -n coditect-app
Pod logskubectl logs <pod-name> -n coditect-app
Exec into podkubectl exec -it <pod-name> -n coditect-app -- sh
Restart deploymentkubectl rollout restart deployment/<name> -n coditect-app
Rollback deploymentkubectl rollout undo deployment/<name> -n coditect-app
View serviceskubectl get svc -n coditect-app
View ingresskubectl get ingress -n coditect-app
Check certificateskubectl get managedcertificate -n coditect-app
View eventskubectl get events -n coditect-app --sort-by='.lastTimestamp'
Resource usagekubectl top pods -n coditect-app

🐳 Containerization & Pod Management​

User Pod Architecture​

Each user gets an isolated development pod with:

  • Base Image: Custom container with development tools
  • Persistent Storage: User workspace mounted as PVC
  • CODI2 Integration: Local monitoring and logging
  • Network: Isolated namespace with controlled egress

Container Registry​

# Google Container Registry
gcr.io/serene-voltage-464305-n2/
β”œβ”€β”€ coditect-api-v2:latest # API server
β”œβ”€β”€ coditect-frontend:latest # React frontend
β”œβ”€β”€ coditect-websocket:latest # WebSocket gateway
β”œβ”€β”€ coditect-user-pod:latest # User development pod
└── codi2:latest # CODI2 monitoring service

Pod Deployment Process​

# 1. User registers/logs in
# 2. API creates workspace in FDB
# 3. Kubernetes creates user pod:
kubectl create namespace user-${USER_ID}
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: workspace
namespace: user-${USER_ID}
spec:
containers:
- name: workspace
image: gcr.io/serene-voltage-464305-n2/coditect-user-pod:latest
env:
- name: USER_ID
value: "${USER_ID}"
- name: CODI2_SERVER
value: "codi2-service.coditect-app:8765"
volumeMounts:
- name: workspace
mountPath: /workspace
- name: codi2-config
mountPath: /workspace/.codi2
volumes:
- name: workspace
persistentVolumeClaim:
claimName: user-workspace
- name: codi2-config
configMap:
name: codi2-config
EOF

CODI2 Integration​

  • Local CODI2: Runs in user pod for file monitoring
  • Server CODI2: Central service for log aggregation
  • Communication: WebSocket connection for real-time sync
  • Storage: Logs persisted to FoundationDB

Authentication Flow​

  1. User logs in via frontend
  2. API validates JWT and creates session
  3. WebSocket gateway authenticates with same JWT
  4. User pod created with session credentials
  5. CODI2 tracks all activity with session ID

🚨 Directory Boundary​

STAY WITHIN v4/ - Do not navigate outside this directory.

πŸ“‚ Directory Organization​

For detailed information about where files belong and directory structure guidelines, see: DIRECTORY-ORGANIZATION-GUIDE.md

πŸ“ Directory Organization (Updated 2025-09-21)​

Core Structure​

v4/
β”œβ”€β”€ src/ # Source code (API, frontend, websocket, scripts)
β”œβ”€β”€ agents/ # Agent system (10 CODITECT + 11 Claude sub-agents)
β”œβ”€β”€ docs/ # Diamond core documentation (214 essential files)
β”œβ”€β”€ standards/ # Foundation standards (logging, testing, errors)
β”œβ”€β”€ .codi/ # Monitoring infrastructure
β”œβ”€β”€ codi2/ # CODI2 development (metrics engine)
β”œβ”€β”€ tests/ # Test suites
β”œβ”€β”€ config/ # Configuration files
β”œβ”€β”€ infrastructure/ # Deployment configs
└── .archived/ # Historical files (1,874+ items)

πŸš€ Session Instantiation Protocol​

Step 1: Set GCP Project (MANDATORY)​

# ALWAYS set the correct GCP project first
gcloud config set project serene-voltage-464305-n2
# Project name: Google-GCP-CLI
# Project number: 1059494892139

# Set billing quota project (ensures consistent billing across all projects)
gcloud config set billing/quota_project serene-voltage-464305-n2

# Verify settings
gcloud config list | grep -E "project|quota"

Step 2: Initialize Infrastructure (MANDATORY)​

# Navigate to repository root
cd /home/hal/CODITECTv4 # or cd /home/halcasteel/v4

# Start all monitoring infrastructure
source .codi/scripts/infrastructure/session-start-hook.sh

# This automatically:
# βœ“ Starts file monitor (tracks all file operations)
# βœ“ Starts export watcher (archives conversation exports)
# βœ“ Starts API server (web tools on port 8080)
# βœ“ Sets up logging to .codi/logs/codi-ps.log
# βœ“ Detects AI environment (Claude Code, CODITECT, etc.)

Step 3: Set Session Identity (REQUIRED FOR AGENTS)​

# For ORCHESTRATOR sessions:
export SESSION_ID="ORCHESTRATOR-SESSION-$(date +%Y-%m-%d)-01"

# For specialist agents:
export SESSION_ID="[AGENT-TYPE]-SESSION-$(date +%Y-%m-%d)-[SEQUENCE]"
# Examples:
# export SESSION_ID="RUST-DEVELOPER-SESSION-2025-09-23-01"
# export SESSION_ID="QA-REVIEWER-SESSION-2025-09-23-02"

# IMPORTANT: Initialize the session identity (sets actor attribution)
source /home/halcasteel/v4/.codi/scripts/session-management/identity/set-session-identity.sh "$SESSION_ID"
# This ensures all codi-log commands are properly attributed to your session

Step 4: Log Session Start (REQUIRED)​

# Log the session initialization
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Starting work on [TASK_DESCRIPTION]" "SESSION_START"

# For ORCHESTRATOR, include role announcement:
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Ready for task distribution and agent coordination" "ORCHESTRATION"

Step 5: Create Project Directory (FOR NEW TASKS)​

# Generate project directory with ISO date and UUID
PROJECT_NAME="$(date +%Y%m%d-%H%M%S)-[TASK_NAME]-$(uuidgen | cut -c1-8)"
mkdir -p "projects/$PROJECT_NAME"
cd "projects/$PROJECT_NAME"

# Example:
# PROJECT_NAME="20250923-120530-SubAgentSprint-a3f2b1c9"
# Creates: projects/20250923-120530-SubAgentSprint-a3f2b1c9/

Step 5: Create Task Checklist (MANDATORY FOR ORCHESTRATOR)​

# Create task checklist with checkboxes
TASK_LIST="TASK-LIST-$(date +%Y%m%d-%H%M%S).md"
cat > "$TASK_LIST" << 'EOF'
# Task List: [PROJECT_NAME]
**Session**: $SESSION_ID
**Created**: $(date +"%Y-%m-%d %H:%M:%S")
**Status**: IN_PROGRESS

## Primary Objectives
- [ ] [Objective 1]
- [ ] [Objective 2]
- [ ] [Objective 3]

## Implementation Tasks
- [ ] Task 1: Description
- [ ] Task 2: Description
- [ ] Task 3: Description

## Quality Gates
- [ ] ADR Compliance (40/40)
- [ ] Test Coverage (β‰₯95%)
- [ ] Integration Tests Pass
- [ ] Documentation Complete
EOF

# Log task list creation
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Created task list $TASK_LIST" "CREATE"

Step 6: Verify Initialization​

# Check all services are running
ps aux | grep -E "(file-monitor|export-watcher|api-server)" | grep -v grep

# Verify session is logged
grep "$SESSION_ID" .codi/logs/codi-ps.log | tail -5

# Access web tools
echo "Web tools available at:"
echo " - Hub: http://localhost:8080"
echo " - Logs: http://localhost:8080/log-viewer.html"
echo " - Dashboard: http://localhost:8080/agent-dashboard.html"

πŸ“ Logging System Documentation​

Log Format (JSON)​

All logs are stored as newline-delimited JSON in .codi/logs/codi-ps.log:

{
"timestamp": "2025-09-23T21:08:22Z",
"level": "INFO",
"component": "ai.activity",
"action": "TEST",
"message": "Testing feature implementation",
"sessionId": "ORCHESTRATOR-SESSION-2025-09-23-02",
"actor": {
"type": "agent",
"id": "ORCHESTRATOR-SESSION-2025-09-23-02"
},
"details": {
"ai_id": "ORCHESTRATOR-SESSION-2025-09-23-02",
"ai_role": "agent",
"ai_details": "Multi-agent orchestration session",
"working_directory": "/home/halcasteel/v4"
}
}

Logging Commands​

# βœ… CORRECT - Session attribution formats:

# Method 1: Include session ID in message (RECOMMENDED)
./.codi/scripts/logging/actors/codi-log-ai.sh "ORCHESTRATOR-SESSION-2025-09-23-02: Implementing user API" "CREATE"

# Method 2: Set SESSION_ID environment variable
export SESSION_ID="ORCHESTRATOR-SESSION-2025-09-23-02"
./.codi/scripts/logging/actors/codi-log-ai.sh "Implementing user API" "CREATE"

# ❌ INCORRECT - Broken formats:
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: message" "ACTION" # Variable not expanded
echo '{"message": "test"}' >> .codi/logs/codi-ps.log # Breaks JSON format

Session ID Format​

Session IDs follow strict naming conventions for proper attribution:

PatternExamplePurpose
AGENT-TYPE-SESSION-YYYY-MM-DD-NNORCHESTRATOR-SESSION-2025-09-23-01Date-based with sequence
AGENT-TYPE-SESSION-NRUST-DEVELOPER-SESSION-7Simple numbering
AGENT-TYPE-SESSION-YYYYMMDD-HHMMQA-REVIEWER-SESSION-20250923-1430Time-specific

Action Types​

ActionUse CaseExample
SESSION_STARTBeginning work"Starting orchestration session"
CREATENew files/resources"Created user API endpoint"
UPDATEModifying existing"Updated authentication logic"
FIXBug fixes"Fixed null pointer in auth"
TESTRunning tests"Executed integration tests"
ANALYZECode analysis"Reviewing ADR compliance"
COORDINATEMulti-agent work"Assigning tasks to agents"
HANDOFFTask completion"Handing off to QA review"
FILE_CLAIMClaiming ownership"Claiming src/api/users.rs"
FILE_RELEASEReleasing files"Releasing src/api/users.rs"
ORCHESTRATIONOrchestrator tasks"Distributing work to agents"
MILESTONEMajor achievements"Completed API v2 migration"

Log File Conventions​

# Primary log file (all sessions)
.codi/logs/codi-ps.log

# Session-specific logs (optional)
.codi/logs/session-ORCHESTRATOR-SESSION-2025-09-23-01.log
.codi/logs/session-RUST-DEVELOPER-SESSION-7.log

# Monitor output
.codi/logs/file-monitor.out
.codi/logs/export-watcher.out

# Archived logs
.codi/logs/archive/codi-ps-20250923-backup.log

Path Details in Logs​

Include sufficient path context without redundancy:

# βœ… GOOD - Specific and actionable
"ORCHESTRATOR-SESSION-2025-09-23-02: Created src/api-v2/handlers/user.rs"
"RUST-DEVELOPER-SESSION-7: Updated tests/integration/auth_test.rs line 142"
"QA-REVIEWER-SESSION-3: Reviewing docs/api/v2/endpoints.md for completeness"

# ❌ BAD - Too vague or too verbose
"Updated a file" # Which file?
"Modified /home/halcasteel/v4/src/api-v2/handlers/auth/jwt/token/validation/expiry.rs line 2341 character 42" # Too detailed

Log Management Scripts​

ScriptPurposeUsage
codi-log-ai.shAI session logging./.codi/scripts/logging/actors/codi-log-ai.sh "message" "ACTION"
codi-log-human.shHuman developer logs./.codi/scripts/logging/actors/codi-log-human.sh "message" "ACTION"
codi-log.shGeneral logging./.codi/scripts/logging/codi-log.sh "message" "ACTION"
aggregate-logs.shCombine session logs./.codi/scripts/aggregate-logs.sh
sync-logs.shSync between sessions./.codi/scripts/sync-logs.sh
split-logs-by-session.shSplit by session./.codi/scripts/split-logs-by-session.sh

Viewing Logs​

# Real-time monitoring
tail -f .codi/logs/codi-ps.log | jq '.'

# Filter by session
grep "ORCHESTRATOR-SESSION-2025-09-23-02" .codi/logs/codi-ps.log | jq '.'

# Filter by action
jq 'select(.action == "CREATE")' .codi/logs/codi-ps.log

# Today's activity
grep "$(date +%Y-%m-%d)" .codi/logs/codi-ps.log | jq '.'

# Web interface
open http://localhost:8080/log-viewer.html

Best Practices​

  1. Always include session ID in logs for proper attribution
  2. Use appropriate action types for better filtering
  3. Include file paths when relevant (src/api/file.rs)
  4. Keep messages concise but informative
  5. Never write directly to log files - use scripts
  6. Backup logs before Git operations (see Git Playbook)

🎯 Session Types​

Session PatternUse For
ORCHESTRATOR-SESSION-YYYY-MM-DD-NNMulti-agent coordination (NN = sequence)
RUST-DEVELOPER-SESSION-NRust backend
QA-REVIEWER-SESSION-NDocumentation review
CLOUD-ARCHITECT-SESSION-NGCP/CI/CD
FRONTEND-EXPERT-SESSION-NReact/TypeScript
GENERAL-SESSION-NMixed tasks

Note: When multiple sessions occur on same day, add sequence number (01, 02) or time (HHMM) to differentiate

πŸ“¦ Monitoring Infrastructure​

What Gets Monitored: All file operations in /home/halcasteel/v4/*

  • βœ“ Creates, updates, deletes, moves
  • βœ“ All subdirectories (including deep nesting)
  • βœ“ Logged as JSON to .codi/logs/codi-ps.log
  • βœ“ Console output to .codi/logs/file-monitor.out

Recent Changes (2025-09-21):

  • Consolidated 5 file monitors β†’ 1 working script
  • Fixed JSON logging (was broken in "file-monitor-fixed.sh")
  • Updated session-start-hook.sh to use consolidated monitor

πŸ› οΈ Common Commands​

# Check status
ps aux | grep -E "(file-monitor-consolidated|export-watcher|api-server)" | grep -v grep

# View logs (stored in .codi/logs/codi-ps.log)
tail -f .codi/logs/codi-ps.log | jq '.'

# See active sessions
grep "SESSION_START" .codi/logs/codi-ps.log | tail -10

# Stop everything
pkill -f "file-monitor|export-watcher|api-server"

# Check essential scripts
cat .codi/scripts/ESSENTIAL-SCRIPTS.md

🌐 Web Tools (http://localhost:8080)​

πŸ€– Agent System​

Available Agents​

CODITECT Agents (Primary - Strategic decisions):

  1. orchestrator - Multi-agent coordination, task distribution
  2. rust-developer - Rust backend, API implementation
  3. frontend-developer - React/TypeScript UI development
  4. database-specialist - FoundationDB schemas, multi-tenant patterns
  5. cloud-architect - GCP infrastructure, CI/CD
  6. security-specialist - Authentication, hardening
  7. testing-specialist - Test coverage (95%), TDD
  8. monitoring-specialist - Observability, metrics
  9. qa-reviewer - Documentation, ADR compliance
  10. ai-specialist - AI provider integration

Claude Sub-Agents (Tactical - Deep analysis): Available for delegation when CODITECT agents need specialized expertise.

🎯 ORCHESTRATOR First Steps​

Creating New ORCHESTRATOR Sessions​

To start a new ORCHESTRATOR session in Claude Code:

  1. Copy the First Instruction:

    cat docs/ORCHESTRATOR-FIRST-INSTRUCTION.md

    Copy everything between the "---" markers

  2. Paste into new Claude Code session as the very first message

  3. The ORCHESTRATOR will then:

    • Read all required documents
    • Initialize infrastructure
    • Set session identity
    • Confirm readiness

Documents ORCHESTRATOR Must Read​

When instantiated as ORCHESTRATOR, ALWAYS read these documents in order:

  1. README.md - Project overview and links to all resources
  2. agents/coditect/orchestrator.md - Your complete role definition
  3. docs/CODITECT-GIT-PLAYBOOK.md - Git management strategies
  4. docs/LOG-MANAGEMENT-SCRIPT-GUIDE.md - Log synchronization

Orchestrator Key Responsibilities​

  1. Project Setup - Create standardized project directories

    ./.codi/scripts/create-project-directory.sh "Task Name"
  2. Task Management - Maintain task checklists with checkboxes

    • Location: projects/YYYYMMDD-HHMMSS-TaskName-UUID/TASK-LIST-*.md
    • Track: Objectives, assignments, quality gates, coordination
  3. Multi-Session Coordination - Prevent conflicts between agents

    • Monitor: Active sessions via CODI logs
    • Enforce: File claiming protocol
    • Coordinate: Handoffs and integration
  4. Quality Enforcement - 40/40 ADR compliance on all deliverables

  • Project Management: See project structure in README.md
  • Git Workflows: docs/CODITECT-GIT-PLAYBOOK.md
  • Log Management: docs/LOG-MANAGEMENT-SCRIPT-GUIDE.md
  • Agent Guides: agents/coditect/*.md
  • Sprint Planning: agents/sub-agent-build-sprint-checklist.md

πŸš€ Agent Instantiation Sequence​

# 1. ALWAYS start infrastructure first
cd /home/hal/CODITECTv4
source .codi/scripts/infrastructure/session-start-hook.sh

# 2. Set your agent identity
export SESSION_ID="[AGENT-TYPE]-SESSION-$(date +%Y-%m-%d)-[SEQUENCE]"
# Example: export SESSION_ID="RUST-DEVELOPER-SESSION-2025-09-23-01"

# 3. Log session start with role details
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Starting work as [AGENT_ROLE]" "SESSION_START"

# 4. For ORCHESTRATOR: Create project structure
if [[ "$SESSION_ID" =~ ORCHESTRATOR ]]; then
PROJECT_DIR="projects/$(date +%Y%m%d-%H%M%S)-[TASK]-$(uuidgen | cut -c1-8)"
mkdir -p "$PROJECT_DIR"
cd "$PROJECT_DIR"

# Create task checklist
cat > "TASK-LIST-$(date +%Y%m%d).md" << 'EOF'
# Task Checklist
**Orchestrator**: $SESSION_ID
**Created**: $(date)

## Tasks
- [ ] Task 1
- [ ] Task 2
- [ ] Task 3
EOF
fi

# 5. Announce capabilities
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Ready. Capabilities: [LIST_CAPABILITIES]" "ORCHESTRATION"

πŸ”„ Multi-Session Coordination​

Prevent Collisions:

# Claim your files
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID claiming src/api/" "FILE_CLAIM"

# Check active sessions
grep "SESSION_START" .codi/logs/codi-ps.log | tail -10

# Coordinate shared work
./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Need coordination on shared file" "COORDINATE"

Complete Guide: See agents/multi-session-guide.md

πŸ—οΈ Agent Instantiation Protocol​

When starting as an agent, ALWAYS:

  1. Read your agent guide first:

    cat agents/coditect/[YOUR-AGENT].md
  2. Set your session ID:

    export SESSION_ID="[AGENT-TYPE]-SESSION-N"
    # Example: export SESSION_ID="RUST-DEVELOPER-SESSION-7"
  3. Log your session start:

    ./.codi/scripts/logging/actors/codi-log-ai.sh "$SESSION_ID: Starting work on [task]" "SESSION_START"
  4. Understand your boundaries:

    • File ownership: Respect file boundaries in your agent guide
    • Dependencies: Check which other agents you depend on
    • Quality standards: Follow ADR requirements and coverage targets
    • Integration points: Understand CODI logging and coordination
  5. Know when to delegate:

    • Complex analysis beyond your scope β†’ Use Claude sub-agents
    • Cross-agent coordination needed β†’ Escalate to orchestrator
    • Quality gates required β†’ Coordinate with qa-reviewer

πŸ”„ Delegation to Claude Sub-Agents​

CODITECT agents can delegate complex tasks to Claude specialists:

# Example: Rust developer delegating security review
Task(
description="Security audit of auth module",
prompt="Review auth.rs for OWASP compliance and ADR-024 hardening",
subagent_type="security-specialist"
)

Available Claude Sub-Agents: See agents/claude-subagents/README.md

πŸ“š Essential Reading for New Agents​

  • Your specific guide: agents/coditect/[AGENT-NAME].md
  • System overview: agents/README.md
  • Integration patterns: CODITECT-CLAUDE-AGENT-INTEGRATION.md
  • Migration info: agents/migration-guide.md

πŸŽ›οΈ Agent Dashboard​

Monitor all agent activity: http://localhost:8080/agent-dashboard.html

πŸ“‚ Moved Files Reference​

Scripts Organization​

All operational scripts have been organized from root into categorized directories:

Deployment Scripts (moved to scripts/deployment/)​

Testing Scripts (moved to scripts/testing/)​

Documentation Organization​

Documentation has been organized into appropriate subdirectories:

Guides (moved to docs/guides/)​

Implementation (moved to docs/implementation/)​

Architecture (moved to docs/architecture/)​

Workflows (moved to docs/workflows/)​

Testing (moved to docs/testing/)​

Integration Test Scripts (moved to tests/integration/)​

Archived Session Data​

Old session data moved to .session/archive/:

  • 2025-09-23-api-v2-frontend-testing/ - Previous testing session files