ADR-001-v4: Container Execution Architecture

User Story

"As a developer, I want to access a full Linux development environment through my browser with persistent terminal access and file operations, so that I can develop applications with the same tools and workflows I use locally while having my work automatically saved and synchronized."

Status

Status: Accepted
Date: 2025-08-27
Deciders: Architecture Team, Technical Lead
Supersedes: ADR-052-v3, ADR-054-v3, ADR-074-v3
Dependencies: None (foundational)

Context

Problem Statement

CODITECT needs to provide persistent development environments that:

Scale efficiently from 1 to millions of users
Provide full Linux terminal access with file operations
Maintain session state across browser reconnections
Support real-time development workflows
Leverage existing deployed infrastructure

Current State Analysis

Already Deployed and Working:

API service on Cloud Run at https://coditect-api-1059494892139.us-central1.run.app
WebSocket Gateway on GKE with HPA (3-20 replicas)
terminal bridge with PTY support (output working)
JWT authentication and multi-tenant workspace model
File operation handlers for basic CRUD

Current Issues:

PTY write error (File exists (os error 17)) blocking terminal input
WebSocket gateway on expensive GKE instead of cost-effective Cloud Run
AI memory system implemented but not responding
No frontend UI deployed

v1/v2 Lessons Learned

v1: Assumed persistent VMs per user (too expensive)
v2: Complex per-workspace container orchestration (over-engineered)
Current: Shared services with session isolation (working model)

Decision

Core Architecture

Maintain the existing shared service architecture while fixing critical issues and optimizing deployment costs.

System Components

WebSocket Gateway: Shared service handling all WebSocket connections
terminal Bridge: PTY process management within gateway containers
Session Manager: workspace isolation within shared processes
File Operation Handler: CRUD operations on workspace files
State Synchronizer: Persistent state management in FoundationDB

Implementation Details

Current Architecture (Working)

// From src/gateway/websocket_server.rs - EXISTING CODE
pub struct WebSocketGateway {
    connections: Arc<DashMap<String, Connection>>,
    container_service: Arc<AgentExecutionService>,
    terminal_bridge: Arc<terminalBridge>,
    file_handler: Arc<FileOperationHandler>,
    shutdown_tx: broadcast::Sender<()>,
    jwt_secret: String,
}

impl WebSocketGateway {
    pub fn new(
        container_service: Arc<AgentExecutionService>,
        jwt_secret: String,
    ) -> (Self, broadcast::Receiver<()>) {
        let (shutdown_tx, shutdown_rx) = broadcast::channel(1);
        
        let terminal_config = terminalConfig::default();
        let terminal_bridge = Arc::new(terminalBridge::new(terminal_config.clone()));
        let file_handler = Arc::new(FileOperationHandler::new(terminal_config.working_dir));
        
        Self {
            connections: Arc::new(DashMap::new()),
            container_service,
            terminal_bridge,
            file_handler,
            shutdown_tx,
            jwt_secret,
        }
    }
}

terminal Bridge (Needs PTY Fix)

// From src/gateway/terminal_bridge/bridge.rs - EXISTING WITH BUG
pub struct terminalBridge {
    sessions: Arc<DashMap<String, terminalSession>>,
    config: terminalConfig,
}

impl terminalBridge {
    // EXISTING: Creates PTY processes per workspace
    pub async fn create_session(
        &self,
        workspace_id: &str,
        cols: u16,
        rows: u16,
    ) -> Result<String> {
        let session_id = Uuid::new_v4().to_string();
        let pty_session = PtySession::new(cols, rows)?;
        
        // BUG FIX NEEDED: PTY write operation fails with "File exists"
        // Current error in src/gateway/terminal_bridge/pty.rs
        let session = terminalSession::new(session_id.clone(), pty_session);
        self.sessions.insert(session_id.clone(), session);
        
        Ok(session_id)
    }
    
    // ENHANCEMENT NEEDED: Add state persistence to FDB
    pub async fn persist_session_state(
        &self,
        workspace_id: &str,
        session_id: &str,
    ) -> Result<()> {
        // TODO: Save terminal history, environment vars to FDB
        // Pattern: {workspace_id}/terminal/{session_id}
        Ok(())
    }
}

Required Bug Fixes

1. PTY Write Error Fix

// In src/gateway/terminal_bridge/pty.rs
// CURRENT ISSUE: File exists (os error 17) on PTY write

impl PtyProcess {
    // NEEDS FIX: Write operation failing
    pub async fn write_input(&mut self, data: &[u8]) -> Result<()> {
        // CURRENT CODE (failing):
        // self.pty_master.write_all(data).await?;
        
        // PROPOSED FIX: Check PTY state before write
        if !self.is_pty_ready()? {
            self.reinitialize_pty().await?;
        }
        
        // Retry mechanism for PTY writes
        let mut attempts = 0;
        while attempts < 3 {
            match self.pty_master.write_all(data).await {
                Ok(_) => return Ok(()),
                Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {
                    attempts += 1;
                    tokio::time::sleep(Duration::from_millis(10)).await;
                    continue;
                }
                Err(e) => return Err(e.into()),
            }
        }
        
        Err(anyhow!("PTY write failed after 3 attempts"))
    }
}

2. WebSocket Gateway Migration (GKE → Cloud Run)

// DEPLOYMENT CHANGE NEEDED:
// Current: Kubernetes deployment on GKE
// Target: Cloud Run service with session affinity

// Cloud Run configuration needed:
// - Session affinity for WebSocket connections  
// - Horizontal scaling 0-100 instances
// - Memory: 2-4GB per instance
// - CPU: 1-2 vCPU per instance
// - Port: 8080 for WebSocket connections

FoundationDB Schema (Existing + Extensions)

// EXISTING: User and workspace models in src/models/
// From src/db/repositories/ - CURRENT PATTERNS WORKING

// Session state persistence (NEW - needed for recovery)
"{workspace_id}/terminal_sessions/{session_id}" -> terminalSessionState {
    id: String,
    workspace_id: UUID,
    started_at: DateTime,
    last_active: DateTime,
    shell_type: String, // bash, zsh, fish
    working_directory: String,
    environment_vars: HashMap<String, String>,
    command_history: Vec<String>, // Last 1000 commands
}

// terminal output buffer (NEW - for session recovery)
"{workspace_id}/terminal_output/{session_id}" -> terminalOutput {
    session_id: String,
    lines: Vec<terminalLine>, // Circular buffer, max 1000 lines
    cursor_position: (u16, u16),
    screen_size: (u16, u16),
}

// File operation tracking (EXTEND existing patterns)
"{workspace_id}/files/{file_path}" -> FileMetadata {
    path: String,
    size: u64,
    modified_at: DateTime,
    content_hash: String,
    storage_location: StorageLocation, // Local, GCS, GitHub
}

// WebSocket connection tracking (NEW)
"{workspace_id}/connections/{connection_id}" -> ConnectionState {
    connection_id: String,
    session_id: String,
    connected_at: DateTime,
    last_ping: DateTime,
    client_ip: String,
    user_agent: String,
}

WebSocket Message Protocol (Existing + Fixes)

// From src/gateway/message_router.rs - EXISTING STRUCTURE
#[derive(Serialize, Deserialize)]
pub enum terminalMessage {
    // EXISTING - working
    Input { data: String },
    Output { data: String },
    Resize { cols: u16, rows: u16 },
    
    // ENHANCEMENT NEEDED - session management
    CreateSession { workspace_id: String, cols: u16, rows: u16 },
    RestoreSession { workspace_id: String, session_id: String },
    SaveSession { workspace_id: String, session_id: String },
    
    // FIX NEEDED - proper error handling
    Error { code: ErrorCode, message: String },
}

#[derive(Serialize, Deserialize)]
pub enum FileOperationMessage {
    // EXISTING - partially working
    ReadFile { path: String },
    WriteFile { path: String, content: String },
    ListDirectory { path: String },
    CreateDirectory { path: String },
    DeleteFile { path: String },
    
    // ENHANCEMENT - file watching
    WatchFile { path: String },
    FileChanged { path: String, change_type: ChangeType },
}

#[derive(Serialize, Deserialize)]
pub enum ErrorCode {
    PtyWriteError,
    SessionNotFound,
    FileNotFound,
    PermissionDenied,
    ConnectionLost,
}

Migration Path

Phase 1: Fix Critical Bugs (Week 1)

Fix PTY Write Error: Debug and resolve "File exists (os error 17)"
Test terminal Input: Verify keystrokes reach PTY processes
Add Error Recovery: Implement PTY reconnection logic
Session Persistence: Save terminal state to FDB

Phase 2: Cost Optimization (Week 2)

Migrate to Cloud Run: Move WebSocket gateway from GKE
Configure Session Affinity: Maintain WebSocket connections
Optimize Scaling: Set appropriate min/max instances
Monitor Costs: Compare GKE vs Cloud Run expenses

Phase 3: Feature Completion (Week 3)

Deploy Frontend: Basic terminal and file browser UI
Fix AI Memory: Resolve context/assistance endpoints
Add File Watching: Real-time file change notifications
Session Recovery: Restore terminal state after disconnection

Security Considerations

Multi-Tenant Isolation (Already Implemented)

JWT authentication validates workspace access
Connection mapping isolates user sessions
File operations scoped to workspace directories
terminal sessions isolated by workspace ID

Enhancements Needed

terminal command auditing to FDB audit logs
File operation permissions beyond basic workspace isolation
Rate limiting on terminal input to prevent abuse
Secure cleanup of terminal history containing sensitive data

Performance & Scalability

Current Performance (Measured)

WebSocket connection setup: ~200ms (working)
terminal output streaming: ~50ms latency (working)
File operations: ~100-300ms (working)
terminal input: BROKEN (PTY write error)

Target Performance (Post-Fix)

terminal input latency: <100ms (after PTY fix)
Session recovery time: <2 seconds
WebSocket reconnection: <1 second
File operation throughput: 100 ops/second per workspace

Scalability Targets

Concurrent WebSocket connections: 1,000 per gateway instance
terminal sessions per instance: 500 active sessions
Gateway instances: Auto-scale 1-20 (current GKE) → 1-100 (Cloud Run)
Cost reduction: 60-80% from GKE migration

Testing Strategy

Critical Tests Needed

#[tokio::test]
async fn test_pty_write_fix() {
    let mut pty = PtyProcess::new(80, 24).await.unwrap();
    
    // Test the specific failure case
    let input = "echo hello\n";
    let result = pty.write_input(input.as_bytes()).await;
    
    assert!(result.is_ok(), "PTY write should succeed");
    
    // Verify output appears
    let output = pty.read_output().await.unwrap();
    assert!(output.contains("hello"));
}

#[tokio::test]
async fn test_session_persistence() {
    let bridge = terminalBridge::new(terminalConfig::default());
    let workspace_id = "test-workspace";
    
    // Create session and run commands
    let session_id = bridge.create_session(workspace_id, 80, 24).await.unwrap();
    bridge.execute_command(&session_id, "export TEST=value").await.unwrap();
    
    // Save state
    bridge.persist_session_state(workspace_id, &session_id).await.unwrap();
    
    // Simulate restart and restore
    let restored = bridge.restore_session(workspace_id, &session_id).await.unwrap();
    
    // Verify environment persisted
    let env_output = bridge.execute_command(&session_id, "echo $TEST").await.unwrap();
    assert!(env_output.contains("value"));
}

#[tokio::test]
async fn test_websocket_migration_compatibility() {
    // Test that existing WebSocket clients work with Cloud Run deployment
    let gateway = WebSocketGateway::new(mock_container_service(), "secret".to_string());
    
    // Simulate connection with existing message format
    let connection = test_websocket_connection().await;
    let result = gateway.handle_message(connection, existing_terminal_message()).await;
    
    assert!(result.is_ok(), "Existing WebSocket protocol should work");
}

Success Criteria

Functional Requirements

terminal input/output fully working (currently broken)
Session persistence across reconnections
File operations with proper workspace isolation
WebSocket gateway stable on Cloud Run

Performance Requirements

terminal latency: <100ms for input/output
Session recovery: <2 seconds
File operations: <300ms for typical files
Gateway throughput: 1,000 concurrent connections

Business Requirements

Cost reduction: 60%+ from GKE → Cloud Run migration
Reliability: 99.9% uptime for WebSocket connections
Scalability: Support 10,000+ concurrent users per region

Consequences

Positive

Builds on Working Code: Enhances existing deployed services
Targeted Fixes: Addresses specific known issues
Cost Optimization: Significant savings from Cloud Run migration
Proven Architecture: Shared services model already working
Quick Implementation: Leverages existing codebase

Negative

Shared Resource Limits: All users share gateway instances
Session Affinity Complexity: WebSocket routing requirements
Debugging Challenges: PTY issues may be system-level

Mitigations

Resource Monitoring: Track per-workspace usage
Graceful Degradation: Fall back to basic terminal if PTY fails
Comprehensive Testing: Verify PTY fixes across environments

QA Certification