Multi-AI CLI Generalization Analysis

Analysis Date: 2025-10-14
Focus: Generalizing HumanLayer/CodeLayer for multiple AI CLI tools
Scope: Gemini CLI, Grok CLI, OpenAI CLI, and extensible architecture

Executive Summary

The current HumanLayer/CodeLayer system is tightly coupled to Claude Code through MCP (Model Context Protocol) integration. To generalize this system for multiple AI CLI tools, we need to abstract the integration layer and create a pluggable architecture that can accommodate different communication protocols, command structures, and approval mechanisms.

Current State Analysis

Claude Code Integration Architecture

The existing system implements a sophisticated but Claude-specific integration:

1. MCP Protocol Dependency: Direct integration via hlyr/src/mcp.ts with hardcoded tool name request_permission
2. Binary Discovery: Claude-specific path detection and validation
3. Configuration Injection: MCP server configuration directly embedded in session management
4. Event Stream Processing: Assumes Claude Code's JSON event format
5. Permission Model: Single approval workflow designed for Claude's tool calling

Limitations for Multi-AI Support

1. Protocol Lock-in: MCP protocol assumption throughout the stack
2. Configuration Rigidity: No abstraction for different CLI parameter structures
3. Binary-Specific Logic: Hardcoded paths and validation rules for Claude
4. Event Format Coupling: Direct dependency on Claude's event stream format
5. Permission Tool Naming: Fixed tool names without provider abstraction

AI CLI Landscape Analysis

Comparison Matrix

Feature	Claude Code	Gemini CLI	Grok CLI	OpenAI CLI
Protocol	MCP (Model Context Protocol)	MCP + Extensions	OpenAI-compatible API	OpenAI API
Tool Integration	MCP tools via stdio	Built-in + MCP servers	Tool execution rounds	Function calling
Permission Model	Tool-level approval	Extension-based	Built-in tool limits	Function-level controls
Configuration	JSON config files	GEMINI.md + settings	Configuration files	API keys + params
Session Management	Native sessions	Conversation checkpoints	Context windows	Chat completions
Extension System	MCP servers	Extensions ecosystem	Third-party tools	Custom implementations

Common Patterns Identified

1. Tool/Function Calling: All systems support some form of external tool integration
2. Configuration Files: Each system uses configuration for customization (JSON, Markdown, YAML)
3. Session Persistence: All maintain conversation/session state in some form
4. Extension Mechanisms: Plugin/extension systems for third-party integrations
5. Command-Line Interface: Standard CLI patterns with subcommands and flags

Unique Characteristics

Gemini CLI

• Extensions Ecosystem: Rich partner integrations (Stripe, Figma, Dynatrace)
• Context Files: GEMINI.md for persistent context injection
• Enterprise Features: Sandboxing, trusted folders, enterprise deployment
• IDE Integration: Direct integration with Zed, VS Code support

Grok CLI

• Multi-Round Execution: 400+ tool execution rounds for complex tasks
• High-Speed Editing: Morph Fast Apply at 4,500+ tokens/sec
• OpenAI Compatibility: Can use any OpenAI-compatible API provider
• Community-Driven: Multiple open-source implementations

OpenAI CLI (Codex)

• Auto-Approval Mode: Direct code modification without explicit approval
• Image Support: Native image input and processing
• Rust Performance: Built in Rust for speed and efficiency
• Local Execution: Direct filesystem access and command execution

Generalized Architecture Design

Core Abstraction Layers

1. AI Provider Interface

// Core interface for AI CLI providers
type AIProvider interface {
    // Provider metadata
    GetProviderInfo() ProviderInfo
    
    // Binary management
    DiscoverBinary() (string, error)
    ValidateBinary(path string) error
    
    // Session lifecycle
    CreateSession(config SessionConfig) (AISession, error)
    
    // Tool integration
    GetToolIntegrationMethod() ToolIntegrationMethod
    CreateToolServer(config ToolServerConfig) (ToolServer, error)
    
    // Configuration
    PrepareConfiguration(session SessionConfig) (ProviderConfig, error)
    InjectApprovalMechanism(config ProviderConfig) (ProviderConfig, error)
}

type ProviderInfo struct {
    Name            string
    Version         string
    SupportedModels []string
    Capabilities    []Capability
    ConfigSchema    JSONSchema
}

type ToolIntegrationMethod string
const (
    ToolIntegrationMCP       ToolIntegrationMethod = "mcp"
    ToolIntegrationOpenAI    ToolIntegrationMethod = "openai"
    ToolIntegrationCustom    ToolIntegrationMethod = "custom"
    ToolIntegrationBuiltIn   ToolIntegrationMethod = "builtin"
)

2. Session Management Abstraction

type AISession interface {
    GetID() string
    GetStatus() SessionStatus
    GetProvider() string
    
    // Lifecycle
    Start() error
    Stop() error
    Interrupt() error
    
    // Communication
    GetEventStream() <-chan SessionEvent
    SendCommand(cmd Command) error
    
    // State
    GetMetadata() SessionMetadata
    GetConfiguration() SessionConfig
}

type SessionConfig struct {
    Provider        string
    Model          string
    WorkingDir     string
    Environment    map[string]string
    ProviderConfig map[string]interface{}
    
    // Approval configuration
    ApprovalMode   ApprovalMode
    ToolServerConfig *ToolServerConfig
}

type SessionEvent struct {
    ID        string
    Type      SessionEventType
    Timestamp time.Time
    Provider  string
    SessionID string
    Data      map[string]interface{}
}

3. Tool Server Abstraction

type ToolServer interface {
    Start() error
    Stop() error
    GetAddress() string
    GetProtocol() string
    
    // Tool registration
    RegisterTool(tool Tool) error
    UnregisterTool(toolName string) error
    ListTools() []Tool
    
    // Approval integration
    SetApprovalHandler(handler ApprovalHandler) error
}

type Tool struct {
    Name        string
    Description string
    Schema      JSONSchema
    Handler     ToolHandler
    
    // Approval configuration
    RequiresApproval bool
    ApprovalPolicy   ApprovalPolicy
}

type ApprovalHandler func(request ToolRequest) (ToolResponse, error)

Provider Implementations

Claude Code Provider

type ClaudeProvider struct {
    binaryPath string
    config     *config.Config
}

func (cp *ClaudeProvider) GetProviderInfo() ProviderInfo {
    return ProviderInfo{
        Name:            "claude-code",
        Version:         "latest",
        SupportedModels: []string{"sonnet", "haiku", "opus"},
        Capabilities:    []Capability{CapabilityMCP, CapabilityFileOps, CapabilityBash},
        ConfigSchema:    loadClaudeConfigSchema(),
    }
}

func (cp *ClaudeProvider) CreateSession(config SessionConfig) (AISession, error) {
    claudeConfig := &claudecode.Config{
        Model:       config.Model,
        Directory:   config.WorkingDir,
        MCPServers: map[string]claudecode.MCPServer{},
    }
    
    // Inject MCP server for approvals if enabled
    if config.ApprovalMode != ApprovalModeDisabled {
        claudeConfig.MCPServers["humanlayer"] = claudecode.MCPServer{
            Command: "hlyr",
            Args:    []string{"mcp", "claude_approvals"},
            Env: map[string]string{
                "HUMANLAYER_SESSION_ID":    config.SessionID,
                "HUMANLAYER_DAEMON_SOCKET": cp.config.SocketPath,
            },
        }
    }
    
    return NewClaudeSession(claudeConfig), nil
}

func (cp *ClaudeProvider) CreateToolServer(config ToolServerConfig) (ToolServer, error) {
    return NewMCPServer(config), nil
}

Gemini CLI Provider

type GeminiProvider struct {
    binaryPath string
    config     *GeminiConfig
}

func (gp *GeminiProvider) GetProviderInfo() ProviderInfo {
    return ProviderInfo{
        Name:            "gemini-cli",
        Version:         "latest", 
        SupportedModels: []string{"gemini-2.5-pro", "gemini-2.5-flash"},
        Capabilities:    []Capability{CapabilityMCP, CapabilityExtensions, CapabilitySearch},
        ConfigSchema:    loadGeminiConfigSchema(),
    }
}

func (gp *GeminiProvider) PrepareConfiguration(session SessionConfig) (ProviderConfig, error) {
    // Create GEMINI.md context file
    contextFile := filepath.Join(session.WorkingDir, "GEMINI.md")
    contextContent := generateGeminiContext(session)
    
    if err := ioutil.WriteFile(contextFile, []byte(contextContent), 0644); err != nil {
        return nil, fmt.Errorf("failed to create GEMINI.md: %w", err)
    }
    
    // Configure extensions
    extensionConfig := map[string]interface{}{
        "extensions": []string{"humanlayer-approval"},
        "settings": map[string]interface{}{
            "approval_mode": session.ApprovalMode,
            "daemon_socket": gp.config.DaemonSocket,
        },
    }
    
    return ProviderConfig{
        Type: "gemini",
        Data: extensionConfig,
    }, nil
}

func (gp *GeminiProvider) CreateToolServer(config ToolServerConfig) (ToolServer, error) {
    // Gemini uses extensions instead of direct MCP servers
    return NewGeminiExtensionServer(config), nil
}

Grok CLI Provider

type GrokProvider struct {
    binaryPath string
    apiKey     string
}

func (gp *GrokProvider) GetProviderInfo() ProviderInfo {
    return ProviderInfo{
        Name:            "grok-cli",
        Version:         "latest",
        SupportedModels: []string{"grok-4-latest", "grok-3-latest", "grok-3-fast"},
        Capabilities:    []Capability{CapabilityOpenAI, CapabilityMultiRound, CapabilityHighSpeed},
        ConfigSchema:    loadGrokConfigSchema(),
    }
}

func (gp *GrokProvider) CreateSession(config SessionConfig) (AISession, error) {
    grokConfig := GrokSessionConfig{
        Model:              config.Model,
        WorkingDirectory:   config.WorkingDir,
        MaxToolRounds:      400, // Grok's default
        ToolExecutionLimit: config.ProviderConfig["tool_limit"].(int),
        APIEndpoint:        "https://api.x.ai/v1/chat/completions",
    }
    
    return NewGrokSession(grokConfig), nil
}

func (gp *GrokProvider) CreateToolServer(config ToolServerConfig) (ToolServer, error) {
    // Grok uses OpenAI-compatible function calling
    return NewOpenAICompatibleToolServer(config), nil
}

Multi-Protocol Tool Server

type MultiProtocolToolServer struct {
    servers map[string]ToolServer
    router  *ToolRouter
}

func NewMultiProtocolToolServer() *MultiProtocolToolServer {
    return &MultiProtocolToolServer{
        servers: make(map[string]ToolServer),
        router:  NewToolRouter(),
    }
}

func (mpts *MultiProtocolToolServer) AddProtocolServer(protocol string, server ToolServer) error {
    mpts.servers[protocol] = server
    return mpts.router.RegisterProtocol(protocol, server)
}

func (mpts *MultiProtocolToolServer) Start() error {
    // Start all protocol servers
    for protocol, server := range mpts.servers {
        if err := server.Start(); err != nil {
            return fmt.Errorf("failed to start %s server: %w", protocol, err)
        }
    }
    
    return mpts.router.Start()
}

// MCP Server Implementation
type MCPToolServer struct {
    server *mcp.Server
    tools  map[string]Tool
    approvalHandler ApprovalHandler
}

func (mts *MCPToolServer) RegisterTool(tool Tool) error {
    mts.tools[tool.Name] = tool
    
    // Register with MCP server
    mcpTool := mcp.Tool{
        Name:        tool.Name,
        Description: tool.Description,
        InputSchema: tool.Schema,
    }
    
    return mts.server.RegisterTool(mcpTool, func(req mcp.ToolRequest) (mcp.ToolResponse, error) {
        if tool.RequiresApproval && mts.approvalHandler != nil {
            toolReq := ToolRequest{
                Name:       tool.Name,
                Parameters: req.Parameters,
                SessionID:  req.SessionID,
            }
            
            return mts.approvalHandler(toolReq)
        }
        
        return tool.Handler(req)
    })
}

// OpenAI Compatible Server Implementation  
type OpenAIToolServer struct {
    httpServer *http.Server
    tools      map[string]Tool
    approvalHandler ApprovalHandler
}

func (ots *OpenAIToolServer) RegisterTool(tool Tool) error {
    ots.tools[tool.Name] = tool
    
    // Register OpenAI function
    function := openai.Function{
        Name:        tool.Name,
        Description: tool.Description,
        Parameters:  tool.Schema,
    }
    
    return ots.registerOpenAIFunction(function, func(req openai.FunctionCall) (openai.FunctionResponse, error) {
        if tool.RequiresApproval && ots.approvalHandler != nil {
            toolReq := ToolRequest{
                Name:       req.Name,
                Parameters: req.Arguments,
                SessionID:  extractSessionID(req),
            }
            
            return ots.approvalHandler(toolReq)
        }
        
        return tool.Handler(req)
    })
}

Configuration Management

Provider Registry

type ProviderRegistry struct {
    providers map[string]AIProvider
    configs   map[string]ProviderConfig
}

func NewProviderRegistry() *ProviderRegistry {
    registry := &ProviderRegistry{
        providers: make(map[string]AIProvider),
        configs:   make(map[string]ProviderConfig),
    }
    
    // Register built-in providers
    registry.RegisterProvider("claude", NewClaudeProvider())
    registry.RegisterProvider("gemini", NewGeminiProvider()) 
    registry.RegisterProvider("grok", NewGrokProvider())
    registry.RegisterProvider("openai", NewOpenAIProvider())
    
    return registry
}

func (pr *ProviderRegistry) CreateSession(providerName string, config SessionConfig) (AISession, error) {
    provider, exists := pr.providers[providerName]
    if !exists {
        return nil, fmt.Errorf("unknown provider: %s", providerName)
    }
    
    return provider.CreateSession(config)
}

func (pr *ProviderRegistry) ListProviders() []ProviderInfo {
    var providers []ProviderInfo
    for _, provider := range pr.providers {
        providers = append(providers, provider.GetProviderInfo())
    }
    return providers
}

Configuration Schema

# Multi-provider configuration
providers:
  claude:
    binary_path: "/usr/local/bin/claude-code"
    default_model: "sonnet"
    capabilities:
      - mcp
      - file_operations
      - bash_execution
    approval:
      mode: "required"
      timeout: 300
      
  gemini:
    binary_path: "/usr/local/bin/gemini"
    api_key_env: "GEMINI_API_KEY"
    default_model: "gemini-2.5-pro"
    capabilities:
      - extensions
      - search_grounding
      - context_files
    approval:
      mode: "extension"
      extension: "humanlayer-approval"
      
  grok:
    binary_path: "/usr/local/bin/grok-cli"
    api_endpoint: "https://api.x.ai/v1/chat/completions"
    default_model: "grok-4-latest"
    capabilities:
      - openai_compatible
      - multi_round_execution
      - high_speed_editing
    approval:
      mode: "function_calling"
      max_rounds: 400
      
  openai:
    binary_path: "/usr/local/bin/codex"
    api_key_env: "OPENAI_API_KEY"
    default_model: "gpt-4"
    capabilities:
      - function_calling
      - image_processing
      - auto_approval
    approval:
      mode: "optional"
      auto_approve: ["read_file", "list_directory"]

# Session defaults
session_defaults:
  approval_mode: "required"
  tool_timeout: 30
  max_tool_rounds: 10
  
# Tool server configuration  
tool_servers:
  mcp:
    enabled: true
    address: "stdio"
    
  openai:
    enabled: true
    port: 8081
    
  custom:
    enabled: false

Integration API

REST API Extensions

# Extended REST API for multi-provider support
paths:
  /api/v1/providers:
    get:
      summary: List available AI providers
      responses:
        '200':
          content:
            application/json:
              schema:
                type: object
                properties:
                  providers:
                    type: array
                    items:
                      $ref: '#/components/schemas/ProviderInfo'
                      
  /api/v1/sessions:
    post:
      summary: Create session with provider selection
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                provider:
                  type: string
                  enum: [claude, gemini, grok, openai]
                model:
                  type: string
                directory:
                  type: string
                approval_mode:
                  type: string
                  enum: [disabled, required, optional, auto]
                provider_config:
                  type: object
                  additionalProperties: true
              required: [provider, directory]
              
  /api/v1/sessions/{session_id}/tools:
    get:
      summary: List available tools for session
      responses:
        '200':
          content:
            application/json:
              schema:
                type: object
                properties:
                  tools:
                    type: array
                    items:
                      $ref: '#/components/schemas/ToolInfo'

CLI Command Extensions

# Multi-provider CLI commands
hlyr launch --provider claude --model sonnet --directory ./project
hlyr launch --provider gemini --model gemini-2.5-pro --directory ./project  
hlyr launch --provider grok --model grok-4-latest --directory ./project
hlyr launch --provider openai --model gpt-4 --directory ./project

# Provider management
hlyr providers list
hlyr providers install gemini-cli
hlyr providers configure claude --binary-path /custom/path
hlyr providers test grok

# Tool server management
hlyr tools start --protocols mcp,openai
hlyr tools list --provider gemini
hlyr tools register custom-tool --provider all

Implementation Strategy

Phase 1: Core Abstraction (4-6 weeks)

1. Interface Definition: Define core AIProvider, AISession, ToolServer interfaces
2. Provider Registry: Implement provider registration and discovery system
3. Session Manager Refactoring: Abstract session management from Claude-specific logic
4. Configuration System: Implement multi-provider configuration management

Phase 2: Claude Provider Implementation (2-3 weeks)

1. Claude Provider: Wrap existing Claude Code integration in provider interface
2. MCP Tool Server: Abstract existing MCP server implementation
3. Backward Compatibility: Ensure existing Claude Code workflows continue working
4. Migration Path: Provide clear migration for existing users

Phase 3: Additional Providers (6-8 weeks each)

1. Gemini Provider: Implement Gemini CLI integration with extensions support
2. Grok Provider: Implement Grok CLI with OpenAI-compatible functions
3. OpenAI Provider: Implement Codex/OpenAI CLI with function calling
4. Testing: Comprehensive integration testing for each provider

Phase 4: Advanced Features (4-6 weeks)

1. Multi-Protocol Tool Servers: Support MCP + OpenAI functions simultaneously
2. Provider Auto-Discovery: Automatic binary detection and capability assessment
3. Plugin System: Third-party provider plugin architecture
4. Performance Optimization: Optimize for multi-provider session management

Benefits of Generalization

For Users

1. Provider Choice: Use preferred AI CLI while maintaining approval workflows
2. Unified Interface: Same UI/CLI experience across different AI providers
3. Migration Flexibility: Easy switching between providers based on needs
4. Cost Optimization: Use different providers for different use cases

For Developers

1. Extensibility: Easy to add new AI CLI providers
2. Protocol Agnostic: Support multiple integration protocols (MCP, OpenAI, custom)
3. Testing: Better isolation and testing of provider-specific logic
4. Maintenance: Clear separation of concerns reduces complexity

For Enterprise

1. Vendor Independence: Avoid lock-in to single AI provider
2. Compliance: Choose providers based on compliance requirements
3. Performance: Select optimal provider for specific tasks
4. Cost Control: Optimize costs across multiple AI service providers

Challenges and Mitigation

Technical Challenges

1. Protocol Differences: Each provider uses different integration methods
   • Mitigation: Multi-protocol tool server with adapter pattern
2. Configuration Complexity: Managing multiple provider configurations
   • Mitigation: Structured configuration with validation and defaults
3. Event Format Variations: Different event stream formats across providers
   • Mitigation: Normalized event interface with provider-specific adapters

User Experience Challenges

1. Configuration Complexity: Users need to understand multiple providers
   • Mitigation: Auto-discovery, sensible defaults, guided configuration
2. Feature Parity: Not all providers support same capabilities
   • Mitigation: Capability-based feature detection and graceful degradation
3. Migration Complexity: Moving between providers
   • Mitigation: Export/import tools, configuration migration utilities

Conclusion

Generalizing HumanLayer/CodeLayer for multiple AI CLI tools requires significant architectural changes but provides substantial benefits in terms of flexibility, vendor independence, and user choice. The proposed abstraction layer maintains the core approval workflow concepts while enabling support for diverse AI providers with different integration patterns.

The implementation strategy provides a clear path from the current Claude-specific system to a generalized multi-provider platform, ensuring backward compatibility while opening new possibilities for AI coding assistant orchestration across the entire ecosystem of available tools.