Skip to main content

OpenAI CLI: Autonomous Workflow Adaptation

Implementation Status: 🔧 Requires Adaptation
Analysis Date: 2025-10-14
Estimated Implementation: 12-16 weeks

Overview​

OpenAI CLI (Codex) adaptation presents unique challenges due to its stateless API nature and auto-approval capabilities. The implementation focuses on client-side session state management, function calling integration, and selective approval workflows while leveraging Codex's image processing and direct execution capabilities.

OpenAI CLI Unique Capabilities​

Core Features​

  • Auto-Approval Mode: Direct code modification without explicit approval for safe operations
  • Image Processing: Native image input and analysis capabilities
  • Rust Performance: Built in Rust for speed and efficiency
  • Direct Execution: Local filesystem access and command execution
  • Function Calling: OpenAI-style function definitions and execution

Architecture Characteristics​

  • Stateless API: No built-in session persistence
  • Request-Response: No native streaming (polling required)
  • HTTP-Based: All communication via HTTPS API calls
  • Function Calling: Native OpenAI function calling protocol

Autonomous Workflow Adaptation Strategy​

Configuration System: Enhanced Auto-Approval​

# openai-autonomous-config.yaml
openai_session:
provider: "openai"
model: "gpt-4"
api_endpoint: "https://api.openai.com/v1/chat/completions"

# Auto-approval configuration
auto_approval:
enabled: true
safe_operations:
- "read_file"
- "list_directory"
- "analyze_code"
- "explain_function"
- "generate_documentation"

requires_approval:
- "write_file"
- "delete_file"
- "execute_command"
- "install_package"
- "modify_config"

# HumanLayer integration
approval_integration:
enabled: true
daemon_socket: "~/.humanlayer/daemon.sock"
session_id: "${HUMANLAYER_SESSION_ID}"
polling_interval: 3000 # Higher latency for stateless API
timeout: 600000 # 10 minutes

# State management (client-side)
state_management:
persistence_mode: "client_side"
context_preservation: true
max_context_tokens: 128000 # GPT-4 limit
context_compression_threshold: 100000

# Autonomous workflow settings
autonomous_mode:
enabled: true
phase: "DELIBERATION"
artifact_management: true
session_reconstruction: true

# Image processing integration
image_support:
enabled: true
formats: ["png", "jpg", "jpeg", "gif", "webp"]
max_size_mb: 20

Function-Based Integration with Auto-Approval​

Selective Approval Function Implementation​

// openai-integrations/selective-approval.ts
export class OpenAISelectiveApproval {
private daemonClient: DaemonClient
private config: OpenAISessionConfig
private autoApprovedOperations: Set<string>

constructor(config: OpenAISessionConfig) {
this.daemonClient = new DaemonClient(config.daemonSocket)
this.config = config
this.autoApprovedOperations = new Set(config.autoApproval.safeOperations)
}

// Smart approval function that auto-approves safe operations
async registerApprovalFunctions(): Promise<OpenAIFunction[]> {
return [
{
name: "execute_operation",
description: "Execute an operation with intelligent approval routing",
parameters: {
type: "object",
properties: {
operation_type: {
type: "string",
description: "Type of operation (read_file, write_file, execute_command, etc.)"
},
operation_details: {
type: "object",
description: "Detailed parameters for the operation"
},
risk_assessment: {
type: "string",
enum: ["safe", "moderate", "high"],
description: "Risk level of the operation"
},
justification: {
type: "string",
description: "Explanation of why this operation is necessary"
},
image_data: {
type: "string",
description: "Base64 encoded image data if operation involves image processing"
}
},
required: ["operation_type", "operation_details", "justification"]
}
},
{
name: "batch_operations",
description: "Execute multiple operations with batch approval",
parameters: {
type: "object",
properties: {
operations: {
type: "array",
items: {
type: "object",
properties: {
operation_type: { type: "string" },
operation_details: { type: "object" },
risk_assessment: { type: "string" }
}
},
description: "Array of operations to execute"
},
batch_justification: {
type: "string",
description: "Overall justification for the batch of operations"
}
},
required: ["operations", "batch_justification"]
}
}
]
}

// Handle operation execution with intelligent routing
async handleOperationExecution(functionCall: OpenAIFunctionCall): Promise<OpenAIFunctionResult> {
const params = JSON.parse(functionCall.arguments)

// Auto-approve safe operations
if (this.autoApprovedOperations.has(params.operation_type)) {
return await this.executeOperationDirectly(params)
}

// Require human approval for risky operations
try {
const approval = await this.daemonClient.createApproval({
session_id: this.config.sessionId,
tool_name: params.operation_type,
tool_input: JSON.stringify(params.operation_details),
explanation: params.justification,
provider_metadata: {
provider: "openai",
risk_level: params.risk_assessment,
auto_approved: false,
image_included: !!params.image_data
}
})

const result = await this.pollApprovalResult(approval.id)

if (result.decision === 'approve') {
return await this.executeOperationDirectly(params)
} else {
return {
success: false,
message: `Operation denied: ${result.explanation}`,
operation_type: params.operation_type
}
}
} catch (error) {
return {
success: false,
message: `Approval request failed: ${error.message}`,
error: error.message
}
}
}

// Direct execution for auto-approved operations
async executeOperationDirectly(params: OperationParams): Promise<OperationResult> {
switch (params.operation_type) {
case "read_file":
return await this.readFile(params.operation_details.path)
case "list_directory":
return await this.listDirectory(params.operation_details.path)
case "analyze_code":
return await this.analyzeCode(params.operation_details)
case "process_image":
return await this.processImage(params.image_data, params.operation_details)
default:
throw new Error(`Unknown auto-approved operation: ${params.operation_type}`)
}
}
}

Client-Side Session State Management​

Stateless Session Reconstruction​

interface OpenAISessionState {
sessionId: string
provider: 'openai'
phase: 'DELIBERATION' | 'RESEARCH' | 'ACTION'

// Client-side conversation history
messages: OpenAIMessage[]
functionCalls: OpenAIFunctionCall[]

// Context management
totalTokens: number
contextTokenLimit: number
compressionHistory: CompressionEvent[]

// Auto-approval tracking
autoApprovedOperations: OperationLog[]
pendingApprovals: ApprovalRequest[]

// Workflow state
completedPhases: string[]
activeArtifacts: string[]
artifactHistory: ArtifactVersion[]

// Image processing history
processedImages: ImageProcessingRecord[]
}

class OpenAISessionStateManager {
private state: OpenAISessionState
private daemonClient: DaemonClient

constructor(sessionId: string, daemonClient: DaemonClient) {
this.daemonClient = daemonClient
this.state = this.initializeState(sessionId)
}

// Reconstruct session from conversation history
async reconstructSession(sessionId: string): Promise<OpenAISessionState> {
const session = await this.daemonClient.getSession(sessionId)
const events = await this.daemonClient.getConversationEvents(sessionId)

const reconstructedState: OpenAISessionState = {
sessionId: sessionId,
provider: 'openai',
phase: this.inferPhase(events),
messages: this.reconstructMessages(events),
functionCalls: this.extractFunctionCalls(events),
totalTokens: this.calculateTokenUsage(events),
contextTokenLimit: 128000,
// ... other fields reconstructed from events
}

return reconstructedState
}

// Persist state to daemon for cross-session continuity
async persistState(): Promise<void> {
await this.daemonClient.updateSessionMetadata(this.state.sessionId, {
provider_type: 'openai',
provider_state: this.state,
conversation_history: this.state.messages,
last_updated: new Date().toISOString()
})
}

// Context compression when approaching token limits
async compressContext(): Promise<void> {
if (this.state.totalTokens > this.state.contextTokenLimit * 0.8) {
const compressionResult = await this.compressConversationHistory(this.state.messages)

this.state.messages = compressionResult.compressedMessages
this.state.compressionHistory.push({
timestamp: new Date(),
originalTokens: this.state.totalTokens,
compressedTokens: compressionResult.tokenCount,
compressionRatio: compressionResult.tokenCount / this.state.totalTokens
})

this.state.totalTokens = compressionResult.tokenCount
}
}
}

Four-Phase Workflow Implementation​

Phase 1: DELIBERATION with Image Analysis Support​

class OpenAIDeliberationManager {
private stateManager: OpenAISessionStateManager
private approvalIntegration: OpenAISelectiveApproval

async startDeliberationPhase(
requirements: string,
projectPath: string,
images?: ImageInput[]
): Promise<DeliberationResult> {

const systemMessage: OpenAIMessage = {
role: "system",
content: `You are operating in autonomous development mode with selective human oversight.

PHASE: DELIBERATION
CAPABILITIES:
- Function calling with selective auto-approval
- Image processing and analysis (if images provided)
- Direct file system access for safe operations
- Client-side session state management

AUTO-APPROVED OPERATIONS: read_file, list_directory, analyze_code, explain_function, generate_documentation
REQUIRES APPROVAL: write_file, delete_file, execute_command, install_package, modify_config

TASK: ${requirements}

Follow this structured approach:

ANALYSIS:
- Core requirements: [identify 3-5 essential capabilities]
- Technical constraints: [runtime, dependencies, performance needs]
- Image analysis: [if images provided, analyze visual requirements]
- Risk assessment: [identify operations requiring human approval]

DECOMPOSITION:
- Phase 1: [foundation - data models, core abstractions]
- Phase 2: [business logic - core features implementation]
- Phase 3: [integration - APIs, persistence, UI components]
- Phase 4: [polish - error handling, testing, documentation]

APPROVAL STRATEGY:
- Auto-approve: [safe read-only operations during analysis]
- Request approval: [any write operations or destructive changes]
- Batch operations: [group related operations for efficient approval]

Use execute_operation function for all operations. Safe operations will auto-approve.
Request approval to proceed to RESEARCH phase when analysis is complete.`
}

const userMessage: OpenAIMessage = {
role: "user",
content: requirements
}

// Add image content if provided
if (images && images.length > 0) {
userMessage.content = [
{ type: "text", text: requirements },
...images.map(img => ({
type: "image_url",
image_url: {
url: `data:${img.mimeType};base64,${img.base64Data}`,
detail: "high"
}
}))
]
}

const messages = [systemMessage, userMessage]

const session = await this.createOpenAISession({
model: "gpt-4",
messages: messages,
functions: await this.approvalIntegration.registerApprovalFunctions(),
function_call: "auto"
})

const deliberationResponse = await session.getCompletion()

// Update state with deliberation results
await this.stateManager.updateState({
phase: 'DELIBERATION',
messages: [...messages, deliberationResponse],
completedPhases: ['DELIBERATION']
})

return {
analysis: this.extractAnalysis(deliberationResponse),
decomposition: this.extractDecomposition(deliberationResponse),
approvalStrategy: this.extractApprovalStrategy(deliberationResponse),
imageAnalysis: images ? this.extractImageAnalysis(deliberationResponse) : null
}
}
}

Phase 2: RESEARCH with Auto-Approved Exploration​

class OpenAIResearchManager {
async executeResearchPhase(
session: OpenAISession,
knowledgeGaps: string[]
): Promise<ResearchResult> {

const researchPrompt = `PHASE: RESEARCH
KNOWLEDGE GAPS: ${knowledgeGaps.join(', ')}

Research strategy using auto-approved operations:
1. Use read_file and list_directory to explore existing codebase
2. Use analyze_code to understand current implementations
3. Generate documentation and explanations for complex parts
4. Request approval only for external data fetching or API calls

Research approach:
- Leverage auto-approved file operations for rapid exploration
- Analyze existing patterns and architectures
- Document findings without requiring approval
- Request approval for external research or risky operations

Begin comprehensive research using available auto-approved functions:`

// Execute research with heavy use of auto-approved operations
const researchMessages = [
...session.state.messages,
{ role: "user", content: researchPrompt }
]

const researchResults: ResearchFinding[] = []

// Research each knowledge gap with auto-approved operations
for (const gap of knowledgeGaps) {
const gapResearch = await session.researchWithAutoApproval({
topic: gap,
autoApprovedOps: ["read_file", "list_directory", "analyze_code"],
maxAutoOps: 20 // Limit to prevent runaway execution
})

researchResults.push({
topic: gap,
findings: gapResearch.findings,
sources: gapResearch.sources,
confidence: gapResearch.confidence,
autoOpsUsed: gapResearch.autoOpsUsed
})
}

// Synthesize research findings
const synthesis = await session.synthesizeResearch(researchResults)

return {
findings: researchResults,
synthesis: synthesis,
autoApprovedOpsUsed: researchResults.reduce((acc, r) => acc + r.autoOpsUsed, 0),
manualApprovalsRequired: synthesis.riskOperations.length
}
}
}

Phase 3: PLAN with Intelligent Batch Planning​

class OpenAIPlanManager {
async executePlanPhase(
session: OpenAISession,
researchResults: ResearchResult[]
): Promise<PlanResult> {

// Update session state for planning phase
await session.stateManager.updateState({
phase: 'PLAN',
research_completed: true,
research_findings: researchResults,
planning_mode: 'intelligent_batch_specification'
})

const planPrompt = `PHASE: PLAN - Intelligent Implementation Specification
RESEARCH FINDINGS:
${researchResults.map(r => `- ${r.topic}: ${r.findings} (confidence: ${r.confidence})`).join('\n')}

CREATE COMPREHENSIVE IMPLEMENTATION PLAN optimized for OpenAI's selective auto-approval:

## 1. System Architecture Design
Based on research findings, design the complete system:
- Component relationships and data flow
- Technology stack decisions with auto-approval optimization
- Integration patterns optimized for batch operations
- Image processing integration points if applicable
- API design and external service connections

## 2. Selective Auto-Approval Implementation Strategy
Break development into phases optimized for efficient approval workflow:
- Phase 1: Foundation (auto-approved analysis + batch file creation)
- Phase 2: Business Logic (mixed auto-approved reads + batch writes)
- Phase 3: Integration (batch API creation + auto-approved testing)
- Phase 4: UI/Frontend (image processing + batch component creation)
- Phase 5: Polish (auto-approved analysis + batch deployment)

For each phase, specify:
- Auto-approved operations (reads, analysis, documentation)
- Batch approval operations (writes, installs, commands)
- Risk assessment and operation classification
- Dependencies and approval gate optimization
- Image processing requirements if applicable

## 3. Batch Approval Optimization Plan
Optimize human oversight efficiency:
- Group related file operations for single approval
- Batch configuration changes and installations
- Separate high-risk from low-risk operations
- Plan approval checkpoints at logical boundaries
- Estimate approval overhead and optimization opportunities

## 4. Auto-Approval Strategy
Maximize workflow efficiency through intelligent classification:
- Safe operations: file reads, code analysis, documentation generation
- Moderate operations: configuration reads, dependency analysis
- High-risk operations: file writes, command execution, external calls
- Batch grouping strategies for efficient approval cycles

## 5. Image Processing Integration Plan
If visual components are involved:
- Design-to-code generation workflows
- Visual requirement extraction strategies
- UI component generation from mockups
- Auto-approved design analysis processes

## 6. Context Management Strategy
Handle stateless session requirements:
- Context compression and reconstruction plans
- Session state persistence strategies
- Conversation history management
- Token budget optimization across phases

Request approval to proceed to ACTION phase with optimized batch approval plan.
`

const planMessages = [
...session.state.messages,
{ role: "user", content: planPrompt }
]

const planResponse = await session.getCompletion({
messages: planMessages,
functions: await session.approvalIntegration.registerApprovalFunctions(),
function_call: "auto"
})

// Parse comprehensive implementation plan
const implementationPlan = this.parseIntelligentImplementationPlan(planResponse)

// Update session state with complete plan
await session.stateManager.updateState({
phase: 'PLAN',
implementation_plan: implementationPlan,
planning_complete: true,
batch_approval_strategy: implementationPlan.batchApprovalStrategy,
auto_approval_optimization: implementationPlan.autoApprovalOptimization,
ready_for_action: true,
next_steps: ['Request approval for ACTION phase', 'Begin batch-optimized implementation']
})

return {
plan: implementationPlan,
architecture: implementationPlan.architecture,
phases: implementationPlan.phases,
batchApprovalStrategy: implementationPlan.batchApprovalStrategy,
autoApprovalOptimization: implementationPlan.autoApprovalOptimization,
contextStrategy: implementationPlan.contextStrategy,
imageProcessingPlan: implementationPlan.imageProcessingPlan
}
}

parseIntelligentImplementationPlan(planResponse: string): IntelligentImplementationPlan {
return {
phases: this.extractOptimizedPhases(planResponse),
architecture: this.extractSystemDesign(planResponse),
batchApprovalStrategy: {
batchGroups: this.extractBatchGroups(planResponse),
approvalCheckpoints: this.extractApprovalCheckpoints(planResponse),
riskClassification: this.extractRiskClassification(planResponse),
efficiencyTargets: this.extractEfficiencyTargets(planResponse)
},
autoApprovalOptimization: {
safeOperations: this.extractSafeOperations(planResponse),
batchCandidates: this.extractBatchCandidates(planResponse),
workflowEfficiency: this.extractWorkflowEfficiency(planResponse)
},
contextStrategy: {
compressionStrategy: this.extractCompressionStrategy(planResponse),
sessionReconstruction: this.extractReconstructionStrategy(planResponse),
tokenOptimization: this.extractTokenOptimization(planResponse)
},
imageProcessingPlan: this.extractImageProcessingPlan(planResponse),
timeline: this.extractOptimizedTimeline(planResponse),
successCriteria: this.extractIntelligentSuccessCriteria(planResponse)
}
}
}

interface IntelligentImplementationPlan extends ImplementationPlan {
batchApprovalStrategy: {
batchGroups: BatchGroup[]
approvalCheckpoints: ApprovalCheckpoint[]
riskClassification: RiskClassification[]
efficiencyTargets: EfficiencyTarget[]
}
autoApprovalOptimization: {
safeOperations: string[]
batchCandidates: BatchCandidate[]
workflowEfficiency: WorkflowEfficiency
}
contextStrategy: {
compressionStrategy: CompressionStrategy
sessionReconstruction: ReconstructionStrategy
tokenOptimization: TokenOptimization
}
imageProcessingPlan?: ImageProcessingPlan
}

Phase 4: ACTION with Batch Approval Strategy​

class OpenAIActionManager {
async executeActionPhase(session: OpenAISession, plan: ImplementationPlan): Promise<ActionResult> {
const actionPrompt = `PHASE: ACTION - Implementation
PLAN: ${JSON.stringify(plan, null, 2)}

Implementation strategy:
- Use batch_operations function for efficient approval of write operations
- Auto-approve safe operations (reading, analyzing existing code)
- Group related file operations for single approval cycle
- Leverage image processing capabilities if UI/visual components involved

Batch approval strategy:
- Group file creations by module/component
- Batch configuration changes
- Request approval for command execution and package installations
- Auto-approve documentation generation and code analysis

Begin implementation with optimized approval workflow:`

const artifacts: CodeArtifact[] = []
const batches = this.createOperationBatches(plan)

for (const batch of batches) {
// Separate auto-approved from approval-required operations
const autoOps = batch.operations.filter(op =>
session.config.autoApproval.safeOperations.includes(op.type)
)
const approvalOps = batch.operations.filter(op =>
session.config.autoApproval.requiresApproval.includes(op.type)
)

// Execute auto-approved operations immediately
const autoResults = await Promise.all(
autoOps.map(op => session.executeAutoApproved(op))
)

// Batch approval for risky operations
if (approvalOps.length > 0) {
const batchApproval = await session.callFunction('batch_operations', {
operations: approvalOps,
batch_justification: `Implementing ${batch.description}: ${approvalOps.length} file operations`
})

if (!batchApproval.success) {
throw new Error(`Batch approval failed: ${batchApproval.message}`)
}
}

// Execute approved operations
const approvalResults = await Promise.all(
approvalOps.map(op => session.executeApproved(op))
)

// Collect artifacts from batch
const batchArtifacts = [...autoResults, ...approvalResults]
.filter(result => result.artifact)
.map(result => result.artifact)

artifacts.push(...batchArtifacts)

// Update session state after each batch
await session.stateManager.updateState({
completedBatches: [...session.state.completedBatches, batch.id],
activeArtifacts: artifacts.map(a => a.filename)
})
}

return {
artifacts: artifacts,
batchesExecuted: batches.length,
autoApprovedOps: batches.reduce((acc, b) => acc + b.autoOpsCount, 0),
manualApprovals: batches.reduce((acc, b) => acc + b.approvalOpsCount, 0),
efficiency: this.calculateEfficiency(batches)
}
}

// Optimize operation batching for approval efficiency
createOperationBatches(plan: ImplementationPlan): OperationBatch[] {
const batches: OperationBatch[] = []

// Group by component/module
const componentGroups = this.groupOperationsByComponent(plan.operations)

for (const [component, operations] of componentGroups.entries()) {
// Further group by risk level for optimal batching
const riskGroups = this.groupOperationsByRisk(operations)

for (const [risk, ops] of riskGroups.entries()) {
batches.push({
id: `${component}-${risk}`,
description: `${component} ${risk} operations`,
operations: ops,
autoOpsCount: ops.filter(op => this.isAutoApproved(op.type)).length,
approvalOpsCount: ops.filter(op => !this.isAutoApproved(op.type)).length
})
}
}

return batches
}
}

Image Processing Integration​

Visual Analysis for Development​

class OpenAIImageProcessor {
async processDesignImages(images: ImageInput[], requirements: string): Promise<VisualAnalysis> {
const analysis = await this.session.callFunction('execute_operation', {
operation_type: 'process_image',
operation_details: {
analysis_type: 'ui_design',
requirements: requirements
},
risk_assessment: 'safe',
justification: 'Analyzing design images to extract UI requirements',
image_data: images.map(img => img.base64Data)
})

return {
components: this.extractUIComponents(analysis),
layout: this.extractlayoutStructure(analysis),
styling: this.extractStylingRequirements(analysis),
interactions: this.extractInteractionPatterns(analysis)
}
}

async generateCodeFromDesign(visualAnalysis: VisualAnalysis): Promise<GeneratedCode> {
// Auto-approved code generation based on visual analysis
const codeGeneration = await this.session.callFunction('execute_operation', {
operation_type: 'generate_code',
operation_details: {
generation_type: 'ui_components',
visual_analysis: visualAnalysis,
target_framework: 'react'
},
risk_assessment: 'safe',
justification: 'Generating UI code from visual analysis'
})

return codeGeneration.generated_code
}
}

Implementation Timeline​

Phase 1: Core Integration (6 weeks)​

  • Develop OpenAI function calling adapter with selective auto-approval
  • Implement client-side session state management
  • Create stateless session reconstruction system

Phase 2: Auto-Approval System (4 weeks)​

  • Build intelligent operation classification
  • Implement batch approval optimization
  • Create risk assessment automation

Phase 3: Image Processing (3 weeks)​

  • Integrate image analysis capabilities
  • Build visual-to-code generation workflows
  • Implement design analysis automation

Phase 4: Performance Optimization (5 weeks)​

  • Optimize context compression strategies
  • Implement efficient polling mechanisms
  • Build session recovery and continuation systems

Success Metrics​

  • Auto-Approval Efficiency: 70%+ of operations auto-approved for faster workflow
  • Batch Approval Optimization: 50% reduction in human approval interactions
  • Session Reconstruction: 95%+ accuracy in stateless session recovery
  • Image Processing Integration: Support for visual design-to-code workflows

This adaptation leverages OpenAI CLI's unique auto-approval capabilities and image processing while maintaining essential human oversight through selective approval integration with HumanLayer's infrastructure.