OpenAI CLI: Autonomous Workflow Adaptation
Implementation Status: 🔧 Requires Adaptation
Analysis Date: 2025-10-14
Estimated Implementation: 12-16 weeks
Overview​
OpenAI CLI (Codex) adaptation presents unique challenges due to its stateless API nature and auto-approval capabilities. The implementation focuses on client-side session state management, function calling integration, and selective approval workflows while leveraging Codex's image processing and direct execution capabilities.
OpenAI CLI Unique Capabilities​
Core Features​
- Auto-Approval Mode: Direct code modification without explicit approval for safe operations
- Image Processing: Native image input and analysis capabilities
- Rust Performance: Built in Rust for speed and efficiency
- Direct Execution: Local filesystem access and command execution
- Function Calling: OpenAI-style function definitions and execution
Architecture Characteristics​
- Stateless API: No built-in session persistence
- Request-Response: No native streaming (polling required)
- HTTP-Based: All communication via HTTPS API calls
- Function Calling: Native OpenAI function calling protocol
Autonomous Workflow Adaptation Strategy​
Configuration System: Enhanced Auto-Approval​
# openai-autonomous-config.yaml
openai_session:
provider: "openai"
model: "gpt-4"
api_endpoint: "https://api.openai.com/v1/chat/completions"
# Auto-approval configuration
auto_approval:
enabled: true
safe_operations:
- "read_file"
- "list_directory"
- "analyze_code"
- "explain_function"
- "generate_documentation"
requires_approval:
- "write_file"
- "delete_file"
- "execute_command"
- "install_package"
- "modify_config"
# HumanLayer integration
approval_integration:
enabled: true
daemon_socket: "~/.humanlayer/daemon.sock"
session_id: "${HUMANLAYER_SESSION_ID}"
polling_interval: 3000 # Higher latency for stateless API
timeout: 600000 # 10 minutes
# State management (client-side)
state_management:
persistence_mode: "client_side"
context_preservation: true
max_context_tokens: 128000 # GPT-4 limit
context_compression_threshold: 100000
# Autonomous workflow settings
autonomous_mode:
enabled: true
phase: "DELIBERATION"
artifact_management: true
session_reconstruction: true
# Image processing integration
image_support:
enabled: true
formats: ["png", "jpg", "jpeg", "gif", "webp"]
max_size_mb: 20
Function-Based Integration with Auto-Approval​
Selective Approval Function Implementation​
// openai-integrations/selective-approval.ts
export class OpenAISelectiveApproval {
private daemonClient: DaemonClient
private config: OpenAISessionConfig
private autoApprovedOperations: Set<string>
constructor(config: OpenAISessionConfig) {
this.daemonClient = new DaemonClient(config.daemonSocket)
this.config = config
this.autoApprovedOperations = new Set(config.autoApproval.safeOperations)
}
// Smart approval function that auto-approves safe operations
async registerApprovalFunctions(): Promise<OpenAIFunction[]> {
return [
{
name: "execute_operation",
description: "Execute an operation with intelligent approval routing",
parameters: {
type: "object",
properties: {
operation_type: {
type: "string",
description: "Type of operation (read_file, write_file, execute_command, etc.)"
},
operation_details: {
type: "object",
description: "Detailed parameters for the operation"
},
risk_assessment: {
type: "string",
enum: ["safe", "moderate", "high"],
description: "Risk level of the operation"
},
justification: {
type: "string",
description: "Explanation of why this operation is necessary"
},
image_data: {
type: "string",
description: "Base64 encoded image data if operation involves image processing"
}
},
required: ["operation_type", "operation_details", "justification"]
}
},
{
name: "batch_operations",
description: "Execute multiple operations with batch approval",
parameters: {
type: "object",
properties: {
operations: {
type: "array",
items: {
type: "object",
properties: {
operation_type: { type: "string" },
operation_details: { type: "object" },
risk_assessment: { type: "string" }
}
},
description: "Array of operations to execute"
},
batch_justification: {
type: "string",
description: "Overall justification for the batch of operations"
}
},
required: ["operations", "batch_justification"]
}
}
]
}
// Handle operation execution with intelligent routing
async handleOperationExecution(functionCall: OpenAIFunctionCall): Promise<OpenAIFunctionResult> {
const params = JSON.parse(functionCall.arguments)
// Auto-approve safe operations
if (this.autoApprovedOperations.has(params.operation_type)) {
return await this.executeOperationDirectly(params)
}
// Require human approval for risky operations
try {
const approval = await this.daemonClient.createApproval({
session_id: this.config.sessionId,
tool_name: params.operation_type,
tool_input: JSON.stringify(params.operation_details),
explanation: params.justification,
provider_metadata: {
provider: "openai",
risk_level: params.risk_assessment,
auto_approved: false,
image_included: !!params.image_data
}
})
const result = await this.pollApprovalResult(approval.id)
if (result.decision === 'approve') {
return await this.executeOperationDirectly(params)
} else {
return {
success: false,
message: `Operation denied: ${result.explanation}`,
operation_type: params.operation_type
}
}
} catch (error) {
return {
success: false,
message: `Approval request failed: ${error.message}`,
error: error.message
}
}
}
// Direct execution for auto-approved operations
async executeOperationDirectly(params: OperationParams): Promise<OperationResult> {
switch (params.operation_type) {
case "read_file":
return await this.readFile(params.operation_details.path)
case "list_directory":
return await this.listDirectory(params.operation_details.path)
case "analyze_code":
return await this.analyzeCode(params.operation_details)
case "process_image":
return await this.processImage(params.image_data, params.operation_details)
default:
throw new Error(`Unknown auto-approved operation: ${params.operation_type}`)
}
}
}
Client-Side Session State Management​
Stateless Session Reconstruction​
interface OpenAISessionState {
sessionId: string
provider: 'openai'
phase: 'DELIBERATION' | 'RESEARCH' | 'ACTION'
// Client-side conversation history
messages: OpenAIMessage[]
functionCalls: OpenAIFunctionCall[]
// Context management
totalTokens: number
contextTokenLimit: number
compressionHistory: CompressionEvent[]
// Auto-approval tracking
autoApprovedOperations: OperationLog[]
pendingApprovals: ApprovalRequest[]
// Workflow state
completedPhases: string[]
activeArtifacts: string[]
artifactHistory: ArtifactVersion[]
// Image processing history
processedImages: ImageProcessingRecord[]
}
class OpenAISessionStateManager {
private state: OpenAISessionState
private daemonClient: DaemonClient
constructor(sessionId: string, daemonClient: DaemonClient) {
this.daemonClient = daemonClient
this.state = this.initializeState(sessionId)
}
// Reconstruct session from conversation history
async reconstructSession(sessionId: string): Promise<OpenAISessionState> {
const session = await this.daemonClient.getSession(sessionId)
const events = await this.daemonClient.getConversationEvents(sessionId)
const reconstructedState: OpenAISessionState = {
sessionId: sessionId,
provider: 'openai',
phase: this.inferPhase(events),
messages: this.reconstructMessages(events),
functionCalls: this.extractFunctionCalls(events),
totalTokens: this.calculateTokenUsage(events),
contextTokenLimit: 128000,
// ... other fields reconstructed from events
}
return reconstructedState
}
// Persist state to daemon for cross-session continuity
async persistState(): Promise<void> {
await this.daemonClient.updateSessionMetadata(this.state.sessionId, {
provider_type: 'openai',
provider_state: this.state,
conversation_history: this.state.messages,
last_updated: new Date().toISOString()
})
}
// Context compression when approaching token limits
async compressContext(): Promise<void> {
if (this.state.totalTokens > this.state.contextTokenLimit * 0.8) {
const compressionResult = await this.compressConversationHistory(this.state.messages)
this.state.messages = compressionResult.compressedMessages
this.state.compressionHistory.push({
timestamp: new Date(),
originalTokens: this.state.totalTokens,
compressedTokens: compressionResult.tokenCount,
compressionRatio: compressionResult.tokenCount / this.state.totalTokens
})
this.state.totalTokens = compressionResult.tokenCount
}
}
}
Four-Phase Workflow Implementation​
Phase 1: DELIBERATION with Image Analysis Support​
class OpenAIDeliberationManager {
private stateManager: OpenAISessionStateManager
private approvalIntegration: OpenAISelectiveApproval
async startDeliberationPhase(
requirements: string,
projectPath: string,
images?: ImageInput[]
): Promise<DeliberationResult> {
const systemMessage: OpenAIMessage = {
role: "system",
content: `You are operating in autonomous development mode with selective human oversight.
PHASE: DELIBERATION
CAPABILITIES:
- Function calling with selective auto-approval
- Image processing and analysis (if images provided)
- Direct file system access for safe operations
- Client-side session state management
AUTO-APPROVED OPERATIONS: read_file, list_directory, analyze_code, explain_function, generate_documentation
REQUIRES APPROVAL: write_file, delete_file, execute_command, install_package, modify_config
TASK: ${requirements}
Follow this structured approach:
ANALYSIS:
- Core requirements: [identify 3-5 essential capabilities]
- Technical constraints: [runtime, dependencies, performance needs]
- Image analysis: [if images provided, analyze visual requirements]
- Risk assessment: [identify operations requiring human approval]
DECOMPOSITION:
- Phase 1: [foundation - data models, core abstractions]
- Phase 2: [business logic - core features implementation]
- Phase 3: [integration - APIs, persistence, UI components]
- Phase 4: [polish - error handling, testing, documentation]
APPROVAL STRATEGY:
- Auto-approve: [safe read-only operations during analysis]
- Request approval: [any write operations or destructive changes]
- Batch operations: [group related operations for efficient approval]
Use execute_operation function for all operations. Safe operations will auto-approve.
Request approval to proceed to RESEARCH phase when analysis is complete.`
}
const userMessage: OpenAIMessage = {
role: "user",
content: requirements
}
// Add image content if provided
if (images && images.length > 0) {
userMessage.content = [
{ type: "text", text: requirements },
...images.map(img => ({
type: "image_url",
image_url: {
url: `data:${img.mimeType};base64,${img.base64Data}`,
detail: "high"
}
}))
]
}
const messages = [systemMessage, userMessage]
const session = await this.createOpenAISession({
model: "gpt-4",
messages: messages,
functions: await this.approvalIntegration.registerApprovalFunctions(),
function_call: "auto"
})
const deliberationResponse = await session.getCompletion()
// Update state with deliberation results
await this.stateManager.updateState({
phase: 'DELIBERATION',
messages: [...messages, deliberationResponse],
completedPhases: ['DELIBERATION']
})
return {
analysis: this.extractAnalysis(deliberationResponse),
decomposition: this.extractDecomposition(deliberationResponse),
approvalStrategy: this.extractApprovalStrategy(deliberationResponse),
imageAnalysis: images ? this.extractImageAnalysis(deliberationResponse) : null
}
}
}
Phase 2: RESEARCH with Auto-Approved Exploration​
class OpenAIResearchManager {
async executeResearchPhase(
session: OpenAISession,
knowledgeGaps: string[]
): Promise<ResearchResult> {
const researchPrompt = `PHASE: RESEARCH
KNOWLEDGE GAPS: ${knowledgeGaps.join(', ')}
Research strategy using auto-approved operations:
1. Use read_file and list_directory to explore existing codebase
2. Use analyze_code to understand current implementations
3. Generate documentation and explanations for complex parts
4. Request approval only for external data fetching or API calls
Research approach:
- Leverage auto-approved file operations for rapid exploration
- Analyze existing patterns and architectures
- Document findings without requiring approval
- Request approval for external research or risky operations
Begin comprehensive research using available auto-approved functions:`
// Execute research with heavy use of auto-approved operations
const researchMessages = [
...session.state.messages,
{ role: "user", content: researchPrompt }
]
const researchResults: ResearchFinding[] = []
// Research each knowledge gap with auto-approved operations
for (const gap of knowledgeGaps) {
const gapResearch = await session.researchWithAutoApproval({
topic: gap,
autoApprovedOps: ["read_file", "list_directory", "analyze_code"],
maxAutoOps: 20 // Limit to prevent runaway execution
})
researchResults.push({
topic: gap,
findings: gapResearch.findings,
sources: gapResearch.sources,
confidence: gapResearch.confidence,
autoOpsUsed: gapResearch.autoOpsUsed
})
}
// Synthesize research findings
const synthesis = await session.synthesizeResearch(researchResults)
return {
findings: researchResults,
synthesis: synthesis,
autoApprovedOpsUsed: researchResults.reduce((acc, r) => acc + r.autoOpsUsed, 0),
manualApprovalsRequired: synthesis.riskOperations.length
}
}
}
Phase 3: PLAN with Intelligent Batch Planning​
class OpenAIPlanManager {
async executePlanPhase(
session: OpenAISession,
researchResults: ResearchResult[]
): Promise<PlanResult> {
// Update session state for planning phase
await session.stateManager.updateState({
phase: 'PLAN',
research_completed: true,
research_findings: researchResults,
planning_mode: 'intelligent_batch_specification'
})
const planPrompt = `PHASE: PLAN - Intelligent Implementation Specification
RESEARCH FINDINGS:
${researchResults.map(r => `- ${r.topic}: ${r.findings} (confidence: ${r.confidence})`).join('\n')}
CREATE COMPREHENSIVE IMPLEMENTATION PLAN optimized for OpenAI's selective auto-approval:
## 1. System Architecture Design
Based on research findings, design the complete system:
- Component relationships and data flow
- Technology stack decisions with auto-approval optimization
- Integration patterns optimized for batch operations
- Image processing integration points if applicable
- API design and external service connections
## 2. Selective Auto-Approval Implementation Strategy
Break development into phases optimized for efficient approval workflow:
- Phase 1: Foundation (auto-approved analysis + batch file creation)
- Phase 2: Business Logic (mixed auto-approved reads + batch writes)
- Phase 3: Integration (batch API creation + auto-approved testing)
- Phase 4: UI/Frontend (image processing + batch component creation)
- Phase 5: Polish (auto-approved analysis + batch deployment)
For each phase, specify:
- Auto-approved operations (reads, analysis, documentation)
- Batch approval operations (writes, installs, commands)
- Risk assessment and operation classification
- Dependencies and approval gate optimization
- Image processing requirements if applicable
## 3. Batch Approval Optimization Plan
Optimize human oversight efficiency:
- Group related file operations for single approval
- Batch configuration changes and installations
- Separate high-risk from low-risk operations
- Plan approval checkpoints at logical boundaries
- Estimate approval overhead and optimization opportunities
## 4. Auto-Approval Strategy
Maximize workflow efficiency through intelligent classification:
- Safe operations: file reads, code analysis, documentation generation
- Moderate operations: configuration reads, dependency analysis
- High-risk operations: file writes, command execution, external calls
- Batch grouping strategies for efficient approval cycles
## 5. Image Processing Integration Plan
If visual components are involved:
- Design-to-code generation workflows
- Visual requirement extraction strategies
- UI component generation from mockups
- Auto-approved design analysis processes
## 6. Context Management Strategy
Handle stateless session requirements:
- Context compression and reconstruction plans
- Session state persistence strategies
- Conversation history management
- Token budget optimization across phases
Request approval to proceed to ACTION phase with optimized batch approval plan.
`
const planMessages = [
...session.state.messages,
{ role: "user", content: planPrompt }
]
const planResponse = await session.getCompletion({
messages: planMessages,
functions: await session.approvalIntegration.registerApprovalFunctions(),
function_call: "auto"
})
// Parse comprehensive implementation plan
const implementationPlan = this.parseIntelligentImplementationPlan(planResponse)
// Update session state with complete plan
await session.stateManager.updateState({
phase: 'PLAN',
implementation_plan: implementationPlan,
planning_complete: true,
batch_approval_strategy: implementationPlan.batchApprovalStrategy,
auto_approval_optimization: implementationPlan.autoApprovalOptimization,
ready_for_action: true,
next_steps: ['Request approval for ACTION phase', 'Begin batch-optimized implementation']
})
return {
plan: implementationPlan,
architecture: implementationPlan.architecture,
phases: implementationPlan.phases,
batchApprovalStrategy: implementationPlan.batchApprovalStrategy,
autoApprovalOptimization: implementationPlan.autoApprovalOptimization,
contextStrategy: implementationPlan.contextStrategy,
imageProcessingPlan: implementationPlan.imageProcessingPlan
}
}
parseIntelligentImplementationPlan(planResponse: string): IntelligentImplementationPlan {
return {
phases: this.extractOptimizedPhases(planResponse),
architecture: this.extractSystemDesign(planResponse),
batchApprovalStrategy: {
batchGroups: this.extractBatchGroups(planResponse),
approvalCheckpoints: this.extractApprovalCheckpoints(planResponse),
riskClassification: this.extractRiskClassification(planResponse),
efficiencyTargets: this.extractEfficiencyTargets(planResponse)
},
autoApprovalOptimization: {
safeOperations: this.extractSafeOperations(planResponse),
batchCandidates: this.extractBatchCandidates(planResponse),
workflowEfficiency: this.extractWorkflowEfficiency(planResponse)
},
contextStrategy: {
compressionStrategy: this.extractCompressionStrategy(planResponse),
sessionReconstruction: this.extractReconstructionStrategy(planResponse),
tokenOptimization: this.extractTokenOptimization(planResponse)
},
imageProcessingPlan: this.extractImageProcessingPlan(planResponse),
timeline: this.extractOptimizedTimeline(planResponse),
successCriteria: this.extractIntelligentSuccessCriteria(planResponse)
}
}
}
interface IntelligentImplementationPlan extends ImplementationPlan {
batchApprovalStrategy: {
batchGroups: BatchGroup[]
approvalCheckpoints: ApprovalCheckpoint[]
riskClassification: RiskClassification[]
efficiencyTargets: EfficiencyTarget[]
}
autoApprovalOptimization: {
safeOperations: string[]
batchCandidates: BatchCandidate[]
workflowEfficiency: WorkflowEfficiency
}
contextStrategy: {
compressionStrategy: CompressionStrategy
sessionReconstruction: ReconstructionStrategy
tokenOptimization: TokenOptimization
}
imageProcessingPlan?: ImageProcessingPlan
}
Phase 4: ACTION with Batch Approval Strategy​
class OpenAIActionManager {
async executeActionPhase(session: OpenAISession, plan: ImplementationPlan): Promise<ActionResult> {
const actionPrompt = `PHASE: ACTION - Implementation
PLAN: ${JSON.stringify(plan, null, 2)}
Implementation strategy:
- Use batch_operations function for efficient approval of write operations
- Auto-approve safe operations (reading, analyzing existing code)
- Group related file operations for single approval cycle
- Leverage image processing capabilities if UI/visual components involved
Batch approval strategy:
- Group file creations by module/component
- Batch configuration changes
- Request approval for command execution and package installations
- Auto-approve documentation generation and code analysis
Begin implementation with optimized approval workflow:`
const artifacts: CodeArtifact[] = []
const batches = this.createOperationBatches(plan)
for (const batch of batches) {
// Separate auto-approved from approval-required operations
const autoOps = batch.operations.filter(op =>
session.config.autoApproval.safeOperations.includes(op.type)
)
const approvalOps = batch.operations.filter(op =>
session.config.autoApproval.requiresApproval.includes(op.type)
)
// Execute auto-approved operations immediately
const autoResults = await Promise.all(
autoOps.map(op => session.executeAutoApproved(op))
)
// Batch approval for risky operations
if (approvalOps.length > 0) {
const batchApproval = await session.callFunction('batch_operations', {
operations: approvalOps,
batch_justification: `Implementing ${batch.description}: ${approvalOps.length} file operations`
})
if (!batchApproval.success) {
throw new Error(`Batch approval failed: ${batchApproval.message}`)
}
}
// Execute approved operations
const approvalResults = await Promise.all(
approvalOps.map(op => session.executeApproved(op))
)
// Collect artifacts from batch
const batchArtifacts = [...autoResults, ...approvalResults]
.filter(result => result.artifact)
.map(result => result.artifact)
artifacts.push(...batchArtifacts)
// Update session state after each batch
await session.stateManager.updateState({
completedBatches: [...session.state.completedBatches, batch.id],
activeArtifacts: artifacts.map(a => a.filename)
})
}
return {
artifacts: artifacts,
batchesExecuted: batches.length,
autoApprovedOps: batches.reduce((acc, b) => acc + b.autoOpsCount, 0),
manualApprovals: batches.reduce((acc, b) => acc + b.approvalOpsCount, 0),
efficiency: this.calculateEfficiency(batches)
}
}
// Optimize operation batching for approval efficiency
createOperationBatches(plan: ImplementationPlan): OperationBatch[] {
const batches: OperationBatch[] = []
// Group by component/module
const componentGroups = this.groupOperationsByComponent(plan.operations)
for (const [component, operations] of componentGroups.entries()) {
// Further group by risk level for optimal batching
const riskGroups = this.groupOperationsByRisk(operations)
for (const [risk, ops] of riskGroups.entries()) {
batches.push({
id: `${component}-${risk}`,
description: `${component} ${risk} operations`,
operations: ops,
autoOpsCount: ops.filter(op => this.isAutoApproved(op.type)).length,
approvalOpsCount: ops.filter(op => !this.isAutoApproved(op.type)).length
})
}
}
return batches
}
}
Image Processing Integration​
Visual Analysis for Development​
class OpenAIImageProcessor {
async processDesignImages(images: ImageInput[], requirements: string): Promise<VisualAnalysis> {
const analysis = await this.session.callFunction('execute_operation', {
operation_type: 'process_image',
operation_details: {
analysis_type: 'ui_design',
requirements: requirements
},
risk_assessment: 'safe',
justification: 'Analyzing design images to extract UI requirements',
image_data: images.map(img => img.base64Data)
})
return {
components: this.extractUIComponents(analysis),
layout: this.extractlayoutStructure(analysis),
styling: this.extractStylingRequirements(analysis),
interactions: this.extractInteractionPatterns(analysis)
}
}
async generateCodeFromDesign(visualAnalysis: VisualAnalysis): Promise<GeneratedCode> {
// Auto-approved code generation based on visual analysis
const codeGeneration = await this.session.callFunction('execute_operation', {
operation_type: 'generate_code',
operation_details: {
generation_type: 'ui_components',
visual_analysis: visualAnalysis,
target_framework: 'react'
},
risk_assessment: 'safe',
justification: 'Generating UI code from visual analysis'
})
return codeGeneration.generated_code
}
}
Implementation Timeline​
Phase 1: Core Integration (6 weeks)​
- Develop OpenAI function calling adapter with selective auto-approval
- Implement client-side session state management
- Create stateless session reconstruction system
Phase 2: Auto-Approval System (4 weeks)​
- Build intelligent operation classification
- Implement batch approval optimization
- Create risk assessment automation
Phase 3: Image Processing (3 weeks)​
- Integrate image analysis capabilities
- Build visual-to-code generation workflows
- Implement design analysis automation
Phase 4: Performance Optimization (5 weeks)​
- Optimize context compression strategies
- Implement efficient polling mechanisms
- Build session recovery and continuation systems
Success Metrics​
- Auto-Approval Efficiency: 70%+ of operations auto-approved for faster workflow
- Batch Approval Optimization: 50% reduction in human approval interactions
- Session Reconstruction: 95%+ accuracy in stateless session recovery
- Image Processing Integration: Support for visual design-to-code workflows
This adaptation leverages OpenAI CLI's unique auto-approval capabilities and image processing while maintaining essential human oversight through selective approval integration with HumanLayer's infrastructure.