ADR-018: Local Filesystem Integration

Status: Accepted Date: 2025-10-06 Deciders: Development Team, Architecture Team Related: ADR-017 (WebSocket), ADR-006 (OPFS), ADR-004 (FoundationDB)

Context

The AZ1.AI llm IDE needs access to the user's local filesystem for:

Project files (read, write, delete, rename)
File watching (detect external changes)
Directory operations (create, delete, move)
Large file handling (GB+ files, streaming)
Multi-session isolation (session-specific workspaces)
Offline capability (work without backend connection)

Current State

Eclipse theia has built-in filesystem provider (abstract)
No implementation for browser-based local filesystem
OPFS (Origin Private File System) available in modern browsers
Native File System Access API for Chrome/Edge
No integration with backend filesystem

Requirements

Local Access: Read/write files on user's local machine
Browser Storage: Cache files in OPFS for offline work
Synchronization: Sync between local FS and browser storage
Performance: Efficient for large files, minimal latency
Security: Sandboxed, user permission-based
Cross-Platform: Works on Windows, macOS, Linux

Decision

We will implement a hybrid filesystem architecture with multiple storage backends:

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Browser (theia Frontend)                 │
│                                                             │
│  ┌───────────────────────────────────────────────────────┐ │
│  │          Unified Filesystem Service                   │ │
│  │  ┌──────────────────────────────────────────────────┐ │ │
│  │  │     File System Provider (theia API)             │ │ │
│  │  └──────────────────────────────────────────────────┘ │ │
│  │                         │                              │ │
│  │         ┌───────────────┼───────────────┐             │ │
│  │         │               │               │             │ │
│  │         ▼               ▼               ▼             │ │
│  │  ┌──────────┐   ┌─────────────┐  ┌────────────┐     │ │
│  │  │   OPFS   │   │File System  │  │  Backend   │     │ │
│  │  │ Provider │   │Access API   │  │   HTTP     │     │ │
│  │  │(Fallback)│   │(Preferred)  │  │(WebSocket) │     │ │
│  │  └──────────┘   └─────────────┘  └────────────┘     │ │
│  └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                           │
                           │ WebSocket (filesystem/ methods)
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                    Backend (Node.js)                        │
│                                                             │
│  ┌───────────────────────────────────────────────────────┐ │
│  │        Filesystem Service                             │ │
│  │  ┌──────────────────────────────────────────────────┐ │ │
│  │  │         Node.js fs/promises                      │ │ │
│  │  └──────────────────────────────────────────────────┘ │ │
│  │                         │                              │ │
│  │                         ▼                              │ │
│  │              Local Filesystem (OS)                     │ │
│  │          /workspace/user-projects/                     │ │
│  └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                           │
                           ▼
                    ┌─────────────┐
                    │FoundationDB │
                    │(File Metadata)│
                    └─────────────┘

Storage Hierarchy

Tier 1: File System Access API (Chrome/Edge, preferred)

Direct access to user's local filesystem
Full read/write permissions with user consent
Handles files of any size
Survives browser restart

Tier 2: Backend WebSocket (all browsers)

Backend has access to local filesystem
Files transferred via WebSocket
Chunked transfer for large files
Session-specific workspaces

Tier 3: OPFS (modern browsers, fallback)

Browser-native file storage
Isolated from OS filesystem
Good performance (up to several GB)
Offline-capable

Tier 4: FoundationDB (metadata only)

File metadata (path, size, modified time)
File relationships (imports, dependencies)
Session → workspace mapping
Search index

Implementation

1. Unified Filesystem Service

// src/browser/services/filesystem-service.ts

import { injectable, inject } from '@theia/core/shared/inversify';
import { FileSystemProvider, FileChangeEvent } from '@theia/filesystem/lib/common/files';

export enum FilesystemBackend {
  FileSystemAccessAPI = 'fs-access-api',
  BackendHTTP = 'backend-http',
  OPFS = 'opfs',
}

@injectable()
export class UnifiedFilesystemService implements FileSystemProvider {
  private backend: FilesystemBackend;
  private fileHandleCache = new Map<string, FileSystemFileHandle>();

  @inject(WebSocketClient)
  protected readonly ws!: WebSocketClient;

  async initialize(): Promise<void> {
    // Detect best available backend
    if (await this.isFileSystemAccessAPIAvailable()) {
      this.backend = FilesystemBackend.FileSystemAccessAPI;
      console.log('Using File System Access API');
    } else if (this.ws.isConnected()) {
      this.backend = FilesystemBackend.BackendHTTP;
      console.log('Using Backend HTTP filesystem');
    } else if (await this.isOPFSAvailable()) {
      this.backend = FilesystemBackend.OPFS;
      console.log('Using OPFS filesystem');
    } else {
      throw new Error('No filesystem backend available');
    }
  }

  // theia FileSystemProvider implementation
  async readFile(uri: URI): Promise<Uint8Array> {
    switch (this.backend) {
      case FilesystemBackend.FileSystemAccessAPI:
        return this.readFileFromFSAccessAPI(uri);
      case FilesystemBackend.BackendHTTP:
        return this.readFileFromBackend(uri);
      case FilesystemBackend.OPFS:
        return this.readFileFromOPFS(uri);
    }
  }

  async writeFile(uri: URI, content: Uint8Array): Promise<void> {
    switch (this.backend) {
      case FilesystemBackend.FileSystemAccessAPI:
        await this.writeFileToFSAccessAPI(uri, content);
        break;
      case FilesystemBackend.BackendHTTP:
        await this.writeFileToBackend(uri, content);
        break;
      case FilesystemBackend.OPFS:
        await this.writeFileToOPFS(uri, content);
        break;
    }

    // Always sync metadata to FoundationDB
    await this.syncMetadataToFDB(uri, content);
  }

  async delete(uri: URI): Promise<void> {
    switch (this.backend) {
      case FilesystemBackend.FileSystemAccessAPI:
        await this.deleteFromFSAccessAPI(uri);
        break;
      case FilesystemBackend.BackendHTTP:
        await this.deleteFromBackend(uri);
        break;
      case FilesystemBackend.OPFS:
        await this.deleteFromOPFS(uri);
        break;
    }

    await this.deleteMetadataFromFDB(uri);
  }

  async readdir(uri: URI): Promise<[string, FileType][]> {
    switch (this.backend) {
      case FilesystemBackend.FileSystemAccessAPI:
        return this.readdirFromFSAccessAPI(uri);
      case FilesystemBackend.BackendHTTP:
        return this.readdirFromBackend(uri);
      case FilesystemBackend.OPFS:
        return this.readdirFromOPFS(uri);
    }
  }

  async watch(uri: URI): Promise<Disposable> {
    // File watching implementation
    switch (this.backend) {
      case FilesystemBackend.FileSystemAccessAPI:
        return this.watchFSAccessAPI(uri);
      case FilesystemBackend.BackendHTTP:
        return this.watchViaBackend(uri);
      case FilesystemBackend.OPFS:
        return this.watchOPFS(uri);
    }
  }

  // File System Access API implementation
  private async isFileSystemAccessAPIAvailable(): Promise<boolean> {
    return 'showDirectoryPicker' in window;
  }

  private async readFileFromFSAccessAPI(uri: URI): Promise<Uint8Array> {
    const fileHandle = await this.getFileHandle(uri.path);
    const file = await fileHandle.getFile();
    const arrayBuffer = await file.arrayBuffer();
    return new Uint8Array(arrayBuffer);
  }

  private async writeFileToFSAccessAPI(uri: URI, content: Uint8Array): Promise<void> {
    const fileHandle = await this.getFileHandle(uri.path, { create: true });
    const writable = await fileHandle.createWritable();
    await writable.write(content);
    await writable.close();
  }

  private async getFileHandle(path: string, options?: { create?: boolean }): Promise<FileSystemFileHandle> {
    if (this.fileHandleCache.has(path)) {
      return this.fileHandleCache.get(path)!;
    }

    // Request directory access if not already granted
    const dirHandle = await window.showDirectoryPicker();
    const pathParts = path.split('/').filter(p => p);

    let currentHandle: FileSystemDirectoryHandle | FileSystemFileHandle = dirHandle;

    for (let i = 0; i < pathParts.length; i++) {
      const part = pathParts[i];
      const isLastPart = i === pathParts.length - 1;

      if (isLastPart) {
        currentHandle = await (currentHandle as FileSystemDirectoryHandle).getFileHandle(part, options);
      } else {
        currentHandle = await (currentHandle as FileSystemDirectoryHandle).getDirectoryHandle(part, { create: true });
      }
    }

    const fileHandle = currentHandle as FileSystemFileHandle;
    this.fileHandleCache.set(path, fileHandle);
    return fileHandle;
  }

  // Backend HTTP implementation
  private async readFileFromBackend(uri: URI): Promise<Uint8Array> {
    const result = await this.ws.request('filesystem/read', {
      path: uri.path,
      encoding: 'base64'
    });

    return this.base64ToUint8Array(result.content);
  }

  private async writeFileToBackend(uri: URI, content: Uint8Array): Promise<void> {
    const base64 = this.uint8ArrayToBase64(content);

    await this.ws.request('filesystem/write', {
      path: uri.path,
      content: base64,
      encoding: 'base64'
    });
  }

  private async deleteFromBackend(uri: URI): Promise<void> {
    await this.ws.request('filesystem/delete', { path: uri.path });
  }

  private async readdirFromBackend(uri: URI): Promise<[string, FileType][]> {
    const result = await this.ws.request('filesystem/readdir', { path: uri.path });
    return result.entries;
  }

  private async watchViaBackend(uri: URI): Promise<Disposable> {
    await this.ws.request('filesystem/watch', { path: uri.path });

    const listener = this.ws.onMessage((message) => {
      if (message.method === 'filesystem/changed' && message.params.path === uri.path) {
        this.fireFileChangeEvent({
          type: message.params.changeType,
          resource: uri
        });
      }
    });

    return {
      dispose: () => {
        listener.dispose();
        this.ws.notify('filesystem/unwatch', { path: uri.path });
      }
    };
  }

  // OPFS implementation
  private async isOPFSAvailable(): Promise<boolean> {
    return 'storage' in navigator && 'getDirectory' in navigator.storage;
  }

  private async readFileFromOPFS(uri: URI): Promise<Uint8Array> {
    const root = await navigator.storage.getDirectory();
    const fileHandle = await this.getOPFSFileHandle(root, uri.path);
    const file = await fileHandle.getFile();
    const arrayBuffer = await file.arrayBuffer();
    return new Uint8Array(arrayBuffer);
  }

  private async writeFileToOPFS(uri: URI, content: Uint8Array): Promise<void> {
    const root = await navigator.storage.getDirectory();
    const fileHandle = await this.getOPFSFileHandle(root, uri.path, { create: true });
    const writable = await fileHandle.createWritable();
    await writable.write(content);
    await writable.close();
  }

  private async deleteFromOPFS(uri: URI): Promise<void> {
    const root = await navigator.storage.getDirectory();
    const pathParts = uri.path.split('/').filter(p => p);
    const fileName = pathParts.pop()!;

    let dirHandle = root;
    for (const part of pathParts) {
      dirHandle = await dirHandle.getDirectoryHandle(part);
    }

    await dirHandle.removeEntry(fileName);
  }

  private async getOPFSFileHandle(
    root: FileSystemDirectoryHandle,
    path: string,
    options?: { create?: boolean }
  ): Promise<FileSystemFileHandle> {
    const pathParts = path.split('/').filter(p => p);
    const fileName = pathParts.pop()!;

    let dirHandle = root;
    for (const part of pathParts) {
      dirHandle = await dirHandle.getDirectoryHandle(part, { create: options?.create });
    }

    return dirHandle.getFileHandle(fileName, options);
  }

  // FoundationDB metadata sync
  private async syncMetadataToFDB(uri: URI, content: Uint8Array): Promise<void> {
    await this.fdbService.set(`file:${uri.path}`, {
      path: uri.path,
      size: content.length,
      modified: Date.now(),
      checksum: await this.calculateChecksum(content)
    });
  }

  private async deleteMetadataFromFDB(uri: URI): Promise<void> {
    await this.fdbService.delete(`file:${uri.path}`);
  }

  // Utility methods
  private base64ToUint8Array(base64: string): Uint8Array {
    const binary = atob(base64);
    const bytes = new Uint8Array(binary.length);
    for (let i = 0; i < binary.length; i++) {
      bytes[i] = binary.charCodeAt(i);
    }
    return bytes;
  }

  private uint8ArrayToBase64(bytes: Uint8Array): string {
    let binary = '';
    for (let i = 0; i < bytes.length; i++) {
      binary += String.fromCharCode(bytes[i]);
    }
    return btoa(binary);
  }

  private async calculateChecksum(content: Uint8Array): Promise<string> {
    const hashBuffer = await crypto.subtle.digest('SHA-256', content);
    const hashArray = Array.from(new Uint8Array(hashBuffer));
    return hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
  }
}

2. Backend Filesystem Service

// src/backend/services/filesystem-service.ts

import { promises as fs } from 'fs';
import path from 'path';
import chokidar from 'chokidar';

export class BackendFilesystemService {
  private workspaceRoot: string;
  private watchers = new Map<string, chokidar.FSWatcher>();

  constructor(workspaceRoot: string) {
    this.workspaceRoot = workspaceRoot;
  }

  async readFile(filePath: string): Promise<Buffer> {
    const fullPath = this.resolvePath(filePath);
    return fs.readFile(fullPath);
  }

  async writeFile(filePath: string, content: Buffer): Promise<void> {
    const fullPath = this.resolvePath(filePath);
    await fs.mkdir(path.dirname(fullPath), { recursive: true });
    await fs.writeFile(fullPath, content);
  }

  async deleteFile(filePath: string): Promise<void> {
    const fullPath = this.resolvePath(filePath);
    await fs.unlink(fullPath);
  }

  async readDirectory(dirPath: string): Promise<Array<[string, 'file' | 'directory']>> {
    const fullPath = this.resolvePath(dirPath);
    const entries = await fs.readdir(fullPath, { withFileTypes: true });

    return entries.map(entry => [
      entry.name,
      entry.isDirectory() ? 'directory' : 'file'
    ]);
  }

  watchFile(filePath: string, callback: (event: string) => void): () => void {
    const fullPath = this.resolvePath(filePath);

    const watcher = chokidar.watch(fullPath, {
      persistent: true,
      ignoreInitial: true
    });

    watcher
      .on('change', () => callback('changed'))
      .on('unlink', () => callback('deleted'));

    this.watchers.set(filePath, watcher);

    return () => {
      watcher.close();
      this.watchers.delete(filePath);
    };
  }

  unwatchFile(filePath: string): void {
    const watcher = this.watchers.get(filePath);
    if (watcher) {
      watcher.close();
      this.watchers.delete(filePath);
    }
  }

  private resolvePath(filePath: string): string {
    // Security: prevent path traversal
    const resolved = path.resolve(this.workspaceRoot, filePath);
    if (!resolved.startsWith(this.workspaceRoot)) {
      throw new Error('Invalid path: path traversal not allowed');
    }
    return resolved;
  }
}

3. Session-Specific workspaces

// src/browser/services/workspace-service.ts

@injectable()
export class workspaceService {
  @inject(UnifiedFilesystemService)
  protected readonly fs!: UnifiedFilesystemService;

  @inject(FDBService)
  protected readonly fdb!: FDBService;

  async createSessionworkspace(sessionId: string): Promise<string> {
    const workspacePath = `/sessions/${sessionId}`;

    // Create workspace directory
    await this.fs.mkdir(workspacePath);

    // Save session → workspace mapping in FDB
    await this.fdb.set(`session:${sessionId}:workspace`, workspacePath);

    return workspacePath;
  }

  async getSessionworkspace(sessionId: string): Promise<string | null> {
    return this.fdb.get(`session:${sessionId}:workspace`);
  }

  async deleteSessionworkspace(sessionId: string): Promise<void> {
    const workspacePath = await this.getSessionworkspace(sessionId);
    if (workspacePath) {
      await this.fs.delete(workspacePath);
      await this.fdb.delete(`session:${sessionId}:workspace`);
    }
  }
}

Rationale

Why Hybrid Architecture?

Browser Capability Variation:

✅ File System Access API: Best UX, but Chrome/Edge only
✅ Backend WebSocket: Works everywhere, but requires backend
✅ OPFS: Good fallback, offline-capable, but isolated

Graceful Degradation:

✅ Try best option first (FS Access API)
✅ Fall back to backend
✅ Use OPFS for offline work

Why FoundationDB for Metadata?

Performance:

✅ Fast lookups (file search, imports)
✅ Transaction support (atomic operations)
✅ Distributed (scales horizontally)

Features:

✅ Secondary indexes (search by name, date, size)
✅ Relationships (file dependencies)
✅ Session isolation (per-session metadata)

Why Not Store Full Files in FDB?

Size Limitations:

❌ FDB values limited to 100KB
❌ Large files (MB+) would require chunking
❌ High storage cost vs filesystem

Better Alternatives:

✅ Filesystem optimized for large files
✅ OS-level caching
✅ FDB only for metadata

Alternatives Considered

Alternative 1: OPFS Only

Pros:

Simple (single backend)
Offline-capable
No permissions needed

Cons:

❌ Isolated from OS filesystem (can't open local files)
❌ Limited to browser storage quota
❌ No integration with local tools

Rejected: Too limited for IDE use case

Alternative 2: Backend Only

Pros:

Full filesystem access
Works everywhere
No browser limitations

Cons:

❌ Requires backend connection
❌ No offline work
❌ Higher latency

Rejected: Poor offline UX

Alternative 3: IndexedDB

Pros:

Wide browser support
Good for structured data

Cons:

❌ Not designed for files
❌ Poor performance for large files
❌ Complex API

Rejected: OPFS is better for files

Consequences

Positive

✅ Best UX: Use native FS API when available ✅ Offline Capable: OPFS fallback for offline work ✅ Universal: Backend works in all browsers ✅ Performance: Metadata in FDB, files in FS ✅ Secure: Sandboxed, permission-based ✅ Session Isolation: Per-session workspaces

Negative

❌ Complexity: 3 different backends to maintain ❌ Sync Issues: Must keep FDB metadata in sync ❌ Permission UX: Users must grant FS access ❌ Browser Support: FS Access API limited to Chrome/Edge

Mitigation

Complexity:

Abstract behind unified interface
Comprehensive tests for each backend
Document backend selection logic

Sync Issues:

Use checksums to detect drift
Background sync job
Conflict resolution UI

Permission UX:

Clear onboarding flow
Explain why access needed
Remember permissions

Browser Support:

Feature detection
Graceful fallback
Document browser requirements

Implementation Plan

Phase 1: Core Infrastructure ✅

Unified filesystem service interface
Backend selection logic
FoundationDB metadata schema

Phase 2: File System Access API 🔲

Request directory access
File handle caching
Read/write implementation
Directory listing

Phase 3: Backend Integration 🔲

WebSocket filesystem methods
Chunked file transfer
File watching via chokidar
Security (path traversal prevention)

Phase 4: OPFS Fallback 🔲

OPFS detection
Read/write implementation
Directory operations
Quota management

Phase 5: Metadata Sync 🔲

FDB schema for file metadata
Sync on read/write
Background sync job
Conflict resolution

Phase 6: Session workspaces 🔲

Session → workspace mapping
workspace creation/deletion
Isolation enforcement
workspace switching

Success Metrics

Performance:

< 50ms read time for files < 1MB
< 200ms read time for files < 10MB
Streaming for files > 10MB

Reliability:

99.9% metadata sync accuracy
< 1% file corruption rate
Zero data loss on crashes

User Experience:

< 5s permission request flow
Clear error messages
Automatic backend selection

ADR-017: WebSocket Backend - File transfer protocol
ADR-006: OPFS - Browser storage
ADR-004: FoundationDB - Metadata storage

References

File System Access API:

OPFS:

Best Practices:

Status: ✅ Accepted Next Review: 2025-11-06 (1 month) Last Updated: 2025-10-06

Context​

Current State​

Requirements​

Decision​

Architecture​

Storage Hierarchy​

Implementation​

1. Unified Filesystem Service​

2. Backend Filesystem Service​

3. Session-Specific workspaces​

Rationale​

Why Hybrid Architecture?​

Why FoundationDB for Metadata?​

Why Not Store Full Files in FDB?​

Alternatives Considered​

Alternative 1: OPFS Only​

Alternative 2: Backend Only​

Alternative 3: IndexedDB​

Consequences​

Positive​

Negative​

Mitigation​

Implementation Plan​

Phase 1: Core Infrastructure ✅​

Phase 2: File System Access API 🔲​

Phase 3: Backend Integration 🔲​

Phase 4: OPFS Fallback 🔲​

Phase 5: Metadata Sync 🔲​

Phase 6: Session workspaces 🔲​

Success Metrics​

Related Decisions​

References​

Context

Current State

Requirements

Decision

Architecture

Storage Hierarchy

Implementation

1. Unified Filesystem Service

2. Backend Filesystem Service

3. Session-Specific workspaces

Rationale

Why Hybrid Architecture?

Why FoundationDB for Metadata?

Why Not Store Full Files in FDB?

Alternatives Considered

Alternative 1: OPFS Only

Alternative 2: Backend Only

Alternative 3: IndexedDB

Consequences

Positive

Negative

Mitigation

Implementation Plan

Phase 1: Core Infrastructure ✅

Phase 2: File System Access API 🔲

Phase 3: Backend Integration 🔲

Phase 4: OPFS Fallback 🔲

Phase 5: Metadata Sync 🔲

Phase 6: Session workspaces 🔲

Success Metrics

Related Decisions

References