ADR-068: Large Project Plan Token Economics

Status

Accepted - January 13, 2026

Context

The Problem

Large project planning documents in CODITECT (e.g., PILOT-PARALLEL-EXECUTION-PLAN.md) grow beyond practical limits for AI-assisted development:

Metric	PILOT Plan (Current)	Impact
File Size	380+ KB	Exceeds 256KB single-read limit
Estimated Tokens	~95,000	47% of 200K context window
Tracks	7 (A-G)	All loaded regardless of task
Task Entries	500+	80% completed, still loaded
Progress Updates	50+	Historical data always loaded

Token Economics Impact

Loading the full PILOT plan consumes significant context budget:

Context Window: 200,000 tokens (Claude 3.5/Opus)

Current PILOT Plan Load:
├── PILOT Plan:        ~95,000 tokens (47%)
├── CLAUDE.md files:    ~8,000 tokens (4%)
├── Remaining for work: ~97,000 tokens (49%)
└── Effective capacity: REDUCED by 51%

ADR-052: Intent-Aware Context Management
ADR-053: Cloud Context Sync Architecture
ADR-054: Track Nomenclature Extensibility
ADR-062: Statusline Configuration System

Decision

Implement a Track-Based Decomposition Strategy with progressive loading and automated archival.

1. Document Structure

internal/project/plans/
├── PILOT-MASTER-INDEX.md              # ~1,500 tokens - Always loaded
├── PILOT-PROGRESS-LOG.md              # ~2,500 tokens - Always loaded
├── PILOT-CRITICAL-PATH.md             # ~1,500 tokens - Always loaded
│
├── tracks/                            # Load on demand
│   ├── TRACK-A-BACKEND.md             # ~10,000 tokens
│   ├── TRACK-B-FRONTEND.md            # ~7,500 tokens
│   ├── TRACK-C-DEVOPS.md              # ~8,750 tokens
│   ├── TRACK-D-SECURITY.md            # ~3,750 tokens
│   ├── TRACK-E-TESTING.md             # ~5,000 tokens
│   ├── TRACK-F-DOCUMENTATION.md       # ~12,500 tokens
│   └── TRACK-G-DMS.md                 # ~6,250 tokens
│
└── archive/                           # Never loaded automatically
    └── completed/
        ├── 2026-01/                   # Monthly archives
        └── ...

2. Token Budget Allocation

Document	Size	Tokens	Load Strategy
PILOT-MASTER-INDEX.md	6 KB	~1,500	Always - Navigation hub
PILOT-PROGRESS-LOG.md	10 KB	~2,500	Always - Recent updates
PILOT-CRITICAL-PATH.md	6 KB	~1,500	Always - Blocking items
Active Track (1 of 7)	40 KB	~10,000	On demand - Task context
Session Total	~62 KB	~15,500	84% reduction

3. Progressive Loading Protocol

# /orient integration
loading_protocol:
  phase_1_always:
    - PILOT-MASTER-INDEX.md
    - PILOT-PROGRESS-LOG.md
    - PILOT-CRITICAL-PATH.md
    total_tokens: ~5,500

  phase_2_on_demand:
    trigger: task_id_detected
    pattern: "TRACK-{track_letter}-*.md"
    example: "A.9.1.1" → loads TRACK-A-BACKEND.md

  phase_3_never_auto:
    - archive/completed/*
    - Historical progress updates

4. Master Index Format

---
title: PILOT Master Index
type: index
tokens: ~1500
always_load: true
---
# PILOT Parallel Execution Plan - Master Index

## Current Sprint: January 2026

| Track | Domain | Progress | Status | Active Tasks |
|-------|--------|----------|--------|--------------|
| A | Backend | 95% | 🟡 Active | A.10.1-A.10.5 |
| B | Frontend | 85% | 🟢 Stable | B.7.1.5, B.7.2.5 |
| C | DevOps | 80% | 🟡 Active | C.5.1-C.5.4 |
| D | Security | 100% | ✅ Complete | [Archived] |
| E | Testing | 80% | 🟢 Stable | E.3.1-E.3.5 |
| F | Docs | 75% | 🟡 Active | F.10.1-F.10.3 |
| G | DMS | 0% | ⏸️ Deferred | - |

## Critical Path Items
| ID | Task | Track | Blocker? | Est. Hours |
|----|------|-------|----------|------------|
| CP-30 | Discord setup | F | No | 4h |

## Quick Navigation
- Track details: `tracks/TRACK-{A-G}-*.md`
- Progress history: `PILOT-PROGRESS-LOG.md`
- Completed work: `archive/completed/`

5. Archive Strategy

Monthly Archive Rotation:

# Archive completed sections monthly
archive_rules:
  trigger: first_of_month
  criteria:
    - track_progress >= 100%
    - all_tasks_marked_complete
    - no_references_in_critical_path
  destination: archive/completed/YYYY-MM/
  retention: 12 months

Track Completion Archival:

Track	Completion	Archive Action
D (Security)	100%	Archive immediately
A (Backend)	95%	Archive when 100%
Others	<90%	Keep active

6. Context-Aware Loading

Integration with /orient command:

# session-orient.py enhancement
def load_relevant_track(task_id: str) -> str:
    """Load only the track relevant to current task."""
    if not task_id:
        return load_master_index_only()

    track_letter = task_id.split('.')[0]  # "A.9.1.1" → "A"
    track_file = f"tracks/TRACK-{track_letter}-*.md"

    return load_files([
        "PILOT-MASTER-INDEX.md",      # Always
        "PILOT-PROGRESS-LOG.md",       # Always
        "PILOT-CRITICAL-PATH.md",      # Always
        track_file                      # On demand
    ])

7. Cross-Reference Protocol

Maintain navigability without loading all documents:

## In Track Files
<!-- Cross-reference format -->
**Related:** [Track B: B.4.5](TRACK-B-FRONTEND.md#b45-container-session-ui)
**Depends On:** [Track A: A.9.1](TRACK-A-BACKEND.md#a91-context-sync)
**Blocks:** [CP-30](../PILOT-CRITICAL-PATH.md#cp-30)

Consequences

Positive

84% Token Reduction: 95K → 15.5K tokens per session
Faster Loading: Only relevant track loaded
Better Focus: Context limited to active work
Cleaner Archives: Completed work separated
Scalable: Pattern works for 10+ tracks
Searchable: Smaller files = faster grep/search

Negative

Initial Migration Effort: ~4 hours to split existing plan
Navigation Overhead: Must reference index for cross-track work
Sync Complexity: Updates may touch multiple files
Tooling Updates: /orient, /session-log need updates

Mitigations

Risk	Mitigation
Migration effort	Automated split script
Navigation overhead	Master index always loaded
Sync complexity	Pre-commit hooks validate links
Tooling updates	Backward-compatible changes

Implementation

Phase 1: Structure Creation (2 hours)

mkdir -p internal/project/plans/tracks
mkdir -p internal/project/plans/archive/completed

Phase 2: Document Split (4 hours)

python3 scripts/split-pilot-plan.py \
  --source PILOT-PARALLEL-EXECUTION-PLAN.md \
  --output tracks/ \
  --create-index PILOT-MASTER-INDEX.md \
  --extract-progress PILOT-PROGRESS-LOG.md \
  --extract-critical PILOT-CRITICAL-PATH.md

Phase 3: Tooling Updates (4 hours)

Update /orient for progressive loading
Update /session-log for track-aware entries
Update task sync hooks for multi-file support

Phase 4: Archive Migration (2 hours)

# Archive completed Track D
mv tracks/TRACK-D-SECURITY.md archive/completed/2026-01/

Metrics

Success Criteria

Metric	Target	Measurement
Session token usage	<20K tokens	/context command
Plan load time	<2 seconds	Timing measurement
Cross-reference accuracy	100%	Link validation
Archive completeness	Monthly	Automated check

Monitoring

# Token usage check
/cxq --stats | grep "plan_tokens"

# Link validation
python3 scripts/validate-plan-links.py

References

Decision Date: January 13, 2026 Decision Makers: Hal Casteel, Claude Opus 4.5 Implementation Priority: P1 (High) Estimated Effort: 12 hours

Status​

Context​

The Problem​

Token Economics Impact​

Related Decisions​

Decision​

1. Document Structure​

2. Token Budget Allocation​

3. Progressive Loading Protocol​

4. Master Index Format​

5. Archive Strategy​

6. Context-Aware Loading​

7. Cross-Reference Protocol​

Consequences​

Positive​

Negative​

Mitigations​

Implementation​

Phase 1: Structure Creation (2 hours)​

Phase 2: Document Split (4 hours)​

Phase 3: Tooling Updates (4 hours)​

Phase 4: Archive Migration (2 hours)​

Metrics​

Success Criteria​

Monitoring​

References​