Skill Health Tracker - Reference Documentation

Version: 1.0.0 Status: Active Created: 2026-01-11 Owner: CODITECT Framework Team

---

Overview

The Skill Health Framework provides continuous monitoring of component health scores, trend analysis, regression detection, and optimization recommendations. It tracks skill performance over time to ensure consistent quality across the CODITECT framework.

Components

1. Skill Health Tracker (scripts/skill-health-tracker.py)

Purpose: Core health tracking with rolling averages and trend analysis.

Features:

• Historical score storage per skill per session
• Rolling averages (7-day, 30-day)
• Trend detection (improving/stable/degrading)
• Baseline comparison
• Dashboard visualization

Usage:

# Show dashboard
python3 scripts/skill-health-tracker.py --dashboard

# View specific skill
python3 scripts/skill-health-tracker.py --skill <name>

# Record current scores
python3 scripts/skill-health-tracker.py --record

# Create baseline
python3 scripts/skill-health-tracker.py --baseline

# Compare to baseline
python3 scripts/skill-health-tracker.py --compare

# Export data
python3 scripts/skill-health-tracker.py --export --json

2. Regression Detector (hooks/skill-regression-detector.py)

Purpose: Detect skill health regressions after retrospectives.

Features:

• Compares current scores to baseline
• Flags skills with >10% score drop
• Generates regression reports
• Auto-creates improvement tasks (optional)

Usage:

# Check for regressions
python3 hooks/skill-regression-detector.py --check

# Generate improvement tasks
python3 hooks/skill-regression-detector.py --auto-task

# Custom threshold
python3 hooks/skill-regression-detector.py --threshold 15

# Save report
python3 hooks/skill-regression-detector.py --save-report

3. /skill-health Command (commands/skill-health.md)

Purpose: User-facing command for health dashboard.

Usage:

/skill-health                      # Dashboard view
/skill-health --skill <name>       # Specific skill
/skill-health --compare            # Baseline comparison
/skill-health --record             # Record current scores

4. /which Health Integration (commands/which.md)

Purpose: Show health scores in agent recommendations.

Features:

• Health score displayed in output: [XX%] ↑/↓/→
• Health warning for <50% agents
• Health-aware routing (health weighted 40% in ranking)

---

Data Storage

skill-health.json

Location: context-storage/skill-health.json

Schema:

{
  "version": "1.0.0",
  "last_updated": "2026-01-11T12:00:00Z",
  "skills": {
    "skill-name": {
      "scores": [
        {
          "date": "2026-01-11T12:00:00Z",
          "score": 85,
          "invocations": 42
        }
      ],
      "rolling_7d": 82.5,
      "rolling_30d": 78.3,
      "trend": "improving",
      "total_invocations": 156,
      "first_tracked": "2026-01-01T00:00:00Z",
      "last_updated": "2026-01-11T12:00:00Z"
    }
  }
}

Baseline Snapshots

Location: context-storage/baselines/skill-baseline-YYYY-MM-DD.json

Schema:

{
  "version": "1.0.0",
  "created": "2026-01-11T12:00:00Z",
  "name": "2026-01-11",
  "skills": {
    "skill-name": {
      "score": 85,
      "invocations": 42,
      "date": "2026-01-11T12:00:00Z",
      "trend": "stable"
    }
  }
}

Regression Reports

Location: context-storage/regressions/regression-report-YYYYMMDD_HHMMSS.json

Schema:

{
  "status": "complete",
  "baseline_name": "2026-01-01",
  "threshold": 10,
  "checked_at": "2026-01-11T12:00:00Z",
  "regressions": [
    {
      "skill": "problem-skill",
      "baseline_score": 85,
      "current_score": 70,
      "delta": -15,
      "severity": "warning"
    }
  ],
  "warnings": [],
  "improvements": [],
  "summary": {
    "total_checked": 150,
    "regression_count": 3,
    "warning_count": 5,
    "improvement_count": 12
  }
}

---

Health Scoring

Score Calculation

Health scores are calculated from skill invocation history:

health_score = (
    success_rate * 0.5 +      # 50% weight: successful invocations / total
    trend_score * 0.3 +       # 30% weight: recent score trajectory
    (1 - error_penalty) * 0.2 # 20% weight: deduction for errors
)

Status Levels

Score	Status	Icon	Action
90-100	Excellent	🟢	Maintain
70-89	Good	🟡	Monitor
50-69	Needs Work	🟠	Improve
0-49	Critical	🔴	Urgent Fix

Trend Detection

Trend	Criteria	Icon
Improving	Score rising >5%	↑
Stable	Score within ±5%	→
Degrading	Score falling >5%	↓

---

Workflow Integration

Session Retrospective Flow

Session End
    │
    ▼
┌─────────────────────┐
│ Run Retrospective   │
│ (session-           │
│  retrospective.py)  │
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ Record Health       │
│ (skill-health-      │
│  tracker.py)        │
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ Check Regressions   │
│ (skill-regression-  │
│  detector.py)       │
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ Alert if            │
│ Regressions Found   │
└─────────────────────┘

Recommended Schedule

Action	Frequency	Command
View dashboard	Daily	/skill-health
Record scores	After retrospective	--record
Check regressions	Weekly	--compare
Create baseline	Before major changes	--baseline
Full export	Monthly	--export

---

Hook Configuration

Add to ~/.claude/settings.json:

{
  "hooks": {
    "PostToolUse:retrospective": [
      {
        "command": "python3 ~/.coditect/hooks/skill-regression-detector.py --check --save-report",
        "timeout": 30000
      }
    ]
  }
}

---

Related Components

Component	Type	Purpose
skill-pattern-analyzer.py	Script	Analyze patterns, generate recommendations
session-retrospective.py	Hook	End-of-session analysis
skill-learnings.json	Data	Source of skill invocation history
/optimize-skills	Command	Alias for /skill-health
/which	Command	Agent recommendations with health

---

Troubleshooting

No Health Data Available

# Record initial scores
python3 scripts/skill-health-tracker.py --record

# Verify data file exists
ls -la context-storage/skill-health.json

No Baseline Found

# Create baseline
python3 scripts/skill-health-tracker.py --baseline

# Verify baseline exists
ls -la context-storage/baselines/

Scores Not Updating

# Check skill-learnings.json has data
cat context-storage/skill-learnings.json | python3 -m json.tool | head -50

# Run retrospective to populate learnings
python3 hooks/session-retrospective.py --analyze

---

ADR References

• F.5.6: Skill Health Framework Enhancement (PILOT Plan)
• F.5.5: Component Success Rate Improvement (completed prerequisite)
• ADR-061: Skill Improvement Loop (related)

---

Last Updated: 2026-01-11 Maintained By: CODITECT Framework Team