QA Self-Healing Loop - User Guide

Version: 1.0.0 Status: Production Ready Created: 2025-12-22 Pattern Origin: Inspired by Auto-Claude autonomous validation patterns

Overview

The QA Self-Healing Loop is an autonomous quality assurance system that automatically validates artifacts, detects issues, applies fixes, and escalates to humans only when truly necessary. It implements a sophisticated loop with recurring issue detection, intelligent escalation, and comprehensive state tracking.

Key Features

Autonomous Validation: Automatically runs multi-agent QA reviews
Auto-Remediation: Applies fixes using specialized QA agents
Recurring Issue Detection: Identifies and escalates persistent problems
Intelligent Escalation: Only escalates when automated fixes fail
State Tracking: Comprehensive history and iteration tracking
Configurable Thresholds: Customize loop behavior for your needs

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    QA Self-Healing Loop                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  1. Run QA Review (council-orchestrator)                    │
│     ├─ Multi-agent review council                           │
│     ├─ Consensus scoring                                    │
│     └─ Structured verdict                                   │
│                                                              │
│  2. Check Result                                            │
│     ├─ APPROVED → SUCCESS ✓                                 │
│     └─ REJECTED → Continue                                  │
│                                                              │
│  3. Detect Recurring Issues                                 │
│     ├─ Normalize issue text                                 │
│     ├─ Calculate similarity (>0.8)                          │
│     ├─ Track occurrence counts                              │
│     └─ If 3+ occurrences → ESCALATE                         │
│                                                              │
│  4. Run Fixer Agent (qa-reviewer)                           │
│     ├─ Apply targeted fixes                                 │
│     ├─ Update artifact                                      │
│     └─ Loop to Step 1                                       │
│                                                              │
│  5. Escalation Triggers                                     │
│     ├─ Recurring issues (3+ similar)                        │
│     ├─ Consecutive errors (3+)                              │
│     ├─ Max iterations (50)                                  │
│     └─ Generate human escalation report                     │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Quick Start

Basic Usage

# Validate an artifact with default settings
python3 scripts/qa-validation-loop.py path/to/artifact.md

# Dry run (preview without applying fixes)
python3 scripts/qa-validation-loop.py path/to/artifact.md --dry-run

# Verbose output
python3 scripts/qa-validation-loop.py path/to/artifact.md --verbose

# Custom configuration
python3 scripts/qa-validation-loop.py path/to/artifact.md --config custom-config.json

Return Codes

0 - QA validation passed successfully
1 - QA validation failed
2 - Human escalation required (see escalation report)

Configuration

Default Configuration

Location: config/qa-loop-config.json

{
  "loop_parameters": {
    "max_iterations": 50,
    "max_consecutive_errors": 3,
    "recurring_threshold": 3,
    "similarity_threshold": 0.8
  },
  "agents": {
    "qa_reviewer": {
      "name": "council-orchestrator",
      "min_reviewers": 4,
      "consensus_threshold": 0.6
    },
    "fixer": {
      "name": "qa-reviewer",
      "max_fix_attempts_per_issue": 3
    }
  },
  "decision_thresholds": {
    "critical_findings": 0,
    "high_findings": 3,
    "aggregate_score": 0.7,
    "consensus_score": 0.5
  },
  "escalation_triggers": {
    "recurring_issue_count": 3,
    "consecutive_errors": 3,
    "max_iterations_reached": 50
  },
  "state_tracking": {
    "history_file": ".coditect/qa-history.json",
    "max_history_entries": 1000,
    "retention_days": 90
  }
}

Custom Configuration

Create a custom config file:

cp config/qa-loop-config.json config/my-qa-config.json
# Edit thresholds as needed
python3 scripts/qa-validation-loop.py artifact.md --config config/my-qa-config.json

Key Parameters

Parameter	Default	Description
`max_iterations`	50	Maximum loop iterations before escalation
`max_consecutive_errors`	3	Consecutive errors before escalation
`recurring_threshold`	3	Occurrences needed to classify as recurring
`similarity_threshold`	0.8	Similarity score for matching issues (0.0-1.0)

Recurring Issue Detection

How It Works

Normalization: Convert issue text to lowercase, remove punctuation
Similarity Calculation: Use difflib.SequenceMatcher to compare issues
Threshold Check: If similarity > 0.8, consider issues the same
Occurrence Tracking: Count occurrences of each normalized issue
Escalation: If any issue occurs 3+ times, escalate to human

Example

Iteration 1: "Missing code example in section 3.2"
Iteration 3: "Missing code example in section 3.2!" (98% similar)
Iteration 5: "missing code example section 3 2" (85% similar)
→ ESCALATE: Issue occurred 3 times

Why This Matters

Prevents infinite loops where the fixer keeps attempting the same unsuccessful fix. If an issue recurs, it indicates a deeper problem requiring human analysis.

State Tracking

History File

Location: .coditect/qa-history.json

Structure:

{
  "version": "1.0.0",
  "runs": [
    {
      "artifact": "path/to/artifact.md",
      "timestamp": "2025-12-22T12:00:00Z",
      "total_iterations": 5,
      "result": "success",
      "iterations": [
        {
          "iteration": 1,
          "status": "rejected",
          "issues_found": 3,
          "duration_seconds": 2.5,
          "timestamp": "2025-12-22T12:00:00Z"
        }
      ]
    }
  ]
}

Viewing History

# View recent runs
cat .coditect/qa-history.json | jq '.runs[-5:]'

# Count total runs
cat .coditect/qa-history.json | jq '.runs | length'

# Average iterations per run
cat .coditect/qa-history.json | jq '[.runs[].total_iterations] | add / length'

Escalation Reports

When Escalation Occurs

Recurring Issues: Same issue detected 3+ times
Consecutive Errors: 3 consecutive QA review errors
Max Iterations: 50 iterations reached without approval
Low Consensus: Significant reviewer disagreement with blocking issues

Report Contents

Generated at: .coditect/qa-escalations/QA_ESCALATION-{artifact}-{timestamp}.md

Includes:

Summary of escalation reason
Recurring issues detected (with counts)
Complete iteration history table
Last QA review result (JSON)
Issue frequency analysis
Recommended actions
Configuration snapshot

Example Report Section

## Recurring Issues Detected

### Most Recurring Issues

- **5 occurrences:** missing code example in implementation section
- **4 occurrences:** mermaid diagram syntax error line 234
- **3 occurrences:** broken cross reference to adr 025

## Issue Frequency Analysis

- **5x:** missing code example in implementation section...
- **4x:** mermaid diagram syntax error line 234...
- **3x:** broken cross reference to adr 025...

Integration with Existing Agents

Council Orchestrator

The loop uses council-orchestrator for QA reviews:

# Conceptual integration (implemented in script)
Task(
    subagent_type="council-orchestrator",
    prompt=f"""
    Review {artifact_path} with:
    - Security reviewer
    - Compliance reviewer
    - Performance reviewer
    - Testing reviewer

    Threshold: consensus >= 0.6, no critical findings
    """
)

QA Reviewer (Fixer)

The loop uses qa-reviewer for applying fixes:

# Conceptual integration (implemented in script)
Task(
    subagent_type="qa-reviewer",
    prompt=f"""
    Fix the following issues in {artifact_path}:

    {formatted_issues}

    Apply fixes following CODITECT v4 standards.
    """
)

Advanced Usage

Custom Escalation Logic

Modify the should_escalate() method to add custom triggers:

def should_escalate(self) -> Tuple[bool, Optional[str]]:
    # Custom: Escalate if score hasn't improved in 5 iterations
    if len(self.iterations) >= 5:
        scores = [it.qa_result.score for it in self.iterations[-5:]]
        if max(scores) - min(scores) < 0.05:
            return True, "Score plateau detected"

    # ... existing escalation checks ...

Integration with CI/CD

# .github/workflows/qa-validation.yml
name: QA Validation

on: [pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run QA Loop
        run: |
          python3 scripts/qa-validation-loop.py docs/architecture/decisions/*.md

      - name: Upload Escalation Reports
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: qa-escalation-reports
          path: .coditect/qa-escalations/

Batch Processing

# Validate multiple artifacts
for file in docs/architecture/decisions/*.md; do
    python3 scripts/qa-validation-loop.py "$file" --verbose
done

# Or use parallel processing
find docs/ -name "*.md" | xargs -P 4 -I {} \
    python3 scripts/qa-validation-loop.py {} --verbose

Performance Considerations

Iteration Duration

Typical iteration: 2-5 seconds

Factors:

QA review complexity (number of reviewers)
Artifact size
Fix complexity

Resource Usage

CPU: Low (mostly I/O bound)
Memory: ~50MB per run
Disk: History file grows ~1KB per run

Optimization Tips

Reduce reviewers for faster iterations (edit config)
Increase similarity threshold to catch more recurring issues earlier
Lower recurring threshold to escalate sooner
Use --dry-run for testing configuration changes

Troubleshooting

Common Issues

Script fails immediately:

# Check if config file exists
ls -l config/qa-loop-config.json

# Validate JSON syntax
python3 -m json.tool config/qa-loop-config.json

Infinite loop without escalation:

# Check recurring threshold
cat config/qa-loop-config.json | jq '.loop_parameters.recurring_threshold'

# Lower the threshold temporarily
# Edit config: "recurring_threshold": 2

Too many escalations:

# Increase thresholds
# Edit config:
# - "max_iterations": 100
# - "recurring_threshold": 5
# - "similarity_threshold": 0.9

Fixes not being applied:

# Ensure not in dry-run mode
python3 scripts/qa-validation-loop.py artifact.md  # No --dry-run flag

# Check fixer agent configuration
cat config/qa-loop-config.json | jq '.agents.fixer'

Best Practices

1. Start with Dry Run

Always test with --dry-run first:

python3 scripts/qa-validation-loop.py artifact.md --dry-run --verbose

2. Review Escalation Reports

When escalation occurs, carefully review the report:

# Find latest escalation report
ls -lt .coditect/qa-escalations/ | head -2

# View with formatting
cat .coditect/qa-escalations/QA_ESCALATION-*.md

3. Tune Configuration for Your Project

Documentation projects: Lower thresholds (faster escalation)
Code reviews: Higher thresholds (more thorough validation)

4. Monitor History

Periodically review history to identify patterns:

# Runs this month
cat .coditect/qa-history.json | jq '.runs[] | select(.timestamp | startswith("2025-12"))'

# Success rate
cat .coditect/qa-history.json | jq '[.runs[].result] | group_by(.) | map({key: .[0], count: length})'

5. Integrate with Git Hooks

# .git/hooks/pre-commit
#!/bin/bash
python3 scripts/qa-validation-loop.py $(git diff --cached --name-only --diff-filter=ACM | grep '\.md$')

Examples

Example 1: ADR Validation

# Validate a new ADR before commit
python3 scripts/qa-validation-loop.py docs/architecture/decisions/ADR-030.md --verbose

# Output:
# [INFO] Starting QA self-healing loop for ADR-030.md
# [INFO] === Iteration 1/50 ===
# [INFO] QA Review complete: REJECTED (score: 0.65)
# [INFO] Running fixer agent for 3 issues
# [INFO] === Iteration 2/50 ===
# [INFO] QA Review complete: APPROVED (score: 0.85)
# [INFO] ✓ QA approved after 2 iterations

Example 2: Bulk Documentation Review

# Review all guides
for guide in docs/guides/*.md; do
    echo "Validating $guide..."
    python3 scripts/qa-validation-loop.py "$guide"
done

Example 3: Custom Configuration for Stricter Standards

{
  "loop_parameters": {
    "max_iterations": 30,
    "recurring_threshold": 2,
    "similarity_threshold": 0.85
  },
  "decision_thresholds": {
    "critical_findings": 0,
    "high_findings": 1,
    "aggregate_score": 0.9
  }
}

python3 scripts/qa-validation-loop.py critical-doc.md --config config/strict-qa.json

Future Enhancements

Planned Features

Machine learning for issue classification
Automatic fix suggestion generation
Integration with external QA tools
Real-time progress dashboard
Parallel artifact validation
Fix quality scoring

Contributing

To contribute enhancements:

Review scripts/qa-validation-loop.py implementation
Add new features with proper documentation
Update configuration schema if needed
Add tests for new functionality
Submit PR with examples

Support

Questions? Review escalation reports and history files first.

Issues? Check troubleshooting section above.

Feature Requests? See future enhancements and contributing section.

Last Updated: 2025-12-22 Version: 1.0.0 Pattern Origin: Inspired by Auto-Claude autonomous validation patterns License: Part of CODITECT Core Framework

Overview​

Key Features​

Architecture​

Quick Start​

Basic Usage​

Return Codes​

Configuration​

Default Configuration​

Custom Configuration​

Key Parameters​

Recurring Issue Detection​

How It Works​

Example​

Why This Matters​

State Tracking​

History File​

Viewing History​

Escalation Reports​

When Escalation Occurs​

Report Contents​

Example Report Section​

Integration with Existing Agents​

Council Orchestrator​

QA Reviewer (Fixer)​

Advanced Usage​

Custom Escalation Logic​

Integration with CI/CD​

Batch Processing​

Performance Considerations​

Iteration Duration​

Resource Usage​

Optimization Tips​

Troubleshooting​

Common Issues​

Best Practices​

1. Start with Dry Run​

2. Review Escalation Reports​

3. Tune Configuration for Your Project​

4. Monitor History​

5. Integrate with Git Hooks​

Examples​

Example 1: ADR Validation​

Example 2: Bulk Documentation Review​

Example 3: Custom Configuration for Stricter Standards​

Future Enhancements​

Planned Features​

Contributing​

Related Documentation​

Support​