Test Automation Skill

Test Automation Skill

When to Use This Skill

Use this skill when implementing test automation patterns in your codebase.

How to Use This Skill

1. Review the patterns and examples below
2. Apply the relevant patterns to your implementation
3. Follow the best practices outlined in this skill

Comprehensive test automation patterns for validating CODITECT framework components across 13 test categories with quality gate enforcement.

Test Suite Architecture

Project Structure

test-results/
├── test-results.json       # Machine-readable results
├── TEST-RESULTS.md         # Human-readable summary
└── coverage/               # Coverage reports

scripts/
└── test-suite.py           # Main test execution script

Test Categories

TEST_CATEGORIES = {
    "AGENTS": "Agent frontmatter, tools, model validation",
    "COMMANDS": "Command name field, description validation",
    "SKILLS": "SKILL.md presence, metadata completeness",
    "SCRIPTS": "Python syntax, executable permissions",
    "CONFIG": "JSON schema, cross-reference integrity",
    "DOCS": "Link validation, required sections",
    "SECURITY": "Secret patterns, dangerous commands",
    "CONSISTENCY": "Naming conventions, version formats",
    "STRUCTURE": "Symlinks, directories, required files",
    "REGISTRY": "Component registration, metadata",
    "HOOKS": "Trigger events, blocking behavior",
    "QUALITY": "Coverage thresholds, broken links",
    "XREF": "Bidirectional references, orphans"
}

Quick Start

Run Full Test Suite

# Execute comprehensive tests
python3 scripts/test-suite.py

# View results
cat test-results/TEST-RESULTS.md

Run Specific Category

# Test only agents
python3 scripts/test-suite.py --category AGENTS

# Test multiple categories
python3 scripts/test-suite.py --category AGENTS,COMMANDS,SKILLS

Test Implementation Patterns

YAML Frontmatter Validation

def validate_frontmatter(file_path: str, required_fields: list) -> TestResult:
    """Validate YAML frontmatter in markdown files."""
    content = read_file(file_path)

    if not content.startswith('---'):
        return TestResult(
            status="FAIL",
            message=f"{file_path}: Missing YAML frontmatter"
        )

    try:
        # Extract frontmatter
        parts = content.split('---', 2)
        if len(parts) < 3:
            return TestResult(status="FAIL", message="Malformed frontmatter")

        frontmatter = yaml.safe_load(parts[1])

        # Check required fields
        missing = [f for f in required_fields if f not in frontmatter]
        if missing:
            return TestResult(
                status="FAIL",
                message=f"Missing fields: {', '.join(missing)}"
            )

        return TestResult(status="PASS", message="Frontmatter valid")

    except yaml.YAMLError as e:
        return TestResult(status="FAIL", message=f"YAML error: {e}")

Python Syntax Validation

def validate_python_syntax(file_path: str) -> TestResult:
    """Validate Python file compiles without errors."""
    try:
        with open(file_path, 'r') as f:
            source = f.read()

        compile(source, file_path, 'exec')
        return TestResult(status="PASS", message="Syntax valid")

    except SyntaxError as e:
        return TestResult(
            status="FAIL",
            message=f"Line {e.lineno}: {e.msg}"
        )

JSON Schema Validation

def validate_json_schema(file_path: str, schema: dict) -> TestResult:
    """Validate JSON file against schema."""
    try:
        with open(file_path, 'r') as f:
            data = json.load(f)

        jsonschema.validate(data, schema)
        return TestResult(status="PASS", message="Schema valid")

    except json.JSONDecodeError as e:
        return TestResult(status="FAIL", message=f"JSON error: {e}")
    except jsonschema.ValidationError as e:
        return TestResult(status="FAIL", message=f"Schema error: {e.message}")

Symlink Validation

def validate_symlink(path: str, expected_target: str) -> TestResult:
    """Validate symlink exists and points to expected target."""
    if not os.path.islink(path):
        return TestResult(
            status="FAIL",
            message=f"{path}: Not a symlink"
        )

    actual_target = os.readlink(path)
    if actual_target != expected_target:
        return TestResult(
            status="WARN",
            message=f"Target mismatch: {actual_target} != {expected_target}"
        )

    return TestResult(status="PASS", message="Symlink valid")

Security Pattern Detection

def detect_security_issues(file_path: str) -> list[TestResult]:
    """Scan for potential security issues."""
    results = []
    content = read_file(file_path)

    # Secret patterns
    SECRET_PATTERNS = [
        (r'sk-[a-zA-Z0-9]{48}', 'OpenAI API key'),
        (r'ghp_[a-zA-Z0-9]{36}', 'GitHub token'),
        (r'AKIA[0-9A-Z]{16}', 'AWS access key'),
        (r'password\s*=\s*["\'][^"\']+["\']', 'Hardcoded password'),
    ]

    for pattern, description in SECRET_PATTERNS:
        if re.search(pattern, content):
            results.append(TestResult(
                status="FAIL",
                severity="critical",
                message=f"Potential {description} detected"
            ))

    # Dangerous commands
    DANGEROUS_COMMANDS = [
        (r'rm\s+-rf\s+/', 'Dangerous rm -rf /'),
        (r'>\s*/dev/[sh]d[a-z]', 'Write to device'),
        (r'mkfs\.', 'Filesystem creation'),
    ]

    for pattern, description in DANGEROUS_COMMANDS:
        if re.search(pattern, content):
            results.append(TestResult(
                status="FAIL",
                severity="critical",
                message=f"Dangerous command: {description}"
            ))

    return results if results else [TestResult(status="PASS")]

Markdown Link Validation

def validate_markdown_links(file_path: str) -> list[TestResult]:
    """Validate internal markdown links exist."""
    results = []
    content = read_file(file_path)
    base_dir = os.path.dirname(file_path)

    # Find markdown links: [text](#)
    link_pattern = r'\[([^\]]+)\]\(([^)]+)\)'

    for match in re.finditer(link_pattern, content):
        text, link = match.groups()

        # Skip external links
        if link.startswith(('http://', 'https://', 'mailto:')):
            continue

        # Remove anchors
        link_path = link.split('#')[0]
        if not link_path:
            continue

        # Resolve relative path
        full_path = os.path.normpath(os.path.join(base_dir, link_path))

        if not os.path.exists(full_path):
            results.append(TestResult(
                status="WARN",
                message=f"Broken link: {link}"
            ))

    return results if results else [TestResult(status="PASS")]

Test Result Structure

TestResult Class

@dataclass
class TestResult:
    category: str
    component: str
    test_name: str
    status: str        # "PASS" | "FAIL" | "WARN"
    message: str
    severity: str = "medium"  # "critical" | "high" | "medium" | "low"

    def to_dict(self) -> dict:
        return {
            "category": self.category,
            "component": self.component,
            "test_name": self.test_name,
            "status": self.status,
            "message": self.message,
            "severity": self.severity
        }

Quality Gate Decision

def quality_gate_verdict(results: list[TestResult]) -> dict:
    """Determine if quality gate passes."""
    failures = [r for r in results if r.status == "FAIL"]
    warnings = [r for r in results if r.status == "WARN"]
    passes = [r for r in results if r.status == "PASS"]

    total = len(results)
    pass_rate = len(passes) / total if total > 0 else 0

    # Critical failures block immediately
    critical = [f for f in failures if f.severity == "critical"]
    if critical:
        return {
            "passed": False,
            "reason": f"{len(critical)} critical failures",
            "pass_rate": pass_rate,
            "details": [f.message for f in critical]
        }

    # Pass rate threshold (95%)
    if pass_rate < 0.95:
        return {
            "passed": False,
            "reason": f"Pass rate {pass_rate:.1%} below 95% threshold",
            "pass_rate": pass_rate
        }

    return {
        "passed": True,
        "reason": f"All gates passed ({pass_rate:.1%})",
        "pass_rate": pass_rate,
        "warnings": len(warnings)
    }

CI/CD Integration

GitHub Actions Workflow

name: CODITECT Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Run test suite
        run: python3 scripts/test-suite.py

      - name: Check quality gate
        run: |
          PASS_RATE=$(jq -r '.summary.pass_rate' test-results/test-results.json)
          if (( $(echo "$PASS_RATE < 95" | bc -l) )); then
            echo "Quality gate failed: $PASS_RATE% < 95%"
            exit 1
          fi

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results/

Pre-Commit Integration

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: coditect-tests
        name: CODITECT Framework Tests
        entry: python3 scripts/test-suite.py --quick
        language: system
        pass_filenames: false
        always_run: true

Coverage Tracking

Category Coverage

def calculate_category_coverage() -> dict:
    """Calculate test coverage by category."""
    coverage = {}

    for category in TEST_CATEGORIES:
        total = count_components(category)
        tested = count_tested_components(category)
        coverage[category] = {
            "total": total,
            "tested": tested,
            "percentage": (tested / total * 100) if total > 0 else 100
        }

    return coverage

Coverage Thresholds

COVERAGE_THRESHOLDS = {
    "AGENTS": 100,      # All agents must be tested
    "COMMANDS": 100,    # All commands must be tested
    "SKILLS": 95,       # 95% skill coverage
    "SCRIPTS": 90,      # 90% script coverage
    "CONFIG": 100,      # All configs validated
    "DOCS": 85,         # 85% doc link validation
    "SECURITY": 100,    # All security patterns checked
    "CONSISTENCY": 95,  # 95% naming compliance
}

Usage Examples

Validate New Agent

Apply test-automation skill to validate new agent meets all frontmatter requirements and quality gates

Run Security Scan

Apply test-automation skill to scan codebase for security issues including secrets and dangerous commands

Generate Coverage Report

Apply test-automation skill to generate comprehensive coverage report across all test categories

Pre-Commit Validation

Apply test-automation skill for pre-commit hook validation ensuring no regressions before commit

Output Formats

JSON Output

{
  "summary": {
    "total": 1975,
    "passed": 1933,
    "failed": 0,
    "warnings": 42,
    "pass_rate": 97.9
  },
  "by_category": {
    "AGENTS": {"passed": 107, "failed": 0, "warnings": 0},
    "COMMANDS": {"passed": 103, "failed": 0, "warnings": 0}
  },
  "failures": [],
  "warnings": [
    {"category": "QUALITY", "message": "Broken link in docs/example.md"}
  ],
  "quality_gate": {
    "passed": true,
    "reason": "All gates passed (97.9%)"
  }
}

Markdown Output

# Test Results

| Category | Passed | Failed | Warnings | Rate |
|----------|--------|--------|----------|------|
| AGENTS   | 107    | 0      | 0        | 100% |
| COMMANDS | 103    | 0      | 0        | 100% |
| ...      | ...    | ...    | ...      | ...  |

**Quality Gate: PASS**

---

Success Output

When successful, this skill MUST output:

✅ SKILL COMPLETE: test-automation

Completed:
- [x] All 13 test categories executed
- [x] Quality gate threshold met (≥95%)
- [x] Test results generated
- [x] Coverage reports created

Outputs:
- Test results: test-results/test-results.json
- Summary report: test-results/TEST-RESULTS.md
- Coverage report: test-results/coverage/

Test Summary:
- Total tests: {total}
- Passed: {passed} ({pass_rate}%)
- Failed: {failed}
- Warnings: {warnings}
- Quality gate: PASS/FAIL

Completion Checklist

Before marking this skill as complete, verify:

• All 13 test categories executed
• AGENTS: 100% frontmatter validation
• COMMANDS: 100% name field validation
• SKILLS: 95%+ metadata completeness
• SCRIPTS: Python syntax validated
• CONFIG: JSON schema validated
• DOCS: Link validation complete
• SECURITY: No secrets detected
• CONSISTENCY: Naming conventions enforced
• STRUCTURE: Symlinks and directories verified
• REGISTRY: Component registration validated
• HOOKS: Trigger events validated
• QUALITY: Coverage thresholds met
• XREF: Cross-references validated
• Pass rate ≥95%
• No critical failures
• Test results saved

Failure Indicators

This skill has FAILED if:

• ❌ Pass rate below 95% threshold
• ❌ Any critical security issues detected
• ❌ Quality gate blocked by critical failures
• ❌ Test execution crashed or incomplete
• ❌ Test results not generated
• ❌ Coverage below required thresholds
• ❌ Broken symlinks detected
• ❌ Invalid JSON/YAML configuration files

When NOT to Use

Do NOT use this skill when:

• Testing a single component in isolation (use specific category test instead)
• Need fast iteration during development (run quick smoke tests only)
• Working in local development (full suite too slow, use --category flag)
• Pre-commit validation needed (use pre-commit-config.yaml hooks instead)
• Only checking syntax (use language-specific linters directly)
• Need unit tests for application code (this is for framework validation)
• Testing external services/APIs (use integration-test-patterns instead)

Anti-Patterns (Avoid)

Anti-Pattern	Problem	Solution
Running full suite every commit	Slow development cycle	Use --category for targeted testing
Ignoring warnings	Technical debt accumulation	Fix warnings incrementally
Not updating tests with code changes	False negatives	Update tests when changing structure
Skipping security scans	Security vulnerabilities	Always run SECURITY category
No coverage tracking	Blind spots in testing	Monitor coverage thresholds
Hardcoding test paths	Brittle tests	Use relative paths and glob patterns
Not testing edge cases	Incomplete validation	Add negative test cases
Skipping CI/CD integration	No automated quality gates	Add to GitHub Actions workflow
Not documenting test failures	Recurring issues	Document failure patterns
Running tests without cleanup	Polluted test state	Implement proper teardown

Principles

This skill embodies:

• #1 Full Automation - Comprehensive automated validation across 13 categories
• #5 Eliminate Ambiguity - Clear pass/fail criteria with 95% threshold
• #6 Clear, Understandable, Explainable - Human-readable reports and JSON output
• #7 Evidence-Based Decision Making - Quality gate based on objective metrics
• #8 No Assumptions - Explicit validation of all component types
• #10 Fail Closed - Critical failures block quality gate
• #13 Measure, Don't Guess - Coverage tracking and metrics

Full Standard: CODITECT-STANDARD-AUTOMATION.md

---