Skip to main content

CODITECT Test Categories Reference

CODITECT Test Categories Reference

Version: 2.0.0 Total Categories: 34 Total Tests: 4,714+

Core Component Tests​

AGENTS (107 tests)​

Validates all agent definition files in agents/ directory.

Checks:

  • YAML frontmatter required fields (name, description, version)
  • Valid tools list
  • Proper markdown structure
  • Cross-reference integrity

Files: agents/*.md

COMMANDS (105 tests)​

Validates slash command definitions in commands/ directory.

Checks:

  • Frontmatter validation (name, description, allowed-tools)
  • Arguments documentation
  • Usage examples present
  • Model specification

Files: commands/*.md

SKILLS (62 tests)​

Validates skill definitions in skills/*/SKILL.md.

Checks:

  • Skill structure and frontmatter
  • Pattern definitions
  • Integration examples
  • Cross-skill references

Files: skills/*/SKILL.md

SCRIPTS (150 tests)​

Validates Python and shell scripts in scripts/ directory.

Checks:

  • Python syntax validity
  • Shebang lines present
  • Docstrings for modules
  • Import statement validation

Files: scripts/*.py, scripts/*.sh

HOOKS (23 tests)​

Validates git hooks and automation triggers.

Checks:

  • Hook file structure
  • Trigger definitions
  • Integration with framework

Files: hooks/*.md

REGISTRY (453 tests)​

Validates component registry integrity.

Checks:

  • framework-registry.json structure
  • Component metadata completeness
  • Version consistency
  • Cross-registry references

Files: config/framework-registry.json, config/**/*.json

XREF (97 tests)​

Validates cross-reference integrity across components.

Checks:

  • Agent-to-skill references valid
  • Command-to-agent references valid
  • Documentation links resolve
  • No orphaned references

CONFIG (300 tests)​

Validates all configuration files.

Checks:

  • JSON syntax validity
  • Required fields present
  • Schema compliance
  • Environment-specific configs

Files: config/**/*.json, .coditect/*.json

DOCS (13 tests)​

Validates documentation structure.

Checks:

  • Required documentation files exist
  • Directory structure matches standard
  • README.md presence in key directories

Files: docs/**/*.md

STRUCTURE (10 tests)​

Validates overall project structure.

Checks:

  • Required directories exist
  • Proper nesting depth
  • Standard naming conventions

QUALITY (274 tests)​

Content quality checks.

Checks:

  • Markdown formatting
  • Code block language specifications
  • Link formatting
  • Heading hierarchy

SECURITY (135 tests)​

Security best practices validation.

Checks:

  • No hardcoded secrets
  • No API keys in source
  • Secure default configurations
  • Input validation patterns

CONSISTENCY (245 tests)​

Cross-component consistency checks.

Checks:

  • Naming conventions consistent
  • Version numbers aligned
  • Metadata format uniform
  • Style guide compliance

ACTIVATION (407 tests)​

Component activation workflow validation.

Checks:

  • Activation status tracking
  • Dependency resolution
  • State transitions valid
  • Audit trail integrity

Files: .coditect/component-activation-status.json

Infrastructure Tests​

INTEGRATION (810 tests)​

Component interaction tests.

Checks:

  • Agent-command integration
  • Skill-script integration
  • Registry-component sync
  • Cross-module dependencies

PERFORMANCE (70 tests)​

Performance metric validation.

Checks:

  • File sizes reasonable
  • Path depths acceptable
  • Load times nominal
  • Memory patterns valid

DEPENDENCIES (655 tests)​

Import and dependency validation.

Checks:

  • Python stdlib imports valid
  • External dependencies declared
  • No circular imports
  • Version constraints met

SCHEMAS (84 tests)​

Schema validation tests.

Checks:

  • JSON Schema compliance
  • YAML structure validity
  • Data type correctness
  • Required field presence

DOC_COVERAGE (288 tests)​

Documentation coverage metrics.

Checks:

  • All components documented
  • API documentation complete
  • User guides present
  • Examples provided

WORKFLOWS (18 tests)​

CI/CD workflow validation.

Checks:

  • GitHub Actions syntax
  • Workflow triggers valid
  • Job dependencies correct
  • Environment configurations

New Test Categories (v2.0)​

UNIT_TESTS (18 tests)​

Pytest unit test execution.

What it validates:

  • Test files in tests/ directory can execute
  • pytest infrastructure works
  • Test discovery succeeds

Status interpretation:

  • PASS = Tests ran (even with assertion failures)
  • SKIP = Missing dependencies

IMPORTS (130 tests)​

Python import validation.

Checks:

  • All .py files compile without syntax errors
  • Module imports resolve

Symlink chain validation.

Checks:

  • All symlinks resolve to valid targets
  • No broken symlink chains
  • Framework symlinks (.coditect/) valid

GIT_INTEGRITY (5 tests)​

Repository health checks.

Checks:

  • Is a valid git repository
  • Working tree status
  • Current branch
  • Submodule status

LINT (5 tests)​

Code quality via ruff.

Checks:

  • Python lint issues (informational)
  • Syntax validation
  • Anti-pattern detection

CLI (82 tests)​

CLI argument validation.

Checks:

  • Scripts support --help
  • argparse usage present
  • Help text quality

Markdown link validation.

Checks:

  • Internal links resolve
  • Relative paths correct
  • Anchor references valid

PARSE (101 tests)​

Configuration file parsing.

Checks:

  • All JSON files parse
  • All YAML files parse
  • No syntax errors

DOCKER (3 tests)​

Docker configuration validation.

Checks:

  • Dockerfile structure valid
  • docker-compose.yml valid
  • .dockerignore present (optional)

TYPES (2 tests)​

Type checking with mypy.

Checks:

  • Type annotations present
  • Type errors (informational)

E2E (2 tests)​

End-to-end test setup.

Checks:

  • Playwright configuration
  • Test file discovery

COVERAGE (4 tests)​

Test coverage metrics.

Checks:

  • Coverage reports exist
  • Threshold compliance (informational)

SMOKE (13 tests)​

Critical path validation.

Checks:

  • Core files exist
  • Essential scripts executable
  • Framework structure valid

API_CONTRACT (3 tests)​

API schema validation.

Checks:

  • OpenAPI schemas valid
  • Contract definitions present

Running Specific Categories​

# Single category
python3 scripts/test-suite.py --category agents

# Multiple categories
python3 scripts/test-suite.py --category agents,commands,skills

# All infrastructure tests
python3 scripts/test-suite.py --category integration,performance,dependencies

Last Updated: December 10, 2025