Agent Skills Framework Extension

Novelty Detection Patterns Skill

When to Use This Skill

Use this skill when implementing novelty detection patterns patterns in your codebase.

How to Use This Skill

1. Review the patterns and examples below
2. Apply the relevant patterns to your implementation
3. Follow the best practices outlined in this skill

Detect novel situations, assess scenarios, adapt strategies, and handle unknown contexts intelligently.

Core Capabilities

1. Situation Classification - Identify scenario type
2. Novelty Detection - Detect unfamiliar patterns
3. Adaptive Strategy - Select appropriate response
4. Confidence Scoring - Quantify certainty
5. Fallback Mechanisms - Handle unknowns gracefully
6. Pattern Learning - Update from experience

Situation Classifier

# scripts/situation-classifier.py
from dataclasses import dataclass
from typing import List, Dict, Optional
from enum import Enum

class SituationType(Enum):
    KNOWN = "known"
    SIMILAR = "similar"
    NOVEL = "novel"
    AMBIGUOUS = "ambiguous"

@dataclass
class Situation:
    description: str
    features: Dict[str, any]
    classification: SituationType
    confidence: float
    similar_cases: List[str]
    recommended_strategy: str

class SituationClassifier:
    """Classify situations as known, similar, or novel"""

    def __init__(self):
        self.known_patterns: Dict[str, Dict] = {}
        self.similarity_threshold = 0.7

    def classify(self, situation_desc: str, context: Dict) -> Situation:
        """Classify situation"""
        features = self._extract_features(situation_desc, context)

        # Check for exact match
        if self._is_known_pattern(features):
            return Situation(
                description=situation_desc,
                features=features,
                classification=SituationType.KNOWN,
                confidence=0.95,
                similar_cases=[],
                recommended_strategy="apply_known_solution"
            )

        # Check for similar patterns
        similar_cases, similarity = self._find_similar_patterns(features)
        if similarity >= self.similarity_threshold:
            return Situation(
                description=situation_desc,
                features=features,
                classification=SituationType.SIMILAR,
                confidence=similarity,
                similar_cases=similar_cases,
                recommended_strategy="adapt_similar_solution"
            )

        # Check if ambiguous
        if self._is_ambiguous(features):
            return Situation(
                description=situation_desc,
                features=features,
                classification=SituationType.AMBIGUOUS,
                confidence=0.5,
                similar_cases=[],
                recommended_strategy="request_clarification"
            )

        # Novel situation
        return Situation(
            description=situation_desc,
            features=features,
            classification=SituationType.NOVEL,
            confidence=0.3,
            similar_cases=[],
            recommended_strategy="cautious_exploration"
        )

    def _extract_features(self, description: str, context: Dict) -> Dict:
        """Extract situation features"""
        features = {
            'has_code': 'code' in description.lower(),
            'has_error': 'error' in description.lower(),
            'complexity': self._estimate_complexity(description),
            'domain': self._detect_domain(description),
            'urgency': context.get('urgency', 'normal'),
        }
        return features

    def _estimate_complexity(self, text: str) -> str:
        """Estimate problem complexity"""
        word_count = len(text.split())
        if word_count < 20:
            return 'simple'
        elif word_count < 100:
            return 'moderate'
        else:
            return 'complex'

    def _detect_domain(self, text: str) -> str:
        """Detect problem domain"""
        domains = {
            'web': ['react', 'vue', 'angular', 'html', 'css'],
            'backend': ['api', 'database', 'server', 'microservice'],
            'devops': ['docker', 'kubernetes', 'ci/cd', 'deployment'],
            'ml': ['machine learning', 'neural network', 'model', 'training'],
        }

        text_lower = text.lower()
        for domain, keywords in domains.items():
            if any(kw in text_lower for kw in keywords):
                return domain

        return 'general'

    def _is_known_pattern(self, features: Dict) -> bool:
        """Check if pattern is known"""
        feature_key = self._features_to_key(features)
        return feature_key in self.known_patterns

    def _find_similar_patterns(
        self,
        features: Dict
    ) -> tuple[List[str], float]:
        """Find similar known patterns"""
        if not self.known_patterns:
            return [], 0.0

        best_similarity = 0.0
        similar_cases = []

        for pattern_key, pattern_data in self.known_patterns.items():
            similarity = self._calculate_similarity(features, pattern_data['features'])
            if similarity > best_similarity:
                best_similarity = similarity
                similar_cases = [pattern_data['description']]
            elif similarity == best_similarity:
                similar_cases.append(pattern_data['description'])

        return similar_cases, best_similarity

    def _calculate_similarity(self, features1: Dict, features2: Dict) -> float:
        """Calculate feature similarity"""
        matching = 0
        total = len(features1)

        for key, value in features1.items():
            if key in features2 and features2[key] == value:
                matching += 1

        return matching / total if total > 0 else 0.0

    def _is_ambiguous(self, features: Dict) -> bool:
        """Check if situation is ambiguous"""
        # Simple heuristic: multiple possible domains
        return features.get('complexity') == 'complex' and \
               features.get('domain') == 'general'

    def _features_to_key(self, features: Dict) -> str:
        """Convert features to string key"""
        return '_'.join(f"{k}={v}" for k, v in sorted(features.items()))

    def learn(self, situation: Situation, outcome: str):
        """Learn from situation outcome"""
        feature_key = self._features_to_key(situation.features)
        self.known_patterns[feature_key] = {
            'features': situation.features,
            'description': situation.description,
            'strategy': situation.recommended_strategy,
            'outcome': outcome
        }

# Usage
classifier = SituationClassifier()

situation = classifier.classify(
    "Help me implement async API calls in React",
    context={'urgency': 'normal'}
)

print(f"Classification: {situation.classification.value}")
print(f"Confidence: {situation.confidence:.2f}")
print(f"Strategy: {situation.recommended_strategy}")

Adaptive Strategy Selector

# scripts/adaptive-strategy.py
from dataclasses import dataclass
from typing import List, Callable, Optional

@dataclass
class Strategy:
    name: str
    applicability: Callable[[Situation], float]
    execute: Callable[[Situation], str]
    fallback: Optional[str]

class AdaptiveStrategySelector:
    """Select and execute appropriate strategies"""

    def __init__(self):
        self.strategies: List[Strategy] = self._initialize_strategies()

    def select_and_execute(self, situation: Situation) -> str:
        """Select best strategy and execute"""
        # Score all strategies
        scored = []
        for strategy in self.strategies:
            score = strategy.applicability(situation)
            scored.append((score, strategy))

        # Sort by score
        scored.sort(reverse=True, key=lambda x: x[0])

        # Execute best strategy
        if scored:
            best_score, best_strategy = scored[0]

            if best_score >= 0.7:
                try:
                    return best_strategy.execute(situation)
                except Exception as e:
                    if best_strategy.fallback:
                        fallback_strat = self._get_strategy(best_strategy.fallback)
                        if fallback_strat:
                            return fallback_strat.execute(situation)
                    return f"Strategy failed: {e}"

        # No suitable strategy
        return self._handle_unknown(situation)

    def _initialize_strategies(self) -> List[Strategy]:
        """Initialize available strategies"""
        return [
            Strategy(
                name="apply_known_solution",
                applicability=lambda s: 0.95 if s.classification == SituationType.KNOWN else 0.0,
                execute=lambda s: f"Applying known solution for: {s.description}",
                fallback="cautious_exploration"
            ),
            Strategy(
                name="adapt_similar_solution",
                applicability=lambda s: 0.8 if s.classification == SituationType.SIMILAR else 0.0,
                execute=lambda s: f"Adapting solution from: {s.similar_cases[0] if s.similar_cases else 'similar case'}",
                fallback="request_clarification"
            ),
            Strategy(
                name="request_clarification",
                applicability=lambda s: 0.9 if s.classification == SituationType.AMBIGUOUS else 0.3,
                execute=lambda s: f"Need clarification: What aspect of {s.description} should I focus on?",
                fallback=None
            ),
            Strategy(
                name="cautious_exploration",
                applicability=lambda s: 0.7 if s.classification == SituationType.NOVEL else 0.2,
                execute=lambda s: f"Novel situation: Exploring options for {s.description}",
                fallback="request_clarification"
            ),
        ]

    def _get_strategy(self, name: str) -> Optional[Strategy]:
        """Get strategy by name"""
        for strategy in self.strategies:
            if strategy.name == name:
                return strategy
        return None

    def _handle_unknown(self, situation: Situation) -> str:
        """Handle completely unknown situations"""
        return f"Unable to determine strategy for: {situation.description}. Requesting human guidance."

# Usage
selector = AdaptiveStrategySelector()

# Known situation
known_sit = Situation(
    description="Add JWT authentication",
    features={'domain': 'backend'},
    classification=SituationType.KNOWN,
    confidence=0.95,
    similar_cases=[],
    recommended_strategy="apply_known_solution"
)

result = selector.select_and_execute(known_sit)
print(result)

# Novel situation
novel_sit = Situation(
    description="Implement quantum error correction",
    features={'domain': 'general'},
    classification=SituationType.NOVEL,
    confidence=0.3,
    similar_cases=[],
    recommended_strategy="cautious_exploration"
)

result = selector.select_and_execute(novel_sit)
print(result)

Anomaly Detector

// scripts/anomaly-detection.ts
interface Observation {
  timestamp: Date;
  features: Record<string, number>;
  label?: string;
}

class AnomalyDetector {
  private observations: Observation[] = [];
  private threshold: number = 2.5; // Standard deviations

  /**
   * Add observation to history
   */
  observe(observation: Observation): void {
    this.observations.push(observation);
  }

  /**
   * Detect if observation is anomalous
   */
  isAnomaly(observation: Observation): boolean {
    if (this.observations.length < 30) {
      return false; // Not enough data
    }

    const scores = this.calculateAnomalyScores(observation);
    const meanScore = scores.reduce((a, b) => a + b, 0) / scores.length;

    return meanScore > this.threshold;
  }

  private calculateAnomalyScores(obs: Observation): number[] {
    const scores: number[] = [];

    for (const [feature, value] of Object.entries(obs.features)) {
      const historicalValues = this.observations
        .map(o => o.features[feature])
        .filter(v => v !== undefined);

      if (historicalValues.length > 0) {
        const mean = historicalValues.reduce((a, b) => a + b, 0) / historicalValues.length;
        const variance = historicalValues
          .map(v => Math.pow(v - mean, 2))
          .reduce((a, b) => a + b, 0) / historicalValues.length;
        const stdDev = Math.sqrt(variance);

        const zScore = stdDev > 0 ? Math.abs((value - mean) / stdDev) : 0;
        scores.push(zScore);
      }
    }

    return scores;
  }

  /**
   * Get anomaly report
   */
  getReport(observation: Observation): string {
    const isAnom = this.isAnomaly(observation);
    const scores = this.calculateAnomalyScores(observation);
    const meanScore = scores.reduce((a, b) => a + b, 0) / scores.length;

    return `Anomaly: ${isAnom}, Score: ${meanScore.toFixed(2)}, Threshold: ${this.threshold}`;
  }
}

// Usage
const detector = new AnomalyDetector();

// Train with normal observations
for (let i = 0; i < 50; i++) {
  detector.observe({
    timestamp: new Date(),
    features: { value1: 10 + Math.random() * 2, value2: 20 + Math.random() * 2 }
  });
}

// Test anomaly
const anomaly = {
  timestamp: new Date(),
  features: { value1: 50, value2: 100 } // Far from normal
};

console.log(detector.getReport(anomaly));

Usage Examples

Situation Classification

Apply novelty-detection-patterns skill to classify request and recommend strategy

Adaptive Strategy Selection

Apply novelty-detection-patterns skill to select appropriate response strategy for novel situation

Anomaly Detection

Apply novelty-detection-patterns skill to detect anomalous patterns in session behavior

Success Output

When successfully applying novelty detection patterns:

✅ SKILL COMPLETE: novelty-detection-patterns

Completed:
- [x] SituationClassifier initialized with known patterns
- [x] Situation features extracted (domain, complexity, urgency)
- [x] Situation classified (KNOWN/SIMILAR/NOVEL/AMBIGUOUS)
- [x] Confidence score calculated (0.0-1.0)
- [x] Recommended strategy determined based on classification
- [x] AdaptiveStrategySelector executed appropriate strategy
- [x] AnomalyDetector (if applicable) identified anomalous patterns

Outputs:
- Situation classification (type, confidence, similar_cases)
- Recommended strategy (apply_known_solution, adapt_similar_solution, etc.)
- Strategy execution result
- Anomaly detection report (if anomaly detector used)
- Pattern learning update to known_patterns

Completion Checklist

Before marking this skill as complete, verify:

• SituationClassifier initialized and configured (similarity_threshold set)
• Features extracted from situation description and context
• Classification logic executed (known → similar → ambiguous → novel)
• Confidence score calculated and appropriate for classification
• Similar patterns found if classification is SIMILAR (similarity >= threshold)
• Recommended strategy matches classification type
• AdaptiveStrategySelector selected and executed best strategy
• Strategy applicability scored for all available strategies
• Fallback strategy triggered if primary strategy failed
• Pattern learned and added to known_patterns (if learn method called)

Failure Indicators

This skill has FAILED if:

• ❌ Classification returns KNOWN for clearly novel situation (false positive)
• ❌ Classification returns NOVEL for well-known pattern (false negative)
• ❌ Confidence score too high (>0.9) for NOVEL situation
• ❌ Similarity calculation always returns 0.0 or 1.0 (broken logic)
• ❌ No strategy selected despite multiple strategies available
• ❌ Strategy applicability score negative or >1.0 (invalid range)
• ❌ Fallback strategy not triggered when primary strategy fails
• ❌ Pattern not learned after successful outcome (learn not called)
• ❌ AnomalyDetector classifies normal data as anomaly (>2.5 stdev threshold violated)

When NOT to Use

Do NOT use this skill when:

• All situations are well-defined and known (no novelty)
• Classification not needed (direct strategy selection)
• No learning or adaptation required (static responses)
• Real-time performance critical (classification adds latency)
• Simple if-else logic sufficient (no fuzzy matching needed)

Use alternatives instead:

• For known-only situations → Use direct lookup table/switch statement
• For simple classification → Use rule-based logic without similarity scoring
• For no learning → Skip pattern learning, use static strategy map
• For real-time → Cache classifications, skip feature extraction
• For deterministic logic → Use traditional control flow

Anti-Patterns (Avoid)

Anti-Pattern	Problem	Solution
Not updating known_patterns	Repeatedly treats known as novel	Call learn() after successful outcomes
Similarity threshold too low (<0.5)	Everything classified as similar	Use 0.7+ threshold for high confidence
Ignoring confidence scores	Low confidence treated as certain	Check confidence, request clarification if <0.6
Fixed strategy selection	Ignores classification type	Use AdaptiveStrategySelector for dynamic selection
No fallback strategy	Failures block execution	Define fallback for each strategy
Hardcoded feature extraction	Can't adapt to new domains	Use configurable feature extractors

Principles

This skill embodies CODITECT automation principles:

• #2 First Principles - Understands WHY novelty detection matters (handle unknowns gracefully)
• #5 Eliminate Ambiguity - Explicit classification types (KNOWN/SIMILAR/NOVEL/AMBIGUOUS)
• #6 Clear, Understandable, Explainable - Confidence scores and reasoning make decisions transparent
• #8 No Assumptions - Detects ambiguity explicitly, requests clarification when needed
• #9 Research When in Doubt - Novelty detection triggers cautious exploration for unknowns
• Continual Learning - Pattern learning updates known_patterns from experience

Full Standard: CODITECT-STANDARD-AUTOMATION.md

---

Integration Points

• uncertainty-quantification-patterns - Confidence scoring
• prompt-analysis-patterns - Request analysis
• session-analysis-patterns - Behavior patterns