Skip to main content

Log Aggregation Pipeline

Set up centralized logging with log collection, parsing, indexing, searching, and retention policies.

Complexity: Moderate | Duration: 15-30m | Category: Devops

Tags: infrastructure logging elk observability

Workflow Diagram​

Steps​

Step 1: Stack selection​

Agent: devops

engineer - Choose ELK, Loki, CloudWatch Logs

Step 2: Log shipper deployment​

Agent: devops

engineer - Deploy Filebeat/Fluentd/Promtail

Step 3: Log parsing​

Agent: devops

engineer - Configure grok patterns for parsing

Step 4: Index creation​

Agent: devops

engineer - Create Elasticsearch indexes

Step 5: Retention policy​

Agent: devops

engineer - Set retention (hot 7d, warm 30d, cold 90d)

Step 6: Search setup​

Agent: devops

engineer - Configure Kibana for log searching

Step 7: Alerting​

Agent: devops

engineer - Create alerts on error patterns

Step 8: Dashboard​

Agent: devops

engineer - Build log volume, error rate dashboards

Usage​

To execute this workflow:

/workflow devops/log-aggregation-pipeline.workflow

See other workflows in this category for related automation patterns.