Skip to main content

Data Collection Pipeline Setup

Design and implement automated data collection pipelines for research and analytics workflows

Complexity: Complex | Duration: 30m+ | Category: Research/Intelligence

Tags: data-pipeline ETL data-engineering automation infrastructure

Workflow Diagram​

Steps​

Step 1: Requirements gathering​

Agent: data

engineering - Define data sources, frequency, volume, and quality requirements

Step 2: Architecture design​

Agent: backend

architect - Design pipeline architecture with extraction, transformation, loading

Step 3: Source integration​

Agent: data

engineering - Implement data source connectors (APIs, databases, files, streams)

Step 4: Transformation logic​

Agent: data

engineering - Build data cleaning, normalization, enrichment transformations

Step 5: Storage setup​

Agent: backend

architect - Configure data storage (data lake, warehouse, database)

Step 6: Scheduling​

Agent: devops

engineer - Implement scheduling and orchestration (Airflow, Prefect, cron)

Step 7: Monitoring​

Agent: devops

engineer - Add data quality checks, error handling, alerting

Step 8: Testing​

Agent: QA

automation - Test pipeline end-to-end with production-like data volumes

Usage​

To execute this workflow:

/workflow research/intelligence/data-collection-pipeline-setup.workflow

See other workflows in this category for related automation patterns.