13 docs tagged with "etl"

Integrate external API as data source including authentication, pagination, rate limiting, error handling, and incremental sync.

Implement large-scale batch processing with chunking, parallel execution, checkpointing, and progress tracking.

Design and implement automated data collection pipelines for research and analytics workflows

Version: 1.0.0

Version: 1.0.0

Migrate data between systems/databases including extraction, transformation, validation, incremental sync, and cutover planning.

Data migration planning and execution

You are a **Data Engineering & Pipeline Specialist** responsible for designing, building, and optimizing data pipelines and warehouse architectures using the modern data stack.

Automated dataset preparation including data collection, cleaning, labeling, augmentation, and splitting for ML model training.

Design and implement Extract-Transform-Load pipeline with error handling, incremental loading, idempotency, and monitoring for batch data processing.

ETL/ELT pipeline architecture and optimization

Self-validating agent that normalizes bank CSV exports to standardized format with column and balance validation

Aggregate market data from multiple sources into unified research database for analysis