Skip to main content

Api Data Integration

Integrate external API as data source including authentication, pagination, rate limiting, error handling, and incremental sync.

Complexity: Moderate | Duration: 15-30m | Category: Devops

Tags: data-engineering api integration etl

Workflow Diagram​

Steps​

Step 1: API exploration​

Agent: backend

architect - Review API docs, authentication, endpoints

Step 2: Authentication setup​

Agent: backend

architect - Implement OAuth, API key, JWT

Step 3: Client implementation​

Agent: backend

architect - Create API client with retry logic

Step 4: Pagination handling​

Agent: data

engineer - Implement cursor/offset pagination

Step 5: Rate limiting​

Agent: backend

architect - Respect API rate limits with backoff

Step 6: Incremental sync​

Agent: data

engineer - Track last sync timestamp, fetch new records

Step 7: Error handling​

Agent: data

engineer - Handle 4xx/5xx errors, network failures

Step 8: Testing​

Agent: backend

architect - Mock API responses, test edge cases

Usage​

To execute this workflow:

/workflow devops/api-data-integration.workflow

See other workflows in this category for related automation patterns.