Data Pipeline Orchestration

Scheduling and sequencing interdependent pipeline steps as a DAG, with retries and backfills.

What it is

Orchestration tools (Airflow, Dagster, Prefect) schedule and sequence pipeline steps that depend on each other, modeled as a Directed Acyclic Graph (DAG) of tasks.

Key points

DAG structure: each task declares its upstream dependencies; the orchestrator figures out execution order and what can run in parallel.
Idempotency matters — a task should be safe to rerun without producing duplicate or corrupted results, since retries and backfills are routine operational realities.
Backfilling: rerunning a pipeline for a past date range, e.g. after fixing a bug — a first-class scenario orchestration tools are built to handle, not an afterthought.
Sensors/triggers: some pipelines run on a schedule, others wait for an upstream event (a file landing, an API signal) before starting.