Skip to content
Tech Interview Prep
Technical interview guide

Data Pipeline Orchestration

Scheduling and sequencing interdependent pipeline steps as a DAG, with retries and backfills.

Read
29 min
Practice MCQs
25
Interview Q&A
25
Edition
v4
Editorial status
Reviewed

Scope: Apache Airflow 3.3.x, Prefect 3, Kubernetes 1.36, and OpenLineage 1.53 documentation current 2026-08-31.

Curated: · Written: · Reviewed:

25 questions

MCQ-1A nightly pipeline is modelled as a DAG. extract_a (10 min) and extract_b (6 min) have no dependencies and can start together. transform (8 min) depends on both extracts. load (4 min) and report (5 min) each depend only on transform. The scheduler starts any task as soon as its dependencies finish and runs as many tasks concurrently as it likes. What is the shortest possible wall-clock time for the whole pipeline?
MCQ-3A daily DAG is scheduled at 00:00 UTC. The run that fires at 2024-03-02 00:00 UTC must read exactly the events produced during the previous calendar day, and re-running that same scheduled run later must read the same events. Which value should the task use to select the partition?
MCQ-4In Airflow 2.x, the DAG `etl` defines three tasks: `extract`, `transform`, `load`. `transform` is declared with `retries=2`. On Monday's scheduled run, `extract` and `load` succeed on their first attempt, while `transform` fails once, is retried, and succeeds on the second attempt. On Tuesday an engineer backfills the same DAG over the four days preceding Monday, and every task in those four runs succeeds on its first attempt. No other runs of this DAG exist. How many task instance records does the metadata database hold for this DAG across all five runs?
MCQ-2An Airflow DAG runs daily at 02:00 UTC. `extract` writes `s3://raw/dt=<ds>/orders.parquet` and takes about 10 minutes; `rollup` reads the `dt-3` through `dt-1` partitions and takes about 25 minutes; `publish` refreshes the BI dashboard from `warehouse.daily_rollup`; `cleanup` deletes every object under `s3://raw/` whose last-modified time is more than 72 hours old and takes about 40 seconds. The only edges are `extract -> rollup -> publish`; `cleanup` has none. About one run in twenty fails with `FileNotFoundError` on the `dt-3` partition, and the failures cluster on runs that start more than ten minutes late. Which change removes the failure?
MCQ-19The 03:12 nightly `orders_daily` run reports `success` and every task instance shows a terminal state, yet the next morning the `2024-06-11` partition is empty while the source held 1.2M rows. Which of these does that `success` state not prove?
MCQ-24An hourly Airflow pipeline covers the 02:00–03:00 interval. Its first task waits on an upstream dataset that lands at 03:18:00. The sensor polls every 5 minutes and last polled at 03:15:00, so it detects the dataset at 03:20:00. The transform then runs 5 minutes, the load 4 minutes, and a validation task 3 minutes, after which the table is queryable. The team measures freshness as the delay from the close of the data interval to the moment the validated table can be queried. What freshness should the dashboard report for this interval?
MCQ-17A nightly index rebuild runs as a Job with completions: 8, parallelism: 4, and backoffLimit: 2. On three of the Pods the container exits with code 1 because of a bad shard. What does the Job controller do?
MCQ-7A client library retries failed calls to a downstream API immediately, up to 5 times. During a 20-second downstream outage, 800 client instances fail at the same instant and retry in lockstep; the downstream's recovery is delayed to 90 seconds because each retry wave arrives as one burst. The team is considering adding exponential backoff with jitter. What does that change accomplish here?
MCQ-5A payments worker calls an API through a retry wrapper that makes up to 3 attempts with jittered backoff, caps each delay at 30 s, has no token-refresh hook, and sends no idempotency key. Which failure should be added to the wrapper's retryable set?
MCQ-6A queue retries a failed charge job by inserting a new job row with a new id and re-enqueuing it. The worker sets the Idempotency-Key header to the job row's id. The provider's docs say a repeated key returns the original charge and never creates a second one, and keys stay valid for 24 hours. During an incident, 800 charge jobs timed out after the provider had already processed them; each was retried once, minutes later, with the same body. Finance reports 1,600 charges. What is the cause?
MCQ-18A checkout saga charges the customer's card $200 at the payment gateway, then reserves inventory. The reservation returns 409 (sold out), so the orchestrator compensates by calling the gateway's refund endpoint. The refund request times out after 2 s and the orchestrator cannot tell whether the gateway processed it. It retries. What must the retry send so the customer ends up refunded exactly once, whether or not the first attempt landed?
MCQ-14A DAG has 200 tasks. Each task calls a vendor API (about 2 s per call; the vendor allows 8 concurrent requests) and then runs a 60 s local aggregation. The team creates an Airflow pool `vendor_api` with 8 slots and assigns every task to it; the executor can run 32 tasks at once. The vendor never sees more than 8 concurrent requests, but the DAG takes about 26 minutes instead of the ~1 minute the team expected. Which change brings the runtime down to about 1 minute?
MCQ-8An Airflow 2.9 deployment runs a DAG of 40 sensor tasks, each waiting for a file that lands about 45 minutes after the DAG starts. Every sensor uses the default mode with poke_interval=30s and timeout=2h, and all tasks share a worker pool of 16 slots. What happens while the files are pending?
MCQ-25An orchestration cluster runs dozens of DAGs containing sensors waiting on upstream hourly object arrivals. During peak load, all worker execution slots become occupied by these polling sensors, preventing downstream compute tasks from starting and causing execution deadlocks. Which mechanism directly resolves this worker slot starvation while keeping polling active?
MCQ-20A nightly Airflow DAG lists the keys under an S3 prefix in one `@task` and then runs `process.expand(key=keys)`. The prefix normally holds about 300 objects, and the mapped task is configured with `max_active_tis_per_dag=16`. A backfill accidentally writes 40,000 objects into that prefix before the next run. What happens when the run starts?
MCQ-15A rendering service runs 24 workers off a single FIFO queue. Interactive requests arrive at 600/s and consume 10 ms of worker time each; batch jobs consume 60 s each. Load tests show interactive p99 stays under 100 ms only while its lane runs at or below 80% utilization: above that it climbs past 1 s. The team splits the pool into two queues with a fixed worker split, where interactive workers never take batch work and vice versa. Which split maximizes batch throughput while holding interactive p99 under 100 ms?
MCQ-13A nightly pipeline shares a 40-slot worker pool. Live scheduled work needs 30 slots at peak between 01:00 and 04:00 and 6 slots the rest of the night. A 90-day backfill runs against the same pool with a cap of 4 concurrent runs; each run holds 5 slots for its whole duration. The scheduler is FIFO and never preempts. On the first night, live task queue wait p99 rises from 20 s to 11 min inside the 01:00–04:00 window and the 06:00 freshness SLA is missed; the backfill's last run finishes at 05:10. What caused the miss?
MCQ-12A team ships a fix to a transformation that changes the output for every date. They backfill a 30-day range in which all 30 days already have successful runs from before the fix, and they need every day recomputed without discarding the existing run history. Which setting does that?
MCQ-11A team is repairing 14 months of a daily-partitioned table after a transformation bug. The backfill request takes a start date, an end date, and a catchup policy that skips dates already marked successful. Before committing cluster time, the engineer runs the identical request with the dry-run flag. What does that run establish?
MCQ-10A daily Airflow DAG writes one partition per day into `events_daily` with `INSERT OVERWRITE PARTITION (ds = '{{ ds }}')`, joining each event to a per-currency rate row. A bug shipped on 2024-01-01 joined the wrong rate table and was fixed on 2024-04-01; every run in between wrote wrong revenue. The DAG has run successfully every day since, is unpaused, and the source events for those dates are still in the landing bucket. The team needs the 2024-01-01 through 2024-03-31 partitions corrected. What should they do?
MCQ-23An Airflow deployment runs a daily DAG with 24 hourly extract tasks that must be backfilled for 730 days (17,520 task instances) by unpausing the DAG and letting the scheduler create the runs. At default settings this saturates two resources: the shared scheduler metadata DB (the live hourly pipeline's scheduling latency climbs) and the warehouse, which rejects more than 10 concurrent sessions from the backfill. The live pipeline must keep running. Which configuration paces the backfill within both limits?
MCQ-9A DAG's mapped group of 400 tasks each returns a 6 MB DataFrame as its task output. The orchestrator keeps task outputs in the same Postgres metadata store the scheduler uses for task state, and the scheduler reads each output when it renders a downstream task's parameters. The store grows 2.4 GB per run and the scheduler's loop p99 goes from 45 ms to 700 ms. Downstream tasks must keep reading each DataFrame. Which change brings the loop back toward its 45 ms baseline?
MCQ-16A nightly DAG has 12 tasks. The 2025-06-01 run starts at 02:00. Task 7 fails at 02:35 on a transient connection reset; it has retries=1 and retry_delay=15 minutes. At 02:40 a deploy lands that changes task 7's output prefix from `s3://lake/stage/` to `s3://lake/stage_v2/`. The run is not pinned to a bundle version. Where does the 02:50 retry of task 7 write?
MCQ-22A workflow orchestrator writes each task's rendered command and run parameters to its metadata store, and appends stdout/stderr to a log object. A security review finds this log line for a production task: `psql postgres://app:Tr0ub4dor@db.internal:5432/prod -c 'select email from users'`. The same connection string is in the task's run parameters. The log object is readable by every workspace member, and the metadata store is encrypted at rest. Which change removes the credential and customer data from the artifacts while keeping the task runnable?
MCQ-21A lineage graph is built from the read and write events each job reports, tagged with job name, code version, and run window. Which claim does that graph establish on its own?