Skip to content
Tech Interview Prep
Technical interview guide

Distributed Data Processing

How frameworks like Spark parallelize work across a cluster: partitioning, shuffling, and their costs.

Read
30 min
Practice MCQs
25
Interview Q&A
25
Edition
v4
Editorial status
Reviewed

Scope: Apache Spark 4.2 documentation, Apache Beam and Apache Hadoop current 2026-08-31.

Curated: · Written: · Reviewed:

25 questions

MCQ-2A Spark job reads 1 TB of `events` (400 partitions) and 2 GB of `users`, then runs: ```python events.select("user_id", "country", "bytes") \ .filter(col("country") == "US") \ .join(broadcast(users), "user_id") \ .withColumn("gb", col("bytes") / 1e9) \ .groupBy("country").agg(sum("gb")) \ .write.parquet("s3://out/") ``` The Spark UI shows a shuffle read of ~900 GB in a single stage. Which operation is the shuffle boundary?
MCQ-1A PySpark 3.5 job runs this script against a 240 MB Parquet dataset at `s3://events/`: ```python from pyspark.sql import functions as F raw = spark.read.parquet("s3://events/") clean = raw.filter("user_id IS NOT NULL") enriched = clean.withColumn("day", F.to_date("ts")) rejected = clean.filter("user_id IS NULL") rejected.write.mode("overwrite").parquet("s3://rejects/") print(enriched.count()) ``` How many bytes does the job read from `s3://events/`?
MCQ-4A Spark job reads 256 GB of Parquet as 4 partitions of 64 GB each. The cluster has 10 executors, each with 8 cores and 32 GB of heap: 80 cores in total. The first stage shows 4 running tasks, 76 idle cores, and about 50 GB of spill per task. The team re-reads the same 256 GB with a split size that yields 1,024 partitions; the filter in that stage is splittable and the input is evenly sized. Once the stage is running steadily, how many tasks run at once and how much data does each read?
MCQ-5A Spark stage runs on a 200-core cluster. Its input is split into 20,000 partitions, and each task's computation takes 50 ms. The driver's single-threaded task scheduler launches 2,000 tasks per second. Roughly how long does the stage take?
MCQ-3A Spark job's shuffle stage moves 400 GB from 400 map tasks with spark.sql.shuffle.partitions=400 and takes 6 minutes. An engineer raises the setting to 4000 to cut that time; the stage now takes 11 minutes and the driver holds far more shuffle metadata. What explains the regression?
MCQ-6Spark 3.5, `spark.sql.autoBroadcastJoinThreshold` left at its default. A nightly job joins `events` (600 GB Parquet, 2,000 partitions) to `users` (14 MB Parquet; catalog statistics report 14 MB) on `user_id`. Each of the three executors has an 8 GB heap. `EXPLAIN` shows a SortMergeJoin and the shuffle writes 600 GB. Which change makes the planner broadcast `users`?
MCQ-16A Spark 3.5 batch job runs with spark.sql.adaptive.enabled=false and the default spark.sql.autoBroadcastJoinThreshold of 10 MB. It joins events (2.1 TB) to users on user_id. ANALYZE TABLE users last ran before a backfill and still reports 8 MB and 60,000 rows; users is now 18 GB and 90 million rows. Executors die with OutOfMemoryError in the join stage. What most likely happened?
MCQ-15A nightly Spark 3.3 job joins a 1.2 TB fact table to a dimension table. The dimension is rebuilt each night, and the catalog statistics are stale: they still record it at 900 MB, so the planner chooses a sort-merge join. At run time the dimension is actually 12 MB, and the fact side is shuffled into 4,000 partitions. Adaptive query execution is enabled with default settings. What does the plan do with this join?
MCQ-7A Spark 3.5 job reads a Parquet table partitioned by `event_date`: 1,095 daily partitions of about 4 GB each. The query filters `WHERE to_date(event_ts) = '2024-03-01'`, where `event_ts` is a timestamp column stored inside the files. The job reads 4.4 TB and takes 38 minutes. `EXPLAIN` shows `PartitionFilters: []` on the `FileScan parquet` node. What does the plan show that the source text does not?
MCQ-17A Spark 3.5 job joins two tables with a sort-merge join. The shuffle writes 480 GB, and with `spark.sql.shuffle.partitions = 200` each reduce task sorts about 2.4 GB. Each executor has an 8 GB heap and 4 cores, and `spark.memory.fraction` is the default 0.6, so the execution pool is about 4.6 GB and each of the 4 concurrent tasks can hold roughly 1.2 GB of sort buffer before spilling. Every task in the join stage reports spill (memory) of about 2.4 GB. Which change removes the spill?
MCQ-18A Spark job on a 20-node cluster (256 GB RAM, 32 cores per node) runs with spark.executor.memory=64g, spark.executor.cores=16 and 40 executors. The Spark UI shows GC time at 38% of total executor time, p99 task duration 180 ms against a 40 ms median, and no disk spill. Losing one executor forces the stage to recompute the partitions whose cached blocks it held, adding about 6 minutes. Which change removes the largest of these costs?
MCQ-11A nightly pipeline builds `enriched`, a 40 GB DataFrame produced by a 9-minute shuffle join over a 1.1 TB Parquet scan. Four downstream aggregations each consume it once; with no cache, each one re-runs the join. The cluster has 600 GB of executor memory, 480 GB of which is free while the downstream stages run. What should the team do?
MCQ-12A nightly Spark 3.x batch job builds `enriched`, a 1.2 TB DataFrame, from Parquet on S3 through 50 narrow transformations, then runs four aggregations over it. The cluster runs on spot instances and loses about 4 of its 8 executors per run, taking roughly half of `enriched`'s partitions with them. With `enriched.cache()` alone the run takes 5 h 10 m, and the team measures 2 h 40 m of that as recomputing `enriched` partitions lost with the preempted executors. They add `enriched.checkpoint()` to S3 before the aggregations; the cluster's measured aggregate S3 throughput is 400 MB/s. What is the most likely new runtime?
MCQ-22A Spark job reads version 42 of `events`, joins it to a live `region_map` table to attach a `region`, and writes to `enriched` with `insertInto`. The team re-runs the job on version 42 to backfill a downstream fix; `region_map` was edited between the two runs. A dedup job keyed on `(event_id, region)` now reports 900 duplicate `event_id`s, each pair carrying a different `region`. What is the cause?
MCQ-9A Spark job's final stage runs 20,000 tasks. One task reads 40 million rows; the other 19,999 read about 200,000 each. The team grows the cluster from 50 to 200 executors (4 cores each) and the stage's wall-clock time barely moves. Which explanation accounts for the unchanged stage time?
MCQ-23A Spark 3.5.0 job's final reduce stage runs for 6 minutes 40 seconds and is the job's longest stage. The Spark UI reports 400 reduce tasks with shuffle read median 95 MB, p99 3.8 GB, max 3.9 GB; shuffle spill (memory) 0 B on every task; task duration median 11 s, p99 5 min 50 s; and executor GC time 2.4% of task runtime. The event log shows no fetch failures and no executor losses. Which diagnosis does this telemetry support?
MCQ-8A Spark 3.5 job joins a 2 TB fact table to a 40 MB dimension table on `tenant_id`. Stage 7 has 400 tasks: 399 read 45–60 MB of shuffle and finish in 40–55 s; task 217 reads 2.4 GB and runs 38 minutes. The cluster has 200 cores, and 199 of them sit idle for the last 37 minutes. `spark.sql.autoBroadcastJoinThreshold` is at its default of 10 MB. Which change removes the 38-minute tail?
MCQ-10A Spark job joins a 1B-row fact table to a 1M-row dimension table on user_id. user_42 has 400M rows in the fact table and all 1M rows in the dimension table. To break the skew, the team appends a salt 0–15 to user_42 in the fact table and replicates the dimension's user_42 rows once per salt. How many rows does the shuffle write in total?
MCQ-25A team capacity-tests a nightly Spark aggregation over 2 TB of clickstream data on a 40-executor cluster. The test replays 3× the daily volume with synthetic rows whose keys are uniformly distributed and finishes in 50 minutes, so the team signs off 3× headroom. In production the same 3× volume takes 4 hours; the Spark UI shows 12 of 400 tasks at 20× the median duration, one of them at 48×, all reading partitions that hold 40% of the rows, while the other 388 finish in the first 50 minutes. Which measurement should the capacity test have taken to predict the production run?
MCQ-13A Spark 3.5 batch job writes each partition to Postgres with a plain INSERT inside foreachPartition. spark.task.maxFailures is 4 and speculation is enabled. Partition 7's task inserts 10,000 rows; the server commits them, then the connection drops before the task reports success. Spark retries the task, the retry inserts the same 10,000 rows and succeeds, and no other attempt of partition 7 runs. When the job finishes, how many rows does the sink hold for partition 7's key range?
MCQ-14A Spark stage has 200 partitions. 199 tasks finish in about 45 s; one task's partition holds 8× the median bytes and has been running for 12 min. Speculation is enabled. The task's final step appends one ledger row per record through a non-idempotent API. What is the most likely outcome?
MCQ-21A nightly job reads 4.2 billion raw events, groups them by (user_id, event_date), and writes Parquet partitioned by date to object storage. Mid-run, 37 executors were preempted and the scheduler retried those tasks; the job then reported SUCCESS. Which evidence best establishes that the written output is correct?
MCQ-19A Spark 3.5 batch job runs on Kubernetes with dynamic allocation enabled: minExecutors=4, maxExecutors=40, executorIdleTimeout=30s, shuffle tracking disabled, and no external shuffle service. Stage 2 writes 0.7 TB of shuffle output. During the lull before stage 3 the driver removes idle executors, dropping the count from 12 to 4. Stage 3 then fails with FetchFailedException naming block locations on executors that are no longer running; stage 2 is re-executed and total shuffle read reaches 2.1 TB. Which explanation accounts for this?
MCQ-20A 200-core cluster runs a fair-share scheduler with two equal-weight queues, `live` and `backfill`, and preemption disabled. Live jobs hold 120 cores steady and spike to 180 cores at the 09:00 peak. At 02:00, with live at 120 cores, a backfill of 500 single-core tasks (each ~30 min) is submitted to `backfill` and fills the 80 free cores. At 09:00, what happens to live jobs?
MCQ-24A team runs Spark 3.5 with `spark.eventLog.enabled=true`; event logs land in a bucket the whole analytics org can read. A security review greps the event log files and finds `123-45-6789`, an SSN that appeared only in the `WHERE` clause of a query run last week. That query matched zero rows. What put the SSN in the event log?