Distributed Data Processing

What it is

Distributed processing frameworks (Spark, and similar) split data and computation across many machines to process datasets too large for a single node.

Key points

• Partitioning: data is split into chunks distributed across workers — how well the partitioning matches the workload (e.g. by join key) heavily affects performance.
• Shuffling: redistributing data across the cluster (needed for operations like group-by or join on a non-partition key) is the most expensive operation in distributed processing — it involves network transfer between every node.
• Lazy evaluation: frameworks like Spark build up a computation plan and only execute it when a result is actually needed, allowing the engine to optimize the whole plan rather than executing each step eagerly.
• Data skew — when one partition is much larger than others — creates a straggler that the whole job waits on, often the real-world cause of a "why is this job slow" investigation.