Data Engineering & Pipelines
Moving and shaping data reliably at scale: ETL/ELT, orchestration, warehousing, and data quality.
Subject: Data & Statistics · Roles: Data Architect, Data Engineer
Concepts
ETL vs. ELT
Transform-before-load versus load-then-transform, and why modern warehouses shifted the order.
Batch vs. Streaming Processing
Processing data in scheduled chunks versus continuously as it arrives — and the latency/complexity tradeoff.
Data Warehousing & Modeling
Star and snowflake schemas, and the fact/dimension split that makes analytical queries fast.
Data Pipeline Orchestration
Scheduling and sequencing interdependent pipeline steps as a DAG, with retries and backfills.
Data Quality & Validation
Catching bad data before it reaches downstream consumers — schema checks, freshness, and anomaly detection.
Distributed Data Processing
How frameworks like Spark parallelize work across a cluster — partitioning, shuffling, and their costs.