Data quality is a decision-risk contract, not a dashboard score

Data quality asks whether a specific dataset version is fit for a named consumer decision under explicit semantics. Generic labels—complete, accurate, fresh, valid, unique, consistent—become useful only when tied to a population, grain, business time, source of truth, threshold, severity, owner and action. A table can be 100% non-null yet wrong, fresh yet incomplete, internally consistent yet inconsistent with the source, or statistically normal after a systematic corruption. Begin with decisions and failure consequences, then design controls at the earliest boundary that can detect or prevent harm.

Schema and contract checks validate structure, types, required fields, domains, compatible evolution and producer identity. They do not establish semantic correctness. Record-level rules cover values and cross-field invariants. Dataset-level rules cover row populations, uniqueness at declared grain, referential coverage, control totals, distribution, duplicates, ordering and time bounds. Cross-system reconciliation compares source and target counts, amounts, keys and positions with reason-coded expected differences. Consumer-level checks validate metric formulas, join cardinality and fitness for the actual decision.

Freshness needs event/business time, ingestion time, processing time and consumer publication time. A recently updated table may contain yesterday's source position; a quiet source may legitimately produce zero rows. Define expected intervals, watermark/position, completeness signal, late-data policy and deadline. Separate timeliness from completeness. For streaming or incremental data, validate accepted, quarantined, corrected and explicitly dropped populations so missing records do not disappear from the denominator.

Static constraints catch known invalid states. Statistical and anomaly checks compare metrics such as volume, null rate, distinctness, quantiles or category proportions with suitable historical peers. Baselines must segment by weekday, season, region, tenant, version and known events; otherwise normal launches page operators while slow corruption becomes the new normal. An anomaly is evidence for investigation, not proof of bad data. Store observed metrics and model/version, monitor threshold performance, and retain deterministic guardrails for impossible states.

Severity and result are different. An assertion can fail while policy chooses warning, quarantine, degraded publication or hard block. Preserve both facts. Consequence depends on affected population, magnitude, downstream use, reversibility and confidence. Critical financial identity or cross-tenant violations usually block; an experimental nullable description may warn. Do not allow warning accumulation to become permanent acceptance. Every failure path needs owner, evidence, deadline, replay/repair and consumer communication.

Quality execution is versioned production software. Bind results to dataset/snapshot/partition, source position, schema, code/config and run identity. Publish quality evidence with the data version; never mark a new version current while validating an older one. Protect samples, failed rows, queries, logs and profiling metrics because they can expose sensitive values and distributions. Use stable assertion identifiers and version material rule changes so trends remain interpretable.

Validation must itself be reliable. Test rules with known-good and deliberately bad fixtures, including null, duplicate, boundary, timezone, late event, deletion, SCD interval, fanout and malformed encoding. Detect checks that never execute, always pass, scan the wrong partition or compare incompatible populations. Monitor coverage, execution freshness, cost, false positives, false negatives, warning age and time-to-resolution. Optimize shared scans and incremental metric state only when merge semantics are valid.

Tool mechanics: expectations, analyzers, and facets

Great Expectations treats an Expectation as a named, parameterized assertion over a batch: the definition is reusable; a Checkpoint or validation run binds it to a dataset version and produces a result with success, observed value, and unexpected counts. Running validations without binding batch identifiers, suite versions, and action handlers leaves you with a screenshot, not an operational control. dbt data tests are SQL assertions over models, sources, seeds, and snapshots; generic tests (unique, not_null, accepted_values, relationships) are cheap coverage, while singular tests encode cross-table invariants. A green dbt test on yesterday's incremental slice does not certify today's untested partition.

Deequ's key concepts separate Analyzers (how a metric is computed, often sharing a scan) from Constraints (predicates over those metrics) and Verification (pass/fail against a dataset). Store analyzer results in a metrics repository so anomaly detection can compare volume, null rates, or distinctness against segmented history rather than a static threshold. Anomaly detectors that fire on every Monday after a weekend dip are mis-baselined, not “sensitive.” OpenLineage's data-quality assertions facet records assertion identity, column, success, severity, and expected versus actual; the metrics facet records row counts, nulls, distinctness, ranges, and quantiles. Publish both on the same run as the dataset version. JSON Schema 2020-12 and OpenAPI Schema Objects catch structural and domain violations at producer boundaries; they do not know whether a valid UUID still identifies the intended customer.

Failure modes and figures that dashboards hide

A uniqueness test that excludes nulls can pass while multiple NULL keys hide duplicate unresolved identities. Referential tests that ignore effective dates will pass facts that have no time-valid parent as long as a Type-2 current row exists for the key. Unstratified sampling of a 10-billion-row table can miss a 50-row poison tenant that is 100% of one consumer's traffic. Record true-positive and false-positive rates against labeled incidents: a rule with 40% precision will be muted. Track assertion coverage as (critical elements with an owned rule) / (critical elements), execution freshness versus the data's watermark, warning age in days, and time-to-quarantine. If restore from backup skips validators, treat absence after restore as unknown, not pass—even when the overview of the last successful run is still green.

Operate quality through prevention, detection and correction. Producer contracts and typed interfaces prevent classes of defects. Quarantine preserves invalid evidence without contaminating trusted outputs. Idempotent replay and versioned restatement repair history. Lineage supports impact analysis and routes alerts, but lineage and a green test suite do not prove correctness. Periodically reconcile against an independent source or hand-computed oracle and review whether controls still protect the decisions that justify them.

Worked example: unique skipping nulls hides 200 people

customers snapshot 2026-09-10: 10,000,000 rows. Finance joins on customer_id.

assertion	observed	decision
not_null customer_id	10,000,000 pass	green
unique customer_id (nulls excluded)	pass	400 NULL keys, 200 duplicate humans
unique treating NULL as unknown, plus source count 10,000,200	fail	quarantine; do not publish

A 100% non-null dashboard can still be the wrong grain. The interview is the excluded population.