Big data pipelines businesses can trust: quality, lineage and recovery

Choose batch, streaming or a combination

Batch processing is a good fit when information can be updated on a schedule, such as nightly finance reporting or periodic catalogue refreshes. Stream processing supports use cases where events need to be processed continuously, such as operational monitoring or near-real-time activity. Many organisations use both: streaming for timely signals and batch jobs for reconciliation, historical backfills and scheduled aggregates.

Define a data contract at each boundary

A data contract describes the structure and meaning of a dataset. It can include field names, types, nullability, units, identifiers, allowed values and freshness expectations. Contracts reduce ambiguity between producers and consumers. When a schema changes, teams can detect the impact before downstream dashboards and models silently break.

Design for late, duplicated and out-of-order events

Real systems do not always deliver events once, in order or immediately. A robust pipeline considers duplicate messages, delayed events, replay, corrections and clock differences. Use stable event identifiers, explicit event time, idempotent writes and documented rules for handling late-arriving records. The correct approach depends on the business meaning of the data.

Build data quality into the pipeline

Validate records as close as possible to the point where they enter the platform. Useful checks include schema validity, required fields, accepted ranges, uniqueness, referential integrity, freshness and reconciliation with source totals. Route invalid records to a controlled quarantine or dead-letter path with enough context to investigate them.

Make lineage and ownership visible

Lineage helps a team understand which source tables, transformations and jobs contribute to a report or model feature. It becomes especially useful during incidents, audits and schema changes. Assign owners to important datasets and publish a simple data catalogue that explains purpose, freshness, definitions and support expectations.

Monitor outcomes, not only job status

A green scheduler status does not prove that a dataset is correct. Monitor row counts, freshness, null rates, distribution shifts, processing lag, rejected records and reconciliation results. Alert on meaningful changes with a named owner and a clear runbook. Avoid alerts that are noisy enough to be ignored.

Plan for recovery and reprocessing

Keep raw inputs or a suitable replay source where policy permits. Make transformations repeatable and track the input range or offset that each run processed. Document how to backfill a date range, rebuild a derived table and prevent duplicate downstream effects. Test recovery procedures before a production incident makes them urgent.

Connect platform quality to business decisions

Work backwards from the reports, operational decisions or machine-learning features that consume the data. Define what freshness and accuracy mean for each use case. A daily management report may tolerate a scheduled update; an inventory alert may need a much shorter delay. Engineering choices should reflect those requirements rather than an abstract goal of “real time.”