Data Pipeline
intermediateDefinition
An automated flow that extracts data from a source, transforms it into a useful shape, and loads it into a destination. Often called ETL (Extract, Transform, Load). Data pipelines run on a schedule to keep dashboards, reports, and databases up to date.
In the wild
A nightly pipeline pulls raw sales data from Stripe (extract), calculates daily revenue and groups it by product (transform), and writes the summary into a reporting database (load). The dashboard reads from the reporting database each morning.
More from Data & Transformation
Aggregation
Combining many individual data points into a summary. Aggregations answer questions like 'how many,' 'what's the total,' or 'what's the average' by collapsing a list of values into a single result.
Deduplication
Removing duplicate entries from a dataset so each record appears only once. Duplicates creep in when data is imported from multiple sources, forms are submitted twice, or syncs overlap. Deduplication keeps your data clean and your counts accurate.
Normalization
Restructuring data to reduce duplication and inconsistency. Instead of repeating the same information in multiple places, normalization splits data into related tables or structures and connects them with references.
Null Value
A special placeholder meaning 'no data here.' Null is different from zero, an empty string, or false. Those are all real values. Null means the value is deliberately absent or unknown. Forgetting to handle null is one of the most common sources of bugs.
Parsing
Reading raw text and extracting structured information from it. Parsing turns unstructured data (a string, a file, a response body) into something your code can work with: objects, arrays, numbers, dates.
Schema
A blueprint that defines the expected shape of your data: what fields exist, what types they are, and which ones are required. Schemas catch bad data early by rejecting anything that doesn't match the expected structure.