Blog · Pipelines
What is a data pipeline? Seven steps from raw records to answers
A data pipeline is a repeatable set of steps that takes records from the places they are published and turns them into something a person can query and trust. The useful ones do seven jobs in order: gather, keep receipts, link, review, answer, watch and deliver.
Why build a pipeline at all
Most analysis starts as a one-off. Someone downloads a few files, tidies them in a spreadsheet and answers the question of the day. That works once. It fails the second time, when the source has changed, the person has moved on and nobody can say which rows were edited by hand.
A pipeline writes those steps down as code and configuration so they can run again, the same way, on new data. The older name for this in data engineering is ETL: extract, transform, load. Data is extracted from a source, transformed into a consistent shape and loaded into a database or warehouse. A common variant, ELT, loads the raw data first and transforms it inside the warehouse. Both terms describe the plumbing. The seven steps below cover the plumbing and also the parts that make the output worth trusting.
1. Gather
Gathering means connecting to each source and pulling records on a schedule. Sources vary: bulk downloads, programming interfaces (APIs), files sent by a partner, exports from internal systems and sensor feeds.
Two habits matter here. First, store the raw response before changing anything, so you can reprocess it later when you find a bug in your cleaning code. Second, respect the source. Follow its terms of use, its rate limits and any licence attached to the data.
2. Keep receipts
For every record, keep a note of where it came from: the publisher, the request that fetched it, when it was fetched and a checksum of the file. A checksum is a short fingerprint computed from the file's contents. If the file changes by even one character, the fingerprint changes.
This is called provenance. It lets you answer the question every careful reader asks, which is "where did this number come from?" There is more detail in data provenance: keeping receipts for every record.
3. Link
Different sources describe the same real thing in different ways. One register spells a company name in full, another abbreviates it and a third uses a trading name. Linking, also called entity resolution or record linkage, works out which records refer to the same company, person, field or patent.
Linking is where much of the value and much of the risk sits. A missed link hides a fact. A false link puts one organisation's history onto another. See entity resolution explained for how matching works.
4. Review
Automated matching produces three piles: clear matches, clear non-matches and uncertain cases. A person should look at the uncertain pile, and at any finding that carries consequences.
The important design point is memory. If a reviewer rejects a match, the pipeline must remember that decision and not propose the same match again next week. Review decisions are data and deserve the same care as the records themselves. Good approval design is covered in human in the loop.
5. Answer
Once records are gathered, sourced, linked and reviewed, people can ask questions of them. The interface may be a dashboard, a search box, a query language or an AI model that writes an answer in plain language.
If an AI model is involved, one rule protects you: every claim in the answer must point to a stored record. A model that answers from its general training can state things that sound right and are wrong. A model limited to citing stored evidence can be checked, and an answer without a citation can be refused. Why AI answers need sources explains the approach.
6. Watch
A pipeline that runs once is a report. One that runs on a schedule can tell you what is different from last time: new records, removed records and changed values. For many users this is the main product. They do not want to re-read everything each month. They want the short list of things that moved. See change detection.
Watching also covers the pipeline's own health. Did every source respond? Did the row count drop sharply? Did a column change its format?
7. Deliver
The last step is putting the result where people work: a scheduled report, an export, an application or an alert. Delivery includes access control, since not everyone should see everything. It also includes versioning, so a reader can tell which run produced the figure in front of them.
Common ways pipelines fail
- Silent source changes. A publisher renames a column and the pipeline keeps running with empty values. Checks on row counts and expected formats catch this.
- Overwriting history. Replacing last month's data with this month's makes "what changed?" impossible to answer. Keep dated snapshots or a change log.
- Hand edits outside the pipeline. A fix made in a spreadsheet is lost on the next run. Corrections belong inside the pipeline as recorded decisions.
- No owner. A pipeline needs someone who is told when a run fails and who has time to fix it.
- Matching without review. Fully automatic linking looks fine until a wrong merge reaches a report.
Questions to ask before you build one
| Question | Why it matters |
|---|---|
| What decision will this support? | It sets which sources you need and how accurate the result must be |
| How often does the source change? | It sets the schedule, and there is no point running hourly against a quarterly file |
| Is there a reliable shared identifier? | If not, plan time for linking and review |
| Who reviews uncertain cases? | Without a named person, the uncertain pile grows and is then ignored |
| What happens when a source disappears? | Sources get withdrawn, moved and restructured, and your stored raw copies are the fallback |
| Are you allowed to store and reuse this data? | Licences and privacy law apply, so check with your own adviser |
A small pipeline that answers one question well is a better start than a large one that gathers everything. You can add sources later. It is much harder to add provenance later, because the receipts for old records were never kept.
Where Prism fits
Prism Labs builds an engine that follows these seven steps: gather, keep receipts, link, review, answer, watch and deliver. Every record keeps its publisher, request and file checksum. A person approves or rejects links, and a rejected link stays rejected. AI answers must cite stored evidence, and uncited claims are refused automatically. The result is packaged as a product that installs through Prism Desktop. You can read more on the industries page.