Blog · Data custody
Data provenance: keeping receipts for every record
Data provenance is the recorded history of a piece of data: who published it, how and when you obtained it, and what has been done to it since. Keeping a receipt for every record means that any number or claim in your system can be traced back to the exact file it came from.
The idea is borrowed from archives and the art world, where provenance is the documented history of an object. For data the purpose is the same. Without it you have a value. With it you have evidence.
Why bother
Most data problems surface late. A figure in a report is challenged. Two systems disagree. A source quietly corrects a dataset and your totals shift. At that moment the only useful question is "where did this come from?", and the honest answer in many organisations is "a spreadsheet someone downloaded a while ago".
Provenance turns that into a lookup. It pays off in several ways:
- Defensibility. You can show a funder, auditor or reviewer the source behind a number.
- Debugging. When output looks wrong, you can tell whether the source was wrong, the parser was wrong, or a later step was wrong.
- Reproducibility. You can rebuild a result from the same inputs and get the same answer.
- Change detection. You can tell that a publisher has altered a file, not only that your numbers moved. See what changed since last time.
- Licensing. You know the terms under which each record was obtained.
- Trustworthy AI answers. A citation is only as good as the record it points to. See why AI answers need sources.
What goes on a receipt
A shop receipt records who sold what, when, and for how much. A data receipt is the same idea. For each file or response you fetch, store:
| Field | Why it matters |
|---|---|
| Publisher | The organisation responsible for the data, not only the website it sat on |
| Request | The exact address and parameters used, so the fetch can be repeated |
| Retrieved at | When you fetched it, which is not the same as when it was published |
| Response details | Status, content type, and any last-modified or version information the source gave |
| Checksum | A fingerprint of the raw bytes, to prove the file has not changed |
| Raw copy | The file itself, stored untouched |
| Licence or terms | What you are allowed to do with it |
| Fetcher version | Which version of your code did the fetching |
Each record you later extract from that file carries a pointer back to the receipt, ideally with its position in the file, such as a row number or page.
Checksums in plain terms
A checksum, in this context, is the output of a cryptographic hash function such as SHA-256. The function reads a file of any size and produces a short fixed-length string. The same file always gives the same string. Change a single character and the string changes completely. It is not practical to craft a different file that gives the same string.
That gives you three useful things. You can prove the stored file is the one you fetched. You can notice when a publisher changes a file without announcing it, because a fresh fetch gives a different checksum. And you can spot duplicates without comparing whole files.
A checksum proves a file is unchanged since you hashed it. It says nothing about whether the content was correct in the first place.
Keep the raw file, separately from the parsed data
The most common regret in data work is having kept only the cleaned version. Parsers have bugs. Requirements change. A field you ignored becomes important. If the raw file is gone, you cannot go back, and the source may no longer offer the old version.
So the rule is: store raw responses exactly as received, never edit them, and derive everything else from them. Storage is cheap compared with the cost of not being able to explain a number.
Lineage: provenance through the pipeline
Provenance at the point of collection is only the first link. Data is then parsed, cleaned, matched and combined. Lineage is the record of those steps.
For each derived record or result, note which inputs it came from, which version of the code produced it, and when. When two records are judged to be the same company or person, keep both originals and record the decision, including who made it and on what grounds. The guide on entity resolution explains why those decisions need a trail of their own.
There is a formal vocabulary for this. The W3C PROV standard describes provenance in terms of entities, the activities that produced them, and the agents responsible. You do not need to adopt it to benefit from the idea, but it is a useful reference if you need to exchange provenance with others.
Append, do not overwrite
If you replace last month's value with this month's, you have destroyed the ability to say what you knew at the time. Keep versions. A new fetch creates a new receipt and new records; the old ones stay, marked as superseded. Corrections by a person are recorded as corrections, with the original preserved.
This is the same discipline as a chain of custody for physical evidence: every hand-off is written down, and nothing is quietly swapped.
Common mistakes
- Recording the website, not the publisher. Data portals host material from many bodies. Note who is actually responsible.
- Storing a link and not the file. Links break and pages change.
- Timestamps without time zones.
- Provenance in someone's head. If only one person knows where a file came from, you have no provenance.
- Hand edits to data files. If a value must be corrected, record the correction as a step, not as a silent change.
- Provenance added later. It is far easier to capture at the moment of fetching than to reconstruct.
Starting small
You do not need a platform. A folder of raw files named by their checksum, and one table of receipts, is a sound beginning. Add the pointer from each parsed record back to its receipt. Add lineage when you add processing steps. The guide to the seven steps of a data pipeline shows where this sits in the whole.
If the data includes personal information, or is subject to licence limits on storage, keeping raw copies has its own obligations. Check with your own adviser.
Where Prism fits
"Keep receipts" is the second of the seven steps in Prism's engine: every record keeps its publisher, request and file checksum. Later steps depend on it. A person approves or rejects links, and the AI must cite stored evidence, with uncited claims refused automatically. See industries for where the engine is applied.