Blog · Pipelines
Entity resolution: working out which records are the same thing
Entity resolution is the work of deciding which records, within one dataset or across several, refer to the same real-world thing. It is needed whenever records lack a shared, reliable identifier, which is most of the time.
You will also see it called record linkage, deduplication, identity resolution or simply matching. The names come from different fields, but the problem is the same.
Why it is harder than it looks
Imagine three records: "Northfield Robotics Ltd", "Northfield Robotics Limited" and "NORTHFIELD ROBOTICS". A person sees one company at a glance. A database sees three different strings.
Now add the harder cases:
- Spelling mistakes and typing errors
- Abbreviations and legal suffixes such as Ltd, Inc or GmbH
- Trading names that differ from registered names
- Companies that change their name, merge or move
- People who share a common name, or who change theirs
- Names written in a different script and spelt several ways in English
- Many unrelated businesses registered at the same address
- Missing fields, so there is little to compare
Exact matching catches almost none of this. Loose matching catches it and also joins things that should stay apart. Entity resolution is the discipline of finding a sensible middle, and of being honest about what is left uncertain.
The usual steps
1. Standardise
Put each field into a consistent form before comparing. Convert to one case, trim spaces, remove punctuation, expand or strip common suffixes, format dates one way and split addresses into parts. Keep the original value too. Standardising is for comparison, not for replacing what the source said.
2. Block
Comparing every record with every other record does not scale. The number of pairs grows with the square of the number of records, so a million records gives roughly five hundred billion pairs. Blocking cuts this down by only comparing records that share something simple, such as the same postcode, the same first few letters of a name or the same year. Good blocking uses several overlapping keys, so that a typing error in one field does not hide a true match.
3. Compare
For each candidate pair, compare field by field. Names are often compared with string similarity measures. Edit distance counts how many single-character changes turn one string into another. Other measures give more weight to agreement at the start of a string, compare sets of words regardless of order, or compare how names sound. Dates, numbers and places each need their own comparison.
4. Score and decide
There are three broad approaches.
| Approach | How it works | Trade-off |
|---|---|---|
| Rules | "Match if the registration number agrees, or if name and postcode both agree" | Easy to explain, but brittle and hard to tune |
| Probabilistic | Each field's agreement or disagreement adds or subtracts weight, and the total is compared with thresholds | Handles partial evidence well, but needs care to set weights |
| Machine learning | A model learns from pairs that people have labelled as match or non-match | Can be accurate, but needs labelled examples and is harder to explain |
A useful idea from probabilistic linkage is that agreement is worth more when the value is rare. Two records sharing an unusual surname is stronger evidence than two records sharing a very common one.
Most systems use two thresholds rather than one. Pairs above the upper line are accepted, pairs below the lower line are rejected, and pairs in between go to a person.
5. Cluster
Decisions are made on pairs, but the goal is groups. If A matches B and B matches C, are all three the same entity? Often yes, but chains like this can join A and C even when they plainly differ. Clustering has to check that the group holds together as a whole and not only link by link.
6. Review
Uncertain pairs go to a person, who approves or rejects them. The decision must be stored and respected on every later run. A system that keeps proposing a match you have already rejected teaches reviewers to stop paying attention. For more on this, see human in the loop.
Two kinds of error
A false match joins records that belong to different entities. A missed match leaves records apart that belong together.
Two standard measures describe this. Precision is the share of the matches you made that are correct. Recall is the share of the true matches that you found. Tightening the rules raises precision and lowers recall. Loosening them does the opposite.
Which error is worse depends on the use. If you are building a mailing list, a missed match means someone gets two letters. If you are attaching a court record or a debt to a person, a false match can do real harm. Decide which error matters more before you set thresholds, and measure both against a sample that people have labelled by hand.
Practical habits
- Use real identifiers where they exist. Registration numbers, publication numbers and similar keys beat any name comparison. Treat them with some caution, since they can be mistyped or missing.
- Never merge destructively. Keep every source record as it arrived and store the link as a separate fact. A wrong link can then be undone without losing anything.
- Store the evidence for each link. Which fields agreed, what the score was, who approved it and when. This is the linking equivalent of keeping receipts.
- Expect entities to change. A company renames itself. A person moves. Keep old names and addresses as history, since an old record will still use them.
- Run it again. New data brings new matches and sometimes shows that an old match was wrong. Entity resolution is a continuing process inside a data pipeline, not a one-off clean-up.
- Mind privacy. Linking records about people can be regulated, and the combined record may be more sensitive than its parts. Check with your own legal adviser before linking personal data.
What good looks like
A good system can answer three questions about any entity it shows you. Which source records sit behind it? Why were they joined? Who decided the uncertain ones? If it cannot answer these, treat its totals with care, because every count of "how many companies" or "how many people" depends on the linking underneath.
Where Prism fits
Linking is the third of seven steps in the engine Prism Labs builds: gather, keep receipts, link, review, answer, watch and deliver. The link step works out which records are the same company, field, person or patent. The review step puts a person in charge of approving or rejecting, and a rejected link stays rejected. You can read more on the industries page.