What makes data good — or bad?
It sounds like a simple question and it is not. Ask a FHIR implementer what data quality means and you get the validator: profiles, cardinalities, terminology bindings, invariants. Ask a data engineer the same question and you get dbt tests: not-null, unique, accepted values, freshness.
Both answers are half right, and the halves do not overlap the way either side assumes. The confusion has a price: teams either rebuild the validator in SQL, or they read a green validation run as "the data is fine" — and both mistakes surface late, usually when a quality measure returns a number nobody can defend.
The good news is that the health-data world already worked out a precise answer to "what is good data" a decade ago — the Kahn framework. This article walks through it and maps it onto FHIR: what your validator already covers, and what it structurally cannot.
I went looking for it while porting OMOP's data quality checks onto SQL on FHIR. The mechanics were easy; saying which checks belong to the validator and which do not was the hard part — and this framework is what draws the line.
Kahn, in three questions
Kahn et al., A Harmonized Data Quality Assessment Terminology and Framework (2016). It is not a metric set and not a tool — it is a terminology, written to merge a dozen incompatible vocabularies different groups had each invented. That is exactly why it stuck. Arguing about names is cheaper than arguing about measurements.
It asks three questions of the data:
| Question | Kahn calls it | Example |
|---|---|---|
| Is it recorded correctly? | Conformance | Sex only has values M, F or U |
| Is the value there at all? | Completeness | 40% of observations carry no value |
| Can the value be believed? | Plausibility | A body weight of 1000 kg |
Two details are load-bearing. Completeness is "without reference to data values" — it counts how often something is present, never what it says. And plausibility is about believability, not truth: whether 78 kg is plausible, not whether the patient actually weighs 78 kg. Nothing in the data can answer the second question, and the framework does not pretend otherwise.
The trap: "validation" means two different things
Before going further — this word will bite a FHIR audience, so let me defuse it.
In FHIR, validation means checking a resource against a StructureDefinition. In Kahn, validation means comparing data against an external benchmark, as opposed to verification against your own expectations. Unrelated concepts, same word, and they overlap just enough to mislead:
- Validating a resource against US Core is Kahn-validation — the rule came from outside.
- Validating the same resource against a profile you wrote yourself is Kahn-verification — the rule is yours.
The validator does identical work in both cases. Only the provenance of the yardstick changed. A practical test: can you run this check with nothing but your own database? A weight of 1000 kg — yes. Diabetes prevalence matching the national rate — no.
One warning, because the wrong version circulates widely: verification = conformance, validation = completeness + plausibility is not what the framework says. The two axes genuinely cross. OHDSI's Data Quality Dashboard, the framework's reference implementation, populates all six cells.
The real line is scope, not category
Here is the part most write-ups get wrong. The split is not "the validator does conformance, quality checks do the rest." FHIR validation reaches considerably further than that.
It does plausibility, as long as the question fits inside one resource. minValue[x] / maxValue[x] is literally a plausible-range constraint:
{
"path": "Observation.value[x]",
"minValueQuantity": { "value": 0.5, "unit": "kg" },
"maxValueQuantity": { "value": 650, "unit": "kg" }
}
It does temporal plausibility. Every Period in FHIR already carries per-1:
per-1: "If present, start SHALL have a lower or equal value than end"
start.hasValue().not() or end.hasValue().not() or (start <= end)
And it does cross-field logic within a resource, via invariants:
obs-7: "If Observation.code is the same as a Observation.component.code
then the value element associated with the code SHALL NOT be present"
So the validator is not confined to structure. What it cannot do is anything that needs a population or a second resource:
| The question needs… | Example | Validator |
|---|---|---|
| one resource | weight within 0.5–650 kg; end not before start | ✅ |
| a proportion | 40% of Observations have no value | ❌ no denominator |
| a tolerance | 5% missing is fine here, 0% required there | ❌ validity is binary |
| another resource | Observation dated before the patient's birth | ❌ references not resolved |
| all records | one MRN per patient | ❌ no dataset in view |
| a distribution | mean weight, prevalence, drift | ❌ needs a population |
| an external benchmark | prevalence matches the national rate | ❌ nothing to compare to |
Which gives the honest one-liner:
A validator answers every question that fits inside a single resource, and no question that does not.
The one case that sits exactly on the line
An element is min=1 and absent. The validator reports it, and that looks like a completeness check. By Kahn it is conformance — a structural rule was violated, not a frequency expectation.
Real completeness is about optional elements populated so sparsely the column is useless. Same SQL either way; different category, different mechanism, decided entirely by whether the element was required.
What only a dataset can answer
Everything that is a property of the dataset rather than of any record in it. In SQL on FHIR these are ordinary queries over a flattened view — the whole point being that they return the offending rows:
-- cross-record uniqueness: one MRN per patient
SELECT mrn FROM patient_flat
GROUP BY mrn HAVING count(DISTINCT id) > 1
-- referential integrity: subject points at a Patient that isn't here
-- (a Bulk FHIR export fails this more often than you'd think)
SELECT o.id, o.patient_id
FROM obs o LEFT JOIN pat ON o.patient_id = pat.id
WHERE o.patient_id IS NOT NULL AND pat.id IS NULL
-- cross-resource temporal: observed before the patient was born
SELECT o.id FROM obs o JOIN pat ON o.patient_id = pat.id
WHERE o.effective < pat.birth_date
Plus the ones with no single-record analogue at all: proportions (40% of values missing), tolerances (5% acceptable here, 0% there), distributions (mean, quantiles, outliers, drift), timeliness (newest record is six weeks old — nothing about any record is wrong, the dataset is stale), and the roll-up that becomes the dashboard.
None of this is a gap a better validator could close. It is a different question.
And the reverse deserves saying, because a clean split needs both halves: SQL checks see only what a projection exposed. The validator inspects the whole resource — unprojected elements, slicing, extensions, invariants over nested structures, ValueSet expansion with hierarchy. Checks neither do that nor should. A check that re-implements a profile constraint is a second source of truth, and it will drift from the first.
Where Kahn runs out
Two gaps you hit immediately if you build on it.
No timeliness. "Is data still arriving?" is neither conformance, completeness nor plausibility. It is a real and common failure — Databricks' anomaly detection reduces to freshness and completeness — but the framework predates that framing. Either add a fourth category or file freshness under temporal plausibility and live with the awkwardness.
No accuracy. Plausibility is believability, not truth. Whether a recorded weight is the patient's actual weight is unanswerable from the data.
Two things OHDSI had to add in practice are worth adopting alongside the taxonomy: a level (dataset / column / code) and a severity (fatal / convention / characterization). Their severity split across 27 check types is instructive — 12 characterization, 8 convention, 7 fatal. Most checks describe rather than judge, which is a useful expectation to set before someone builds a dashboard that colours everything red.
Where this leaves us
A validator answers "is this resource well-formed?" A check answers "is this dataset fit to use?"
The second question is meaningless without the first, and the first is insufficient without the second. FHIR has an excellent answer to the first and, so far, no standard mechanism for the second.
The encouraging part is how little needs inventing. The vocabulary is settled. The reference implementation has run in OMOP-land for a decade. And in FHIR the pieces already exist: a ViewDefinition is the dataset, an SQLQuery is the check, and relatedArtifact already declares the dependency graph. What is missing is only the semantics — saying this query is a check, this is what it measures, and this much failure is acceptable. That is what the SQL on FHIR data quality work is about, and it is open.
Sources
- Kahn MG et al., A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data, eGEMs 4(1):1244, 2016 — PMC5051581. All quotations are from this paper.
- OHDSI Data Quality Dashboard — the 27-type classification matrix lives in
inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv.
