SQL on FHIR WG Meetings
SQL on FHIR WG Meeting — July 2, 2024
Jul 2, 2024

Topics discussed:

  • Medplum's implementation is about 250 lines of TypeScript, written largely by transcribing the pseudocode in the spec, which mapped to TypeScript almost line for line. The test suite was imported with a thin shim around the test runner, passing roughly 98 of 120, with most failures being lowBoundary and highBoundary not yet implemented. It replaces a custom $csv operation that had two problems: it was proprietary, so anyone using it had to learn something that existed nowhere else, and it was nowhere near as capable as the spec.
  • It is exposed as an SDK primitive rather than a server operation — run bulk export, get NDJSON, crank it through the client-side SDK, pipe the result into a warehouse. That is deliberate: give it a view definition and a resource and you get a row back, which is fine for testing, but millions of rows need streaming and batching, so the scaling decisions are left to the developers using it. Asked why not compile the ViewDefinition into SQL instead, the answer was that it would not be a big lift — the server already translates search parameters into nested SQL joins — it just was not the first thing to build.
  • They use a clean-room FHIRPath rather than fhirpath.js, with the caveat that the comparison is a couple of years old: the client-side payload was heavy where structure definitions have to be available up front, and their model lazily fetches StructureDefinitions from the server as it goes. Their engine is well hardened by running on every write, evaluating every search parameter's FHIRPath to precompute values into relational columns across tens of millions of resources.
  • The other showcase came with a warning. A first-generation product on Athena over Parquet foundered on schema management: a programmatically generated FHIR schema became huge and unworkable, attributes had to be cut, nesting had to be capped, and different accounts ended up with different schemas. Their conclusion is that sharing FHIR via Parquet tends to mean custom Parquet files per share rather than one portable format.
  • The replacement puts top-level attributes into real columns in FHIR's own element order — not alphabetical, so it reads like the spec — with primitives in native types and complex types in Redshift's super, which holds JSON with no predefined schema. Primitive extensions are moved to a side field so the primitive column keeps its intended value. The smallest single node reportedly outran the Athena setup by around ten to one. The gotcha: below the first level there is no schema, so a missing attribute returns null and a join on that null collapses the whole result set — each unnesting belongs in its own CTE. And putting the entire resource in one super would defeat columnar storage, since you would read every field on every query.