The Parquet session — three jobs called one problem, and two ways to handle extensions — May 8, 2024
Three jobs people had been calling one problem
- View output to Parquet — easy, because views are already flat
- Generic FHIR-in-Parquet data lake — queryable with off-the-shelf tools
- Pure transport — where a columnar format may be the wrong choice altogether
The motivation for any of it is the threshold argument — if converting costs extra effort, fewer people do it. JSON and XML are nearly isomorphic, and that small difference in friendliness changed what got built.
A devil's advocate — should bulk export use Avro?
If bulk export gains a binary format it should be Avro, since Avro to Parquet is trivial — solve Avro's schema and everyone downstream gets Parquet for free.
The counters:
- Avro buys you nothing if your destination is a columnar store
- Gzipped NDJSON compresses about as well as Avro anyway, because it is the repeated field names taking up the space
Noted as considered, not settled.
What makes lossless FHIR-to-Parquet hard
- Recursion has no limit — so implementations make you nominate a maximum nesting depth when encoding
- Primitive extensions need a parallel field for every single field
- Open types mean encoding a full nested structure for every type an element could be
Lossless is probably achievable, but the schema becomes massive and the queries over it complex — and in Spark, schema size tracks performance directly.
The counter-position: a generic lossless schema is impossible, while a lossless schema for your own data is fine if you know your recursion limits.
Two designs for extensions
First-class columns — a race column for US Core — are fast and obvious to query, but make the schema profile-specific: you must know the profiles at encode time, and files from different profiles do not merge without extra work.
Generic alternative — put a field identifier on each backbone element plus a map at the root from field id to extension structures, joined back with SQL and hidden behind FHIRPath's extension().
A third option came out of the argument — make first-class extension columns optional and lean on Parquet's schema evolution, since extension columns are purely additive and a file with none could merge with a file with many.
Pushing processing to the source
Rather than export everything losslessly, say export according to this ViewDefinition. Broad agreement that it is a clear win — you do not ship data you do not need and you skip a serialise/deserialise round trip — tempered by nobody expecting EHRs to support it soon.
Whether it is an extension to bulk export or a separate operation was left open.