Topics discussed:
- The session started by pulling apart three things people had been calling one problem: writing view output to Parquet, which is easy because views are already flat; a generic FHIR-in-Parquet data lake you can query with off-the-shelf tools; and pure transport, where a columnar format may be the wrong choice altogether. The motivation for any of it is the threshold argument — if converting costs extra effort, fewer people do it. JSON and XML are nearly isomorphic, and that small difference in friendliness changed what got built.
- A devil's advocate proposal: if bulk export gains a binary format it should be Avro, since Avro to Parquet is trivial — solve Avro's schema and everyone downstream gets Parquet for free. The counters were that Avro buys you nothing if your destination is a columnar store, and that gzipped NDJSON compresses about as well as Avro anyway, because it is the repeated field names taking up the space. Noted as considered, not settled.
- What makes a lossless FHIR-to-Parquet schema hard: recursion has no limit, so implementations make you nominate a maximum nesting depth when encoding; primitive extensions need a parallel field for every single field; and open types mean encoding a full nested structure for every type an element could be. Lossless is probably achievable, but the schema becomes massive and the queries over it complex — and in Spark, schema size tracks performance directly. The counter-position was that a generic lossless schema is impossible, while a lossless schema for your own data is fine if you know your recursion limits.
- Two designs for extensions. First-class columns — a
racecolumn for US Core — are fast and obvious to query, but make the schema profile-specific: you must know the profiles at encode time, and files from different profiles do not merge without extra work. The generic alternative puts a field identifier on each backbone element plus a map at the root from field id to extension structures, joined back with SQL and hidden behind FHIRPath'sextension(). A third option came out of the argument: make first-class extension columns optional and lean on Parquet's schema evolution, since extension columns are purely additive and a file with none could merge with a file with many. - Pushing processing to the source: rather than "export everything losslessly", say "export according to this ViewDefinition". Broad agreement that it is a clear win — you do not ship data you do not need and you skip a serialise/deserialise round trip — tempered by nobody expecting EHRs to support it soon. Whether it is an extension to bulk export or a separate operation was left open.