Topics discussed:
- Several Parquet schemas for FHIR already exist — Pathling, Open Health Stack, others — most descending from the Bunsen library and loosely based on SQL on FHIR v1. They diverge for a good reason: a schema that can hold any possible FHIR data is effectively infinite, because open types let an element be any of twenty or thirty types and nesting recurses. So everyone picks their own cut-off — maximum depth, which types to encode — which is fine locally and useless for sharing, since the writer and the reader may not have picked the same limits.
- John presented the alternative: a spec that says how each element must be represented if it is present, without requiring it to be present. One schema per resource, fitted to the data actually in the file. It stays very close to the JSON structure — extensions live in an extension child of each complex type, primitive extensions use the same underscore convention JSON uses — which means JSON schema inference already gets you most of the way to a conforming file.
- Keep FHIR's original representation and add annotation columns beside it rather than converting in place. Dates stay strings because casting a partial date to a timestamp silently loses the precision it was captured at; decimals stay strings to preserve scale; quantities get a UCUM-canonicalised twin so a Celsius reading can be compared with a Fahrenheit one. Names in the standard annotation namespace carry standard semantics, and you can add your own without clashing. The idea traces back to prior work in the Sync for Science project.
- The objections were all about tooling. Schema evolution is well supported in Spark and shakier elsewhere, particularly for nested fields; merging schemas is expensive enough that Spark disables auto-merge by default; and SQL written against one file breaks on another that lacks a column. The answer offered was that this is precisely the view runner's job — Parquet is self-describing, so a runner can check whether a column exists and return an empty collection when it doesn't, which is exactly what absence means in FHIRPath.
- Left open: is Parquet here a transport format — a columnar NDJSON for bulk export — or something you point a query tool at directly? The requirements differ, and missing columns only really hurt in the second case. Also unresolved was whether a subgroup should own this, since the weekly call can't carry it.