A Parquet schema that fits the data instead of covering all of FHIR — Jul 16, 2024

Why several FHIR-Parquet schemas already exist and diverge

Pathling, Open Health Stack, others — most descending from the Bunsen library and loosely based on SQL on FHIR v1.

They diverge for a good reason: a schema that can hold any possible FHIR data is effectively infinite, because open types let an element be any of twenty or thirty types and nesting recurses. So everyone picks their own cut-off — maximum depth, which types to encode — which is fine locally and useless for sharing, since the writer and the reader may not have picked the same limits.

John's alternative — say how each element is represented if present

One schema per resource, fitted to the data actually in the file. It stays very close to the JSON structure — extensions live in an extension child of each complex type, primitive extensions use the same underscore convention JSON uses — which means JSON schema inference already gets you most of the way to a conforming file.

Original representation + annotation columns

Keep FHIR's original representation and add annotation columns beside it rather than converting in place.

• Dates stay strings because casting a partial date to a timestamp silently loses the precision it was captured at
• Decimals stay strings to preserve scale
• Quantities get a UCUM-canonicalised twin so a Celsius reading can be compared with a Fahrenheit one

Names in the standard annotation namespace carry standard semantics, and you can add your own without clashing. The idea traces back to prior work in the Sync for Science project.

The objections — all about tooling

• Schema evolution is well supported in Spark and shakier elsewhere, particularly for nested fields
• Merging schemas is expensive enough that Spark disables auto-merge by default
• SQL written against one file breaks on another that lacks a column

The answer offered: this is precisely the view runner's job — Parquet is self-describing, so a runner can check whether a column exists and return an empty collection when it doesn't, which is exactly what absence means in FHIRPath.

Left open

Is Parquet here a transport format — a columnar NDJSON for bulk export — or something you point a query tool at directly? The requirements differ, and missing columns only really hurt in the second case.

Also unresolved: whether a subgroup should own this, since the weekly call can't carry it.