Three Parquet-for-FHIR schemas compared, and the case for not fixing your schema in advance — Feb 7, 2025

Walking through three schemas

John walked through Pathling's Parquet schema, FHIR Data Pipes' schema, and a newer Parquet on FHIR spec meant to harmonise them.

Both existing ones build one big static schema per resource type by walking the StructureDefinition. Extensions are where they part ways:

• FHIR Data Pipes — promotes known extensions to top-level fields, so you can query US Core ethnicity directly — but only if you knew in advance you wanted it
• Pathling — keeps a generic extension structure with field IDs mapping into it, close to lossless

The maximal schema isn't strictly possible

FHIR's JSON structure is theoretically infinite, so you cap recursion depth, pick which open types to admit, and the schema balloons anyway.

Parquet on FHIR's answer: the spec describes the infinite schema and any given file merely conforms to a subset of it. The price is that whoever receives the data must tolerate absent fields and merge schemas across files that don't match.

The goal question

Someone pushed back: what is this schema for? If you want to write SQL that runs against anything conforming to the spec, you need firm expectations about what will be there — a fluid schema and portable SQL pull against each other.

Left open, with the profile idea borrowed from SQL on FHIR floated as a way to have several variants for different uses.

Schema size isn't only a performance question

In Spark the schema compiles into an expression shipped between workers at query time, so a 2,000-line schema is real overhead — and some systems will not accept schemas that size at all.

Cutting the other way: a minimal schema slots into a full one without loss, so a less capable client can still be handed the fat schema and never know.

Round-tripping — an explicit design goal

Store a decimal in a numeric type and you've lost the precision and scale it was captured at; convert a date to a native date and you've lost fidelity.

The approach: keep the awkward primitives as strings and layer optional annotations carrying native typed forms for efficient querying, so the original FHIR can still be reconstructed.

Next step agreed: prove it on an engine other than Spark, and publish sample Parquet files people can just download and poke at.