Topics discussed:
- John walked through Pathling's Parquet schema, FHIR Data Pipes' schema, and a newer Parquet on FHIR spec meant to harmonise them. Both existing ones build one big static schema per resource type by walking the StructureDefinition. Extensions are where they part ways: FHIR Data Pipes promotes known extensions to top-level fields, so you can query US Core ethnicity directly — but only if you knew in advance you wanted it. Pathling keeps a generic extension structure with field IDs mapping into it, which is close to lossless.
- The maximal schema isn't strictly possible: FHIR's JSON structure is theoretically infinite, so you cap recursion depth, pick which open types to admit, and the schema balloons anyway. Parquet on FHIR's answer is that the spec describes the infinite schema and any given file merely conforms to a subset of it. The price is that whoever receives the data must tolerate absent fields and merge schemas across files that don't match.
- Someone pushed back with the goal question: what is this schema for? If you want to write SQL that runs against anything conforming to the spec, you need firm expectations about what will be there — a fluid schema and portable SQL pull against each other. Left open, with the profile idea borrowed from SQL on FHIR floated as a way to have several variants for different uses.
- Schema size isn't only a performance question. In Spark the schema compiles into an expression shipped between workers at query time, so a 2,000-line schema is real overhead; and some systems will not accept schemas that size at all, full stop. Cutting the other way: a minimal schema slots into a full one without loss, so a less capable client can still be handed the fat schema and never know.
- Round-tripping was promoted to an explicit design goal, and it's where FHIR bites. Store a decimal in a numeric type and you've lost the precision and scale it was captured at; convert a date to a native date and you've lost fidelity. The approach: keep the awkward primitives as strings and layer optional annotations carrying native typed forms for efficient querying, so the original FHIR can still be reconstructed. Next step agreed — prove it on an engine other than Spark, and publish sample Parquet files people can just download and poke at.