The roadmap — publish v1, wire ViewDefinition into bulk export, spin FHIR-on-Parquet out — Feb 8, 2024
Publish what exists — stop adding features
The strongest roadmap opinion: publish the ViewDefinition resource, the documentation, a stable base with test cases that multiple implementations pass — call that version one and draw a line in the sand.
No ambition for new features that haven't been tested; declaring intent might just squeak in. The group agreed on that scope, which leaves the tests as the real blocker: two efforts need merging before they drift to cross purposes, and the highest-value thing anyone who has implemented this can do is plug the tests in even if they fail, because that exercise is what validates the tests and surfaces the hairy parts.
How do you tell a FHIR server to run a ViewDefinition?
One implementation does it implicitly — create the ViewDefinition and the view exists — but an explicit operation would let you parameterise what actually varies: temporary or unlogged table, materialised view or not.
The most wanted piece was the ability to ask in advance whether a server would run a given view, rather than turning it on and finding out — much the way SubscriptionTopic lets a system declare up front what it will do. Not before v1, though.
Bulk export with a ViewDefinition
Instead of exporting everything and flattening it afterwards, say give me a bulk export using this ViewDefinition, as Parquet or CSV. You get a bespoke extract, the implementation details stay under the hood, and it sidesteps what makes bulk data expensive today — serialising enormous amounts of data just to hand it all over.
A supporting number from one pipeline: rebuilding a view by reading from the FHIR server is 20 to 30 times slower than reading Parquet and applying the view to that. The database is the slow part.
Spin the columnar/JSON guidance out into a separate spec
The columnar and JSON schema guidance sits awkwardly inside a spec that is otherwise data-layer agnostic. The proposal: a separate repository and spec — narrow, just the Parquet schema and recommendations, sitting between two standards — which SQL on FHIR and bulk data could both point at.
One caution: whatever gets published as a recommendation or a sample becomes a de facto standard the moment people implement it, so be careful what goes out.
Is Parquet even good ground for standardisation?
Against: there's no single perfect generalised mapping, and even people who have built a generalised one find themselves reaching for implementation-guide-specific schemas to fit the actual use case.
For: without a defined mapping, Parquet cannot be a bulk export output format at all — if you want it as an output, someone has to write down how extensions and the rest are represented.
The two positions turned out to be about different things — a near-lossless rendering for passing data around, versus tuning the shape once it has landed — and both were accepted.