--- { "title": "$batch-validate: Validate Every Stored Resource Against Its Profiles, at Scale", "description": "Aidbox 2607 replaces the old batch-validation API with the $batch-validate operation — validate a whole resource type already in your database, sync or async, and drill into exactly which resources are non-compliant and why.", "date": "2026-07-13", "author": "Andrew Listopadov", "reading-time": "9 min read", "tags": ["Aidbox", "FHIR Profiling", "Compliance", "Database"], "utm-campaign": "feature", "utm-content": "batch-validate", "tldr": "$batch-validate checks every resource of a type already stored in Aidbox against its FHIR schema and any profiles you name, synchronously or asynchronously. Results are aggregated into a compact, issue-indexed form, so validating 100 GB of non-conformant data doesn't add 100 GB to your database. Available from Aidbox 2607.", "seo-tags": ["FHIR validation", "batch validation", "FHIR profile validation", "Aidbox", "FHIR conformance", "healthcare data quality"] } --- A FHIR server accumulates complex, deeply structured medical records from every system that feeds it, and it's easy to assume they're all correct. Trusting the data is one thing; being able to check it — robustly, at scale — is another. With millions of resources, doing manual validation of each is tedious, and probably unreasonable. And you rarely validate just once, because profiles are not static. Implementation guides like US Core and the HL7 Da Vinci guides publish new versions regularly, each one adding elements, tightening cardinalities, or changing the value sets they bind to. Every time you adopt a new profile version — or publish one of your own — the same concern returns: how much of what you already hold still conforms. So how do you check all of it, repeatedly, without it becoming a project? FHIR's `$validate` operation, in theory, could be automated, but you'll have to fetch each resource and POST it back, collecting and filtering results. This approach has a lot of problems, and is hard to scale. Ideally, you'd want to ask the server to validate a specific resource type, and tell you what's wrong in one call, with proper ability to scale both horizontally and vertically. That's precisely what `$batch-validate` does. It ships in Aidbox 2607, and it replaces the previous batch-validation API entirely. ## Why we rebuilt batch validation Aidbox has had asynchronous batch validation for years, exposed through a set of RPCs (`aidbox.validation/batch-validation` and friends). It worked, but it had several problems. For one, **every validation error was stored as its own `BatchValidationError` resource.** It was also quite slow, making validation of lots of resources an unnecessarily long task. Finally, validating a large dataset could produce so many results that it could rival the data itself in size. A hundred gigabytes of non-conformant resources could produce something close to a hundred gigabytes of error resources. The mechanism you reached for to *understand* a data-quality problem made your storage problem worse. It was also async-only, RPC-shaped rather than a FHIR operation, and it handed you a pile of error resources to query rather than an answer. `$batch-validate` keeps the good part — validate what's already stored, in parallel — and fixes the rest. | | Old batch validation | `$batch-validate` | |----------------|-----------------------------------------------|---------------------------------------------------------------------| | Interface | Proprietary RPCs | FHIR operation (`Parameters` in and out) | | Modes | Asynchronous only | Synchronous **or** asynchronous | | Result storage | One `BatchValidationError` resource per error | One row per **distinct** issue plus a tiny table of resource IDs | | Storage cost | Grows with the error count | Bounded — resource bodies are never copied | | Output | A pile of error resources to query | Worst-first issue summary with on-demand drill-down | | Scaling | Fixed | Hash-partitioned chunks, streamed, parallel across nodes, indexable | The old `aidbox.validation/*` RPCs and the `BatchValidationRun` / `BatchValidationError` resources no longer exist. This is a breaking change; if you used them, migrate to the operation below. ## How to use `$batch-validate` runs against a single resource type. The only required parameter is `_since`, a lower bound on `meta.lastUpdated`. It forces every run to declare a window instead of scanning the whole dataset by accident — to validate everything, pass the epoch. So to validate every Observation updated in April 2026, we can call `$batch-validate` like so: ```yaml POST /fhir/Observation/$batch-validate Content-Type: application/json resourceType: Parameters parameter: - {name: _since, valueInstant: "2026-04-01T00:00:00Z"} - {name: _until, valueInstant: "2026-05-01T00:00:00Z"} ``` By default the call is **synchronous**: it blocks and returns a `Parameters` summary with the headline counts and one entry per distinct issue, worst first. ```yaml resourceType: Parameters parameter: - {name: task-id, valueString: "b1f9..."} - {name: validated, valueUnsignedInt: 1804646} # resources checked - {name: valid, valueUnsignedInt: 1317494} # no issues - {name: invalid, valueUnsignedInt: 487152} # total invalid resources - {name: invalid-resources, valueUrl: "/fhir/$batch-validate/b1f9.../invalid-resources"} - name: issue part: - {name: code, valueCode: invalid-slice-cardinality} - {name: expression, valueString: category} - {name: profile, valueString: "http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab"} - {name: count, valueUnsignedInt: 486018} # resources hitting this exact issue - {name: diagnostics, valueString: "Observation.category: element count is outside the allowed range"} ``` `count` is the number of distinct resources that hit that exact issue — the fastest way to see whether a problem touches six resources or six hundred thousand. Synchronous calls are a good fit for a small set of resources, where you want the answer right away. The work still runs in parallel: `number-of-chunks` (set per call) splits the resources into that many chunks, and the [`scheduler-executors`](https://www.health-samurai.io/docs/aidbox/reference/all-settings#scheduler-executors) setting controls how many run at once on the node. Together they trade chunk granularity against how hard one machine works — that's your vertical scaling knob. However, when you want to validate a substantially larger dataset, you may want to do it in an asynchronous way. ## Async for big datasets To make any `$batch-validate` call asynchronous, all you need is to add a `Prefer: respond-async` header to the call. Aidbox then schedules the work on its task engine, spreading it across nodes, allowing for both horizontal and vertical scaling. ```yaml POST /fhir/Observation/$batch-validate Prefer: respond-async resourceType: Parameters parameter: - {name: _since, valueInstant: "1970-01-01T00:00:00Z"} # basically will validate every Observation in the database ``` You get a `202` with a `Content-Location` header, which you can poll to see the progress: ```http GET /fhir/$batch-validate/b1f9... ``` While it runs you get `202` with an `X-Progress: 45%` header. When it finishes, you get the same `Parameters` summary a synchronous call returns. Both synchronous and asynchronous calls persist their results under a `task-id`, so there's no difference in how you analyze results. ## Explore invalid resources The summary tells you which issues exist and how many resources each one hits. To see the actual resources, follow the `invalid-resources` link — filter it to a single issue with `_issue`, and page with `_count` / `_page`: ```http GET /fhir/$batch-validate/b1f9.../invalid-resources?_count=50&_page=1 ``` Each invalid resource comes back with a **version-specific** `fullUrl` pointing at the exact version that was validated, the resource body, and an `OperationOutcome` listing every issue that resource has: ```yaml - name: resource part: - {name: fullUrl, valueUrl: "/Observation/obs-42/_history/7"} - name: resource resource: {resourceType: Observation} - name: outcome resource: resourceType: OperationOutcome issue: - {severity: fatal, code: invalid, expression: [Observation.category], diagnostics: "..."} ``` The outcome lists a resource's **full** issue set, even when `_issue` narrows which resources come back — so you never fix one problem only to discover a second one on the next pass. ## Test a profile before you enforce it The most common reason to run this is a new profile: you want to know what breaks before you make it a requirement. Pass one or more `profile` canonicals, and every resource is validated against them on top of its base schema. ```yaml resourceType: Parameters parameter: - {name: _since, valueInstant: "1970-01-01T00:00:00Z"} - {name: profile, valueCanonical: "http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab"} ``` Multiple profiles are conjunctive (AND): a resource is compliant only if it conforms to every one, and the issues are the union across them. So you can turn US Core on knowing exactly what would fail, instead of finding out in production. ## Tune the validator per run Sometimes you want a fast structural sweep and don't care about terminology yet; sometimes you want to be stricter than the box defaults. Each `disable-*` / `strict-*` parameter overrides one validator setting for that run only — omit it to keep the box's configured behavior. | Parameter | Effect | |----------------------------------|----------------------------------------------------------------| | `disable-terminology-validation` | Skip coded-binding / terminology checks | | `disable-primitive-validation` | Skip primitive type and format checks | | `disable-slicing-validation` | Skip slice validation | | `disable-constraint-validation` | Skip **all** FHIRPath invariants (or check all when `false`) | | `disable-constraint` | Skip specific invariants by key (e.g. `us-core-8`) | | `strict-profile-resolution` | Treat an unresolved profile as an error instead of skipping it | | `strict-extension-resolution` | Treat an unresolved extension as an error | `strict-profile-resolution` is worth calling out. Without it, a profile URL that doesn't resolve is skipped, so a typo turns into a "compliant" report that is quietly wrong. Turn it on when you want the run to fail loudly instead. ## Built to scale Under the hood, `$batch-validate` hash-partitions the resources into a fixed number of chunks (`number-of-chunks`, default 12). Each chunk validates its `mod(hash(id), N)` slice, and the chunks aggregate into one result: ```mermaid flowchart LR A["POST /fhir/Observation/$batch-validate"] --> B{Hash-partition by id} B --> C[chunk 0] B --> D[chunk 1] B --> E[chunk ...] C --> F[(Aggregated results)] D --> F E --> F ``` Each chunk is streamed, so heap stays bounded no matter how large the dataset is. A synchronous run executes its chunks on a dedicated worker pool — a fixed thread pool Aidbox spins up for that run and tears down when it finishes, sized by the [`scheduler-executors`](https://www.health-samurai.io/docs/aidbox/reference/all-settings#scheduler-executors) setting and separate from the threads that serve your regular API traffic. At most that many chunks run at once, so there's no upper limit on `number-of-chunks` — a larger count simply queues instead of growing the heap. An async run doesn't use this local pool; instead it runs on the task engine's own pool, writing one scheduler row per chunk for any node to claim. Since it neither borrows request threads nor connection-pool slots, it's safe against a live box; CPU is what you spend, so favor async for a big first sweep. The async path is also what lets a run scale across machines. Each chunk is a job on a scheduler that lives in the shared Postgres, so you can run several Aidbox instances on different machines against one database and they split the chunks between them — a run gets faster as you add nodes. A chunk is claimed by exactly one instance, so it never runs twice; and if an instance dies mid-run, the scheduler reclaims its chunk and retries it on another, with idempotent writes so a reclaimed chunk never double-counts. ```mermaid flowchart LR A["POST + Prefer: respond-async"] --> Q[(Shared Postgres chunk queue)] Q -->|claim| N1[Aidbox node 1] Q -->|claim| N2[Aidbox node 2] Q -->|claim| N3[Aidbox node ...] N1 --> R[(Aggregated results)] N2 --> R N3 --> R ``` Because `N` is fixed for a run, the partition predicate is a constant expression you can index. For a very large dataset validated with a high chunk count, a matching expression index turns each chunk from a full scan into a selective index scan: ```sql CREATE INDEX CONCURRENTLY observation_batch_validate_10000 ON observation (mod(abs(hashtextextended(id, 0)), 10000)); ``` Then run with `number-of-chunks: 10000`. The index modulus must match the chunk count, or PostgreSQL won't use it — confirm with `EXPLAIN`. ## Compact by design Here's why validating 100 GB of bad data no longer costs you another 100 GB. Aidbox stores results in an aggregated, issue-indexed form in a dedicated `aidbox_batch_validation` schema: | Table | Holds | |--------------------|--------------------------------------------------------------------| | `issue` | one row per **distinct** error | | `invalid_resource` | a tiny `(issue, resource_id, version)` row per resource — ids only | | `chunk_stat` | one row per chunk with its metrics | All occurrences that share a profile, resource type, normalized path, code, and constraint collapse into a single issue whose count is the number of distinct resources that hit it. Aidbox never copies the invalid resource bodies or their `OperationOutcome`s: the drill-down re-reads each body from history at the validated version and reconstructs the outcome from the stored fields. Distinct problems, not raw volume, drive the storage cost. ## Try it `$batch-validate` is available in Aidbox 2607. Point it at a resource type, pass an epoch `_since`, and you get a worst-first map of your data-quality problems in one call — then drill into the exact resources behind each one. Want to see it work without writing anything? We published an interactive notebook that runs the whole flow end to end: it loads a set of sample Patients, creates a profile modeled on US Core Patient, validates them in one call, and charts the results — the valid/invalid split, the invalid Patients by issue, and where the problems concentrate. Open the **Batch validation** notebook from the Notebooks section in Aidbox and run it top to bottom. Full reference: [Batch resource validation](https://www.health-samurai.io/docs/aidbox/modules/profiling-and-validation/batch-resource-validation).