Batch resource validation

Overview

Batch validation checks resources already in the database against the active FHIR schemas and an optional set of profiles. Use it when you loaded data with validation off, or when you publish a new profile version and want to know how many existing resources are non-compliant and why.

$batch-validate runs against one resource type. Aidbox hash-partitions it into a fixed number of tasks (number-of-chunks, default 12), each task validating its mod(hash(id), N) slice, and aggregates the results into a compact, offender-indexed form (see How results are stored).

It works two ways, chosen by the Prefer header:

Both paths produce the same result under a task-id and persist it the same way, so you drill into a synchronous run the same as an asynchronous one.

Start a validation

POST /fhir/<type>/$batch-validate with a FHIR Parameters body:

POST /fhir/Observation/$batch-validate
content-type: application/json

resourceType: Parameters
parameter:
  # required: only resources whose meta.lastUpdated >= _since
  - {name: _since, valueInstant: '2025-06-02T00:00:00Z'}
  # optional upper bound (exclusive)
  - {name: _until, valueInstant: '2025-06-09T00:00:00Z'}
  # validate against these profiles (conjunctive, see Profiles)
  - {name: profile, valueCanonical: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'}
  # tuning (optional): number of parallel tasks (default 12)
  - {name: number-of-chunks, valuePositiveInt: 24}
  # validator options (optional): override the box's validation settings for this run
  - {name: disable-terminology-validation, valueBoolean: true}

Validator options

The disable-* and strict-* parameters tune what the validator checks, for this run only. Each is three-state: omit it to keep the box's configured setting (see FHIR Schema Validator), or pass it to override that setting (true/false). Use them to trade completeness for speed on a huge type (e.g. skip the terminology and slicing passes for a structural-only sweep), or to tighten a run beyond the box defaults (e.g. strict-profile-resolution to surface resources whose declared profiles don't resolve, which otherwise read as compliant).

Constraints (FHIRPath invariants) have two controls that compose:

• disable-constraint-validation is the blanket switch: true skips every invariant, false checks every invariant (including any the box normally mutes).
• disable-constraint names specific invariants to skip (repeat it per key).
• When both are given, the blanket wins: true skips everything (the list is moot); with false, only the listed keys are skipped and all others are checked. With the blanket omitted, the listed keys are skipped on top of the box's defaults.

Indexing the hash partition

Each task validates its mod(abs(hashtextextended(id, 0)), N) slice of the type, where N is number-of-chunks. N stays fixed for a run, so the partition predicate is a constant expression that a PostgreSQL expression index can cover. Without such an index, every task scans the whole table to find its slice, so a run with N tasks costs N full scans (the scan cost the number-of-chunks note above describes). With a matching index, each task reads only its slice through the index.

The gain grows as the slice shrinks, so an index matters most for a large type validated with a high number-of-chunks. Build it on the resource's storage table (named after the lowercased resource type) with the same modulus as the number-of-chunks the run passes:

-- Observation, validated in 10000 chunks
CREATE INDEX CONCURRENTLY observation_batch_validate_10000
  ON observation (mod(abs(hashtextextended(id, 0)), 10000));

Then run with the matching number-of-chunks:

POST /fhir/Observation/$batch-validate
resourceType: Parameters
parameter:
  - {name: _since, valueInstant: '1970-01-01T00:00:00Z'}
  - {name: number-of-chunks, valuePositiveInt: 10000}

Synchronous response

A Parameters resource holds the task-id, the headline counts, a link to the offending resources, and one issue per distinct error (each with its own filtered drill-down link).

status: 200

resourceType: Parameters
parameter:
  - {name: task-id,   valueString: '<task-id>'}
  - {name: validated, valueUnsignedInt: 1804646}   # resources validated
  - {name: valid,     valueUnsignedInt: 1317494}    # resources with no issues
  - {name: invalid,   valueUnsignedInt: 487152}     # distinct resources with ≥1 issue
  - {name: bytes,     valueDecimal: 5242880000}     # total bytes of resource JSON processed
  - {name: invalid-resources, valueUrl: '/fhir/$batch-validate/<task-id>/invalid-resources'}
  - name: issue
    part:
      - {name: id,                valueString: '5b4b07e4…'}   # issue-id (for drill-down)
      - {name: invalid-resources, valueUrl: '/fhir/$batch-validate/<task-id>/invalid-resources?_issue=5b4b07e4…'}
      - {name: code,        valueCode: invalid-slice-cardinality}
      - {name: expression,  valueString: category}      # the element
      - {name: profile,     valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'}
      - {name: count,       valueUnsignedInt: 486018}   # distinct offending resources
      - {name: constraint,  valueString: us-core-8}      # for invariants
      - {name: diagnostics, valueString: '…human-readable message…'}

• bytes is the total size of the resource JSON the run processed, a decimal because the total overflows unsignedInt at scale.
• count is the number of distinct offending resources for the issue, derived from the offender index.
• For invariant issues, the constraint part carries the constraint key. The diagnostics part is a generated summary keyed off the issue code (for an invariant, <expression>: constraint <key> is not satisfied). It is not the validator's original message text, which is not stored (see How results are stored).
• Other issue kinds add a type-specific part: slice (the slice name) for slice issues, binding (the value-set URL) for terminology-binding issues, and unknown-profile (the unresolved canonical) for an unresolved profile / meta.profile. A part with no value is omitted.
• Each issue carries its own invalid-resources link, pre-filtered to that issue.
• The summary lists at most 10,000 distinct issues, worst first. If a run produces more, it lists the worst 10,000 and adds issues-total (the true count) and issues-truncated: true, so a truncated list is not mistaken for the whole.
• On failure the response is an OperationOutcome.

Asynchronous response

status: 202 Accepted
Content-Location: /fhir/$batch-validate/<task-id>

The endpoints below are system-level, keyed by task-id alone (no resource type in the path).

Poll the Content-Location:

GET /fhir/$batch-validate/<task-id>

• In progress → 202 Accepted with an X-Progress header (percent of tasks completed, e.g. 45%).
• Complete → 200 with the same Parameters summary as the synchronous response.
• Cancelled → 200, an informational OperationOutcome (issue code informational).
• Failed → 200 with the partial Parameters summary from the tasks that completed, plus a status: failed parameter; if no task completed, a 200 OperationOutcome.
• Unknown task → 404.

Drill into the offending resources

The summary tells you which issues occur and how many resources hit each. To get the offending resources, each linked to the version that was validated and carrying its full OperationOutcome, call invalid-resources:

GET /fhir/$batch-validate/<task-id>/invalid-resources
    ?_issue=<issue-id>&_issue=<issue-id2>&_count=50&_page=1&_fullurl-only=false

The response is a Parameters report rather than a Bundle (see Response format): a total, flat paging links, and one repeated resource parameter per offending resource.

resourceType: Parameters
parameter:
  - {name: total, valueUnsignedInt: 1114}
  # paging: flat links. first/previous appear past page 1; next/last before the last page
  - {name: self, valueUrl: '/fhir/$batch-validate/<task-id>/invalid-resources?_count=50&_page=1'}
  - {name: next, valueUrl: '/fhir/$batch-validate/<task-id>/invalid-resources?_count=50&_page=2'}
  - {name: last, valueUrl: '/fhir/$batch-validate/<task-id>/invalid-resources?_count=50&_page=23'}
  - name: resource
    part:
      - {name: fullUrl, valueUrl: '/Observation/<id>/_history/<version>'}
      - name: resource                                  # omitted when _fullurl-only=true
        resource: {resourceType: Observation, meta: {versionId: '<version>'}, …}
      - name: outcome                                   # omitted when _fullurl-only=true
        resource:
          resourceType: OperationOutcome
          issue:
            - {severity: fatal, code: invalid, expression: [Observation.category],
               diagnostics: '…', details: {coding: [{code: invalid-slice-cardinality}]}}

• Each resource parameter is one distinct offending resource: its versioned fullUrl, the resource body (read from history at the validated version), and its outcome, an OperationOutcome listing every issue that resource has (its full issue set, even when _issue narrows which resources come back).
• _fullurl-only=true drops the resource and outcome parts and keeps the fullUrl.
• Paging is flat: self always; first and previous once past page 1; next and last while before the last page.
• An unknown _issue on a known task returns an empty report (total: 0). An unknown task returns 404.

Cancel

DELETE /fhir/$batch-validate/<task-id>

Responds 202 Accepted. Cancellation removes the run's pending tasks and marks it cancelled (a later poll reports cancelled). Aidbox does not interrupt a task already running, and keeps partial results.

Profiles

Pass profile (as valueCanonical) to validate every resource against those URLs (via meta.profile), say to check the impact of a new profile version on existing data. The profiles must be loaded (an installed IG package, or a StructureDefinition you created).

Filtering by date

_since and _until are FHIR instants that filter on meta.lastUpdated, following the bulk-export _since/_until semantics:

parameter:
  - {name: _since, valueInstant: '2025-06-02T00:00:00Z'}   # lastUpdated in [2025-06-02, 2025-06-09)
  - {name: _until, valueInstant: '2025-06-09T00:00:00Z'}

• _since is an inclusive lower bound (lastUpdated >= _since); _until is an exclusive upper bound (lastUpdated < _until).
• _since is required: every run declares a window, so no call scans a whole (possibly huge) type by accident. To validate everything, pass an epoch _since (1970-01-01T00:00:00Z).
• A narrow window keeps a synchronous run small. Date filters compose with profile.

How results are stored

Aidbox stores results in an aggregated, compact form, so validating 100 GB of non-conformant data does not add 100 GB to the database. The aidbox_batch_validation schema holds:

The aggregation key is profile, resource_type, index-normalized path (identifier[2].system → identifier.system), code, constraint_key, and (where they apply) the slice name, binding value set, and unresolved profile canonical. All occurrences that share these collapse into one issue; the issue's count is the number of offender rows (distinct resources).

Aidbox does not store the invalid resource bodies, their OperationOutcomes, or the validator's original message text. The drill-down re-reads the body from history at the validated version and reconstructs the OperationOutcome from the stored machine fields (code, expression, constraint, and the type-specific parts), synthesizing the diagnostics message from the code.

Both synchronous and asynchronous runs persist these tables under the run's task-id, so you poll and drill into either until you cancel it.

Response format

The invalid-resources response is a Parameters resource rather than a Bundle. A searchset or collection Bundle cannot carry a total, version-specific links, a per-offender OperationOutcome, and the invalid resource bodies together while remaining FHIR-valid: Bundle invariants prohibit a version-specific fullUrl, permit total and entry.response only on certain Bundle types, and require each embedded resource to be valid in its own right, which the invalid bodies are not. A Parameters resource is subject to none of these constraints: it preserves the version-specific drill-down links and embeds each OperationOutcome beside the resource it describes. The invalid resource bodies are the report's content: the data under review.

Terminology

Coded-binding and slice validation may call the configured terminology server. Bindings that resolve locally (local code systems, or a hybrid engine with local content) validate offline and surface invalid codes as ordinary terminology-binding-error issues. But if validation needs the configured terminology server and it is unreachable or errors, the validator cannot complete and the whole run fails: a synchronous call returns 422 (OperationOutcome, "Batch validation failed…"), an asynchronous run reports failed. Point fhir.terminology.service-base-url at a reachable server (a local or hybrid engine works best) so coded validation is accurate and does not fail the run.