$import and /fhir/$import

$import is an implementation of the upcoming FHIR Bulk Import API. This is an asynchronous Operation, which returns url to monitor progress. There are two versions of this operation - /fhir/$import accepts data in FHIR format, /$import works with Aidbox format.

Resource requirements for all import operations:

Operation	id	resourceType
`/$import`	Required	Not required
`/fhir/$import`	Required	Not required

Keep in mind that $import does not validate inserted resources for the sake of performance. Pay attention to the structure of data you insert and use the correct URL for your data format, i.e.: use /fhir prefix for FHIR data.

Please consider using Batch validation API to validate data after $import

Example

POST /fhir/$import
Accept: text/yaml
Content-Type: text/yaml

id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz

HTTP 200 OK

The response body is empty. A 200 status code indicates the import has started successfully.

Parameters

Parameter	Description
`id`	Identifier of the import
`contentEncoding`	Supports `gzip` or `plain` (non-gzipped .ndjson files)
`inputs`	Resources to import
`update`	Update history for updated resources (false by default)

You can monitor progress by using id you provided in request body.

GET /BulkImportStatus/synthea

Status

Body

time:
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'

Status

Body

time:
  end: '2023-05-15T14:45:33.820465+02:00'
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - ts: '2023-05-15T14:45:33.819425+02:00'
    url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    total: 3460
    status: finished
    duration: 530
    resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'

Status

Body

time:
  end: '2023-05-15T14:45:33.820465+02:00'
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - ts: '2023-05-15T14:45:33.819425+02:00'
    url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    error: '403: Forbidden'
    status: failed
    resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'

If you didn't provide id in request body, you can use content-location in response header.

Result

Parameter	Type	Description
`id`	string	Identifier of the import
`resourceType`	string	Type of resource where the progress of import operation is recorded. Possible value: `BulkImportStatus`
`meta`	object
`meta.createdAt`	string	Timestamp string at which the resource was created
`meta.lastUpdated`	string	Timestamp string at which the resource was updated last time
`meta.versionId`	string	Version id of this resource
`contentEncoding`	string	`gzip` or `plain`
`time`	object
`time.start`	string	Timestamp string at which the operation started in ISO format
`time.end`	string	Timestamp string at which the operation was completed in ISO format. Only present after the entire import operation has been completed
`type`	string	Data format type to be loaded. Possible values: `aidbox`, `fhir`
`inputs`	object[]
`inputs[].url`	string	URL from which load resources
`inputs[].resourceType`	string	Resource type to be loaded
`inputs[].status`	string	Load status for each input. Only present after the operation for this input has been completed. Possible values: `finished`, `failed`
`inputs[].total`	integer	The number of loaded resources. Only present after the operation for this input has been completed successfully
`inputs[].ts`	string	Timestamp string at which the loading was completed in ISO format. Only present after the operation for this input has been completed
`inputs[].duration`	integer	Duration of loading in milliseconds. Only present after the operation for this input has been completed successfully
`status`	string	Load status for all inputs. Only present after the entire import operation has been completed. After completed, this value is always `finished`, regardless of whether each input is `finished` or `failed`. Possible value: `finished`

Note

For performance reasons $import does raw upsert into the resource table without history update. If you want to store the previous version of resources in history, please set update = true

With this flag, Aidbox will update the history for updated resources. For each resource:

if the resource was not present in DB before the import, the import time will be the same.
if the resource was present in DB before and it's updated during the import, it will double the time importing this resource because of the additional insert operation into the _history table.

/v2/$import

Improved version of the $import operation with enhanced reliability and performance. This version continues work after restarts, handles errors correctly, accepts multiple requests executing them from a queue, and simultaneously processes multiple items from the "inputs" field (with a default of two items processed simultaneously).

Changes in the new $import API:

Executing more than one import with the same id is not possible. Users can omit the `id` field from the request, allowing Aidbox to generate the ID.
The status of the workflow can be accessed with a GET request to /v2/$import/<id> instead of /BulkImportStatus/<id>. The URL for the import status is returned in the content-location header of the $import request.

This feature is not available in Multibox

To start import make a POST request to /v2[/fhir]/$import:

POST /v2/fhir/$import
Accept: text/yaml
Content-Type: text/yaml

id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz

Status

200 OK

Headers

Content-Location:  /v2/$import/synthea

Parameters

Parameter	Description
`id`	Identifier of the import. If you don't provide this, the id will be auto-generated. You can check it on `Content-Location` header in the response
`contentEncoding`	Supports `gzip` or `plain` (non-gzipped .ndjson files)
`inputs` (required)	Resources to import `url` - URL from which load resources `resourceType` - Resource type to be loaded
`update`	Update history for updated resources (false by default)
`allowedRetryCount`	Set the maximum number of import retries for each input (2 by default)

To check the status of the import make a GET request to /v2/$import/<id>:

GET /v2/$import/<id>

Status

200 OK

Body

type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: in-progress
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: waiting
  - url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: waiting
contentEncoding: gzip
status: in-progress

Status

200 OK

Body

type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: done
    outcome: succeeded
    result:
      imported-resources: 0
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: done
    outcome: succeeded
    result:
      imported-resources: 3460
  - url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: done
    outcome: succeeded
    result:
      imported-resources: 124
contentEncoding: gzip
status: done
outcome: succeeded
result:
  message: All input files imported, 3584 new resources loaded
  total-files: 3
  total-imported-resources: 3584

Status

200 OK

Body

type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: done
    outcome: succeeded
    result:
      imported-resources: 225
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: done
    outcome: failed
    error:
      message: '403: Forbidden'
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: done
    outcome: failed
    error:
      message: '403: Forbidden'
contentEncoding: gzip
status: done
outcome: failed
error:
  message: >-
    Import for some files failed with an error: task 'Encounter
    https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    failed

Authenticated sources

$import fetches each inputs[].url with a plain, unauthenticated HTTP GET. There is no way to attach an Authorization header, bearer token, or other credential to the request — the url field is the only thing Aidbox sends.

This means the URL must be either:

publicly/anonymously readable, or
a pre-signed URL that carries its credentials in the query string, e.g. an Azure SAS token or an S3/GCS presigned URL.

If the source returns a non-2xx response (401, 403, etc.), Aidbox reports it as a generic Can't open connection for uri error rather than surfacing the actual status code. If you see this error, check that the URL is public or properly pre-signed before assuming a network or DNS issue.

For Azure Blob Storage, generate a SAS URL for each blob (see Azure Blob Storage) and use that signed URL — with the sv/sig query parameters included — as the url value. A bearer token obtained separately (e.g. via az CLI) cannot be used here, since there is no header to put it in.

Import local file

Sometimes you want to import local file into local Aidbox. Possible solutions for local development:

Add volume to the `aidboxone` container (not `aidboxdb`):

volumes:
- ./Encounter.ndjson.gz:/resources/Encounter.ndjson.gz
# url: file:///resources/Encounter.ndjson.gz

Use tunneling e.g. ngrok:

python3 -m http.server 
ngrok http 8000
# url: https://<...>.ngrok-free.app/Encounter.ndjson.gz

$import and /fhir/$import

Resource requirements for all import operations:

Example

Parameters

Result

Note

/v2/$import

Parameters

Authenticated sources

Import local file

Add volume to the aidboxone container (not aidboxdb):

Use tunneling e.g. ngrok:

Add volume to the `aidboxone` container (not `aidboxdb`):