Offload base64Binary data to external storage

FHIR resources carry binary payloads in base64Binary elements. Binary.data, DocumentReference.content.attachment.data, and Patient.photo.data are common examples, many other resource types have such elements, and extensions with a valueBase64Binary value can appear on any element. Stored inline, these payloads grow the resource tables and everything built on top of them: history, backups, replication.

Data offload moves the payloads out of the database. On create and update, Aidbox uploads the decoded bytes to external blob storage and stores the resource with a pointer to the blob in place of the data. On read, Aidbox downloads the bytes and returns the resource with the data inlined, so API clients work with the resource as if nothing was offloaded.

Offload is a property of an API. You configure it with the dataOffloadToExternalStorage parameter of $create-api or $configure-api. Azure Blob Storage, AWS S3, and GCP Cloud Storage are the storage providers supported today.

How it works

On create (POST) and update (PUT), Aidbox finds the configured base64Binary elements in the incoming resource. For each element that has a value, Aidbox:

1. Decodes the base64 value and uploads the bytes to the configured storage as a blob named by a random UUID.
2. Removes the element value from the resource.
3. Adds an extension on the element (a primitive extension, in the _data form) that records the blob location and the hash of the data.

The blob holds the raw decoded bytes, not the base64 string. Downloading it with the provider's own tooling yields the original file.

Aidbox commits the blob after the resource write succeeds. If the write fails, Aidbox abandons the upload and no blob appears in the storage.

The stored extension looks like this:

{
  "resourceType": "Binary",
  "id": "b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02",
  "contentType": "application/octet-stream",
  "_data": {
    "extension": [
      {
        "url": "http://health-samurai.io/fhir/core/data-offloaded-to-external-storage",
        "extension": [
          { "url": "hash", "valueString": "Kq5sNclPz7QV2+lfQIuc6R7oRu0=" },
          { "url": "location", "valueString": "azure://my-container/mystorageaccount/4f1f61a2-9e3b-4b0e-bb1d-6a1a1c2f7e58" }
        ]
      }
    ]
  }
}

On instance read (GET /fhir/{resourceType}/{id}), Aidbox downloads the blob, restores the element value, and removes the offload extension from the response. Other extensions on the same element stay in place, both in storage and in responses.

Configuration

Pass the dataOffloadToExternalStorage parameter to $create-api or $configure-api. Its parts:

Element path expressions

fhirpathToBase64BinaryElement accepts a restricted FHIRPath subset: element names separated by dots, with an optional [n] index on array elements. Functions and filters are not supported.

When a path segment names an array and carries no index, the expression matches every item. Missing and empty elements are skipped.

Storage providers

storageProvider selects where the blobs go, and each provider brings its own configuration parts and prerequisites.

Azure Blob Storage

Set storageProvider to azure and reference an AzureContainer resource in the azureContainer part. Blobs land in that container, and the location sub-extension records them as azure://{container-name}/{storage-account}/{blob-name}.

Offload uses the same AzureAccount and AzureContainer resources as the Azure Blob Storage file storage integration.

Create an AzureAccount with one of the supported credential sets:

• key: a storage account access key.
• tenantId, clientId, and clientSecret: Azure AD application credentials.
• No credentials: Aidbox falls back to DefaultAzureCredential, which picks up workload identity in environments configured for it.

For the Azure AD methods, the identity needs a role that grants blob read and write access on the container, such as Storage Blob Data Contributor.

PUT /fhir/AzureAccount/my-account
Content-Type: application/json

{
  "key": "<storage-account-key>"
}

Create an AzureContainer that points to the storage account and container:

PUT /fhir/AzureContainer/my-container
Content-Type: application/json

{
  "account": { "id": "my-account", "resourceType": "AzureAccount" },
  "storage": "mystorageaccount",
  "container": "my-container"
}

AWS S3

Set storageProvider to aws, reference an AwsAccount resource in the awsAccount part, and name the bucket in awsBucket. Objects land in that bucket, and the location sub-extension records them as s3://{bucket}/{object-name}.

Offload uses the same AwsAccount resource as the AWS S3 file storage integration. The region field is required. Pass access-key-id and secret-access-key for explicit credentials, or omit them so Aidbox uses the default credentials provider chain (environment variables, instance profile, or pod identity).

PUT /fhir/AwsAccount/my-aws-account
Content-Type: application/json

{
  "region": "us-east-1",
  "access-key-id": "<access-key-id>",
  "secret-access-key": "<secret-access-key>"
}

The examples below configure Azure. For AWS, the offload parameter carries awsAccount and awsBucket instead of azureContainer:

{
  "name": "dataOffloadToExternalStorage",
  "part": [
    { "name": "fhirpathToBase64BinaryElement", "valueString": "data" },
    { "name": "storageProvider", "valueCode": "aws" },
    { "name": "awsAccount", "valueReference": { "reference": "AwsAccount/my-aws-account" } },
    { "name": "awsBucket", "valueString": "my-bucket" }
  ]
}

GCP Cloud Storage

Set storageProvider to gcp, reference a GcpServiceAccount resource in the gcpServiceAccount part, and name the bucket in gcpBucket. Objects land in that bucket, and the location sub-extension records them as gs://{bucket}/{object-name}.

Offload uses the same GcpServiceAccount resource as the GCP Cloud Storage file storage integration. Pass service-account-email and private-key for explicit credentials, or create the resource without them so Aidbox falls back to Application Default Credentials, which picks up Workload Identity in GKE and Cloud Run.

PUT /fhir/GcpServiceAccount/my-gcp-account
Content-Type: application/json

{
  "service-account-email": "storage-access@my-project.iam.gserviceaccount.com",
  "private-key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
}

The service account needs read and write access to the objects in the bucket, including multipart uploads. roles/storage.objectAdmin on the bucket covers all of them.

With explicit credentials, Aidbox takes the GCP project from service-account-email: the part between @ and the first dot. With Application Default Credentials, the project comes from the runtime environment.

For GCP, the offload parameter carries gcpServiceAccount and gcpBucket instead of azureContainer:

{
  "name": "dataOffloadToExternalStorage",
  "part": [
    { "name": "fhirpathToBase64BinaryElement", "valueString": "data" },
    { "name": "storageProvider", "valueCode": "gcp" },
    { "name": "gcpServiceAccount", "valueReference": { "reference": "GcpServiceAccount/my-gcp-account" } },
    { "name": "gcpBucket", "valueString": "my-bucket" }
  ]
}

Example: offload Binary.data

Create a storage for Binary and connect it to an API with offload enabled. See Storages for the $create-storage step; the example below uses the storageId it returned.

POST /fhir/$create-api
Content-Type: application/json

{
  "resourceType": "Parameters",
  "parameter": [
    { "name": "resourceType", "valueString": "Binary" },
    { "name": "storageId", "valueString": "2791c25a-c28d-47ea-ab96-3e13162a5b58" },
    { "name": "apiTemplate", "valueString": "pre-2604" },
    {
      "name": "dataOffloadToExternalStorage",
      "part": [
        { "name": "fhirpathToBase64BinaryElement", "valueString": "data" },
        { "name": "storageProvider", "valueCode": "azure" },
        { "name": "azureContainer", "valueReference": { "reference": "AzureContainer/my-container" } }
      ]
    }
  ]
}

{
  "resourceType": "Parameters",
  "parameter": [
    { "name": "apiId", "valueString": "39473529-a37e-4b98-afc2-bea014bbe68e" },
    { "name": "apiTemplate", "valueString": "pre-2604" },
    { "name": "resourceType", "valueString": "Binary" },
    { "name": "storageId", "valueString": "2791c25a-c28d-47ea-ab96-3e13162a5b58" }
  ]
}

Create a Binary with raw content. The response returns the resource as stored: data is absent and the offload extension holds the hash and the blob location.

POST /fhir/Binary
Content-Type: application/octet-stream

hello world

{
  "resourceType": "Binary",
  "id": "b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02",
  "contentType": "application/octet-stream",
  "_data": {
    "extension": [
      {
        "url": "http://health-samurai.io/fhir/core/data-offloaded-to-external-storage",
        "extension": [
          { "url": "hash", "valueString": "Kq5sNclPz7QV2+lfQIuc6R7oRu0=" },
          { "url": "location", "valueString": "azure://my-container/mystorageaccount/4f1f61a2-9e3b-4b0e-bb1d-6a1a1c2f7e58" }
        ]
      }
    ]
  }
}

Read the resource. Aidbox fetches the blob and returns the data inlined, without the offload extension:

GET /fhir/Binary/b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02
Accept: application/fhir+json

{
  "resourceType": "Binary",
  "id": "b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02",
  "contentType": "application/octet-stream",
  "data": "aGVsbG8gd29ybGQ="
}

Raw reads work the same way. When the Accept header contains the type stored in Binary.contentType, Aidbox serves the decoded bytes:

GET /fhir/Binary/b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02
Accept: application/octet-stream

hello world

Example: offload Patient.photo.data

Offload works for any resource type and any base64Binary element. This configuration offloads every Patient.photo.data value:

{
  "name": "dataOffloadToExternalStorage",
  "part": [
    { "name": "fhirpathToBase64BinaryElement", "valueString": "photo.data" },
    { "name": "storageProvider", "valueCode": "azure" },
    { "name": "azureContainer", "valueReference": { "reference": "AzureContainer/my-container" } }
  ]
}

Create a Patient with a photo. The response returns each photo's data replaced by the extension:

POST /fhir/Patient
Content-Type: application/json

{
  "resourceType": "Patient",
  "name": [{ "given": ["Amy"] }],
  "photo": [
    {
      "contentType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAA..."
    }
  ]
}

{
  "resourceType": "Patient",
  "id": "5f0c7e2a-8d31-4f5e-9b1a-2c3d4e5f6a7b",
  "name": [{ "given": ["Amy"] }],
  "photo": [
    {
      "contentType": "image/png",
      "_data": {
        "extension": [
          {
            "url": "http://health-samurai.io/fhir/core/data-offloaded-to-external-storage",
            "extension": [
              { "url": "hash", "valueString": "L4pJTPTGwsyBb1TgAAmVGmPKmoc=" },
              { "url": "location", "valueString": "azure://my-container/mystorageaccount/9d2c5a11-7b4f-4e0a-8f26-3c1d9e0b5a44" }
            ]
          }
        ]
      }
    }
  ]
}

GET /fhir/Patient/5f0c7e2a-8d31-4f5e-9b1a-2c3d4e5f6a7b returns the Patient with photo[0].data restored.

Behavior and limitations

• Offload runs on the FHIR REST create (POST) and update (PUT) interactions, and on the entries of transaction bundles. When a transaction fails, its blobs never become visible in the storage. Conditional update and PATCH store the data inline.
• Aidbox restores data on instance read and version read. Search and history responses return the stored form: the element is absent and the extension holds the location.
• When the upload to the external storage fails, the request fails with a 500 OperationOutcome and the resource is not written. When the download fails on read, the request fails with a 500 OperationOutcome as well.
• A configured element whose value is not valid base64 fails the write with 422.
• Aidbox does not delete blobs. Deleting a resource leaves its blob in the storage, and updating a resource uploads a new blob while the old one stays.
• On read, Aidbox resolves the storage from the current API configuration, and the blob name comes from the stored location. Repointing the API at a different container or bucket makes data offloaded through the old one unreadable.

See also

• Storage and API configuration
• Binary resource
• Azure Blob Storage
• AWS S3
• GCP Cloud Storage