Amazon S3

Use an S3 feed when the sending system puts files into an Amazon S3 bucket, or into an S3-compatible store such as MinIO.

Setup

Interbox needs read access to the bucket. Grant:

s3:ListBucket on the bucket, limited to the feed's prefix if you like;
s3:GetObject on the prefix;
kms:Decrypt on the key, when the bucket uses SSE-KMS.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-bucket",
      "Condition": { "StringLike": { "s3:prefix": "clinical/*" } }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::example-bucket/clinical/*"
    }
  ]
}

Interbox finds its credentials in one of two ways:

`credentials`	Interbox uses	Default
`"chain"`	the AWS default chain: environment keys, the shared credentials file, IRSA, EKS Pod Identity, an ECS task role, an EC2 instance role	without `endpoint`
`"keys"`	`S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY`, else `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`	with `endpoint`

On AWS, give Interbox a role and let the chain find it: no key is stored anywhere. For an S3-compatible store, set endpoint and the keys.

Interbox refuses credentials: "chain" together with endpoint, so AWS identity tokens never go to a host named in the configuration.

EKS Pod Identity

Install the Pod Identity agent on the cluster:

aws eks create-addon --cluster-name my-cluster --addon-name eks-pod-identity-agent

Create an IAM role that EKS pods can assume, with the policy above attached:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}

Associate the role with the service account Interbox runs as:

aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace interbox \
  --service-account interbox \
  --role-arn arn:aws:iam::111122223333:role/interbox-s3-read

Run Interbox under that service account (serviceAccountName: interbox) and leave credentials unset.

With IRSA instead, annotate the service account with eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/interbox-s3-read; the chain picks it up the same way.

A missing identity or permission fails each poll with an error that says what is missing. Interbox skips no file over it, and reads as soon as the access is fixed.

Configuration

import { env } from "@health-samurai/interbox";
import { aidbox, fhirItems, skipUnchanged } from "@health-samurai/interbox/aidbox";
import { pipeline, postgresStore } from "@health-samurai/interbox/chain";
import type { Bundle, Resource } from "@health-samurai/interbox/fhir/4.0.1";
import { deliveryEnd, fileDropSource, ndjson, records, s3, unpack, type NdjsonRecord } from "@health-samurai/interbox/files";

const toAidbox = aidbox({
  url: env("AIDBOX_URL"),
  auth: { kind: "basic", user: env("AIDBOX_CLIENT_ID", "root"), password: env("AIDBOX_CLIENT_SECRET") },
});

function bundleToResources(row: NdjsonRecord): Resource[] {
  const bundle = row as unknown as Bundle;
  return (bundle.entry ?? []).flatMap((entry) => (entry.resource === undefined ? [] : [entry.resource]));
}

pipeline("clinical")
  .source(
    fileDropSource({
      transport: s3({ bucket: env("BUCKET"), prefix: "clinical" }),
    }),
  )
  .step(unpack())
  .step(ndjson())
  .store(postgresStore({ codec: "json" }))
  .step(deliveryEnd())
  .step(records(bundleToResources))
  .step(fhirItems())
  .step(skipUnchanged())
  .sender(toAidbox);

Field	Required	Default	Sets
`bucket`	yes
`prefix`	yes		the feed; `""` for the whole bucket
`region`		`S3_REGION` or `AWS_REGION` from Interbox's environment, else `us-east-1`; with `endpoint`, the region an `s3.<region>.amazonaws.com` address names, else `auto`	the bucket's region, whatever `credentials` is; a wrong one fails with an error naming the right one
`endpoint`	for an S3-compatible store		the store's URL; unset means Amazon S3
`credentials`		`"chain"`, or `"keys"` with `endpoint`	see Setup
`timeout`		`30s`	the longest Interbox waits for any one answer

To try a feed locally, run an S3-compatible store such as MinIO, point endpoint at it, and set S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY to its keys.

With endpoint unset, Interbox refuses to read when S3_ENDPOINT or AWS_ENDPOINT is set in the environment: the transport would ignore the variable and read from AWS, which is rarely what was meant. Name the endpoint on the transport instead.

Files Interbox cannot read

Archived files. Interbox skips a file in Glacier Flexible Retrieval or Glacier Deep Archive and logs why. Restoring it is not enough, since the listing still shows it archived: to have it read, copy the restored file to a key in another storage class. Glacier Instant Retrieval reads like any other file.
Files encrypted with your own key (SSE-C). Interbox never has that key, so it skips the file. Default encryption (SSE-S3) and KMS keys (SSE-KMS) work.

Network

HTTPS 443 to s3.<region>.amazonaws.com or to endpoint. With "chain", also to the AWS identity endpoint in use: STS or instance metadata. See Network Access for egress rules.

Deliveries, file formats and pipeline settings shared by every storage are on Reading Files from Storage.