Operating Pipelines

What an operator sees and does once pipelines are running: the dashboard's pages for them, the actions it offers on units, the engine instances behind it, and the inputs refused before they were stored. For units, statuses and items themselves, see Units and State. For the same actions from a script, see Admin Scripts.

The dashboard

Pipelines and Instances appear in the sidebar once at least one pipeline has created its tables. From then on Dashboard opens on the 2.0 home, Sources shows the pipelines' sources, and the sidebar no longer lists the 1.x message pages. Reference stays listed: one entry for the HL7v2 and C-CDA references, with a switch between the two in the page header. Code mappings stays listed too, and lists the codes the pipelines' units wait on. Units and Unit open from the Pipelines page, the Dashboard and the Sources page.

A page that refreshes gives up on a read that has no answer within 20 seconds, or within two of its refresh intervals when that is longer, and shows it as a read that failed until a later one succeeds.

Dashboard

The 2.0 home covers every pipeline. Its live rates refresh every second, its counts every 10 seconds, and its top error kinds every minute:

Each tile also draws its last 60 seconds and gives the change from the first half of that minute to the second. On Error rate both are of the units that reached error per second, not of the percent.

• The rates count units as the engine moves them. Each stage of a pipeline counts a unit once the transaction that moved it commits, and the counts of every engine instance are summed:

  • Received: the units ingest created, at received or skipped, and the units a bridge created in the pipeline it feeds;
  • Processed: the units whose steps closed them out of received, to processed, skipped or sent, or on through a bridge. A unit that fails counts as an error instead. Its items/s is the sender items those units queued;
  • Sent: the units that reached sent, whether the sender closed them, they had nothing to send, or a bridge passed them on. Its items/s is the items the sender delivered;
  • errors, the chart's fourth line and Error rate's sparkline: the units that reached error, from a step or from the sender. A unit counts once however many of its items were rejected.

  Nothing is counted for a notReady(), an item answered wait, or a failure that is retried, until the retries run out and the unit reaches error. Nor are an operator's Retry, Defer and item deletion: the API makes those moves, and only the engine counts.

• The rates keep no history. Each engine holds the last few minutes of its counts in memory and writes them every second. Anything that restarts the pipeline workers — an engine restart, a reload after a new commit, or a worker restarted after a crash — starts them from zero, and the chart fills again over its minute. When the rates cannot be read, the three rate tiles read — and say live rates unavailable, and the chart and the sparklines stay empty until the rates can be read again.

• Error rate and the queue depths count every unit, so they cost a full count. Each pipeline's units are counted in full at most once every 15 seconds, so a total can be that old when it is shown. A pipeline's count is given up after 2 seconds and then not tried again for a minute, and all the counts together are given up after half the request's time budget, INTERBOX_API_READ_TIMEOUT_MS. The rest of the page still loads. Both then add up the pipelines that were counted and say how many were not, such as 1 pipeline not counted, and read — only when none was.

• Each pipeline counts its own units. An input a bridge passes on becomes a unit in each pipeline it reaches, so the rates, the error rate and the queue depths count it once per pipeline.

• When Postgres cannot be reached, System health reads Postgres and the engine down and the mapping coverage unknown.

When no pipeline has created its tables, or the pipeline list cannot be read, Dashboard is the 1.x home.

Pipelines

One row per pipeline. Its run status and its graph refresh every 5 seconds, the other columns every 10 seconds:

• The run status comes from the engine instances' heartbeats. A pipeline runs as up to three stages, chain-source:, chain-segment: and chain-sender: followed by its name. A pipeline fed by another through a bridge has no source stage, and one that bridges on into another has no sender stage. Only fresh, licensed instances count — see Engine instances:
  • running: every instance that runs any of the pipeline's stages runs all of them, each running;
  • degraded: an instance has one of its stages starting, in backoff or exited, or runs some of its stages but not all;
  • stopped: no instance runs any of its stages, because the engine is stopped, parked after a failed load, or not licensed.
• The graph comes from the engine's live counters, the ones behind the Dashboard's rates, drawn as one point per 5 seconds. A pipeline fed by a bridge counts as received the units the bridge created in it. Like the Dashboard's rates, the graph keeps no history: anything that restarts the pipeline workers — an engine restart, a reload after a new commit, or a worker restarted after a crash — starts them from zero, and the graph fills again over its 5 minutes.
• When the live status cannot be read, the row shows no status and its graph reads — until it can be read again, rather than the last one drawn.

Units opens the pipeline's units. Sender opens its sender backlog under the table, refreshed every 5 seconds:

Below them the items are grouped by outcome and kind, such as wait · sender/reference_missing, so you can see what a backlog waits for.

Units

The units of one pipeline, newest first, 50 at a time. The newest page refreshes every 4 seconds; loading older pages with Load more pauses that until you go Back to live. When a refresh fails, the list keeps the rows it last read and says what time they are from and why the refresh failed, and the field counts do the same, until a refresh succeeds.

• The field list counts the units by status, error_kind, error_step and each attr the store declares in index. Click a value to filter by it, and negate a filter to exclude it. The counts come from the newest 10,000 matching units, and say so when they reach that.
• Search appears when the store declares search: "trgm". It finds units whose search text contains every word typed; each word needs at least three letters or digits.
• The time window bounds the units by when they were created.
• The filters, the search, the time window, the bounds of a created_at or order_at chip and the similar-error filter are kept in the page's address, so opening that address again opens the list the same way. A chip read back from the address is labelled with its bounds. A link from the Dashboard's top error kinds sets the created_at bounds to the 24 hours its count covers, with the time window at All.
• Click a row to expand it in place:
  • its error, if it has one. An error that carries a FHIR OperationOutcome shows its diagnostics under the request that failed, with the whole message one click away;
  • its fields as chips. Click status, error_kind, error_step or a declared attr to filter by its value. The created_at and order_at chips filter to within 1, 5 or 30 minutes of the unit's own time, or to before or after it;
  • the first 20 lines of its value, loaded when the row opens. JSON is shown indented, and a value over 1 MiB or stored as bytes is shown by its size. When the store declares a search index, clicking a token of an HL7v2 value adds it to the search, or takes it out again, and so does selecting text in it;
  • Code mapping on a unit that failed on an unmapped observation code, as on the Unit page.
• Similar errors. The ~ error_message chip of an expanded unit at error keeps the units at error whose message is at least 50% similar to it, as trigram similarity rates it over the first 512 characters of each message. It applies together with every other filter. It looks only among the newest 5,000 units at error in the time range, and when there are more, the list says that older matches are not listed.
• The row menu offers Retry and Defer when the unit can take them, Open details, Filter to this patient when the store declares patient_id in index, Filter to this sender when it declares sending_facility, and Copy unit id.
• Selecting. Click a row's checkbox to select it. Shift-click another to select every row between the two, or drag over the checkboxes to select, or unselect, every row you pass.
• Retry and Defer act on the selected units. Each appears when at least one selected unit can take it. The rows that moved flash, and the result says how many units moved and lists those left unchanged by reason (see Operator actions).
• Ask assistant appears beside Retry and Defer while the assistant is closed. The selected units go to the assistant as their id, pipeline, status, error_kind and message_type attr, never their value or error message.
• Columns chooses which columns show. Views offers All, Errors and Deferred, and Save keeps the current filters, search, time chips and similar-error filter as a view of your own. Both are kept per pipeline, in your browser only.

Unit

One unit, opened from the Units page or by id, in one inspector: a top bar, the unit's error under it, and tabs.

The top bar names the unit, such as Unit #1042, then gives its status and type, its patient, MRN, sender and creation time, each when it is known, and its actions. Its back and close buttons return to the pipeline's Units page: to the list the unit was opened from, with its filters, search, time window, time chips and similar-error filter, or to the whole list for a unit opened any other way.

• Type is an HL7v2 message's MSH-9 type and trigger event, such as ADT^A01. Any other unit, and an HL7v2 message with an empty MSH-9, takes its message_type attr, joined to its event_type attr with ^ when it has one.
• Patient is PID-5's given and family name, and sender the first component of MSH-4, both read from an HL7v2 value's text. MRN is the first component of PID-3, or else the patient_id attr.
• Code mapping appears on a unit that failed on an unmapped observation code, and opens that code's concept map on the Code mappings page at that code.
• Reprocess runs Retry and Defer runs Defer (see Operator actions), each when the unit's status allows it.

Ask assistant, in the page header, opens the assistant with this unit as its context, shared as on the Units page. Refresh reads the unit again.

A unit that holds an error shows it in a strip under the top bar: at error, and at deferred, which keeps the error it was deferred at. It names the error_kind and error_step, and gives the message. The result of Reprocess, Defer or an item deletion appears in the same strip.

• An Aidbox rejection is shown as its status line and the diagnostics of its issues, each once. raw response opens the whole message as stored.
• An unmapped observation code (code/unmapped_observation_code) names the code, its display and system, and the concept map.
• Any other error is shown as its message.

The tabs, in this order:

The value's own tabs follow what it holds. A value over 1 MiB gets Raw, whatever it holds.

• An HL7v2 message, text that starts MSH|. Raw HL7v2 shows its segments, each field with its spec, its table's meaning and the spec warnings; Parsed shows the parsed value with its TypeScript accessor.
• A C-CDA document, XML whose root element is ClinicalDocument. Document shows its document tree, parsed on read; Raw XML the text.
• JSON, a json value or text holding a JSON object or array. Message shows it pretty-printed.
• Anything else gets Raw: the stored value, a bytes value as the start of its base64, and a value over 1 MiB as its size only.

The FHIR resources are read from the unit's sender items, the waiting ones and a record of the finished ones. A waiting item shows its JSON. Once the sender is done with an item, the record of it shows how it finished:

• sent: a summary, its type, id and when it was sent, and no JSON, as the body of a sent resource is not kept. Open it in Aidbox links to the resource in Aidbox, when its id or its key names it and Aidbox's address is one your browser can reach.
• rejected: its JSON as it was, and the reason as an error, its kind and message. An item is rejected by the sender, and this is the only finished item whose body is kept.
• dropped: a summary and no JSON, and the reason as an error, its kind and message. An item is dropped when its unit was rejected or deferred before it was sent. Its body is not kept: the unit's stored value still holds the input it came from, and Reprocess sends the unit again.
• deleted: a summary and no JSON, and a plain note, not an error, that an operator action deleted it — deleting waiting items, Reprocess or Defer — with the reason recorded, if any.

An item listed as waiting that has finished by the time you pick it is shown as finished.

Finished items are kept for INTERBOX_DONE_ITEMS_RETENTION_DAYS days, 7 by default, and then purged. It takes from one hour, 0.042, to 36500 days, and 0 keeps them for good. A unit at sent or error with no item left says that no record of its resources is kept any more: they were purged after the retention period, or it never had any. When the items cannot be read the tab says so, and when the inspector itself cannot load it asks you to reload the page.

The cards of the Details tab:

On a unit at processed, the Sender items card offers Delete selected and Delete all waiting items.

Sources

A tree of the pipelines' sources, in the order of the Pipelines page, each followed by its senders, refreshed every 10 seconds. A source's row shows its kind — mllp or webhook for a pipeline that listens, bridge for one fed by fromPipeline() — its address or upstream, how many senders its breakdown lists (see below), and its health:

Collapse a source to hide its senders. Selecting a source, or one of its senders, opens the source's detail. With a sender selected, its Status mix and Volume are that sender's own:

The address is the source's configuration with its env() values resolved. A webhook source takes no host and listens on every interface, shown as [::]. No secret is shown. Units opens the pipeline's units.

The sender breakdown appears for a pipeline whose store declares a btree index on sending_facility in index; without one there is none. It lists the 50 senders with the most units created in the last 24 hours, busiest first, and says so when there are more; the sender count in the page header then reads "top". Each sender has the same counts as the source, and is active with a unit in the last minute, recent with one in the last hour, and idle otherwise. Units with no sending_facility count under (not set). Units on a sender opens the pipeline's units filtered to that sender.

A source's volume or sender breakdown that cannot be counted in time is shown as not counted, never as zero: its figures read — and say not counted in time, its tree row reads — snd, and the page header says some senders were not counted. When the indexes a pipeline's store declares cannot be read in time either, its breakdown is shown as not counted too, whether or not it declares one on sending_facility.

Until the volume has loaded, and while its latest read is failing, the detail shows it the same way rather than as zero or as the last figures read: the Status mix and the Volume read —, and Senders says the breakdown has not loaded. A failed read also says not read: the latest read failed, and the page header says volume unavailable.

Simulate ingest sends messages to an MLLP source the way a sender would, and shows the ACK each one got. Paste them, or add or drop files; a line starting MSH| starts a new message, and blank lines are dropped. Up to 100 messages and 4 MiB in all go in one request. They are framed one at a time over one connection, each waiting up to 10 seconds for its ACK. A message answered AE or AR shows the ACK's text. When the connection fails or an ACK does not come, that message shows the reason and the messages after it are not sent.

The connection is made from the API to the address the source's instance reports, never to one given in the request, and only ever over the loopback, so the API must run on the same host as the engine:

A port outside 1 to 65535, 0 included, also answers 400, and so does a webhook or bridge source: a webhook's secret never leaves the engine, and a bridge listens on nothing. A source whose instance does not report it running answers 409; try again once it runs. A refused request sends nothing.

Who may simulate follows the login. With it on, that is a signed-in dashboard session. With INTERBOX_AUTH_ENABLED=false it is anyone who can reach the dashboard, within the Host allowlist and cross-site rules that stay on — see Turning it off. A service token cannot simulate either way. Each API process runs at most four simulations at once, each ending within about a minute of starting; another answers 429 until one ends.

The messages are ingested like any other, so they become real units. Each simulation request that gets past the login and the cross-site checks writes an audit line to the API's log, whether it was answered, refused, aborted or failed. It records the session's short reference (none with the login off), the route, the client's agent and forwardedFor when it sends them, and under simulation the pipeline, how many messages and bytes, the outcome, the HTTP status (499 when the client went away) and how many messages were answered AA, AE, AR or not at all. A refused or failed request adds its reason. The line never holds a message or an ACK's text:

{"at":"2026-10-01T12:00:00.000Z","audit":"source-simulated",
 "route":"POST /api/pipelines/adt-feed/source/test","session":"3f9a1c2e",
 "simulation":{"pipeline":"adt-feed","outcome":"exchanged","status":200,
  "messages":3,"bytes":1536,"acks":{"AA":2,"AE":1,"AR":0,"none":0}}}

Instances

One row per engine instance seen in the last hour, fresh ones first, refreshed every 5 seconds: its id, whether it is fresh or stale, its version, whether it is licensed, when it was last seen and started, its stages by state (starting, running, backoff, exited), the inputs it refused, and its reload error. See Engine instances below for what these mean.

Code mappings

The concept maps under src/terminology/concept-maps/ in the workspace, and the codes the pipelines' units wait on, refreshed every 4 seconds. A unit waits on a code when a step failed it with domainError("code", "unmapped_observation_code", …) and a message laid out as

unmapped code "<code>" (<display>) · system <system> · map <concept map id>

where (<display>) may be left out and (none) stands for no system, such as unmapped code "NA" (SODIUM) · system (none) · map hl7v2-lab-hosp-observation-code-loinc.

• The queue lists each code once per concept map, with its display and how many units wait on it: the units at error of that kind whose message names it, from every pipeline, counted together. A code the map already holds is not listed, nor is a message laid out any other way, nor a unit at deferred. GET /api/unmapped-codes answers the same list, and adds when the first of those units was created and, as pipeline:<name> origins, how many wait in each pipeline.
• Each pipeline is read within 2 seconds, and all of them within the request's budget, INTERBOX_API_READ_TIMEOUT_MS. A pipeline whose units take longer, or that is reached once the budget is spent, is left out of that refresh, and the others are still listed.
• A sender is named only by the map. The page labels a concept map with its title, or else with the part of its id between hl7v2- and the mapping type, so hl7v2-lab-hosp-observation-code-loinc reads Lab Hosp. The step builds that id from the sending application and facility, and the page shows neither on its own.
• A map no file holds yet reads no file yet where the others show how many codes they map. It is listed only because units wait on its codes, and approving one of them creates its file.
• Approve writes the code's target into the concept map's file, and creates the file when the map has none yet, with no useContext: a step finds the map by its id. It then retries, in every pipeline, every unit at error whose message names that map and that code, as Retry does and in runs of 1000, and says how many it re-queued. Each run takes only the units still at error on that code when it starts, so a unit another retry has moved on is not sent again. Nothing has to be reloaded first: a step's ctx.translate that misses reads the map files again when they changed, so a retried unit finds the new mapping.
• When Approve fails after writing the mapping, for example when the units cannot be read within the budget, the mapping stays written and the code leaves the queue, but units not yet retried stay at error. Find them on the pipeline's Units page by error_kind code/unmapped_observation_code and retry them there.
• Approve needs a worktree workspace, the directory the engine bundles in place. A clone workspace is reset by the engine on its next reload, so Approve refuses it with 409: add the element to the concept map in the workspace repository and push it instead, then retry the units once the engine has reloaded.

Operator actions

Three actions change units: Retry, Defer, and deleting waiting items. Each runs in one short transaction and answers at once.

Retry and Defer

A Retry runs every step after the store again, with the workspace's current code. The Aidbox sender does not send again a resource it already delivered unchanged, so retrying a unit that partly reached Aidbox sends only what is missing or changed.

• Up to 1000 units per request. The dashboard sends a larger selection in parts of 1000.
• Leftover items are cleared first. A unit the sender rejected may still have items that another send held at that moment. Both actions delete those before they change the unit. An item a send still holds stays, and the unit is left unchanged as item_in_flight.

Each answers the units it moved, and the units it left unchanged with a reason:

Deleting waiting items

A unit at processed stays there until its last item is delivered. An item that waits counts no error and has no limit, so a unit whose resource waits for a reference that will never arrive stays processed until someone acts. Deleting its waiting items closes it.

• Only a unit at processed. Any other status is refused with unit is not processed and the unit's status.
• All, or the ones you name, up to 1000 item ids.
• Items a send holds right now are kept. A named item already gone, delivered or deleted meanwhile, is missing. The rest are deleted.
• When no item is left, the unit becomes sent.
• A named item of another unit refuses the request, and so does a unit another transaction holds (busy).

The answer lists the items deleted, kept and missing, and the unit's status after it.

They never wait on the engine

The actions do not wait for a lock. A unit the engine is working on is in work, and an operator does not change it underneath; waiting could also deadlock against the engine, and would spend the request's time. So a unit another transaction holds is answered busy at once, and the action can simply be repeated.

Every statement of an action, and of the reads behind the Dashboard's totals and error kinds and the Pipelines, Units and Unit pages, runs within one time budget for the whole request, INTERBOX_API_READ_TIMEOUT_MS. A request that spends it answers 504, and changes nothing.

A unit an action moves enters a new status, so a pipeline that subscribes to received, deferred or sent gets that event, delivered by the engine shortly after the action — see Webhooks.

Retry and Defer are also open to the dashboard assistant's grant; deleting items is not — see Securing Dashboard Access.

Engine instances

Every engine instance writes a heartbeat row of its own every 2 seconds, keyed by its id: INTERBOX_INSTANCE_ID, or the host name when that is unset. The row carries its version, license state, CPU and memory, the state of each of its stages, its pipelines' sources, the inputs it refused, and its reload error.

• Fresh or stale. An instance seen within the last 6 seconds is fresh. A stale one has stopped, crashed, or cannot reach the database. Rows not seen for an hour are deleted, so an instance that is gone for good, such as one that restarted under a new host name, drops off within the hour.
• One id, one instance. Two live engines with the same id write one row, and the dashboard shows them as one. When an instance finds its row written by another live process, it logs one error naming the duplicate INTERBOX_INSTANCE_ID. Give each engine its own.
• Reload error. The instance's current workspace load failure: { at, ref, message }, with credentials in URLs removed. It is set when the workspace fails to load, at boot or on a hot reload, and cleared when a load succeeds. A failed hot reload leaves the previous workspace running. A failed boot leaves the engine parked: the dashboard stays up, no stage runs, and the instance keeps beating with no stages. Either way the engine loads the workspace again when a new commit arrives, on Sync now, or on SIGHUP — see The Workspace.

Refused inputs

An input refused before the store left nothing behind: no unit, no stored value. The sending side was told, and retries on its own. Each refusal is counted, so a feed whose every input is refused can be told apart from one that went quiet.

• Per pipeline, per instance, since its source started. Each instance keeps its counts in memory and writes them with its heartbeat. A restart starts from zero, and the refusals of the last seconds before a crash are lost.
• The Pipelines page sums the fresh instances, with the reason and time of the latest refusal. The Instances page shows each instance's total.
• Only the code is counted. The refusal itself, with its message, is in the engine's log, naming the pipeline.
• Not counted: an input abandoned before its commit by a stop or by an HTTP client that disconnected, and a request refused for its method or its secret.

Environment

2.0 adds these variables: