Silver bullet DIY: Aidbox and MatrixCare EHR integration

Abstract

As of yet, there is no silver bullet for establishing a working ETL pipeline for transmitting an HL7v2 message feed from EHR into FHIR data out of the box. MatrixCare EHR is not an exclusion as soon as there are customization points. In this article, we are going to give examples of how we managed to ease the pain of integration and which instruments we used (spoiler: Aidbox). We will also describe the core principles of the HL7v2-to-FHIR system configuration, for the benefit of anyone who may wish to build their own system from scratch.

FHIR data is a perfect informational model for: 

In order to give you a brief overview of the whole system, please take a look at this diagram which shows the full ETL pipeline for MatrixCare EHR by Health Samurai:

MatrixCare & Aidbox Integration pipeline

Common architecture

In this post, we will be discussing MatrixCare EHR, though attentive readers may notice that the described experience is absolutely applicable to any other EHR because almost every system that is built in order to execute an effective ETL pipeline operates on the same principles.

So, consuming data from somewhere else starts with the SOURCE of that data!

Data source

MatrixCare HL7v2 Feed

Our data source is MatrixCare EHR. It produces RAW HL7v2 messages. If you forgot or did not know what raw HL7v2 messages look like, here is a brief example. It is quite an old data format and is actually just a string divided into segments (and subsegments), and each of them stores useful medical information.

Strings are not the most convenient datatype, so they are usually parsed into human and machine-readable data structures (JSON, YAML, or EDN). There are open-source parsers for HL7v2 messages (e.g. https://github.com/HL7/v2-to-fhir, https://github.com/LinuxForHealth/hl7v2-fhir-converter), though Aidbox provides a built-in feature for parsing messages and persisting the result (e.g. https://docs.aidbox.app/modules-1/hl7-v2-integration).

Our source produces a huge number of HL7v2 messages like this:

HL7v2 Message Example

MatrixCare EHR sends an ADT (Admit, Discharge, Transfer) HL7v2 message. It uses the 2.5 version of the HL7 standard. A message consists of the following segments: 

Queue

MatrixCare & Amazon SQS Integration

It is hard to imagine a complex system without using data structures that provide disciplined access. Medical systems use queues for the following purposes: 

In our example, we use Amazon SQS (though we may configure our Application Adapter to work with almost any queue service). Amazon SQS has some interesting features. As well as parallel access, a deduplication mechanism, long polling and beautiful visualization it has one feature that we like the most. This is its DLQ or dead letters queue. If something unexpected happens during message consumption, SQS will try to deliver a message to the consumer, but after several attempts, it will be stored in the DLQ, from where it may be consumed manually or programmatically (spoiler: - yeah, our Lilly Adapter application has an API for reprocessing DLQ messages – POST “/$sqs-dlq-reprocess”).

Where the magic starts!

Amazon SQS & DLQ & Aidbox Integration

After the extraction of the HL7v2 message, we need to perform a transformation to FHIR. We actually have a video guide for this process: 

As we mentioned earlier, there are various HL7v2 open-source parsers, but Aidbox provides built-in features to cover the whole ETL pipeline

That is how it exactly happens: 

HL7v2 to Aidbox

HL7v2 statuses within Aidbox


outcome = all_mappings_from_hl7v2config(parsed_field_data)

FHIR mapping

A little bit more on mappings

Mappings serve several functions: 

Mappings are written in a special DSL developed by the Health Samurai team – the JUTE mapping language. It is a lightweight and well-documented DSL with almost no learning curve and is designed in such a way that you do not even have to be a programmer to write your mappings. Nevertheless, you will sometimes face sophisticated cases that require sophisticated mappings.  

Link: https://github.com/HealthSamurai/jute.clj

You might ask how mapping applications result in persisting data. We will try to explain this to you:

Aidbox Mapping Resource example / JUTE

Mappings should contain a ‘request’ field which is actually an Aidbox API call, which will be executed as a result of a mapping application. As a consequence of the mapping application here, resources will be created and persisted in Aidbox Database.

To sum up

Here are the steps you need to take to establish integration with MatrixCare: 

 

You might actually reinvent the wheel, but Aidbox is already configured to make all the magic happen inside it. We persist, parse messages, translate them to the FHIR format and then persist the result. As you may have noticed, the adapter application is lightweight and you certainly can develop your own version of it, but we do not think this is necessary. 

Mappings (used to transform data from parsed HL7v2 to FHIR resources) are a point of customization. Each EHR will probably need a unique set of mappings, but we also have some pre-made mappings for MatrixCare EHR that might be handy. We can provide any support at any point of the integration process.

---

Author: A. Alexeev

Silver bullet DIY: Aidbox and MatrixCare EHR integration

Abstract

As of yet, there is no silver bullet for establishing a working ETL pipeline for transmitting an HL7v2 message feed from EHR into FHIR data out of the box. MatrixCare EHR is not an exclusion as soon as there are customization points. In this article, we are going to give examples of how we managed to ease the pain of integration and which instruments we used (spoiler: Aidbox). We will also describe the core principles of the HL7v2-to-FHIR system configuration, for the benefit of anyone who may wish to build their own system from scratch.

FHIR data is a perfect informational model for: 

  • data aggregation for further analytics;
  • creating an FHIR App store for health and providing data for smart apps;
  • evolutionary enhancement of the current application or finding custom solutions for special tasks.

In order to give you a brief overview of the whole system, please take a look at this diagram which shows the full ETL pipeline for MatrixCare EHR by Health Samurai:

MatrixCare & Aidbox Integration pipeline

Common architecture

In this post, we will be discussing MatrixCare EHR, though attentive readers may notice that the described experience is absolutely applicable to any other EHR because almost every system that is built in order to execute an effective ETL pipeline operates on the same principles.

So, consuming data from somewhere else starts with the SOURCE of that data!

Data source

MatrixCare HL7v2 Feed

Our data source is MatrixCare EHR. It produces RAW HL7v2 messages. If you forgot or did not know what raw HL7v2 messages look like, here is a brief example. It is quite an old data format and is actually just a string divided into segments (and subsegments), and each of them stores useful medical information.

Strings are not the most convenient datatype, so they are usually parsed into human and machine-readable data structures (JSON, YAML, or EDN). There are open-source parsers for HL7v2 messages (e.g. https://github.com/HL7/v2-to-fhir, https://github.com/LinuxForHealth/hl7v2-fhir-converter), though Aidbox provides a built-in feature for parsing messages and persisting the result (e.g. https://docs.aidbox.app/modules-1/hl7-v2-integration).

Our source produces a huge number of HL7v2 messages like this:

HL7v2 Message Example

MatrixCare EHR sends an ADT (Admit, Discharge, Transfer) HL7v2 message. It uses the 2.5 version of the HL7 standard. A message consists of the following segments: 

Queue

MatrixCare & Amazon SQS Integration

It is hard to imagine a complex system without using data structures that provide disciplined access. Medical systems use queues for the following purposes: 

  • delayed computation - no one expects that all systems will handle all incoming data immediately, we just need to know that it will happen at a predictable time in the nearest future;
  • scalability issues - we may add more consumers in order to process data faster;
  • peak loads - we do not expect that data flow will always be linear.

In our example, we use Amazon SQS (though we may configure our Application Adapter to work with almost any queue service). Amazon SQS has some interesting features. As well as parallel access, a deduplication mechanism, long polling and beautiful visualization it has one feature that we like the most. This is its DLQ or dead letters queue. If something unexpected happens during message consumption, SQS will try to deliver a message to the consumer, but after several attempts, it will be stored in the DLQ, from where it may be consumed manually or programmatically (spoiler: - yeah, our Lilly Adapter application has an API for reprocessing DLQ messages – POST “/$sqs-dlq-reprocess”).

Where the magic starts!

Amazon SQS & DLQ & Aidbox Integration

After the extraction of the HL7v2 message, we need to perform a transformation to FHIR. We actually have a video guide for this process: 

As we mentioned earlier, there are various HL7v2 open-source parsers, but Aidbox provides built-in features to cover the whole ETL pipeline

That is how it exactly happens: 

  • Lilly Adapter provides a listener (or consumer) that constantly asks SQS for new messages and, if there are any, initiates HL7v2 message processing,
  • new incoming HL7v2 messages are saved in the HL7v2Message resource: src: "HL7V2MESSAGE" status: received  
  • Aidbox provides a job that is waiting for new messages with ‘received’ status,

HL7v2 to Aidbox

  • after the message is seen by the Aidbox system, the message is parsed by Aidbox, the raw HL7v2 message becomes a human-readable data structure that may be returned as YAML or JSON, and is stored under the ‘parsed’ key in the same HL7v2Message resource:

HL7v2 statuses within Aidbox

  • the config field should also be supported – config defines which set of mappings (you can read about them at https://docs.aidbox.app/tools/mappings) will be applied. Config is stored in the HL7v2Config resource, mappings are stored as a Mapping resource;  
  • so parsed data is input data for mappings and the result of the mapping application is stored in the outcome field:

outcome = all_mappings_from_hl7v2config(parsed_field_data)
  • depending on whether both the parsing and resource creation process succeed, the message status is changed from ‘received’ to ‘processed’. And if it fails, it is assigned an ‘error’ status. The Lilly Adapter application has an API for message reprocessing. It may reprocess messages by different criteria as well as status, type of event, time period, etc. ‘Processed’ status means that all required FHIR resources were successfully created:

FHIR mapping

A little bit more on mappings

Mappings serve several functions: 

Mappings are written in a special DSL developed by the Health Samurai team – the JUTE mapping language. It is a lightweight and well-documented DSL with almost no learning curve and is designed in such a way that you do not even have to be a programmer to write your mappings. Nevertheless, you will sometimes face sophisticated cases that require sophisticated mappings.  

Link: https://github.com/HealthSamurai/jute.clj

You might ask how mapping applications result in persisting data. We will try to explain this to you:

Aidbox Mapping Resource example / JUTE

Mappings should contain a ‘request’ field which is actually an Aidbox API call, which will be executed as a result of a mapping application. As a consequence of the mapping application here, resources will be created and persisted in Aidbox Database.

To sum up

Here are the steps you need to take to establish integration with MatrixCare: 

  • provide an entity that fetches messages from MatrixCare (it can be almost any entity including Kafka or RabbitMQ, but we used AmazonSQS here);
  • find a way to consume that message (from a queue or directly from EHR);
  • parse the HL7v2 message from string to data structure;
  • transform parsed HL7v2 into FHIR resources;
  • save the result.

 

You might actually reinvent the wheel, but Aidbox is already configured to make all the magic happen inside it. We persist, parse messages, translate them to the FHIR format and then persist the result. As you may have noticed, the adapter application is lightweight and you certainly can develop your own version of it, but we do not think this is necessary. 

Mappings (used to transform data from parsed HL7v2 to FHIR resources) are a point of customization. Each EHR will probably need a unique set of mappings, but we also have some pre-made mappings for MatrixCare EHR that might be handy. We can provide any support at any point of the integration process.

---

Author: A. Alexeev

Silver bullet DIY: Aidbox and MatrixCare EHR integration

ReAD MORE

How to load 1 billion FHIR resources into Aidbox in 5 hours

ReAD MORE

Standardized API for EHRs | Cheat Sheet - §170.315(g)(10)

ReAD MORE

Bringing data from wearables and medical IoT devices to FHIR solutions

ReAD MORE

Your path to compliance with the 21st Century Cures Act

ReAD MORE

Aidbox HIPAA book. Part 1. Technical safeguards

ReAD MORE

Why do you need to enable 2FA in your healthcare application?

ReAD MORE

The 2020 X-mas Hackathon

ReAD MORE

A quick guide to telemedicine software alternatives

ReAD MORE

About FHIR facades (part I) - two approaches

ReAD MORE

Two-phase FHIR terminology

ReAD MORE

First Fhirbase Release

ReAD MORE

FHIR Storage and Analytics in Baltimore

ReAD MORE

SQL on FHIR

ReAD MORE

Should you use FHIR resources as your data storage format?

ReAD MORE

The FHIR Guide for CTOs and technical leaders

ReAD MORE

FHIR and Machine Learning (ML)

ReAD MORE

Announcing the FHIRbase Dojo!

ReAD MORE

FHIR Starter 2018: Recap of the first FHIR conference in Eastern Europe

ReAD MORE

Using FHIR to Simplify Healthcare Application Development

ReAD MORE

FHIR: What's great, what isn't so good, and what it's not built to do

ReAD MORE

Moving Cardiovascular Disease Detection to the Cloud

ReAD MORE

FHIR Success Story: Narus Health Connects Patients, Families and Providers

ReAD MORE

Thoughts on the Duke University FHIR Applications Roundtable

ReAD MORE

Choosing Access Control Model for a Generic FHIR Server

ReAD MORE

Implementing FHIR in Dynamic Languages

ReAD MORE

Choosing FHIR for Laboratory Integration

ReAD MORE

Thoughts About Microservices

ReAD MORE

Meaningful Use Stage 3: ONC-certified API for your health care application

ReAD MORE

Transforming Healthcare IT

ReAD MORE

Start a Health IT project with the FHIR standard at your organization

ReAD MORE

How do we build the best Convenient Healthcare Clinics?

ReAD MORE

Accelerating Healthcare Innovation with HL7 FHIR

ReAD MORE

Contact us

Let us know if you want to discuss it further.

By submitting the form you agree to Privacy Policy and Cookie Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
3415 S Sepulveda Blvd Ste 1000 Los Angeles, CA 90034
+1 (818) 731-1279
hello@health-samurai.io