Migrating to Aidbox: How Deep 6 AI enhanced AI pipeline performance for clinical trial recruitment

Executive Summary

Deep 6 AI partnered with Health Samurai to enhance their healthcare data pipeline by implementing the Aidbox FHIR server. This migration reduced data loading times by 50% while improving data quality by 90% through validation against Deep 6's FHIR Implementation Guide.

The cloud-native solution on AWS with Kubernetes provides real-time monitoring and elastic scaling capabilities. Using Aidbox's PostgreSQL engine, automated IG processing, and standardized FHIR integration patterns, Deep 6 AI improved operational efficiency and accelerated partner onboarding.

Company Background

Deep 6 AI is an innovative healthcare technology company that leverages artificial intelligence to accelerate and improve clinical trial recruitment. Their platform uses natural language processing (NLP) and machine learning (ML) to analyze structured and unstructured clinical data from electronic health records (EHRs) to match patients to appropriate clinical trials.

Health Samurai is the company behind Aidbox, a high-performance FHIR server platform designed to meet the demanding requirements of modern healthcare applications.

Challenge

Deep 6 AI faced several challenges with their previous FHIR implementation:

Performance Bottlenecks -- The initial data loading process from healthcare systems was lengthy and tedious, creating delays in onboarding new healthcare partners.
Limited Scalability -- As Deep 6 AI expanded to more healthcare systems and processed larger volumes of data, the previous architecture struggled to scale efficiently.
Monitoring Deficiencies -- The legacy system provided limited visibility into the data processing pipeline, making it difficult to track progress and identify bottlenecks.
FHIR IG versioning -- Frictions on loading new versions of Deep 6 AI custom FHIR IG.

Solution

In collaboration with Health Samurai, Deep 6 AI implemented the Aidbox FHIR server with the following key components:

Aidbox FHIR Server

Replacing the previous implementation with Aidbox provided immediate benefits including:

Improved error handling with full visibility into continuous real-time data ingestion.
Efficient query capabilities for better visibility into the entire client data set.

Deep 6 FHIR Implementation Guide

Aidbox offers a fully-automated process for loading FHIR IGs with all their dependencies. Deep 6 AI developed their own FHIR IG that allows for standardized data validation, ensuring that all incoming data meets specific requirements for AI processing and clinical trial matching.

Kubernetes Cluster Deployment

The solution was deployed on a highly available Amazon EKS cluster, ensuring resilience, scalability, and efficient resource utilization. This architecture allowed Deep 6 AI to dynamically scale Aidbox instances during initial data loading for maximum throughput and then downscale resources afterward to optimize costs.

Real-time Monitoring Dashboard

Aidbox comes with monitoring dashboards based on Prometheus and Grafana, and a transaction log based on Elasticsearch, providing visibility into the data loading process with real-time metrics on progress, errors, and data quality.

Standardized Integration Patterns

Implementation of consistent integration approaches for connecting to healthcare systems included support for direct FHIR APIs and bulk data transfer mechanisms. After data was validated in Aidbox, it was passed downstream through FHIR topic-based subscriptions.

Results

Configuration

Database:

AWS RDS Postgres m7g.2xlarge [8 vCPU 32 GiB]

Aidbox:

48 Aidbox instances deployed to AWS EKS Nodes 3 x m7g.2xlarge [8 vCPU 32 GiB]
Average of 2,400 FHIR resources per second with synchronous validation
6-day load time for a total of 1.2 billion FHIR resources

Key Achievements

Dramatic Performance Improvements -- Initial data loading time reduced by 50%.
Enhanced Data Quality -- 90% reduction in data validation errors with synchronous validation against the custom Deep 6 AI IG.
Operational Efficiency -- Real-time visibility into data processing status and errors.
Scalability Achievements -- Dynamic scaling of the Kubernetes cluster during peak periods.

Deep 6 AI has successfully implemented their platform with Aidbox for three priority customers with the highest requirements for data volumes and throughput. The biggest historical data set loaded to a single Aidbox instance was over one billion FHIR resources (over 1TB of data).

Solution Architecture

Deep 6 AI Implementation

Conclusion

The partnership between Deep 6 AI and Health Samurai demonstrates the significant impact that high-performance FHIR server technology can have on healthcare data processing pipelines. For healthcare technology companies facing similar challenges with data processing performance or quality, this case study offers compelling evidence for considering specialized FHIR server implementations like Aidbox.