Real-Time Discovery Services over Large, Heterogeneous and Complex Healthcare Datasets Using Schema-Less, Column-Oriented Methods

Conference · Thu Dec 31 23:00:00 EST 2015

OSTI ID:1295155

Begoli, Edmon ^[1]; Dunning, Ted ^[2]; Charlie, Frasure ^[3]

---

1. ORNL
2. MapR Technologies
3. PYA Analytics

We present a service platform for schema-leess exploration of data and discovery of patient-related statistics from healthcare data sets. The architecture of this platform is motivated by the need for fast, schema-less, and flexible approaches to SQL-based exploration and discovery of information embedded in the common, heterogeneously structured healthcare data sets and supporting components (electronic health records, practice management systems, etc.) The motivating use cases described in the paper are clinical trials candidate discovery, and a treatment effectiveness analysis. Following the use cases, we discuss the key features and software architecture of the platform, the underlying core components (Apache Parquet, Drill, the web services server), and the runtime profiles and performance characteristics of the platform. We conclude by showing dramatic speedup with some approaches, and the performance tradeoffs and limitations of others.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1295155
• Country of Publication: United States
• Language: English

Related Subjects

Apache Drill Apache Parquet Column Oriented Stores Data Analysis Data Cyclone Healthcare Schema-less data management Services Platform