A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Begoli, Edmon; Goethert, Ian; Knight, Kathryn

doi:10.1109/BigData52589.2021.9671534

A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Conference · 30 November 2021

DOI:https://doi.org/10.1109/BigData52589.2021.9671534· OSTI ID:1865733

^[1]; ^[1]; Knight, Kathryn ^[1]

ORNL

Data Lakehouse is a new paradigm in data architectures that embodies and integrates already established concepts for the systematic management of disparate, large-scale data – a data lake for heterogeneous data management, use of open standards for high-performance querying, and systematic maintenance of the data "freshness". In addition to being a new concept, the data lakehouse is also still a conceptual construct. Many projects that use the lakehouse require maturing, empirical studies, and specific implementations. In this paper, we present our implementation of the data lakehouse concept in a biomedical research and health data analytics domain, and we discuss the implementation of some unique and novel features such as support for specialized access controls in support of HIPAA regulation and IRB protocols, and support for the FAIR standard.

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1865733

Country of Publication:: United States

Language:: English

References (12)

Million Veteran Program: A mega-biobank to study genetic influences on health and disease Gaziano, John Michael; Concato, John; Brophy, Mary Journal of Clinical Epidemiology, Vol. 70 https://doi.org/10.1016/j.jclinepi.2015.09.016	journal	February 2016
Hadoop, MapReduce and HDFS: A Developers Perspective Ghazi, Mohd Rehan; Gangodkar, Durgaprasad Procedia Computer Science, Vol. 48 https://doi.org/10.1016/j.procs.2015.04.108	journal	January 2015
Delta lake Armbrust, Michael; Das, Tathagata; Sun, Liwen Proceedings of the VLDB Endowment, Vol. 13, Issue 12 https://doi.org/10.14778/3415478.3415560	journal	August 2020
HIPAA Regulations — A New Era of Medical-Record Privacy? Annas, George J. New England Journal of Medicine, Vol. 348, Issue 15 https://doi.org/10.1056/NEJMlim035027	journal	April 2003
Apache Parquet Vohra, Deepak Practical Hadoop Ecosystem https://doi.org/10.1007/978-1-4842-2199-0_8	book	January 2016
The variant call format and VCFtools Danecek, P.; Auton, A.; Abecasis, G. Bioinformatics, Vol. 27, Issue 15 https://doi.org/10.1093/bioinformatics/btr330	journal	June 2011
The FAIR Guiding Principles for scientific data management and stewardship Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.18	journal	March 2016
Information system security compliance to FISMA standard: a quantitative measure Hulitt, Elaine; Vaughn, Rayford B. Telecommunication Systems, Vol. 45, Issue 2-3 https://doi.org/10.1007/s11235-009-9248-8	journal	December 2009
Promoting an open research culture Nosek, B. A.; Alter, G.; Banks, G. C. Science, Vol. 348, Issue 6242 https://doi.org/10.1126/science.aab2374	journal	June 2015
Apache Spark: a unified engine for big data processing Zaharia, Matei; Franklin, Michael J.; Ghodsi, Ali Communications of the ACM, Vol. 59, Issue 11 https://doi.org/10.1145/2934664	journal	October 2016
Risks and Wrongs in Social Science Research Oakes, J. Michael Evaluation Review, Vol. 26, Issue 5 https://doi.org/10.1177/019384102236520	journal	October 2002
National population-based biobanks for genetic research Swede, Helen; Stone, Carol L.; Norwood, Alyssa R. Genetics in Medicine, Vol. 9, Issue 3 https://doi.org/10.1097/GIM.0b013e3180330039	journal	March 2007

Similar Records

Knowledge Beacons: Web services for data harvesting of distributed biomedical knowledge

Journal Article · 2021 · PLoS ONE · OSTI ID:1826904

Standardized Architecture for a Mega-Biobank Phenomic Library: The Million Veteran Program (MVP)

Conference · 2020 · OSTI ID:1649606

Standardized Architecture for a Mega-Biobank Phenomic Library: The Million Veteran Program (MVP)

Conference · 2020 · OSTI ID:1884042

A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Citation Formats

References (12)

Similar Records

Related Subjects