The Modern Research Data Portal: a design pattern for networked, data-intensive science

Chard, Kyle; Dart, Eli; Foster, Ian; Shifflett, David; Tuecke, Steven; Williams, Jason

doi:10.7717/peerj-cs.144

Title: The Modern Research Data Portal: a design pattern for networked, data-intensive science

Journal Article · Mon Jan 15 00:00:00 EST 2018 · PeerJ. Computer Science

DOI:https://doi.org/10.7717/peerj-cs.144· OSTI ID:1425432

Chard, Kyle ^[1]; Dart, Eli ^[2]; Foster, Ian ^[1]; Shifflett, David ^[1]; Tuecke, Steven ^[1]; Williams, Jason ^[1]

Univ. of Chicago, IL (United States); Argonne National Lab. (ANL), Argonne, IL (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Energy Sciences Network

We describe best practices for providing convenient, high-speed, secure access to large data via research data portals. Here, we capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services; review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities. Sample code at a companion web site,https://docs.globus.org/mrdp, provides application skeletons that readers can adapt to realize their own research data portals.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1425432

Journal Information:: PeerJ. Computer Science, Vol. 4, Issue 1; ISSN 2376-5992

Publisher:: PeerJ Inc.Copyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 11 works

Citation information provided by
Web of Science

References (28)

The iPlant Collaborative: Cyberinfrastructure for Plant Biology Goff, Stephen A.; Vaughn, Matthew; McKay, Sheldon Frontiers in Plant Science, Vol. 2 https://doi.org/10.3389/fpls.2011.00034	journal	January 2011
Cloud Kotta: Enabling secure and scalable data analytics in the cloud Babuji, Yadu N.; Chard, Kyle; Gerow, Aaron 2016 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2016.7840616	conference	December 2016
The conundrum of sharing research data Borgman, Christine L. Journal of the American Society for Information Science and Technology, Vol. 63, Issue 6 https://doi.org/10.1002/asi.22634	journal	April 2012
EUDAT: A New Cross-Disciplinary Data Infrastructure for Science Lecarpentier, Damien; Wittenburg, Peter; Elbers, Willem International Journal of Digital Curation, Vol. 8, Issue 1 https://doi.org/10.2218/ijdc.v8i1.260	journal	June 2013
UDT: UDP-based data transfer for high-speed wide area networks Gu, Yunhong; Grossman, Robert L. Computer Networks, Vol. 51, Issue 7 https://doi.org/10.1016/j.comnet.2006.11.009	journal	May 2007
Cloud computing and SaaS as new computing platforms Cusumano, Michael Communications of the ACM, Vol. 53, Issue 4 https://doi.org/10.1145/1721654.1721667	journal	April 2010
Science gateways today and tomorrow: positive perspectives of nearly 5000 members of the research community: Science Gateways Today and Tomorrow Lawrence, Katherine A.; Zentner, Michael; Wilkins-Diehr, Nancy Concurrency and Computation: Practice and Experience, Vol. 27, Issue 16 https://doi.org/10.1002/cpe.3526	journal	May 2015
Collaboration gets the most out of software Morin, Andrew; Eisenbraun, Ben; Key, Jason eLife, Vol. 2 https://doi.org/10.7554/eLife.01456	journal	September 2013
Efficient and Secure Transfer, Synchronization, and Sharing of Big Data Chard, Kyle; Tuecke, Steven; Foster, Ian IEEE Cloud Computing, Vol. 1, Issue 3 https://doi.org/10.1109/MCC.2014.52	journal	September 2014
Improving throughput and maintaining fairness using parallel TCP Hacker, T. J.; Noble, B. D.; Athey, B. D. IEEE INFOCOM 2004 https://doi.org/10.1109/INFCOM.2004.1354669	conference	January 2004
PhEDEx Data Service Egeland, Ricky; Wildish, Tony; Huang, Chih-Hao Journal of Physics: Conference Series, Vol. 219, Issue 6 https://doi.org/10.1088/1742-6596/219/6/062010	journal	April 2010
FAST TCP: from theory to experiments IEEE Network, Vol. 19, Issue 1 https://doi.org/10.1109/MNET.2005.1383434	journal	January 2005
Globus Nexus: A Platform-as-a-Service provider of research identity, profile, and group management Chard, Kyle; Lidman, Mattias; McCollam, Brendan Future Generation Computer Systems, Vol. 56 https://doi.org/10.1016/j.future.2015.09.006	journal	March 2016
Scalable TCP: improving performance in highspeed wide area networks Kelly, Tom ACM SIGCOMM Computer Communication Review, Vol. 33, Issue 2 https://doi.org/10.1145/956981.956989	journal	April 2003
Data Sharing by Scientists: Practices and Perceptions Tenopir, Carol; Allard, Suzie; Douglass, Kimberly PLoS ONE, Vol. 6, Issue 6 https://doi.org/10.1371/journal.pone.0021101	journal	June 2011
The Science DMZ: a network design pattern for data-intensive science Dart, Eli; Rotman, Lauren; Tierney, Brian Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503245	conference	January 2013
iRODS Primer: Integrated Rule-Oriented Data System Rajasekar, Arcot; Moore, Reagan; Hou, Chien-Yi Synthesis Lectures on Information Concepts, Retrieval, and Services, Vol. 2, Issue 1 https://doi.org/10.2200/S00233ED1V01Y200912ICR012	journal	January 2010
The Dataverse Network®: An Open-Source Application for Sharing, Discovering and Preserving Data Crosas, Mercè D-Lib Magazine, Vol. 17, Issue 1/2 https://doi.org/10.1045/january2011-crosas	journal	January 2011
Data publication with the structural biology data grid supports live analysis Meyer, Peter A.; Socias, Stephanie; Key, Jason Nature Communications, Vol. 7, Issue 1 https://doi.org/10.1038/ncomms10882	journal	March 2016
TeraGrid Science Gateways and Their Impact on Science Wilkins-Diehr, N.; Gannon, D.; Klimeck, G. Computer, Vol. 41, Issue 11 https://doi.org/10.1109/MC.2008.470	journal	November 2008
Apache Airavata: design and directions of a science gateway framework: Design and Directions of a Science Gateway Framework Pierce, Marlon E.; Marru, Suresh; Gunathilake, Lahiru Concurrency and Computation: Practice and Experience, Vol. 27, Issue 16 https://doi.org/10.1002/cpe.3534	journal	May 2015
HUBzero: A Platform for Dissemination and Collaboration in Computational Science and Engineering McLennan, Michael; Kennell, Rick Computing in Science & Engineering, Vol. 12, Issue 2 https://doi.org/10.1109/MCSE.2010.41	journal	March 2010
nanoHUB.org: Advancing Education and Research in Nanotechnology Klimeck, Gerhard; McLennan, Michael; Brophy, Sean P. Computing in Science & Engineering, Vol. 10, Issue 5 https://doi.org/10.1109/MCSE.2008.120	journal	September 2008
Globus Data Publication as a Service: Lowering Barriers to Reproducible Science Chard, Kyle; Pruyne, Jim; Blaiszik, Ben 2015 IEEE 11th International Conference on e-Science (e-Science) https://doi.org/10.1109/escience.2015.68	conference	August 2015
An OAuth service for issuing certificates to science gateways for TeraGrid users Basney, Jim; Gaynor, Jeff Proceedings of the 2011 TeraGrid Conference on Extreme Digital Discovery - TG '11 https://doi.org/10.1145/2016741.2016776	conference	January 2011
Programming the Grid with gLite* Laure, E.; Fisher, S. M.; Frohner, A. Computational Methods in Science and Technology, Vol. 12, Issue 1 https://doi.org/10.12921/cmst.2006.12.01.33-45	journal	January 2006
A reference panel of 64,976 haplotypes for genotype imputation. McCarthy, Shane; Das, Sayantan; Kretzschmar, Warren Springer Science and Business Media LLC https://doi.org/10.17863/cam.25404	journalarticle	January 2016
Cloud Kotta: Enabling Secure and Scalable Data Analytics in the Cloud Babuji, Yadu N.; Chard, Kyle; Gerow, Aaron arXiv https://doi.org/10.48550/arxiv.1610.03108	preprint	January 2016

Cited By (1)

Qresp, a tool for curating, discovering and exploring reproducible scientific papers Govoni, Marco; Munakami, Milson; Tanikanti, Aditya Scientific Data, Vol. 6, Issue 1 https://doi.org/10.1038/sdata.2019.2	journal	January 2019

Similar Records

The Modern Research Data Portal: A Design Pattern for Networked, Data-Intensive Science

Journal Article · Tue Sep 12 00:00:00 EDT 2017 · PeerJ Preprints · OSTI ID:1425432

Chard, Kyle; Dart, Eli; Foster, Ian; +3 more

The Astronomy Commons Platform: A Deployable Cloud-based Analysis Platform for Astronomy

Journal Article · Tue Jul 26 00:00:00 EDT 2022 · The Astronomical Journal · OSTI ID:1425432

Stetzler, Steven; Jurić, Mario; Boone, Kyle; +3 more

Catalysis-Hub.org, an open electronic structure database for surface reactions

Journal Article · Tue May 28 00:00:00 EDT 2019 · Scientific Data · OSTI ID:1425432

Winther, Kirsten T.; Hoffmann, Max J.; Boes, Jacob R.; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Portal
High-speed network
Globus
Science DMZ
Data transfer node

Title: The Modern Research Data Portal: a design pattern for networked, data-intensive science

Citation Formats

References (28)

Cited By (1)

Similar Records

Related Subjects