U.S. Department of EnergyOffice of Scientific and Technical Information
A View from ORNL: Scientific Data Research Opportunities in the Big Data Age
Abstract
One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of "Big Data". In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].
- Authors:
-
[1];
[1];
[1];
[1];
[1];
[1];
[1];
[1];
[1];
[1];
[1];
[1]
more »
- ORNL
- Kitware
- Georgia Institute of Technology, Atlanta
- Rutgers University
- Publication Date:
- Research Org.:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- OSTI Identifier:
- 1468120
- DOE Contract Number:
- AC05-00OR22725
- Resource Type:
- Conference
- Resource Relation:
- Conference: IEEE 38th International Conference on Distributed Computing Systems (ICDCS) - Vienna, , Austria - 7/2/2018 4:00:00 AM-7/5/2018 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Citation Formats
Klasky, Scott A., Wolf, Matthew D., Ainsworth, Mark, Atkins, Chuck, Choi, Jong Youl, Eisenhauer, Greg, Geveci, Berk, Godoy, William F., Kim, Mark B., Kress, James M., Kurc, Tahsin M., Liu, Qing Gary, Logan, Jeremy S., Maccabe, Arthur Barney, Mehta, Kshitij V., Ostrouchov, George, Parashar, Manish, Podhorszki, Norbert, Pugmire, Dave, Suchyta, Eric D., Wan, Lipeng, and Wang, Ruonan. A View from ORNL: Scientific Data Research Opportunities in the Big Data Age. United States: N. p., 2018.
Web. doi:10.1109/ICDCS.2018.00136.
Klasky, Scott A., Wolf, Matthew D., Ainsworth, Mark, Atkins, Chuck, Choi, Jong Youl, Eisenhauer, Greg, Geveci, Berk, Godoy, William F., Kim, Mark B., Kress, James M., Kurc, Tahsin M., Liu, Qing Gary, Logan, Jeremy S., Maccabe, Arthur Barney, Mehta, Kshitij V., Ostrouchov, George, Parashar, Manish, Podhorszki, Norbert, Pugmire, Dave, Suchyta, Eric D., Wan, Lipeng, & Wang, Ruonan. A View from ORNL: Scientific Data Research Opportunities in the Big Data Age. United States. https://doi.org/10.1109/ICDCS.2018.00136
Klasky, Scott A., Wolf, Matthew D., Ainsworth, Mark, Atkins, Chuck, Choi, Jong Youl, Eisenhauer, Greg, Geveci, Berk, Godoy, William F., Kim, Mark B., Kress, James M., Kurc, Tahsin M., Liu, Qing Gary, Logan, Jeremy S., Maccabe, Arthur Barney, Mehta, Kshitij V., Ostrouchov, George, Parashar, Manish, Podhorszki, Norbert, Pugmire, Dave, Suchyta, Eric D., Wan, Lipeng, and Wang, Ruonan. 2018.
"A View from ORNL: Scientific Data Research Opportunities in the Big Data Age". United States. https://doi.org/10.1109/ICDCS.2018.00136. https://www.osti.gov/servlets/purl/1468120.
@article{osti_1468120,
title = {A View from ORNL: Scientific Data Research Opportunities in the Big Data Age},
author = {Klasky, Scott A. and Wolf, Matthew D. and Ainsworth, Mark and Atkins, Chuck and Choi, Jong Youl and Eisenhauer, Greg and Geveci, Berk and Godoy, William F. and Kim, Mark B. and Kress, James M. and Kurc, Tahsin M. and Liu, Qing Gary and Logan, Jeremy S. and Maccabe, Arthur Barney and Mehta, Kshitij V. and Ostrouchov, George and Parashar, Manish and Podhorszki, Norbert and Pugmire, Dave and Suchyta, Eric D. and Wan, Lipeng and Wang, Ruonan},
abstractNote = {One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of "Big Data". In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].},
doi = {10.1109/ICDCS.2018.00136},
url = {https://www.osti.gov/biblio/1468120},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {7}
}
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.