Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++

Godoy, William; Malviya, Addi Thakur; Hahn, Steven

doi:10.1007/978-981-19-4453-6_14

Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++

Conference · 01 November 2022

DOI:https://doi.org/10.1007/978-981-19-4453-6_14· OSTI ID:1899022

^[1]; Malviya, Addi Thakur ^[1]; ^[1]

ORNL

General-purpose library application programming interfaces (APIs) for self-describing hierarchical scientific data storage, such as the HDF5 and NetCDF libraries, are traditionally of runtime nature. Runtime errors for entry existence and data types are typically caught later in the development process of higher-level application-specific APIs. In this paper, we propose exploiting modern C++ metaprogramming features to add compile-time type-safety to improve the interaction with a well-defined metadata-rich scientific schema in domain-specific hierarchical datasets. We tackle two aspects of common use: (i) direct data access, (ii) flexible “in-memory” index models for efficient search and data processing. The proposed APIs use C++17’s template type auto deduction features, C++11’s enum class for type-safety and C-style preprocessor macros for generative templated code. We showcase the pros and cons of our initial work on the standard NeXus schema used for annotating and storing experimental neutron scattering data at several facilities around the world on top of HDF5. Extendable compile-time type-safe APIs are a desirable feature that could be indexed by any modern integrated development environment (IDE). Hence, such APIs can help ease the learning curve for domain scientists using a less error-prone software interaction to enhance the findability of their data without resorting to a domain-specific language (DSL).

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1899022

Country of Publication:: United States

Language:: English

References (25)

ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management Godoy, William F.; Podhorszki, Norbert; Wang, Ruonan SoftwareX, Vol. 12 https://doi.org/10.1016/j.softx.2020.100561	journal	July 2020
Miqs Zhang, Wei; Byna, Suren; Tang, Houjun Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356146	conference	November 2019
The NeXus data format Könnecke, Mark; Akeroyd, Frederick A.; Bernstein, Herbert J. Journal of Applied Crystallography, Vol. 48, Issue 1 https://doi.org/10.1107/S1600576714027575	journal	January 2015
Domain specific language implementation via compile-time meta-programming Tratt, Laurence ACM Transactions on Programming Languages and Systems, Vol. 30, Issue 6 https://doi.org/10.1145/1391956.1391958	journal	October 2008
Template based HDF5 satellite digital data product generation software Gaur, Neha; Mahammad, Sazid; Sharma, Vivek 2012 International Conference on Data Science & Engineering (ICDSE) https://doi.org/10.1109/ICDSE.2012.6281903	conference	July 2012
Insights for exascale IO APIs from building a petascale IO API Lofstead, Jay; Ross, Robert Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503238	conference	January 2013
The UNIX time-sharing system Ritchie, Dennis M.; Thompson, Ken Communications of the ACM, Vol. 17, Issue 7 https://doi.org/10.1145/361011.361061	journal	July 1974
Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid Godoy, William F.; Peterson, Peter F.; Hahn, Steven E. Communications in Computer and Information Science https://doi.org/10.1007/978-3-030-63393-6_12	book	January 2020
Software Development for Reproducible Research Hinsen, Konrad Computing in Science & Engineering, Vol. 15, Issue 4 https://doi.org/10.1109/MCSE.2013.91	journal	July 2013
NetCDF: an interface for scientific data access Rew, R.; Davis, G. IEEE Computer Graphics and Applications, Vol. 10, Issue 4 https://doi.org/10.1109/38.56302	journal	July 1990
An overview of the HDF5 technology suite and its applications Folk, Mike; Heber, Gerd; Koziol, Quincey Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases - AD '11 https://doi.org/10.1145/1966895.1966900	conference	January 2011
Improving access to multi-dimensional self-describing scientific datasets Nam, B.; Sussman, A. CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings. https://doi.org/10.1109/CCGRID.2003.1199366	conference	January 2003
Abstracting the template instantiation relation in C++ Sutton, Andrew; Holeman, Ryan; Maletic, Jonathan I. 2009 IEEE International Conference on Software Maintenance https://doi.org/10.1109/ICSM.2009.5306392	conference	September 2009
The ANTAREX domain specific language for high performance computing Silvano, Cristina; Agosta, Giovanni; Bartolini, Andrea Microprocessors and Microsystems, Vol. 68 https://doi.org/10.1016/j.micpro.2019.05.005	journal	July 2019
A Brief Introduction on Big Data 5Vs Characteristics and Hadoop Technology No authors listed Procedia Computer Science, Vol. 48 https://doi.org/10.1016/j.procs.2015.04.188	journal	January 2015
Scientific data management in the coming decade Gray, Jim; Liu, David T.; Nieto-Santisteban, Maria ACM SIGMOD Record, Vol. 34, Issue 4 https://doi.org/10.1145/1107499.1107503	journal	December 2005
Armadillo: a template-based C++ library for linear algebra Sanderson, Conrad; Curtin, Ryan The Journal of Open Source Software, Vol. 1, Issue 2 https://doi.org/10.21105/joss.00026	journal	June 2016
ROOT — A C++ framework for petabyte data storage, statistical analysis and visualization Antcheva, I.; Ballintijn, M.; Bellenot, B. Computer Physics Communications, Vol. 180, Issue 12 https://doi.org/10.1016/j.cpc.2009.08.005	journal	December 2009
The FAIR Guiding Principles for scientific data management and stewardship Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.18	journal	March 2016
Exploring Metadata Search Essentials for Scientific Data Management Zhang, Wei; Byna, Suren; Niu, Chenxu 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC) https://doi.org/10.1109/HiPC.2019.00021	conference	December 2019
ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems Byna, Suren; Breitenfeld, M. Scot; Dong, Bin Journal of Computer Science and Technology, Vol. 35, Issue 1 https://doi.org/10.1007/s11390-020-9822-9	journal	January 2020
Guidelines for a Standardized Filesystem Layout for Scientific Data Spreckelsen, Florian; Rüchardt, Baltasar; Lebert, Jan Data, Vol. 5, Issue 2 https://doi.org/10.3390/data5020043	journal	April 2020
Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities Godoy, William F.; Savici, Andrei T.; Hahn, Steven E. 2021 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData52589.2021.9671354	conference	December 2021
Expressing and Applying C++ Code Transformations for the HDF5 API Through a DSL Golasowski, Martin; Bispo, João; Martinovič, Jan Computer Information Systems and Industrial Management https://doi.org/10.1007/978-3-319-59105-6_26	book	January 2017
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL Godoy, William F.; Peterson, Peter F.; Hahn, Steven E. 2020 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData50022.2020.9377836	conference	December 2020

Similar Records

HDF5-FastQuery: An API for Simplifying Access to Data Storage,Retrieval, Indexing and Querying

Technical Report · 2006 · OSTI ID:888964

DARMA v. Beta 0.5

Software · 2017 · OSTI ID:1349227

Automatic Offloading C++ Expression Templates to CUDA Enabled GPUs

Conference · 2012 · OSTI ID:1080421

Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++

Citation Formats

References (25)

Similar Records

Related Subjects