Scientific data analysis on data-parallel platforms.

Technical Report · Wed Sep 01 04:00:00 EDT 2010

DOI:https://doi.org/10.2172/1011199· OSTI ID:1011199

Ulmer, Craig D; Bayer, Gregory W; Choe, Yung Ryn; Roe, Diana C

As scientific computing users migrate to petaflop platforms that promise to generate multi-terabyte datasets, there is a growing need in the community to be able to embed sophisticated analysis algorithms in the computing platforms' storage systems. Data Warehouse Appliances (DWAs) are attractive for this work, due to their ability to store and process massive datasets efficiently. While DWAs have been utilized effectively in data-mining and informatics applications, they remain largely unproven in scientific workloads. In this paper we present our experiences in adapting two mesh analysis algorithms to function on five different DWA architectures: two Netezza database appliances, an XtremeData dbX database, a LexisNexis DAS, and multiple Hadoop MapReduce clusters. The main contribution of this work is insight into the differences between these DWAs from a user's perspective. In addition, we present performance measurements for ten DWA systems to help understand the impact of different architectural trade-offs in these systems.

• Research Organization: Sandia National Laboratories
• Sponsoring Organization: USDOE
• DOE Contract Number: AC04-94AL85000
• OSTI ID: 1011199
• Report Number(s): SAND2010-7471
• Country of Publication: United States
• Language: English

Related Subjects

99 GENERAL AND MISCELLANEOUS
ALGORITHMS
APPLIANCES
DATA ANALYSIS
PERFORMANCE
STORAGE