skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Choosing the best partition of the output from a large-scale simulation

Abstract

Data partitioning becomes necessary when a large-scale simulation produces more data than can be feasibly stored. The goal is to partition the data, typically so that every element belongs to one and only one partition, and store summary information about the partition, either a representative value plus an estimate of the error or a distribution. Once the partitions are determined and the summary information stored, the raw data is discarded. This process can be performed in-situ; meaning while the simulation is running. When creating the partitions there are many decisions that researchers must make. For instance, how to determine once an adequate number of partitions have been created, how are the partitions created with respect to dividing the data, or how many variables should be considered simultaneously. In addition, decisions must be made for how to summarize the information within each partition. Because of the combinatorial number of possible ways to partition and summarize the data, a method of comparing the different possibilities will help guide researchers into choosing a good partitioning and summarization scheme for their application.

Authors:
 [1];  [1]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Office of Science (SC). Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1396090
Report Number(s):
LA-UR-17-28730
DOE Contract Number:
AC52-06NA25396
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Challacombe, Chelsea Jordan, and Casleton, Emily Michele. Choosing the best partition of the output from a large-scale simulation. United States: N. p., 2017. Web. doi:10.2172/1396090.
Challacombe, Chelsea Jordan, & Casleton, Emily Michele. Choosing the best partition of the output from a large-scale simulation. United States. doi:10.2172/1396090.
Challacombe, Chelsea Jordan, and Casleton, Emily Michele. 2017. "Choosing the best partition of the output from a large-scale simulation". United States. doi:10.2172/1396090. https://www.osti.gov/servlets/purl/1396090.
@article{osti_1396090,
title = {Choosing the best partition of the output from a large-scale simulation},
author = {Challacombe, Chelsea Jordan and Casleton, Emily Michele},
abstractNote = {Data partitioning becomes necessary when a large-scale simulation produces more data than can be feasibly stored. The goal is to partition the data, typically so that every element belongs to one and only one partition, and store summary information about the partition, either a representative value plus an estimate of the error or a distribution. Once the partitions are determined and the summary information stored, the raw data is discarded. This process can be performed in-situ; meaning while the simulation is running. When creating the partitions there are many decisions that researchers must make. For instance, how to determine once an adequate number of partitions have been created, how are the partitions created with respect to dividing the data, or how many variables should be considered simultaneously. In addition, decisions must be made for how to summarize the information within each partition. Because of the combinatorial number of possible ways to partition and summarize the data, a method of comparing the different possibilities will help guide researchers into choosing a good partitioning and summarization scheme for their application.},
doi = {10.2172/1396090},
journal = {},
number = ,
volume = ,
place = {United States},
year = 2017,
month = 9
}

Technical Report:

Save / Share:
  • We present an approach to simulate time-synchronized, one-minute power output from large photovoltaic (PV) generation plants in locations where only hourly irradiance estimates are available from satellite sources. The approach uses one-minute irradiance measurements from ground sensors in a climatically and geographically similar area. Irradiance is translated to power using the Sandia Array Performance Model. Power output is generated for 2007 in southern Nevada are being used for a Solar PV Grid Integration Study to estimate the integration costs associated with various utility-scale PV generation levels. Plant designs considered include both fixed-tilt thin-film, and single-axis-tracked polycrystalline Si systems ranging inmore » size from 5 to 300 MW{sub AC}. Simulated power output profiles at one-minute intervals were generated for five scenarios defined by total PV capacity (149.5 MW, 222 WM, 292 MW, 492 MW, and 892 MW) each comprising as many as 10 geographically separated PV plants.« less
  • Over the course of the past two decades, quantum mechanical calculations have emerged as a key component of modern materials research. However, the solution of the required quantum mechanical equations is a formidable task and this has severely limited the range of materials systems which can be investigated by such accurate, quantum mechanical means. The current state of the art for large-scale quantum simulations is the planewave (PW) method, as implemented in now ubiquitous VASP, ABINIT, and QBox codes, among many others. However, since the PW method uses a global Fourier basis, with strictly uniform resolution at all points inmore » space, and in which every basis function overlaps every other at every point, it suffers from substantial inefficiencies in calculations involving atoms with localized states, such as first-row and transition-metal atoms, and requires substantial nonlocal communications in parallel implementations, placing critical limits on scalability. In recent years, real-space methods such as finite-differences (FD) and finite-elements (FE) have been developed to address these deficiencies by reformulating the required quantum mechanical equations in a strictly local representation. However, while addressing both resolution and parallel-communications problems, such local real-space approaches have been plagued by one key disadvantage relative to planewaves: excessive degrees of freedom (grid points, basis functions) needed to achieve the required accuracies. And so, despite critical limitations, the PW method remains the standard today. In this work, we show for the first time that this key remaining disadvantage of real-space methods can in fact be overcome: by building known atomic physics into the solution process using modern partition-of-unity (PU) techniques in finite element analysis. Indeed, our results show order-of-magnitude reductions in basis size relative to state-of-the-art planewave based methods. The method developed here is completely general, applicable to any crystal symmetry and to both metals and insulators alike. We have developed and implemented a full self-consistent Kohn-Sham method, including both total energies and forces for molecular dynamics, and developed a full MPI parallel implementation for large-scale calculations. We have applied the method to the gamut of physical systems, from simple insulating systems with light atoms to complex d- and f-electron systems, requiring large numbers of atomic-orbital enrichments. In every case, the new PU FE method attained the required accuracies with substantially fewer degrees of freedom, typically by an order of magnitude or more, than the current state-of-the-art PW method. Finally, our initial MPI implementation has shown excellent parallel scaling of the most time-critical parts of the code up to 1728 processors, with clear indications of what will be required to achieve comparable scaling for the rest. Having shown that the key remaining disadvantage of real-space methods can in fact be overcome, the work has attracted significant attention: with sixteen invited talks, both domestic and international, so far; two papers published and another in preparation; and three new university and/or national laboratory collaborations, securing external funding to pursue a number of related research directions. Having demonstrated the proof of principle, work now centers on the necessary extensions and optimizations required to bring the prototype method and code delivered here to production applications.« less
  • Schools in California often have a choice between multiple electricity rate options. For schools with photovoltaic (PV) installations, choosing the right rate is essential to maximize the value of PV generation. The rate option that minimizes a school?s electricity expenses often does not remain the most economical choice after the school installs a PV system. The complex interaction between PV generation, building load, and rate structure makes determining the best rate a challenging task. This report evaluates 22 rate structures across three of California?s largest electric utilities--Pacific Gas and Electric Co. (PG&E), Southern California Edison (SCE), and San Diego Gasmore » and Electric (SDG&E)--in order to identify common rate structure attributes that are favorable to PV installations.« less
  • A rule for selecting the weighting fuhctions in the weighted residual method for obthining approximate solutions to various types of equations and boundary value problems is derived. (D.C.W.)
  • The primary purpose of this paper is to provide managers and group facilitators with a tool for selecting the best technique(s) to help a group solve a problem. The selection tool was created by matching available techniques to problem component examples using a contour map. The secondary purpose is to: (1) describe (by example) three components of a problem that requires a group to solve, and (2) describe available techniques for group problem solving.