The Case for Columnar Analysis (a Two-Part Series) [PowerPoint]

doi:https://doi.org/10.2172/1570209

Title: The Case for Columnar Analysis (a Two-Part Series) [PowerPoint]

Full Record
Other Related Research

Abstract

This talk provides prologue terminology and technology. Part I covers analyzer experience including user experience, code samples, domain of applicability, and scalability. Part II discusses technical underpinnings including theoretical motivation, the Coffea framework, factorized data delivery, package ecosystem, performance, and future directions.

Authors:

Smith, Nick ^[1]; Gray, Lindsey ^[1]; Cremonisi, Matteo ^[1]; Jayatilaka, Bo ^[1]; Gutsche, Oliver ^[1]; Hall, Allison ^[1]; Pedro, Kevin ^[1]; Melo, Andrew ^[2]; Pivarski, Jim ^[3]; Galewsky, Ben ^[4]; Neubauer, Mark ^[5]

Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Vanderbilt Univ., Nashville, TN (United States)
Princeton Univ., NJ (United States)
Univ. of Illinois at Urbana-Champaign, IL (United States). National Center for Supercomputing Applications (NCSA)
Univ. of Illinois at Urbana-Champaign, IL (United States)

Publication Date:: 2019-03-21

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

Contributing Org.:: Coffea Team; Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP)

OSTI Identifier:: 1570209

Report Number(s):: FERMILAB-SLIDES-19-007-T
oai:inspirehep.net:1757703; TRN: US2100578

DOE Contract Number:: AC02-07CH11359

Resource Type:: Technical Report

Resource Relation:: Conference: 2019 Joint HSF/OSG/WLCG Workshop (HOW 2019), Newport News, VA (United States), 18-22 Mar 2019

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS; 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Smith, Nick, Gray, Lindsey, Cremonisi, Matteo, Jayatilaka, Bo, Gutsche, Oliver, Hall, Allison, Pedro, Kevin, Melo, Andrew, Pivarski, Jim, Galewsky, Ben, and Neubauer, Mark. The Case for Columnar Analysis (a Two-Part Series) [PowerPoint].  United States: N. p., 2019. 
        Web.  doi:10.2172/1570209.

Copy to clipboard


                    Smith, Nick, Gray, Lindsey, Cremonisi, Matteo, Jayatilaka, Bo, Gutsche, Oliver, Hall, Allison, Pedro, Kevin, Melo, Andrew, Pivarski, Jim, Galewsky, Ben, & Neubauer, Mark. The Case for Columnar Analysis (a Two-Part Series) [PowerPoint].  United States.  https://doi.org/10.2172/1570209

Copy to clipboard


                    Smith, Nick, Gray, Lindsey, Cremonisi, Matteo, Jayatilaka, Bo, Gutsche, Oliver, Hall, Allison, Pedro, Kevin, Melo, Andrew, Pivarski, Jim, Galewsky, Ben, and Neubauer, Mark. Thu .  
        "The Case for Columnar Analysis (a Two-Part Series) [PowerPoint]".  United States.  https://doi.org/10.2172/1570209.  https://www.osti.gov/servlets/purl/1570209.

Copy to clipboard


                    
@article{osti_1570209,

  title        = {The Case for Columnar Analysis (a Two-Part Series) [PowerPoint]},

  author       = {Smith, Nick and Gray, Lindsey and Cremonisi, Matteo and Jayatilaka, Bo and Gutsche, Oliver and Hall, Allison and Pedro, Kevin and Melo, Andrew and Pivarski, Jim and Galewsky, Ben and Neubauer, Mark},

  abstractNote = {This talk provides prologue terminology and technology. Part I covers analyzer experience including user experience, code samples, domain of applicability, and scalability. Part II discusses technical underpinnings including theoretical motivation, the Coffea framework, factorized data delivery, package ecosystem, performance, and future directions.},

  doi          = {10.2172/1570209},

  url          = {https://www.osti.gov/biblio/1570209},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2019},

  month        = {3}

}

Copy to clipboard

Technical Report:

View Technical Report

https://doi.org/10.2172/1570209

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Coffea -- Columnar Object Framework For Effective Analysis

Conference Smith, Nicholas ; Gray, Lindsey ; Cremonesi, Matteo ; ...

The coffea framework provides a new approach to High-Energy Physics analysis, via columnar operations, that improves time-to-insight, scalability, portability, and reproducibility of analysis. It is implemented with the Python programming language, the scientific python package ecosystem, and commodity big data technologies. To achieve this suite of improvements across many use cases, coffea takes a factorized approach, separating the analysis implementation and data delivery scheme. All analysis operations are implemented using the NumPy or awkward-array packages which are wrapped to yield user code whose purpose is quickly intuited. Various data delivery schemes are wrapped into a common front-end which accepts usermore »« less
Full Text Available
COFFEA - Columnar Object Framework For Effective Analysis

Technical Report Smith, Nick ; Gray, Lindsey ; Cremonesi, Matteo ; ...

The COFFEA Framework provides a new approach to HEP analysis, via columnar operations, that improves time-to-insight, scalability, portability, and reproducibility of analysis. It is implemented with the Python programming language and commodity big data technologies such as Apache Spark and NoSQL databases. To achieve this suite of improvements across many use cases, COFFEA takes a factorized approach, separating the analysis implementation and data delivery scheme. All analysis operations are implemented using the NumPy or awkward-array packages which are wrapped to yield user code whose purpose is quickly intuited. Various data delivery schemes are wrapped into a common front-end which acceptsmore »« less
https://doi.org/10.5281zenodo.3598789

Full Text Available
2020 Exascale Computing Project Annual Meeting (Executive Summary Report)

Technical Report Garcia, Marta ; Munson, Todd

The Exascale Computing Project (ECP) delivers specific applications, software products, and outcomes on DOE computing facilities. Integration across these elements for specific hardware technologies for exascale system instantiations is fundamental to ECP success. The outcome of the ECP is the delivery of a capable exascale computing ecosystem to provide breakthrough solutions addressing our most critical challenges in scientific discovery, energy assurance, economic competitiveness, and national security. This outcome is not a matter of ensuring more powerful computing systems. The ECP is designed to create more valuable and rapid insights from a wide variety of applications (“capable”), which requires a muchmore »« less
https://doi.org/10.2172/1649172

Full Text Available
SDN for End-to-end Networked Science at the Exascale (SENSE) - Final Technical Report

Technical Report Yang, Xi

The Software-defined network for End-to-end Networked Science at Exascale (SENSE) research developed intelligent network services to accelerate scientific discovery in the era of big data driven by Exascale, cloud computing, machine learning and artificial intelligence. The SENSE system allows National Labs and Universities to request and provision end-to-end intelligent network services for their application workflows leveraging Software Defined Network (SDN) capabilities. The SENSE system includes a model-based architecture, implementation, and deployment which enables end-to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements. Intelligent orchestrator and software defined systems allow formore »« less
https://doi.org/10.2172/1577127

Full Text Available
National Geothermal Data System: Transforming the Discovery, Access, and Analytics of Data for Geothermal Exploration

Conference Patten, Kim

Compendium of Papers from the 38th Workshop on Geothermal Reservoir Engineering Stanford University, Stanford, California February 11-13, 2013 The National Geothermal Data System (NGDS) is a distributed, interoperable network of data collected from state geological surveys across all fifty states and the nation’s leading academic geothermal centers. The system serves as a platform for sharing consistent, reliable, geothermal-relevant technical data with users of all types, while supplying tools relevant for their work. As aggregated data supports new scientific findings, this content-rich linked data ultimately broadens the pool of knowledge available to promote discovery and development of commercial-scale geothermal energy production.more »« less
Full Text Available

Similar Records