Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores

He, Yun; Cook, Brandon; Deslippe, Jack; Friesen, Brian; Gerber, Richard; Hartman-Baker, Rebecca; Koniges, Alice; Kurth, Thorsten; Leak, Stephen; Yang, Woo -Sun; Zhao, Zhengji; Baron, Eddie; Hauschildt, Peter

doi:10.1002/cpe.4291

Title: Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores

Abstract

The newest NERSC supercomputer Cori is a Cray XC40 system consisting of 2,388 Intel Xeon Haswell nodes and 9,688 Intel Xeon-Phi “Knights Landing” (KNL) nodes. Compared to the Xeon-based clusters NERSC users are familiar with, optimal performance on Cori requires consideration of KNL mode settings; process, thread, and memory affinity; fine-grain parallelization; vectorization; and use of the high-bandwidth MCDRAM memory. This paper describes our efforts preparing NERSC users for KNL through the NERSC Exascale Science Application Program, Web documentation, and user training. We discuss how we configured the Cori system for usability and productivity, addressing programming concerns, batch system configurations, and default KNL cluster and memory modes. Here, system usage data, job completion analysis, programming and running jobs issues, and a few successful user stories on KNL are presented.

Authors:

^[1]; Cook, Brandon ^[1]; Deslippe, Jack ^[1];

^[1];

^[1]; Yang, Woo -Sun ^[1]; Zhao, Zhengji ^[1];

^[2]; Hauschildt, Peter ^[3]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Univ. of Oklahoma, Norman, OK (United States)
Hamburger Sternwarte, Hamburg (Germany)

Publication Date:: Fri Aug 25 00:00:00 EDT 2017

Research Org.:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1459400

Grant/Contract Number:: AC02-05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Concurrency and Computation. Practice and Experience

Additional Journal Information:: Journal Volume: 30; Journal Issue: 1; Related Information: Copyright © 2017 John Wiley & Sons, Ltd.; Journal ID: ISSN 1532-0626

Publisher:: Wiley

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; cross compilation; Intel Xeon Phi; KNL; performance optimization; process and thread affinity; user support; training; web documentation; heterogeneous; cluster and memory modes; NESAP

Citation Formats


                    He, Yun, Cook, Brandon, Deslippe, Jack, Friesen, Brian, Gerber, Richard, Hartman-Baker, Rebecca, Koniges, Alice, Kurth, Thorsten, Leak, Stephen, Yang, Woo -Sun, Zhao, Zhengji, Baron, Eddie, and Hauschildt, Peter. Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores.  United States: N. p., 2017. 
Web.  doi:10.1002/cpe.4291.

Copy to clipboard


                    He, Yun, Cook, Brandon, Deslippe, Jack, Friesen, Brian, Gerber, Richard, Hartman-Baker, Rebecca, Koniges, Alice, Kurth, Thorsten, Leak, Stephen, Yang, Woo -Sun, Zhao, Zhengji, Baron, Eddie, & Hauschildt, Peter. Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores.  United States.  https://doi.org/10.1002/cpe.4291

Copy to clipboard


                    He, Yun, Cook, Brandon, Deslippe, Jack, Friesen, Brian, Gerber, Richard, Hartman-Baker, Rebecca, Koniges, Alice, Kurth, Thorsten, Leak, Stephen, Yang, Woo -Sun, Zhao, Zhengji, Baron, Eddie, and Hauschildt, Peter. Fri .  
"Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores".  United States.  https://doi.org/10.1002/cpe.4291.  https://www.osti.gov/servlets/purl/1459400.

Copy to clipboard


                    
@article{osti_1459400,

  title        = {Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores},

  author       = {He, Yun and Cook, Brandon and Deslippe, Jack and Friesen, Brian and Gerber, Richard and Hartman-Baker, Rebecca and Koniges, Alice and Kurth, Thorsten and Leak, Stephen and Yang, Woo -Sun and Zhao, Zhengji and Baron, Eddie and Hauschildt, Peter},

  abstractNote = {The newest NERSC supercomputer Cori is a Cray XC40 system consisting of 2,388 Intel Xeon Haswell nodes and 9,688 Intel Xeon-Phi “Knights Landing” (KNL) nodes. Compared to the Xeon-based clusters NERSC users are familiar with, optimal performance on Cori requires consideration of KNL mode settings; process, thread, and memory affinity; fine-grain parallelization; vectorization; and use of the high-bandwidth MCDRAM memory. This paper describes our efforts preparing NERSC users for KNL through the NERSC Exascale Science Application Program, Web documentation, and user training. We discuss how we configured the Cori system for usability and productivity, addressing programming concerns, batch system configurations, and default KNL cluster and memory modes. Here, system usage data, job completion analysis, programming and running jobs issues, and a few successful user stories on KNL are presented.},

  doi          = {10.1002/cpe.4291},

  journal      = {Concurrency and Computation. Practice and Experience},

  number       = 1,

  volume       = 30,

  place        = {United States},

  year         = {Fri Aug 25 00:00:00 EDT 2017},

  month        = {Fri Aug 25 00:00:00 EDT 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1002/cpe.4291

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 8 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

BerkeleyGW: A massively parallel computer package for the calculation of the quasiparticle and optical properties of materials and nanostructures
journal, June 2012

Deslippe, Jack; Samsonidze, Georgy; Strubbe, David A.
Computer Physics Communications, Vol. 183, Issue 6
DOI: 10.1016/j.cpc.2011.12.006

Evaluating and Optimizing the NERSC Workload on Knights Landing
conference, November 2016

Barnes, Taylor; Cook, Brandon; Deslippe, Jack
2016 7th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)
DOI: 10.1109/PMBS.2016.010

Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set
journal, July 1996

Kresse, G.; Furthmüller, J.
Computational Materials Science, Vol. 6, Issue 1, p. 15-50
DOI: 10.1016/0927-0256(96)00008-0

Roofline: an insightful visual performance model for multicore architectures
journal, April 2009

Williams, Samuel; Waterman, Andrew; Patterson, David
Communications of the ACM, Vol. 52, Issue 4
DOI: 10.1145/1498765.1498785

Works referencing / citing this record:

Harnessing billions of tasks for a scalable portable hydrodynamic simulation of the merger of two stars
journal, September 2018

Heller, Thomas; Lelbach, Bryce Adelstein; Huck, Kevin A.
The International Journal of High Performance Computing Applications, Vol. 33, Issue 4
DOI: 10.1177/1094342018819744

MCtandem: an efficient tool for large-scale peptide identification on many integrated core (MIC) architecture
journal, July 2019

Li, Chuang; Li, Kenli; Li, Keqin
BMC Bioinformatics, Vol. 20, Issue 1
DOI: 10.1186/s12859-019-2980-5

Similar Records in DOE PAGES and OSTI.GOV collections:

Roofline Analysis in the Intel® Advisor to Deliver Optimized Performance for applications on Intel® Xeon Phi™ Processor

Conference Koskela, Tuomas S. ; Lobet, Mathieu ; Deslippe, Jack ; ...

In this session we show, in two case studies, how the roofline feature of Intel Advisor has been utilized to optimize the performance of kernels of the XGC1 and PICSAR codes in preparation for Intel Knights Landing architecture. The impact of the implemented optimizations and the benefits of using the automatic roofline feature of Intel Advisor to study performance of large applications will be presented. This demonstrates an effective optimization strategy that has enabled these science applications to achieve up to 4.6 times speed-up and prepare for future exascale architectures. # Goal/Relevance of Session The roofline model [1,2] is amore »« less
Full Text Available
Understanding the I/O Performance Gap Between Cori KNL and Haswell

Conference Liu, Jialin ; Koziol, Quincey ; Tang, Houjun ; ...

The Cori system at NERSC has two compute partitions with different CPU architectures: a 2,004 node Haswell partition and a 9,688 node KNL partition, which ranked as the 5th most powerful and fastest supercomputer on the November 2016 Top 500 list. The compute partitions share a common storage configuration, and understanding the IO performance gap between them is important, impacting not only to NERSC/LBNL users and other national labs, but also to the relevant hardware vendors and software developers. In this paper, we have analyzed performance of single core and single node IO comprehensively on the Haswell and KNL partitions,more »« less
Full Text Available
Performance and Energy Usage of Workloads on KNL and Haswell Architectures. In: High Performance Computing Systems. Performance Modeling, Benchmarking, and Simulation

Conference Allen, Tyler ; Daley, Christopher S. ; Doerfler, Douglas ; ... - Lecture Notes in Computer Science

Manycore architectures are an energy-efficient step towards exascale computing within a constrained power budget. The Intel Knights Landing (KNL) manycore chip is a specific example of this and has seen early adoption by a number of HPC facilities. It is therefore important to understand the performance and energy usage characteristics of KNL. In this paper, we evaluate the performance and energy efficiency of KNL in contrast to the Xeon (Haswell) architecture for applications representative of the workload of users at NERSC. We consider the optimal MPI/OpenMP configuration of each application and use the results to characterize KNL in contrast tomore »« less
https://doi.org/10.1007/978-3-319-72971-8_12
Evaluating the networking characteristics of the Cray XC-40 Intel Knights Landing-based Cori supercomputer at NERSC

Conference Doerfler, Douglas ; Austin, Brian ; Cook, Brandon ; ... - Concurrency and Computation. Practice and Experience

There are many potential issues associated with deploying the Intel Xeon Phi™ (code named Knights Landing [KNL]) manycore processor in a large-scale supercomputer. One in particular is the ability to fully utilize the high-speed communications network, given that the serial performance of a Xeon Phi TM core is a fraction of a Xeon®core. In this paper, we take a look at the trade-offs associated with allocating enough cores to fully utilize the Aries high-speed network versus cores dedicated to computation, e.g., the trade-off between MPI and OpenMP. In addition, we evaluate new features of Cray MPI in support of KNL,more »« less
https://doi.org/10.1002/cpe.4297
Electronic Structure Theory Calculations Using Modern Architectures: KNL vs Haswell

Journal Article Harville, Taylor ; Gordon, Mark S. - Journal of Chemical Theory and Computation

The time to solution and parallel efficiency of several commonly used electronic structure methods (Hartree–Fock, density functional theory, second order perturbation theory, resolution of the identity second order perturbation theory, coupled cluster) are evaluated on both the Intel Xeon Haswell and the Intel Xeon Phi Knights Landing (KNL) architectures. The Haswell completes the benchmark calculations with a faster time to solution than the KNL for all molecules and methods tested. While the Haswell exhibits an average speedup of at least 3.5 relative to the KNL for all nonthreaded computations, the KNL has a better parallel efficiency than the Haswell withmore »« less
https://doi.org/10.1021/acs.jctc.1c00705

Similar Records

Title: Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores

Abstract

Citation Formats

BerkeleyGW: A massively parallel computer package for the calculation of the quasiparticle and optical properties of materials and nanostructures journal, June 2012

Evaluating and Optimizing the NERSC Workload on Knights Landing conference, November 2016

Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set journal, July 1996

Roofline: an insightful visual performance model for multicore architectures journal, April 2009

Harnessing billions of tasks for a scalable portable hydrodynamic simulation of the merger of two stars journal, September 2018

MCtandem: an efficient tool for large-scale peptide identification on many integrated core (MIC) architecture journal, July 2019

BerkeleyGW: A massively parallel computer package for the calculation of the quasiparticle and optical properties of materials and nanostructures
journal, June 2012

Evaluating and Optimizing the NERSC Workload on Knights Landing
conference, November 2016

Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set
journal, July 1996

Roofline: an insightful visual performance model for multicore architectures
journal, April 2009

Harnessing billions of tasks for a scalable portable hydrodynamic simulation of the merger of two stars
journal, September 2018

MCtandem: an efficient tool for large-scale peptide identification on many integrated core (MIC) architecture
journal, July 2019