FireWorks: a Dynamic Workflow System Designed for High-Throughput Applications

Jain, Anubhav; Ong, Shyue Ping; Chen, Wei; Medasani, Bharat; Qu, Xiaohui; Kocher, Michael; Brafman, Miriam; Petretto, Guido; Rignanese, Gian-Marco; Hautier, Geoffroy; Gunter, Daniel; Persson, Kristin A.

doi:10.1002/cpe.3505

FireWorks: a Dynamic Workflow System Designed for High-Throughput Applications

Journal Article · Mon May 18 00:00:00 EDT 2015 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.3505· OSTI ID:1474899

Jain, Anubhav ^[1]; Ong, Shyue Ping ^[2]; Chen, Wei ^[1]; Medasani, Bharat ^[3]; Qu, Xiaohui ^[1]; Kocher, Michael ^[1]; Brafman, Miriam ^[3]; Petretto, Guido ^[4]; Rignanese, Gian-Marco ^[4]; Hautier, Geoffroy ^[4]; Gunter, Daniel ^[3]; Persson, Kristin A. ^[1]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Environmental Energy and Technologies Division
Univ. of California, San Diego, CA (United States). Dept. of Nanoengineering
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
Catholic Univ. of Louvain, Louvain-la-Neuve (Belgium). Inst. of Condensed Matter and Nanosciences (IMCN) and European Theoretical Spectroscopy Facility (ETSF)

This work introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: European Union (EU); USDOE Office of Energy Efficiency and Renewable Energy (EERE); USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22). Joint Center for Energy Storage Research (JCESR); USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1474899

Alternate ID(s):: OSTI ID: 1401389

Journal Information:: Concurrency and Computation. Practice and Experience, Journal Name: Concurrency and Computation. Practice and Experience Journal Issue: 17 Vol. 27; ISSN 1532-0626

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

References (32)

User tools and languages for graph-based Grid workflows Hoheisel, Andreas Concurrency and Computation: Practice and Experience, Vol. 18, Issue 10 https://doi.org/10.1002/cpe.1002	journal	January 2006
Adaptive exception handling for scientific workflows Tolosana-Calasanz, Rafael; Bañares, José A.; Rana, Omer F. Concurrency and Computation: Practice and Experience https://doi.org/10.1002/cpe.1487	journal	January 2009
Distributed computing in practice: the Condor experience Thain, Douglas; Tannenbaum, Todd; Livny, Miron Concurrency and Computation: Practice and Experience, Vol. 17, Issue 2-4, p. 323-356 https://doi.org/10.1002/cpe.938	journal	January 2005
Scientific workflow management and the Kepler system Ludäscher, Bertram; Altintas, Ilkay; Berkley, Chad Concurrency and Computation: Practice and Experience, Vol. 18, Issue 10 https://doi.org/10.1002/cpe.994	journal	January 2006
Workflow Management in Condor Couvares, Peter; Kosar, Tevfik; Roy, Alain Workflows for e-Science, p. 357-375 https://doi.org/10.1007/978-1-84628-757-2_22	book	January 2007
Python Scripting for Computational Science Langtangen, Hans Petter Texts in Computational Science and Engineering https://doi.org/10.1007/978-3-540-73916-6	book	January 2008
Visual Grid Workflow in Triana Taylor, Ian; Shields, Matthew; Wang, Ian Journal of Grid Computing, Vol. 3, Issue 3-4 https://doi.org/10.1007/s10723-005-9007-3	journal	September 2005
Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis Ong, Shyue Ping; Richards, William Davidson; Jain, Anubhav Computational Materials Science, Vol. 68 https://doi.org/10.1016/j.commatsci.2012.10.028	journal	February 2013
The Materials Application Programming Interface (API): A simple, flexible and efficient API for materials data based on REpresentational State Transfer (REST) principles Ong, Shyue Ping; Cholia, Shreyas; Jain, Anubhav Computational Materials Science, Vol. 97 https://doi.org/10.1016/j.commatsci.2014.10.037	journal	February 2015
ABINIT: First-principles approach to material and nanosystem properties Gonze, X.; Amadon, B.; Anglade, P. -M. Computer Physics Communications, Vol. 180, Issue 12 https://doi.org/10.1016/j.cpc.2009.07.007	journal	December 2009
Swift: A language for distributed parallel scripting Wilde, Michael; Hategan, Mihael; Wozniak, Justin M. Parallel Computing, Vol. 37, Issue 9 https://doi.org/10.1016/j.parco.2011.05.005	journal	September 2011
High-Throughput ab Initio Screening for Two-Dimensional Electride Materials Tada, Tomofumi; Takemoto, Seiji; Matsuishi, Satoru Inorganic Chemistry, Vol. 53, Issue 19 https://doi.org/10.1021/ic501362b	journal	September 2014
Accelerating Electrolyte Discovery for Energy Storage with High-Throughput Screening Cheng, Lei; Assary, Rajeev S.; Qu, Xiaohui The Journal of Physical Chemistry Letters, Vol. 6, Issue 2 https://doi.org/10.1021/jz502319n	journal	January 2015
The rechargeable revolution: A better battery Van Noorden, Richard Nature, Vol. 507, Issue 7490 https://doi.org/10.1038/507026a	journal	March 2014
Charting the complete elastic properties of inorganic crystalline compounds de Jong, Maarten; Chen, Wei; Angsten, Thomas Scientific Data, Vol. 2, Issue 1 https://doi.org/10.1038/sdata.2015.9	journal	March 2015
Advances in methods and algorithms in a modern quantum chemistry program package Shao, Yihan; Molnar, Laszlo Fusti; Jung, Yousung Phys. Chem. Chem. Phys., Vol. 8, Issue 27 https://doi.org/10.1039/B517914A	journal	January 2006
Commentary: The Materials Project: A materials genome approach to accelerating materials innovation Jain, Anubhav; Ong, Shyue Ping; Hautier, Geoffroy APL Materials, Vol. 1, Issue 1 https://doi.org/10.1063/1.4812323	journal	July 2013
Taverna: a tool for the composition and enactment of bioinformatics workflows Oinn, T.; Addis, M.; Ferris, J. Bioinformatics, Vol. 20, Issue 17 https://doi.org/10.1093/bioinformatics/bth361	journal	June 2004
Scientific workflow systems - can one size fit all? Curcin, V.; Ghanem, M. 2008 Cairo International Biomedical Engineering Conference (CIBEC) https://doi.org/10.1109/CIBEC.2008.4786077	conference	December 2008
NEWT: A RESTful service for building High Performance Computing web applications Cholia, Shreyas; Skinner, David; Boverhof, Joshua 2010 Gateway Computing Environments Workshop (GCE) https://doi.org/10.1109/GCE.2010.5676125	conference	November 2010
Python for Scientific Computing Oliphant, Travis E. Computing in Science & Engineering, Vol. 9, Issue 3 https://doi.org/10.1109/MCSE.2007.58	journal	January 2007
Python: An Ecosystem for Scientific Computing Perez, Fernando; Granger, Brian E.; Hunter, John D. Computing in Science & Engineering, Vol. 13, Issue 2 https://doi.org/10.1109/MCSE.2010.119	journal	March 2011
Many-task computing for grids and supercomputers Raicu, Ioan; Foster, Ian T. 2008 Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) https://doi.org/10.1109/MTAGS.2008.4777912	conference	November 2008
Community Accessible Datastore of High-Throughput Calculations: Experiences from the Materials Project Gunter, Dan; Cholia, Shreyas; Jain, Anubhav 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis https://doi.org/10.1109/SC.Companion.2012.150	conference	November 2012
The Trident Scientific Workflow Workbench Barga, Roger; Jackson, Jared; Araujo, Nelson 2008 IEEE Fourth International Conference on eScience (eScience) https://doi.org/10.1109/eScience.2008.126	conference	December 2008
COMPUTER SCIENCE: Beyond the Data Deluge Bell, G.; Hey, T.; Szalay, A. Science, Vol. 323, Issue 5919 https://doi.org/10.1126/science.1170411	journal	March 2009
Weaver: integrating distributed computing abstractions into scientific workflows using Python Bui, Peter; Yu, Li; Thain, Douglas Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851570	conference	January 2010
Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems Deelman, Ewa; Singh, Gurmeet; Su, Mei-Hui Scientific Programming, Vol. 13, Issue 3 https://doi.org/10.1155/2005/128026	journal	January 2005
Workflow Systems for Science: Concepts and Tools Talia, Domenico ISRN Software Engineering, Vol. 2013 https://doi.org/10.1155/2013/404525	journal	January 2013
Globus: a Metacomputing Infrastructure Toolkit Foster, Ian; Kesselman, Carl The International Journal of Supercomputer Applications and High Performance Computing, Vol. 11, Issue 2 https://doi.org/10.1177/109434209701100205	journal	June 1997
Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences Goecks, Jeremy; Nekrutenko, Anton; Taylor, James Genome Biology, Vol. 11, Issue 8 https://doi.org/10.1186/gb-2010-11-8-r86	journal	January 2010
Density functional calculations of the enthalpies of formation of rare-earth orthophosphates Rustad, J. R. American Mineralogist, Vol. 97, Issue 5-6 https://doi.org/10.2138/am.2012.3948	journal	May 2012

Cited By (3)

Convergence and pitfalls of density functional perturbation theory phonons calculations from a high-throughput perspective Petretto, Guido; Gonze, Xavier; Hautier, Geoffroy Computational Materials Science, Vol. 144 https://doi.org/10.1016/j.commatsci.2017.12.040	journal	March 2018
Semantic Interoperability and Characterization of Data Provenance in Computational Molecular Engineering Horsch, Martin Thomas; Niethammer, Christoph; Boccardo, Gianluca Journal of Chemical & Engineering Data, Vol. 65, Issue 3 https://doi.org/10.1021/acs.jced.9b00739	journal	December 2019
BioExcel-2 Deliverable 2.1 – State of the Art and Initial Roadmap Hospital, Adam; Soiland-Reyes, Stian; Gelpí, Josep Lluís Zenodo https://doi.org/10.5281/zenodo.4604606	text	January 2019