The Astronomy Commons Platform: A Deployable Cloud-based Analysis Platform for Astronomy

Journal Article · 25 July 2022 · The Astronomical Journal

DOI: https://doi.org/10.3847/1538-3881/ac77fb · OSTI ID:1877932

Stetzler, Steven ; Jurić, Mario ; Boone, Kyle ; Connolly, Andrew ; Slater, Colin T. ; Zečević, Petar

Abstract We present a scalable, cloud-based science platform solution designed to enable next-to-the-data analyses of terabyte-scale astronomical tabular data sets. The presented platform is built on Amazon Web Services (over Kubernetes and S3 abstraction layers), utilizes Apache Spark and the Astronomy eXtensions for Spark for parallel data analysis and manipulation, and provides the familiar JupyterHub web-accessible front end for user access. We outline the architecture of the analysis platform, provide implementation details and rationale for (and against) technology choices, verify scalability through strong and weak scaling tests, and demonstrate usability through an example science analysis of data from the Zwicky Transient Facility’s 1Bn+ light-curve catalog. Furthermore, we show how this system enables an end user to iteratively build analyses (in Python) that transparently scale processing with no need for end-user interaction. The system is designed to be deployable by astronomers with moderate cloud engineering knowledge, or (ideally) IT groups. Over the past 3 yr, it has been utilized to build science platforms for the DiRAC Institute, the ZTF partnership, the LSST Solar System Science Collaboration, and the LSST Interdisciplinary Network for Collaboration and Computing, as well as for numerous short-term events (with over 100 simultaneous users). In a live demo instance, the deployment scripts, source code, and cost calculators are accessible. ^{4 4} http://hub.astronomycommons.org/

• Research Organization: Krell Institute, Ames, IA (United States)
• Sponsoring Organization: National Science Foundation (NSF); USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• Grant/Contract Number: SC0019323
• OSTI ID: 1877932
• Journal Information: The Astronomical Journal, Journal Name: The Astronomical Journal Journal Issue: 2 Vol. 164; ISSN 0004-6256
• Publisher: American Astronomical SocietyCopyright Statement
• Country of Publication: United States
• Language: English