Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

The Photometric LSST Astronomical Time-series Classification Challenge PLAsTiCC: Selection of a Performance Metric for Classification Probabilities Balancing Diverse Science Goals

Journal Article · · The Astronomical Journal (Online)
 [1];  [2];  [3];  [4];  [5];  [6];  [7];  [8];  [6];  [9];  [10];  [11];  [12];  [13];  [14];  [3];  [15];  [16];  [17];  [4] more »;  [18];  [5] « less
  1. German Centre of Cosmological Lensing, Ruhr-Universität Bochum, Universitätsstraße 150, Bochum, Germany; New York Univ., New York, NY (United States). Center for Cosmology and Particle Physics; New York Univ., New York, NY (United States). Dept. of Physics; OSTI
  2. Univ. of Toronto, Toronto, ON, Canada. Dept. of Astronomy and Astrophysics; Univ. of Toronto, Toronto, ON, Canada. Dunlap Institute for Astronomy and Astrophysics
  3. University College London, Dorking, United Kingdom. Mullard Space Science Lab, Dept. of Space and Climate Physics
  4. Univ. of Toronto, Toronto, ON, Canada. Dunlap Institute for Astronomy and Astrophysics
  5. Stockholm University, AlbaNova, Stockholm, Sweden. The Oskar Klein Centre for Cosmoparticle Physics, Dept. of Physics
  6. Rutgers, the State University of New Jersey, Piscataway, NJ (United States)
  7. Univ. of Pittsburgh, Pittsburgh, PA (United States); Departamento de Física Teórica y del Cosmos, Universidad de Granada, Granada, Spain
  8. Université Clermont Auvergne, CNRS/IN2P3, LPC, F-63000 Clermont-Ferrand, France
  9. Univ. of California, Santa Cruz, Santa Cruz, CA (United States)
  10. KICP, Chicago, IL (United States)
  11. African Institute for Mathematical Sciences, 6 Melrose Road, Muizenberg, 7945, South Africa; South African Radio Astronomy Observatory, The Park, Park Road, Pinelands, Cape Town 7405, South Africa
  12. California Institute of Technology, Pasadena, CA (United States). Div. of Physics, Mathematics, and Astronomy; California Institute of Technology, Pasadena, CA (United States). Center for Data Driven Discovery
  13. Institute of Astronomy and Kavli Institute for Cosmology, Madingley Road, Cambridge, United Kingdom; Univ. of Cambridge, Wilberforce Road, Cambridge, United Kingdom. Statistical Laboratory, DPMMS
  14. Harvard-Smithsonian Center for Astrophysics, Cambridge, MA (United States)
  15. Institute of Astronomy and Kavli Institute for Cosmology, Madingley Road, Cambridge, United Kingdom
  16. Space Telescope Science Institute, Baltimore, MD (United States)
  17. Stockholm University, AlbaNova, Stockholm, Sweden. The Oskar Klein Centre for Cosmoparticle Physics, Dept. of Physics; Department of Physics and Astronomy, University College London, Gower Street, London, United Kingdom
  18. Univ. of California Berkeley, Berkeley, CA (United States). Berkeley Center for Cosmological Physics
+ Show Author Affiliations
Classification of transient and variable light curves is an essential step in using astronomical observations to develop an understanding of the underlying physical processes from which they arise. However, upcoming deep photometric surveys, including the Large Synoptic Survey Telescope (LSST), will produce a deluge of low signal-to-noise data for which traditional type estimation procedures are inappropriate. Probabilistic classification is more appropriate for such data but is incompatible with the traditional metrics used on deterministic classifications. Furthermore, large survey collaborations like LSST intend to use the resulting classification probabilities for diverse science objectives, indicating a need for a metric that balances a variety of goals. We describe the process used to develop an optimal performance metric for an open classification challenge that seeks to identify probabilistic classifiers that can serve many scientific interests. The Photometric LSST Astronomical Time-series Classification Challenge (PLASTICC) aims to identify promising techniques for obtaining classification probabilities of transient and variable objects by engaging a broader community beyond astronomy. Using mock classification probability submissions emulating realistically complex archetypes of those anticipated of PLASTICC, we compare the sensitivity of two metrics of classification probabilities under various weighting schemes, finding that both yield results that are qualitatively consistent with intuitive notions of classification performance. We thus choose as a metric for PLASTICC a weighted modification of the cross-entropy because it can be meaningfully interpreted in terms of information content. Finally, we propose extensions of our methodology to ever more complex challenge goals and suggest some guiding principles for approaching the choice of a metric of probabilistic data products.
Research Organization:
Rutgers Univ., Piscataway, NJ (United States); Stanford Univ., CA (United States); Univ. of California, Oakland, CA (United States)
Sponsoring Organization:
Ajax Foundation; ERC; Marie Sklodowska-Curie; National Science Foundation (NSF); Swedish Research Council; USDOE Office of Science (SC)
Contributing Organization:
LSST Dark Energy Science Collaboration; LSST Transients and Variable Stars Science Collaboration
Grant/Contract Number:
AC02-05CH11231; AC02-76SF00515; SC0011636
OSTI ID:
1802229
Journal Information:
The Astronomical Journal (Online), Journal Name: The Astronomical Journal (Online) Journal Issue: 5 Vol. 158; ISSN 1538-3881
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

References (63)

Automated probabilistic classification of transients and variables journal March 2008
Ensemble Modeling of CMEs Using the WSA–ENLIL+Cone Model journal May 2015
Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning journal January 2018
Observing Dark Worlds: A crowdsourcing experiment for dark matter mapping journal July 2014
Measuring photometric redshifts using galaxy images and Deep Neural Networks journal July 2016
Separation of pulsar signals from noise using supervised machine learning algorithms journal April 2018
Validation of the NOAA Space Weather Prediction Center's solar flare forecasting look-up table and forecaster-issued probabilities: VALIDATION OF SWPC FLARE PROBABILITIES journal June 2012
Observation of a Rapidly Pulsating Radio Source journal February 1968
K ‐Corrections and Extinction Corrections for Type Ia Supernovae
  • Nugent, Peter; Kim, Alex; Perlmutter, Saul
  • Publications of the Astronomical Society of the Pacific, Vol. 114, Issue 798 https://doi.org/10.1086/341707
journal August 2002
SiFTO: An Empirical Method for Fitting SN Ia Light Curves journal July 2008
Results from the Supernova Photometric Classification Challenge
  • Kessler, Richard; Bassett, Bruce; Belov, Pavel
  • Publications of the Astronomical Society of the Pacific, Vol. 122, Issue 898 https://doi.org/10.1086/657607
journal December 2010
Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era
  • Bloom, J. S.; Richards, J. W.; Nugent, P. E.
  • Publications of the Astronomical Society of the Pacific, Vol. 124, Issue 921 https://doi.org/10.1086/668468
journal November 2012
PHOTOMETRIC TYPE Ia SUPERNOVA CANDIDATES FROM THE THREE-YEAR SDSS-II SN SURVEY DATA journal August 2011
Unity: Confronting Supernova Cosmology’S Statistical and Systematic Uncertainties in a Unified Bayesian Framework journal November 2015
CONSTRUCTION OF A CALIBRATED PROBABILISTIC CLASSIFICATION CATALOG: APPLICATION TO 50k VARIABLE SOURCES IN THE ALL-SKY AUTOMATED SURVEY journal December 2012
The Third Gravitational Lensing Accuracy Testing (Great3) Challenge Handbook journal April 2014
Bayesian High-Redshift Quasar Classification from Optical and Mid-Ir Photometry journal August 2015
Gravity Spy: integrating advanced LIGO detector characterization, machine learning, and citizen science journal February 2017
Photometric classification of type Ia supernovae in the SuperNova Legacy Survey with supervised learning journal December 2016
zBEAMS: a unified solution for supernova cosmology with redshift uncertainties journal October 2017
Kernel PCA for Type Ia supernovae photometric classification journal January 2013
A hybrid ensemble learning approach to star–galaxy classification journal August 2015
K2 variable catalogue – II. Machine learning classification of variable stars and eclipsing binaries in K2 fields 0–4 journal December 2015
Rotation-invariant convolutional neural networks for galaxy morphology prediction journal April 2015
Star–galaxy classification using deep convolutional neural networks journal October 2016
Deep learning classification in asteroseismology journal May 2017
Radio Galaxy Zoo: Claran – a deep learning classifier for radio morphologies journal October 2018
Deep learning classification in asteroseismology using an improved neural network: results on 15 000 Kepler red giants and applications to K2 and TESS data journal February 2018
Detection of bars in galaxies using a deep convolutional neural network journal March 2018
Machine-learning selection of optical transients in the Subaru/Hyper Suprime-Cam survey journal October 2016
Classification and unsupervised clustering of LIGO data with Deep Transfer Learning journal May 2018
Detecting Quasars in Large-Scale Astronomical Surveys
  • Gieseke, Fabian; Polsterer, Kai Lars; Thom, Andreas
  • 2010 International Conference on Machine Learning and Applications (ICMLA), 2010 Ninth International Conference on Machine Learning and Applications https://doi.org/10.1109/ICMLA.2010.59
conference December 2010
Matplotlib: A 2D Graphics Environment journal January 2007
Python for Scientific Computing journal January 2007
The NumPy Array: A Structure for Efficient Numerical Computation journal March 2011
Statistical classification techniques for photometric supernova typing: Statistical methods for photometric typing journal April 2011
Verification of Forecasts Expressed in Terms of Probability journal January 1950
Verification of Forecasts Expressed in Terms of Probability journal January 1950
Gravitational Lensing Accuracy Testing 2010 (GREAT10) Challenge Handbook journal September 2011
The measurement of radio source diameters using a diffraction method text January 1969
Photometric Supernova Classification with Machine Learning journal August 2016
Approximating Photo- z PDFs for Large Surveys journal June 2018
Deep-HiTS: Rotation Invariant Convolutional Neural Network for Transient Detection journal February 2017
Measuring Dark Energy Properties with Photometrically Classified Pan-STARRS Supernovae. II. Cosmological Parameters journal April 2018
Detecting Solar-like Oscillations in Red Giants with Deep Learning journal May 2018
Machine-learning-based Brokers for Real-time Classification of the LSST Alert Stream journal May 2018
Results from the Supernova Photometric Classification Challenge text January 2010
Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era text January 2011
Kernel PCA for type Ia supernovae photometric classification text January 2012
The Third Gravitational Lensing Accuracy Testing (GREAT3) Challenge Handbook text January 2013
Ensemble modeling of CMEs using the WSA-ENLIL+Cone model text January 2015
Measuring photometric redshifts using galaxy images and Deep Neural Networks preprint January 2015
A Hybrid Ensemble Learning Approach to Star-Galaxy Classification text January 2015
UNITY: Confronting Supernova Cosmology's Statistical and Systematic Uncertainties in a Unified Bayesian Framework text January 2015
K2 Variable Catalogue II: Machine Learning Classification of Variable Stars and Eclipsing Binaries in K2 Fields 0-4 text January 2015
Photometric Supernova Classification With Machine Learning text January 2016
Photometric classification of type Ia supernovae in the SuperNova Legacy Survey with supervised learning text January 2016
Machine-learning Selection of Optical Transients in Subaru/Hyper Suprime-Cam Survey text January 2016
Gravity Spy: Integrating Advanced LIGO Detector Characterization, Machine Learning, and Citizen Science text January 2016
Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning text January 2018
Machine Learning-based Brokers for Real-time Classification of the LSST Alert Stream text January 2018
Detecting Solar-like Oscillations in Red Giants with Deep Learning text January 2018
K-corrections and Extinction Corrections for Type Ia Supernovae text January 2002

Cited By (2)

Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation journal December 2019
Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation text January 2019

Similar Records

Results of the Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC)
Journal Article · Thu Jul 20 20:00:00 EDT 2023 · The Astrophysical Journal. Supplement Series · OSTI ID:1991984

Related Subjects

79 ASTRONOMY AND ASTROPHYSICS
astronomy & astrophysics
data analysis
photometric
statistical methods
supernovae
surveys
variables