DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

Journal Article · · The Astrophysical Journal
ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo more »; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo « less

Abstract We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.

Sponsoring Organization:
USDOE
OSTI ID:
2467566
Journal Information:
The Astrophysical Journal, Journal Name: The Astrophysical Journal Journal Issue: 2 Vol. 974; ISSN 0004-637X
Publisher:
American Astronomical SocietyCopyright Statement
Country of Publication:
United States
Language:
English

References (181)

The Levenberg-Marquardt algorithm: Implementation and theory book January 1978
Astronomaly: Personalised active anomaly detection in astronomical data journal July 2021
The influence of host galaxy morphology on the properties of Type Ia supernovae from the JLA compilation journal February 2017
SNAD transient miner: Finding missed transient events in ZTF DR4 using k-D trees journal October 2022
Random Forests journal January 2001
An unusual supernova in the error box of the γ-ray burst of 25 April 1998 journal October 1998
A new class of flares from accreting supermassive black holes journal January 2019
The core-collapse rate from the Supernova Legacy Survey journal April 2009
Astropy: A community Python package for astronomy journal September 2013
The EPOCH Project: I. Periodic variable stars in the EROS-2 LMC database⋆ journal June 2014
Automated novelty detection in the WISE survey with one-class support vector machines journal October 2017
PELICAN: deeP architecturE for the LIght Curve ANalysis journal June 2019
Transient processing and analysis using AMPEL: alert management, photometry, and evaluation of light curves journal November 2019
Type IIn supernova light-curve properties measured from an untargeted survey sample journal May 2020
Active anomaly detection for time-domain discoveries journal June 2021
Fink: Early supernovae Ia classification using active learning journal July 2022
Astromer journal February 2023
Understanding of the properties of neural network approaches for transient light curve approximations journal August 2023
The broad-lined Type-Ic supernova SN 2022xxf and its extraordinary two-humped light curves journal October 2023
RAINBOW: A colorful approach to multipassband light-curve estimation journal March 2024
The VizieR database of astronomical catalogues journal April 2000
The frontier of simulation-based inference journal May 2020
An optimal extraction algorithm for CCD spectroscopy journal June 1986
On the Automatic Determination of Light-Curve Parameters for Cepheid Variables journal October 1996
Spectroscopy and photometry of elliptical galaxies. I - A new distance estimator journal February 1987
The relationship between infrared, optical, and ultraviolet extinction journal October 1989
Photometry of a complete sample of faint galaxies journal June 1980
Measurements of the Cosmological Parameters Ω and Λ from the First Seven Supernovae at z ≥ 0.35 journal July 1997
The High‐Z Supernova Search: Measuring Cosmic Deceleration and Global Curvature of the Universe Using Type Ia Supernovae journal November 1998
Color Separation of Galaxy Types in the Sloan Digital Sky Survey Imaging Data journal October 2001
The Application of Photometric Redshifts to the SDSS Early Data Release journal February 2003
Galactic Stellar and Substellar Initial Mass Function journal July 2003
Optical Photometry and Spectroscopy of the SN 1998bw–like Type Ic Supernova 2002ap
  • Foley, Ryan J.; Papenkova, Marina S.; Swift, Brandon J.
  • Publications of the Astronomical Society of the Pacific, Vol. 115, Issue 812 https://doi.org/10.1086/378242
journal October 2003
Twenty‐Three High‐Redshift Supernovae from the Institute for Astronomy Deep Survey: Doubling the Supernova Sample at z > 0.7  journal February 2004
Nearly 5000 Distant Early‐Type Galaxies in COMBO‐17: A Red Sequence and Its Evolution since z  ∼ 1 journal June 2004
Gemini Spectroscopy of Supernovae from the Supernova Legacy Survey: Improving High‐Redshift Supernova Selection and Classification journal December 2005
The Two Micron All Sky Survey (2MASS) journal February 2006
Determining the Type, Redshift, and Age of a Supernova Spectrum journal September 2007
SNANA: A Public Software Package for Supernova Analysis
  • Kessler, Richard; Bernstein, Joseph P.; Cinabro, David
  • Publications of the Astronomical Society of the Pacific, Vol. 121, Issue 883 https://doi.org/10.1086/605984
journal September 2009
emcee : The MCMC Hammer
  • Foreman-Mackey, Daniel; Hogg, David W.; Lang, Dustin
  • Publications of the Astronomical Society of the Pacific, Vol. 125, Issue 925 https://doi.org/10.1086/670067
journal March 2013
The Wide-Field Infrared Survey Explorer (Wise): Mission Description and Initial On-Orbit Performance journal November 2010
Absolute-Magnitude Distributions of Supernovae journal April 2014
Optical Spectra of 73 Stripped-Envelope Core-Collapse Supernovae journal March 2014
The Propagation of Uncertainties in Stellar Population Synthesis Modeling. i. the Relevance of Uncertain Aspects of Stellar Evolution and the Initial mass Function to the Derived Physical Properties of Galaxies journal June 2009
The Propagation of Uncertainties in Stellar Population Synthesis Modeling. iii. Model Calibration, Comparison, and Evaluation journal March 2010
HUBBLE RESIDUALS OF NEARBY TYPE Ia SUPERNOVAE ARE CORRELATED WITH HOST GALAXY MASSES journal May 2010
THE SPITZER - WISE SURVEY OF THE ECLIPTIC POLES journal June 2011
THE Pan-STARRS1 PHOTOMETRIC SYSTEM journal April 2012
MID-INFRARED SELECTION OF ACTIVE GALACTIC NUCLEI WITH THE WIDE-FIELD INFRARED SURVEY EXPLORER . I. CHARACTERIZING WISE -SELECTED ACTIVE GALACTIC NUCLEI IN COSMOS journal June 2012
THEGALEXTIME DOMAIN SURVEY. I. SELECTION AND CLASSIFICATION OF OVER A THOUSAND ULTRAVIOLET VARIABLE SOURCES journal March 2013
Classifying Supernovae Using only Galaxy data journal November 2013
Supervised Detection of Anomalous Light Curves in Massive Astronomical Catalogs journal September 2014
Improving Cosmological Distance Measurements Using twin type ia Supernovae journal December 2015
A Catalog of Detailed Visual Morphological Classifications for 14,034 Galaxies in the Sloan Digital sky Survey journal February 2010
Extending Supernova Spectral Templates for Next-generation Space Telescope Observations journal October 2018
A Morphological Classification Model to Identify Unresolved PanSTARRS1 Sources: Application in the ZTF Real-time Pipeline journal November 2018
The Zwicky Transient Facility Alert Distribution System journal November 2018
The Zwicky Transient Facility: System Overview, Performance, and First Results journal December 2018
RAPID: Early Classification of Explosive Transients Using Deep Learning journal September 2019
YSE-PZ: A Transient Survey Management Platform that Empowers the Human-in-the-loop journal June 2023
Rubin Observatory LSST Transients and Variable Stars Roadmap journal October 2023
Exploring the Chemical link Between Local Ellipticals and Their High-Redshift Progenitors journal November 2013
DIGS: deep inference of galaxy spectra with neural posterior estimation journal December 2022
Unsupervised machine learning for transient discovery in deeper, wider, faster light curves journal September 2020
PS1-STRM: neural network source classification and photometric redshift catalogue for PS1 3π DR1 journal August 2020
Density-based outlier scoring on Kepler data journal September 2020
Fink , a new generation of broker for the LSST community journal November 2020
The ZTF Source Classification Project – II. Periodicity and variability processing metrics journal May 2021
A method for finding anomalous astronomical light curves and their analogues journal September 2021
Anomaly detection in Hyper Suprime-Cam galaxy images with generative adversarial networks journal September 2021
Anomaly detection in the Zwicky Transient Facility DR3 journal February 2021
Real-time detection of anomalies in large-scale transient surveys journal September 2022
Pan-chromatic photometric classification of supernovae from multiple surveys and transfer learning for future surveys journal December 2022
Concerning colour: The effect of environment on type Ia supernova colour in the dark energy survey journal December 2022
SN 2022ann: a Type Icn supernova from a dwarf galaxy that reveals helium in its circumstellar environment journal May 2023
Fast and efficient identification of anomalous galaxy spectra with neural density estimation journal September 2023
The simulated catalogue of optical transients and correlated hosts (SCOTCH) journal January 2023
Scalable hierarchical BayeSN inference: investigating dependence of SN Ia host galaxy dust properties on stellar mass and redshift journal May 2024
Astronomaly at scale: searching for anomalies amongst 4 million galaxies journal February 2024
SN Ia host galaxy properties from Sloan Digital Sky Survey-II spectroscopy journal September 2013
An analysis of feature relevance in the classification of astronomical transients with machine learning methods journal February 2016
Investigating the diversity of supernovae type Iax: a MUSE and NOT spectroscopic study of their environments journal September 2017
GLADE: A galaxy catalogue for multimessenger searches in the advanced gravitational-wave detector era journal June 2018
Systematic serendipity: a test of unsupervised machine learning as a method for anomaly detection journal December 2018
Anomaly Detection in the Open Supernova Catalog journal August 2019
Spectrophotometric templates for core-collapse supernovae and their application in simulations of time-domain surveys journal September 2019
Supernovae and their host galaxies – VI. Normal Type Ia and 91bg-like supernovae in ellipticals journal September 2019
SuperNNova: an open-source framework for Bayesian, neural network-based supernova classification journal December 2019
ADASYN: Adaptive synthetic sampling approach for imbalanced learning conference June 2008
Matplotlib: A 2D Graphics Environment journal January 2007
The NumPy Array: A Structure for Efficient Numerical Computation journal March 2011
Skysurveys, Light Curves and Statistical Challenges journal September 2015
The ages and metallicities of galaxies in the local universe journal September 2005
The UKIRT Infrared Deep Sky Survey (UKIDSS) journal August 2007
Spectral analysis of the 91bg-like Type Ia SN 2005bl: low luminosity, low velocities, incomplete burning journal November 2009
The dependence of Type Ia Supernovae luminosities on their host galaxies: SN Ia host galaxies journal May 2010
Nearby supernova rates from the Lick Observatory Supernova Search - II. The observed luminosity functions and fractions of supernovae in a complete sample: Nearby supernova rates from LOSS - II journal March 2011
Berkeley Supernova Ia Program - I. Observations, data reduction and spectroscopic sample of 582 low-redshift Type Ia supernovae: BSNIP I: SN Ia spectra journal August 2012
Observed versus modelledu-,g-,r-,i-,z-band photometry of local galaxies – evaluation of model performance journal November 2012
Isolation-Based Anomaly Detection journal March 2012
The Most Luminous Supernovae journal August 2019
Unsupervised Learning With Random Forest Predictors journal March 2006
SMOTE: Synthetic Minority Over-sampling Technique journal January 2002
UMAP: Uniform Manifold Approximation and Projection journal September 2018
Ensemble Learning Method for Outlier Detection and its Application to Astronomical Light Curves journal August 2016
Mesa Isochrones and Stellar Tracks (Mist). i. Solar-Scaled Models journal May 2016
Analyzing the Largest Spectroscopic data set of Stripped Supernovae to Improve Their Identifications and Constrain Their Progenitors journal August 2016
THE SPECTRAL SN-GRB CONNECTION: SYSTEMATIC SPECTRAL COMPARISONS BETWEEN TYPE Ic SUPERNOVAE AND BROAD-LINED TYPE Ic SUPERNOVAE WITH AND WITHOUT GAMMA-RAY BURSTS journal November 2016
Mesa Isochrones and Stellar Tracks (Mist) 0: Methods for the Construction of Stellar Isochrones journal January 2016
Photometric Supernova Classification with Machine Learning journal August 2016
The Astropy Project: Building an Open-science Project and Status of the v2.0 Core Package journal August 2018
Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation journal December 2019
Alert Classification for the ALeRCE Broker System: The Light Curve Classifier journal February 2021
The ANTARES Astronomical Time-domain Event Broker journal February 2021
The ZTF Source Classification Project. I. Methods and Infrastructure journal May 2021
The Automatic Learning for the Rapid Classification of Events (ALeRCE) Alert Broker journal April 2021
SCONE: Supernova Classification with a Convolutional Neural Network journal July 2021
Alert Classification for the ALeRCE Broker System: The Real-time Stamp Classifier journal November 2021
ParSNIP: Generative Models of Transient Light Curves with Physics-enabled Deep Learning journal December 2021
DELIGHT: Deep Learning Identification of Galaxy Hosts of Transients using Multiresolution Images journal October 2022
Deep Attention-based Supernovae Classification of Multiband Light Curves journal December 2022
Alert Classification for the ALeRCE Broker System: The Anomaly Detector journal September 2023
Autoencoding Galaxy Spectra. II. Redshift Invariance and Outlier Detection journal July 2023
LOSS Revisited. II. The Relative Rates of Different Types of Supernovae Vary between Low- and High-mass Galaxies journal March 2017
Deriving Physical Properties from Broadband Photometry with Prospector: Description of the Model and a Demonstration of its Accuracy Using 129 Galaxies in the Local Universe journal March 2017
Nebular Continuum and Line Emission in Stellar Population Synthesis Models journal May 2017
Spectral Sequences of Type Ia Supernovae. I. Connecting Normal and Subluminous SNe Ia and the Presence of Unburned Carbon journal August 2017
Type II Supernova Spectral Diversity. I. Observations, Sample Characterization, and Spectral Line Evolution journal November 2017
Near-infrared Variability of Obscured and Unobscured X-Ray-selected AGNs in the COSMOS Field journal November 2017
Spectra of Hydrogen-poor Superluminous Supernovae from the Palomar Transient Factory journal February 2018
An Embedded X-Ray Source Shines through the Aspherical AT 2018cow: Revealing the Inner Workings of the Most Luminous Fast-evolving Optical Transients journal February 2019
The First Tidal Disruption Flare in ZTF: From Photometric Selection to Multi-wavelength Characterization journal February 2019
LSST: From Science Drivers to Reference Design and Anticipated Data Products journal March 2019
How to Measure Galaxy Star Formation Histories. II. Nonparametric Models journal April 2019
An Older, More Quiescent Universe from Panchromatic SED Fitting of the 3D- HST Survey journal June 2019
Supernova Photometric Classification Pipelines Trained on Spectroscopically Classified Supernovae from the Pan-STARRS1 Medium-deep Survey journal October 2019
A Classification Algorithm for Time-domain Novelties in Preparation for LSST Alerts. Application to Variable Stars and Transients Detected with DECam in the Galactic Bulge journal April 2020
The Zwicky Transient Facility Bright Transient Survey. I. Spectroscopic Classification and the Redshift Completeness of Local Galaxy Catalogs journal May 2020
Star Formation and Morphological Properties of Galaxies in the Pan-STARRS 3π Survey. I. A Machine-learning Approach to Galaxy and Supernova Classification journal October 2020
FLEET: A Redshift-agnostic Machine Learning Pipeline to Rapidly Identify Hydrogen-poor Superluminous Supernovae journal November 2020
The Distant, Galaxy Cluster Environment of the Short GRB 161104A at z ∼ 0.8 and a Comparison to the Short GRB Host Population journal November 2020
SuperRAENN: A Semisupervised Supernova Photometric Classification Pipeline Trained on Pan-STARRS1 Medium-Deep Survey Supernovae journal December 2020
GHOST: Using Only Host Galaxy Information to Accurately Associate and Distinguish Supernovae journal February 2021
It’s Dust: Solving the Mysteries of the Intrinsic Scatter and Host-galaxy Dependence of Standardized Type Ia Supernova Brightnesses journal March 2021
The Young Supernova Experiment: Survey Goals, Overview, and Operations journal February 2021
The Twins Embedding of Type Ia Supernovae. II. Improving Cosmological Distance Estimates journal May 2021
Improved Treatment of Host-galaxy Correlations in Cosmological Analyses with Type Ia Supernovae journal May 2021
Fast-transient Searches in Real Time with ZTFReST: Identification of Three Optically Discovered Gamma-Ray Burst Afterglows and New Constraints on the Kilonova Rate journal September 2021
A Family Tree of Optical Transients from Narrow-line Seyfert 1 Galaxies journal October 2021
SALT3: An Improved Type Ia Supernova Model for Measuring Cosmic Distances journal December 2021
An Early-time Optical and Ultraviolet Excess in the Type-Ic SN 2020oi journal January 2022
Final Moments. I. Precursor Emission, Envelope Inflation, and Enhanced Mass Loss Preceding the Luminous Type II Supernova 2020tlf journal January 2022
Less Than 1% of Core-collapse Supernovae in the Local Universe Occur in Elliptical Galaxies journal March 2022
The Type Icn SN 2021csp: Implications for the Origins of the Fastest Supernovae and the Fates of Wolf–Rayet Stars journal March 2022
The Circumstellar Environments of Double-peaked, Calcium-strong Transients 2021gno and 2021inl journal June 2022
Accelerated Bayesian SED Modeling Using Amortized Neural Posterior Estimation journal October 2022
A Systematic Study of Ia-CSM Supernovae from the ZTF Bright Transient Survey journal May 2023
Identifying Tidal Disruption Events with an Expansion of the FLEET Machine-learning Algorithm journal June 2023
The First Two Years of FLEET: An Active Search for Superluminous Supernovae journal June 2023
LensWatch. I. Resolved HST Observations and Constraints on the Strongly Lensed Type Ia Supernova 2022qmx (“SN Zwicky”) journal May 2023
Corrected SFD: A More Accurate Galactic Dust Map with Minimal Extragalactic Contamination journal November 2023
Final Moments. II. Observational Properties and Physical Modeling of Circumstellar-material-interacting Type II Supernovae journal July 2024
Pan-STARRS Photometric and Astrometric Calibration journal October 2020
Stellar Population Inference with Prospector journal May 2021
The Palomar Transient Factory Core-collapse Supernova Host-galaxy Sample. I. Host-galaxy Distribution Functions and Environment Dependence of Core-collapse Supernovae journal August 2021
A Deep-learning Approach for Live Anomaly Detection of Extragalactic Transients journal August 2021
Linking Extragalactic Transients and Their Host Galaxy Properties: Transient Sample, Multiwavelength Host Identification, and Database Construction journal February 2022
Considerations for Optimizing the Photometric Classification of Supernovae from the Rubin Observatory journal January 2022
Preparing to Discover the Unknown with Rubin LSST: Time Domain journal December 2021
Neural Stellar Population Synthesis Emulator for the DESI PROVABGS journal March 2023
The Young Supernova Experiment Data Release 1 (YSE DR1): Light Curves and Photometric Classification of 1975 Supernovae journal May 2023
Results of the Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC) journal July 2023
Identifying Light-curve Signals with a Deep-learning-based Object Detection Algorithm. II. A General Light-curve Classification Framework journal September 2024
Deep Recurrent Neural Networks for Supernovae Classification journal March 2017
Optimal Classification and Outlier Detection for Stripped-envelope Core-collapse Supernovae journal July 2019
Strong Calcium Emission Indicates that the Ultraviolet-flashing SN Ia 2019yvq Was the Result of a Sub-Chandrasekar-mass Double-detonation Explosion journal September 2020
Optical Rebrightening of Extragalactic Transients from the Zwicky Transient Facility journal February 2022
Multiscale Stamps for Real-time Classification of Alert Streams journal August 2023
pandas-dev/pandas: Pandas software January 2024
YSE-PZ: An Open-source Target and Observation Management System software November 2022
Gaussian Processes for Machine Learning book January 2005