DMTN-260: Failure Modes and Error Handling for Prompt Processing

Findeisen, Krzysztof; Lim, Kian-Tat; Bellm, Eric C.; Chiang, Hsin-Fang; Parejko, John K.

doi:10.71929/rubin/2998133

Title: DMTN-260: Failure Modes and Error Handling for Prompt Processing

Dataset
Other Related Research

Abstract

The Prompt Processing system will be responsible for processing roughly a thousand visits per night, and distributing the results in near real time, for at least ten years of Rubin Observatory operations. As such, it must be highly robust to algorithmic, network, and infrastructure failures, ranging from momentary glitches to extended downtimes. DMTN-219 introduced the initial design for the Prompt Processing framework; this document expands on the design to address expected failure modes and recovery strategies for each.

Authors:

;

Dept. of Astronomy, University of Washington
SLAC National Accelerator Laboratory

Publication Date:: Wed May 08 04:00:00 UTC 2024

Other Number(s):: Vera C. Rubin Observatory Data Management Technical Note DMTN-260

DOE Contract Number:: AC02-76SF00515

Research Org.:: NSF-DOE Vera C. Rubin Observatory

Sponsoring Org.:: U.S. National Science Foundation; U.S. Department of Energy Office of Science

Subject:: 79 ASTRONOMY AND ASTROPHYSICS

OSTI Identifier:: 2998133

DOI:: https://doi.org/10.71929/rubin/2998133

Citation Formats


                    Findeisen, Krzysztof, Lim, Kian-Tat, Bellm, Eric C., Chiang, Hsin-Fang, and Parejko, John K. DMTN-260: Failure Modes and Error Handling for Prompt Processing.  United States: N. p., 2024. 
        Web.  doi:10.71929/rubin/2998133.

Copy to clipboard


                    Findeisen, Krzysztof, Lim, Kian-Tat, Bellm, Eric C., Chiang, Hsin-Fang, & Parejko, John K. DMTN-260: Failure Modes and Error Handling for Prompt Processing.  United States.  doi:https://doi.org/10.71929/rubin/2998133

Copy to clipboard


                    Findeisen, Krzysztof, Lim, Kian-Tat, Bellm, Eric C., Chiang, Hsin-Fang, and Parejko, John K. 2024.  
"DMTN-260: Failure Modes and Error Handling for Prompt Processing".  United States.  doi:https://doi.org/10.71929/rubin/2998133.  https://www.osti.gov/servlets/purl/2998133. Pub date:Wed May 08 04:00:00 UTC 2024

Copy to clipboard


                    
@article{osti_2998133,

  title        = {DMTN-260: Failure Modes and Error Handling for Prompt Processing},

  author       = {Findeisen, Krzysztof and Lim, Kian-Tat and Bellm, Eric C. and Chiang, Hsin-Fang and Parejko, John K.},

  abstractNote = {The Prompt Processing system will be responsible for processing roughly a thousand visits per night, and distributing the results in near real time, for at least ten years of Rubin Observatory operations. As such, it must be highly robust to algorithmic, network, and infrastructure failures, ranging from momentary glitches to extended downtimes. DMTN-219 introduced the initial design for the Prompt Processing framework; this document expands on the design to address expected failure modes and recovery strategies for each.},

  doi          = {10.71929/rubin/2998133},

  journal      = {},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {Wed May 08 04:00:00 UTC 2024},

  month        = {Wed May 08 04:00:00 UTC 2024}

}

Copy to clipboard

Dataset:

View Dataset

DOI: https://doi.org/10.71929/rubin/2998133

Save / Share:

Export Metadata

Save to My Library

Similar records in DOE Data Explorer and OSTI.GOV collections:

DMTN-219: Proposal and Prototype for Prompt Processing

Dataset Lim, Kian-Tat

The Prompt Processing framework has long been a gray area in the overall Rubin Observatory architecture. This document attempts to fill that hole by proposing a concrete design with an accompanying prototype implementation that should meet minimum requirements. Additional features and optimizations that could be incorporated later are also described.
DMTN-197: Streak Masking in DM Image Processing

Dataset Saunders, Clare

Streaks caused by satellites are a persistent problem in optical images, and their occurrence will likely increase in the coming years. Many Hyper Suprime-Cam images are already affected by streaks, and the same is expected for Rubin Observatory data. In the LSST Data Release Production (DRP), most streaks and other artifacts are already detected and masked by the algorithm CompareWarpAssembleCoadd. However, some streaks are not caught at this stage and can contaminate the final coadds and detection catalogs. To find and mask these, we adopt a morphologically-based method for detecting streaks, which uses the Kernel-Based Hough Transform to detect straightmore »« less
DMTN-015: Flavors of Coadds

Dataset Bosch, James F.

A glossary of different kinds of coadded images, with brief descriptions of the algorithms behind them.
DMTN-118: Review of Timeseries Features

Dataset Bellm, Eric C.

Rubin Observatory will compute timeseries variability features on lightcurves to aid users in identifying objects of interest, both during realtime Alert Production as well as in the annual Data Releases. The Data Products Definition Document (DPDD; LSE-163) allocates space for pre-computed timeseries features, and a sample set is baselined in LDM-151. However, in the subsequent decade the scientfic community has made a great deal of further progress in this area. This technote reviews the relevant literature, grouping related features where possible; discusses potential concerns and open questions; and proposes a new baseline feature set.
DMTN-171: Fall 2020 status of crowded field processing with the LSST Alert Production Pipelines

Dataset Sullivan, Ian S. ; Bellm, Eric C.

A report on the Fall 2020 state of the LSST Alert Production Pipelines' ability to process deep images in the galactic bulge.

Similar Records