Resimulation-based self-supervised learning for pretraining physics foundation models

Harris, P.; Krupa, J.; Kagan, M.; Maier, B.; Woodward, N.

doi:10.1103/physrevd.111.032010

Resimulation-based self-supervised learning for pretraining physics foundation models

Journal Article · Thu Feb 20 23:00:00 EST 2025 · Physical Review. D.

DOI:https://doi.org/10.1103/physrevd.111.032010· OSTI ID:2575499

Harris, P. ^[1]; ^[2]; ^[3]; ^[4]; ^[5]

Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States); Institute for Artificial Intelligence and Fundamental Interactions, Cambridge, MA (United States)
Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States); Institute for Artificial Intelligence and Fundamental Interactions, Cambridge, MA (United States); SLAC National Accelerator Laboratory (SLAC), Stanford, CA (United States)
SLAC National Accelerator Laboratory (SLAC), Stanford, CA (United States)
Imperial College, London (United Kingdom)
Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)

Self-supervised learning (SSL) is at the core of training modern large machine learning models, providing a scheme for learning powerful representations that can be used in a variety of downstream tasks. However, SSL strategies must be adapted to the type of training data and downstream tasks required. We propose resimulation-based self-supervised representation learning (RS3L), a novel simulation-based SSL strategy that employs a method of resimulation to drive data augmentation for contrastive learning in the physical sciences, particularly, in fields that rely on stochastic simulators. By intervening in the middle of the simulation process and rerunning simulation components downstream of the intervention, we generate multiple realizations of an event, thus producing a set of augmentations covering all physics-driven variations available in the simulator. Using experiments from high-energy physics, we explore how this strategy may enable the development of a foundation model; we show how RS3L pretraining enables powerful performance in downstream tasks such as discrimination of a variety of objects and uncertainty mitigation. In addition to our results, we make the RS3L dataset publicly available for further studies on how to improve SSL strategies.

View Accepted Manuscript (DOE)

Research Organization:: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER). Biological Systems Science (BSS); National Science Foundation (NSF)

Grant/Contract Number:: AC02-76SF00515

OSTI ID:: 2575499

Journal Information:: Physical Review. D., Journal Name: Physical Review. D. Journal Issue: 3 Vol. 111; ISSN 2470-0010; ISSN 2470-0029

Publisher:: American Physical Society (APS)Copyright Statement

Country of Publication:: United States

Language:: English

References (33)

Search for low mass vector resonances decaying into quark-antiquark pairs in proton-proton collisions at s = 13 $$ \sqrt{s}=13 $$ TeV Sirunyan, A. M.; Tumasyan, A.; Adam, W. Journal of High Energy Physics, Vol. 2018, Issue 1 https://doi.org/10.1007/JHEP01(2018)097	journal	January 2018
DELPHES 3: a modular framework for fast simulation of a generic collider experiment de Favereau, J.; Delaere, C.; Demin, P. Journal of High Energy Physics, Vol. 2014, Issue 2 https://doi.org/10.1007/JHEP02(2014)057	journal	February 2014
Search for dark matter particles produced in association with a Higgs boson in proton-proton collisions at s$$ \sqrt{\mathrm{s}} $$ = 13 TeV Sirunyan, A. M.; Tumasyan, A.; Adam, W. Journal of High Energy Physics, Vol. 2020, Issue 3 https://doi.org/10.1007/JHEP03(2020)025	journal	March 2020
Parton distributions for the LHC run II Ball, Richard D.; Bertone, Valerio; Carrazza, Stefano Journal of High Energy Physics, Vol. 2015, Issue 4 https://doi.org/10.1007/JHEP04(2015)040	journal	April 2015
The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations Alwall, J.; Frederix, R.; Frixione, S. Journal of High Energy Physics, Vol. 2014, Issue 7 https://doi.org/10.1007/JHEP07(2014)079	journal	July 2014
New angles on energy correlation functions Moult, Ian; Necib, Lina; Thaler, Jesse Journal of High Energy Physics, Vol. 2016, Issue 12 https://doi.org/10.1007/JHEP12(2016)153	journal	December 2016
Inclusive search for highly boosted Higgs bosons decaying to bottom quark-antiquark pairs in proton-proton collisions at $$ \sqrt{s} $$ = 13 TeV Sirunyan, A. M.; Tumasyan, A.; Adam, W. Journal of High Energy Physics, Vol. 2020, Issue 12 https://doi.org/10.1007/JHEP12(2020)085	journal	December 2020
An introduction to PYTHIA 8.2 Sjöstrand, Torbjörn; Ask, Stefan; Christiansen, Jesper R. Computer Physics Communications, Vol. 191 https://doi.org/10.1016/j.cpc.2015.01.024	journal	June 2015
Observation of H → b b ¯ decays and VH production with the ATLAS detector Aaboud, M.; Aad, G.; Abbott, B. Physics Letters B, Vol. 786 https://doi.org/10.1016/j.physletb.2018.09.013	journal	November 2018
Measurement of the associated production of a Higgs boson decaying into b-quarks with a vector boson at high transverse momentum in pp collisions at s = 13 TeV with the ATLAS detector Aad, G.; Abbott, B.; Abbott, D. C. Physics Letters B, Vol. 816 https://doi.org/10.1016/j.physletb.2021.136204	journal	May 2021
Large-scale chemical language representations capture molecular structure and properties Ross, Jerret; Belgodere, Brian; Chenthamarakshan, Vijil Nature Machine Intelligence, Vol. 4, Issue 12 https://doi.org/10.1038/s42256-022-00580-7	journal	December 2022
Large language model for molecular chemistry Pan, Jie Nature Computational Science, Vol. 3, Issue 1 https://doi.org/10.1038/s43588-023-00399-1	journal	January 2023
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences Rives, Alexander; Meier, Joshua; Sercu, Tom Proceedings of the National Academy of Sciences, Vol. 118, Issue 15 https://doi.org/10.1073/pnas.2016239118	journal	April 2021
The anti- k _t jet clustering algorithm Cacciari, Matteo; Salam, Gavin P.; Soyez, Gregory Journal of High Energy Physics, Vol. 2008, Issue 04 https://doi.org/10.1088/1126-6708/2008/04/063	journal	April 2008
LHC Machine Evans, Lyndon; Bryant, Philip Journal of Instrumentation, Vol. 3, Issue 08 https://doi.org/10.1088/1748-0221/3/08/S08001	journal	August 2008
The ATLAS Experiment at the CERN Large Hadron Collider Collaboration, The ATLAS; Aad, G.; Abat, E. Journal of Instrumentation, Vol. 3, Issue 08 https://doi.org/10.1088/1748-0221/3/08/S08003	journal	August 2008
The CMS experiment at the CERN LHC Collaboration, The CMS; Chatrchyan, S.; Hmayakyan, G. Journal of Instrumentation, Vol. 3, Issue 08 https://doi.org/10.1088/1748-0221/3/08/S08004	journal	August 2008
Graph neural networks in particle physics Shlomi, Jonathan; Battaglia, Peter; Vlimant, Jean-Roch Machine Learning: Science and Technology, Vol. 2, Issue 2 https://doi.org/10.1088/2632-2153/abbf9a	journal	January 2021
Chemformer: a pre-trained transformer for computational chemistry Irwin, Ross; Dimitriadis, Spyridon; He, Jiazhen Machine Learning: Science and Technology, Vol. 3, Issue 1 https://doi.org/10.1088/2632-2153/ac3ffb	journal	January 2022
Pile-up mitigation using attention Maier, B.; Narayanan, S. M.; de Castro, G. Machine Learning: Science and Technology, Vol. 3, Issue 2 https://doi.org/10.1088/2632-2153/ac7198	journal	June 2022
Finetuning foundation models for joint analysis optimization in High Energy Physics Vigl, Matthias; Hartman, Nicole; Heinrich, Lukas Machine Learning: Science and Technology, Vol. 5, Issue 2 https://doi.org/10.1088/2632-2153/ad55a3	journal	June 2024
Masked particle modeling on sets: towards self-supervised high energy physics foundation models Golling, Tobias; Heinrich, Lukas; Kagan, Michael Machine Learning: Science and Technology, Vol. 5, Issue 3 https://doi.org/10.1088/2632-2153/ad64a8	journal	September 2024
OmniJet-α: the first cross-task foundation model for particle physics Birk, Joschka; Hallin, Anna; Kasieczka, Gregor Machine Learning: Science and Technology, Vol. 5, Issue 3 https://doi.org/10.1088/2632-2153/ad66ad	journal	August 2024
Data compression and inference in cosmology with self-supervised machine learning Akhmetzhanova, Aizhan; Mishra-Sharma, Siddharth; Dvorkin, Cora Monthly Notices of the Royal Astronomical Society, Vol. 527, Issue 3 https://doi.org/10.1093/mnras/stad3646	journal	November 2023
Search for low mass vector resonances decaying into quark-antiquark pairs in proton-proton collisions at s = 13 TeV Sirunyan, A. M.; Tumasyan, A.; Adam, W. Physical Review D, Vol. 100, Issue 11 https://doi.org/10.1103/PhysRevD.100.112007	journal	December 2019
Measurement of simplified template cross sections of the Higgs boson produced in association with W or Z bosons in the H→bb¯ decay channel in proton-proton collisions at s=13 TeV Tumasyan, A.; Adam, W.; Andrejkovic, J. W. Physical Review D, Vol. 109, Issue 9 https://doi.org/10.1103/PhysRevD.109.092011	journal	May 2024
Observation of Higgs Boson Decay to Bottom Quarks Sirunyan, A. M.; Tumasyan, A.; Adam, W. Physical Review Letters, Vol. 121, Issue 12 https://doi.org/10.1103/PhysRevLett.121.121801	journal	September 2018
Search for dark matter produced in association with a Higgs boson decaying to a pair of bottom quarks in proton–proton collisions at $$\sqrt{s}=13\,\text {Te}\text {V} $$ s = 13 Te Sirunyan, A. M.; Tumasyan, A.; Adam, W. The European Physical Journal C, Vol. 79, Issue 3 https://doi.org/10.1140/epjc/s10052-019-6730-7	journal	March 2019
Extraction and validation of a new set of CMS pythia8 tunes from underlying-event measurements Sirunyan, A. M.; Tumasyan, A.; Adam, W. The European Physical Journal C, Vol. 80, Issue 1 https://doi.org/10.1140/epjc/s10052-019-7499-4	journal	January 2020
Measurements of WH and ZH production in the $$H \rightarrow b\bar{b}$$ decay channel in pp collisions at $$13\,\text {Te}\text {V}$$ with the ATLAS detector Aad, G.; Abbott, B.; Abbott, D. C. The European Physical Journal C, Vol. 81, Issue 2 https://doi.org/10.1140/epjc/s10052-020-08677-2	journal	February 2021
Mathematical Methods of Organizing and Planning Production Kantorovich, L. V. Management Science, Vol. 6, Issue 4 https://doi.org/10.1287/mnsc.6.4.366	journal	July 1960
Symmetries, safety, and self-supervision Dillon, Barry; Kasieczka, Gregor; Olischlager, Hans SciPost Physics, Vol. 12, Issue 6 https://doi.org/10.21468/SciPostPhys.12.6.188	journal	June 2022
Event generators for high-energy physics experiments Campbell, J. M.; Diefenthaler, M.; Hobbs, T. J. SciPost Physics, Vol. 16, Issue 5 https://doi.org/10.21468/SciPostPhys.16.5.130	journal	May 2024

Similar Records

Data compression and inference in cosmology with self-supervised machine learning

Journal Article · Sun Nov 26 23:00:00 EST 2023 · Monthly Notices of the Royal Astronomical Society · OSTI ID:2578898

Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models

Journal Article · Wed Jul 17 00:00:00 EDT 2024 · Machine Learning: Science and Technology · OSTI ID:2403614

Self-supervised Representation Learning for Astronomical Images

Journal Article · Mon Apr 26 00:00:00 EDT 2021 · The Astrophysical Journal. Letters · OSTI ID:1813371

Related Subjects

Artificial intelligence
Artificial neural networks
Stochastic processes & statistics

Resimulation-based self-supervised learning for pretraining physics foundation models

Citation Formats

References (33)

Similar Records

Related Subjects