skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities

Abstract

As the amount of available data increases, the human ability to locate, process, and analyze it is strained and eventually overwhelmed. To address this challenge for nonproliferation analysts, the authors have been creating a large-scale multimodal retrieval system to help analysts triage and search open source science, technology, and news data for indicators of nuclear proliferation capabilities and activities. The system relies on a set of deep neural networks (DNNs) trained to evaluate conceptual similarities across data modalities, such as text, image, and video. These DNNs can be used to search and prioritize data, according to a nuclear fuel cycle (NFC) process template, that are conceptually closest to the seed query items regardless of data modality. The DNNs have been trained to map conceptually related words, sentences, and images to nearby points in a multimodal feature space, enabling intra- and intermodal retrieval via nearest-neighbor calculations to seed query points. The authors evaluate the system’s ability to retrieve NFC-related data that have been purposely hidden in collections of unrelated background data. They demonstrate quantitative and qualitative results for text-to-image, image-to-image, and image-to-video retrievals. This paper discusses data challenges confronting nonproliferation analysts, describes the DNNs designed to retrieve multimodal data that aremore » proximal in the semantic space, and demonstrates their effectiveness by applying the DNNs to nonproliferation-specific multimodal data sets.« less

Authors:
 [1];  [1];  [1];  [1];  [1]
  1. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Publication Date:
Research Org.:
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1488807
Report Number(s):
LLNL-JRNL-751737
Journal ID: ISSN 0893-6188; 937574
Grant/Contract Number:  
AC52-07NA27344
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Journal of Nuclear Materials Management
Additional Journal Information:
Journal Volume: 46; Journal Issue: 3; Journal ID: ISSN 0893-6188
Publisher:
Institute of Nuclear Materials Management
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Feldman, Yana, Arno, Margaret, Carrano, Carmen, Ng, Brenda, and Chen, Barry. Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities. United States: N. p., 2018. Web.
Feldman, Yana, Arno, Margaret, Carrano, Carmen, Ng, Brenda, & Chen, Barry. Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities. United States.
Feldman, Yana, Arno, Margaret, Carrano, Carmen, Ng, Brenda, and Chen, Barry. Mon . "Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities". United States.
@article{osti_1488807,
title = {Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities},
author = {Feldman, Yana and Arno, Margaret and Carrano, Carmen and Ng, Brenda and Chen, Barry},
abstractNote = {As the amount of available data increases, the human ability to locate, process, and analyze it is strained and eventually overwhelmed. To address this challenge for nonproliferation analysts, the authors have been creating a large-scale multimodal retrieval system to help analysts triage and search open source science, technology, and news data for indicators of nuclear proliferation capabilities and activities. The system relies on a set of deep neural networks (DNNs) trained to evaluate conceptual similarities across data modalities, such as text, image, and video. These DNNs can be used to search and prioritize data, according to a nuclear fuel cycle (NFC) process template, that are conceptually closest to the seed query items regardless of data modality. The DNNs have been trained to map conceptually related words, sentences, and images to nearby points in a multimodal feature space, enabling intra- and intermodal retrieval via nearest-neighbor calculations to seed query points. The authors evaluate the system’s ability to retrieve NFC-related data that have been purposely hidden in collections of unrelated background data. They demonstrate quantitative and qualitative results for text-to-image, image-to-image, and image-to-video retrievals. This paper discusses data challenges confronting nonproliferation analysts, describes the DNNs designed to retrieve multimodal data that are proximal in the semantic space, and demonstrates their effectiveness by applying the DNNs to nonproliferation-specific multimodal data sets.},
doi = {},
journal = {Journal of Nuclear Materials Management},
issn = {0893-6188},
number = 3,
volume = 46,
place = {United States},
year = {2018},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on November 5, 2019
Publisher's Version of Record
The DOI is not currently available

Save / Share: