skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

This content will become publicly available on May 14, 2020

Title: Computationally Efficient Training of Deep Neural Networks via Transfer Learning

Abstract

Transfer learning is a highly successful approach to train deep neural networks for customized image recognition tasks. Transfer learning leverages deep learning models that are previously trained on massive datasets and re-trains the network for a novel image recognition dataset. Typically, the advantage of transfer learning has been measured in sample efficiency, but instead, we investigate the computational efficiency of transfer learning. A good pre-trained model provides features that can be used as input to a new classifier (usually the top layers of a neural network). We show that if a good pre-trained model is selected, that training a new classifier can be much more computationally efficient than training a deep neural network without transfer learning. The first step in transfer learning is to select a pre-trained model to use as a feed-forward network for generating features. Here this selection process either uses human intuition/convenience or a methodical but computationally intensive validation loop. Instead, we would prefer a method to select the pre-trained model that will produce the best transfer results for the least amount of computation. To this end, we provide a computationally efficient metric for the fit between a pre-trained model and a novel image recognition task. Themore » better the fit, the less computation will be needed to re-train the pre-trained model for the novel task. As machine learning becomes ubiquitous, highly-accurate trained models will proliferate and computationally efficient transfer learning methods will enable rapid development of new image recognition models.« less

Authors:
 [1]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Laboratory Directed Research and Development (LDRD) Program
OSTI Identifier:
1565903
Report Number(s):
LA-UR-19-22785
Journal ID: ISSN 0277-786X
Grant/Contract Number:  
89233218CNA000001
Resource Type:
Accepted Manuscript
Journal Name:
Proceedings of SPIE - The International Society for Optical Engineering
Additional Journal Information:
Journal Volume: 10996; Conference: SPIE DEFENSE + COMMERCIAL SENSING, Baltimore, MD (United States), 14-18 Apr 2019; Journal ID: ISSN 0277-786X
Publisher:
SPIE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Mathematics

Citation Formats

Oyen, Diane. Computationally Efficient Training of Deep Neural Networks via Transfer Learning. United States: N. p., 2019. Web. doi:10.1117/12.2519097.
Oyen, Diane. Computationally Efficient Training of Deep Neural Networks via Transfer Learning. United States. doi:10.1117/12.2519097.
Oyen, Diane. Tue . "Computationally Efficient Training of Deep Neural Networks via Transfer Learning". United States. doi:10.1117/12.2519097.
@article{osti_1565903,
title = {Computationally Efficient Training of Deep Neural Networks via Transfer Learning},
author = {Oyen, Diane},
abstractNote = {Transfer learning is a highly successful approach to train deep neural networks for customized image recognition tasks. Transfer learning leverages deep learning models that are previously trained on massive datasets and re-trains the network for a novel image recognition dataset. Typically, the advantage of transfer learning has been measured in sample efficiency, but instead, we investigate the computational efficiency of transfer learning. A good pre-trained model provides features that can be used as input to a new classifier (usually the top layers of a neural network). We show that if a good pre-trained model is selected, that training a new classifier can be much more computationally efficient than training a deep neural network without transfer learning. The first step in transfer learning is to select a pre-trained model to use as a feed-forward network for generating features. Here this selection process either uses human intuition/convenience or a methodical but computationally intensive validation loop. Instead, we would prefer a method to select the pre-trained model that will produce the best transfer results for the least amount of computation. To this end, we provide a computationally efficient metric for the fit between a pre-trained model and a novel image recognition task. The better the fit, the less computation will be needed to re-train the pre-trained model for the novel task. As machine learning becomes ubiquitous, highly-accurate trained models will proliferate and computationally efficient transfer learning methods will enable rapid development of new image recognition models.},
doi = {10.1117/12.2519097},
journal = {Proceedings of SPIE - The International Society for Optical Engineering},
number = ,
volume = 10996,
place = {United States},
year = {2019},
month = {5}
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on May 14, 2020
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning
journal, May 2016

  • Shin, Hoo-Chang; Roth, Holger R.; Gao, Mingchen
  • IEEE Transactions on Medical Imaging, Vol. 35, Issue 5
  • DOI: 10.1109/TMI.2016.2528162