skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer

Abstract

Here, we present a scalable object detection workflow for detecting objects, such as settlements, from remotely sensed (RS) imagery. We have successfully deployed this workflow on Titan supercomputer and utilized it for the task of mapping human settlement at a country scale. The performance of various stages in the workflow was analyzed before making it operational. The workflow implemented various strategies to address issues such as suboptimal resource utilization and long-tail effects due to unbalanced image workload, data loss due to runtime failures, and maximum wall-time constraints imposed by Titan's job scheduling policy. A mean shift clustering–based static load balancing strategy was implemented, which partitions the image load such that each partition contained similar-sized images. Furthermore, a checkpoint-restart strategy was added in the workflow as a fault-tolerance mechanism to prevent the data losses due to unforeseen runtime failures. The performance of the above-mentioned strategies was observed in various scenarios, such as node failure, exceeding wall time, and successful completion. Using this workflow, we have examined an RS data set that has a spatial resolution of 0.31 m and is comprised of 685 675 km 2 of area of the Republic of Zambia in under six hours using 5426 nodes ofmore » the Titan supercomputer.« less

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1];  [1]; ORCiD logo [1]
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22). Scientific User Facilities Division
OSTI Identifier:
1511944
Alternate Identifier(s):
OSTI ID: 1511755
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Concurrency and Computation. Practice and Experience
Additional Journal Information:
Journal Volume: 31; Journal Issue: 20; Journal ID: ISSN 1532-0626
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; convolutional neural network; deep learning; fault tolerance; HPC; human settlement mapping; load balancing

Citation Formats

Kurte, Kuldeep, Sanyal, Jibonananda, Berres, Anne, Lunga, Dalton, Coletti, Mark, Yang, Hsiuhan Lexie, Graves, Daniel, Liebersohn, Benjamin, and Rose, Amy. Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer. United States: N. p., 2019. Web. doi:10.1002/cpe.5305.
Kurte, Kuldeep, Sanyal, Jibonananda, Berres, Anne, Lunga, Dalton, Coletti, Mark, Yang, Hsiuhan Lexie, Graves, Daniel, Liebersohn, Benjamin, & Rose, Amy. Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer. United States. doi:10.1002/cpe.5305.
Kurte, Kuldeep, Sanyal, Jibonananda, Berres, Anne, Lunga, Dalton, Coletti, Mark, Yang, Hsiuhan Lexie, Graves, Daniel, Liebersohn, Benjamin, and Rose, Amy. Wed . "Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer". United States. doi:10.1002/cpe.5305. https://www.osti.gov/servlets/purl/1511944.
@article{osti_1511944,
title = {Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer},
author = {Kurte, Kuldeep and Sanyal, Jibonananda and Berres, Anne and Lunga, Dalton and Coletti, Mark and Yang, Hsiuhan Lexie and Graves, Daniel and Liebersohn, Benjamin and Rose, Amy},
abstractNote = {Here, we present a scalable object detection workflow for detecting objects, such as settlements, from remotely sensed (RS) imagery. We have successfully deployed this workflow on Titan supercomputer and utilized it for the task of mapping human settlement at a country scale. The performance of various stages in the workflow was analyzed before making it operational. The workflow implemented various strategies to address issues such as suboptimal resource utilization and long-tail effects due to unbalanced image workload, data loss due to runtime failures, and maximum wall-time constraints imposed by Titan's job scheduling policy. A mean shift clustering–based static load balancing strategy was implemented, which partitions the image load such that each partition contained similar-sized images. Furthermore, a checkpoint-restart strategy was added in the workflow as a fault-tolerance mechanism to prevent the data losses due to unforeseen runtime failures. The performance of the above-mentioned strategies was observed in various scenarios, such as node failure, exceeding wall time, and successful completion. Using this workflow, we have examined an RS data set that has a spatial resolution of 0.31 m and is comprised of 685 675 km2 of area of the Republic of Zambia in under six hours using 5426 nodes of the Titan supercomputer.},
doi = {10.1002/cpe.5305},
journal = {Concurrency and Computation. Practice and Experience},
issn = {1532-0626},
number = 20,
volume = 31,
place = {United States},
year = {2019},
month = {5}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Parallel computation of PDFs on big spatial data using Spark
journal, February 2019


DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images
conference, June 2018

  • Demir, Ilke; Koperski, Krzysztof; Lindenbaum, David
  • 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • DOI: 10.1109/CVPRW.2018.00031

Simulation and big data challenges in tuning building energy models
conference, May 2013

  • Sanyal, Jibonananda; New, Joshua
  • 2013 Workshop on Modeling and Simulation of Cyber-Physical Energy Systems (MSCPES)
  • DOI: 10.1109/MSCPES.2013.6623320

Evolving Deep Networks Using HPC
conference, January 2017

  • Young, Steven R.; Rose, Derek C.; Johnston, Travis
  • Proceedings of the Machine Learning on HPC Environments - MLHPC'17
  • DOI: 10.1145/3146347.3146355

Semantics and High Performance Computing Driven Approaches for Enhanced Exploitation of Earth Observation (EO) Data: State of the Art
journal, November 2017

  • Durbha, Surya S.; Kurte, Kuldeep R.; Bhangale, Ujwala
  • Proceedings of the National Academy of Sciences, India Section A: Physical Sciences, Vol. 87, Issue 4
  • DOI: 10.1007/s40010-017-0432-z

Scheduling many-task workloads on supercomputers: Dealing with trailing tasks
conference, November 2010

  • Armstrong, Timothy G.; Zhang, Zhao; Katz, Daniel S.
  • 2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS)
  • DOI: 10.1109/MTAGS.2010.5699433

Parallelizing maximum likelihood classification on computer cluster and graphics processing unit for supervised image classification
journal, November 2016


A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers
journal, July 2018

  • Potok, Thomas E.; Schuman, Catherine; Young, Steven
  • ACM Journal on Emerging Technologies in Computing Systems, Vol. 14, Issue 2
  • DOI: 10.1145/3178454

High Performance Computing for Hyperspectral Remote Sensing
journal, September 2011

  • Plaza, Antonio; Du, Qian; Chang, Yang-Lang
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 4, Issue 3
  • DOI: 10.1109/JSTARS.2010.2095495

Exploiting Different Types of Parallelism in Distributed Analysis of Remote Sensing Data
journal, August 2017

  • Costa, Gilson A. O. P.; Bentes, Cristiana; Ferreira, Rodrigo S.
  • IEEE Geoscience and Remote Sensing Letters, Vol. 14, Issue 8
  • DOI: 10.1109/LGRS.2017.2709700

FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters
conference, June 2016

  • Iandola, Forrest N.; Moskewicz, Matthew W.; Ashraf, Khalid
  • 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • DOI: 10.1109/CVPR.2016.284

Development of a graph-based approach for building detection
journal, January 1999


A first order approximation to the optimum checkpoint interval
journal, September 1974


Automatic Rooftop Extraction in Nadir Aerial Imagery of Suburban Regions Using Corners and Variational Level Set Evolution
journal, January 2013

  • Cote, Melissa; Saeedi, Parvaneh
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 51, Issue 1
  • DOI: 10.1109/TGRS.2012.2200689

Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery
journal, September 2018


Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters
journal, January 2018

  • Xu, Yongyang; Wu, Liang; Xie, Zhong
  • Remote Sensing, Vol. 10, Issue 1
  • DOI: 10.3390/rs10010144

Automatic building footprint extraction from high-resolution satellite image using mathematical morphology
journal, December 2017


A Probabilistic Framework for Building Extraction From Airborne Color Image and DSM
journal, March 2017

  • Chai, Dengfeng
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, Issue 3
  • DOI: 10.1109/JSTARS.2016.2616446

Automated detection of buildings from single VHR multispectral images using shadow information and graph cuts
journal, December 2013


A higher order estimate of the optimum checkpoint interval for restart dumps
journal, February 2006


Accelerating Big Data processing chain in Image Information Mining using a hybrid HPC approach
conference, July 2016

  • Kurte, Kuldeep R.; Bhangale, Ujwala M.; Durbha, Surya S.
  • IGARSS 2016 - 2016 IEEE International Geoscience and Remote Sensing Symposium, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
  • DOI: 10.1109/IGARSS.2016.7730981

Mean shift: a robust approach toward feature space analysis
journal, May 2002

  • Comaniciu, D.; Meer, P.
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, Issue 5
  • DOI: 10.1109/34.1000236

Multisource Remote Sensing Data Classification Based on Convolutional Neural Network
journal, February 2018

  • Xu, Xiaodong; Li, Wei; Ran, Qiong
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, Issue 2
  • DOI: 10.1109/TGRS.2017.2756851

Deep Learning Earth Observation Classification Using ImageNet Pretrained Networks
journal, January 2016

  • Marmanis, Dimitrios; Datcu, Mihai; Esch, Thomas
  • IEEE Geoscience and Remote Sensing Letters, Vol. 13, Issue 1
  • DOI: 10.1109/LGRS.2015.2499239

Supercomputer assisted generation of machine learning agents for the calibration of building energy models
conference, January 2013

  • Sanyal, Jibonananda; New, Joshua; Edwards, Richard
  • Proceedings of the Conference on Extreme Science and Engineering Discovery Environment Gateway to Discovery - XSEDE '13
  • DOI: 10.1145/2484762.2484818

An object-based convolutional neural network (OCNN) for urban land use classification
journal, October 2018


pipsCloud: High performance cloud computing for remote sensing big data management and processing
journal, January 2018


Shape-Based Building Detection in Visible Band Images Using Shadow Information
journal, March 2017

  • Ngo, Tran-Thanh; Mazet, Vincent; Collet, Christophe
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, Issue 3
  • DOI: 10.1109/JSTARS.2016.2598856

Evaluation of Automatic Building Detection Approaches Combining High Resolution Images and LiDAR Data
journal, June 2011

  • Hermosilla, Txomin; Ruiz, Luis A.; Recio, Jorge A.
  • Remote Sensing, Vol. 3, Issue 6
  • DOI: 10.3390/rs3061188

Automatic Building Segmentation of Aerial Imagery Using Multi-Constraint Fully Convolutional Networks
journal, March 2018

  • Wu, Guangming; Shao, Xiaowei; Guo, Zhiling
  • Remote Sensing, Vol. 10, Issue 3
  • DOI: 10.3390/rs10030407

Google Earth Engine: Planetary-scale geospatial analysis for everyone
journal, December 2017


Implementation of the parallel mean shift-based image segmentation algorithm on a GPU cluster
journal, January 2018


CNTK: Microsoft's Open-Source Deep-Learning Toolkit
conference, January 2016

  • Seide, Frank; Agarwal, Amit
  • Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16
  • DOI: 10.1145/2939672.2945397

Titan - Early Experience with the Titan System at Oak Ridge National Laboratory
conference, November 2012

  • Bland, Buddy
  • 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
  • DOI: 10.1109/SC.Companion.2012.356

Exploiting convolutional representations for multiscale human settlement detection: Preliminary results
conference, July 2017

  • Lunga, Dalton; Patlolla, Dilip; Yang, Hsiuhan Lexie
  • 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
  • DOI: 10.1109/IGARSS.2017.8127822