Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer

Kurte, Kuldeep; Sanyal, Jibonananda; Berres, Anne; Lunga, Dalton; Coletti, Mark; Yang, Hsiuhan Lexie; Graves, Daniel; Liebersohn, Benjamin; Rose, Amy

doi:10.1002/cpe.5305

Title: Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer

Journal Article · Wed May 08 00:00:00 EDT 2019 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.5305· OSTI ID:1511944

^[1];

^[1]; Liebersohn, Benjamin ^[1];

^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Summary This paper presents a scalable object detection workflow for detecting objects, such as settlements, from remotely sensed (RS) imagery. We have successfully deployed this workflow on Titan supercomputer and utilized it for the task of mapping human settlement at a country scale. The performance of various stages in the workflow was analyzed before making it operational. The workflow implemented various strategies to address issues such as suboptimal resource utilization and long‐tail effects due to unbalanced image workload, data loss due to runtime failures, and maximum wall‐time constraints imposed by Titan's job scheduling policy. A mean shift clustering–based static load balancing strategy was implemented, which partitions the image load such that each partition contained similar‐sized images. Furthermore, a checkpoint‐restart strategy was added in the workflow as a fault‐tolerance mechanism to prevent the data losses due to unforeseen runtime failures. The performance of the above‐mentioned strategies was observed in various scenarios, such as node failure, exceeding wall time, and successful completion. Using this workflow, we have processed an RS data set that has a spatial resolution of 0.31 m and is comprised of 685 675 km ² of area of the Republic of Zambia in under six hours using 5426 nodes of the Titan supercomputer.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities Division

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1511944

Alternate ID(s):: OSTI ID: 1511755

Journal Information:: Concurrency and Computation. Practice and Experience, Vol. 31, Issue 20; ISSN 1532-0626

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 6 works

Citation information provided by
Web of Science

References (36)

Parallel computation of PDFs on big spatial data using Spark Liu, Ji; Lemus, Noel Moreno; Pacitti, Esther Distributed and Parallel Databases https://doi.org/10.1007/s10619-019-07260-3	journal	February 2019
DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images Demir, Ilke; Koperski, Krzysztof; Lindenbaum, David 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) https://doi.org/10.1109/CVPRW.2018.00031	conference	June 2018
Simulation and big data challenges in tuning building energy models Sanyal, Jibonananda; New, Joshua 2013 Workshop on Modeling and Simulation of Cyber-Physical Energy Systems (MSCPES) https://doi.org/10.1109/MSCPES.2013.6623320	conference	May 2013
Evolving Deep Networks Using HPC Young, Steven R.; Rose, Derek C.; Johnston, Travis Proceedings of the Machine Learning on HPC Environments - MLHPC'17 https://doi.org/10.1145/3146347.3146355	conference	January 2017
Semantics and High Performance Computing Driven Approaches for Enhanced Exploitation of Earth Observation (EO) Data: State of the Art Durbha, Surya S.; Kurte, Kuldeep R.; Bhangale, Ujwala Proceedings of the National Academy of Sciences, India Section A: Physical Sciences, Vol. 87, Issue 4 https://doi.org/10.1007/s40010-017-0432-z	journal	November 2017
Scheduling many-task workloads on supercomputers: Dealing with trailing tasks Armstrong, Timothy G.; Zhang, Zhao; Katz, Daniel S. 2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) https://doi.org/10.1109/MTAGS.2010.5699433	conference	November 2010
Parallelizing maximum likelihood classification on computer cluster and graphics processing unit for supervised image classification Shi, Xuan; Xue, Bowei International Journal of Digital Earth, Vol. 10, Issue 7 https://doi.org/10.1080/17538947.2016.1251502	journal	November 2016
A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers Potok, Thomas E.; Schuman, Catherine; Young, Steven ACM Journal on Emerging Technologies in Computing Systems, Vol. 14, Issue 2 https://doi.org/10.1145/3178454	journal	July 2018
High Performance Computing for Hyperspectral Remote Sensing Plaza, Antonio; Du, Qian; Chang, Yang-Lang IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 4, Issue 3 https://doi.org/10.1109/JSTARS.2010.2095495	journal	September 2011
Exploiting Different Types of Parallelism in Distributed Analysis of Remote Sensing Data Costa, Gilson A. O. P.; Bentes, Cristiana; Ferreira, Rodrigo S. IEEE Geoscience and Remote Sensing Letters, Vol. 14, Issue 8 https://doi.org/10.1109/LGRS.2017.2709700	journal	August 2017
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters Iandola, Forrest N.; Moskewicz, Matthew W.; Ashraf, Khalid 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.284	conference	June 2016
Development of a graph-based approach for building detection Kim, Taejung; Muller, Jan-Peter Image and Vision Computing, Vol. 17, Issue 1 https://doi.org/10.1016/S0262-8856(98)00092-4	journal	January 1999
A first order approximation to the optimum checkpoint interval Young, John W. Communications of the ACM, Vol. 17, Issue 9 https://doi.org/10.1145/361147.361115	journal	September 1974
Automatic Rooftop Extraction in Nadir Aerial Imagery of Suburban Regions Using Corners and Variational Level Set Evolution Cote, Melissa; Saeedi, Parvaneh IEEE Transactions on Geoscience and Remote Sensing, Vol. 51, Issue 1 https://doi.org/10.1109/TGRS.2012.2200689	journal	January 2013
Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery Huang, Bo; Zhao, Bei; Song, Yimeng Remote Sensing of Environment, Vol. 214 https://doi.org/10.1016/j.rse.2018.04.050	journal	September 2018
Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters Xu, Yongyang; Wu, Liang; Xie, Zhong Remote Sensing, Vol. 10, Issue 1 https://doi.org/10.3390/rs10010144	journal	January 2018
Automatic building footprint extraction from high-resolution satellite image using mathematical morphology Gavankar, Nitin L.; Ghosh, Sanjay Kumar European Journal of Remote Sensing, Vol. 51, Issue 1 https://doi.org/10.1080/22797254.2017.1416676	journal	December 2017
A Probabilistic Framework for Building Extraction From Airborne Color Image and DSM Chai, Dengfeng IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, Issue 3 https://doi.org/10.1109/JSTARS.2016.2616446	journal	March 2017
Automated detection of buildings from single VHR multispectral images using shadow information and graph cuts Ok, Ali Ozgun ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 86 https://doi.org/10.1016/j.isprsjprs.2013.09.004	journal	December 2013
A higher order estimate of the optimum checkpoint interval for restart dumps Daly, J. T. Future Generation Computer Systems, Vol. 22, Issue 3, p. 303-312 https://doi.org/10.1016/j.future.2004.11.016	journal	February 2006
Accelerating Big Data processing chain in Image Information Mining using a hybrid HPC approach Kurte, Kuldeep R.; Bhangale, Ujwala M.; Durbha, Surya S. IGARSS 2016 - 2016 IEEE International Geoscience and Remote Sensing Symposium, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) https://doi.org/10.1109/IGARSS.2016.7730981	conference	July 2016
Mean shift: a robust approach toward feature space analysis Comaniciu, D.; Meer, P. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, Issue 5 https://doi.org/10.1109/34.1000236	journal	May 2002
Multisource Remote Sensing Data Classification Based on Convolutional Neural Network Xu, Xiaodong; Li, Wei; Ran, Qiong IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, Issue 2 https://doi.org/10.1109/TGRS.2017.2756851	journal	February 2018
Deep Learning Earth Observation Classification Using ImageNet Pretrained Networks Marmanis, Dimitrios; Datcu, Mihai; Esch, Thomas IEEE Geoscience and Remote Sensing Letters, Vol. 13, Issue 1 https://doi.org/10.1109/LGRS.2015.2499239	journal	January 2016
Supercomputer assisted generation of machine learning agents for the calibration of building energy models Sanyal, Jibonananda; New, Joshua; Edwards, Richard Proceedings of the Conference on Extreme Science and Engineering Discovery Environment Gateway to Discovery - XSEDE '13 https://doi.org/10.1145/2484762.2484818	conference	January 2013
An object-based convolutional neural network (OCNN) for urban land use classification Zhang, Ce; Sargent, Isabel; Pan, Xin Remote Sensing of Environment, Vol. 216 https://doi.org/10.1016/j.rse.2018.06.034	journal	October 2018
pipsCloud: High performance cloud computing for remote sensing big data management and processing Wang, Lizhe; Ma, Yan; Yan, Jining Future Generation Computer Systems, Vol. 78 https://doi.org/10.1016/j.future.2016.06.009	journal	January 2018
Shape-Based Building Detection in Visible Band Images Using Shadow Information Ngo, Tran-Thanh; Mazet, Vincent; Collet, Christophe IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, Issue 3 https://doi.org/10.1109/JSTARS.2016.2598856	journal	March 2017
Evaluation of Automatic Building Detection Approaches Combining High Resolution Images and LiDAR Data Hermosilla, Txomin; Ruiz, Luis A.; Recio, Jorge A. Remote Sensing, Vol. 3, Issue 6 https://doi.org/10.3390/rs3061188	journal	June 2011
Automatic Building Segmentation of Aerial Imagery Using Multi-Constraint Fully Convolutional Networks Wu, Guangming; Shao, Xiaowei; Guo, Zhiling Remote Sensing, Vol. 10, Issue 3 https://doi.org/10.3390/rs10030407	journal	March 2018
Google Earth Engine: Planetary-scale geospatial analysis for everyone Gorelick, Noel; Hancher, Matt; Dixon, Mike Remote Sensing of Environment, Vol. 202 https://doi.org/10.1016/j.rse.2017.06.031	journal	December 2017
Implementation of the parallel mean shift-based image segmentation algorithm on a GPU cluster Huang, Fang; Chen, Yinjie; Li, Li International Journal of Digital Earth, Vol. 12, Issue 3 https://doi.org/10.1080/17538947.2018.1432709	journal	January 2018
CNTK: Microsoft's Open-Source Deep-Learning Toolkit Seide, Frank; Agarwal, Amit Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16 https://doi.org/10.1145/2939672.2945397	conference	January 2016
Titan - Early Experience with the Titan System at Oak Ridge National Laboratory Bland, Buddy 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis https://doi.org/10.1109/SC.Companion.2012.356	conference	November 2012
Exploiting convolutional representations for multiscale human settlement detection: Preliminary results Lunga, Dalton; Patlolla, Dilip; Yang, Hsiuhan Lexie 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) https://doi.org/10.1109/IGARSS.2017.8127822	conference	July 2017
Ecosystem state change in the Arabian Sea fuelled by the recent loss of snow over the Himalayan-Tibetan Plateau region Goes, Joaquim I.; Tian, Hongzhen; Gomes, Helga do Rosario Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-020-64360-2	journal	May 2020

Similar Records

Resiliency in numerical algorithm design for extreme scale simulations

Journal Article · Fri Dec 10 00:00:00 EST 2021 · International Journal of High Performance Computing Applications · OSTI ID:1511944

Agullo, Emmanuel; Altenbernd, Mirco; Anzt, Hartwig; +33 more

Towards Low-Overhead Resilience for Data Parallel Deep Learning

Conference · Sat Jan 01 00:00:00 EST 2022 · OSTI ID:1511944

Nicolae, Bogdan; Hobson, Tanner; Yildiz, Orcun; +2 more

Analyzing the Interplay of Failures and Workload on a Leadership-Class Supercomputer

Conference · Thu Jan 01 00:00:00 EST 2015 · OSTI ID:1511944

Meneses, Esteban; Ni, Xiang; Jones, Terry R; +1 more

Related Subjects

97 MATHEMATICS AND COMPUTING
convolutional neural network
deep learning
fault tolerance
HPC
human settlement mapping
load balancing

Title: Performance analysis and optimization for scalable deployment of deep learning models for country-scale settlement mapping on Titan supercomputer

Citation Formats

References (36)

Similar Records

Related Subjects