Coarse-to-fine Task-driven Inpainting for Geoscience Images

Sun, Huiming; Ma, Jin; Guo, Qing; Zou, Qin; Song, Shaoyue; Lin, Yuewei; Yu, Hongkai

doi:10.1109/tcsvt.2023.3276719

Coarse-to-fine Task-driven Inpainting for Geoscience Images

Journal Article · Tue May 16 00:00:00 EDT 2023 · IEEE Transactions on Circuits and Systems for Video Technology

DOI:https://doi.org/10.1109/tcsvt.2023.3276719· OSTI ID:1993157

^[1]; Ma, Jin ^[1]; Guo, Qing ^[2]; ^[3]; ^[4]; Lin, Yuewei ^[5]; ^[1]

Cleveland State Univ., Cleveland, OH (United States)
Agency for Science, Technology and Research (A*STAR), Singapore. Centre for Frontier AI Research (CFAR)
Wuhan Univ. (China)
Beijing University of Technology (China)
Brookhaven National Laboratory (BNL), Upton, NY (United States)

The processing and recognition of geoscience images have wide applications. Most of existing researches focus on understanding the high-quality geoscience images by assuming that all the images are clear. However, in many real-world cases, the geoscience images might contain occlusions during the image acquisition. This problem actually implies the image inpainting problem in computer vision and multimedia. As far as we know, all the existing image inpainting algorithms learn to repair the occluded regions for a better visualization quality, they are excellent for natural images but not good enough for geoscience images, and they never consider the following geoscience task when developing inpainting methods. Here, this paper aims to repair the occluded regions for a better geoscience task performance and advanced visualization quality simultaneously, without changing the current deployed deep learning based geoscience models. Because of the complex context of geoscience images, we propose a coarse-to-fine encoder-decoder network with the help of designed coarse-to-fine adversarial context discriminators to reconstruct the occluded image regions. Due to the limited data of geoscience images, we propose a MaskMix based data augmentation method, which augments inpainting masks instead of augmenting original images, to exploit the limited geoscience image data. The experimental results on three public geoscience datasets for remote sensing scene recognition, cross-view geolocation and semantic segmentation tasks respectively show the effectiveness and accuracy of the proposed method. The code is available at: https://github.com/HMS97/Task-driven-Inpainting.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0012704

OSTI ID:: 1993157

Report Number(s):: BNL-224638-2023-JAAM

Journal Information:: IEEE Transactions on Circuits and Systems for Video Technology, Journal Name: IEEE Transactions on Circuits and Systems for Video Technology Journal Issue: 12 Vol. 33; ISSN 1051-8215

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (62)

Image Inpainting for Irregular Holes Using Partial Convolutions Liu, Guilin; Reda, Fitsum A.; Shih, Kevin J. Computer Vision – ECCV 2018 https://doi.org/10.1007/978-3-030-01252-6_6	book	January 2018
VCNet: A Robust Approach to Blind Image Inpainting Wang, Yi; Chen, Ying-Cong; Tao, Xin Computer Vision – ECCV 2020 https://doi.org/10.1007/978-3-030-58595-2_45	book	January 2020
Automatic pancreas segmentation based on lightweight DCNN modules and spatial prior propagation Zhang, Dingwen; Zhang, Jiajia; Zhang, Qiang Pattern Recognition, Vol. 114 https://doi.org/10.1016/j.patcog.2020.107762	journal	June 2021
Context Encoders: Feature Learning by Inpainting Pathak, Deepak; Krahenbuhl, Philipp; Donahue, Jeff 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.278	conference	June 2016
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Image-to-Image Translation with Conditional Adversarial Networks Isola, Phillip; Zhu, Jun-Yan; Zhou, Tinghui 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.632	conference	July 2017
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric Zhang, Richard; Isola, Phillip; Efros, Alexei A. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00068	conference	June 2018
DOTA: A Large-Scale Dataset for Object Detection in Aerial Images Xia, Gui-Song; Bai, Xiang; Ding, Jian 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00418	conference	June 2018
Generative Image Inpainting with Contextual Attention Yu, Jiahui; Lin, Zhe; Yang, Jimei 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2018.00577	conference	June 2018
Learning RoI Transformer for Oriented Object Detection in Aerial Images Ding, Jian; Xue, Nan; Long, Yang 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00296	conference	June 2019
BASNet: Boundary-Aware Salient Object Detection Qin, Xuebin; Zhang, Zichen; Huang, Chenyang 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00766	conference	June 2019
Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting Yi, Zili; Tang, Qiang; Azizi, Shekoofeh 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00753	conference	June 2020
Recurrent Feature Reasoning for Image Inpainting Li, Jingyuan; Wang, Ning; Zhang, Lefei 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00778	conference	June 2020
PD-GAN: Probabilistic Diverse GAN for Image Inpainting Liu, Hongyu; Wan, Ziyu; Huang, Wei 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR46437.2021.00925	conference	June 2021
MAT: Mask-Aware Transformer for Large Hole Image Inpainting Li, Wenbo; Lin, Zhe; Zhou, Kun 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.01049	conference	June 2022
Coherent Semantic Attention for Image Inpainting Liu, Hongyu; Jiang, Bin; Xiao, Yi 2019 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2019.00427	conference	October 2019
Dense-Add Net: An Novel Convolutional Neural Network for Remote Sensing Image Inpainting Lin, Daoyu; Xu, Guangluan; Wang, Yang IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium https://doi.org/10.1109/IGARSS.2018.8519156	conference	July 2018
Remote Sensing Image Inpainting with Generative Adversarial Networks Kuznetsov, Andrey; Gashnikov, Mikhail 2020 8th International Symposium on Digital Forensics and Security (ISDFS) https://doi.org/10.1109/ISDFS49300.2020.9116347	conference	June 2020
Remote Sensing Image Scene Classification: Benchmark and State of the Art Cheng, Gong; Han, Junwei; Lu, Xiaoqiang Proceedings of the IEEE, Vol. 105, Issue 10 https://doi.org/10.1109/JPROC.2017.2675998	journal	October 2017
Multilevel Cloud Detection in Remote Sensing Images Based on Deep Learning Xie, Fengying; Shi, Mengyun; Shi, Zhenwei IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, Issue 8 https://doi.org/10.1109/JSTARS.2017.2686488	journal	August 2017
Deep Learning Based Feature Selection for Remote Sensing Scene Classification Zou, Qin; Ni, Lihao; Zhang, Tong IEEE Geoscience and Remote Sensing Letters, Vol. 12, Issue 11 https://doi.org/10.1109/LGRS.2015.2475299	journal	November 2015
Inpainting of Remote Sensing SST Images With Deep Convolutional Generative Adversarial Network Dong, Junyu; Yin, Ruiying; Sun, Xin IEEE Geoscience and Remote Sensing Letters, Vol. 16, Issue 2 https://doi.org/10.1109/LGRS.2018.2870880	journal	February 2019
Domain Adaptation for Convolutional Neural Networks-Based Remote Sensing Scene Classification Song, Shaoyue; Yu, Hongkai; Miao, Zhenjiang IEEE Geoscience and Remote Sensing Letters, Vol. 16, Issue 8 https://doi.org/10.1109/LGRS.2019.2896411	journal	August 2019
Denoising-Based Multiscale Feature Fusion for Remote Sensing Image Captioning Huang, Wei; Wang, Qi; Li, Xuelong IEEE Geoscience and Remote Sensing Letters, Vol. 18, Issue 3 https://doi.org/10.1109/LGRS.2020.2980933	journal	March 2021
Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging Techniques Hnewa, Mazin; Radha, Hayder IEEE Signal Processing Magazine, Vol. 38, Issue 1 https://doi.org/10.1109/MSP.2020.2984801	journal	January 2021
Patch-Sparsity-Based Image Inpainting Through a Facet Deduced Directional Derivative Jin, Darui; Bai, Xiangzhi IEEE Transactions on Circuits and Systems for Video Technology, Vol. 29, Issue 5 https://doi.org/10.1109/TCSVT.2018.2839351	journal	May 2019
Low Rank Component Induced Spatial-Spectral Kernel Method for Hyperspectral Image Classification Sun, Le; Ma, Chenyang; Chen, Yunjie IEEE Transactions on Circuits and Systems for Video Technology, Vol. 30, Issue 10 https://doi.org/10.1109/TCSVT.2019.2946723	journal	October 2020
Multiscale Densely-Connected Fusion Networks for Hyperspectral Images Classification Xie, Jie; He, Nanjun; Fang, Leyuan IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, Issue 1 https://doi.org/10.1109/TCSVT.2020.2975566	journal	January 2021
E2I: Generative Inpainting From Edge to Image Xu, Shunxin; Liu, Dong; Xiong, Zhiwei IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, Issue 4 https://doi.org/10.1109/TCSVT.2020.3001267	journal	April 2021
Structure-Guided Deep Video Inpainting Wang, Chaoqun; Chen, Xuejin; Min, Shaobo IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, Issue 8 https://doi.org/10.1109/TCSVT.2020.3034422	journal	August 2021
Each Part Matters: Local Patterns Facilitate Cross-View Geo-Localization Wang, Tingyu; Zheng, Zhedong; Yan, Chenggang IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, Issue 2 https://doi.org/10.1109/TCSVT.2021.3061265	journal	February 2022
IID-Net: Image Inpainting Detection Network via Neural Architecture Search and Attention Wu, Haiwei; Zhou, Jiantao IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, Issue 3 https://doi.org/10.1109/TCSVT.2021.3075039	journal	March 2022
Global-Local Balanced Low-Rank Approximation of Hyperspectral Images for Classification Liu, Hui; Jia, Yuheng; Hou, Junhui IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, Issue 4 https://doi.org/10.1109/TCSVT.2021.3095250	journal	April 2022
Dual-Pyramidal Image Inpainting With Dynamic Normalization Wang, Chao; Shao, Mingwen; Meng, Deyu IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, Issue 9 https://doi.org/10.1109/TCSVT.2022.3165587	journal	September 2022
Deep Image Inpainting With Enhanced Normalization and Contextual Attention Liu, Jialu; Gong, Maoguo; Tang, Zedong IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, Issue 10 https://doi.org/10.1109/TCSVT.2022.3175171	journal	October 2022
Reference-Guided Landmark Image Inpainting With Deep Feature Matching Li, Jiacheng; Xiong, Zhiwei; Liu, Dong IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, Issue 12 https://doi.org/10.1109/TCSVT.2022.3193893	journal	December 2022
A MAP-Based Algorithm for Destriping and Inpainting of Remotely Sensed Images No authors listed IEEE Transactions on Geoscience and Remote Sensing, Vol. 47, Issue 5 https://doi.org/10.1109/TGRS.2008.2005780	journal	May 2009
Inpainting for Remotely Sensed Images With a Multichannel Nonlocal Total Variation Model Cheng, Qing; Shen, Huanfeng; Zhang, Liangpei IEEE Transactions on Geoscience and Remote Sensing, Vol. 52, Issue 1 https://doi.org/10.1109/TGRS.2012.2237521	journal	January 2014
Unsupervised Deep Slow Feature Analysis for Change Detection in Multi-Temporal Remote Sensing Images Du, Bo; Ru, Lixiang; Wu, Chen IEEE Transactions on Geoscience and Remote Sensing, Vol. 57, Issue 12 https://doi.org/10.1109/TGRS.2019.2930682	journal	December 2019
Truncation Cross Entropy Loss for Remote Sensing Image Captioning Li, Xuelong; Zhang, Xueting; Huang, Wei IEEE Transactions on Geoscience and Remote Sensing, Vol. 59, Issue 6 https://doi.org/10.1109/TGRS.2020.3010106	journal	June 2021
AFNet: Adaptive Fusion Network for Remote Sensing Image Semantic Segmentation Liu, Rui; Mi, Li; Chen, Zhenzhong IEEE Transactions on Geoscience and Remote Sensing, Vol. 59, Issue 9 https://doi.org/10.1109/TGRS.2020.3034123	journal	September 2021
VD-GAN: A Unified Framework for Joint Prototype and Representation Learning From Contaminated Single Sample per Person Pang, Meng; Wang, Binghui; Cheung, Yiu-ming IEEE Transactions on Information Forensics and Security, Vol. 16 https://doi.org/10.1109/TIFS.2021.3050055	journal	January 2021
An adaptive semiparametric and context-based approach to unsupervised change detection in multitemporal remote-sensing images Bruzzone, L.; Prieto, D. F. IEEE Transactions on Image Processing, Vol. 11, Issue 4 https://doi.org/10.1109/TIP.2002.999678	journal	April 2002
Image Quality Assessment: From Error Visibility to Structural Similarity Wang, Z.; Bovik, A. C.; Sheikh, H. R. IEEE Transactions on Image Processing, Vol. 13, Issue 4 https://doi.org/10.1109/TIP.2003.819861	journal	April 2004
No-Reference Image Quality Assessment in the Spatial Domain Mittal, A.; Moorthy, A. K.; Bovik, A. C. IEEE Transactions on Image Processing, Vol. 21, Issue 12 https://doi.org/10.1109/TIP.2012.2214050	journal	December 2012
Polarimetric SAR Image Semantic Segmentation With 3D Discrete Wavelet Transform and Markov Random Field Bi, Haixia; Xu, Lin; Cao, Xiangyong IEEE Transactions on Image Processing, Vol. 29 https://doi.org/10.1109/TIP.2020.2992177	journal	January 2020
X-Invariant Contrastive Augmentation and Representation Learning for Semi-Supervised Skeleton-Based Action Recognition Xu, Binqian; Shu, Xiangbo; Song, Yan IEEE Transactions on Image Processing, Vol. 31 https://doi.org/10.1109/TIP.2022.3175605	journal	January 2022
Salient Band Selection for Hyperspectral Image Classification via Manifold Ranking Wang, Qi; Lin, Jianzhe; Yuan, Yuan IEEE Transactions on Neural Networks and Learning Systems, Vol. 27, Issue 6 https://doi.org/10.1109/TNNLS.2015.2477537	journal	June 2016
DisP+V: A Unified Framework for Disentangling Prototype and Variation From Single Sample per Person Pang, Meng; Wang, Binghui; Ye, Mang IEEE Transactions on Neural Networks and Learning Systems, Vol. 34, Issue 2 https://doi.org/10.1109/TNNLS.2021.3103194	journal	February 2023
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation Badrinarayanan, Vijay; Kendall, Alex; Cipolla, Roberto IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12 https://doi.org/10.1109/TPAMI.2016.2644615	journal	December 2017
Places: A 10 Million Image Database for Scene Recognition Zhou, Bolei; Lapedriza, Agata; Khosla, Aditya IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 40, Issue 6 https://doi.org/10.1109/TPAMI.2017.2723009	journal	June 2018
Effects of Image Degradation and Degradation Removal to CNN-Based Image Classification Pei, Yanting; Huang, Yaping; Zou, Qi IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, Issue 4 https://doi.org/10.1109/TPAMI.2019.2950923	journal	April 2021
Deep High-Resolution Representation Learning for Visual Recognition Wang, Jingdong; Sun, Ke; Cheng, Tianheng IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, Issue 10 https://doi.org/10.1109/TPAMI.2020.2983686	journal	October 2021
Evaluation of Image Inpainting for Classification and Retrieval Black, Samuel; Keshavarz, Somayeh; Souvenir, Richard 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV45572.2020.9093362	conference	March 2020
Cross-View Image Matching for Geo-Localization in Urban Environments Tian, Yicong; Chen, Chen; Shah, Mubarak 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/cvpr.2017.216	conference	July 2017
Predicting Ground-Level Scene Layout from Aerial Imagery Zhai, Menghua; Bessinger, Zachary; Workman, Scott 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/cvpr.2017.440	conference	July 2017
MISF:Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting Li, Xiaoguang; Guo, Qing; Lin, Di 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/cvpr52688.2022.00191	conference	June 2022
JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting Guo, Qing; Li, Xiaoguang; Juefei-Xu, Felix Proceedings of the 29th ACM International Conference on Multimedia https://doi.org/10.1145/3474085.3475170	conference	October 2021
R³Net: Recurrent Residual Refinement Network for Saliency Detection Deng, Zijun; Hu, Xiaowei; Zhu, Lei Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence https://doi.org/10.24963/ijcai.2018/95	conference	July 2018
Context-Aware Image Inpainting with Learned Semantic Priors Zhang, Wendong; Zhu, Junwei; Tai, Ying Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence https://doi.org/10.24963/ijcai.2021/183	conference	August 2021
Cloud Extraction from Chinese High Resolution Satellite Imagery by Probabilistic Latent Semantic Analysis and Object-Based Machine Learning Tan, Kai; Zhang, Yongjun; Tong, Xin Remote Sensing, Vol. 8, Issue 11 https://doi.org/10.3390/rs8110963	journal	November 2016
LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation Junjue, Wang; Zhuo, Zheng; Ailong, Ma Zenodo https://doi.org/10.5281/zenodo.5706578	dataset	January 2021

Similar Records

Inpainting with sparse linear combinations of exemplars

Journal Article · Mon Dec 31 23:00:00 EST 2007 · OSTI ID:960615

Load Profile Inpainting for Missing Load Data Restoration and Baseline Estimation

Journal Article · Thu Jul 06 20:00:00 EDT 2023 · IEEE Transactions on Smart Grid · OSTI ID:2329385

Related Subjects

97 MATHEMATICS AND COMPUTING
coarse-to-fine
geoscience images
image inpainting
task-driven

Coarse-to-fine Task-driven Inpainting for Geoscience Images

Citation Formats

References (62)

Similar Records

Related Subjects