PlantSegNet: 3D point cloud instance segmentation of nearby plant organs with identical semantics

Zarei, Ariyan; Li, Bosheng; Schnable, James C.; Lyons, Eric; Pauli, Duke; Barnard, Kobus; Benes, Bedrich

doi:10.1016/j.compag.2024.108922

Title: PlantSegNet: 3D point cloud instance segmentation of nearby plant organs with identical semantics

Journal Article · 16 April 2024 · Computers and Electronics in Agriculture

DOI: https://doi.org/10.1016/j.compag.2024.108922 · OSTI ID:2339863

^[1]; Li, Bosheng ^[2]; Schnable, James C. ^[3]; Lyons, Eric ^[4]; Pauli, Duke ^[4]; Barnard, Kobus ^[4]; Benes, Bedrich ^[2]

University of Arizona, Tucson, AZ (United States); Donald Danforth Plant Science Center
Purdue University, West Lafayette, LA (United States)
University of Nebraska–Lincoln, NE (United States)
University of Arizona, Tucson, AZ (United States)

In this study, we introduce PlantSegNet, a novel neural network model for instance segmentation of nearby objects with similar geometric structures. Our work addresses the challenges of instance segmentation of plant point clouds, including the difficulty of annotating and labeling point clouds, the loss of local structural information in neural network components, and the generation of large numbers of incorrect small clusters due to poor choices of the loss function. One of the key contributions of our approach is a digital twin of sorghum, i.e., a procedural sorghum model, which was used to generate point clouds of sorghum fields. This allowed us to create a large-scale, annotated, synthetic dataset of sorghum plants that we used to train our PlantSegNet model. We demonstrated the effectiveness of our method in segmenting instances of sorghum leaves grown in outdoor field settings. To the best of our knowledge, this is the first study to address this specific instance segmentation problem for plants grown in such a setting. We compared our proposed method with other state-of-the-art methods for indoor settings, including SGPN and TreePartNet, on both synthetic and real data. Furthermore, our results show that PlantSegNet outperforms these methods regarding accuracy, robustness, and efficiency.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Donald Danforth Plant Science Center, St. Louis, MO (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER); USDOE Advanced Research Projects Agency - Energy (ARPA-E)

Grant/Contract Number:: SC0020401; AR0001101; SC0023305; AR0000594

OSTI ID:: 2339863

Journal Information:: Computers and Electronics in Agriculture, Journal Name: Computers and Electronics in Agriculture Vol. 221; ISSN 0168-1699

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (34)

OctNet: Learning Deep 3D Representations at High Resolutions Riegler, Gernot; Ulusoy, Ali Osman; Geiger, Andreas 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.701	conference	July 2017
VIBUS: Data-efficient 3D scene parsing with VIewpoint Bottleneck and Uncertainty-Spectrum modeling Tian, Beiwen; Luo, Liyi; Zhao, Hao ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 194 https://doi.org/10.1016/j.isprsjprs.2022.10.013	journal	December 2022
Automatic organ-level point cloud segmentation of maize shoots by integrating high-throughput data acquisition and deep learning Li, Yinglun; Wen, Weiliang; Miao, Teng Computers and Electronics in Agriculture, Vol. 193 https://doi.org/10.1016/j.compag.2022.106702	journal	February 2022
GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud Yi, Li; Zhao, Wang; Wang, He 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00407	conference	June 2019
Label3DMaize: toolkit for 3D point cloud data annotation of maize shoots Miao, Teng; Wen, Weiliang; Li, Yinglun GigaScience, Vol. 10, Issue 5 https://doi.org/10.1093/gigascience/giab031	journal	May 2021
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans Hou, Ji; Dai, Angela; NieBner, Matthias 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00455	conference	June 2019
Dynamic quantification of canopy structure to characterize early plant vigour in wheat genotypes Duan, T.; Chapman, S. C.; Holland, E. Journal of Experimental Botany, Vol. 67, Issue 15 https://doi.org/10.1093/jxb/erw227	journal	June 2016
Latent L-systems: Transformer-based Tree Generator Lee, Jae Joong; Li, Bosheng; Benes, Bedrich ACM Transactions on Graphics, Vol. 43, Issue 1 https://doi.org/10.1145/3627101	journal	November 2023
PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding Mo, Kaichun; Zhu, Shilin; Chang, Angel X. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00100	conference	June 2019
Plant-part segmentation using deep learning and multi-view vision Shi, Weinan; van de Zedde, Rick; Jiang, Huanyu Biosystems Engineering, Vol. 187 https://doi.org/10.1016/j.biosystemseng.2019.08.014	journal	November 2019
3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation Liang, Zhidong; Yang, Ming; Li, Hao IEEE Robotics and Automation Letters, Vol. 5, Issue 3 https://doi.org/10.1109/LRA.2020.3004802	journal	July 2020
Point Cloud Pre-training with Natural 3D Structures Yamada, Ryosuke; Kataoka, Hirokatsu; Chiba, Naoya 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.02060	conference	June 2022
Associatively Segmenting Instances and Semantics in Point Clouds Wang, Xinlong; Liu, Shu; Shen, Xiaoyong 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00422	conference	June 2019
3D Point Cloud Instance Segmentation of Lettuce Based on PartNet Wang, Luhan; Zheng, Lihua; Wang, Minjuan 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) https://doi.org/10.1109/CVPRW56347.2022.00171	conference	June 2022
Deep Learning for 3D Point Clouds: A Survey Guo, Yulan; Wang, Hanyun; Hu, Qingyong IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, Issue 12 https://doi.org/10.1109/TPAMI.2020.3005434	journal	December 2021
PointVST: Self-Supervised Pre-Training for 3D Point Clouds Via View-Specific Point-to-Image Translation Zhang, Qijian; Hou, Junhui IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2023.3345353	journal	January 2023
Adaptive Graph Convolution for Point Cloud Analysis Zhou, Haoran; Feng, Yidan; Fang, Mingsheng 2021 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV48922.2021.00492	conference	October 2021
Dynamic Graph CNN for Learning on Point Clouds Wang, Yue; Sun, Yongbin; Liu, Ziwei ACM Transactions on Graphics, Vol. 38, Issue 5 https://doi.org/10.1145/3326362	journal	October 2019
RegGeoNet: Learning Regular Representations for Large-Scale 3D Point Clouds Zhang, Qijian; Hou, Junhui; Qian, Yue International Journal of Computer Vision, Vol. 130, Issue 12 https://doi.org/10.1007/s11263-022-01682-w	journal	September 2022
3D Tree Reconstruction from Simulated Small Footprint Waveform Lidar Wu, Jiaying; Cawse-Nicholson, Kerry; van Aardt, Jan Photogrammetric Engineering & Remote Sensing, Vol. 79, Issue 12 https://doi.org/10.14358/PERS.79.12.1147	journal	December 2013
TreePartNet Liu, Yanchao; Guo, Jianwei; Benes, Bedrich ACM Transactions on Graphics, Vol. 40, Issue 6 https://doi.org/10.1145/3478513.3480486	journal	December 2021
Grid-GCN for Fast and Scalable Point Cloud Learning Xu, Qiangeng; Sun, Xudong; Wu, Cho-Ying 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00570	conference	June 2020
Point Transformer Zhao, Hengshuang; Jiang, Li; Jia, Jiaya 2021 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV48922.2021.01595	conference	October 2021
PlantNet: A dual-function point cloud segmentation network for multiple plant species Li, Dawei; Shi, Guoliang; Li, Jinsheng ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 184 https://doi.org/10.1016/j.isprsjprs.2022.01.007	journal	February 2022
JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields Pham, Quang-Hieu; Nguyen, Thanh; Hua, Binh-Son 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00903	conference	June 2019
Multi-view Convolutional Neural Networks for 3D Shape Recognition Su, Hang; Maji, Subhransu; Kalogerakis, Evangelos 2015 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2015.114	conference	December 2015
Rhizomorph: The Coordinated Function of Shoots and Roots Li, Bosheng; Klein, Jonathan; Michels, Dominik L. ACM Transactions on Graphics, Vol. 42, Issue 4 https://doi.org/10.1145/3592145	journal	July 2023
Stratified Transformer for 3D Point Cloud Segmentation Lai, Xin; Liu, Jianhui; Jiang, Li 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.00831	conference	June 2022
Multi-view Harmonized Bilinear Network for 3D Object Recognition Yu, Tan; Meng, Jingjing; Yuan, Junsong 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00027	conference	June 2018
MVS-Pheno: A Portable and Low-Cost Phenotyping Platform for Maize Shoots Using Multiview Stereo 3D Reconstruction Wu, Sheng; Wen, Weiliang; Wang, Yongjian Plant Phenomics, Vol. 2020 https://doi.org/10.34133/2020/1848437	journal	January 2020
View-GCN: View-Based Graph Convolutional Network for 3D Shape Analysis Wei, Xin; Yu, Ruixuan; Sun, Jian 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00192	conference	June 2020
3D Semantic Parsing of Large-Scale Indoor Spaces Armeni, Iro; Sener, Ozan; Zamir, Amir R. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.170	conference	June 2016
SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation Wang, Weiyue; Yu, Ronald; Huang, Qiangui 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00272	conference	June 2018
MegaStitch: Robust Large-Scale Image Stitching Zarei, Ariyan; Gonzalez, Emmanuel; Merchant, Nirav IEEE Transactions on Geoscience and Remote Sensing, Vol. 60 https://doi.org/10.1109/TGRS.2022.3141907	journal	January 2022

Similar Records

PhytoOracle: Scalable, modular phenomics data processing pipelines

Journal Article · 2023 · Frontiers in Plant Science · OSTI ID:1960057

Gonzalez, Emmanuel M.; Zarei, Ariyan; Hendler, Nathanial; +16 more

Automatic Crack Segmentation and Feature Extraction in Electroluminescence Images of Solar Modules

Journal Article · 2023 · IEEE Journal of Photovoltaics · OSTI ID:1973673

Chen, Xin; Karin, Todd; Libby, Cara; +4 more

A Two-Tier Convolutional Neural Network for Combined Detection and Segmentation in Biological Imagery

Conference · 2019 · OSTI ID:1761680

Ziabari, Amir; Shirinifard, Abbas; Eicholtz, Matthew; +2 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Phenotyping
Digital twins
Point clouds
Procedural modeling
Plant geometry
Sorghum

Title: PlantSegNet: 3D point cloud instance segmentation of nearby plant organs with identical semantics

Citation Formats

References (34)

Similar Records

Related Subjects