Tightly-coupled camera/LiDAR integration for point cloud generation from GNSS/INS-assisted UAV mapping systems

Zhou, Tian; Hasheminasab, Seyyed Meghdad; Habib, Ayman

doi:10.1016/j.isprsjprs.2021.08.020

Tightly-coupled camera/LiDAR integration for point cloud generation from GNSS/INS-assisted UAV mapping systems

Journal Article · Thu Sep 09 00:00:00 EDT 2021 · ISPRS Journal of Photogrammetry and Remote Sensing

DOI:https://doi.org/10.1016/j.isprsjprs.2021.08.020· OSTI ID:1977227

Zhou, Tian ^[1]; Hasheminasab, Seyyed Meghdad ^[2]; Habib, Ayman ^[2]

Purdue Univ., West Lafayette, IN (United States); OSTI
Purdue Univ., West Lafayette, IN (United States)

Unmanned aerial vehicles (UAVs) equipped with integrated global navigation satellite systems/inertial navigation systems (GNSS/INS) together with cameras and/or LiDAR sensors are being widely used for topographic mapping in a variety of applications such as precision agriculture, coastal monitoring, and archaeological documentation. Integration of image-based and LiDAR point clouds can provide a comprehensive 3D model of the area of interest. For such integration, ensuring a good alignment between data from the different sources is critical. Although many works have been conducted on this topic, there is still a need for a rigorous integration approach that minimizes the discrepancy between camera and LiDAR data caused by inaccurate system calibration parameters and/or trajectory artifacts. This study proposes an automated tightly-coupled camera/LiDAR integration workflow for GNSS/INS-assisted UAV systems. The proposed strategy is conducted in three main steps. First, an image-based point cloud is generated using a LiDAR/GNSS/INS-assisted structure from motion (SfM) strategy. Then, feature correspondences between image-based and LiDAR point clouds are automatically identified. Finally, an integrated-bundle adjustment procedure including image points, LiDAR raw measurements, and GNSS/INS information is conducted to minimize the discrepancy between point clouds from different sensors while estimating system calibration parameters and refining the trajectory information. The proposed SfM strategy and integration framework are evaluated using five datasets. The SfM results show that using LiDAR data can facilitate feature matching and further increase the number of reconstructed 3D points. The experimental results also illustrate that the developed automated camera/LiDAR integration strategy is capable of accurately estimating system calibration parameters to achieve good alignment among camera/LiDAR data from single/multiple systems. Finally, an absolute accuracy in the range of 3–5 cm is achieved for the image/LiDAR point clouds after the integration process.

View Accepted Manuscript (DOE)

Research Organization:: Purdue Univ., West Lafayette, IN (United States)

Sponsoring Organization:: USDOE Advanced Research Projects Agency - Energy (ARPA-E)

Grant/Contract Number:: AR0001135

OSTI ID:: 1977227

Alternate ID(s):: OSTI ID: 1819466

Journal Information:: ISPRS Journal of Photogrammetry and Remote Sensing, Journal Name: ISPRS Journal of Photogrammetry and Remote Sensing Journal Issue: C Vol. 180; ISSN 0924-2716

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (25)

Topographic structure from motion: a new development in photogrammetric measurement: TOPOGRAPHIC STRUCTURE FROM MOTION Fonstad, Mark A.; Dietrich, James T.; Courville, Brittany C. Earth Surface Processes and Landforms, Vol. 38, Issue 4 https://doi.org/10.1002/esp.3366	journal	January 2013
UAV for 3D mapping applications: a review Nex, Francesco; Remondino, Fabio Applied Geomatics, Vol. 6, Issue 1 https://doi.org/10.1007/s12518-013-0120-x	journal	November 2013
Principal component analysis Wold, Svante; Esbensen, Kim; Geladi, Paul Chemometrics and Intelligent Laboratory Systems, Vol. 2, Issue 1-3 https://doi.org/10.1016/0169-7439(87)80084-9	journal	August 1987
Object modelling by registration of multiple range images Chen, Yang; Medioni, Gérard Image and Vision Computing, Vol. 10, Issue 3 https://doi.org/10.1016/0262-8856(92)90066-C	journal	April 1992
‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications Westoby, M. J.; Brasington, J.; Glasser, N. F. Geomorphology, Vol. 179 https://doi.org/10.1016/j.geomorph.2012.08.021	journal	December 2012
In-flight photogrammetric camera calibration and validation via complementary lidar Gneeniss, A. S.; Mills, J. P.; Miller, P. E. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 100 https://doi.org/10.1016/j.isprsjprs.2014.04.019	journal	February 2015
NRLI-UAV: Non-rigid registration of sequential raw laser scans and images for low-cost UAV LiDAR point cloud quality improvement Li, Jianping; Yang, Bisheng; Chen, Chi ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 158 https://doi.org/10.1016/j.isprsjprs.2019.10.009	journal	December 2019
A flexible targetless LiDAR–GNSS/INS–camera calibration method for UAV platforms Pentek, Quentin; Kennel, Pol; Allouis, Tristan ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 166 https://doi.org/10.1016/j.isprsjprs.2020.05.014	journal	August 2020
Distinctive Image Features from Scale-Invariant Keypoints Lowe, David G. International Journal of Computer Vision, Vol. 60, Issue 2, p. 91-110 https://doi.org/10.1023/B:VISI.0000029664.99615.94	journal	November 2004
Structure-from-Motion Revisited Schonberger, Johannes L.; Frahm, Jan-Michael 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.445	conference	June 2016
Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences Zhou, Lipu; Li, Zimo; Kaess, Michael 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) https://doi.org/10.1109/IROS.2018.8593660	conference	October 2018
Automatic extrinsic calibration for lidar-stereo vehicle sensor setups Guindel, Carlos; Beltran, Jorge; Martin, David 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) https://doi.org/10.1109/ITSC.2017.8317829	conference	October 2017
Infrastructure Based Calibration of a Multi-Camera and Multi-LiDAR System Using Apriltags Xie, Yuanfan; Shao, Rui; Guli, Popo 2018 IEEE Intelligent Vehicles Symposium (IV) https://doi.org/10.1109/IVS.2018.8500646	conference	June 2018
Simultaneous System Calibration of a Multi-LiDAR Multicamera Mobile Mapping Platform Ravi, Radhika; Lin, Yun-Jou; Elbahnasawy, Magdy IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 11, Issue 5 https://doi.org/10.1109/JSTARS.2018.2812796	journal	May 2018
Bias Impact Analysis and Calibration of Terrestrial Mobile LiDAR System With Several Spinning Multibeam Laser Scanners Ravi, Radhika; Lin, Yun-Jou; Elbahnasawy, Magdy IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, Issue 9 https://doi.org/10.1109/TGRS.2018.2812782	journal	September 2018
Accurate, Dense, and Robust Multiview Stereopsis Furukawa, Yasutaka; Ponce, Jean IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, Issue 8 https://doi.org/10.1109/TPAMI.2009.161	journal	August 2010
Stability Analysis and Geometric Calibration of Off-the-Shelf Digital Cameras Habib, Ayman; Morgan, Michel Photogrammetric Engineering & Remote Sensing, Vol. 71, Issue 6 https://doi.org/10.14358/PERS.71.6.733	journal	June 2005
Automated Relative Orientation of UAV-Based Imagery in the Presence of Prior Information for the Flight Trajectory He, Fangning; Habib, Ayman Photogrammetric Engineering & Remote Sensing, Vol. 82, Issue 11 https://doi.org/10.14358/PERS.82.11.879	journal	November 2016
Rigorous Strip Adjustment of UAV-based Laserscanning Data Including Time-Dependent Correction of Trajectory Errors Glira, Philipp; Pfeifer, Norbert; Mandlburger, Gottfried Photogrammetric Engineering & Remote Sensing, Vol. 82, Issue 12 https://doi.org/10.14358/PERS.82.12.945	journal	December 2016
GNSS/INS-Assisted Structure from Motion Strategies for UAV-Based Imagery over Mechanized Agricultural Fields Hasheminasab, Seyyed Meghdad; Zhou, Tian; Habib, Ayman Remote Sensing, Vol. 12, Issue 3 https://doi.org/10.3390/rs12030351	journal	January 2020
LiDAR-Aided Interior Orientation Parameters Refinement Strategy for Consumer-Grade Cameras Onboard UAV Remote Sensing Systems Zhou, Tian; Hasheminasab, Seyyed Meghdad; Ravi, Radhika Remote Sensing, Vol. 12, Issue 14 https://doi.org/10.3390/rs12142268	journal	July 2020
Accurate Calibration of Multi-LiDAR-Multi-Camera Systems Pusztai, Zoltán; Eichhardt, Iván; Hajder, Levente Sensors, Vol. 18, Issue 7 https://doi.org/10.3390/s18072139	journal	July 2018
A Novel Calibration Board and Experiments for 3D LiDAR and Camera Calibration Cai, Huaiyu; Pang, Weisong; Chen, Xiaodong Sensors, Vol. 20, Issue 4 https://doi.org/10.3390/s20041130	journal	February 2020
Hybrid Orientation of Airborne Lidar Point Clouds and Aerial Images Glira, P.; Pfeifer, N.; Mandlburger, G. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. IV-2/W5 https://doi.org/10.5194/isprs-annals-IV-2-W5-567-2019	journal	January 2019
Investigations on the Quality of the Interior Orientation and its Impact in Object Space for uav Photogrammetry Hastedt, H.; Luhmann, T. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XL-1/W4 https://doi.org/10.5194/isprsarchives-XL-1-W4-321-2015	journal	January 2015

Similar Records

LiDAR-Aided Interior Orientation Parameters Refinement Strategy for Consumer-Grade Cameras Onboard UAV Remote Sensing Systems

Journal Article · Wed Jul 15 00:00:00 EDT 2020 · Remote Sensing · OSTI ID:2503553

Related Subjects

42 ENGINEERING
Bundle adjustment
Camera/LiDAR integration
GNSS/INS-assisted mapping
Geology
Imaging Science & Photographic Technology
Physical Geography
Remote Sensing
Structure from motion
System calibration
Unmanned aerial vehicles

Tightly-coupled camera/LiDAR integration for point cloud generation from GNSS/INS-assisted UAV mapping systems

Citation Formats

References (25)

Similar Records

Related Subjects