Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Elmes, Arthur; Alemohammad, Hamed; Avery, Ryan; Caylor, Kelly; Eastman, J. Ronald; Fishgold, Lewis; Friedl, Mark A.; Jain, Meha; Kohli, Divyani; Laso Bayas, Juan Carlos; Lunga, Dalton; McCarty, Jessica L.; Pontius, Jr., Robert Gilmore; Reinmann, Andrew B.; Rogan, John; Song, Lei; Stoynova, Hristiana; Ye, Su; Yi, Zhuang-Fang; Estes, Lyndon

doi:10.3390/rs12061034

Title: Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Journal Article · Mon Mar 23 00:00:00 EDT 2020 · Remote Sensing

DOI:https://doi.org/10.3390/rs12061034· OSTI ID:1608215

^[1];

^[2]; Avery, Ryan ^[3]; Caylor, Kelly ^[3]; Eastman, J. Ronald ^[4]; Fishgold, Lewis ^[5]; Friedl, Mark A. ^[6]; Jain, Meha ^[7]; Kohli, Divyani ^[8];

^[9];

^[10];

^[11];

^[4]; Reinmann, Andrew B. ^[12]; Rogan, John ^[4]; Song, Lei ^[4]; Stoynova, Hristiana ^[12]; Ye, Su ^[4]; Yi, Zhuang-Fang ^[13];

^[4]

Clark Univ., Worcester, MA (United States); Univ. of Massachusetts, Boston (United States)
Radiant Earth Foundation, San Francisco, CA (United States)
Univ. of California, Santa Barbara, CA (United States)
Clark Univ., Worcester, MA (United States)
Azavea, Inc., Philadelphia, PA (United States)
Boston Univ., MA (United States)
Univ. of Michigan, Ann Arbor, MI (United States)
Univ. of Twente, Enschede (The Netherlands)
International Inst. for Applied Systems Analysis (IIASA), Laxenburg (Austria)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Miami Univ., Oxford, OH (United States)
City Univ. of New York (CUNY), NY (United States). Advanced Science Research Center; Hunter College, New York, NY (United States)
Development Seed,Washington, DC (United States)

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; Omidyar Network’s Property Rights Initiative, now PlaceFund; National Aeronautics and Space Administration (NASA); National Science Foundation (NSF); National Institute of Standards and Technology (NIST); New York State Department of Environmental Conservation

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1608215

Journal Information:: Remote Sensing, Vol. 12, Issue 6; Conference: Quantifying Error in Training Data for Mapping and Monitoring the Earth System, Worcester, MA (United States), 8-9 Jan 2019; ISSN 2072-4292

Publisher:: MDPICopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 25 works

Citation information provided by
Web of Science

References (192)

Influence of carbon mapping and land change modelling on the prediction of carbon emissions from deforestation Gutierrez-Velez, Victor Hugo; Pontius, Robert Gilmore Environmental Conservation, Vol. 39, Issue 4 https://doi.org/10.1017/S0376892912000173	journal	June 2012
Orthorectification of VHR optical satellite data exploiting the geometric accuracy of TerraSAR-X data Reinartz, Peter; Müller, Rupert; Schwind, Peter ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 66, Issue 1 https://doi.org/10.1016/j.isprsjprs.2010.10.003	journal	January 2011
Toward intelligent training of supervised image classifications: directing training data acquisition for SVM classification Foody, Giles M.; Mathur, Ajay Remote Sensing of Environment, Vol. 93, Issue 1-2 https://doi.org/10.1016/j.rse.2004.06.017	journal	October 2004
Global land cover classifications at 8 km spatial resolution: The use of training data derived from Landsat imagery in decision tree classifiers De Fries, R. S.; Hansen, M.; Townshend, J. R. G. International Journal of Remote Sensing, Vol. 19, Issue 16 https://doi.org/10.1080/014311698214235	journal	January 1998
The impact of imperfect ground reference data on the accuracy of land cover change estimation Foody, Giles M. International Journal of Remote Sensing, Vol. 30, Issue 12 https://doi.org/10.1080/01431160902755346	journal	June 2009
Mapping smallholder and large-scale cropland dynamics with a flexible classification system and pixel-based composites in an emerging frontier of Mozambique Bey, Adia; Jetimane, Julieta; Lisboa, Sá Nogueira Remote Sensing of Environment, Vol. 239 https://doi.org/10.1016/j.rse.2019.111611	journal	March 2020
Citizens as sensors: the world of volunteered geography Goodchild, Michael F. GeoJournal, Vol. 69, Issue 4 https://doi.org/10.1007/s10708-007-9111-y	journal	November 2007
Object based image analysis for remote sensing Blaschke, T. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 65, Issue 1 https://doi.org/10.1016/j.isprsjprs.2009.06.004	journal	January 2010
A Pixel-Based Landsat Compositing Algorithm for Large Area Land Cover Mapping Griffiths, Patrick; van der Linden, Sebastian; Kuemmerle, Tobias IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 6, Issue 5 https://doi.org/10.1109/JSTARS.2012.2228167	journal	October 2013
Exploring the Use of Google Earth Imagery and Object-Based Methods in Land Use/Cover Mapping Hu, Qiong; Wu, Wenbin; Xia, Tian Remote Sensing, Vol. 5, Issue 11 https://doi.org/10.3390/rs5116026	journal	November 2013
Artificial neural network response to mixed pixels in coarse-resolution satellite data Moody, Aaron; Gopal, Sucharita; Strahler, Alan H. Remote Sensing of Environment, Vol. 58, Issue 3 https://doi.org/10.1016/S0034-4257(96)00107-1	journal	December 1996
On the nature of models in remote sensing Strahler, Alan H.; Woodcock, Curtis E.; Smith, James A. Remote Sensing of Environment, Vol. 20, Issue 2 https://doi.org/10.1016/0034-4257(86)90018-0	journal	October 1986
Mapping land-cover modifications over large areas: A comparison of machine learning algorithms Rogan, John; Franklin, Janet; Stow, Doug Remote Sensing of Environment, Vol. 112, Issue 5 https://doi.org/10.1016/j.rse.2007.10.004	journal	May 2008
Deep learning in remote sensing applications: A meta-analysis and review Ma, Lei; Liu, Yu; Zhang, Xueliang ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 152 https://doi.org/10.1016/j.isprsjprs.2019.04.015	journal	June 2019
Status of land cover classification accuracy assessment Foody, Giles M. Remote Sensing of Environment, Vol. 80, Issue 1 https://doi.org/10.1016/S0034-4257(01)00295-4	journal	April 2002
Water, Energy, and Carbon with Artificial Neural Networks (WECANN): a statistically based estimate of global surface turbulent fluxes and gross primary productivity using solar-induced fluorescence Alemohammad, Seyed Hamed; Fang, Bin; Konings, Alexandra G. Biogeosciences, Vol. 14, Issue 18 https://doi.org/10.5194/bg-14-4101-2017	journal	January 2017
Satellite-based assessment of yield variation and its determinants in smallholder African systems Burke, Marshall; Lobell, David B. Proceedings of the National Academy of Sciences, Vol. 114, Issue 9 https://doi.org/10.1073/pnas.1616919114	journal	February 2017
Land-Use Mapping in a Mixed Urban-Agricultural Arid Landscape Using Object-Based Image Analysis: A Case Study from Maricopa, Arizona Galletti, Christopher; Myint, Soe Remote Sensing, Vol. 6, Issue 7 https://doi.org/10.3390/rs6076089	journal	June 2014
Strategies for Incorporating High-Resolution Google Earth Databases to Guide and Validate Classifications: Understanding Deforestation in Borneo Dorais, Alexis; Cardille, Jeffrey Remote Sensing, Vol. 3, Issue 6 https://doi.org/10.3390/rs3061157	journal	June 2011
A survey of image classification methods and techniques for improving classification performance Lu, D.; Weng, Q. International Journal of Remote Sensing, Vol. 28, Issue 5 https://doi.org/10.1080/01431160600746456	journal	March 2007
Exploring diversity in ensemble classification: Applications in large area land cover mapping Mellor, Andrew; Boukir, Samia ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 129 https://doi.org/10.1016/j.isprsjprs.2017.04.017	journal	July 2017
Recommendations for using the relative operating characteristic (ROC) Pontius, Robert Gilmore; Parmentier, Benoit Landscape Ecology, Vol. 29, Issue 3 https://doi.org/10.1007/s10980-013-9984-8	journal	January 2014
New refinements and validation of the MODIS Land-Surface Temperature/Emissivity products Wan, Z. Remote Sensing of Environment, Vol. 112, Issue 1 https://doi.org/10.1016/j.rse.2006.06.026	journal	January 2008
A Novel Context-Sensitive Semisupervised SVM Classifier Robust to Mislabeled Training Samples Bruzzone, L.; Persello, C. IEEE Transactions on Geoscience and Remote Sensing, Vol. 47, Issue 7 https://doi.org/10.1109/TGRS.2008.2011983	journal	July 2009
High-Resolution Global Maps of 21st-Century Forest Cover Change Hansen, M. C.; Potapov, P. V.; Moore, R. Science, Vol. 342, Issue 6160 https://doi.org/10.1126/science.1244693	journal	November 2013
Mapping global cropland and field size Fritz, Steffen; See, Linda; McCallum, Ian Global Change Biology, Vol. 21, Issue 5 https://doi.org/10.1111/gcb.12838	journal	January 2015
Automatic identification of building types based on topographic databases – a comparison of different data sources Hecht, Robert; Meinel, Gotthard; Buchroithner, Manfred International Journal of Cartography, Vol. 1, Issue 1 https://doi.org/10.1080/23729333.2015.1055644	journal	January 2015
Land cover mapping of wetland areas in an agricultural landscape using SAR and Landsat imagery Castañeda, C.; Ducrot, D. Journal of Environmental Management, Vol. 90, Issue 7 https://doi.org/10.1016/j.jenvman.2007.06.030	journal	May 2009
Harshness in image classification accuracy assessment Foody, Giles M. International Journal of Remote Sensing, Vol. 29, Issue 11 https://doi.org/10.1080/01431160701442120	journal	May 2008
Large-scale land cover mapping with the integration of multi-source information based on the Dempster–Shafer theory Ran, Y. H.; Li, X.; Lu, L. International Journal of Geographical Information Science, Vol. 26, Issue 1 https://doi.org/10.1080/13658816.2011.577745	journal	January 2012
Classification in the Presence of Label Noise: A Survey Frenay, Benoit; Verleysen, Michel IEEE Transactions on Neural Networks and Learning Systems, Vol. 25, Issue 5 https://doi.org/10.1109/TNNLS.2013.2292894	journal	May 2014
First operational BRDF, albedo nadir reflectance products from MODIS Schaaf, Crystal B.; Gao, Feng; Strahler, Alan H. Remote Sensing of Environment, Vol. 83, Issue 1-2 https://doi.org/10.1016/S0034-4257(02)00091-3	journal	November 2002
Comparison of support vector machine, neural network, and CART algorithms for the land-cover classification using limited training data points Shao, Yang; Lunetta, Ross S. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 70 https://doi.org/10.1016/j.isprsjprs.2012.04.001	journal	June 2012
The Sensitivity of Mapping Methods to Reference Data Quality: Training Supervised Image Classifications with Imperfect Reference Data Foody, Giles; Pal, Mahesh; Rocchini, Duccio ISPRS International Journal of Geo-Information, Vol. 5, Issue 11 https://doi.org/10.3390/ijgi5110199	journal	November 2016
A neural network approach using multi-scale textural metrics from very high-resolution panchromatic imagery for urban land-use classification Pacifici, Fabio; Chini, Marco; Emery, William J. Remote Sensing of Environment, Vol. 113, Issue 6 https://doi.org/10.1016/j.rse.2009.02.014	journal	June 2009
Assessing geometric accuracy of the orthorectification process from GeoEye-1 and WorldView-2 panchromatic images Aguilar, Manuel A.; Saldaña, María del Mar; Aguilar, Fernando J. International Journal of Applied Earth Observation and Geoinformation, Vol. 21 https://doi.org/10.1016/j.jag.2012.06.004	journal	April 2013
Effect of errors in ground truth on classification accuracy Carlotto, Mark J. International Journal of Remote Sensing, Vol. 30, Issue 18 https://doi.org/10.1080/01431160802672864	journal	August 2009
Comparison of Random Forest, k-Nearest Neighbor, and Support Vector Machine Classifiers for Land Cover Classification Using Sentinel-2 Imagery Thanh Noi, Phan; Kappas, Martin Sensors, Vol. 18, Issue 2 https://doi.org/10.3390/s18010018	journal	December 2017
Spatiotemporal characteristics, patterns, and causes of land-use changes in China since the late 1980s Liu, Jiyuan; Kuang, Wenhui; Zhang, Zengxiang Journal of Geographical Sciences, Vol. 24, Issue 2 https://doi.org/10.1007/s11442-014-1082-6	journal	January 2014
Good practices for estimating area and assessing accuracy of land change Olofsson, Pontus; Foody, Giles M.; Herold, Martin Remote Sensing of Environment, Vol. 148 https://doi.org/10.1016/j.rse.2014.02.015	journal	May 2014
A review of accuracy assessment for object-based image analysis: From per-pixel to per-polygon approaches Ye, Su; Pontius, Robert Gilmore; Rakshit, Rahul ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 141 https://doi.org/10.1016/j.isprsjprs.2018.04.002	journal	July 2018
Remote sensing of impervious surfaces in the urban areas: Requirements, methods, and trends Weng, Qihao Remote Sensing of Environment, Vol. 117 https://doi.org/10.1016/j.rse.2011.02.030	journal	February 2012
A scalable satellite-based crop yield mapper Lobell, David B.; Thau, David; Seifert, Christopher Remote Sensing of Environment, Vol. 164 https://doi.org/10.1016/j.rse.2015.04.021	journal	July 2015
Conflation of expert and crowd reference data to validate global binary thematic maps Waldner, François; Schucknecht, Anne; Lesiv, Myroslava Remote Sensing of Environment, Vol. 221 https://doi.org/10.1016/j.rse.2018.10.039	journal	February 2019
Towards a set of agrosystem-specific cropland mapping methods to address the global cropland diversity Waldner, François; De Abelleyra, Diego; Verón, Santiago R. International Journal of Remote Sensing, Vol. 37, Issue 14 https://doi.org/10.1080/01431161.2016.1194545	journal	June 2016
Making better use of accuracy data in land change studies: Estimating accuracy and area and quantifying uncertainty using stratified estimation Olofsson, Pontus; Foody, Giles M.; Stehman, Stephen V. Remote Sensing of Environment, Vol. 129 https://doi.org/10.1016/j.rse.2012.10.031	journal	February 2013
Utilizing Temporally Invariant Calibration Sites to Classify Multiple Dates and Types of Satellite Imagery Fortier, Joe; Rogan, John; Woodcock, Curtis E. Photogrammetric Engineering & Remote Sensing, Vol. 77, Issue 2 https://doi.org/10.14358/PERS.77.2.181	journal	February 2011
Object-based Vegetation Mapping in the Kissimmee River Watershed Using HyMap Data and Machine Learning Techniques Zhang, Caiyun; Xie, Zhixiao Wetlands, Vol. 33, Issue 2 https://doi.org/10.1007/s13157-012-0373-x	journal	January 2013
Sub-pixel land-cover mapping with improved fraction images upon multiple-point simulation Ge, Yong International Journal of Applied Earth Observation and Geoinformation, Vol. 22 https://doi.org/10.1016/j.jag.2012.04.013	journal	June 2013
On the use of dimensioned measures of error to evaluate the performance of spatial interpolators Willmott, C. J.; Matsuura, K. International Journal of Geographical Information Science, Vol. 20, Issue 1 https://doi.org/10.1080/13658810500286976	journal	January 2006
Using active learning to adapt remote sensing image classifiers Tuia, D.; Pasolli, E.; Emery, W. J. Remote Sensing of Environment, Vol. 115, Issue 9 https://doi.org/10.1016/j.rse.2011.04.022	journal	September 2011
Carbon consequences of land cover change and expansion of urban lands: A case study in the Seattle metropolitan region Hutyra, Lucy R.; Yoon, Byungman; Hepinstall-Cymerman, Jeffrey Landscape and Urban Planning, Vol. 103, Issue 1 https://doi.org/10.1016/j.landurbplan.2011.06.004	journal	October 2011
Incorporating Land Use Mapping and Participation in Jordan: An Approach to Sustainable Management of Two Mountainous Areas Al-Bakri, Jawad Taleb; Ajlouni, Mohammad; Abu-Zanat, Mahfouz Mountain Research and Development, Vol. 28, Issue 1 https://doi.org/10.1659/mrd.0863	journal	February 2008
Comparison of methods for land-use classification incorporating remote sensing and GIS inputs Rozenstein, Offer; Karnieli, Arnon Applied Geography, Vol. 31, Issue 2 https://doi.org/10.1016/j.apgeog.2010.11.006	journal	April 2011
A fuzzy classification of sub-urban land cover from remotely sensed imagery Zhang, J.; Foody, G. M. International Journal of Remote Sensing, Vol. 19, Issue 14 https://doi.org/10.1080/014311698214479	journal	January 1998
The modifiable areal unit problem and implications for landscape ecology Jelinski, Dennis E.; Wu, Jianguo Landscape Ecology, Vol. 11, Issue 3 https://doi.org/10.1007/BF02447512	journal	June 1996
Criteria to Confirm Models that Simulate Deforestation and Carbon Disturbance Pontius, Robert Land, Vol. 7, Issue 3 https://doi.org/10.3390/land7030105	journal	September 2018
Comment on “Tropical forests are a net carbon source based on aboveground measurements of gain and loss” Hansen, Matthew C.; Potapov, Peter; Tyukavina, Alexandra Science, Vol. 363, Issue 6423 https://doi.org/10.1126/science.aar3629	journal	January 2019
Sampling designs for accuracy assessment of land cover Stehman, Stephen V. International Journal of Remote Sensing, Vol. 30, Issue 20 https://doi.org/10.1080/01431160903131000	journal	September 2009
A platform for crowdsourcing the creation of representative, accurate landcover maps Estes, L. D.; McRitchie, D.; Choi, J. Environmental Modelling & Software, Vol. 80 https://doi.org/10.1016/j.envsoft.2016.01.011	journal	June 2016
A framework for selecting appropriate remotely sensed data dimensions for environmental monitoring and management Phinn, S. R. International Journal of Remote Sensing, Vol. 19, Issue 17 https://doi.org/10.1080/014311698214136	journal	November 1998
Capturing rapid land surface dynamics with Collection V006 MODIS BRDF/NBAR/Albedo (MCD43) products Wang, Zhuosen; Schaaf, Crystal B.; Sun, Qingsong Remote Sensing of Environment, Vol. 207 https://doi.org/10.1016/j.rse.2018.02.001	journal	March 2018
A review of assessing the accuracy of classifications of remotely sensed data Congalton, Russell G. Remote Sensing of Environment, Vol. 37, Issue 1 https://doi.org/10.1016/0034-4257(91)90048-B	journal	July 1991
Spatio-temporal dynamics and evolution of land use change and landscape pattern in response to rapid urbanization Deng, Jin S.; Wang, Ke; Hong, Yang Landscape and Urban Planning, Vol. 92, Issue 3-4 https://doi.org/10.1016/j.landurbplan.2009.05.001	journal	September 2009
Challenges in using land use and land cover data for global change studies: LAND USE AND LAND COVER DATA FOR GLOBAL CHANGE STUDIES Verburg, Peter H.; Neumann, Kathleen; Nol, Linda Global Change Biology, Vol. 17, Issue 2 https://doi.org/10.1111/j.1365-2486.2010.02307.x	journal	January 2011
Accuracy of forest inventory mapping: Some implications for boreal forest management Thompson, Ian D.; Maher, Scott C.; Rouillard, Daniel P. Forest Ecology and Management, Vol. 252, Issue 1-3 https://doi.org/10.1016/j.foreco.2007.06.033	journal	November 2007
Development of a global land cover characteristics database and IGBP DISCover from 1 km AVHRR data Loveland, T. R.; Reed, B. C.; Brown, J. F. International Journal of Remote Sensing, Vol. 21, Issue 6-7 https://doi.org/10.1080/014311600210191	journal	January 2000
Land Surface Temperature Retrieval Methods From Landsat-8 Thermal Infrared Sensor Data Jimenez-Munoz, Juan C.; Sobrino, Jose A.; Skokovic, Drazen IEEE Geoscience and Remote Sensing Letters, Vol. 11, Issue 10 https://doi.org/10.1109/LGRS.2014.2312032	journal	October 2014
An Enhanced TIMESAT Algorithm for Estimating Vegetation Phenology Metrics From MODIS Data Tan, B.; Morisette, J. T.; Wolfe, R. E. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 4, Issue 2 https://doi.org/10.1109/JSTARS.2010.2075916	journal	June 2011
A global map of rainfed cropland areas (GMRCA) at the end of last millennium using remote sensing Biradar, Chandrashekhar M.; Thenkabail, Prasad S.; Noojipady, Praveen International Journal of Applied Earth Observation and Geoinformation, Vol. 11, Issue 2 https://doi.org/10.1016/j.jag.2008.11.002	journal	April 2009
Land cover change assessment using decision trees, support vector machines and maximum likelihood classification algorithms Otukei, J. R.; Blaschke, T. International Journal of Applied Earth Observation and Geoinformation, Vol. 12 https://doi.org/10.1016/j.jag.2009.11.002	journal	February 2010
Smallholder maize area and yield mapping at national scales with Google Earth Engine Jin, Zhenong; Azzari, George; You, Calum Remote Sensing of Environment, Vol. 228 https://doi.org/10.1016/j.rse.2019.04.016	journal	July 2019
Cartography: uncertainty, interventions, and dynamic display Monmonier, Mark Progress in Human Geography, Vol. 30, Issue 3 https://doi.org/10.1191/0309132506ph612pr	journal	June 2006
Assessing the accuracy of land cover change with imperfect ground reference data Foody, Giles M. Remote Sensing of Environment, Vol. 114, Issue 10 https://doi.org/10.1016/j.rse.2010.05.003	journal	October 2010
A Coefficient of Agreement for Nominal Scales Cohen, Jacob Educational and Psychological Measurement, Vol. 20, Issue 1 https://doi.org/10.1177/001316446002000104	journal	April 1960
A global reference database of crowdsourced cropland data collected using the Geo-Wiki platform Laso Bayas, Juan Carlos; Lesiv, Myroslava; Waldner, François Scientific Data, Vol. 4, Issue 1 https://doi.org/10.1038/sdata.2017.136	journal	September 2017
Modeling Percent Tree Canopy Cover: A Pilot Study Coulston, John W.; Moisen, Gretchen G.; Wilson, Barry T. Photogrammetric Engineering & Remote Sensing, Vol. 78, Issue 7 https://doi.org/10.14358/PERS.78.7.715	journal	July 2012
MODIS Collection 5 global land cover: Algorithm refinements and characterization of new datasets Friedl, Mark A.; Sulla-Menashe, Damien; Tan, Bin Remote Sensing of Environment, Vol. 114, Issue 1 https://doi.org/10.1016/j.rse.2009.08.016	journal	January 2010
Edge effects enhance carbon uptake and its vulnerability to climate change in temperate broadleaf forests Reinmann, Andrew B.; Hutyra, Lucy R. Proceedings of the National Academy of Sciences, Vol. 114, Issue 1 https://doi.org/10.1073/pnas.1612369114	journal	December 2016
Global characterization and monitoring of forest cover using Landsat data: opportunities and challenges Townshend, John R.; Masek, Jeffrey G.; Huang, Chengquan International Journal of Digital Earth, Vol. 5, Issue 5 https://doi.org/10.1080/17538947.2012.713190	journal	August 2012
Soil Moisture Remote Sensing: State-of-the-Science Mohanty, Binayak P.; Cosh, Michael H.; Lakshmi, Venkat Vadose Zone Journal, Vol. 16, Issue 1 https://doi.org/10.2136/vzj2016.10.0105	journal	January 2017
A generalized cross‐tabulation matrix to compare soft‐classified maps at multiple resolutions Pontius, R. G.; Cheuk, M. L. International Journal of Geographical Information Science, Vol. 20, Issue 1 https://doi.org/10.1080/13658810500391024	journal	January 2006
A review of methods for the assessment of prediction errors in conservation presence/absence models Fielding, Alan H.; Bell, John F. Environmental Conservation, Vol. 24, Issue 1 https://doi.org/10.1017/S0376892997000088	journal	March 1997
From Guesstimates to GPStimates: Land Area Measurement and Implications for Agricultural Analysis Carletto, Calogero; Gourlay, Sydney; Winters, Paul Journal of African Economies, Vol. 24, Issue 5 https://doi.org/10.1093/jae/ejv011	journal	May 2015
Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance Willmott, Cj; Matsuura, K. Climate Research, Vol. 30 https://doi.org/10.3354/cr030079	journal	January 2005
Characterizing the Spatial and Temporal Availability of Very High Resolution Satellite Imagery in Google Earth and Microsoft Bing Maps as a Source of Reference Data Lesiv, Myroslava; See, Linda; Laso Bayas, Juan Land, Vol. 7, Issue 4 https://doi.org/10.3390/land7040118	journal	October 2018
Combining satellite imagery and machine learning to predict poverty Jean, N.; Burke, M.; Xie, M. Science, Vol. 353, Issue 6301 https://doi.org/10.1126/science.aaf7894	journal	August 2016
Range of Categorical Associations for Comparison of Maps with Mixed Pixels Pontius, Jr., Robert Gilmore; Connors, John Photogrammetric Engineering & Remote Sensing, Vol. 75, Issue 8 https://doi.org/10.14358/PERS.75.8.963	journal	August 2009
Crowdsourcing In-Situ Data on Land Cover and Land Use Using Gamification and Mobile Technology Laso Bayas, Juan; See, Linda; Fritz, Steffen Remote Sensing, Vol. 8, Issue 11 https://doi.org/10.3390/rs8110905	journal	November 2016
Quality control and assessment of interpreter consistency of annual land cover reference data in an operational national monitoring program Pengra, Bruce W.; Stehman, Stephen V.; Horton, Josephine A. Remote Sensing of Environment, Vol. 238 https://doi.org/10.1016/j.rse.2019.111261	journal	March 2020
Hybrid object-based approach for land use/land cover mapping using high spatial resolution imagery Malinverni, Eva Savina; Tassetti, Anna Nora; Mancini, Adriano International Journal of Geographical Information Science, Vol. 25, Issue 6 https://doi.org/10.1080/13658816.2011.566569	journal	June 2011
Recent Advances in the Measurement Error Literature Schennach, Susanne M. Annual Review of Economics, Vol. 8, Issue 1 https://doi.org/10.1146/annurev-economics-080315-015058	journal	October 2016
A scalable approach to mapping annual land cover at 250 m using MODIS time series data: A case study in the Dry Chaco ecoregion of South America Clark, Matthew L.; Aide, T. Mitchell; Grau, H. Ricardo Remote Sensing of Environment, Vol. 114, Issue 11 https://doi.org/10.1016/j.rse.2010.07.001	journal	November 2010
Global land cover mapping at 30m resolution: A POK-based operational approach Chen, Jun; Chen, Jin; Liao, Anping ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 103 https://doi.org/10.1016/j.isprsjprs.2014.09.002	journal	May 2015
Detecting trends in forest disturbance and recovery using yearly Landsat time series: 1. LandTrendr — Temporal segmentation algorithms Kennedy, Robert E.; Yang, Zhiqiang; Cohen, Warren B. Remote Sensing of Environment, Vol. 114, Issue 12 https://doi.org/10.1016/j.rse.2010.07.008	journal	December 2010
Long-term land cover dynamics by multi-temporal classification across the Landsat-5 record Sexton, Joseph O.; Urban, Dean L.; Donohue, Michael J. Remote Sensing of Environment, Vol. 128 https://doi.org/10.1016/j.rse.2012.10.010	journal	January 2013
An ontology of slums for image-based classification Kohli, Divyani; Sliuzas, Richard; Kerle, Norman Computers, Environment and Urban Systems, Vol. 36, Issue 2 https://doi.org/10.1016/j.compenvurbsys.2011.11.001	journal	March 2012
Exploring issues of training data imbalance and mislabelling on random forest performance for large area land cover classification using the ensemble margin Mellor, Andrew; Boukir, Samia; Haywood, Andrew ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 105 https://doi.org/10.1016/j.isprsjprs.2015.03.014	journal	July 2015
Comparison of Office and Field Techniques for Validating Landscape Change Classification in Pacific Northwest National Parks Copass, Catharine; Antonova, Natalya; Kennedy, Robert Remote Sensing, Vol. 11, Issue 1 https://doi.org/10.3390/rs11010003	journal	December 2018
Towards a Reproducible LULC Hierarchical Class Legend for Use in the Southwest of Pará State, Brazil: A Comparison with Remote Sensing Data-Driven Hierarchies Reis, Mariane; Escada, Maria; Dutra, Luciano Land, Vol. 7, Issue 2 https://doi.org/10.3390/land7020065	journal	May 2018
Component intensities to relate difference by category with difference overall Pontius, Robert Gilmore International Journal of Applied Earth Observation and Geoinformation, Vol. 77 https://doi.org/10.1016/j.jag.2018.07.024	journal	May 2019
An Evaluation of Bagging, Boosting, and Random Forests for Land-Cover Classification in Cape Cod, Massachusetts, USA Ghimire, Bardan; Rogan, John; Galiano, Víctor Rodríguez GIScience & Remote Sensing, Vol. 49, Issue 5 https://doi.org/10.2747/1548-1603.49.5.623	journal	September 2012
Improved land cover mapping using high resolution multiangle 8-band WorldView-2 satellite remote sensing data Jawak, Shridhar D.; Luis, Alvarinho J. Journal of Applied Remote Sensing, Vol. 7, Issue 1 https://doi.org/10.1117/1.JRS.7.073573	journal	January 2013
On the Use of Unmanned Aerial Systems for Environmental Monitoring Manfreda, Salvatore; McCabe, Matthew; Miller, Pauline Remote Sensing, Vol. 10, Issue 4 https://doi.org/10.3390/rs10040641	journal	April 2018
Evaluating effectiveness of down-sampling for stratified designs and unbalanced prevalence in Random Forest models of tree species distributions in Nevada Freeman, Elizabeth A.; Moisen, Gretchen G.; Frescino, Tracey S. Ecological Modelling, Vol. 233 https://doi.org/10.1016/j.ecolmodel.2012.03.007	journal	May 2012
While Boolean sets non-gently rip: A theoretical framework on fuzzy sets for mapping landscape patterns Rocchini, Duccio Ecological Complexity, Vol. 7, Issue 1 https://doi.org/10.1016/j.ecocom.2009.08.002	journal	March 2010
Supervised methods of image segmentation accuracy assessment in land cover mapping Costa, Hugo; Foody, Giles M.; Boyd, Doreen S. Remote Sensing of Environment, Vol. 205 https://doi.org/10.1016/j.rse.2017.11.024	journal	February 2018
Extended triple collocation: Estimating errors and correlation coefficients with respect to an unknown target: EXTENDED TRIPLE COLLOCATION McColl, Kaighin A.; Vogelzang, Jur; Konings, Alexandra G. Geophysical Research Letters, Vol. 41, Issue 17 https://doi.org/10.1002/2014GL061322	journal	September 2014
An assessment of the effectiveness of a random forest classifier for land-cover classification Rodriguez-Galiano, V. F.; Ghimire, B.; Rogan, J. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 67 https://doi.org/10.1016/j.isprsjprs.2011.11.002	journal	January 2012
Mapping vegetation in a heterogeneous mountain rangeland using landsat data: an alternative method to define and classify land-cover units Cingolani, A. Remote Sensing of Environment, Vol. 92, Issue 1 https://doi.org/10.1016/j.rse.2004.05.008	journal	July 2004
Implementation of machine-learning classification in remote sensing: an applied review Maxwell, Aaron E.; Warner, Timothy A.; Fang, Fang International Journal of Remote Sensing, Vol. 39, Issue 9 https://doi.org/10.1080/01431161.2018.1433343	journal	January 2018
Multiscale analysis and validation of the MODIS LAI productI. Uncertainty assessment Tian, Y.; Woodcock, C.; Wang, Y. Remote Sensing of Environment, Vol. 83, Issue 3 https://doi.org/10.1016/S0034-4257(02)00047-0	journal	December 2002
Sub-pixel confusion–uncertainty matrix for assessing soft classifications Silván-Cárdenas, J. L.; Wang, L. Remote Sensing of Environment, Vol. 112, Issue 3 https://doi.org/10.1016/j.rse.2007.07.017	journal	March 2008
The factor of scale in remote sensing Woodcock, Curtis E.; Strahler, Alan H. Remote Sensing of Environment, Vol. 21, Issue 3 https://doi.org/10.1016/0034-4257(87)90015-0	journal	April 1987
Ground reference data error and the mis-estimation of the area of land cover change as a function of its abundance Foody, Giles M. Remote Sensing Letters, Vol. 4, Issue 8 https://doi.org/10.1080/2150704X.2013.798708	journal	August 2013
Radiative forcing and temperature response to changes in urban albedos and associated CO ₂ offsets Menon, Surabi; Akbari, Hashem; Mahanama, Sarith Environmental Research Letters, Vol. 5, Issue 1 https://doi.org/10.1088/1748-9326/5/1/014005	journal	January 2010
An assessment of support vector machines for land cover classification Huang, C.; Davis, L. S.; Townshend, J. R. G. International Journal of Remote Sensing, Vol. 23, Issue 4 https://doi.org/10.1080/01431160110040323	journal	January 2002
Land cover mapping using time series HJ-1/CCD data Zhong, Bo; Ma, Peng; Nie, AiHua Science China Earth Sciences, Vol. 57, Issue 8 https://doi.org/10.1007/s11430-014-4877-5	journal	May 2014
A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists Shen, Chaopeng Water Resources Research, Vol. 54, Issue 11 https://doi.org/10.1029/2018WR022643	journal	November 2018
ImageNet Large Scale Visual Recognition Challenge Russakovsky, Olga; Deng, Jia; Su, Hao International Journal of Computer Vision, Vol. 115, Issue 3 https://doi.org/10.1007/s11263-015-0816-y	journal	April 2015
Components of information for multiple resolution comparison between maps that share a real variable Pontius, Robert Gilmore; Thontteh, Olufunmilayo; Chen, Hao Environmental and Ecological Statistics, Vol. 15, Issue 2 https://doi.org/10.1007/s10651-007-0043-y	journal	October 2007
Statistics notes: Measurement error Bland, J. M.; Altman, D. G. BMJ, Vol. 312, Issue 7047 https://doi.org/10.1136/bmj.312.7047.1654	journal	June 1996
Land change for all municipalities in Latin America and the Caribbean assessed from 250-m MODIS imagery (2001–2010) Clark, Matthew L.; Aide, T. Mitchell; Riner, George Remote Sensing of Environment, Vol. 126 https://doi.org/10.1016/j.rse.2012.08.013	journal	November 2012
Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS): Assessing the accuracy of distribution models Allouche, Omri; Tsoar, Asaf; Kadmon, Ronen Journal of Applied Ecology, Vol. 43, Issue 6 https://doi.org/10.1111/j.1365-2664.2006.01214.x	journal	September 2006
Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data Graves, Sarah; Asner, Gregory; Martin, Roberta Remote Sensing, Vol. 8, Issue 2 https://doi.org/10.3390/rs8020161	journal	February 2016
Nominal 30-m Cropland Extent Map of Continental Africa by Integrating Pixel-Based and Object-Based Algorithms Using Sentinel-2 and Landsat-8 Data on Google Earth Engine Xiong, Jun; Thenkabail, Prasad; Tilton, James Remote Sensing, Vol. 9, Issue 10 https://doi.org/10.3390/rs9101065	journal	October 2017
Land cover mapping of the tropical savanna region in Brazil Sano, Edson E.; Rosa, Roberto; Brito, Jorge L. S. Environmental Monitoring and Assessment, Vol. 166, Issue 1-4 https://doi.org/10.1007/s10661-009-0988-4	journal	June 2009
A generalized computer vision approach to mapping crop fields in heterogeneous agricultural landscapes Debats, Stephanie R.; Luo, Dee; Estes, Lyndon D. Remote Sensing of Environment, Vol. 179 https://doi.org/10.1016/j.rse.2016.03.010	journal	June 2016
A global dataset of crowdsourced land cover and land use reference data Fritz, Steffen; See, Linda; Perger, Christoph Scientific Data, Vol. 4, Issue 1 https://doi.org/10.1038/sdata.2017.75	journal	June 2017
Importance of Matrix Construction for Multiple-Resolution Categorical Map Comparison Kuzera, Kristopher; Pontius, Robert Gilmore GIScience & Remote Sensing, Vol. 45, Issue 3 https://doi.org/10.2747/1548-1603.45.3.249	journal	July 2008
Accounting for urban biogenic fluxes in regional carbon budgets Hardiman, Brady S.; Wang, Jonathan A.; Hutyra, Lucy R. Science of The Total Environment, Vol. 592 https://doi.org/10.1016/j.scitotenv.2017.03.028	journal	August 2017
Estimating the global distribution of field size using crowdsourcing Lesiv, Myroslava; Laso Bayas, Juan Carlos; See, Linda Global Change Biology, Vol. 25, Issue 1 https://doi.org/10.1111/gcb.14492	journal	November 2018
The total operating characteristic to measure diagnostic ability for multiple thresholds Pontius Jr, Robert Gilmore; Si, Kangping International Journal of Geographical Information Science, Vol. 28, Issue 3 https://doi.org/10.1080/13658816.2013.862623	journal	November 2013
Optimizing Remotely Sensed Solutions for Monitoring, Modeling, and Managing Coastal Environments Phinn, Stuart R.; Menges, Carl; Hill, Greg J. E. Remote Sensing of Environment, Vol. 73, Issue 2 https://doi.org/10.1016/S0034-4257(00)00087-0	journal	August 2000
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification Helber, Patrick; Bischke, Benjamin; Dengel, Andreas IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 12, Issue 7 https://doi.org/10.1109/JSTARS.2019.2918242	journal	July 2019
What is Land Cover? Comber, Alexis; Fisher, Peter; Wadsworth, Richard Environment and Planning B: Planning and Design, Vol. 32, Issue 2 https://doi.org/10.1068/b31135	journal	April 2005
Uncovering Ecological Patterns with Convolutional Neural Networks Brodrick, Philip G.; Davies, Andrew B.; Asner, Gregory P. Trends in Ecology & Evolution, Vol. 34, Issue 8 https://doi.org/10.1016/j.tree.2019.03.006	journal	August 2019
High-resolution mapping of global surface water and its long-term changes Pekel, Jean-François; Cottam, Andrew; Gorelick, Noel Nature, Vol. 540, Issue 7633 https://doi.org/10.1038/nature20584	journal	December 2016
Learning Multiscale Deep Features for High-Resolution Satellite Image Scene Classification Liu, Qingshan; Hang, Renlong; Song, Huihui IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, Issue 1 https://doi.org/10.1109/TGRS.2017.2743243	journal	January 2018
Mapping and interpreting historical land cover/land use changes in a Natura 2000 site using earth observational data: The case of Nestos delta, Greece Mallinis, Georgios; Emmanoloudis, Dimitrios; Giannakopoulos, Vasileios Applied Geography, Vol. 31, Issue 1 https://doi.org/10.1016/j.apgeog.2010.07.002	journal	January 2011
Mapping Urban Land Use by Using Landsat Images and Open Social Data Hu, Tengyun; Yang, Jun; Li, Xuecao Remote Sensing, Vol. 8, Issue 2 https://doi.org/10.3390/rs8020151	journal	February 2016
Ambiguities inherent in sums-of-squares-based error statistics Willmott, Cort J.; Matsuura, Kenji; Robeson, Scott M. Atmospheric Environment, Vol. 43, Issue 3 https://doi.org/10.1016/j.atmosenv.2008.10.005	journal	January 2009
Fully fuzzy supervised classification of land cover from remotely sensed imagery with an artificial neural network Foody, G. M. Neural Computing & Applications, Vol. 5, Issue 4 https://doi.org/10.1007/BF01424229	journal	December 1997
Global land change from 1982 to 2016 Song, Xiao-Peng; Hansen, Matthew C.; Stehman, Stephen V. Nature, Vol. 560, Issue 7720 https://doi.org/10.1038/s41586-018-0411-9	journal	August 2018
Key issues in rigorous accuracy assessment of land cover products Stehman, Stephen V.; Foody, Giles M. Remote Sensing of Environment, Vol. 231 https://doi.org/10.1016/j.rse.2019.05.018	journal	September 2019
Learning Aerial Image Segmentation From Online Maps Kaiser, Pascal; Wegner, Jan Dirk; Lucchi, Aurelien IEEE Transactions on Geoscience and Remote Sensing, Vol. 55, Issue 11 https://doi.org/10.1109/TGRS.2017.2719738	journal	November 2017
Geo-Wiki: An online platform for improving global land cover Fritz, Steffen; McCallum, Ian; Schill, Christian Environmental Modelling & Software, Vol. 31 https://doi.org/10.1016/j.envsoft.2011.11.015	journal	May 2012
Global forecasts of urban expansion to 2030 and direct impacts on biodiversity and carbon pools Seto, K. C.; Guneralp, B.; Hutyra, L. R. Proceedings of the National Academy of Sciences, Vol. 109, Issue 40 https://doi.org/10.1073/pnas.1211658109	journal	September 2012
Lessons learned implementing an operational continuous United States national land change monitoring capability: The Land Change Monitoring, Assessment, and Projection (LCMAP) approach Brown, Jesslyn F.; Tollerud, Heather J.; Barber, Christopher P. Remote Sensing of Environment, Vol. 238 https://doi.org/10.1016/j.rse.2019.111356	journal	March 2020
Validating MODIS Terrestrial Ecology Products Cohen, Warren B.; Justice, Christopher O. Remote Sensing of Environment, Vol. 70, Issue 1 https://doi.org/10.1016/S0034-4257(99)00053-X	journal	October 1999
Finer resolution observation and monitoring of global land cover: first mapping results with Landsat TM and ETM+ data Gong, Peng; Wang, Jie; Yu, Le International Journal of Remote Sensing, Vol. 34, Issue 7 https://doi.org/10.1080/01431161.2012.748992	journal	December 2012
Explaining the unsuitability of the kappa coefficient in the assessment and comparison of the accuracy of thematic maps obtained by image classification Foody, Giles M. Remote Sensing of Environment, Vol. 239 https://doi.org/10.1016/j.rse.2019.111630	journal	March 2020
Variability of operator performance in remote-sensing image interpretation: the importance of human and external factors Van Coillie, Frieke M. B.; Gardin, Soetkin; Anseel, Frederik International Journal of Remote Sensing, Vol. 35, Issue 2 https://doi.org/10.1080/01431161.2013.873152	journal	January 2014
Visualizing Uncertain Information MacEachren, Alan M. Cartographic Perspectives, Issue 13 https://doi.org/10.14714/CP13.1000	journal	June 1992
Design and Interpretation of Intensity Analysis Illustrated by Land Change in Central Kalimantan, Indonesia Pontius, Robert; Gao, Yan; Giner, Nicholas Land, Vol. 2, Issue 3 https://doi.org/10.3390/land2030351	journal	July 2013
Assessing the global warming potential of human settlement expansion in a mesic temperate landscape from 2005 to 2050 Reinmann, Andrew B.; Hutyra, Lucy R.; Trlica, Andrew Science of The Total Environment, Vol. 545-546 https://doi.org/10.1016/j.scitotenv.2015.12.033	journal	March 2016
Land Use and Land Cover Mapping in the Brazilian Amazon Using Polarimetric Airborne P-Band SAR Data Freitas, Corina da Costa; Soler, Luciana de Souza; Sant'Anna, Sidnei JoÃo Siqueira IEEE Transactions on Geoscience and Remote Sensing, Vol. 46, Issue 10 https://doi.org/10.1109/TGRS.2008.2000630	journal	October 2008
Using volunteered geographic information (VGI) in design-based statistical inference for area estimation and accuracy assessment of land cover Stehman, Stephen V.; Fonte, Cidália C.; Foody, Giles M. Remote Sensing of Environment, Vol. 212 https://doi.org/10.1016/j.rse.2018.04.014	journal	June 2018
The effects of imperfect reference data on remote sensing-assisted estimators of land cover class proportions McRoberts, Ronald E.; Stehman, Stephen V.; Liknes, Greg C. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 142 https://doi.org/10.1016/j.isprsjprs.2018.06.002	journal	August 2018
Review article Synergy in remote sensing-what's in a pixel? Cracknell, A. P. International Journal of Remote Sensing, Vol. 19, Issue 11 https://doi.org/10.1080/014311698214848	journal	January 1998
Uncertainty analysis for image interpretations of urban slums Kohli, Divyani; Stein, Alfred; Sliuzas, Richard Computers, Environment and Urban Systems, Vol. 60 https://doi.org/10.1016/j.compenvurbsys.2016.07.010	journal	November 2016
A large-area, spatially continuous assessment of land cover map error and its impact on downstream analyses Estes, Lyndon; Chen, Peng; Debats, Stephanie Global Change Biology, Vol. 24, Issue 1 https://doi.org/10.1111/gcb.13904	journal	October 2017
Collect Earth: Land Use and Land Cover Assessment through Augmented Visual Interpretation Bey, Adia; Sánchez-Paus Díaz, Alfonso; Maniatis, Danae Remote Sensing, Vol. 8, Issue 10 https://doi.org/10.3390/rs8100807	journal	September 2016
The dimensions of global urban expansion: Estimates and projections for all countries, 2000–2050 Angel, Shlomo; Parent, Jason; Civco, Daniel L. Progress in Planning, Vol. 75, Issue 2 https://doi.org/10.1016/j.progress.2011.04.001	journal	February 2011
The spatial and temporal domains of modern ecology Estes, Lyndon; Elsen, Paul R.; Treuer, Timothy Nature Ecology & Evolution, Vol. 2, Issue 5 https://doi.org/10.1038/s41559-018-0524-4	journal	April 2018
Hierarchical mapping of annual global land cover 2001 to present: The MODIS Collection 6 Land Cover product Sulla-Menashe, Damien; Gray, Josh M.; Abercrombie, S. Parker Remote Sensing of Environment, Vol. 222 https://doi.org/10.1016/j.rse.2018.12.013	journal	March 2019
Intensity analysis to unify measurements of size and stationarity of land changes by interval, category, and transition Aldwaik, Safaa Zakaria; Pontius, Robert Gilmore Landscape and Urban Planning, Vol. 106, Issue 1 https://doi.org/10.1016/j.landurbplan.2012.02.010	journal	May 2012
A critical synthesis of remotely sensed optical image change detection techniques Tewkesbury, Andrew P.; Comber, Alexis J.; Tate, Nicholas J. Remote Sensing of Environment, Vol. 160 https://doi.org/10.1016/j.rse.2015.01.006	journal	April 2015
Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment Pontius, Robert Gilmore; Millones, Marco International Journal of Remote Sensing, Vol. 32, Issue 15 https://doi.org/10.1080/01431161.2011.552923	journal	August 2011
Evaluation of the VIIRS BRDF, Albedo and NBAR products suite and an assessment of continuity with the long term MODIS record Liu, Yan; Wang, Zhuosen; Sun, Qingsong Remote Sensing of Environment, Vol. 201 https://doi.org/10.1016/j.rse.2017.09.020	journal	November 2017
Land use and land cover change in Greater Dhaka, Bangladesh: Using remote sensing to promote sustainable urbanization Dewan, Ashraf M.; Yamaguchi, Yasushi Applied Geography, Vol. 29, Issue 3 https://doi.org/10.1016/j.apgeog.2008.12.005	journal	July 2009
Evaluation of land surface phenology from VIIRS data using time series of PhenoCam imagery Zhang, Xiaoyang; Jayavelu, Senthilnath; Liu, Lingling Agricultural and Forest Meteorology, Vol. 256-257 https://doi.org/10.1016/j.agrformet.2018.03.003	journal	June 2018
Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources Zhu, Xiao Xiang; Tuia, Devis; Mou, Lichao IEEE Geoscience and Remote Sensing Magazine, Vol. 5, Issue 4 https://doi.org/10.1109/MGRS.2017.2762307	journal	December 2017
Uncertainty in ecosystem mapping by remote sensing Rocchini, Duccio; Foody, Giles M.; Nagendra, Harini Computers & Geosciences, Vol. 50 https://doi.org/10.1016/j.cageo.2012.05.022	journal	January 2013
Integrating OpenStreetMap crowdsourced data and Landsat time-series imagery for rapid land use/land cover (LULC) mapping: Case study of the Laguna de Bay area of the Philippines Johnson, Brian A.; Iizuka, Kotaro Applied Geography, Vol. 67 https://doi.org/10.1016/j.apgeog.2015.12.006	journal	February 2016
Global vegetation phenology from Moderate Resolution Imaging Spectroradiometer (MODIS): Evaluation of global patterns and comparison with in situ measurements: GLOBAL PHENOLOGY FROM MODIS Zhang, Xiaoyang; Friedl, Mark A.; Schaaf, Crystal B. Journal of Geophysical Research: Biogeosciences, Vol. 111, Issue G4 https://doi.org/10.1029/2006JG000217	journal	December 2006
How good is good enough? Data requirements for reliable crop yield simulations and yield-gap analysis Grassini, Patricio; van Bussel, Lenny G. J.; Van Wart, Justin Field Crops Research, Vol. 177 https://doi.org/10.1016/j.fcr.2015.03.004	journal	June 2015
Identifying Mislabeled Training Data Brodley, C. E.; Friedl, M. A. Journal of Artificial Intelligence Research, Vol. 11 https://doi.org/10.1613/jair.606	journal	July 1999
Smallholder crop area mapped with wall-to-wall WorldView sub-meter panchromatic image texture: A test case for Tigray, Ethiopia Neigh, Christopher S. R.; Carroll, Mark L.; Wooten, Margaret R. Remote Sensing of Environment, Vol. 212 https://doi.org/10.1016/j.rse.2018.04.025	journal	June 2018
Fuzzy set theory and thematic maps: accuracy assessment and area estimation Woodcock, Curtis E.; Gopal, Sucharita International Journal of Geographical Information Science, Vol. 14, Issue 2 https://doi.org/10.1080/136588100240895	journal	March 2000
Random Forests Breiman, Leo Machine Learning, Vol. 45, Issue 1, p. 5-32 https://doi.org/10.1023/A:1010933404324	journal	January 2001
Machine learning in geosciences and remote sensing Lary, David J.; Alavi, Amir H.; Gandomi, Amir H. Geoscience Frontiers, Vol. 7, Issue 1 https://doi.org/10.1016/j.gsf.2015.07.003	journal	January 2016
Geological mapping using remote sensing data: A comparison of five machine learning algorithms, their response to variations in the spatial distribution of training data and the use of explicit spatial information Cracknell, Matthew J.; Reading, Anya M. Computers & Geosciences, Vol. 63 https://doi.org/10.1016/j.cageo.2013.10.008	journal	February 2014
Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts See, Linda; Comber, Alexis; Salk, Carl PLoS ONE, Vol. 8, Issue 7 https://doi.org/10.1371/journal.pone.0069958	journal	July 2013
Sources of error in accuracy assessment of thematic land-cover maps in the Brazilian Amazon Powell, R. L.; Matzke, N.; de Souza, C. Remote Sensing of Environment, Vol. 90, Issue 2 https://doi.org/10.1016/j.rse.2003.12.007	journal	March 2004
Assessing different remote sensing techniques to detect land use/cover changes in the eastern Mediterranean Berberoglu, S.; Akin, A. International Journal of Applied Earth Observation and Geoinformation, Vol. 11, Issue 1 https://doi.org/10.1016/j.jag.2008.06.002	journal	February 2009
A global reference database of crowdsourced cropland data collected using the Geo-Wiki platform See, Linda PANGAEA - Data Publisher for Earth & Environmental Science https://doi.org/10.1594/pangaea.873912	dataset	January 2017
Implementation of machine-learning classification in remote sensing: an applied review Maxwell, Aaron E.; Warner, Timothy A.; Fang, Fang Taylor & Francis https://doi.org/10.6084/m9.figshare.5849532.v1	dataset	January 2018
Edge effects enhance carbon uptake and its vulnerability to climate change in temperate broadleaf forests University), Reinmann, Andrew (Boston Harvard Dataverse https://doi.org/10.7910/dvn/azgsqv	dataset	January 2016
Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding Sumbul, Gencer; Charfuelan, Marcela; Demir, Begum IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium https://doi.org/10.1109/igarss.2019.8900532	conference	July 2019
Estimating the Global Distribution of Field Size using Crowdsourcing Lesiv, Myroslava; Laso Bayas, Juan Carlos; See, Linda Zenodo https://doi.org/10.5281/zenodo.6651481	dataset	January 2018
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification Helber, Patrick; Bischke, Benjamin; Dengel, Andreas Zenodo https://doi.org/10.5281/zenodo.7711097	dataset	January 2018

Cited By (1)

Using satellite imagery to understand and promote sustainable development Burke, Marshall; Driscoll, Anne; Lobell, David B. arXiv https://doi.org/10.48550/arxiv.2010.06988	preprint	January 2020

Similar Records

Scaling Automatic Vector Data Alignment to Satellite Imagery

Conference · Sun Oct 01 00:00:00 EDT 2023 · OSTI ID:1608215

Potnis, Abhishek; Lunga, Dalton; Ambrozio Dias, Philipe; +3 more

Multispectral Thermal Imagery and Its Application to the Geologic Mapping of the Koobi Fora Formation, Northwestern Kenya

Thesis/Dissertation · Thu Dec 01 00:00:00 EST 2005 · OSTI ID:1608215

Green, Mary K.

Small Scale Field Test Demonstrating CO₂ Sequestration In Arbuckle Saline Aquifer And By CO₂-Eor At Wellington Field, Sumner County, Kansas

Technical Report · Sun Dec 31 00:00:00 EST 2017 · OSTI ID:1608215

Holubnyak, Yevhen Eugene; Watney, Lynn; Hollenbach, Jennifer; +11 more

Related Subjects

58 GEOSCIENCES
training data
machine learning
map accuracy
error propagation

Title: Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Citation Formats

References (192)

Cited By (1)

Similar Records

Related Subjects