Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Abstract

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping,more » global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.« less

Authors:
ORCiD logo [1]; ORCiD logo [2];  [3];  [3];  [4];  [5];  [6];  [7];  [8]; ORCiD logo [9]; ORCiD logo [10]; ORCiD logo [11]; ORCiD logo [4];  [12];  [4];  [4];  [12];  [4];  [13]; ORCiD logo [4]
+ Show Author Affiliations
  1. Clark Univ., Worcester, MA (United States); Univ. of Massachusetts, Boston (United States)
  2. Radiant Earth Foundation, San Francisco, CA (United States)
  3. Univ. of California, Santa Barbara, CA (United States)
  4. Clark Univ., Worcester, MA (United States)
  5. Azavea, Inc., Philadelphia, PA (United States)
  6. Boston Univ., MA (United States)
  7. Univ. of Michigan, Ann Arbor, MI (United States)
  8. Univ. of Twente, Enschede (The Netherlands)
  9. International Inst. for Applied Systems Analysis (IIASA), Laxenburg (Austria)
  10. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  11. Miami Univ., Oxford, OH (United States)
  12. City Univ. of New York (CUNY), NY (United States). Advanced Science Research Center; Hunter College, New York, NY (United States)
  13. Development Seed,Washington, DC (United States)
Publication Date:
Research Org.:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE; Omidyar Network’s Property Rights Initiative, now PlaceFund; National Aeronautics and Space Administration (NASA); National Science Foundation (NSF); National Institute of Standards and Technology (NIST); New York State Department of Environmental Conservation
OSTI Identifier:
1608215
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
Remote Sensing
Additional Journal Information:
Journal Volume: 12; Journal Issue: 6; Conference: Quantifying Error in Training Data for Mapping and Monitoring the Earth System, Worcester, MA (United States), 8-9 Jan 2019; Journal ID: ISSN 2072-4292
Publisher:
MDPI
Country of Publication:
United States
Language:
English
Subject:
58 GEOSCIENCES; training data; machine learning; map accuracy; error propagation

Citation Formats

Elmes, Arthur, Alemohammad, Hamed, Avery, Ryan, Caylor, Kelly, Eastman, J. Ronald, Fishgold, Lewis, Friedl, Mark A., Jain, Meha, Kohli, Divyani, Laso Bayas, Juan Carlos, Lunga, Dalton, McCarty, Jessica L., Pontius, Jr., Robert Gilmore, Reinmann, Andrew B., Rogan, John, Song, Lei, Stoynova, Hristiana, Ye, Su, Yi, Zhuang-Fang, and Estes, Lyndon. Accounting for Training Data Error in Machine Learning Applied to Earth Observations. United States: N. p., 2020. Web. doi:10.3390/rs12061034.
Copy to clipboard
Elmes, Arthur, Alemohammad, Hamed, Avery, Ryan, Caylor, Kelly, Eastman, J. Ronald, Fishgold, Lewis, Friedl, Mark A., Jain, Meha, Kohli, Divyani, Laso Bayas, Juan Carlos, Lunga, Dalton, McCarty, Jessica L., Pontius, Jr., Robert Gilmore, Reinmann, Andrew B., Rogan, John, Song, Lei, Stoynova, Hristiana, Ye, Su, Yi, Zhuang-Fang, & Estes, Lyndon. Accounting for Training Data Error in Machine Learning Applied to Earth Observations. United States. https://doi.org/10.3390/rs12061034
Copy to clipboard
Elmes, Arthur, Alemohammad, Hamed, Avery, Ryan, Caylor, Kelly, Eastman, J. Ronald, Fishgold, Lewis, Friedl, Mark A., Jain, Meha, Kohli, Divyani, Laso Bayas, Juan Carlos, Lunga, Dalton, McCarty, Jessica L., Pontius, Jr., Robert Gilmore, Reinmann, Andrew B., Rogan, John, Song, Lei, Stoynova, Hristiana, Ye, Su, Yi, Zhuang-Fang, and Estes, Lyndon. Mon . "Accounting for Training Data Error in Machine Learning Applied to Earth Observations". United States. https://doi.org/10.3390/rs12061034. https://www.osti.gov/servlets/purl/1608215.
Copy to clipboard
@article{osti_1608215,
title = {Accounting for Training Data Error in Machine Learning Applied to Earth Observations},
author = {Elmes, Arthur and Alemohammad, Hamed and Avery, Ryan and Caylor, Kelly and Eastman, J. Ronald and Fishgold, Lewis and Friedl, Mark A. and Jain, Meha and Kohli, Divyani and Laso Bayas, Juan Carlos and Lunga, Dalton and McCarty, Jessica L. and Pontius, Jr., Robert Gilmore and Reinmann, Andrew B. and Rogan, John and Song, Lei and Stoynova, Hristiana and Ye, Su and Yi, Zhuang-Fang and Estes, Lyndon},
abstractNote = {Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.},
doi = {10.3390/rs12061034},
journal = {Remote Sensing},
number = 6,
volume = 12,
place = {United States},
year = {Mon Mar 23 00:00:00 EDT 2020},
month = {Mon Mar 23 00:00:00 EDT 2020}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.3390/rs12061034
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 25 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Influence of carbon mapping and land change modelling on the prediction of carbon emissions from deforestation
journal, June 2012

  • Gutierrez-Velez, Victor Hugo; Pontius, Robert Gilmore
  • Environmental Conservation, Vol. 39, Issue 4
  • DOI: 10.1017/S0376892912000173

Orthorectification of VHR optical satellite data exploiting the geometric accuracy of TerraSAR-X data
journal, January 2011

  • Reinartz, Peter; Müller, Rupert; Schwind, Peter
  • ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 66, Issue 1
  • DOI: 10.1016/j.isprsjprs.2010.10.003

Toward intelligent training of supervised image classifications: directing training data acquisition for SVM classification
journal, October 2004

Global land cover classifications at 8 km spatial resolution: The use of training data derived from Landsat imagery in decision tree classifiers
journal, January 1998

  • De Fries, R. S.; Hansen, M.; Townshend, J. R. G.
  • International Journal of Remote Sensing, Vol. 19, Issue 16
  • DOI: 10.1080/014311698214235

The impact of imperfect ground reference data on the accuracy of land cover change estimation
journal, June 2009

Citizens as sensors: the world of volunteered geography
journal, November 2007

Object based image analysis for remote sensing
journal, January 2010

A Pixel-Based Landsat Compositing Algorithm for Large Area Land Cover Mapping
journal, October 2013

  • Griffiths, Patrick; van der Linden, Sebastian; Kuemmerle, Tobias
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 6, Issue 5
  • DOI: 10.1109/JSTARS.2012.2228167

Exploring the Use of Google Earth Imagery and Object-Based Methods in Land Use/Cover Mapping
journal, November 2013

  • Hu, Qiong; Wu, Wenbin; Xia, Tian
  • Remote Sensing, Vol. 5, Issue 11
  • DOI: 10.3390/rs5116026

Artificial neural network response to mixed pixels in coarse-resolution satellite data
journal, December 1996

On the nature of models in remote sensing
journal, October 1986

Mapping land-cover modifications over large areas: A comparison of machine learning algorithms
journal, May 2008

Deep learning in remote sensing applications: A meta-analysis and review
journal, June 2019

Status of land cover classification accuracy assessment
journal, April 2002

Satellite-based assessment of yield variation and its determinants in smallholder African systems
journal, February 2017

  • Burke, Marshall; Lobell, David B.
  • Proceedings of the National Academy of Sciences, Vol. 114, Issue 9
  • DOI: 10.1073/pnas.1616919114

A survey of image classification methods and techniques for improving classification performance
journal, March 2007

Exploring diversity in ensemble classification: Applications in large area land cover mapping
journal, July 2017

Recommendations for using the relative operating characteristic (ROC)
journal, January 2014

New refinements and validation of the MODIS Land-Surface Temperature/Emissivity products
journal, January 2008

A Novel Context-Sensitive Semisupervised SVM Classifier Robust to Mislabeled Training Samples
journal, July 2009

  • Bruzzone, L.; Persello, C.
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 47, Issue 7
  • DOI: 10.1109/TGRS.2008.2011983

High-Resolution Global Maps of 21st-Century Forest Cover Change
journal, November 2013

Mapping global cropland and field size
journal, January 2015

  • Fritz, Steffen; See, Linda; McCallum, Ian
  • Global Change Biology, Vol. 21, Issue 5
  • DOI: 10.1111/gcb.12838

Automatic identification of building types based on topographic databases – a comparison of different data sources
journal, January 2015

  • Hecht, Robert; Meinel, Gotthard; Buchroithner, Manfred
  • International Journal of Cartography, Vol. 1, Issue 1
  • DOI: 10.1080/23729333.2015.1055644

Land cover mapping of wetland areas in an agricultural landscape using SAR and Landsat imagery
journal, May 2009

Harshness in image classification accuracy assessment
journal, May 2008

Large-scale land cover mapping with the integration of multi-source information based on the Dempster–Shafer theory
journal, January 2012

Classification in the Presence of Label Noise: A Survey
journal, May 2014

  • Frenay, Benoit; Verleysen, Michel
  • IEEE Transactions on Neural Networks and Learning Systems, Vol. 25, Issue 5
  • DOI: 10.1109/TNNLS.2013.2292894

First operational BRDF, albedo nadir reflectance products from MODIS
journal, November 2002

Comparison of support vector machine, neural network, and CART algorithms for the land-cover classification using limited training data points
journal, June 2012

The Sensitivity of Mapping Methods to Reference Data Quality: Training Supervised Image Classifications with Imperfect Reference Data
journal, November 2016

  • Foody, Giles; Pal, Mahesh; Rocchini, Duccio
  • ISPRS International Journal of Geo-Information, Vol. 5, Issue 11
  • DOI: 10.3390/ijgi5110199

A neural network approach using multi-scale textural metrics from very high-resolution panchromatic imagery for urban land-use classification
journal, June 2009

  • Pacifici, Fabio; Chini, Marco; Emery, William J.
  • Remote Sensing of Environment, Vol. 113, Issue 6
  • DOI: 10.1016/j.rse.2009.02.014

Assessing geometric accuracy of the orthorectification process from GeoEye-1 and WorldView-2 panchromatic images
journal, April 2013

  • Aguilar, Manuel A.; Saldaña, María del Mar; Aguilar, Fernando J.
  • International Journal of Applied Earth Observation and Geoinformation, Vol. 21
  • DOI: 10.1016/j.jag.2012.06.004

Effect of errors in ground truth on classification accuracy
journal, August 2009

Spatiotemporal characteristics, patterns, and causes of land-use changes in China since the late 1980s
journal, January 2014

  • Liu, Jiyuan; Kuang, Wenhui; Zhang, Zengxiang
  • Journal of Geographical Sciences, Vol. 24, Issue 2
  • DOI: 10.1007/s11442-014-1082-6

Good practices for estimating area and assessing accuracy of land change
journal, May 2014

A review of accuracy assessment for object-based image analysis: From per-pixel to per-polygon approaches
journal, July 2018

Remote sensing of impervious surfaces in the urban areas: Requirements, methods, and trends
journal, February 2012

A scalable satellite-based crop yield mapper
journal, July 2015

Conflation of expert and crowd reference data to validate global binary thematic maps
journal, February 2019

  • Waldner, François; Schucknecht, Anne; Lesiv, Myroslava
  • Remote Sensing of Environment, Vol. 221
  • DOI: 10.1016/j.rse.2018.10.039

Towards a set of agrosystem-specific cropland mapping methods to address the global cropland diversity
journal, June 2016

  • Waldner, François; De Abelleyra, Diego; Verón, Santiago R.
  • International Journal of Remote Sensing, Vol. 37, Issue 14
  • DOI: 10.1080/01431161.2016.1194545

Making better use of accuracy data in land change studies: Estimating accuracy and area and quantifying uncertainty using stratified estimation
journal, February 2013

  • Olofsson, Pontus; Foody, Giles M.; Stehman, Stephen V.
  • Remote Sensing of Environment, Vol. 129
  • DOI: 10.1016/j.rse.2012.10.031

Utilizing Temporally Invariant Calibration Sites to Classify Multiple Dates and Types of Satellite Imagery
journal, February 2011

  • Fortier, Joe; Rogan, John; Woodcock, Curtis E.
  • Photogrammetric Engineering & Remote Sensing, Vol. 77, Issue 2
  • DOI: 10.14358/PERS.77.2.181

Sub-pixel land-cover mapping with improved fraction images upon multiple-point simulation
journal, June 2013

On the use of dimensioned measures of error to evaluate the performance of spatial interpolators
journal, January 2006

  • Willmott, C. J.; Matsuura, K.
  • International Journal of Geographical Information Science, Vol. 20, Issue 1
  • DOI: 10.1080/13658810500286976

Using active learning to adapt remote sensing image classifiers
journal, September 2011

Carbon consequences of land cover change and expansion of urban lands: A case study in the Seattle metropolitan region
journal, October 2011

Incorporating Land Use Mapping and Participation in Jordan: An Approach to Sustainable Management of Two Mountainous Areas
journal, February 2008

  • Al-Bakri, Jawad Taleb; Ajlouni, Mohammad; Abu-Zanat, Mahfouz
  • Mountain Research and Development, Vol. 28, Issue 1
  • DOI: 10.1659/mrd.0863

Comparison of methods for land-use classification incorporating remote sensing and GIS inputs
journal, April 2011

A fuzzy classification of sub-urban land cover from remotely sensed imagery
journal, January 1998

The modifiable areal unit problem and implications for landscape ecology
journal, June 1996

  • Jelinski, Dennis E.; Wu, Jianguo
  • Landscape Ecology, Vol. 11, Issue 3
  • DOI: 10.1007/BF02447512

Criteria to Confirm Models that Simulate Deforestation and Carbon Disturbance
journal, September 2018

Comment on “Tropical forests are a net carbon source based on aboveground measurements of gain and loss”
journal, January 2019

  • Hansen, Matthew C.; Potapov, Peter; Tyukavina, Alexandra
  • Science, Vol. 363, Issue 6423
  • DOI: 10.1126/science.aar3629

Sampling designs for accuracy assessment of land cover
journal, September 2009

A platform for crowdsourcing the creation of representative, accurate landcover maps
journal, June 2016

A framework for selecting appropriate remotely sensed data dimensions for environmental monitoring and management
journal, November 1998

Capturing rapid land surface dynamics with Collection V006 MODIS BRDF/NBAR/Albedo (MCD43) products
journal, March 2018

A review of assessing the accuracy of classifications of remotely sensed data
journal, July 1991

Spatio-temporal dynamics and evolution of land use change and landscape pattern in response to rapid urbanization
journal, September 2009

Challenges in using land use and land cover data for global change studies: LAND USE AND LAND COVER DATA FOR GLOBAL CHANGE STUDIES
journal, January 2011

Accuracy of forest inventory mapping: Some implications for boreal forest management
journal, November 2007

  • Thompson, Ian D.; Maher, Scott C.; Rouillard, Daniel P.
  • Forest Ecology and Management, Vol. 252, Issue 1-3
  • DOI: 10.1016/j.foreco.2007.06.033

Development of a global land cover characteristics database and IGBP DISCover from 1 km AVHRR data
journal, January 2000

  • Loveland, T. R.; Reed, B. C.; Brown, J. F.
  • International Journal of Remote Sensing, Vol. 21, Issue 6-7
  • DOI: 10.1080/014311600210191

Land Surface Temperature Retrieval Methods From Landsat-8 Thermal Infrared Sensor Data
journal, October 2014

  • Jimenez-Munoz, Juan C.; Sobrino, Jose A.; Skokovic, Drazen
  • IEEE Geoscience and Remote Sensing Letters, Vol. 11, Issue 10
  • DOI: 10.1109/LGRS.2014.2312032

An Enhanced TIMESAT Algorithm for Estimating Vegetation Phenology Metrics From MODIS Data
journal, June 2011

  • Tan, B.; Morisette, J. T.; Wolfe, R. E.
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 4, Issue 2
  • DOI: 10.1109/JSTARS.2010.2075916

A global map of rainfed cropland areas (GMRCA) at the end of last millennium using remote sensing
journal, April 2009

  • Biradar, Chandrashekhar M.; Thenkabail, Prasad S.; Noojipady, Praveen
  • International Journal of Applied Earth Observation and Geoinformation, Vol. 11, Issue 2
  • DOI: 10.1016/j.jag.2008.11.002

Land cover change assessment using decision trees, support vector machines and maximum likelihood classification algorithms
journal, February 2010

  • Otukei, J. R.; Blaschke, T.
  • International Journal of Applied Earth Observation and Geoinformation, Vol. 12
  • DOI: 10.1016/j.jag.2009.11.002

Smallholder maize area and yield mapping at national scales with Google Earth Engine
journal, July 2019

Cartography: uncertainty, interventions, and dynamic display
journal, June 2006

Assessing the accuracy of land cover change with imperfect ground reference data
journal, October 2010

A Coefficient of Agreement for Nominal Scales
journal, April 1960

A global reference database of crowdsourced cropland data collected using the Geo-Wiki platform
journal, September 2017

  • Laso Bayas, Juan Carlos; Lesiv, Myroslava; Waldner, François
  • Scientific Data, Vol. 4, Issue 1
  • DOI: 10.1038/sdata.2017.136

Modeling Percent Tree Canopy Cover: A Pilot Study
journal, July 2012

  • Coulston, John W.; Moisen, Gretchen G.; Wilson, Barry T.
  • Photogrammetric Engineering & Remote Sensing, Vol. 78, Issue 7
  • DOI: 10.14358/PERS.78.7.715

MODIS Collection 5 global land cover: Algorithm refinements and characterization of new datasets
journal, January 2010

  • Friedl, Mark A.; Sulla-Menashe, Damien; Tan, Bin
  • Remote Sensing of Environment, Vol. 114, Issue 1
  • DOI: 10.1016/j.rse.2009.08.016

Edge effects enhance carbon uptake and its vulnerability to climate change in temperate broadleaf forests
journal, December 2016

  • Reinmann, Andrew B.; Hutyra, Lucy R.
  • Proceedings of the National Academy of Sciences, Vol. 114, Issue 1
  • DOI: 10.1073/pnas.1612369114

Global characterization and monitoring of forest cover using Landsat data: opportunities and challenges
journal, August 2012

  • Townshend, John R.; Masek, Jeffrey G.; Huang, Chengquan
  • International Journal of Digital Earth, Vol. 5, Issue 5
  • DOI: 10.1080/17538947.2012.713190

Soil Moisture Remote Sensing: State-of-the-Science
journal, January 2017

  • Mohanty, Binayak P.; Cosh, Michael H.; Lakshmi, Venkat
  • Vadose Zone Journal, Vol. 16, Issue 1
  • DOI: 10.2136/vzj2016.10.0105

A generalized cross‐tabulation matrix to compare soft‐classified maps at multiple resolutions
journal, January 2006

  • Pontius, R. G.; Cheuk, M. L.
  • International Journal of Geographical Information Science, Vol. 20, Issue 1
  • DOI: 10.1080/13658810500391024

A review of methods for the assessment of prediction errors in conservation presence/absence models
journal, March 1997

From Guesstimates to GPStimates: Land Area Measurement and Implications for Agricultural Analysis
journal, May 2015

  • Carletto, Calogero; Gourlay, Sydney; Winters, Paul
  • Journal of African Economies, Vol. 24, Issue 5
  • DOI: 10.1093/jae/ejv011

Combining satellite imagery and machine learning to predict poverty
journal, August 2016

Range of Categorical Associations for Comparison of Maps with Mixed Pixels
journal, August 2009

  • Pontius, Jr., Robert Gilmore; Connors, John
  • Photogrammetric Engineering & Remote Sensing, Vol. 75, Issue 8
  • DOI: 10.14358/PERS.75.8.963

Crowdsourcing In-Situ Data on Land Cover and Land Use Using Gamification and Mobile Technology
journal, November 2016

  • Laso Bayas, Juan; See, Linda; Fritz, Steffen
  • Remote Sensing, Vol. 8, Issue 11
  • DOI: 10.3390/rs8110905

Quality control and assessment of interpreter consistency of annual land cover reference data in an operational national monitoring program
journal, March 2020

  • Pengra, Bruce W.; Stehman, Stephen V.; Horton, Josephine A.
  • Remote Sensing of Environment, Vol. 238
  • DOI: 10.1016/j.rse.2019.111261

Hybrid object-based approach for land use/land cover mapping using high spatial resolution imagery
journal, June 2011

  • Malinverni, Eva Savina; Tassetti, Anna Nora; Mancini, Adriano
  • International Journal of Geographical Information Science, Vol. 25, Issue 6
  • DOI: 10.1080/13658816.2011.566569

Recent Advances in the Measurement Error Literature
journal, October 2016

A scalable approach to mapping annual land cover at 250 m using MODIS time series data: A case study in the Dry Chaco ecoregion of South America
journal, November 2010

  • Clark, Matthew L.; Aide, T. Mitchell; Grau, H. Ricardo
  • Remote Sensing of Environment, Vol. 114, Issue 11
  • DOI: 10.1016/j.rse.2010.07.001

Global land cover mapping at 30m resolution: A POK-based operational approach
journal, May 2015

Detecting trends in forest disturbance and recovery using yearly Landsat time series: 1. LandTrendr — Temporal segmentation algorithms
journal, December 2010

  • Kennedy, Robert E.; Yang, Zhiqiang; Cohen, Warren B.
  • Remote Sensing of Environment, Vol. 114, Issue 12
  • DOI: 10.1016/j.rse.2010.07.008

Long-term land cover dynamics by multi-temporal classification across the Landsat-5 record
journal, January 2013

  • Sexton, Joseph O.; Urban, Dean L.; Donohue, Michael J.
  • Remote Sensing of Environment, Vol. 128
  • DOI: 10.1016/j.rse.2012.10.010

An ontology of slums for image-based classification
journal, March 2012

Exploring issues of training data imbalance and mislabelling on random forest performance for large area land cover classification using the ensemble margin
journal, July 2015

Comparison of Office and Field Techniques for Validating Landscape Change Classification in Pacific Northwest National Parks
journal, December 2018

  • Copass, Catharine; Antonova, Natalya; Kennedy, Robert
  • Remote Sensing, Vol. 11, Issue 1
  • DOI: 10.3390/rs11010003

Component intensities to relate difference by category with difference overall
journal, May 2019

  • Pontius, Robert Gilmore
  • International Journal of Applied Earth Observation and Geoinformation, Vol. 77
  • DOI: 10.1016/j.jag.2018.07.024

An Evaluation of Bagging, Boosting, and Random Forests for Land-Cover Classification in Cape Cod, Massachusetts, USA
journal, September 2012

  • Ghimire, Bardan; Rogan, John; Galiano, Víctor Rodríguez
  • GIScience & Remote Sensing, Vol. 49, Issue 5
  • DOI: 10.2747/1548-1603.49.5.623

Improved land cover mapping using high resolution multiangle 8-band WorldView-2 satellite remote sensing data
journal, January 2013

  • Jawak, Shridhar D.; Luis, Alvarinho J.
  • Journal of Applied Remote Sensing, Vol. 7, Issue 1
  • DOI: 10.1117/1.JRS.7.073573

On the Use of Unmanned Aerial Systems for Environmental Monitoring
journal, April 2018

  • Manfreda, Salvatore; McCabe, Matthew; Miller, Pauline
  • Remote Sensing, Vol. 10, Issue 4
  • DOI: 10.3390/rs10040641

Supervised methods of image segmentation accuracy assessment in land cover mapping
journal, February 2018

Extended triple collocation: Estimating errors and correlation coefficients with respect to an unknown target: EXTENDED TRIPLE COLLOCATION
journal, September 2014

  • McColl, Kaighin A.; Vogelzang, Jur; Konings, Alexandra G.
  • Geophysical Research Letters, Vol. 41, Issue 17
  • DOI: 10.1002/2014GL061322

An assessment of the effectiveness of a random forest classifier for land-cover classification
journal, January 2012

Implementation of machine-learning classification in remote sensing: an applied review
journal, January 2018

Multiscale analysis and validation of the MODIS LAI productI. Uncertainty assessment
journal, December 2002

Sub-pixel confusion–uncertainty matrix for assessing soft classifications
journal, March 2008

The factor of scale in remote sensing
journal, April 1987

Radiative forcing and temperature response to changes in urban albedos and associated CO 2 offsets
journal, January 2010

An assessment of support vector machines for land cover classification
journal, January 2002

  • Huang, C.; Davis, L. S.; Townshend, J. R. G.
  • International Journal of Remote Sensing, Vol. 23, Issue 4
  • DOI: 10.1080/01431160110040323

Land cover mapping using time series HJ-1/CCD data
journal, May 2014

A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists
journal, November 2018

ImageNet Large Scale Visual Recognition Challenge
journal, April 2015

  • Russakovsky, Olga; Deng, Jia; Su, Hao
  • International Journal of Computer Vision, Vol. 115, Issue 3
  • DOI: 10.1007/s11263-015-0816-y

Components of information for multiple resolution comparison between maps that share a real variable
journal, October 2007

  • Pontius, Robert Gilmore; Thontteh, Olufunmilayo; Chen, Hao
  • Environmental and Ecological Statistics, Vol. 15, Issue 2
  • DOI: 10.1007/s10651-007-0043-y

Statistics notes: Measurement error
journal, June 1996

Land change for all municipalities in Latin America and the Caribbean assessed from 250-m MODIS imagery (2001–2010)
journal, November 2012

Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data
journal, February 2016

  • Graves, Sarah; Asner, Gregory; Martin, Roberta
  • Remote Sensing, Vol. 8, Issue 2
  • DOI: 10.3390/rs8020161

Land cover mapping of the tropical savanna region in Brazil
journal, June 2009

  • Sano, Edson E.; Rosa, Roberto; Brito, Jorge L. S.
  • Environmental Monitoring and Assessment, Vol. 166, Issue 1-4
  • DOI: 10.1007/s10661-009-0988-4

A generalized computer vision approach to mapping crop fields in heterogeneous agricultural landscapes
journal, June 2016

A global dataset of crowdsourced land cover and land use reference data
journal, June 2017

  • Fritz, Steffen; See, Linda; Perger, Christoph
  • Scientific Data, Vol. 4, Issue 1
  • DOI: 10.1038/sdata.2017.75

Importance of Matrix Construction for Multiple-Resolution Categorical Map Comparison
journal, July 2008

Accounting for urban biogenic fluxes in regional carbon budgets
journal, August 2017

Estimating the global distribution of field size using crowdsourcing
journal, November 2018

  • Lesiv, Myroslava; Laso Bayas, Juan Carlos; See, Linda
  • Global Change Biology, Vol. 25, Issue 1
  • DOI: 10.1111/gcb.14492

The total operating characteristic to measure diagnostic ability for multiple thresholds
journal, November 2013

  • Pontius Jr, Robert Gilmore; Si, Kangping
  • International Journal of Geographical Information Science, Vol. 28, Issue 3
  • DOI: 10.1080/13658816.2013.862623

Optimizing Remotely Sensed Solutions for Monitoring, Modeling, and Managing Coastal Environments
journal, August 2000

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
journal, July 2019

  • Helber, Patrick; Bischke, Benjamin; Dengel, Andreas
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 12, Issue 7
  • DOI: 10.1109/JSTARS.2019.2918242

What is Land Cover?
journal, April 2005

  • Comber, Alexis; Fisher, Peter; Wadsworth, Richard
  • Environment and Planning B: Planning and Design, Vol. 32, Issue 2
  • DOI: 10.1068/b31135

Uncovering Ecological Patterns with Convolutional Neural Networks
journal, August 2019

  • Brodrick, Philip G.; Davies, Andrew B.; Asner, Gregory P.
  • Trends in Ecology & Evolution, Vol. 34, Issue 8
  • DOI: 10.1016/j.tree.2019.03.006

High-resolution mapping of global surface water and its long-term changes
journal, December 2016

  • Pekel, Jean-François; Cottam, Andrew; Gorelick, Noel
  • Nature, Vol. 540, Issue 7633
  • DOI: 10.1038/nature20584

Learning Multiscale Deep Features for High-Resolution Satellite Image Scene Classification
journal, January 2018

  • Liu, Qingshan; Hang, Renlong; Song, Huihui
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, Issue 1
  • DOI: 10.1109/TGRS.2017.2743243

Mapping and interpreting historical land cover/land use changes in a Natura 2000 site using earth observational data: The case of Nestos delta, Greece
journal, January 2011

  • Mallinis, Georgios; Emmanoloudis, Dimitrios; Giannakopoulos, Vasileios
  • Applied Geography, Vol. 31, Issue 1
  • DOI: 10.1016/j.apgeog.2010.07.002

Mapping Urban Land Use by Using Landsat Images and Open Social Data
journal, February 2016

  • Hu, Tengyun; Yang, Jun; Li, Xuecao
  • Remote Sensing, Vol. 8, Issue 2
  • DOI: 10.3390/rs8020151

Ambiguities inherent in sums-of-squares-based error statistics
journal, January 2009

Fully fuzzy supervised classification of land cover from remotely sensed imagery with an artificial neural network
journal, December 1997

Global land change from 1982 to 2016
journal, August 2018

Key issues in rigorous accuracy assessment of land cover products
journal, September 2019

Learning Aerial Image Segmentation From Online Maps
journal, November 2017

  • Kaiser, Pascal; Wegner, Jan Dirk; Lucchi, Aurelien
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 55, Issue 11
  • DOI: 10.1109/TGRS.2017.2719738

Geo-Wiki: An online platform for improving global land cover
journal, May 2012

Global forecasts of urban expansion to 2030 and direct impacts on biodiversity and carbon pools
journal, September 2012

  • Seto, K. C.; Guneralp, B.; Hutyra, L. R.
  • Proceedings of the National Academy of Sciences, Vol. 109, Issue 40
  • DOI: 10.1073/pnas.1211658109

Validating MODIS Terrestrial Ecology Products
journal, October 1999

Finer resolution observation and monitoring of global land cover: first mapping results with Landsat TM and ETM+ data
journal, December 2012

Variability of operator performance in remote-sensing image interpretation: the importance of human and external factors
journal, January 2014

  • Van Coillie, Frieke M. B.; Gardin, Soetkin; Anseel, Frederik
  • International Journal of Remote Sensing, Vol. 35, Issue 2
  • DOI: 10.1080/01431161.2013.873152

Visualizing Uncertain Information
journal, June 1992

Design and Interpretation of Intensity Analysis Illustrated by Land Change in Central Kalimantan, Indonesia
journal, July 2013

Assessing the global warming potential of human settlement expansion in a mesic temperate landscape from 2005 to 2050
journal, March 2016

Land Use and Land Cover Mapping in the Brazilian Amazon Using Polarimetric Airborne P-Band SAR Data
journal, October 2008

  • Freitas, Corina da Costa; Soler, Luciana de Souza; Sant'Anna, Sidnei JoÃo Siqueira
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 46, Issue 10
  • DOI: 10.1109/TGRS.2008.2000630

Using volunteered geographic information (VGI) in design-based statistical inference for area estimation and accuracy assessment of land cover
journal, June 2018

  • Stehman, Stephen V.; Fonte, Cidália C.; Foody, Giles M.
  • Remote Sensing of Environment, Vol. 212
  • DOI: 10.1016/j.rse.2018.04.014

The effects of imperfect reference data on remote sensing-assisted estimators of land cover class proportions
journal, August 2018

  • McRoberts, Ronald E.; Stehman, Stephen V.; Liknes, Greg C.
  • ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 142
  • DOI: 10.1016/j.isprsjprs.2018.06.002

Review article Synergy in remote sensing-what's in a pixel?
journal, January 1998

Uncertainty analysis for image interpretations of urban slums
journal, November 2016

A large-area, spatially continuous assessment of land cover map error and its impact on downstream analyses
journal, October 2017

  • Estes, Lyndon; Chen, Peng; Debats, Stephanie
  • Global Change Biology, Vol. 24, Issue 1
  • DOI: 10.1111/gcb.13904

Collect Earth: Land Use and Land Cover Assessment through Augmented Visual Interpretation
journal, September 2016

  • Bey, Adia; Sánchez-Paus Díaz, Alfonso; Maniatis, Danae
  • Remote Sensing, Vol. 8, Issue 10
  • DOI: 10.3390/rs8100807

The dimensions of global urban expansion: Estimates and projections for all countries, 2000–2050
journal, February 2011

The spatial and temporal domains of modern ecology
journal, April 2018

Hierarchical mapping of annual global land cover 2001 to present: The MODIS Collection 6 Land Cover product
journal, March 2019

  • Sulla-Menashe, Damien; Gray, Josh M.; Abercrombie, S. Parker
  • Remote Sensing of Environment, Vol. 222
  • DOI: 10.1016/j.rse.2018.12.013

Intensity analysis to unify measurements of size and stationarity of land changes by interval, category, and transition
journal, May 2012

A critical synthesis of remotely sensed optical image change detection techniques
journal, April 2015

  • Tewkesbury, Andrew P.; Comber, Alexis J.; Tate, Nicholas J.
  • Remote Sensing of Environment, Vol. 160
  • DOI: 10.1016/j.rse.2015.01.006

Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
journal, August 2011

Evaluation of the VIIRS BRDF, Albedo and NBAR products suite and an assessment of continuity with the long term MODIS record
journal, November 2017

Land use and land cover change in Greater Dhaka, Bangladesh: Using remote sensing to promote sustainable urbanization
journal, July 2009

Evaluation of land surface phenology from VIIRS data using time series of PhenoCam imagery
journal, June 2018

Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources
journal, December 2017

  • Zhu, Xiao Xiang; Tuia, Devis; Mou, Lichao
  • IEEE Geoscience and Remote Sensing Magazine, Vol. 5, Issue 4
  • DOI: 10.1109/MGRS.2017.2762307

Uncertainty in ecosystem mapping by remote sensing
journal, January 2013

Global vegetation phenology from Moderate Resolution Imaging Spectroradiometer (MODIS): Evaluation of global patterns and comparison with in situ measurements: GLOBAL PHENOLOGY FROM MODIS
journal, December 2006

  • Zhang, Xiaoyang; Friedl, Mark A.; Schaaf, Crystal B.
  • Journal of Geophysical Research: Biogeosciences, Vol. 111, Issue G4
  • DOI: 10.1029/2006JG000217

How good is good enough? Data requirements for reliable crop yield simulations and yield-gap analysis
journal, June 2015

  • Grassini, Patricio; van Bussel, Lenny G. J.; Van Wart, Justin
  • Field Crops Research, Vol. 177
  • DOI: 10.1016/j.fcr.2015.03.004

Identifying Mislabeled Training Data
journal, July 1999

  • Brodley, C. E.; Friedl, M. A.
  • Journal of Artificial Intelligence Research, Vol. 11
  • DOI: 10.1613/jair.606

Smallholder crop area mapped with wall-to-wall WorldView sub-meter panchromatic image texture: A test case for Tigray, Ethiopia
journal, June 2018

  • Neigh, Christopher S. R.; Carroll, Mark L.; Wooten, Margaret R.
  • Remote Sensing of Environment, Vol. 212
  • DOI: 10.1016/j.rse.2018.04.025

Fuzzy set theory and thematic maps: accuracy assessment and area estimation
journal, March 2000

  • Woodcock, Curtis E.; Gopal, Sucharita
  • International Journal of Geographical Information Science, Vol. 14, Issue 2
  • DOI: 10.1080/136588100240895

Random Forests
journal, January 2001

Machine learning in geosciences and remote sensing
journal, January 2016

Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts
journal, July 2013

Sources of error in accuracy assessment of thematic land-cover maps in the Brazilian Amazon
journal, March 2004

Assessing different remote sensing techniques to detect land use/cover changes in the eastern Mediterranean
journal, February 2009

  • Berberoglu, S.; Akin, A.
  • International Journal of Applied Earth Observation and Geoinformation, Vol. 11, Issue 1
  • DOI: 10.1016/j.jag.2008.06.002

A global reference database of crowdsourced cropland data collected using the Geo-Wiki platform
dataset, January 2017

Implementation of machine-learning classification in remote sensing: an applied review
dataset, January 2018

Edge effects enhance carbon uptake and its vulnerability to climate change in temperate broadleaf forests
dataset, January 2016

Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding
conference, July 2019

  • Sumbul, Gencer; Charfuelan, Marcela; Demir, Begum
  • IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium
  • DOI: 10.1109/igarss.2019.8900532

Cartography: uncertainty, interventions, and dynamic display
journal, June 2006

On the Use of Unmanned Aerial Systems for Environmental Monitoring
journal, April 2018

  • Manfreda, Salvatore; McCabe, Matthew; Miller, Pauline
  • Remote Sensing, Vol. 10, Issue 4
  • DOI: 10.3390/rs10040641

Estimating the Global Distribution of Field Size using Crowdsourcing
dataset, January 2018

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
dataset, January 2018

    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Using satellite imagery to understand and promote sustainable development
    preprint, January 2020

      Sort by:
      [ × clear filter / sort ]

      Similar Records in DOE PAGES and OSTI.GOV collections: