Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

What can simulation test beds teach us about social science? Results of the ground truth program

Journal Article · · Computational and Mathematical Organization Theory
 [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1]
  1. Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States)
+ Show Author Affiliations
The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.
Research Organization:
Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
Sponsoring Organization:
Defense Advanced Research Projects Agency (DARPA)
Grant/Contract Number:
NA0003525
OSTI ID:
1870475
Report Number(s):
SAND2022-3972J; 704807
Journal Information:
Computational and Mathematical Organization Theory, Journal Name: Computational and Mathematical Organization Theory Vol. 29; ISSN 1381-298X
Publisher:
SpringerCopyright Statement
Country of Publication:
United States
Language:
English

References (14)

Linear Regression Analysis book January 2003
The Ground Truth program: simulations as test beds for social science research methods journal April 2022
SCAMP’s stigmergic model of social conflict journal November 2021
Urban life: a model of people and places journal November 2021
Groups, governance, and greed: the ACCESS world model journal December 2021
Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds journal January 2022
Disaster world journal May 2022
Does big data serve policy? Not without context. An experiment with in silico social science journal November 2022
Aleatory or epistemic? Does it matter? journal March 2009
ImageNet: A large-scale hierarchical image database
  • Deng, Jia; Dong, Wei; Socher, Richard
  • 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), 2009 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2009.5206848
conference June 2009
Meta-research: Evaluation and Improvement of Research Methods and Practices journal October 2015
Why Most Published Research Findings Are False journal August 2005
Model Selection for Multivariate Regression in Small Samples journal March 1994
Statsmodels: Econometric and Statistical Modeling with Python conference January 2010

Similar Records

The Ground Truth Program: Simulations as Test Beds for Social Science Research Methods.
Journal Article · Sun Apr 17 20:00:00 EDT 2022 · Computational and Mathematical Organization Theory · OSTI ID:1894598
Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods
Journal Article · Wed Nov 17 19:00:00 EST 2021 · Computational and Mathematical Organization Theory · OSTI ID:1969000
Ground Truth Collections at the MTI Core Sites
Conference · Wed Jan 24 23:00:00 EST 2001 · OSTI ID:773936

Related Subjects

97 MATHEMATICS AND COMPUTING
causal structure
complexity
data efficiency
metascience
simulation test beds
social science