skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: What can simulation test beds teach us about social science? Results of the ground truth program

Journal Article · · Computational and Mathematical Organization Theory
ORCiD logo [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1]
  1. Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States)
+ Show Author Affiliations

The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.

Research Organization:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Organization:
Defense Advanced Research Projects Agency (DARPA)
Grant/Contract Number:
NA0003525; HR0011937661
OSTI ID:
1870475
Report Number(s):
SAND2022-3972J; 704807
Journal Information:
Computational and Mathematical Organization Theory, Vol. 29; ISSN 1381-298X
Publisher:
SpringerCopyright Statement
Country of Publication:
United States
Language:
English

References (14)

Aleatory or epistemic? Does it matter? journal March 2009
Why Most Published Research Findings Are False journal August 2005
ImageNet: A large-scale hierarchical image database
  • Deng, Jia; Dong, Wei; Socher, Richard
  • 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), 2009 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2009.5206848
conference June 2009
Does big data serve policy? Not without context. An experiment with in silico social science journal November 2022
Meta-research: Evaluation and Improvement of Research Methods and Practices journal October 2015
Groups, governance, and greed: the ACCESS world model journal December 2021
Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds journal January 2022
The Ground Truth program: simulations as test beds for social science research methods journal April 2022
SCAMP’s stigmergic model of social conflict journal November 2021
Urban life: a model of people and places journal November 2021
Model Selection for Multivariate Regression in Small Samples journal March 1994
Statsmodels: Econometric and Statistical Modeling with Python conference January 2010
Disaster world journal May 2022
Linear Regression Analysis book January 2003

Similar Records

Related Subjects

97 MATHEMATICS AND COMPUTING
social science
simulation test beds
complexity
causal structure
data efficiency
metascience