The Ground Truth Program: Simulations as Test Beds for Social Science Research Methods.

Naugle, Asmeret; Russell, Adam; Lakkaraju, Kiran; Swiler, Laura Painton; Verzi, Stephen; Romero, Vicente

doi:10.1007/s10588-021-09346-9

The Ground Truth Program: Simulations as Test Beds for Social Science Research Methods.

Journal Article · Mon Apr 18 00:00:00 EDT 2022 · Computational and Mathematical Organization Theory

DOI:https://doi.org/10.1007/s10588-021-09346-9· OSTI ID:1894598

Naugle, Asmeret ^[1]; Russell, Adam ^[1]; Lakkaraju, Kiran ^[1]; Swiler, Laura Painton ^[1]; Verzi, Stephen ^[1]; Romero, Vicente ^[1]

Sandia National Laboratory (SNL-NM), Albuquerque, NM (United States)

Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world’s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: Defense Advanced Research Projects Agency (DARPA); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: NA0003525

OSTI ID:: 1894598

Report Number(s):: SAND2022-4400J; 704940

Journal Information:: Computational and Mathematical Organization Theory, Journal Name: Computational and Mathematical Organization Theory Journal Issue: 1 Vol. 29; ISSN 1381-298X

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (19)

SCAMP’s stigmergic model of social conflict Parunak, H. Van Dyke; Greanya, Jason; McCarthy, Peggy Computational and Mathematical Organization Theory https://doi.org/10.1007/s10588-021-09347-8	journal	November 2021
Urban life: a model of people and places Züfle, Andreas; Wenk, Carola; Pfoser, Dieter Computational and Mathematical Organization Theory https://doi.org/10.1007/s10588-021-09348-7	journal	November 2021
What can simulation test beds teach us about social science? Results of the ground truth program Naugle, Asmeret; Krofcheck, Daniel; Warrender, Christina Computational and Mathematical Organization Theory, Vol. 29, Issue 1 https://doi.org/10.1007/s10588-021-09349-6	journal	April 2022
Groups, governance, and greed: the ACCESS world model Rager, Scott; Leung, Alice; Pinegar, Shannon Computational and Mathematical Organization Theory https://doi.org/10.1007/s10588-021-09352-x	journal	December 2021
Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds Schmidt, Aurora C.; Cameron, Christopher J.; Lowman, Corey Computational and Mathematical Organization Theory https://doi.org/10.1007/s10588-021-09353-w	journal	January 2022
Disaster world Pynadath, David V.; Dilkina, Bistra; Jeong, David C. Computational and Mathematical Organization Theory https://doi.org/10.1007/s10588-022-09359-y	journal	May 2022
Does big data serve policy? Not without context. An experiment with in silico social science Graziul, Chris; Belikov, Alexander; Chattopadyay, Ishanu Computational and Mathematical Organization Theory https://doi.org/10.1007/s10588-022-09362-3	journal	November 2022
What is a complex system? Ladyman, James; Lambert, James; Wiesner, Karoline European Journal for Philosophy of Science, Vol. 3, Issue 1 https://doi.org/10.1007/s13194-012-0056-8	journal	June 2012
Data-Driven Computational Social Science: A Survey Zhang, Jun; Wang, Wei; Xia, Feng Big Data Research, Vol. 21 https://doi.org/10.1016/j.bdr.2020.100145	journal	September 2020
An Evaluation of Statistical Approaches to Text Categorization Yang, Yiming Information Retrieval, Vol. 1, Issue 1-2, p. 69-90 https://doi.org/10.1023/A:1009982220290	journal	April 1999
Should social science be more solution-oriented? Watts, Duncan J. Nature Human Behaviour, Vol. 1, Issue 1 https://doi.org/10.1038/s41562-016-0015	journal	January 2017
A manifesto for reproducible science Munafò, Marcus R.; Nosek, Brian A.; Bishop, Dorothy V. M. Nature Human Behaviour, Vol. 1, Issue 1 https://doi.org/10.1038/s41562-016-0021	journal	January 2017
Measuring the predictability of life outcomes with a scientific mass collaboration Salganik, Matthew J.; Lundberg, Ian; Kindel, Alexander T. Proceedings of the National Academy of Sciences, Vol. 117, Issue 15 https://doi.org/10.1073/pnas.1915006117	journal	March 2020
Dynamic models of segregation† Schelling, Thomas C. The Journal of Mathematical Sociology, Vol. 1, Issue 2 https://doi.org/10.1080/0022250X.1971.9989794	journal	July 1971
Clustering by Compression Cilibrasi, R.; Vitanyi, P. M. B. IEEE Transactions on Information Theory, Vol. 51, Issue 4 https://doi.org/10.1109/TIT.2005.844059	journal	April 2005
Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences Oreskes, N.; Shrader-Frechette, K.; Belitz, K. Science, Vol. 263, Issue 5147 https://doi.org/10.1126/science.263.5147.641	journal	February 1994
Computational social science: Obstacles and opportunities Lazer, David M. J.; Pentland, Alex; Watts, Duncan J. Science, Vol. 369, Issue 6507 https://doi.org/10.1126/science.aaz8170	journal	August 2020
Hierarchy Measure for Complex Networks Mones, Enys; Vicsek, Lilla; Vicsek, Tamás PLoS ONE, Vol. 7, Issue 3 https://doi.org/10.1371/journal.pone.0033799	journal	March 2012
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries Olteanu, Alexandra; Castillo, Carlos; Diaz, Fernando Frontiers in Big Data, Vol. 2 https://doi.org/10.3389/fdata.2019.00013	journal	July 2019

Similar Records

What can simulation test beds teach us about social science? Results of the ground truth program

Journal Article · Fri Apr 29 20:00:00 EDT 2022 · Computational and Mathematical Organization Theory · OSTI ID:1870475

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Journal Article · Wed Nov 17 19:00:00 EST 2021 · Computational and Mathematical Organization Theory · OSTI ID:1969000

Taxonomy of USA east coast fishing communities in terms of social vulnerability and resilience

Journal Article · Sat Nov 14 23:00:00 EST 2015 · Environmental Impact Assessment Review · OSTI ID:22479768

Related Subjects

97 MATHEMATICS AND COMPUTING
causal ground truth
complexity
human domain
metascience
simulation test beds

The Ground Truth Program: Simulations as Test Beds for Social Science Research Methods.

Citation Formats

References (19)

Similar Records

Related Subjects