A Bayesian Approach for Quantifying Data Scarcity when Modeling Human Behavior via Inverse Reinforcement Learning

Hossain, Tahera; Shen, Wanggang; Antar, Anindya Das; Prabhudesai, Snehal; Inoue, Sozo; Huan, Xun; Banovic, Nikola

doi:10.1145/3551388

Title: A Bayesian Approach for Quantifying Data Scarcity when Modeling Human Behavior via Inverse Reinforcement Learning

Journal Article · 27 July 2022 · ACM Transactions on Computer-Human Interaction

DOI: https://doi.org/10.1145/3551388 · OSTI ID:1907166

^[1];

^[2];

^[3];

^[2];

^[2]

Univ. of Michigan, Ann Arbor, MI (United States); Kyushu Institute of Technology, Kitakyushu (Japan); University of Michigan
Univ. of Michigan, Ann Arbor, MI (United States)
Kyushu Institute of Technology, Kitakyushu (Japan)

Computational models that formalize complex human behaviors enable study and understanding of such behaviors. However, collecting behavior data required to estimate the parameters of such models is often tedious and resource intensive. Thus, estimating dataset size as part of data collection planning (also known as Sample Size Determination) is important to reduce the time and effort of behavior data collection while maintaining an accurate estimate of model parameters. In this paper, we present a sample size determination method based on Uncertainty Quantification (UQ) for a specific Inverse Reinforcement Learning (IRL) model of human behavior, in two cases: 1) pre-hoc experiment design—conducted in the planning stage before any data is collected, to guide the estimation of how many samples to collect; and 2) post-hoc dataset analysis—performed after data is collected, to decide if the existing dataset has sufficient samples and whether more data is needed. Here, we validate our approach in experiments with a realistic model of behaviors of people with Multiple Sclerosis (MS) and illustrate how to pick a reasonable sample size target. Our work enables model designers to perform a deeper, principled investigation of effects of dataset size on IRL.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Univ. of Michigan, Ann Arbor, MI (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0021398

OSTI ID:: 1907166

Journal Information:: ACM Transactions on Computer-Human Interaction, Journal Name: ACM Transactions on Computer-Human Interaction Journal Issue: 1 Vol. 30; ISSN 1073-0516

Publisher:: Association for Computing Machinery (ACM)Copyright Statement

Country of Publication:: United States

Language:: English

References (69)

Multivariate Density Estimation: Theory, Practice, and Visualization Scott, David W. Wiley Series in Probability and Statistics https://doi.org/10.1002/9781118575574	book	March 2015
Monte Carlo Statistical Methods Robert, Christian P.; Casella, George Springer Texts in Statistics https://doi.org/10.1007/978-1-4757-4145-2	book	January 2004
Statistical Decision Theory and Bayesian Analysis Berger, James O. Springer Series in Statistics https://doi.org/10.1007/978-1-4757-4286-2	book	January 1985
Handbook of Uncertainty Quantification Ghanem, Roger; Higdon, David; Owhadi, Houman https://doi.org/10.1007/978-3-319-12385-1	reference-book	January 2017
Predicting the Relationship Between the Size of Training Sample and the Predictive Power of Classifiers Boonyanunta, Natthaphan; Zeephongsekul, Panlop Lecture Notes in Computer Science https://doi.org/10.1007/978-3-540-30134-9_71	book	January 2004
Kullback-Leibler Divergence Joyce, James M. International Encyclopedia of Statistical Science https://doi.org/10.1007/978-3-642-04898-2_327	book	January 2011
Bayesian Inverse Reinforcement Learning for Modeling Conversational Agents in a Virtual Environment Rojas-Barahona, Lina M.; Cerisara, Christophe Computational Linguistics and Intelligent Text Processing https://doi.org/10.1007/978-3-642-54906-9_41	book	January 2014
Eigenbehaviors: identifying structure in routine Eagle, Nathan; Pentland, Alex Sandy Behavioral Ecology and Sociobiology, Vol. 63, Issue 7 https://doi.org/10.1007/s00265-009-0739-0	journal	April 2009
40 years of cognitive architectures: core cognitive abilities and practical applications Kotseruba, Iuliia; Tsotsos, John K. Artificial Intelligence Review, Vol. 53, Issue 1 https://doi.org/10.1007/s10462-018-9646-y	journal	July 2018
Inverse reinforcement learning from summary data Kangasrääsiö, Antti; Kaski, Samuel Machine Learning, Vol. 107, Issue 8-10 https://doi.org/10.1007/s10994-018-5730-4	journal	June 2018
An Efficient Over-sampling Approach Based on Mean Square Error Back-propagation for Dealing with the Multi-class Imbalance Problem Alejo, R.; García, V.; Pacheco-Sánchez, J. H. Neural Processing Letters, Vol. 42, Issue 3 https://doi.org/10.1007/s11063-014-9376-3	journal	August 2014
Development and validation of the positive affect and well-being scale for the neurology quality of life (Neuro-QOL) measurement system Salsman, John M.; Victorson, David; Choi, Seung W. Quality of Life Research, Vol. 22, Issue 9 https://doi.org/10.1007/s11136-013-0382-0	journal	March 2013
Simulation Based Optimal Design Müller, Peter Handbook of Statistics https://doi.org/10.1016/S0169-7161(05)25017-4	book	January 2005
Ecological Momentary Assessment of Pain, Fatigue, Depressive, and Cognitive Symptoms Reveals Significant Daily Variability in Multiple Sclerosis Kratz, Anna L.; Murphy, Susan L.; Braley, Tiffany J. Archives of Physical Medicine and Rehabilitation, Vol. 98, Issue 11 https://doi.org/10.1016/j.apmr.2017.07.002	journal	November 2017
Pain, Fatigue, and Cognitive Symptoms Are Temporally Associated Within but Not Across Days in Multiple Sclerosis Kratz, Anna L.; Murphy, Susan L.; Braley, Tiffany J. Archives of Physical Medicine and Rehabilitation, Vol. 98, Issue 11 https://doi.org/10.1016/j.apmr.2017.07.003	journal	November 2017
How Do Pain, Fatigue, Depressive, and Cognitive Symptoms Relate to Well-Being and Social and Physical Functioning in the Daily Lives of Individuals With Multiple Sclerosis? Kratz, Anna L.; Braley, Tiffany J.; Foxen-Craft, Emily Archives of Physical Medicine and Rehabilitation, Vol. 98, Issue 11 https://doi.org/10.1016/j.apmr.2017.07.004	journal	November 2017
Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts’ law research in HCI Soukoreff, R. William; MacKenzie, I. Scott International Journal of Human-Computer Studies, Vol. 61, Issue 6 https://doi.org/10.1016/j.ijhcs.2004.09.001	journal	December 2004
Simulation-based optimal Bayesian experimental design for nonlinear systems Huan, Xun; Marzouk, Youssef M. Journal of Computational Physics, Vol. 232, Issue 1 https://doi.org/10.1016/j.jcp.2012.08.013	journal	January 2013
The Neurobiology of Multiple Sclerosis: Genes, Inflammation, and Neurodegeneration Hauser, Stephen L.; Oksenberg, Jorge R. Neuron, Vol. 52, Issue 1 https://doi.org/10.1016/j.neuron.2006.09.011	journal	October 2006
Probability Theory Jaynes, E. T. https://doi.org/10.1017/CBO9780511790423	book	January 2003
The need for uncertainty quantification in machine-assisted medical decision making Begoli, Edmon; Bhattacharya, Tanmoy; Kusnezov, Dimitri Nature Machine Intelligence, Vol. 1, Issue 1 https://doi.org/10.1038/s42256-018-0004-1	journal	January 2019
Variational Inference: A Review for Statisticians Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D. Journal of the American Statistical Association, Vol. 112, Issue 518 https://doi.org/10.1080/01621459.2017.1285773	journal	July 2016
`emcee` : The MCMC Hammer Foreman-Mackey, Daniel; Hogg, David W.; Lang, Dustin Publications of the Astronomical Society of the Pacific, Vol. 125, Issue 925 https://doi.org/10.1086/670067	journal	March 2013
Bayesian inference in physics von Toussaint, Udo Reviews of Modern Physics, Vol. 83, Issue 3 https://doi.org/10.1103/RevModPhys.83.943	journal	September 2011
Effects of sample size in classifier design Fukunaga, K.; Hayes, R. R. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11, Issue 8 https://doi.org/10.1109/34.31448	journal	January 1989
Ability-Based Optimization of Touchscreen Interactions Sarcar, Sayan; Jokinen, Jussi P. P.; Oulasvirta, Antti IEEE Pervasive Computing, Vol. 17, Issue 1 https://doi.org/10.1109/MPRV.2018.011591058	journal	January 2018
Control of Gene Regulatory Networks Using Bayesian Inverse Reinforcement Learning Imani, Mahdi; Braga-Neto, Ulisses M. IEEE/ACM Transactions on Computational Biology and Bioinformatics, Vol. 16, Issue 4 https://doi.org/10.1109/TCBB.2018.2830357	journal	July 2019
Bayesian calibration of computer models Kennedy, Marc C.; O'Hagan, Anthony Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 63, Issue 3 https://doi.org/10.1111/1467-9868.00294	journal	August 2001
The choice of sample size Lindley, Dennis V. Journal of the Royal Statistical Society: Series D (The Statistician), Vol. 46, Issue 2 https://doi.org/10.1111/1467-9884.00068	journal	July 1997
Sample size determination: a review Adcock, C. J. Journal of the Royal Statistical Society: Series D (The Statistician), Vol. 46, Issue 2 https://doi.org/10.1111/1467-9884.00082	journal	July 1997
Parameter Inference for Computational Cognitive Models with Approximate Bayesian Computation Kangasrääsiö, Antti; Jokinen, Jussi P. P.; Oulasvirta, Antti Cognitive Science, Vol. 43, Issue 6 https://doi.org/10.1111/cogs.12738	journal	May 2019
Computational Rationality: Linking Mechanism and Behavior Through Bounded Utility Maximization Lewis, Richard L.; Howes, Andrew; Singh, Satinder Topics in Cognitive Science, Vol. 6, Issue 2 https://doi.org/10.1111/tops.12086	journal	March 2014
Verification, validation, and predictive capability in computational engineering and physics Oberkampf, William L.; Trucano, Timothy G.; Hirsch, Charles Applied Mechanics Reviews, Vol. 57, Issue 5 https://doi.org/10.1115/1.1767847	journal	September 2004
Computational rationality: A converging paradigm for intelligence in brains, minds, and machines Gershman, S. J.; Horvitz, E. J.; Tenenbaum, J. B. Science, Vol. 349, Issue 6245 https://doi.org/10.1126/science.aac6076	journal	July 2015
Navigate like a cabbie Ziebart, Brian D.; Maas, Andrew L.; Dey, Anind K. Proceedings of the 10th international conference on Ubiquitous computing https://doi.org/10.1145/1409635.1409678	conference	September 2008
Human model evaluation in interactive supervised learning Fiebrink, Rebecca; Cook, Perry R.; Trueman, Dan Proceedings of the SIGCHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/1978942.1978965	conference	May 2011
Probabilistic pointing target prediction via inverse optimal control Ziebart, Brian; Dey, Anind; Bagnell, J. Andrew Proceedings of the 2012 ACM international conference on Intelligent User Interfaces https://doi.org/10.1145/2166966.2166968	conference	February 2012
Directing exploratory search Glowacka, Dorota; Ruotsalo, Tuukka; Konuyshkova, Ksenia Proceedings of the 2013 international conference on Intelligent user interfaces - IUI '13 https://doi.org/10.1145/2449396.2449413	conference	January 2013
The effect of time-based cost of error in target-directed pointing tasks Banovic, Nikola; Grossman, Tovi; Fitzmaurice, George Proceedings of the SIGCHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/2470654.2466181	conference	April 2013
Modeling and Understanding Human Routine Behavior Banovic, Nikola; Buzali, Tofi; Chevalier, Fanny Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/2858036.2858557	conference	May 2016
Supervised and Unsupervised Transfer Learning for Activity Recognition from Simple In-home Sensors Inoue, Sozo; Pan, Xincheng Proceedings of the 13th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services https://doi.org/10.1145/2994374.2994400	conference	November 2016
Leveraging Human Routine Models to Detect and Generate Human Behaviors Banovic, Nikola; Wang, Anqi; Jin, Yanfeng Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/3025453.3025571	conference	May 2017
A Cognitive Model of How People Make Decisions Through Interaction with Visual Displays Chen, Xiuli; Starke, Sandra Dorothee; Baber, Chris Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/3025453.3025596	conference	May 2017
RDeepSense Yao, Shuochao; Zhao, Yiran; Shao, Huajie Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 1, Issue 4 https://doi.org/10.1145/3161181	journal	January 2018
SenseGAN Yao, Shuochao; Zhao, Yiran; Shao, Huajie Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 2, Issue 3 https://doi.org/10.1145/3264954	journal	September 2018
Computational Modeling in Human-Computer Interaction Banovic, Nikola; Oulasvirta, Antti; Kristensson, Per Ola Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/3290607.3299032	conference	May 2019
Rl-Klm Leino, Katri; Oulasvirta, Antti; Kurimo, Mikko Proceedings of the 24th International Conference on Intelligent User Interfaces https://doi.org/10.1145/3301275.3302285	conference	March 2019
Computer-supported form design using keystroke-level modeling with reinforcement learning Leino, Katri; Todi, Kashyap; Oulasvirta, Antti Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion https://doi.org/10.1145/3308557.3308704	conference	March 2019
Learning Cooperative Personalized Policies from Gaze Data Gebhardt, Christoph; Hecox, Brian; van Opheusden, Bas Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology https://doi.org/10.1145/3332165.3347933	conference	October 2019
Leveraging Active Learning and Conditional Mutual Information to Minimize Data Annotation in Human Activity Recognition Adaimi, Rebecca; Thomaz, Edison Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 3, Issue 3 https://doi.org/10.1145/3351228	journal	September 2019
Integrating Activity Recognition and Nursing Care Records Inoue, Sozo; Lago, Paula; Hossain, Tahera Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 3, Issue 3 https://doi.org/10.1145/3351244	journal	September 2019
Leveraging Routine Behavior and Contextually-Filtered Features for Depression Detection among College Students Xu, Xuhai; Chikersal, Prerna; Doryab, Afsaneh Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 3, Issue 3 https://doi.org/10.1145/3351274	journal	September 2019
A Systematic Study of Unsupervised Domain Adaptation for Robust Human-Activity Recognition Chang, Youngjae; Mathur, Akhil; Isopoussu, Anton Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 4, Issue 1 https://doi.org/10.1145/3380985	journal	March 2020
Computational Rationality as a Theory of Interaction Oulasvirta, Antti; Jokinen, Jussi P. P.; Howes, Andrew CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/3491102.3517739	conference	April 2022
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation Maxwell, Scott E.; Kelley, Ken; Rausch, Joseph R. Annual Review of Psychology, Vol. 59, Issue 1 https://doi.org/10.1146/annurev.psych.59.103006.093735	journal	January 2008
Predicting sample size required for classification performance Figueroa, Rosa L.; Zeng-Treitler, Qing; Kandula, Sasikiran BMC Medical Informatics and Decision Making, Vol. 12, Issue 1 https://doi.org/10.1186/1472-6947-12-8	journal	February 2012
Multiple sclerosis: clinical profiling and data collection as prerequisite for personalized medicine approach Ziemssen, Tjalf; Kern, Raimar; Thomas, Katja BMC Neurology, Vol. 16, Issue 1 https://doi.org/10.1186/s12883-016-0639-7	journal	August 2016
Some Practical Guidelines for Effective Sample Size Determination Lenth, Russell V. The American Statistician, Vol. 55, Issue 3 https://doi.org/10.1198/000313001317098149	journal	August 2001
Handbook of Markov Chain Monte Carlo Brooks, Steve; Gelman, Andrew; Jones, Galin Handbooks of Modern Statistical Methods https://doi.org/10.1201/b10905	book	May 2011
On Information and Sufficiency Kullback, S.; Leibler, R. A. The Annals of Mathematical Statistics, Vol. 22, Issue 1 https://doi.org/10.1214/aoms/1177729694	journal	March 1951
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) Breiman, Leo Statistical Science, Vol. 16, Issue 3 https://doi.org/10.1214/ss/1009213726	journal	August 2001
Bayesian Experimental Design: A Review Chaloner, Kathryn; Verdinelli, Isabella Statistical Science, Vol. 10, Issue 3 https://doi.org/10.1214/ss/1177009939	journal	August 1995
Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning Brown, Daniel; Niekum, Scott Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, Issue 1 https://doi.org/10.1609/aaai.v32i1.11755	journal	April 2018
Reinforcement Learning: A Survey Kaelbling, L. P.; Littman, M. L.; Moore, A. W. Journal of Artificial Intelligence Research, Vol. 4 https://doi.org/10.1613/jair.301	journal	January 1996
Ensemble samplers with affine invariance Goodman, Jonathan; Weare, Jonathan Communications in Applied Mathematics and Computational Science, Vol. 5, Issue 1 https://doi.org/10.2140/camcos.2010.5.65	journal	January 2010
The NASA Langley Multidisciplinary Uncertainty Quantification Challenge Crespo, Luis G.; Kenny, Sean P.; Giesy, Daniel P. 16th AIAA Non-Deterministic Approaches Conference https://doi.org/10.2514/6.2014-1347	conference	January 2014
The class imbalance problem: A systematic study1 Japkowicz, Nathalie; Stephen, Shaju Intelligent Data Analysis, Vol. 6, Issue 5 https://doi.org/10.3233/IDA-2002-6504	journal	November 2002
Discovering hidden time patterns in behavior: T-patterns and their detection Magnusson, Magnus S. Behavior Research Methods, Instruments, & Computers, Vol. 32, Issue 1 https://doi.org/10.3758/BF03200792	journal	March 2000
Ten simple rules for the computational modeling of behavioral data Wilson, Robert C.; Collins, Anne GE eLife, Vol. 8 https://doi.org/10.7554/eLife.49547	journal	November 2019

Similar Records

Inverse Uncertainty Quantification of Reactor Simulation with Polynomial Chaos Surrogate Model

Journal Article · 2016 · Transactions of the American Nuclear Society · OSTI ID:22991923

Wu, Xu; Kozlowski, Tomasz

Hierarchical Bayesian modeling for Inverse Uncertainty Quantification of system thermal-hydraulics code using critical flow experimental data

Journal Article · 2024 · International Journal of Heat and Mass Transfer · OSTI ID:2496317

Xie, Ziyu; Wang, Chen; Wu, Xu

Entropy-Bayesian Inversion of Time-Lapse Tomographic GPR data for Monitoring Dielectric Permittivity and Soil Moisture Variations

Technical Report · 2013 · OSTI ID:1069207

Hou, Zhangshuan; Terry, Neil C.; Hubbard, Susan S.

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian inference
Behavior modeling
Inverse Reinforcement Learning
Sample size determination

Title: A Bayesian Approach for Quantifying Data Scarcity when Modeling Human Behavior via Inverse Reinforcement Learning

Citation Formats

References (69)

Similar Records

Related Subjects