Exploring the role of judgement and shared situation awareness when working with AI recommender systems

Srivastava, Divya; Lilly, J. Mason; Feigh, Karen M.

doi:10.1007/s10111-024-00771-9

Title: Exploring the role of judgement and shared situation awareness when working with AI recommender systems

Journal Article · 26 July 2024 · Cognition, Technology & Work

DOI: https://doi.org/10.1007/s10111-024-00771-9 · OSTI ID:2406706

Srivastava, Divya; Lilly, J. Mason; Feigh, Karen M.

Abstract AI-advised Decision Making is a form of human-autonomy teaming in which an AI recommender system suggests a solution to a human operator, who is responsible for the final decision. This work seeks to examine the importance of judgement and shared situation awareness between humans and automated agents when interacting together in the form of a recommender systems. We propose manipulating both human judgement and shared situation awareness by providing the human decision maker with relevant information that the automated agent (AI), in the form of a recommender system, uses to generate possible courses of action. This paper presents the results of a two-phase between-subjects study in which participants and a recommender system jointly make a high-stakes decision. We varied the amount of relevant information the participant had, the assessment technique of the proposed solution, and the reliability of the recommender system. Findings indicate that this technique of supporting the human’s judgement and establishing a shared situation awareness is effective in (1) boosting the human decision maker’s situation awareness and task performance, (2) calibrating their trust in AI teammates, and (3) reducing overreliance on an AI partner. Additionally, participants were able to pinpoint the limitations and boundaries of the AI partner’s capabilities. They were able to discern situations where the AI’s recommendations could be trusted versus instances when they should not rely on the AI’s advice. This work proposes and validates a way to provide model-agnostic transparency into recommender systems that can support the human decision maker and lead to improved team performance.

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 2406706

Journal Information:: Cognition, Technology & Work, Journal Name: Cognition, Technology & Work; ISSN 1435-5558

Publisher:: Springer Science + Business MediaCopyright Statement

Country of Publication:: Country unknown/Code not available

Language:: English

References (41)

Reflections on shared cognition Cannon‐Bowers, Janis A.; Salas, Eduardo Journal of Organizational Behavior, Vol. 22, Issue 2 https://doi.org/10.1002/job.82	journal	March 2001
Process and context in choice models Ben-Akiva, Moshe; de Palma, André; McFadden, Daniel Marketing Letters, Vol. 23, Issue 2 https://doi.org/10.1007/s11002-012-9180-7	journal	May 2012
Transparency and the Black Box Problem: Why We Do Not Trust AI von Eschenbach, Warren J. Philosophy & Technology, Vol. 34, Issue 4 https://doi.org/10.1007/s13347-021-00477-0	journal	September 2021
Artificial intelligence in recommender systems Zhang, Qian; Lu, Jie; Jin, Yaochu Complex & Intelligent Systems, Vol. 7, Issue 1 https://doi.org/10.1007/s40747-020-00212-w	journal	November 2020
Levels of explainable artificial intelligence for human-aligned conversational explanations Dazeley, Richard; Vamplew, Peter; Foale, Cameron Artificial Intelligence, Vol. 299 https://doi.org/10.1016/j.artint.2021.103525	journal	October 2021
Repeatable effects of synchronizing perceptual tasks with heartbeat on perception-driven situation awareness Vanderhaegen, F.; Wolff, M.; Mollard, R. Cognitive Systems Research, Vol. 81 https://doi.org/10.1016/j.cogsys.2023.05.005	journal	September 2023
The influential factors on nurses’ situation awareness in inpatient settings: A literature review Avalos, Joshua; Roy, Debrina; Asan, Onur Human Factors in Healthcare, Vol. 1 https://doi.org/10.1016/j.hfh.2022.100006	journal	December 2021
The negative consequences of knowledge hiding in NPD project teams: The roles of project work attributes Zhang, Zhen; Min, Min International Journal of Project Management, Vol. 37, Issue 2 https://doi.org/10.1016/j.ijproman.2019.01.006	journal	February 2019
Knowledge hiding in organizational crisis: The moderating role of leadership Nguyen, Tuyet-Mai; Malik, Ashish; Budhwar, Pawan Journal of Business Research, Vol. 139 https://doi.org/10.1016/j.jbusres.2021.09.026	journal	February 2022
Fast, frugal, and fit: Simple heuristics for paired comparison Martignon, Laura; Hoffrage, Ulrich Theory and Decision, Vol. 52, Issue 1, p. 29-71 https://doi.org/10.1023/A:1015516217425	journal	February 2002
The benefits of imperfect diagnostic automation: a synthesis of the literature Wickens, Christopher D.; Dixon, Stephen R. Theoretical Issues in Ergonomics Science, Vol. 8, Issue 3 https://doi.org/10.1080/14639220500370105	journal	May 2007
Decision-Making Tactics and Contextual Features: Strategic, Tactical and Operational Implications Harrington, Robert J.; Ottenbacher, Michael C. International Journal of Hospitality & Tourism Administration, Vol. 10, Issue 1 https://doi.org/10.1080/15256480802557259	journal	January 2009
Visual Cognition: Where Cognition and Culture Meet Gooding, David C. Philosophy of Science, Vol. 73, Issue 5 https://doi.org/10.1086/518523	journal	December 2006
Adaptive enterprise design: The sense‐and‐respond model Haeckel, Stephen H. Planning Review, Vol. 23, Issue 3 https://doi.org/10.1108/eb054506	journal	March 1995
Overtrust of robots in emergency evacuation scenarios Robinette, Paul; Li, Wenchen; Allen, Robert 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) https://doi.org/10.1109/HRI.2016.7451740	conference	March 2016
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems Bussone, Adrian; Stumpf, Simone; O'Sullivan, Dympna 2015 International Conference on Healthcare Informatics https://doi.org/10.1109/ICHI.2015.26	conference	October 2015
An Interactive Bio-inspired Approach to Clustering and Visualizing Datasets Erra, Ugo; Frola, Bernardino; Scarano, Vittorio 2011 15th International Conference on Information Visualisation https://doi.org/10.1109/IV.2011.16	conference	July 2011
Impact of Missing Information and Strategy on Decision Making Performance Sealy, William I. N.; Feigh, Karen M. 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC) https://doi.org/10.1109/SMC52423.2021.9659259	conference	October 2021
Accuracy and Effort of Decision-Making Strategies With Incomplete Information: Implications for Decision Support System Design Canellas, Marc C.; Feigh, Karen M.; Chua, Zarrin K. IEEE Transactions on Human-Machine Systems, Vol. 45, Issue 6 https://doi.org/10.1109/THMS.2015.2420575	journal	December 2015
Modeling Cognitive Loads for Evolving Shared Mental Models in Human–Agent Collaboration IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 41, Issue 2 https://doi.org/10.1109/TSMCB.2010.2053705	journal	April 2011
Piggybacking Robots Booth, Serena; Tompkin, James; Pfister, Hanspeter Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction https://doi.org/10.1145/2909824.3020211	conference	March 2017
Neural Survival Recommender Jing, How; Smola, Alexander J. Proceedings of the Tenth ACM International Conference on Web Search and Data Mining https://doi.org/10.1145/3018661.3018719	conference	February 2017
Reinforcement Learning to Rank in E-Commerce Search Engine Hu, Yujing; Da, Qing; Zeng, Anxiang Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3219819.3219846	conference	July 2018
A Survey of Methods for Explaining Black Box Models Guidotti, Riccardo; Monreale, Anna; Ruggieri, Salvatore ACM Computing Surveys, Vol. 51, Issue 5 https://doi.org/10.1145/3236009	journal	January 2019
Overtrust in the robotic age Wagner, Alan R.; Borenstein, Jason; Howard, Ayanna Communications of the ACM, Vol. 61, Issue 9 https://doi.org/10.1145/3241365	journal	August 2018
Breast cancer: prediction with artificial neural network based on BI-RADS standardized lexicon. Baker, J. A.; Kornguth, P. J.; Lo, J. Y. Radiology, Vol. 196, Issue 3 https://doi.org/10.1148/radiology.196.3.7644649	journal	September 1995
Complacency and Bias in Human Use of Automation: An Attentional Integration Parasuraman, Raja; Manzey, Dietrich H. Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 52, Issue 3 https://doi.org/10.1177/0018720810376055	journal	June 2010
Model blindness: A Framework for Understanding how Model-Based Decision Support Systems can Lead to Performance Degradation Parmar, Sweta; Illingworth, David A.; Thomas, Rickey P. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 65, Issue 1 https://doi.org/10.1177/1071181321651080	journal	September 2021
Nasa-Task Load Index (NASA-TLX); 20 Years Later Hart, Sandra G. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 50, Issue 9 https://doi.org/10.1177/154193120605000909	journal	October 2006
Attention Allocation in Situation Awareness Fracker, Martin L. Proceedings of the Human Factors Society Annual Meeting, Vol. 33, Issue 20 https://doi.org/10.1177/154193128903302007	journal	October 1989
Are they Shooting at Me?: An Approach to Training Situational Awareness Kass, Steven J.; Herschler, Daniel A.; Companion, Michael A. Proceedings of the Human Factors Society Annual Meeting, Vol. 34, Issue 18 https://doi.org/10.1177/154193129003401811	journal	October 1990
Designing for Situation Awareness: An Approach to User-Centered Design Endsley, Mica R.; Bolte, Betty; Jones, Debra G. https://doi.org/10.1201/9780203485088	book	July 2003
Toward a Theory of Situation Awareness in Dynamic Systems Endsley, Mica R. Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 37, Issue 1 https://doi.org/10.1518/001872095779049543	journal	March 1995
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance Bansal, Gagan; Nushi, Besmira; Kamar, Ece Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7 https://doi.org/10.1609/hcomp.v7i1.5285	journal	October 2019
SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets Ie, Eugene; Jain, Vihan; Wang, Jing Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence https://doi.org/10.24963/ijcai.2019/360	conference	August 2019
Contextual Factors and Decision-Making in the Behavior of Finalization in the Positional Attack in Beach Handball: Differences by Gender Through Polar Coordinates Analysis Vázquez-Diz, Juan A.; Morillo-Baro, Juan P.; Reigal, Rafael E. Frontiers in Psychology, Vol. 10 https://doi.org/10.3389/fpsyg.2019.01386	journal	June 2019
The Impact of Multimodal Communication on a Shared Mental Model, Trust, and Commitment in Human–Intelligent Virtual Agent Teams Hanna, Nader; Richards, Deborah Multimodal Technologies and Interaction, Vol. 2, Issue 3 https://doi.org/10.3390/mti2030048	journal	August 2018
Understanding and Avoiding AI Failures: A Practical Guide Williams, Robert; Yampolskiy, Roman Philosophies, Vol. 6, Issue 3 https://doi.org/10.3390/philosophies6030053	journal	June 2021
"Why Should I Trust You?": Explaining the Predictions of Any Classifier Ribeiro, Marco Tulio; Singh, Sameer; Guestrin, Carlos arXiv https://doi.org/10.48550/ARXIV.1602.04938	preprint	January 2016
On the Robustness of Interpretability Methods Alvarez-Melis, David; Jaakkola, Tommi S. arXiv https://doi.org/10.48550/ARXIV.1806.08049	preprint	January 2018
Explaining individual predictions when features are dependent: More accurate approximations to Shapley values Aas, Kjersti; Jullum, Martin; Løland, Anders arXiv https://doi.org/10.48550/ARXIV.1903.10464	preprint	January 2019

Similar Records

“Shoulda, Coulda, Woulda”: Conceptualizing the Differences in Trust Between Human-Human Teaming and Human-Machine Teaming

Technical Report · 2023 · OSTI ID:1999810

Dreslin, Brandon D.; Baweja, Jessica A.

Characterizing Interaction Uncertainty in Human-Machine Teams

Conference · 2024 · OSTI ID:2426428

Wenskovitch, John E.; Fallon, Corey K.; Miller, Kate; +1 more

Yes, No, Maybe So: Human Factors Considerations for Fostering Calibrated Trust in Foundation Models Under Uncertainty

Technical Report · 2025 · OSTI ID:2573046

Dreslin, Brandon D.; Baweja, Jessica A.

Title: Exploring the role of judgement and shared situation awareness when working with AI recommender systems

Citation Formats

References (41)

Similar Records

Related Subjects