Simon W. S. Fischer, Hanna Schraffenberger, Miranda L. van Hooff, Casper Reijnen, Serge Thill, Pim Haselager
Decision support systems (DSS) assist with clinical decision-making. Little is known, however, about the decision-making process with DSS in clinical practice and the extent to which clinicians consider or rely on the predictions. Studies on human-AI decision-making usually focus on laypeople, simplified decision tasks, or one-shot interactions with prototypes that are not integrated into existing workflows. To gain insights from real-world practice, we conducted an online survey (n=12) and semi-structured interviews (n=10) with clinicians from two different medical cohorts each working with a DSS. Findings show, amongst other things, that the interaction with DSS changes as experience increases, the predictions become more important in complex cases, that decisions are discussed with colleagues, and that workflow integration affects decision-making. Our findings on human, technical, contextual, and organisational factors contribute to a socio-technical perspective on DSS reliance behaviour. We conclude by discussing the implications of our findings for the design of DSS.
Simon WS Fischer, Hanna Schraffenberger, Miranda L. van Hooff, Serge Thill, Pim Haselager
A major challenge in AI-supported decision-making is achieving appropriate reliance on AI. To reduce overreliance on AI, approaches for cognitive friction have been proposed that promote reflection and critical thinking. In this paper, we investigate how two such approaches are perceived by medical experts in a real-world decision task. We replicated an existing decision support system used in clinical practice and extended it with outputting reflective questions and what-if scenarios. We gathered clinicians' (n=7) feedback on this prototype through in-situ interviews. While the questions were perceived as unhelpful for reflective thinking in itself, their potential was seen in serving as reminders to consider relevant information, e.g., when attention wanes. Inspecting what-if scenarios was found useful for considering alternative outcomes and informing discussions with patients. Our participants regarded the prototype as a promising training tool for novice clinicians. From the clinicians' feedback, we make recommendations for designing friction in human-AI interactions.