Abstract
The growing capabilities of AI models are leading to their wider use, including in safety-critical domains. Explainable AI (XAI) aims to make these models safer to use by making their inference process more transparent. However, current explainability methods are seldom evaluated in the way they are intended to be used: by real-world end users. To address this, we conducted a large-scale user study with 85 healthcare practitioners in the context of human-AI collaborative chest X-ray analysis. We evaluated three types of explanations: visual explanations (saliency maps), natural language explanations, and a combination of both modalities. We specifically examined how different explanation types influence users depending on whether the AI advice and explanations are factually correct. We find that text-based explanations lead to significant over-reliance, which is alleviated by combining them with saliency maps. We also observe that the quality of explanations, that is, how much factually correct information they entail, and how much this aligns with AI correctness, significantly impacts the usefulness of the different explanation types.
Main results
NLEs on their lead to overreliance: For incorrect advice, there is a significant drop in human accuracy for NLEs compared to combined. This suggests that NLEs make people more likely to agree with the AI when it is incorrect. This could suggest that the assertiveness and/or human-like nature of language-based explanations could lead people to overly trust and rely on AI.
Explanation correctness needs to align with AI output correctness: Our results show that insightful explanations are helpful in a decision-support setting. higher explanation correctness scores harm human accuracy when the AI prediction is incorrect (deceptive explanations) and benefit human accuracy when the AI advice is correct (insightful explanations). These effects are less strong for NLEs than for the visual methods. As a general trend, human accuracy is harmed when explanations are deceptive, and people would be better off seeing no explanation.
When aligned, combine saliency maps and NLEs: For insightful explanations, where correctness aligns with AI correctness, combining saliency maps and NLEs provides significant improvements compared to the other conditions. Interestingly, even though insightful NLEs on their own are worse than “No XAI”, combining them with visual explanations leads to a significant boost.
Exploratory results
In addition to our main research question, we also measured “perceived usefulness”, “decision speed”, and “positive certainty”:
Perceived usefulness: NLEs, in line with our post-study survey, are consistently rated the most useful. Even though this contrasts actual usefulness (human accuracy), there is no significant difference in how perceived and actual usefulness misalign between explanation types.
Positive certainty: We find that it is hard to predict positive certainty and that it does not vary significantly by explanation type.
Decision speed: Decision speed increases significantly with the level of complexity of the explanations, going from 36.0 seconds for no explanation to 39.6 for saliency maps, 42.8 for NLEs, and 43.1 for combined.
Summary & Outlook
We juxtaposed textual (NLEs) and visual (saliency maps) explanations and found that NLEs lead to severe overreliance, but can be helpful when combined with visual explanations. We also show that alignment between explanation and advice correctness is a strong predictor for explanation usefulness. This study sheds light on the pitfalls of convincing-sounding language-based explanations and we hope it enables future research on optimizing such explanations to lead to safe use of AI.
Excited to explore further? Send us an email at [email protected].
