Comprehensive Evaluation of Explanation Types in a Spaceflight-Relevant Human–Autonomy Teaming Task
Journal Article
Overview
abstract
; Objective; This study evaluates how explanation type in an explainable AI (XAI) human–autonomy teaming (HAT) task affects performance, workload, trust, situation awareness (SA), and preference in a dynamic, spaceflight-relevant simulator. Second, we introduce a holistic evaluation method for comparing XAI systems across multiple outcomes.; ; ; Background; XAI aims to improve understanding, calibrate trust, and enhance performance of an HAT, but the impact of explanation type in realistic, high-taskload HAT settings remains underexplored.; ; ; Method; ; Participants (; ; ; ; N; =; 31; ; ; ; ) completed 18 trials in a dual-task simulator requiring manual rover driving while supervising an autonomous exploration agent. Participants received various combinations of global, contrastive, and deductive explanations for AI-generated routes, with incentives tied to performance.; ; ; ; Results; ; Explanation type significantly affected manual performance (; ; ; ; p; =; 0.0003; ; ; ; ), autonomy performance (; ; ; ; p; <; 0.0001; ; ; ; ), team performance (; ; ; ; p; <; 0.0001; ; ; ; ), workload (; ; ; ; p; <; 0.0001; ; ; ; ), trust (; ; ; ; p; <; 0.0001; ; ; ; ), and preference (; ; ; ; p; =; 0.001; ; ; ; ), but not SA (; ; ; ; p; =; 0.41; ; ; ; ). Participants preferred global and contrastive explanations, performing better with their preferred explanation (; ; ; ; p; =; 0.049; ; ; ; ).; ; ; ; Conclusion; Explanation type influences performance and perception in demanding HAT contexts. A standardized, multi-metric evaluation framework is essential for understanding tradeoffs in XAI design.; ; ; Application; In HAT tasks like space exploration where users must quickly make decisions with an AI teammate, designers must consider the explanation method for XAI explanations. Our human-centered evaluation found a contrastive + global explanation combination was the best in our HAT task across a range of performance and preference metrics.;
