Understanding AI Teammates through Grounded Dialogue
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
Effective human–AI coordination requires people to understand and anticipate AI teammates' behavior. Yet plausible post-hoc explanations may not faithfully reflect the policy that selected an action. We present PolicyLens, an interactive system that provides evidence-grounded explanations of AI teammates' actions and the possible consequences of alternative human actions. Before deployment, its regularity-constrained policy distillation module extracts compact executable summaries of neural behavior; during collaboration, PolicyLens receives open-ended questions and constructs answers from interaction records, available program paths, or isolated simulations using the same fixed controller. We implement PolicyLens in Warehouse, Cooperative Pong, and an Overcooked-inspired Cooperative Kitchen, matching explanations to the controller used in each environment. In a 60-participant study across the three environments, 53.3% of the explanation group gave positive ratings (5–7 on a seven-point scale) on the item about understanding when their teammate waited or changed plans, compared with 30.0% of the control group, a 77.8% higher proportion. This overall result provides preliminary support for improving participants' perceived understanding of AI teammates. PolicyLens offers a concrete approach to supporting human–AI collaboration through dialogue grounded in verifiable evidence.
- Primary area:
- interpretability and explainable AI