Not Every Question Is Yours: Learning Whom To Answer In Multi-Party Dialogue

A dialogue agent can give a relevant answer and still speak out of turn. We study selective participation: answering when addressed while leaving questions intended for other participants to them. In controlled multi-party dialogue, better episode scores can hide persistent errors in whom an agent answers. We make participation an explicit speak-or-wait decision and evaluate it on shared contexts at matched answering rates. Our central finding is that learned selectivity can be transferred through a late internal state. In Qwen3-VL-8B, inserting the trained model’s state at one decision position increases the own-versus-other speaking-probability gap by 21 percentage points, while the reverse intervention removes 22 points. Addressee-only counterfactuals and participant renaming show that the effect follows conversational role, while a development-calibrated output bias produces less separation. A five-seed training study finds that four supervision schemes reduce other-addressed responses at a fixed own-address answering rate. Experiments across five models, including a full-duplex speech model, and on When2Speak extend the behavioral evidence. Together, these results connect participation training to internal action selection and show why dialogue agents should be evaluated on whom they answer, not only how often they speak.