- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies selective participation for conversational agents, i.e. the problem of not only answering questions in a multi-party dialogue but identifying which ones are addressed at the agent and only answering those. The authors argue that existing training and evaluation approaches are prone to the "scoreboard masking" phenomenon which they introduce as assigning high scores to an agent that answers questions addressed to someone else. To study this, they build a controlled three-party simulator where an agent explicitly decides whether to speak or to wait at annotated moments (question to self, question to another participant, pause request, bad-news event). They compare several credit assignment strategies and evaluate models at matched answering rates. The authors' main finding is that the supervised models substantially reduce answering questions addressed to others (0.548 → 0.205-0.249 in the controlled study) while maintaining the same own-address answering rate. Among supervision signals, per-turn supervision learns the addressee cue more strongly than per-dialogue supervision. Another interesting result is late-layer activation patch (replacing one of the layers' activations obtained on a slightly different input prompt with a different addressee) transfers much of the learned selectivity, providing a mechanistic result that the distinction is encoded in a localized decision state. Results replicate across several model families and partially transfer to external participation benchmarks while largely preserving capability benchmarks.
Critical strengths
- A new answer-rate matching evaluation protocol that removes the main confound, participation frequency
- Identification and formalization of "scoreboard masking" failure mode that is distinct from answer quality and highly relevant for multi-agent systems
- An efficient supervision strategy for mitigating false addressee answering. The work shows that a relatively simple decision-level labeling scheme substantially reduces false addressee answering while maintaining comparable participation rates.
- Mechanistic evidence via activation patching. The activation-patching experiments provide evidence that the model develops an internal representation related to addressee recognition and participation decisions. This increases confidence that the observed gains reflect genuine learning of selective participation rather than artifact exploitation
Critical weaknesses
- Heavy reliance on a synthetic simulator. Nearly all core results rely heavily on a bespoke simulator whose dynamics, incentives, and interaction patterns are designed by the authors.
- Limited external validation. Transfer to outside datasets is mixed. The external benchmarks only partially capture the selective-participation setting studied in the paper, and the strongest improvements remain concentrated in the authors' custom environment. As a result, it is difficult to assess whether the learned behavior reflects a broadly applicable conversational norm or adaptation to the specific structure and addressing conventions of the simulator.
- Reliance on explicit addressee signals. Much of the training and evaluation setup centers on explicit vocative addressing (e.g., "Ada", "Ben"), making it relatively easy to determine the intended respondent. It remains unclear whether the learned participation policy would generalize to more realistic settings where addressee identification depends on implicit discourse cues, conversational context, expertise assignment, interruptions, overlapping speech.
Questions for authors
Q1. How much of the observed gain remains when explicit addressee cues are removed? Q2. How much transfer occurs without benchmark-specific retraining? Q3. Can scoreboard masking be demonstrated on external benchmarks?