acceptodds
Withdrawn from ICLR 2027

Not Every Question Is Yours: Learning Whom To Answer In Multi-Party Dialogue

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A dialogue agent can give a relevant answer and still speak out of turn. We study selective participation: answering when addressed while leaving questions intended for other participants to them. In controlled multi-party dialogue, better episode scores can hide persistent errors in whom an agent answers. We make participation an explicit speak-or-wait decision and evaluate it on shared contexts at matched answering rates. Our central finding is that learned selectivity can be transferred through a late internal state. In Qwen3-VL-8B, inserting the trained model’s state at one decision position increases the own-versus-other speaking-probability gap by 21 percentage points, while the reverse intervention removes 22 points. Addressee-only counterfactuals and participant renaming show that the effect follows conversational role, while a development-calibrated output bias produces less separation. A five-seed training study finds that four supervision schemes reduce other-addressed responses at a fixed own-address answering rate. Experiments across five models, including a full-duplex speech model, and on When2Speak extend the behavioral evidence. Together, these results connect participation training to internal action selection and show why dialogue agents should be evaluated on whom they answer, not only how often they speak.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-party dialogue, selective participation, turn-taking, conversational agents, addressee selection, activation patching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yqHWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies selective participation for conversational agents, i.e. the problem of not only answering questions in a multi-party dialogue but identifying which ones are addressed at the agent and only answering those. The authors argue that existing training and evaluation approaches are prone to the "scoreboard masking" phenomenon which they introduce as assigning high scores to an agent that answers questions addressed to someone else. To study this, they build a controlled three-party simulator where an agent explicitly decides whether to speak or to wait at annotated moments (question to self, question to another participant, pause request, bad-news event). They compare several credit assignment strategies and evaluate models at matched answering rates. The authors' main finding is that the supervised models substantially reduce answering questions addressed to others (0.548 → 0.205-0.249 in the controlled study) while maintaining the same own-address answering rate. Among supervision signals, per-turn supervision learns the addressee cue more strongly than per-dialogue supervision. Another interesting result is late-layer activation patch (replacing one of the layers' activations obtained on a slightly different input prompt with a different addressee) transfers much of the learned selectivity, providing a mechanistic result that the distinction is encoded in a localized decision state. Results replicate across several model families and partially transfer to external participation benchmarks while largely preserving capability benchmarks.

Critical strengths
  1. A new answer-rate matching evaluation protocol that removes the main confound, participation frequency
  2. Identification and formalization of "scoreboard masking" failure mode that is distinct from answer quality and highly relevant for multi-agent systems
  3. An efficient supervision strategy for mitigating false addressee answering. The work shows that a relatively simple decision-level labeling scheme substantially reduces false addressee answering while maintaining comparable participation rates.
  4. Mechanistic evidence via activation patching. The activation-patching experiments provide evidence that the model develops an internal representation related to addressee recognition and participation decisions. This increases confidence that the observed gains reflect genuine learning of selective participation rather than artifact exploitation
Critical weaknesses
  1. Heavy reliance on a synthetic simulator. Nearly all core results rely heavily on a bespoke simulator whose dynamics, incentives, and interaction patterns are designed by the authors.
  2. Limited external validation. Transfer to outside datasets is mixed. The external benchmarks only partially capture the selective-participation setting studied in the paper, and the strongest improvements remain concentrated in the authors' custom environment. As a result, it is difficult to assess whether the learned behavior reflects a broadly applicable conversational norm or adaptation to the specific structure and addressing conventions of the simulator.
  3. Reliance on explicit addressee signals. Much of the training and evaluation setup centers on explicit vocative addressing (e.g., "Ada", "Ben"), making it relatively easy to determine the intended respondent. It remains unclear whether the learned participation policy would generalize to more realistic settings where addressee identification depends on implicit discourse cues, conversational context, expertise assignment, interruptions, overlapping speech.
Questions for authors

Q1. How much of the observed gain remains when explicit addressee cues are removed? Q2. How much transfer occurs without benchmark-specific retraining? Q3. Can scoreboard masking be demonstrated on external benchmarks?

Reviewer_uSzMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles the problem of teaching LLM agents to answer at appropriate times in group conversations. Prior work has relied mostly on a complete dialogue feedback given after all turns of a conversation are done. The authors challenge that approach by proposing that a finer-grained feedback would be more useful to label the training data. This feedback is divided into a few conversational situations such as when the agent is addressed, when someone else is addressed, when there is a long pause, and when some bad news have been shared by someone. The authors build an environment around the LLM they wish o train composed with other LLM models portraying as chat members in order to collect and annotate the data. The main contribution is their evaluation of different feedback constructions forms. Their results show that a feedback summarizing the whole conversation does not improve a model's capability to select when to answer or stay silent.

Critical strengths
  1. The work addresses an important problem of teaching LLM agents when to speak and when not to. The research field of conversational agents has been moving towards reducing agent's generic response behavior towards more human-like responses that respect context. The authors provide a valuable contribution towards understanding how finer-grained feedback can be the path to more human-like conversational agents.
  2. The paper provides convincing experimental results and a sound methodology. The authors experiment with different feedback constructions on fixed data and on data generated by each during interaction with their experimental setup. Furthermore, they run their models on open benchmarks and compare against fine-tuned baselines. Together, these experiments provide credible evidence supporting the paper's main claims.
Critical weaknesses

Major Concerns:

  1. Clarity and organization are concerning. The paper is very difficult to follow due to unclear explanations and extensive use of paper-specific terminology, especially throughout introductory sections. Terms are often used without their proper introduction and explanation to the reader. Their meaning is often only clear in later sections of the paper due to the context they appear. Section and subsection titles provide little indication of their contents making it difficult to follow the progression of ideas. Additionally, the paper lacks a dedicated "Related Work" section. The only mention of prior work for positioning the paper's methodology and hypothesis is a single paragraph in the introduction. These issues significantly hinder the reader's ability to understand and contextualize the paper.
  2. The authors do not sufficiently interpret or discuss their findings. Although the paper presents convincing experimental results and a sound methodology, it does not establish conclusions from these experiments. The authors often limit their discussion to describing tables and charts numerically rather than examining why the observed outcomes occur, what they reveal, and what broader implications they carry. Without exploring the scientific implications readers are left to infer the significance of the findings which weakens the paper's role in communicating its contribution.
  3. The authors state a supplementary material in the Reproducibility Statement but none was submitted.

Minor Concerns (do not affect score):

  1. Some Reinforcement Learning terms, or RL-inspired are used throughout the paper such as credit assignment, reward, return, critic. however, the paper does not use a Reinforcement Learning loss or setup. Data is annotated and the loss is supervised. I suggest clarifying this in the text, either by using different terms, or by plainly stating that there is no RL applied as not to mislead readers.
  2. Line 155: The statement "dense and episode return flip 77 and 607 targets" is unclear. Does this refer to flipping the target values of 77 and 607 data points, respectively? If so, the authors should clarify the procedure and explain the significance of these numbers in the context of the discussion.
  3. The paper presents an unusual amount of Appendix sections, however, very few of them are actually referenced to in text. I would suggest the authors to review the need for those sections and should a section be judged as important, then make sure it is properly referenced in the main text to guide the reader to it should he or she choose to dive into those details.
Questions for authors
  1. The main text, along with appendix Y suggest that the pipeline study was also executed with multiple seeds and bootstrapped. Could the authors confirm whether this was the case?
  2. In the controlled study the feedback construction modes of dense, flat, and decision credit present very close results. Could the authors clarify why the paper favors decision credit over the others?
  3. Could the authors provide the pipeline study results for the dense, and flat feedback construction methods?
  4. Why was the trained model able to come close to the baseline results in the When2Speak dataset but not in the "Speak or Stay Silent"? Matching or beating a baseline here is not a concern, I am interested in the differences between both datasets that might have lead to this.
  5. The authors say that in the pipeline study it "lets each construction set its own rows and anchors". Does this mean they generate their own data via interactions with the environment? If so, how is comparability between the constructors methods maintained?
Reviewer_FCYLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies selective participation in multi-party dialogue: an agent should answer questions addressed to it and stay silent when a question is addressed to someone else. The authors build a three-party text simulator with LLM-driven peers and a scripted facilitator. The trainee is polled every 2 s to choose speak or wait, and four annotated time windows define the target behavior: answer when addressed, wait when another participant is addressed, wait after a request for thinking time, and respond to bad news. They argue that episode-level scores can improve while the agent still answers for others ("scoreboard masking"). The decision is trained with offline signed-weight SFT on base-policy rollouts, and four ways of assigning credit (per-decision, scenario-centered, episode return, and sign only) are compared on identical rows. At a matched own-address answering rate, all four cut other-addressed responses from 0.548 to about 0.21-0.25. On 41 addressee-swap pairs, only the per-decision variants widen the own-versus-other gap (0.31 to about 0.70). Activation patching places the learned difference in a late residual state at the decision position. Results are also reported on When2Speak, Speak or Stay Silent, AMI, and capability benchmarks.

Critical strengths
  1. The paper points to a question that is easy to overlook when evaluating conversational agents: not only how often an agent speaks, but whose question it answers.

  2. Comparing models at a matched answering rate is a sensible idea, since it keeps a model from looking selective just by speaking less.

  3. The authors report negative results, failed pre-registrations, and costs openly, which makes the limitations of the work easy to see.

Critical weaknesses
  1. The application scenario is narrow, and the paper's premise is questionable. Settings that require an assistant to decide on its own whether to answer in a multi-party conversation are still uncommon. Most current products rely on mentions, wake words, or routing, so the practical scope of the problem is limited. The paper motivates the setting only briefly in the introduction. It also treats a correct answer to a question meant for someone else as an error, yet in real meetings such an answer can often be helpful. The authors acknowledge that waiting is "a specified policy, not a universal rule." Their own rating audit shows that an LLM observer penalizes answering for others only slightly (0.36 SD). The paper does not provide evidence that this behavior harms users.

  2. The scenario design is one-sided. In the main setup, bad news is the only moment where the agent is rewarded for speaking without being asked. Good news also calls for a response, and staying silent there can seem just as cold. Corrections and open requests for help are also missing, and the paper does not explain this choice. With only one such trigger, it is unclear whether the model learns to respond to important news or simply to negative wording. The pause rule has the opposite problem. It forbids speaking while the person asked is hesitating, although helping a hesitating colleague is often appropriate, as the cited work notes (Lerner, 2003; Stivers & Robinson, 2006). It also does not distinguish a request for time from a sign that the person asked cannot answer. Finally, the windows depend only on time and ignore who is talking. In Figure 9e, responding to bad news is marked as the correct action while the facilitator is still speaking, and the trained arms do so with probability 0.15-0.38.

  3. The proposed method is thin, and the main comparison is inconclusive. The method ("decision credit") is fine-tuning on rule-based speak/wait labels, with a penalty for drifting from the base model. It does not introduce a new learning algorithm. Its main competitor, "episode return," gives every decision in an episode the same episode-level score in a single offline pass. This reverses 607 labels, including 82 of the 183 labels on questions addressed to the agent itself, which makes it a weak stand-in for outcome-based RL such as GRPO. Even so, all four variants reach a similar rate of answering for others (0.21-0.25, Table 2), and the evidence that they learn different cues comes only from 41 name-swapped pairs from 22 scenario families. The trained model also speaks more at ordinary moments (0.29 to 0.51, Table 2), and a blind judge rates its utterances 0.45 points lower on a nine-point scale.

  4. The task is mostly name matching, and the baselines are weak. Questions in the simulator address people by name at the start of the sentence. A keyword regex can replace the annotated windows and still retain 84% of the training gain, and the authors note that it is template-matched to their generator. The untrained Qwen3-VL-32B does better than the four main trained 8B arms (0.139 vs. 0.21-0.25). The prompt baseline adds one sentence to a prompt that still asks the agent to contribute actively. The name-based rule baseline answers only 0.167 of the agent's own questions, mainly because it checks only the last utterance. There is no comparison with an addressee-detection step followed by a decision, and no closed model is evaluated in the simulator. Harder cases, such as "what about you?" or questions to the whole group, are left to future work.

Questions for authors
  1. Which real applications need an agent to decide on its own whether to answer, without mentions or routing? Is there evidence that a correct answer to someone else's question harms users?

  2. Why is bad news the only moment where unprompted speaking is rewarded? How does the model respond to good news, corrections, or open requests for help? Can the pause rule tell "give me a moment" from "I don't know"?

  3. Can you compare against on-policy outcome RL (for example GRPO over several iterations) rather than the single-pass relabeling?

  4. How does the model compare with an addressee-detection pipeline, a rewritten prompt, a rule that checks every utterance since the agent last spoke, and a closed model? How does it handle implicit or group addressing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.