acceptodds
Withdrawn from ICLR 2027

Which Messages,at What Budget? A Complete-Subset Audit of Learned Message Sharing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In multi-agent scientific reasoning, a learned selector decides which of several proposed solutions reach the model that writes the final answer. When that choice leaves accuracy unmoved, accuracy alone cannot tell you why: the developer cannot see whether a winning combination was unavailable, available but not selected, or selected but not repeatable. We separate the three by recording what the answering model does under all 32 subsets of five fixed messages, across 975 scientific questions. The result is a diagnosis rather than a verdict. Winning combinations turn out to be plentiful, as some subset succeeds on 91.2% of biology questions against 70.9% for answering alone, yet neither of two independently seeded selectors converts that headroom into a significant gain on any of five cohorts. Within the selector family we evaluate, the shortfall is in selection rather than in what was available to select. The same tables show that a harmful message rarely forces a rebuild: 245 of 255 affected questions still admit an ordering that is optimal at every budget. But structure is easy to overstate, and our own controls temper it: four of five cohorts match a density-preserving null, and the one exception does not recur on fresh executions. This study therefore suggests a practical order of work: diagnose which component limits the pipeline before increasing its message budget. Absent combinations implicate the candidate generators or the answering model; available but unselected ones implicate the selection policy; unstable ones implicate re-execution variance and receiver alignment. Evaluation should indicate which component to improve next, rather than motivate more messages.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, large language models, agent communication, message selection, empirical analysis, evaluation methodology, scientific reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2nsCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an empirical audit of learned message selection under communication budgets. By evaluating all 32 subsets of five candidate messages per question, it distinguishes whether a successful subset exists, whether a selector identifies it, and whether its success persists across executions. It also examines whether optimal selections remain compatible as the budget increases.

Critical strengths

None

Critical weaknesses

The fixed pool of five messages lacks sufficient justification or an ablation over pool size. With so few candidates, restrictive budgets, such as forwarding only two messages, require a clearer practical motivation. Larger pools would make this motivation more compelling, but the exhaustive audit requires 2^M receiver evaluations for M messages. The tension between a practically motivated selection problem and computational tractability remains unresolved.

The paper measures context-dependent message effects but does not systematically model or control whether the underlying evidence is complementary, contradictory, or redundant. These relationships are central to the selection problem. Training selectors on subset outcomes without examining how these relationships affect their behavior provides limited explanation of what makes selection difficult or why the learned policies fail.

The contribution is primarily an evaluation framework rather than a new selection algorithm. Its value therefore depends on the insights gained from the audit. However, the findings remain largely descriptive, and the paper does not clearly establish what new, generalizable guidance they offer for designing effective selectors.

Questions for authors

What practical constraint motivates the five-message setting and its budgets? Can the authors vary the candidate-pool size and report how the main findings and computational costs change?

Reviewer_TDuBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits a single message-sharing stage with five cached candidate messages. It evaluates all 32 subsets on 975 scientific questions, separating recorded availability of successful subsets, a learned selector's ability to choose them, and repeatability under fresh receiver executions. Several Qwen3-4B selectors are trained on GPT-OSS mathematics outcomes and evaluated primarily with a Gemma receiver. The study finds substantial hindsight headroom but no statistically resolved selector improvement over receiver-only answering. It also studies whether optimal subsets across exact message budgets can be represented by one growing ordering, using density-preserving controls and repeat experiments to qualify the apparent structure.

Critical strengths
  1. The "optimal growing path" construct provides a clear lens on whether improvements across budgets can be accomplished via incremental addition without replacement.
  2. The paper separately reports rescued answers, newly incorrect answers, and the amount of communication used, showing that an unchanged net accuracy can conceal substantial countervailing effects.
Critical weaknesses
  1. The audit's availability, rescue, and growing-path results all rest on one fixed pool of five messages, but the paper does not explain how those messages were generated. Section 3.1 only names Diverse Hypothesis Deliberation and cites Yang et al. (2026), stating that different reasoning roles propose solutions and that GPT-OSS-120B generates the messages.
  2. The learned selectors are evaluated only with a Qwen3-4B . It remains unclear whether the observed selection shortfall persists across other selector model families or capacities.
  3. The selectors are trained on outcomes from the GPT-OSS receiver and evaluated primarily with a Gemma receiver, without retraining on that target receiver. The present results therefore do not separate unsuccessful selection learning from the change of receiver. Figure 4c further shows that frozen hindsight choices do not yield a statistically resolved improvement over receiver-only answering under fresh executions.
Questions for authors
  1. Please specify the Diverse Hypothesis Deliberation procedure used to generate the five cached messages.
  2. Does the selection shortfall persist for selectors other than Qwen3-4B, across other model families or capacities?
  3. Without retraining on the Gemma receiver, how can unsuccessful selection learning be separated from the change of receiver? Do the fresh-execution results in Figure 4c change that interpretation?
Reviewer_bwZxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an approach for auditing the internal steps of a multi-agent approach to answering scientific questions. The hypothesis is that this approach can help identify what went wrong when a selector fails to pick a correct answer from an ensemble of input messages from different agents: was the correct answer there in the inputs but not selected by the policy? Or was it not present? The claim is that this can help make the whole system more efficient, by diagnosing whether the components a successful answer might be present before increasing the message budget for the overall system.

Critical strengths

None.

Critical weaknesses

I found the paper hard to follow, and hard to appreciate any meaningful contribution. The claimed contribution at the end of the abstract is one I found to be unsurprising and obvious: "This study therefore suggests a practical order of work: diagnose which component limits the pipeline before increasing its message budget. Absent combinations implicate the candidate generators or the answering model; available but unselected ones implicate the selection policy; unstable ones implicate re-execution variance and receiver alignment. Evaluation should indicate which component to improve next, rather than motivate more messages.

The research appears to be conducted by AI agents, as the authors' AI statement clearly and forthrightly acknowledges. The writing felt lik AI prose, needlessly long and wordy. The combination of the AI writing with an uninteresting and uninspiring research contribution made me resent having to review this paper, honestly, and feel like it was a waste of my time. That could contribute to my low score, but since a research paper is -- I think? Hopefully? -- still intended to be one that human researchers find valuable and not off-putting to read, I think this is a valid critique and weakness in the paper, so I am flagging it here.

Questions for authors

I would appreciate any responses to the weaknesses raised above. I am not confident they would cause me to raise my score, but it's possible they could.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.