acceptodds
Under review as a conference paper at ICLR 2027

RPO-MAS: RECEIVER PREFERENCE OPTIMIZATION FOR SELECTIVE MULTI-AGENT COMMUNICATION

Abstract

Communication is a critical component of multi-agent reasoning, yet peer communication does not always improve the receiver's decision. A peer response that is correct in isolation may be redundant or harmful for a receiver with a different reasoning state, while autonomous refinement or preserving the current response may lead to better outcomes. Existing approaches for selective multi-agent communication mainly rely on source reliability, peer selection, or communication structures, providing limited supervision on whether a particular intervention benefits the receiver. In this work, we propose RPO-MAS, a receiver-conditioned preference optimization framework for selective multi-agent communication that learns intervention preferences from receiver-side outcomes. RPO-MAS formulates communication selection over three intervention types: peer-conditioned updating (SEND), autonomous refinement (SELF), and state preservation (KEEP). Starting from the same receiver state, it compares alternative interventions and constructs outcome-based preferences from their observed correctness outcomes. Based on these preferences, RPO-MAS learns a receiver-conditioned scorer to select appropriate interventions among candidate actions. Evaluation across seven reasoning benchmarks with Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct agents shows that RPO-MAS consistently improves multi-agent reasoning performance over independent aggregation and communication-based baselines. Compared with Majority Voting, RPO-MAS improves average accuracy from 76.91% to 77.72% and from 72.42% to 73.13% on the two backbones, respectively. Further analysis shows that receiver-conditioned supervision improves correction of erroneous states while reducing harmful changes to correct states.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Reasoning, Receiver-Conditioned Communication, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aFwJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RPO-MAS, a receiver-conditioned intervention-selection framework for multi-agent communication. For each receiver state, the method considers three possible interventions: SEND, where the receiver updates its answer based on a peer response; SELF, where the receiver performs an additional autonomous refinement step; and KEEP, where the receiver preserves its current state.

The method constructs preference data by executing all candidate interventions from the same receiver checkpoint and comparing their resulting answer correctness. A lightweight scorer is then trained with a pairwise ranking objective to predict which intervention should be selected at inference time.

Overall, the paper presents a reasonable formulation of receiver-aware communication selection, but the empirical gains are modest and the main methodological contribution appears incremental. More importantly, the method relies on task- and backbone-specific preference supervision constructed using ground-truth correctness, while providing no convincing evidence that the learned intervention policy generalizes beyond the specific task distributions used to train it.

Critical strengths
  1. The intervention decomposition is conceptually clear.
    Separating communication into SEND, autonomous refinement through SELF, and no intervention through KEEP is a useful way to distinguish peer information from additional computation and state preservation.

  2. The matched-counterfactual construction is reasonable.
    Evaluating candidate interventions from the same receiver checkpoint is a sensible attempt to isolate the effect of different intervention choices. The inclusion of SELF and KEEP is also preferable to comparing communication against an insufficient baseline.

  3. The paper is generally well organized.
    The problem formulation, scoring model, and inference procedure are described relatively clearly, and the experimental protocol is sufficiently specified to understand the proposed system.

Critical weaknesses

1. The empirical gains are small and may not be statistically reliable

The improvement over majority voting is only 0.81 percentage points for Qwen2.5-7B-Instruct and 0.71 points for Llama-3.1-8B-Instruct. Several individual results are unchanged or worse than majority voting. Given the small evaluation subsets and stochastic LLM generation, the paper should report results across multiple seeds, at least with mean-std.

2. The method relies on oracle-like, ground-truth-derived supervision

The controller is trained by executing SEND, SELF, and KEEP, then using the ground-truth correctness of their outputs to construct preference labels. This is a valid supervised policy-learning setup, but it is close to hindsight or answer-aware imitation and weakens the claim that the method learns a generally available communication principle. The paper should more clearly position RPO-MAS as task-specific supervised intervention selection.

3. There is no evidence of cross-task or out-of-distribution generalization

Separate controllers are trained for every dataset and backbone, while evaluation remains within the same task and model family. There are no experiments on unseen tasks, cross-dataset transfer, cross-backbone transfer, prompt changes, different team sizes, or distribution shifts. Thus, the results may reflect benchmark-specific pattern fitting rather than a transferable receiver-conditioned policy.

4. The algorithmic novelty is limited and relevant baselines are missing

SEND, SELF, and KEEP are standard communication, self-refinement, and no-op operations. The main addition is a lightweight scorer trained on outcome-derived pairwise labels. The paper does not compare against simple disagreement/confidence heuristics, learned communication gates, contextual bandits, or other reward-based selectors, making it difficult to isolate the value of the proposed preference objective.

5. Preference labels are noisy and based on limited rollouts

Candidate interventions appear to be judged from single sampled outcomes, even though LLM generation is stochastic. A single correct or incorrect completion is a noisy estimate of an intervention's expected utility. The paper should analyze label stability across seeds and repeated rollouts, as well as sensitivity to the amount of preference data.

6. The evaluation protocol is narrow and the receiver-level metrics are difficult to interpret

The experiments use five homogeneous agents, two interaction rounds, two backbones, and task-specific controllers trained from only 400 examples. In addition, low harm can be achieved by selecting KEEP frequently, so correction and harm rates should be interpreted jointly with intervention frequency, computation cost, and final team accuracy. The very low selection rate of SELF also raises the question of whether the full three-way intervention space is necessary.

Questions for authors

1. Does the learned policy generalize? Please evaluate cross-task, cross-backbone, or out-of-distribution transfer instead of training a separate controller for every dataset and backbone.

2. How stable are the preference labels? Since labels appear to be based on single sampled outcomes, please report results using repeated rollouts and analyze preference-label stability.

3. How much of the gain comes from the proposed objective rather than adaptive gating? Please compare against simple disagreement/confidence heuristics and a learned KEEP/UPDATE or contextual action-selection baseline.

4. Is SELF necessary? Given its very low selection frequency, please report an ablation using only SEND and KEEP, together with performance as the number of agents and interaction rounds changes.

5. What is the cost-benefit trade-off? Please report the cost of constructing the preference data and relate intervention frequency, correction rate, harm rate, computation cost, and final team accuracy.

Reviewer_iUrLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RPO-MAS, a receiver-conditioned controller that selects among preserving the current response, independent refinement, and updating with a selected peer’s response. Preference labels are constructed by executing these alternatives from the same receiver state and comparing answer correctness. Experiments on seven benchmarks with two LLM backbones report modest improvements over majority voting and larger gains over conventional debate methods.

Critical strengths
  • Distinguishing message correctness from its usefulness to a particular receiver is an interesting perspective. Including SELF and KEEP provides meaningful controls for additional reasoning and non-intervention.

  • Constructing preferences from matched receiver states is well motivated. The ablations suggest that receiver conditioning can reduce harmful updates compared with source-based scoring.

Critical weaknesses
  • The improvement over the strongest simple baseline is too small to be convincing without uncertainty estimates. Average gains over majority voting are only 0.81 and 0.71 percentage points, with evaluation subsets of 100–300 examples. On GSM8K, the one-point improvement corresponds to three additional correct answers. Without repeated runs or paired significance analysis, these results do not establish a reliable advantage over sampling variation.

  • The implementation does not sufficiently substantiate the proposed receiver-specific reasoning perspective. Although the controller receives reasoning-text embeddings, these are compressed to 32 dimensions and combined with agreement and consensus features to predict action preferences from binary outcomes. The experiments do not show whether it identifies which peer argument addresses a specific reasoning error, or mainly learns task-specific correlations and conservative update patterns. This leaves the connection between the conceptual motivation and the learned behavior unclear.

  • The necessity of the full framework is not established against simpler alternatives. SELF is rarely selected, suggesting that much of the behavior may be captured by a KEEP/SEND gate. Comparisons with a binary gate, a consensus-only controller, and compute-matched independent sampling would help determine whether the task-specific preference training provides sufficient value beyond simpler selection strategies.

Questions for authors
  1. Do the gains over majority voting remain significant across repeated decoding runs and paired statistical tests on larger evaluation sets?

  2. Can you demonstrate that selections depend on useful reasoning content—for example, by varying peer rationales while holding final answers and consensus patterns fixed—and compare against a controller using only structural features?

  3. How does RPO-MAS compare with a KEEP/SEND-only controller and majority voting under a matched inference budget, including scoring and additional generation costs?

Reviewer_trafOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose RPO-MAS, a receiver-conditioned preference optimization framework that learns whether peer communication, autonomous refinement, or preservation of the current response benefits a receiver. For a question xx, receiver response RitR_i^t, and candidate peer message mjt=Rjtm_j^t=R_j^t, the interventions are defined as

Ait=K,S∪Mj:j≠i,\mathcal{A}_i^t=\\{K,S\\}\cup\\{M_j:j\ne i\\},

where KK, SS, and MjM_j denote KEEP, SELF, and SEND$(j)$, respectively. Their effects on the receiver are defined as

\begin{cases} R_i^t, & a=K,\\\\ F(x,R_i^t;\xi), & a=S,\\\\ F(x,R_i^t,m_j^t;\xi), & a=M_j. \end{cases}$$ where $F$ is the underlying language model and $\xi$ represents generation randomness. This formulation provides explicit comparisons between peer-conditioned reasoning, additional autonomous reasoning, and preservation of the existing response. The proposed method defines an intervention's utility through the probability that the resulting receiver answer is correct: $$U(a\mid z_i^t)=\Pr_{\xi}\left[\operatorname{ans}\left(\widehat{R}_i^{t+1}(a;\xi)\right)=y\right],$$ where $y$ is the reference answer and $z_i^t$ contains the question, receiver state, and candidate peer messages. During preference-data construction, the active branches are executed from the same receiver state with fixed candidate messages and a shared decoding seed $\xi_i$. KEEP is evaluated directly on the existing response. The observed outcomes and preference pairs are $$r_i(a;\xi_i)=\mathbf{1}\left\\{\operatorname{ans}\left(\widehat{R}_i^{t+1}(a;\xi_i)\right)=y\right\\},$$ $$\mathcal{P}_i=\left\\{(a^{+},a^{-})\in\mathcal{A}_i^t\times\mathcal{A}_i^t:\\ r_i(a^{+}) > r_i(a^{-})\right\\}.$$ These observed correctness differences supervise the scorer, while equal-outcome actions remain unordered. The architecture represents the question, receiver response, and candidate peer response using a frozen MiniLM encoder followed by a training-fitted, 32-dimensional PCA projection. A two-hidden-layer MLP combines these representations with receiver–peer interaction features and pre-action structural information. It learns action scores through a group-normalized pairwise ranking objective with directional balancing and score regularization, while the backbone LLMs remain frozen. At inference, the scorer first identifies the highest-scoring active intervention and compares it with KEEP: $$a_i^\star=\operatorname*{arg\\,max}_{a\in\mathcal{A}_i^t\setminus\\{K\\}} f_\theta(z_i^t,a),$$ $$a_i^t= \begin{cases} a_i^\star, & f_\theta(z_i^t,a_i^\star)-f_\theta(z_i^t,K) > \tau,\\\\ K, & \text{otherwise}. \end{cases}$$ The experiments use $\tau=0$, so exact threshold ties select KEEP. Only the selected active branch is executed. Receivers decide from the same public snapshot and update synchronously, with early termination when all receivers select KEEP and majority voting over the final parse-able answers. The authors evaluate five-agent homogeneous systems using Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct across seven benchmarks, with a separate controller for each dataset and backbone and up to two interaction rounds. RPO-MAS reports average team accuracy of 77.72% and 73.13%, improving over Majority Voting by 0.81 and 0.71 percentage points, respectively. Both methods improve on five benchmarks, match Majority Voting on Arithmetics, and perform comparably to HH-RLHF. Additionally, ablations on GSM8K and CommonsenseQA support the contributions of receiver conditioning, receiver-side outcome supervision, and directional balancing, while the communication analysis shows increased use of KEEP in the second round and relatively infrequent selection of SELF.
Critical strengths
  • A central strength of RPO-MAS is that it selects communication based on the receiver’s current state, recognizing that the same message can have different values across receiver states. The SEND, SELF, and KEEP actions allow the method to assess whether using peer message benefits the receiver compared with refining its answer independently or keeping it unchanged.

  • RPO-MAS constructs training preferences by checking whether each alternative produces a correct receiver answer from the same starting response. An alternative that produces a correct answer is preferred over one that produces an incorrect answer, while alternatives with the same observed correctness create no preference pair. This grounds the training preferences in the observed effects of peer communication, independent refinement, and answer preservation on receiver correctness.

  • Ablation studies on GSM8K and CommonsenseQA provide supporting evidence for receiver conditioning and receiver-side outcome supervision. On GSM8K, the complete method reports a Harm rate of 6.06%, compared with 12.32% for Source-based Scoring, and a Correction Rate of 51.43%, compared with 41.26% for Source-quality Supervision. Among the active interventions selected by each method, these results indicate a lower rate of changing correct answers into incorrect ones than Source-based Scoring and a higher rate of correcting incorrect answers than Source-quality Supervision. The accompanying receiver and team accuracy improvements provide further support for receiver conditioning and receiver-side outcome supervision in their evaluation.

Critical weaknesses
  • The paper defines the preferred intervention as the one most likely to produce a correct receiver answer, but its group-normalized training objective need not preserve this ordering. In Eq. (10), each pairwise loss is divided by the realized number of preference pairs at its checkpoint. Consequently, comparisons favoring an intervention can receive greater weight when fewer other interventions succeed, potentially favoring an intervention with a lower correctness probability. The authors explain that normalization prevents checkpoints with more pairs from dominating training, and Table 2 supports the usefulness of directional balancing. However, the effect of group normalization on intervention selection remains unclear, since the ablations retain this normalization and do not show whether it changes which actions the controller selects.
Questions for authors
  • How does removing the per-checkpoint normalization in Eq. (10), while retaining directional balancing, affect the actions selected by the controller and the resulting receiver accuracy? This comparison could help establish whether equalizing checkpoint contributions improves intervention selection despite potentially changing the ordering of correctness probabilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.