acceptodds
Under review as a conference paper at ICLR 2027

Improving LLM Collaboration via Multi-Agent Preference Learning

Abstract

Several works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback. Yet, its extension to multi-agent systems remains underexplored. To address this gap, we formulate preference-based multi-agent systems (MAS) from decentralized and centralized collaboration perspectives. We also introduce a general multi-agent preference learning framework (MAPL) to solve these problems. MAPL allows iterative updates by comparing the current solution with decentralized or centralized solutions generated by various agents. We instantiate MAPL using MARL from human feedback (MARLHF) with a learned reward model and multi-agent direct preference optimization (MADPO). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO on most tasks but remains sensitive to data coverage, agent and comparator models, and the underlying MARL algorithms.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Preference-based Reinforcement Learning, LLM Collaboration.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RHrkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies preference-based learning for LLM multi-agent collaboration, motivated by the difficulty of designing reliable scalar rewards. The authors formulate decentralized and centralized collaboration as preference-based multi-agent POMDPs and propose MAPL, a framework that collects preferences over joint trajectories to optimize agent policies. MAPL includes two methods: MARLHF, which combines reward modeling with MARL, and MADPO, which directly optimizes policies from preferences. Experiments cover four tasks: summarization, coding, function calling, and travel planning. Using oracle-generated preferences, MAPL outperforms the evaluated test-time collaboration baselines and approaches reward-based MARL on some metrics. Ablation studies examine different collaboration settings, replay buffers, comparator models, and optimization methods.

Critical strengths

The paper provides a thorough empirical evaluation across four LLM collaboration tasks. MAPL outperforms the evaluated test-time collaboration baselines on task performance and approaches reward-based MARL on some metrics. The comparisons between MARLHF and MADPO, along with ablations on replay buffers, comparator models, and agent configurations, provide useful insights into the effectiveness and limitations of preference-based multi-agent learning.

Critical weaknesses
  1. Preference-based MARL has been studied in prior work [1][2], and preference optimization has also been applied to LLM multi-agent systems [3][4]. The paper does not clearly establish what new challenges arise in its setting or what existing approaches fail to address. MAPL mainly adapts existing techniques, including online preference learning, replay buffers, RLHF, and DPO. While the paper motivates some design choices, its new technical contribution remains unclear. It also does not sufficiently explain how joint-trajectory preferences handle agent-level credit assignment, a problem explored in related work [5][6].

  2. MAPL selects preference pairs using oracle returns, despite being motivated by the difficulty of designing reliable scalar rewards. Appendix B.2 suggests using a learned reward model or human-specified metrics instead. The former requires additional reward modeling, while the latter reintroduces a form of reward design. The experiments compare MAPL with single-agent preference learning and reward-based MARL, but not existing multi-agent preference learning methods [2][5] or related LLM multi-agent preference optimization approaches [3][4]. These methods may require adaptation to the evaluated tasks, but the authors should explain their exclusion or provide comparisons where feasible.

[1] 2024 – AAAI – Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning

[2] 2025 – ICML – O-MAPL: Offline Multi-agent Preference Learning

[3] 2025 – Findings of ACL – Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System

[4] 2026 – ACL – AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answering

[5] 2025 – AAMAS – Offline Multi-Agent Preference-Based Reinforcement Learning with Agent-aware Direct Preference Optimization

[6] 2026 – ACL – MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation

Questions for authors
  1. What specific challenges in LLM multi-agent collaboration are not addressed by existing preference-based MARL methods [1][2] or LLM multi-agent preference optimization approaches [3][4]?
  2. What is the main technical contribution of MAPL beyond adapting existing online preference learning, RLHF, and DPO techniques to multi-agent collaboration?
  3. How does MAPL address agent-level credit assignment when preferences are provided only for joint trajectories? What distinguishes its approach from existing multi-agent credit assignment methods [5][6]?
  4. How can MAPL select the top-k preference pairs when oracle returns are unavailable? Appendix B.2 suggests using a learned reward model or human-specified metrics. How do these alternatives preserve the paper's motivation of avoiding manually designed scalar rewards?
1 comment
AuthorsOpenReview ↗

Official Comments by Authors

We thank the reviewer for helping us improve this paper.

 

Re. W1 and Q2: Lack of Novelty

Our main technical contribution is not simply applying preference learning to LLM-based MAS. Rather, this is the FIRST work to systematically compare and analyze different preference-learning methods in centralized and decentralized LLM-based MAS. We also provide the FIRST general framework explaining why preference learning can and should be used to optimize LLM collaboration (Section 3.3). We elaborate on the differences from related work below.

 

Re. Q1: Difference with Related Works

We appreciate the reviewer can bring additional related work into our attention and will discuss them explicitly in the revised manuscript. However, they have substantial differences with us.

  1. MAPT learns agent-wise rewards and temporal aggregation weights, which follows a complete different problem formulation, i.e., POSG. Its solution concept (Nash equilibrium) is fundamentally different than a Dec-POMDP (global optimum). A fully-cooperative MARL problem should be formulated as a Dec-POMDP. In addition, the experiments in MAPT do not reveal the challenges of using preference learning to optimized LLMs.

  2. O-MAPL is an pure offline method whose direct application to LLM-based MAS faces the challenges discussed in Section 3.3. We also explicitly discuss the unique challenges of this setting in Section 3.3, including why offline PbRL methods [2,5] are GENERALLY impractical for optimizing LLM-based MAS. Its value-factorization component also faces the limitations discussed for value-based methods in the “Baselines” paragraph of Section 5.1. Table 2 includes results for offline MAPL.

  3. We do not merely show that preference learning works well in LLM-based MAS, we distinguish centralized and decentralized settings and characterize the methods’ properties, including why MAPL sometimes fails. AMATA and Optima optimize LLM-based MAS under predefined interaction protocols in which agents exchange information through communication. Our work focuses on fully centralized MAS with access to the full joint action-observation history and fully decentralized MAS where agents act on local histories without inter-agent communication, and highlights their respective pros and cons. In addition, to enable a clean comparison of MAPL post-training methods, we do NOT introduce an additional SFT stage.

 

Re. W2&Q4: Oracle Reward Reliance

We sincerely appreciate the review can look at our Appendix B.2 about different sampling strategy, including top-kk. We first want to clarify that we use oracle rewards to select top-kk samples not because other approaches are difficult to construct, but because this provides the most direct and interpretable way to demonstrate the concept.

MAPL does not inherently rely on oracle rewards. It can also use learned rewards, when available, or heuristic metrics. With MARLHF, R~\tilde{R} learns the corresponding causal metrics internally, so oracle rewards are not required. Even when learned rewards are unreliable or unavailable, we can select dominant trajectories (πR∗(ν),∗≻ϖR∗(ν),∗\boldsymbol{\pi}_{R^{\ast}}^{(\nu),\ast} \succ \boldsymbol{\varpi}_{R^{\ast}}^{(\nu),\ast}) or advantageous trajectories (πR∗(ν),∗≻ϖR∗(ν),∗\boldsymbol{\pi}_{R^{\ast}}^{(\nu),\ast} \succ \boldsymbol{\varpi}_{R^{\ast}}^{(\nu),\ast}) using heuristic metrics (Appendix B.2). This reflects our motivation that optimizing agents does not require a highly accurate reward function or model and can instead be bootstrapped from available metrics or simple heuristics.

We report MAPL results with top-kk sampling using reward-free metrics or an AI judge, as suggested by reviewer mkbZ to demonstrate this concept. The results show that MAPL is not very sensitive to the sampling strategy and can learn effectively even with rough metrics.

Table 3: Top-kk Sampling with Other Metrics

 

Re. Q3: Credit Assignment of MAPL

We do not claim credit assignment as an advantage of MAPL. On the contrary, Appendix H identifies it as a potential limitation in long-horizon settings and a direction for future work.

The main focus of this paper is to propose and compare preference-based MARL methods under different setting, analyze when they could success and why they can potential fail under some settings. To have a direct, autonoumous, and clean comparison between MAPL methods, we don't involve specific credit assignment techniques to remove compounding factor that prevent a clean explanation and analysis. Empirically, we just scaling the preference dataset size to overcome this issue, where each task are given 20 pairs of preferences per iteration based on the 4 given tasks in our experiments, though larger and more complex and fundamental post-training in industry would need far more data.

Reviewer_mkbZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies preference-based optimization of collaborative LLM systems. It formulates decentralized and centralized collaboration using preference-based partially observable decision processes and introduces Multi-Agent Preference Learning (MAPL). MAPL iteratively collects comparisons between agent-generated and comparator-generated trajectories and updates policies using either reward-model-based MARLHF or Multi-Agent Direct Preference Optimization (MADPO). Experiments cover summarization, collaborative coding, function calling, and travel planning. The paper compares decentralized and centralized agent–comparator configurations and studies replay buffers, comparator models, optimization algorithms, and agent configurations. The results show improvements over untrained collaboration in several settings and identify important failure modes, including unstable MADPO training and poor transfer from centralized comparators to decentralized agents. All experimental preferences are generated from deterministic oracle rewards.

Critical strengths

The paper provides a useful comparison of preference-learning configurations for collaborative LLMs. Evaluating MARLHF and MADPO under Dec-Dec, Cen-Cen, and Dec-Cen exposes differences that a single favorable configuration would miss. The negative results concerning centralized comparators and actor-critic optimization are particularly informative for designing subsequent systems.

The experimental analysis extends beyond aggregate performance. The replay-buffer, comparator, model-configuration, and agent-count studies help characterize when training succeeds or collapses. Reporting five runs and documenting prompts, reward construction, hyperparameters, and compute requirements makes the controlled experiments easier to assess.

Critical weaknesses
  1. The evaluated pipeline relies on oracle information beyond pairwise preferences, leaving the central motivation insufficiently validated. Section 4.1 selects the top-k preference pairs according to the oracle returns of their winning trajectories. Binary winner–loser labels alone do not provide this ranking across different pairs. Thus, oracle access influences both label generation and which data are retained for training.

Appendix B.2 acknowledges this issue and proposes using a proxy reward when the oracle is unavailable, but the reported experiments do not establish the effectiveness of that replacement. Consequently, the results demonstrate learning under oracle-derived supervision more directly than they demonstrate a practical alternative when reliable scalar rewards are unavailable. An evaluation using preference-only data selection and independently collected human or AI judgments would materially strengthen the contribution.

  1. The evaluation provides limited evidence of task quality and generalization beyond the constructed objectives. For TLDR, Appendix C.3 defines rewards through response lengths, unique-word ratios, vocabulary overlap, and transition-word occurrence. The described reward contains no explicit measure of factual consistency with the source document or coverage of its important information. High scores therefore need not establish that the generated summaries are accurate or useful.

The other evaluation sets are also small: Table 3 specifies 16 CoopHE examples, 40 BFCL examples, and 22 Travel examples. Five runs help assess training variability, but do not resolve uncertainty arising from such limited task coverage. This matters particularly for small reported differences and broad conclusions about collaboration quality. Independent summarization assessment, larger held-out subsets, and uncertainty estimates for the principal task metrics are needed to substantiate those conclusions.

Questions for authors

Clarification: Which parts of the implemented pipeline access numerical oracle rewards beyond generating pairwise labels? How does performance change if replay-buffer selection uses only preference information, without oracle-based ranking?

Assessment-changing evidence: Do the TLDR improvements persist under source-grounded factuality and information-coverage evaluation, or blinded human assessment? Can high-scoring examples be provided to demonstrate that the metric rewards meaningful summaries?

Clarification and robustness: How were hyperparameters and checkpoints selected relative to the small evaluation subsets in Table 3? Can you provide results on larger held-out subsets with uncertainty estimates for functional correctness and final Travel success?

Positioning: How does MAPL differ from the iterative generation, ranking, selection, and DPO training in Optima? A direct discussion would help establish which advances arise from the decentralized formulation, comparator design, or optimization procedure.

2 comments
AuthorsOpenReview ↗

Official Comments by Authors

We thank the reviewer for the insightful feedback.

 

Re. W2&Q2: Experimental Setup

The reviewer questions whether the TLDR reward captures factuality. Note that our core contribution is to compare different MAPL optimization methods, and this setting serves as a proof of concept for aligning agents with a predefined notion of summarization quality. Once the objective is fixed, the comparison would be fair. Moreover, the quality of a summary is often subjective, multifaceted, and category-dependent.

We use relatively small datasets because datasets suitable for clean comparisons between Cen-MAS and Dec-MAS are limited. The data splits are provided in Table 3. There are total 82 CoopHE instances, 200 selected instances from the BFCL parallel and parallel_multiple partitions, and filtered 3–5-day Travel tasks. We consider those specific partition because we hope the task can be technically solvable by decentralized LLM agents and not much beyond Qwen 4B-8B agents' capabilities to make different MAPL methods or settings separable. Else, it is not very meaningful to optimize and compare different MAPL methods (all result in 0% success or perfectly solvable). According to the suggestion, we report the uncertainty in standard deviation error of Table 1 and 2 as below.

Table 1: Evaluation Results with Standard Deviations

Table 2: Ablation Results with Standard Deviations

 

Re. W1&Q1: Oracle Reward Reliance

Appendix B.2 discusses various sampling strategies, and we will clarify it in the main text. The core purpose of sampling is to select higher-quality samples to facilitate training. Selection can use oracle rewards, learned rewards, or heuristic metrics. For example, in CoopHE, samples can be selected using pass rates or call rates without requiring a learned reward model. We report results using the evaluation metrics in Table 2 for preference labeling and pair selection below. These results show that such metrics can support effective learning in most domains.

Table 3: Table 3: Top-kk Sampling with Other Metrics

To answer Q1, we also ablate different values of kk. At k=20k=20, all non-tied candidate pairs are retained. The results suggest that effective training often requires an appropriate, domain-dependent kk. An excessively large kk can admit low-quality samples and hence dilute the density of good samples, whereas an excessively small kk can have insufficient informative comparisons for effective learning.

Table 4: Top-kk Ablation Results

 

Re. Q4: Comparison with Related Work

We thank the reviewer for bringing additional work to our attention, and we will discuss these studies carefully in the final version. Our core contribution is not simply applying preference-based RL to LLM-based MAS, but systematically comparing and analyzing preference-learning methods across different settings, including comparator information structures, sampling strategies, and online versus offline learning. Notably, we do not merely show that preference learning works well in LLM-based MAS, we distinguish centralized and decentralized settings and characterize the methods' properties, including why MAPL sometimes fails.

Optima optimizes LLM-based MAS under predefined interaction protocols in which agents exchange information through communication. Our work focuses on fully centralized MAS with access to the full joint action-observation history and fully decentralized MAS where agents act on local histories without inter-agent communication, and highlights their respective pros and cons. We also analyze failures of MAPL variants such as Dec-Cen arising from the gap between the two policy classes, and compare the strengths and weaknesses of MARLHF and MADPO. In addition, to enable a clean comparison of MAPL post-training methods, we do NOT introduce an additional SFT stage.

Thanks For your response, after carefully reading the words and the external links, the reviewer still feels unconvinced by most of the main complaints. So, the reviewer decided to maintain the original evaluation

Reviewer_XhpLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAPL, a framework for optimizing cooperative LLM multi-agent systems from preference feedback. It formulates decentralized and centralized collaboration as a preference-based Dec-POMDP and multi-agent POMDP, iteratively builds a replay buffer by comparing the agents’ joint trajectories with those of a comparator policy, and instantiates the framework with MARLHF, a two-stage approach with a learned trajectory reward model and group-relative policy optimization, and MADPO, which applies DPO to joint trajectories. Experiments on summarization, coding, function calling and travel planning show improvements over test-time collaboration and performance close to MARL with oracle rewards.

Critical strengths

S1. The motivation is clear, and the problem of optimizing LLM collaboration without hand-crafted rewards is of practical interest.

S2. Extensive experiments are conducted on four domains, and the code is provided.

Critical weaknesses

W1. The experiments do not test the setting that motivates the paper. First, although this follows the protocol of prior preference-based RL work, all preferences are generated by a deterministic oracle reward, i.e., the same task metric that MAGRPO optimizes directly, so the finding that MAPL approaches MARL with oracle rewards is expected and says nothing about the case where such metrics are unavailable or inaccurate; no human, LLM-judge or noisy preferences are used anywhere. Second, the replay buffer keeps the top-k pairs ranked by the return of the winning trajectory, which requires scalar rewards rather than comparisons, so the method as evaluated does not learn from preferences alone.

W2. The reported improvements are not supported as stated. First, Tables 1 and 2 give averages over the runs without standard deviations, while the margins over the strongest single-agent baseline on the domains where MAPL is said to win are under one point, the single-agent baselines are better on the other domains, and in the matched decentralized comparison MAPL is well below MAGRPO on CoopHE and BFCL. Second, on BFCL the centralized variants issue as many calls per agent as the single agent, so the parallel efficiency that motivates the domain is not obtained, and where the calls are split the correctness is well below the single agent. Please report the standard deviations over the runs for Tables 1 and 2.

W3. The novelty is limited. The method is essentially iterative online preference learning with top-k selection [1] applied to the multi-agent setting of MAGRPO [2], with the reward-model and DPO objectives taken over joint trajectories. Closely related work is also missing: Optima [3] trains LLM multi-agent systems with iterative DPO on reward-ranked trajectory pairs and MALT [4] post-trains a multi-agent pipeline with DPO from outcome-graded trajectories, and neither is cited; the multi-agent preference-based RL works in the reference list, e.g., O-MAPL [5], are not discussed, so it is unclear what the preference-based Dec-POMDP formulation adds to them.

W4. The method is not fully specified in the main text. It is not stated whether the comparator is updated across iterations or kept fixed, how many tasks and preference pairs are used per domain, or how the history-based reward model is parameterized, and the single-agent baseline models are only described as comparable in size.

Minor comments:

  • Sec. 5.2: “MARLHF can even surpass MAGRPO on ... Travel success (13.1% vs. 12.5%)”: a 0.6-point gap without variance does not support “surpass”, and the explanation offered for it is speculative.
  • Table 1: MADPO (Dec-Dec) on Travel is well below the untrained parallel baseline (41.5 vs. 66.2 pass, 0.0 vs. 8.3 success), which is not discussed.
  • Sec. 3.1: the preference function is defined as deterministic, PR∗:Υ×Υ→0,1P_{R*}: Υ×Υ → {0,1}, while Sec. 2 and Eq. 4 use the probabilistic BT model; the two are not reconciled.

References:

[1] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. ICML, 2024.

[2] LLM Collaboration with Multi-Agent Reinforcement Learning. AAAI, 2026.

[3] Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System. Findings of ACL, 2025.

[4] MALT: Improving Reasoning with Multi-Agent LLM Training. COLM, 2025.

[5] O-MAPL: Offline Multi-agent Preference Learning. ICML, 2025.

Questions for authors

Please refer to the Weaknesses above.

1 comment
AuthorsOpenReview ↗

Official Comments by Authors

We thank the reviewer for the valuable comments.

 

Re. W1: Experimental Setting

Weakness 1 questions our use of oracle rewards to generate synthetic preferences and perform top-kk selection. Note that this is just an engineering choice intended to make the experiments intuitive, objective, and interpretable. Our training does not inherently require access to these oracle rewards. As noted at the beginning of Section 5.1, such synthetic preferences are also used in previous work, including influential studies (Christiano et al., 2017; Lee et al., 2021) and references [3,5] cited by the reviewer.

Using human annotators or black-box LLM judges would introduce many confounding factors. Their evaluation criteria and reasoning, particularly regarding cooperation, is difficult to interpreted. Specifically, labeling may depend on annotator persona and biases and the variance may differ across domains.

 

Re. W3: Lack of Novelty

To our knowledge, this is the first work to systematically compare and analyze preference-learning methods in centralized and decentralized LLM-based MAS. We also provide the first general framework explaining the rationale for optimizing LLM collaboration from preferences (Section 3.3). The framework accommodates multiple preference-learning algorithms in a flexible, general setting. We also provide substantial empirical insights into different algorithms and training paradigms in these settings (Figure 3, Table 1(d), and Table 2), and analyze failures of some MAPL variants.

We appreciate the reviewer can bring additional related work into our attention and will discuss them explicitly in the revised manuscript. However, they have substantial differences with us.

  1. Optima optimizes LLM-based MAS under predefined interaction protocols in which agents exchange information through communication. Our work focuses on fully centralized MAS with access to the full joint action-observation history and fully decentralized MAS where agents act on local histories without inter-agent communication, and highlights their respective pros and cons. In addition, to enable a clean comparison of MAPL post-training methods, we do NOT introduce an additional SFT stage.

  2. MALT trains a predefined sequential generator-verifier-refiner workflow using SFT and preference learning. In our setting, agents can learn how to coordinate in both centralized and decentralized MAS.

  3. O-MAPL is a pure offline method whose direct application to LLM-based MAS faces the challenges discussed in Section 3.3.** Its value-factorization component also faces the limitations discussed for value-based methods in the “Baselines” paragraph of Section 5.1.** Table 2 includes results for offline MAPL.

 

Re. W4: Lack Details in the Main

We use the untuned raw models as comparators in Table 2 for a fair comparison, while results with updated comparators appear in Figure 8 in the appendix. Task counts, exact model configurations, and domain-specific hyperparameters are provided in Table 3 and Section 5.3. Table 2 primarily examines design choices for the core modules. The history-based reward model follows Equation 4. When space permits in the final version, we will move more details, including comparator and replay-buffer design, from Appendix D into the main text.

 

Re. W2: Standard Deviations

We provide the standard deviations for the results in Tables 1 and 2 below. The main pattern we observe is that MARLHF exhibits greater variability than MADPO in most domains, as already discussed alongside Figure 3 in the experimental section.

Table 1: Evaluation Results with Standard Deviations

Table 2: Ablation Results with Standard Deviations

 

Re. Minor Issues

Note that we use the word “can” to emphasize MAPL's potential to outperform MARL by learning a reward model that provides better local learning signals. The corresponding variance is reported above.

Appendix D.2 explains MADPO's poor performance on Travel in Table 2: “MADPO training collapses in the middle stage.” We will move this explanation into the main text. As discussed there, this issue occurs across all MADPO variants, including Dec-Dec.

In Section 3.1, we describe “the annotator labels their preference according to PR∗P_{R^*}”, so it includes both the BT model and the sampling step. But we agree that writing it as Δ(0,1)\Delta(\\{0,1\\}) is more clear.

 

References

[1] Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.

[2] Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.