- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies preference-based learning for LLM multi-agent collaboration, motivated by the difficulty of designing reliable scalar rewards. The authors formulate decentralized and centralized collaboration as preference-based multi-agent POMDPs and propose MAPL, a framework that collects preferences over joint trajectories to optimize agent policies. MAPL includes two methods: MARLHF, which combines reward modeling with MARL, and MADPO, which directly optimizes policies from preferences. Experiments cover four tasks: summarization, coding, function calling, and travel planning. Using oracle-generated preferences, MAPL outperforms the evaluated test-time collaboration baselines and approaches reward-based MARL on some metrics. Ablation studies examine different collaboration settings, replay buffers, comparator models, and optimization methods.
Critical strengths
The paper provides a thorough empirical evaluation across four LLM collaboration tasks. MAPL outperforms the evaluated test-time collaboration baselines on task performance and approaches reward-based MARL on some metrics. The comparisons between MARLHF and MADPO, along with ablations on replay buffers, comparator models, and agent configurations, provide useful insights into the effectiveness and limitations of preference-based multi-agent learning.
Critical weaknesses
-
Preference-based MARL has been studied in prior work [1][2], and preference optimization has also been applied to LLM multi-agent systems [3][4]. The paper does not clearly establish what new challenges arise in its setting or what existing approaches fail to address. MAPL mainly adapts existing techniques, including online preference learning, replay buffers, RLHF, and DPO. While the paper motivates some design choices, its new technical contribution remains unclear. It also does not sufficiently explain how joint-trajectory preferences handle agent-level credit assignment, a problem explored in related work [5][6].
-
MAPL selects preference pairs using oracle returns, despite being motivated by the difficulty of designing reliable scalar rewards. Appendix B.2 suggests using a learned reward model or human-specified metrics instead. The former requires additional reward modeling, while the latter reintroduces a form of reward design. The experiments compare MAPL with single-agent preference learning and reward-based MARL, but not existing multi-agent preference learning methods [2][5] or related LLM multi-agent preference optimization approaches [3][4]. These methods may require adaptation to the evaluated tasks, but the authors should explain their exclusion or provide comparisons where feasible.
[1] 2024 – AAAI – Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning
[2] 2025 – ICML – O-MAPL: Offline Multi-agent Preference Learning
[3] 2025 – Findings of ACL – Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System
[4] 2026 – ACL – AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answering
[5] 2025 – AAMAS – Offline Multi-Agent Preference-Based Reinforcement Learning with Agent-aware Direct Preference Optimization
[6] 2026 – ACL – MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation
Questions for authors
- What specific challenges in LLM multi-agent collaboration are not addressed by existing preference-based MARL methods [1][2] or LLM multi-agent preference optimization approaches [3][4]?
- What is the main technical contribution of MAPL beyond adapting existing online preference learning, RLHF, and DPO techniques to multi-agent collaboration?
- How does MAPL address agent-level credit assignment when preferences are provided only for joint trajectories? What distinguishes its approach from existing multi-agent credit assignment methods [5][6]?
- How can MAPL select the top-k preference pairs when oracle returns are unavailable? Appendix B.2 suggests using a learned reward model or human-specified metrics. How do these alternatives preserve the paper's motivation of avoiding manually designed scalar rewards?
Official Comments by Authors
We thank the reviewer for helping us improve this paper.
Re. W1 and Q2: Lack of Novelty
Our main technical contribution is not simply applying preference learning to LLM-based MAS. Rather, this is the FIRST work to systematically compare and analyze different preference-learning methods in centralized and decentralized LLM-based MAS. We also provide the FIRST general framework explaining why preference learning can and should be used to optimize LLM collaboration (Section 3.3). We elaborate on the differences from related work below.
Re. Q1: Difference with Related Works
We appreciate the reviewer can bring additional related work into our attention and will discuss them explicitly in the revised manuscript. However, they have substantial differences with us.
MAPTlearns agent-wise rewards and temporal aggregation weights, which follows a complete different problem formulation, i.e., POSG. Its solution concept (Nash equilibrium) is fundamentally different than a Dec-POMDP (global optimum). A fully-cooperative MARL problem should be formulated as a Dec-POMDP. In addition, the experiments in MAPT do not reveal the challenges of using preference learning to optimized LLMs.O-MAPLis an pure offline method whose direct application to LLM-based MAS faces the challenges discussed in Section 3.3. We also explicitly discuss the unique challenges of this setting in Section 3.3, including why offline PbRL methods [2,5] are GENERALLY impractical for optimizing LLM-based MAS. Its value-factorization component also faces the limitations discussed for value-based methods in the “Baselines” paragraph of Section 5.1. Table 2 includes results for offline MAPL.We do not merely show that preference learning works well in LLM-based MAS, we distinguish centralized and decentralized settings and characterize the methods’ properties, including why MAPL sometimes fails.
AMATAandOptimaoptimize LLM-based MAS under predefined interaction protocols in which agents exchange information through communication. Our work focuses on fully centralized MAS with access to the full joint action-observation history and fully decentralized MAS where agents act on local histories without inter-agent communication, and highlights their respective pros and cons. In addition, to enable a clean comparison of MAPL post-training methods, we do NOT introduce an additional SFT stage.Re. W2&Q4: Oracle Reward Reliance
We sincerely appreciate the review can look at our Appendix B.2 about different sampling strategy, including top-k. We first want to clarify that we use oracle rewards to select top-k samples not because other approaches are difficult to construct, but because this provides the most direct and interpretable way to demonstrate the concept.
MAPL does not inherently rely on oracle rewards. It can also use learned rewards, when available, or heuristic metrics. With MARLHF, R~ learns the corresponding causal metrics internally, so oracle rewards are not required. Even when learned rewards are unreliable or unavailable, we can select dominant trajectories (πR∗(ν),∗≻ϖR∗(ν),∗) or advantageous trajectories (πR∗(ν),∗≻ϖR∗(ν),∗) using heuristic metrics (Appendix B.2). This reflects our motivation that optimizing agents does not require a highly accurate reward function or model and can instead be bootstrapped from available metrics or simple heuristics.
We report MAPL results with top-k sampling using reward-free metrics or an AI judge, as suggested by reviewer
mkbZto demonstrate this concept. The results show that MAPL is not very sensitive to the sampling strategy and can learn effectively even with rough metrics.Table 3: Top-k Sampling with Other Metrics
Re. Q3: Credit Assignment of MAPL
We do not claim credit assignment as an advantage of MAPL. On the contrary, Appendix H identifies it as a potential limitation in long-horizon settings and a direction for future work.
The main focus of this paper is to propose and compare preference-based MARL methods under different setting, analyze when they could success and why they can potential fail under some settings. To have a direct, autonoumous, and clean comparison between MAPL methods, we don't involve specific credit assignment techniques to remove compounding factor that prevent a clean explanation and analysis. Empirically, we just scaling the preference dataset size to overcome this issue, where each task are given 20 pairs of preferences per iteration based on the 4 given tasks in our experiments, though larger and more complex and fundamental post-training in industry would need far more data.