- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes marginal policy gradients under filtered feedback for multi-agent LLMs. This paper discusses an interesting topic, i.e., multi-agent LLM, which may be one of the important directions of LLM tuning study. However, the current form of the paper is very difficult to follow and has many substantive issues that need improvement and clarification. Please see weaknesses and questions.
Critical strengths
-
This paper discusses an interesting topic.
-
The methodology of this paper is supported by a rigorous theoretical analysis.
Critical weaknesses
-
This paper has an unclear objective: why or in which case are multi-agent LLMs more important and preferable to deploy than single LLM tuning or distributed (federated) LLM tuning?
-
This paper has unclear problem formulation: how the multi-agent is modeled in the setting, how the roles (e.g., planner, solver, critic) are assigned to the agents, how data is distributed to the agents, and how the cumulative LLM capability/ability is produced by the agent cooperation.
-
This paper has unclear experiment setting, result, and analysis, including what challenge is represented by the GSM8K dataset and why the dataset is relevant for multi-agent LLM.
-
Given a very few baselines (winner-take-all GRPO and a frozen single agent that is likely a lower bound) and 1 dataset, I believe that the experiment result and analysis are not sufficient.
-
Methodology: It is unclear how interaction between agents to cooperatively train/tune the LLM is unclear. The introduction mentions role distribution between agents, but Figure 1 and section 3 show agents act in a layered model, which is totally different.
-
Methodology cont’: The marginal contributions to the policy gradients are unclear.
-
Methodology cont’: Rather than writing technical components in an oversimplified way added with a very dense theorem, it is better to write a clear storyline about why a technical component is important and clearly explain how it works along with its input and output. Giving an overview of the proposed method and its components, along with a clear motivation, will be helpful for the reader.
-
This paper has no conclusion summarizing the significant contribution and the finding of the study.
-
This paper has a very short (insufficient) literature review.
-
It is advisable to significantly improve the paper both from technical quality and writing before submitting the manuscript for a publication.
Questions for authors
-
What is the new mechanism for multi-agent policy learning offered by the paper? Could you please give a clear design on learning marginal contributions to the policy learning?
-
What is the feedback mechanism to improve each agent to deliver a better marginal contribution on the next iteration of the policy learning?
4 comments
Follow-up
Thank you again for your review. As the discussion closes on Wednesday, Nov 18 (AoE), we wanted to ask whether our reply and the revised PDF address your concerns, and whether any question remains that we can answer. Thank you.
Official Comment by Reviewer 89xx
Thank you for your response,
While the authors address some of my concerns, several issues, mainly W1, W2, Q1, W2, baseline, and performance evaluation, remain unclear/not convincing to me. Thus, I keep my score.
Thank you for the reply. We answer each point here in full, without pointers into the paper. (We read the second "W2" as W3.)
In one sentence. When several LLM agents produce one answer and only that answer is graded, what training signal should each agent receive?
W1. We take multi-agent systems as given (a router over several models, or a pipeline), built for cost or specialization; we do not claim they beat one model. A single model has no credit-assignment problem; federated tuning trains one model on partitioned data, and every output is graded. Neither covers one grade shared by several models.
W2. A question x goes to all K agents. Agent k, with its own parameters θ_k, samples Y_k ~ π_k(· | x; θ_k). A router deploys Y = Y_J, with J ~ p(· | x, Y_1, …, Y_K). Only Y is graded: G ∈ {0, 1}. The objective is E[G]. Roles enter only through π_k and the router. In our experiments each agent is Llama-3.2-3B-Instruct with its own LoRA adapter and one of three role prompts: direct arithmetic, equation-first, quantity-first (App. G).
Q1. Agent k updates θ_k ← θ_k + η · A_k · ∇ log π_k(Y_k | x). Shared reward, A_k = G, is correct on average but noisy; winner-take-all is biased (Theorem 1); ours, A_k = G − B_k, with B_k the estimated grade without agent k's answer, is correct on average (Lemma 1) and less noisy under the condition of Proposition 4. An agent whose answer lifts the grade above what the others reach without it is reinforced.
W3, baselines, evaluation. GSM8K grades every answer exactly, so each signal's noise can be measured. Against shared reward, on all 1,319 test problems with paired seeds: no accuracy difference with three agents; with nine, ours learns faster (24 of 30 paired comparisons, p = 7.5·10⁻⁵). Training uses one model and one dataset; we state this as a limitation.
If a step is still unclear, please tell us which.
Thank you. Several concerns read the paper as being about a different setting, so we first state what it studies, then answer each point.
What the paper studies. Some deployed AI systems use several language models (agents), each with its own parameters, and a mechanism such as a router that picks or combines their answers. Only the final answer is graded. The question is how each agent should learn from that one grade. This is not federated or distributed tuning: no data are split among agents, and every agent sees the same question. Each agent updates its own parameters from its own feedback signal; the system's quality is the average grade of its final answer.
Your review noted the paper's rigorous theoretical analysis. If any result seems unsound, please tell us which, and we will address it. We would be grateful if you would reconsider your score.