acceptodds
Under review as a conference paper at ICLR 2027

Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs

Abstract

We develop a unified treatment of credit assignment for RL training in multi-agent LLM systems. We show that observed reward alone cannot distinguish an agent that determines it from one that never affects it, and that standard shared-reward training performs exact gradient ascent on each agent's private utility rather than system performance. Moreover, we prove no single scalar per agent can consistently account for joint performance once agents interact. We thus develop the unique background-dependent notion of marginal contribution satisfying natural consistency requirements. From it we derive gradient-correct marginal contribution training signals, identify them from filtered feedback, and optimally allocate a budget of exact counterfactual evaluations against learned-signal error. Instantiated in GRPO, our signal improves routed GSM8K accuracy over winner-take-all training at no extra generation cost.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, multi-agent systems, policy gradients, credit assignment, large language models, multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_89xxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes marginal policy gradients under filtered feedback for multi-agent LLMs. This paper discusses an interesting topic, i.e., multi-agent LLM, which may be one of the important directions of LLM tuning study. However, the current form of the paper is very difficult to follow and has many substantive issues that need improvement and clarification. Please see weaknesses and questions.

Critical strengths
  1. This paper discusses an interesting topic.

  2. The methodology of this paper is supported by a rigorous theoretical analysis.

Critical weaknesses
  1. This paper has an unclear objective: why or in which case are multi-agent LLMs more important and preferable to deploy than single LLM tuning or distributed (federated) LLM tuning?

  2. This paper has unclear problem formulation: how the multi-agent is modeled in the setting, how the roles (e.g., planner, solver, critic) are assigned to the agents, how data is distributed to the agents, and how the cumulative LLM capability/ability is produced by the agent cooperation.

  3. This paper has unclear experiment setting, result, and analysis, including what challenge is represented by the GSM8K dataset and why the dataset is relevant for multi-agent LLM.

  4. Given a very few baselines (winner-take-all GRPO and a frozen single agent that is likely a lower bound) and 1 dataset, I believe that the experiment result and analysis are not sufficient.

  5. Methodology: It is unclear how interaction between agents to cooperatively train/tune the LLM is unclear. The introduction mentions role distribution between agents, but Figure 1 and section 3 show agents act in a layered model, which is totally different.

  6. Methodology cont’: The marginal contributions to the policy gradients are unclear.

  7. Methodology cont’: Rather than writing technical components in an oversimplified way added with a very dense theorem, it is better to write a clear storyline about why a technical component is important and clearly explain how it works along with its input and output. Giving an overview of the proposed method and its components, along with a clear motivation, will be helpful for the reader.

  8. This paper has no conclusion summarizing the significant contribution and the finding of the study.

  9. This paper has a very short (insufficient) literature review.

  10. It is advisable to significantly improve the paper both from technical quality and writing before submitting the manuscript for a publication.

Questions for authors
  1. What is the new mechanism for multi-agent policy learning offered by the paper? Could you please give a clear design on learning marginal contributions to the policy learning?

  2. What is the feedback mechanism to improve each agent to deliver a better marginal contribution on the next iteration of the policy learning?

4 comments

Thank you. Several concerns read the paper as being about a different setting, so we first state what it studies, then answer each point.

What the paper studies. Some deployed AI systems use several language models (agents), each with its own parameters, and a mechanism such as a router that picks or combines their answers. Only the final answer is graded. The question is how each agent should learn from that one grade. This is not federated or distributed tuning: no data are split among agents, and every agent sees the same question. Each agent updates its own parameters from its own feedback signal; the system's quality is the average grade of its final answer.

Your concernAnswerWhere
Why multi-agent systems?They are already deployed, because several specialized models can answer better or more cheaply than one. We take them as given and ask how to train them.§1, p. 1
Roles vs layersThe same model: a planner–solver–critic pipeline is a chain of one-agent layers; a pool of agents behind a router is one layer.§1, p. 1; Fig. 1, p. 3
What is the new mechanism? How does each agent improve?No new interaction mechanism; the contribution is the feedback each agent receives. One training step: agents answer, the router picks one answer, it is graded, each agent receives its signal and updates its own parameters.Algorithm 1, p. 9
How do marginal contributions enter the policy gradient?Each agent's signal is the grade minus a baseline that ignores its own answer. This keeps the gradient correct and can reduce its noise.Table 1, p. 8; Lemma 1, p. 6
Dense theorems, no overviewAdded a roadmap, a table of all signals and a two-agent worked example.§1, p. 2; Table 1, p. 8; §3.1, p. 4
Why GSM8K?Answers are graded exactly, so we can grade every agent's answer offline and check each signal; untrained agents solve 51–61% of problems, leaving room to learn.§7, p. 9
Few baselines, one datasetShared reward, winner-take-all and seven variants of our estimator, compared on gradient noise; training comparisons with paired seeds on the full test set; a second dataset (SVAMP) for the noise measurements.§7, p. 9; Tables 3 and 9
No conclusion; short literature reviewBoth added.§8, p. 10; §2, p. 2

Your review noted the paper's rigorous theoretical analysis. If any result seems unsound, please tell us which, and we will address it. We would be grateful if you would reconsider your score.

Thank you again for your review. As the discussion closes on Wednesday, Nov 18 (AoE), we wanted to ask whether our reply and the revised PDF address your concerns, and whether any question remains that we can answer. Thank you.

Reviewer_89xxOpenReview ↗

Official Comment by Reviewer 89xx

Thank you for your response,

While the authors address some of my concerns, several issues, mainly W1, W2, Q1, W2, baseline, and performance evaluation, remain unclear/not convincing to me. Thus, I keep my score.

Thank you for the reply. We answer each point here in full, without pointers into the paper. (We read the second "W2" as W3.)

In one sentence. When several LLM agents produce one answer and only that answer is graded, what training signal should each agent receive?

W1. We take multi-agent systems as given (a router over several models, or a pipeline), built for cost or specialization; we do not claim they beat one model. A single model has no credit-assignment problem; federated tuning trains one model on partitioned data, and every output is graded. Neither covers one grade shared by several models.

W2. A question x goes to all K agents. Agent k, with its own parameters θ_k, samples Y_k ~ π_k(· | x; θ_k). A router deploys Y = Y_J, with J ~ p(· | x, Y_1, …, Y_K). Only Y is graded: G ∈ {0, 1}. The objective is E[G]. Roles enter only through π_k and the router. In our experiments each agent is Llama-3.2-3B-Instruct with its own LoRA adapter and one of three role prompts: direct arithmetic, equation-first, quantity-first (App. G).

Q1. Agent k updates θ_k ← θ_k + η · A_k · ∇ log π_k(Y_k | x). Shared reward, A_k = G, is correct on average but noisy; winner-take-all is biased (Theorem 1); ours, A_k = G − B_k, with B_k the estimated grade without agent k's answer, is correct on average (Lemma 1) and less noisy under the condition of Proposition 4. An agent whose answer lifts the grade above what the others reach without it is reinforced.

W3, baselines, evaluation. GSM8K grades every answer exactly, so each signal's noise can be measured. Against shared reward, on all 1,319 test problems with paired seeds: no accuracy difference with three agents; with nine, ours learns faster (24 of 30 paired comparisons, p = 7.5·10⁻⁵). Training uses one model and one dataset; we state this as a limitation.

If a step is still unclear, please tell us which.

Reviewer_WHrQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies applying RL to multi-agent LLMs under filtered feedback. The setup is that there are K agents, each generating responses. A downstream aggregator picks or combines them into a single deployed output Y and a system reward G is observed. The main contributions are:

  1. Showing that "winner-take-all routing" routing is biased (Theorem 1)
  2. Notion of contribution fields (Sections 4-5): To isolate how much Agent k contributed, the paper suggests the idea of a contribution field, which measures an agent's added value against a fixed subset of teammates.
  3. Policy gradients are unbiased when subtracting the teammate baseline for any set of teammates S
  4. Experiments on GSM8K with 3 routed agents. The new method beats winner-take-all and a frozen baseline.
Critical strengths
  • Theorem 1 is intuitive and useful. Rewarding only the winning model in an ensemble does not make sense and leads to bad results.
Critical weaknesses
  • The primary weakness of this paper is missing an obvious natural baseline that I believe practitioners would use: assigning the final score G to all agents equally. The paper seems to frame the "summing to G" as predeteremined requirement, but I believe it should make clear that all agents receiving the same score as the default baseline.
  • This actually is a special case of the authors proposal, where S = empty set. I believe the paper should be written with this in mind, rather than starting with the (imo slightly unnatural) baseline of giving the credit to the winning agent.
  • The paper frames its theory around general multi-agent mechanisms, but all its experiments apply to single-choice routing. It would be nice to see results with synthesis or aggregation of some sort, which I believe is an important practical tool (eg aggregating subagents).
Questions for authors
  • Would the baseline above lead to high variance such that learning slows? It would be nice to see this comparison
5 comments

Thank you for your precise review. You were right, and the revision is built around your point.

Shared reward is the natural baseline. Agreed. The paper now starts from it: giving every agent the team's reward is correct on average but noisy; winner-take-all is biased (Theorem 1, which you found useful); and our contribution signals are shared reward minus a baseline (your S = ∅ observation), so their only possible benefit is less noise. Training signals no longer need to sum to the reward. Shared reward is the baseline of every comparison (§1, p. 1; §3, p. 3).

Your question: does shared reward's noise slow learning? Yes, when there are many agents.

  • Shared reward mixes every other agent's luck into each agent's signal. With a router that picks at random, our removal estimator has 0.41× shared reward's gradient noise with three agents and 0.09× with nine (Table 18, p. 40).
  • In a pre-registered test with nine agents, three copies of an agent learned side by side, one from each signal. Ours learned faster than shared reward in 24 of 30 paired comparisons (p = 7.5·10⁻⁵) and tolerated twice the learning rate: at that rate 4 of 10 shared-reward runs collapsed and none of ours did (Table 3, p. 9; §7.2, p. 10).
  • With three agents the noise gap is small, and training does not separate the signals (Table 11, p. 35).

Synthesis and aggregation. Not tested in training; our experiments cover selection. We say so (Limitations, p. 10).

We hope this answers your question directly. If so, we would be grateful if you would reconsider your score.

Thank you again for your review. As the discussion closes on Wednesday, Nov 18 (AoE), we wanted to ask whether our reply and the revised PDF address your concerns, and whether any question remains that we can answer. Thank you.

In case it saves time before the discussion closes (end of Nov 18, AoE), our answer in two lines:

  • Your central point is now the paper's framing. Shared reward is the baseline; our signals are shared reward minus a baseline that ignores the agent's own answer (your S = ∅ observation), so their only possible benefit is less noise (§1, §3).
  • Your question: does shared reward's noise slow learning? Yes, with many agents. In a pre-registered nine-agent test, ours learned faster in 24 of 30 paired comparisons (p = 7.5·10⁻⁵) and tolerated twice the learning rate, at which 4 of 10 shared-reward runs collapsed and none of ours did. With three agents the noise gap is small, and training does not separate the signals (§7.2, p. 10).

We would value your view on whether this answers your question.

Reviewer_ZTZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study the credit assignment problem, under the multi-agent setting. In this setting, a mechanism selects or combines agent outputs and only the deployed output can receive a reward, which separates contribution attribution, identification from observed feedback, and optimization, of the system objective.

Theoretical analysis here include two characterizations: policy-gradient bias (winner-take-all) and background-dependent marginal distributions. Also the authors study the conditions which contribution-based signals preserve the expected system gradient. Based on these, the authors develop masked-critic estimation, propensity-corrected removal estimates for routing, and a budget-allocation rule for counterfactual evaluations.

The experiments were conducted on a small GSM8K dataset, with 3 agents. The experiment reported approximately 41% deployed accuracy for removal-signal GRPO versus 37% for winner-take-all GRPO, without additional counterfactual generation. Evaluation uses 128 examples and one reported seed.

Critical strengths

The main strength to me is a precise diagnosis of winner-take-all credit allocation. Theorem 1 identifies the difference between the private-utility gradient and the system gradient through the effect on competitors’ expected returns. This provides a concrete explanation of the objective mismatch and makes the concern mathematically testable.

Also, I found this paper provides a useful separation of attribution, optimization, and identification. In this paper, the authors provide a framework to distinguish the counterfactual being measured from whether its associated signal is suitable for training. Then, the routing estimator further shows how known selection probabilities can recover removal contributions in expectation despite imperfect outcome predictions. This is consequential in settings where only selected outputs receive feedback.

Critical weaknesses

This paper has some issues to improve from the following aspects.

Experimental validation

  • The experiment doesn't include system-objective baseline (§8, Table 1).
    • No shared-reward GRPO is included as the baseline. It is necessary because Lemma 1 establishes the validity of the underlying shared-reward policy gradient.
    • Please include this comparison under matched settings; otherwise, the gain cannot be attributed specifically to marginal-contribution estimation.
  • I also don't find sufficient evidence of robustness
    • In (§8; Appendix E), the approximately four-point accuracy gain is evaluated on 128 examples, with one reported seed, roughly 5 additional correct answers.
    • This is not enough. Please report variation across seeds and uncertainty for the paired accuracy difference.
  • Moreover, estimator components are not isolated, as shown in (§6.1; Appendix D; Table 1).
    • This is necessary to establish whether correcting attribution adds practical value, beyond using a valid learned baseline.
    • Please add comparisons against the direct learned removal baseline without the propensity correction.

Connection between theory and training

  • The variance analysis does not establish a reduction in gradient variance (§5, Proposition 4, Eq. 12).
    • Proposition 4 concerns conditional reward variance, whereas the training update uses the score-weighted residual Ψi(G−Bi)\Psi_i(G-B_i).
    • Lower residual variance can coexist with higher gradient variance. Please clarify this limitation and provide either sufficient conditions or empirical gradient-variance measurements.
  • The theoretical guarantee does not directly cover the experimental GRPO update (§8; Appendix E).
    • Group normalization and clipping transform the contribution signal, so its original expected-gradient identity does not automatically carry over.
    • The manuscript acknowledges this distinction, but the opening claim in §8 about replacing rewards or advantages in standard RLFT updates remains too broad.
    • Please qualify that claim and, if possible, provide a controlled comparison showing whether the implemented update improves estimation of the system gradient.

Scope of the attribution claims

  • The interpretation of the scalar-credit impossibility result needs qualification (§4, Proposition 2).
    • The result requires the same fixed credits to reconstruct every coalition’s value. This is stronger than requiring a useful scalar training signal.
    • Indeed, action-independent weighted averages of retention contributions remain gradient-correct under Corollary 1.
    • Please distinguish the need to represent all coalition values from the requirements of policy optimization.
  • The uniqueness claim depends on choices that remain unspecified by the theorem (§4, Proposition 3; Abstract).
    • Contributions are unique given a valuation and the ordered-totality requirement. The result does not uniquely determine the counterfactual evaluation rule.
    • Please make this qualification explicit, particularly in the abstract, and explain how practitioners should choose between retention and removal.

Learnability and counterfactual evaluation

  • Population identification does not establish practical recovery of the contribution field (§6, Lemma 2).
    • Recovery assumes a sufficiently expressive critic, population-risk minimization, and positive probability for the relevant masks.
    • These assumptions do not establish accurate estimation from finite data across potentially exponentially many backgrounds.
    • Please clarify the scope of the guarantee and, where feasible, report critic or contribution-estimation error.
  • The practical benefit of the evaluation-allocation rule remains unvalidated (§7, Theorem 2, Eq. 22).
    • The optimal probabilities depend on unknown contribution-estimation errors and evaluation variances. Pilot estimation is suggested, but its cost and effect on allocation quality are not evaluated.
    • Please distinguish the oracle optimality result from a practical allocation procedure, and explain how these quantities would be estimated within the evaluation budget.

Writing and organization

  • The relationships among the different training and attribution signals are difficult to track (§§4–8).
    • Exact contributions, factual reward residuals, critic differences, propensity-corrected estimates, and normalized GRPO advantages have different guarantees.
    • Please add a compact table identifying each signal’s purpose, assumptions, and guarantees, with a clear indication of which signal is used experimentally.
  • The abstract formulation would benefit from a running numerical example (§§4–6).
    • Retention averages over omitted agents’ outputs, while removal evaluates a reduced system. Their distinction is central to the subsequent gradient arguments.
    • Please carry a simple two-agent example through both definitions and baseline subtraction to illustrate why accurate attribution and gradient correctness are separate properties.
Questions for authors
  • Is Proposition 4 intended to support a claim about gradient-estimator variance, or only reward-residual variance? If the former, what additional assumptions justify that implication?
  • How does removal-signal GRPO compare with shared-reward GRPO under the same generation budget, optimization settings, and evaluation procedure? Does the improvement persist across training seeds?
  • Can a controlled experiment measure gradient bias or variance before and after the GRPO transformations? Alternatively, does an update directly covered by the theory demonstrate the predicted advantage?
  • What are the paired evaluation outcomes underlying the reported accuracy difference, and what uncertainty can be reported for that difference?
3 comments

Thank you for a careful review; it shaped this revision. Your three main concerns, in brief:

  • No shared-reward baseline; one seed; 128 problems. Shared reward is now the baseline of every comparison, with paired seeds on all 1,319 test problems. The original experiment is withdrawn.
  • Proposition 4 was about reward variance, not gradient variance. It now gives an exact split of gradient variance and the condition you asked for.
  • The theory did not cover the GRPO update. We now test the update the theory covers, in a pre-registered experiment: with nine agents, our removal estimator learns faster than shared reward (24 of 30 paired comparisons, p = 7.5·10⁻⁵).

Your questions

QuestionAnswerWhere
Q1. Does Prop. 4 concern gradient variance?Part (i) concerns reward variance only; the text now says so. New parts (ii)–(iii) concern gradient variance: the best baseline, and an exact split into unavoidable noise + baseline error + estimation noise. A signal is less noisy than shared reward exactly when its baseline error plus estimation noise is smaller.§5, p. 6
Q2. Removal signal vs shared reward under GRPO, same settings, several seeds?Same seeds, budget, optimizer and evaluation. With GRPO's standardization: no difference (0.599 vs 0.601); we withdraw the original gain. Without it, the update the theory covers: no difference with three agents; with nine, ours learns faster.Tables 11–12, pp. 35–36; §7.2, p. 10
Q3. Measure noise or bias before and after GRPO's transformations, or show the advantage in a covered update?Noise, on identical rollouts: the improved estimator (ours without the propensity division, Cor. 5) has 0.73× shared reward's gradient noise without standardization and 1.92× with it, so standardization reverses the order. Bias after standardization: not measured; we claim nothing there. Covered update: the nine-agent test, in which three copies of an agent learn side by side from different signals, so only their noise differs.§7.1, p. 9; §7.2, p. 10
Q4. Paired outcomes and uncertainty?Three agents: 7 paired seeds, paired intervals. Nine agents: paired within each team, exact sign-flip test.Table 11, p. 35; Table 17, p. 39

Your other requests

RequestWhat we didWhere
Compare with the removal baseline without the propensity correctionDone. Your question was whether correcting attribution adds practical value; the answer is no. The correction is not needed for correct gradients, and it is what makes our original estimator noisy (8.4× shared reward). The direct baseline without it has 0.91–1.05× shared reward's noise; dropping it from the baseline, but not from the rest of the estimator, gives the improved estimator (0.73×).Table 9, p. 35; Cor. 5, p. 8
Qualify the opening claim of §8Removed; we state exactly which update the theory covers.Alg. 1, p. 9; Limitations, p. 10
Prop. 2 concerns representing all coalitions, not trainingAgreed and stated.After Prop. 3, p. 5
Prop. 3 is unique only for a given valuation; retention or removal?Agreed; "unique" removed from the abstract; when to use each.p. 5
Lemma 2 is population-levelAgreed and stated; the masked critic is not run.§6, p. 7
Theorem 2 is an oracle; pilot costLabelled oracle; with a pilot whose cost is counted, the rule cuts error to 0.24× uniform allocation's.App. B, p. 14; Table 4, p. 32
A table of signals; a running exampleAdded.Table 1, p. 8; §3.1, p. 4

What we do not claim. A gain under GRPO's standardization; a training gain with three agents; an advantage over winner-take-all in the nine-agent test, whose random router makes winner-take-all unbiased. (With a random router the propensity division is harmless, so that test uses our original estimator with a training-free outcome model.)

Your review credited the paper with a precise diagnosis of winner-take-all (Theorem 1) and a useful separation of attribution, optimization and identification. Both remain. The revision adds when a contribution signal reduces gradient noise, gradient-noise measurements that match it, and a pre-registered training test against the baseline you asked for. We would be grateful if you would reconsider your score.

Thank you again for your review. As the discussion closes on Wednesday, Nov 18 (AoE), we wanted to ask whether our reply and the revised PDF address your concerns, and whether any question remains that we can answer. Thank you.

In case it saves time before the discussion closes (end of Nov 18, AoE), here are your four questions with one-line answers:

  • Q1. Proposition 4 now concerns gradient variance: an exact split of each agent's gradient noise, and the condition under which a signal is less noisy than shared reward (§5, p. 6).
  • Q2. Against shared reward with the same seeds, budget and evaluation: no difference under GRPO's standardization or with three agents; with nine agents, ours learns faster (24 of 30 paired comparisons, p = 7.5·10⁻⁵) (§7.2, p. 10).
  • Q3. On identical rollouts, our improved estimator has 0.73× shared reward's gradient noise without standardization and 1.92× with it; so we test the update the theory covers (§7.1, p. 9).
  • Q4. Paired seeds on all 1,319 test problems, with paired intervals (three agents) and an exact sign-flip test (nine agents) (Tables 11 and 17).

We would value your view on whether these address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision (all reviewers and the AC)

Summary of the revision

We thank the reviewers. The reviews asked for three main things: a comparison with giving every agent the shared reward (ZTZg, WHrQ); a robust experiment in place of the one-seed, 128-problem result (ZTZg, 89xx); and a clearer paper that connects theory to training (all). The revised PDF does all three. Changes are in blue.

  1. Shared reward is now the baseline. Giving every agent the team's reward is correct on average but noisy. Our contribution signals subtract a baseline that ignores the agent's own answer, so they are also correct on average; their only possible benefit is less noise. The paper is now framed this way (§1, §3). A wrong sentence of the submitted abstract, which described winner-take-all as if it were shared reward, is removed.
  2. The theory now concerns gradient noise. Proposition 4 gives an exact split of an agent's gradient noise into three parts, and the condition under which a signal is less noisy than shared reward (§5, p. 6). Two corollaries are new (p. 8); the statements of all existing results are unchanged apart from notation.
  3. New experiments, all against shared reward (§7). Accuracy is measured on all 1,319 GSM8K test problems, with paired seeds. The original experiment is withdrawn.
    • Noise. The submitted estimator is 8.4× as noisy as shared reward. The theory finds the cause (a division by the router's probabilities); removing it gives an estimator 27% less noisy than shared reward (§7.1, p. 9).
    • Training, nine agents, pre-registered. Three copies of an agent learn side by side, one from each signal, under a router that picks at random, so only the noise of their signals differs. Our removal estimator learns faster than shared reward (24 of 30 paired comparisons, p = 7.5·10⁻⁵) and tolerates twice the learning rate, at which 4 of 10 shared-reward runs collapsed and none of ours did (§7.2, p. 10; App. J).
    • Training, three agents. The noise gap is small, and training does not separate the signals. We report this.

Each reply below answers every point of that review, with its location. Code and the pre-registration files: https://anonymous.4open.science/r/mcpg-12632

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.