acceptodds
Under review as a conference paper at ICLR 2027

When Does Local Credit Assignment Improve Shared Policy Gradients in Multi-Agent Reinforcement Learning?

Abstract

Credit assignment in multi-agent reinforcement learning is justified agent by agent: a counterfactual or teammate-marginalized critic can reduce the variance of a policy-gradient term. With parameter sharing, the optimizer updates the sum of agent-time contributions, so local variance reduction need not improve the shared gradient. This gap is fundamental. For every team size, we construct bounded games with an exact critic and a nonzero shared gradient in which every locally variance-optimal correction reduces local variance but increases shared-gradient variance by an arbitrarily large factor. For two agents sharing a Bernoulli policy with the value baseline, we identify a sharp regime in which exact teammate marginalization is guaranteed not to increase shared-gradient covariance, and extend it to an exact criterion for identical agents. These results motivate PLACID, which separates admission - whether and at what strength a correction improves the shared-gradient estimate - from realization - how to compute it with less Monte Carlo noise. On 16 fixed tabular games, empirical admission removes up to 9.21% of sparse population mean-squared error. On the two fitted critic families, it exceeds unit marginalization at every tested batch budget. Its certified variant selects 30,101 corrections with none increasing population shared-gradient variance. In 1,566 enumerated games, even population-moment local screens can select corrections that harm the shared gradient. In shared-actor training, a lagged empirical admission variant increases normalized return AUC on coverage games by over the sparse update and exceeds unit marginalization on team and coverage games with one teammate proposal.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, policy gradients, credit assignment, parameter sharing, variance reduction, control variates, gradient estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tUKpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows local credit assignment that reduces local gradient variance can result in the increment of the joint policy gradient. To address this problem, PLACID is proposed to select the corrections via the change of the joint variance.

Critical strengths
  • The counterexample is intuitive and informative to characterize the failure of variance guarantee.

  • PLACID is well-motivated and supported by a lot of experiments.

Critical weaknesses
  • The relationship between the variance and the performance is my main concern. The theory does not provide value improvement guarantee. I know it might be hard, but more explanation and experiments on non-toy games would strengthen the claim of the paper.
Questions for authors
  • Can you discuss the relationship between PLACID and other variance reduction tricks like advantage normalization and gradient clipping in PPO? Would PLACID's benefits persist when these tricks are combined? In other word, if the goal is to control the gradient, why should I use PLACID instead of all these standard tricks in practice?
Reviewer_j4NfOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors explore the credit-assignment problem in cooperative MARL settings and whether COMA-style per-agent credit-assignment improves the gradient of an actor with shared parameters. The update of this gradient sums every agent and time-periods terms so that cross-agent covariances matter. They formally prove theoretically that for any team-size there are bounded games with an exact critic in which locally optimal (minimal variance) correct credit-assignment makes the shared-gradient variance arbitrarily worse. They study a toy example environment with two-actions. Here they can find when per-action agent behavior is optimal for an arbitrary number of agents. It takes the form of a threshold rule. They propose a new learning rule PLACID which decides when/where to apply local credit assignment by attempting to estimate the effect on the shared gradient via empirical bernstein bounds. On tabular games with known true-gradients PLACID reduces about 9% of the uncorrected estimator's error and doesn't increase variance in any of the 30,001 pruned games.

Critical strengths
  • The theory is pretty interesting in terms of how and when you should do local corrections to reduce shared gradient error.
  • The theoretical results derived are very nice for the simplified setting.
  • The method introduced is pretty interesting as is the use of the empirical Bernstein lower bounds to avoid admitting harmful corrections.
Critical weaknesses
  • Paper is very challenging to read to the point that I had to have a LLM read over and "decompress" the abstract and resulting analysis. While heterogeneous (some parts are better/worse than others), in general it introduces a ton of novel vocabulary for standard concepts and it is not at all clear what the authors were doing. This is unfortunate as the actual underlying theory and experiments are interesting.
  • I am not clear how much of PLACID is effectively recaptiulating Geffner and Domke 2018 and 2020 where optimal coefficients for vector control variables are computed by minimizing the gradient variance. This literature is exceptionally well established and 'standard'. It is also not clear how this compares to other papers like Lapeyre and Lelong and Kim and Henderson 2007 (Adaptive Control Variates for Finite-Horizon Simulation) and more generally the extensive literature on Rao-Blackwellisation for monte-carlo estimation. (See as one reference: Glynn and Szechtman 2002, https://web.stanford.edu/~glynn/papers/2002/GSzechtman02.pdf) In particular, via PLACID are the authors just implementing a simpler/less accurate form of adaptive control variables? If not, what is the primary difference? Similarly, how does this differ other than in minor implementation choices from Greensmith, Bartlett and Baxter 2004: "Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning".
  • Many other works that should probably be cited are missing. Greensmith, Bartlett and Baxter 2004, Geffner and Domke 2018, especially Lemma 4.1 computing the optimal coefficient for a vector control variable (or some other textbook standard reference), Liu et al 2018's Stein control variables, and Grthwohl et al 2018 (RELAX). GradPS is not cited.
  • It is unclear how much the simplified binomial setting can be extended and so how much their theoretical analysis is relevant for the actual results they get. The simplified setting is really simplified.

I think as it stands the paper isn't ready to be published. The results are fundamentally interesting (on that basis I'd give it probably a 3/4 if the innovation is cleared up satisfactorally and it is not just "apply optimal control variance policies in standard setting and it reduces variance", which we already know it theoretically does and is commonly used for that purpose.) But the writing as it stands is not readable. I had to have an AI go over the abstract and several sections and decompress the text. I suspect much of the text may have been compressed in its meaning and intermediate terms dropped. I think realistically the paper needs to be rewritten. For the writing/readability I'd give it a 1.

Between the two, I'd give it a 1, possibly a 2 if the contribution can be clarified. I think if rewritten, then it could well be a useful contribution if the innovation is made clear.

Questions for authors
  • How does this work compare against the standard optimal control literature and rao-blackwellization and other standard variance reduction methods for monte carlo? It seems to just be reapplying them?
  • How does it compare against Kuba et al 2021 and Greensmith et al 2004 in RL/policy gradient settings?
  • What is the primary contribution would you say.

I think you should probably extend the setting/theory to more general settings if possible.

Reviewer_Xs5EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether local credit assignment improves policy-gradient estimation when agents share parameters. It shows that locally variance-reducing corrections can increase shared-gradient variance and derives exact safety conditions for teammate marginalization in shared-Bernoulli settings. The proposed method, PLACID, evaluates corrections using complete shared-gradient statistics, retains screening data as uncorrected gradients, and uses dependency-preserving stratification to reduce teammate-sampling noise. Experiments examine gradient-estimation accuracy and shared-actor learning in controlled games, alongside sampling diagnostics in SMAC and MA-MuJoCo.

Critical strengths
  1. I find the central question interesting: does reducing each agent’s gradient variance actually improve the shared policy update? The paper makes clear why these are not the same thing under parameter sharing, and why looking only at individual agents can miss an important part of the problem.

  2. The theory is a strong part of the paper. The counterexamples show that the problem persists even with bounded rewards, exact critics, and a nonzero shared gradient. I also appreciate that the authors go beyond showing failures and identify sharp conditions under which local improvements do carry over to the shared gradient.

  3. PLACID is well connected to the theoretical findings. It evaluates corrections at the shared-gradient level and separates selecting a correction from applying it to new samples. Retaining the screening data as ordinary gradients is a useful design choice, especially since rejecting a correction recovers the original sparse estimator without discarding those samples.

  4. The experiments help clarify what each part of the method contributes. They distinguish complete-gradient estimation error, teammate-sampling noise, and learning performance, with exact population gradients available in the controlled games. The appendix also reports inconclusive results rather than only positive ones, which makes the scope of the evidence easier to assess.

Critical weaknesses
  1. The practical benefit on standard MARL tasks is still unclear. The paper carefully distinguishes sampling diagnostics from learning results. However, the SMAC and MA-MuJoCo studies do not yet establish end-to-end learning improvements: the larger reductions concern conditional proposal noise, while the complete-gradient results in Appendix L are inconclusive. The learning evidence therefore remains concentrated on controlled games.

  2. The certified version appears difficult to use effectively at smaller batch sizes. PLACID-C admits no corrections at the smaller tested budgets and abstains on all batches in the original SMAC study. The Shared-Relay experiment shows that certification can work with a changing actor, but its return improvement is negligible. Although the paper clearly distinguishes PLACID-C from PLACID-E and reports harmful empirical selections, more practical guidance on choosing between them in small-batch neural training would be useful.

  3. The additional learning benefit over simpler correction rules is mixed. The fixed-policy estimation results provide useful evidence for complete-block screening. In the learning experiments, however, the return differences from unit correction are inconclusive under exact marginalization. The finite-proposal results are more encouraging against unit correction, but do not show a consistent advantage over local admission. Further discussion would help clarify when the estimation gains are expected to translate into better learning performance.

  4. The relationship between the reversal diagnostics and the learning gains could be clearer. In the exact-marginal experiments, dispersion games show the largest gains over sparse despite rare reversals, whereas tabular games show more frequent reversals but no clear return gain. Since these diagnostics are measured along sparse-training trajectories rather than each method’s own trajectory, they do not directly explain the performance differences. Discussing this distinction would help connect the theoretical motivation to the learning results.

Questions for authors
  1. Beyond allowing negative coefficients, how sensitive are the results to the range and spacing of the coefficient grid?
  2. How does the choice of baseline affect the frequency and magnitude of local-to-shared reversals during training?
  3. Does lower raw-gradient covariance also lead to lower variability in the actual Adam parameter updates?
  4. How does prospective recycling compare with reciprocal cross-fitting under a matched computational budget?
  5. How does the cost of computing complete shared-gradient statistics scale with actor size and episode length?
  6. How sensitive is performance to the allocation of samples between critic fitting, screening, and correction, especially at smaller batch sizes?
  7. How does PLACID perform when integrated into standard PPO or MAPPO training, particularly with multiple optimization epochs per rollout batch?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.