acceptodds
Under review as a conference paper at ICLR 2027

Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning

Abstract

Scaling cooperative multi-agent reinforcement learning (MARL) is fundamentally limited by cross-agent noise. When agents share a common reward, each agent's learning signal is computed from a shared return that depends on all agents, so the stochasticity of the other agents enters the signal as cross-agent noise that grows with . Many engineering systems, such as cloud computing and power systems, have differentiable analytical models that prescribe efficient system states, providing a new reference beyond noisy shared returns. In this work, we propose Descent-Guided Policy Gradient (DG-PG), a framework that augments policy-gradient updates with a low-noise descent signal toward this reference. We prove that DG-PG reduces policy-gradient estimator variance from to , preserves the stationary points of the cooperative objective, and achieves agent-independent sample complexity . We evaluate DG-PG on cloud resource scheduling and power dispatch tasks, which involve discrete and continuous decisions, respectively. In both tasks, DG-PG converges within 10 episodes on average at every scale up to 1500 heterogeneous agents.

Primary area:
reinforcement learning
Keywords:
cooperative multi-agent reinforcement learning, large-scale multi-agent systems, policy gradient, variance reduction, engineering systems

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BzpoOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Critical strengths

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Critical weaknesses

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Questions for authors

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Reviewer_suPZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new method for multi-agent RL (MARL) modifying the standard policy gradient objective to take into account distance from a reference state. They provide theoretical proofs and motivation for their new method, claiming that their approach preserves the stationary points of the original objective, reduces the variance of the per-agent gradient estimator to O(1) from O(N), and has a sample complexity of O(1/\eps^3). The authors evaluate the method empirically on three diverse environments (a one-step congestion game, cloud scheduling, and power dispatch) with standard MARL baselines and task-specific heuristics. Their empirical results show an impressive speed-up in convergence compared to baselines.

Critical strengths

The presented method has a clear and large empirical performance benefit over varied baselines, with variance estimates provided. It is clear that the method is effective in the evaluated settings so long as an appropriate reference state can be identified.

The authors motivate their work well by providing proof that the cross-agent empirical noise is a major issue in Section 6.2, and by taking steps to validate their assumptions 4.2 and 4.3 about the reference state actually hold in practical applications.

Critical weaknesses

I believe there is an issue with the derivation of the guidance term in Equation 10. Equation 9 computes the action derivative of d(xt+1,x~t+1)d(x_{t+1},\tilde{x}_{t+1}) . Please explain why multiplying this derivative by the policy score yields $ \nabla_{\theta_i}G $ for the stated visitation-based objective. This equality does not follow directly from the standard score-function identity. This is important as subsequent theoretical results build on Equation 10.

I currently lean toward Weak Reject because the justification of Equation 10 remains unresolved, and subsequent theoretical guarantees rely on this identity. I find the empirical results promising and would reconsider my recommendation if the authors provide a convincing derivation under assumptions applicable to the claimed setting, or a correction that adequately addresses the implications for the subsequent results.

Questions for authors
  • Can you show the derivation of equation 10 more clearly and give more details on how it follows from the policy gradient formulation and equation 9?
  • Can you clarify what the dashed line is in Figure 2 and whether it is the best fit baseline?
4 comments
AuthorsOpenReview ↗

Response to Reviewer suPZ (1/2)

We sincerely thank the reviewer for the careful reading, and for recognizing the clear empirical benefit of DG-PG and our validation of Assumptions 4.2 and 4.3. Beyond cloud scheduling and power dispatch, we have also applied DG-PG to voltage control in distribution networks and fleet rebalancing in robotaxi operations, where references built from the analytical models that operators already use likewise make DG-PG effective.

The reviewer's main concern is the derivation of Equation 10. Below we give the full derivation from the policy-gradient formulation and Equation 9, and state the condition under which it holds. We also answer the question on Figure 2.

1. Concern on whether Equation 10 follows from the policy-gradient formulation and Equation 9.

We thank the reviewer for this careful reading. Equation 10 holds for the guidance direction that DG-PG uses, and we give the full derivation below.

Equations 9 and 10 read

\qquad \nabla_{\theta^i}\mathcal{G} = \mathbb{E}_{s_t \sim \nu^{\boldsymbol{\pi}},\\, \boldsymbol{a}_t \sim \boldsymbol{\pi}}\\!\left[\langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle\\,\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i)\right].$$ The coefficient $\langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle$ differs from $d$ only by terms that do not depend on agent $i$'s action. Fix $s_t$ and the other agents' actions, and let $\boldsymbol{x}_{t+1}^{-i} \triangleq \boldsymbol{x}_{t+1}-\boldsymbol{z}_t^i$ be the next system state without agent $i$'s contribution. Then $$\begin{aligned} d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) - d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1}) &= \frac{1}{2}\\|\boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}\\|^2 - \frac{1}{2}\\|\boldsymbol{x}_{t+1}-\boldsymbol{z}_t^i-\tilde{\boldsymbol{x}}_{t+1}\\|^2 \\\\ &= \langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle - \frac{1}{2}\\|\boldsymbol{z}_t^i\\|^2 . \end{aligned}$$ Since agents sample their actions independently given $s_t$, the term $d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1})$ does not depend on $a_t^i$, and neither does $\\|\boldsymbol{z}_t^i\\|^2$ when agent $i$'s contribution has the same magnitude under every action. The score-function identity then gives $$\begin{aligned} \nabla_{\theta^i}\\,\mathbb{E}\\!\left[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) \mid s_t\right] &= \mathbb{E}\\!\left[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1})\\,\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i) \mid s_t\right] \\\\ &= \mathbb{E}\\!\left[\Big(d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1}) + \langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle - \tfrac{1}{2}\\|\boldsymbol{z}_t^i\\|^2\Big)\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i) \mid s_t\right] \\\\ &= \mathbb{E}\\!\left[\langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle\\,\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i) \mid s_t\right], \end{aligned}$$ where the last equality holds since $d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1})$ and $\tfrac{1}{2}\\|\boldsymbol{z}_t^i\\|^2$ do not depend on $a_t^i$, and thus have zero expectation when multiplied by $\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i)$. By definition, $\mathcal{G} = \mathbb{E}_{s_t \sim \nu^{\boldsymbol{\pi}}}\big[\mathbb{E}[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) \mid s_t]\big]$. With $\nu^{\boldsymbol{\pi}}$ held fixed, its gradient is the average of the per-step gradient above, $$\nabla_{\theta^i}\mathcal{G} = \mathbb{E}_{s_t \sim \nu^{\boldsymbol{\pi}}}\big[\nabla_{\theta^i}\mathbb{E}[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) \mid s_t]\big],$$ which is Equation 10, the direction for which Theorems 5.1–5.3 are stated. In the original submission, Equation 10 and the DG-PG update already compute this per-step direction. To make this explicit, we have added the derivation and its conditions to Section 4 and the appendix.
AuthorsOpenReview ↗

Response to Reviewer suPZ (2/2)

2. Question on whether the dashed line in Figure 2 is the Best-Fit baseline.

We thank the reviewer and apologize that the figure omitted this. The dashed line is the Best-Fit baseline, and we have added it to the legend of Figure 2.


We sincerely thank the reviewer again for the careful reading and the constructive comments. We hope that the full derivation of Equation 10 resolves the concern on which the subsequent theoretical results build, and we would be grateful if the reviewer would reconsider the evaluation in light of it. We would be glad to clarify any further questions during the discussion period.

Thank you for the detailed response and revisions. The additional derivation in Appendix D.2 and the clarification that Equation 10 describes a per-step direction with the visitation distribution held fixed address an important part of my concern. I also appreciate the revised stationary-point claim and the explicit alignment assumption used in the subsequent proofs.

While the applicability of these additional assumptions remains a limitation of the theoretical results, I find the empirical performance benefits compelling. On balance, I am updating my recommendation to Weak Accept. Thank you also for clarifying the Best-Fit baseline in Figure 2.

We sincerely thank the reviewer for raising the recommendation. Assumption G.2 holds exactly, with c=1c = 1, in the congestion game of Appendix H, and in experiments we find DG-PG does not require it to hold strictly.

Compared with the exactness of this alignment, cross-agent noise, whose variance grows with the NN, is the far greater obstacle to learning in large systems, and it is the problem that DG-PG solves. Thanks again!

Reviewer_si7ZOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets cooperative MARL in engineered systems, where all agents share one reward. In this setting the "cross-agent noise" in each agent's policy gradient grows with the number of agents N. DG-PG uses an analytical reference state x̃, for example capacity-proportional server loads. It adds a guidance term −α⟨x_{t+1} − x̃_{t+1}, z_t^i⟩ to each agent's advantage, where z is the agent's local influence on the system state, and it down-weights the MAPPO advantage by (1−α).

The paper claims three guarantees:

  • the stationary points of the original objective are preserved for any α ∈ (0,1) (Thm 5.1);
  • the per-agent gradient variance drops from O(N) to O(1) (Thm 5.2);
  • the sample complexity is O(1/ε³), independent of N (Thm 5.3).

Experiments cover three environments: a one-step congestion game with exact gradients, cloud scheduling and power dispatch with up to 1500 agents. DG-PG converges within about 10 episodes at every scale. In cloud scheduling it beats MAPPO/IPPO at N ≤ 10 and Best-Fit at N ≥ 200. In power dispatch it beats AGC and centralized greedy at all scales.

Critical strengths
  1. A cheap, practical recipe. It is one extra term in the advantage, and it turns domain knowledge into fast, stable training at large N. Training converges in under 10 episodes at every scale up to 1500 agents, with clear gains over AGC in power dispatch (e.g., −13.83 vs −18.49 at N=1500). The appendices on environments, hyperparameters and failed baselines are detailed. For practitioners in scheduling and dispatch, this is a useful data point.
Critical weaknesses
  1. The theoretical guarantees, which are the paper's main claims, do not hold as stated.

    Thm 5.1 (stationary points). The guidance term is a form of reward shaping that is not potential-based, so it can move both the stationary points and the optimum. Both happen in the paper's own congestion game (App. H), where the paper states that Assumptions 4.2–4.3 hold. Take N=3 agents with softmax policies:

    • A stationary point of J is lost. Let the capacities be c = (0.6, 1, 1.4), and let every agent play p = (0.1, 1/3, 17/30). Then ∂J/∂p_n = −7/3 for every n, so the policy gradient of J is exactly zero. But ∂G/∂p = (0.10, 0.17, 0.23) is not constant, so the gradient of J_α is not zero.
    • The optimum moves. Let c = (0.5, 0.5, 1.45). The best assignment for the true reward puts one agent on each resource, loads (1,1,1), with cost 4.690. For any α above about 0.30, the guided objective prefers loads (1,0,2), with cost 4.759. The paper uses α = 0.9 for N ≥ 100.

    The proof breaks at Step 3 of Lemma E.2: a non-zero state-space gradient does not have to give a non-zero parameter gradient. The full-row-rank condition of Remark E.1 can never hold when total load is conserved, because the rows of ∇θ x then always sum to zero.

    Thm 5.2 and 5.3 (variance and sample complexity).

    • For any fixed α < 1, the estimator still contains (1−α)·ĝ_J. With ρ < 0 as the paper assumes, its variance is therefore at least (1−α)²σ_J², which is O(N).
    • With the paper's own numbers (σ_J² = 5.7×10³ at N=1500, α = 0.9), that term alone is about 57, against σ_G² ≈ 2.
    • The O(1) minimum is reached only as α → 1, i.e. by dropping the true policy gradient.
    • Thm 5.3 plugs this O(1) variance into a bound for a fixed α. But the bound in App. G contains σ²_DG(α)/(1−α)², which is at least σ_J² for ρ ≤ 0. By its own expression, the sample complexity is therefore never better than plain policy gradient and still grows with N.
  2. The experiments don't isolate the claimed mechanism, and the baselines don't get the same information.

    • Few learning baselines. MAPPO/IPPO are run only at N ≤ 10, only in cloud scheduling, and only with linear actor/critic models. The neural versions failed even at N = 2 and 5 (App. L, about 20 runs). Yet Section 6.4 says they "fail to converge at these scales (Appendix L)" for large N, and no such runs are shown. Power dispatch has no learning baseline at all.
    • Unequal information. DG-PG gets an analytical reference and model that no baseline sees. Load balancing is even removed from the reward and enters only through the guidance (I.3). No baseline receives the same knowledge as a shaped reward or a difference reward. A single potential-based shaping run for MAPPO at N=5 (App. L) is the only attempt.
    • Small gains over Best-Fit. The margins are 0.07–0.25 at N ≥ 200 (e.g., −10.41 vs −10.66 at N=1500), and DG-PG is worse than Best-Fit for N ≤ 100 (e.g., −13.19 vs −12.15 at N=10).
    • Checkpoint selection on test seeds. The reported checkpoint is picked on the same 10 test seeds that are reported (K.1).
Questions for authors
  1. How do Thm 5.1 and Lemma E.2 hold in the congestion game of App. H, given the two examples above? How can the full-row-rank condition of Remark E.1 be met when total load is conserved?

  2. For which α is the bound in App. G independent of N, and how large is the bias with respect to J at that α? An α = 1 (guidance-only) ablation would help here.

  3. Please add equal-information baselines:

    • MAPPO/IPPO with the same guidance used as a shaped reward or a difference reward;
    • MAPPO/IPPO at N ≥ 50 and in power dispatch.

    Please also select checkpoints on validation seeds that are separate from the test seeds.

  4. For cloud scheduling at N=10, DG-PG scores −9.69 ± 0.23 in Section 6.3 but −13.19 in Table 1. Is this the same environment? If so, why does the linear model do much better than both the neural model and Best-Fit (−12.15)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer si7Z (1/3)

We thank the reviewer for the comments. As the reviewer notes, CMARL scales poorly in engineered systems. DG-PG integrates the analytical models of these systems into the policy gradient and gives each agent a learning signal whose variance does not grow with the number of agents. As a result, DG-PG converges within 10 episodes on average, even with 1,500 agents. Its decentralized decisions match Best-Fit, a centralized heuristic with global information, and surpass it at large scales, and they also outperform AGC, the conventional distributed controller, whereas conventional CMARL methods such as MAPPO already fail at small scales. The guidance enters learning through the gradient rather than through the reward, and the ablation in the original submission shows that the same information does not help MAPPO as a shaped reward. DG-PG is therefore a new learning algorithm, not a variant of reward shaping.

Large-scale engineered systems are among the most important settings for deploying CMARL in practice, and DG-PG is designed for these systems, not as a general method for every CMARL problem. The paper illustrates its theory and learning paradigm with two typical settings, cloud scheduling and power dispatch. Our experiments confirm the central claim that the variance and the sample complexity do not grow with the number of agents, since learning does not slow down even at 1,500 agents. All methods share the same seeds and evaluation settings, and the baselines even receive more training iterations.

Learning in large heterogeneous systems is a recognized bottleneck of CMARL, and DG-PG makes it possible. As the original submission explains, suitable learning baselines are hard to find at these scales, so we compare DG-PG mainly with classical and centralized controllers, and Appendix L shows that conventional CMARL algorithms fail. We therefore believe that the concerns about few baselines and small gains over centralized methods do not hold, since the original submission already explains both points, why MAPPO and IPPO are not reported at large scales (Section 6.4, Appendix L) and why Best-Fit, which uses global information, serves as an upper reference (Sections 6.3 and 6.4).

Finally, the reviewer's objections to the theory mainly arise from misreading Theorem 5.2 as a bound at a fixed α\alpha and the guidance as reward shaping. The variance objection evaluates the variance at a fixed α\alpha, whereas Theorem 5.2 states its minimum over α\alpha, which is O(1)O(1), and so is the sample complexity at that α\alpha. The counterexamples measure the deviation from the reference by an unweighted distance, which does not follow the cost, whereas under the deviation given by the cost the guidance equals −∇J-\nabla J in the congestion game of Appendix H, so neither counterexample applies. We respond to each concern on the theory and the experiments in detail below.

1. Concern that the guidance is reward shaping that moves the stationary points and the optimum, and question on the full-row-rank condition of Remark E.1 (W1, Q1).

We would first like to clarify that the guidance term is not reward shaping, because the reference from the analytical model enters learning through the gradient rather than through the reward. Second, in this congestion game, the guidance follows the gradient of JJ exactly, so neither example applies to DG-PG. Since x~\tilde{x} is proportional to cc and the total load is conserved, the cost satisfies C(x)=C(x~)+∑n(xn−x~n)2/cnC(x) = C(\tilde{x}) + \sum_n (x_n - \tilde{x}_n)^2 / c_n for every load xx. The deviation from the reference measured by this cost, which is twice dd applied to the scaled state xn/cnx_n/\sqrt{c_n}, is therefore G=−J−C(x~)\mathcal{G} = -J - C(\tilde{x}), and Jα=J+α,C(x~)J_\alpha = J + \alpha\\, C(\tilde{x}) has the same stationary points and optimum as JJ for every α\alpha. In the first example, ∇G=−∇J=0\nabla \mathcal{G} = -\nabla J = 0 at the chosen point, and in the second, (1,1,1)(1,1,1) remains the optimum of JαJ_\alpha. Both examples rely on the unweighted distance ∑n(xn−x~n)2\sum_n (x_n - \tilde{x}_n)^2, which Appendix H used to measure gradient variance and which does not follow the cost.

Third, Theorem 5.1 guarantees that every stationary point of DG-PG is a stationary point of JJ, so the guidance never stops learning where JJ can still improve, and this guarantee follows directly from the parameter-space alignment of Assumption G.2. If the DG-PG gradient vanishes, taking its inner product with ∇J\nabla J gives 0≥β∣∇J∣20 \ge \beta \\|\nabla J\\|^2, so ∇J=0\nabla J = 0. We have written Theorem 5.1 in this form, so its proof uses neither Lemma E.2 nor the full-row-rank condition of Remark E.1, which we have removed. In the congestion game, with the deviation measured by the cost, Assumption G.2 holds exactly with c=1c = 1, since ∇G=−∇J\nabla \mathcal{G} = -\nabla J.

AuthorsOpenReview ↗

Response to Reviewer si7Z (2/3)

2. Concern that the variance and sample complexity grow with NN at a fixed α\alpha, and question on the α\alpha at which the bound is independent of NN and the bias at that α\alpha (W1, Q2).

We would first like to clarify that Theorem 5.2 states the minimum variance over α\alpha, not the variance at a fixed α\alpha, and the bound in Appendix G is independent of NN at the α\alpha of Theorem 5.2. Section 5.2 explains this choice of α\alpha. In Theorem 5.3, a fixed guidance weight means that α\alpha is held constant during training, and the condition σDG2=O(1)\sigma_{\mathrm{DG}}^2 = O(1) from Theorem 5.2 sets it to the α\alpha of Theorem 5.2. We have made this choice explicit in the statement of Theorem 5.3.

Second, this α\alpha does not drop the policy gradient, and the guidance does not shift the stationary points of JJ. DG-PG retains (1−α)g^J(1-\alpha)\hat{g}_J, and at this α\alpha, as at every α∈(0,1)\alpha \in (0,1), every stationary point of DG-PG is a stationary point of JJ (Theorem 5.1).

Third, the bound in Appendix G does not contain σDG2(α)/(1−α)2\sigma_{\mathrm{DG}}^2(\alpha)/(1-\alpha)^2, so the sample complexity is not tied to that of plain policy gradient and does not grow with NN. The proof of Theorem 5.3 uses the alignment between ∇J\nabla J and ∇G\nabla \mathcal{G}, which we have made explicit as Assumption G.2, −⟨∇J,∇G⟩≥c,∣∇J∣2-\langle \nabla J, \nabla \mathcal{G} \rangle \ge c\\,\\|\nabla J\\|^2 with cc independent of NN. The coefficient of ∣∇J∣2\\|\nabla J\\|^2 in the descent inequality is therefore β=(1−α)+αc≥min⁡1,c\beta = (1-\alpha) + \alpha c \ge \min\\{1, c\\}, not 1−α1-\alpha, and the variance enters the bound as σDG2(α)/β2\sigma_{\mathrm{DG}}^2(\alpha)/\beta^2, so the bound holds for every α∈(0,1)\alpha \in (0,1) and does not degrade as α→1\alpha \to 1.

For the α=1\alpha = 1 ablation, we refer the reviewer to our reply below on guidance-only learning and to the new Appendix M, which compares guidance-only learning with DG-PG at all scales of Section 6.4.

3. Request for a guidance-only ablation, α=1\alpha = 1 (Q2).

The original submission already defines DG-PG for α∈(0,1)\alpha \in (0,1) only (Section 4) and examines the guidance weight in this range (Section 6.2), where DG-PG is robust across the tested values. Guidance-only learning is excluded by design. At α=1\alpha = 1, the update follows only ∇G\nabla \mathcal{G}, so the policy minimizes the deviation from the reference instead of maximizing the cooperative objective, and Theorem 5.1 no longer applies. DG-PG retains (1−α)g^J(1-\alpha)\hat{g}_J precisely so that every stationary point of DG-PG is a stationary point of JJ for every α∈(0,1)\alpha \in (0,1), which also answers the question on the bias with respect to JJ.

We have nevertheless run guidance-only learning in cloud scheduling at all scales of Section 6.4 with the same seeds, and added the comparison to Appendix M. Up to N=1000N=1000, guidance-only learning and DG-PG reach comparable rewards, with differences within the spread across seeds. At N=1500N=1500, guidance-only learning becomes unstable, and two of the three seeds repeatedly build up long queues during training, giving −14.08±2.52-14.08 \pm 2.52 compared with −10.41±0.02-10.41 \pm 0.02 for DG-PG. The policy-gradient term thus keeps training stable at the largest scale, where the reference alone does not.

4. Request for MAPPO and IPPO at N≥50N \ge 50 and in power dispatch (W2, Q3).

The reviewer's own observation answers this request. As the reviewer notes, neural-network MAPPO and IPPO already fail at N=2N=2 and N=5N=5 despite extensive tuning (Appendix L), so running them at N≥50N \ge 50, where cross-agent noise is far stronger, cannot produce a meaningful comparison. This failure is precisely the scalability bottleneck that DG-PG addresses. For the same reason, we compare DG-PG in power dispatch with AGC, the standard frequency-regulation scheme in power system operation, and with a centralized greedy controller, and DG-PG outperforms both at all scales. We did run MAPPO and IPPO at N≥50N \ge 50, where cross-agent noise dominates the shared return and neither method learns at all, as their results at N≤10N \le 10 already indicate, so we did not report them separately.

AuthorsOpenReview ↗

Response to Reviewer si7Z (3/3)

5. Concern on unequal information, with a request for baselines that receive the same knowledge as a shaped or difference reward (W2, Q3).

The information is indeed unequal, but in favor of the baselines, not DG-PG. Best-Fit assigns tasks with global information on all servers, AGC dispatches by the same economic dispatch model that defines the DG-PG reference, and the centralized greedy controller predicts the system with the full system model (Appendix J). Each DG-PG agent decides from its local observation only, yet DG-PG outperforms all three at large scales. The reviewer also judges DG-PG against Best-Fit, which shows that equal information is not the standard of comparison in this review either.

The requested reinforcement-learning baselines with the same knowledge were already tested in the original submission. Run 7 in Appendix L gives MAPPO the DG-PG reference as a potential-based shaped reward, and it does not help, while MAPPO and IPPO already fall behind at small scales (Figure 2). Adding the knowledge to the reward cannot remove the noise that the other agents' actions inject into the shared reward, and this noise is exactly the bottleneck that DG-PG removes.

6. Concern on the small gains over Best-Fit (W2).

As explained in Sections 6.3 and 6.4 of the original submission, Best-Fit is a centralized heuristic that assigns tasks using global information on all servers, whereas each DG-PG agent decides from its local observation only. Best-Fit therefore serves as an upper reference, and DG-PG matching it at small scales and surpassing it at large scales is the intended result, not a small gain.

7. Concern on selecting checkpoints on the test seeds (W2, Q3).

As stated in Appendix K.1, all learned methods follow the same checkpoint selection rule. The results do not depend on this rule, since in Section 6.3 even the worst DG-PG checkpoint outperforms the best MAPPO and IPPO checkpoints on the test seeds. If anything, the rule favors MAPPO and IPPO, which are selected among 100 checkpoints, whereas DG-PG is selected among only six. Nevertheless, we followed the reviewer's suggestion and further evaluated each reported checkpoint on 10 held-out seeds, where no checkpoint performs worse and DG-PG retains a clear margin.

8. Question on the two different results for cloud scheduling at N=10N=10 (Q4).

These numbers come from different settings, as stated in Appendix K and in the captions of Figure 2 and Table 1. The value −9.69-9.69 is the best training reward of the linear DG-PG in the controlled comparison, whereas −13.19-13.19 is the test reward of the neural-network DG-PG on held-out seeds, so the two are not comparable.

We thank the reviewer again for the detailed review. If our responses resolve the concerns on the theoretical guarantees and the baselines, we would be grateful if the reviewer would reconsider the evaluation, and we would be glad to clarify any further questions during the discussion period.

Reviewer_kxsSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Descent-Guided Policy Gradient (DG-PG), a framework for scalable cooperative multi-agent reinforcement learning that augments conventional policy gradients with low-noise guidance derived from analytical reference states. The proposed guidance aims to alleviate cross-agent noise by providing agent-specific descent directions toward desirable system states. The authors claim that DG-PG preserves the stationary points of the original cooperative objective, reduces gradient-estimator variance from O(N)O(N) to O(1)O(1), and achieves agent-independent sample complexity of O(1/ϵ3)O(1/\epsilon^3). Experiments on cloud resource scheduling and power dispatch, involving discrete and continuous actions respectively, demonstrate rapid convergence and competitive performance with up to 1,500 heterogeneous agents.

Critical strengths
  1. Leveraging analytical models to improve multi-agent credit assignment is a promising direction, particularly for structured engineering systems such as power dispatch and cloud scheduling. 2.The paper evaluates up to 1,500 heterogeneous agents across discrete and continuous control tasks, with detailed experimental settings provided in the appendix.
  2. The paper is well organized, with a clear progression from motivation and methodology to theoretical analysis and experiments.
Critical weaknesses
  1. Theorem 5.2 establishes O(1)O(1) variance for an optimally chosen guidance weight α\alpha, whereas the experiments use a fixed α=0.9\alpha=0.9 at large scales. For fixed α<1\alpha<1, the standard policy-gradient component retains a variance contribution of (1−α)2Θ(N)(1-\alpha)^2\Theta(N), preventing asymptotically agent-independent variance. Theorem 5.3 further relies on the optimized variance bound while assuming fixed α\alpha, leaving the claimed agent-independent sample complexity insufficiently justified.

  2. Since the method relies heavily on analytical references, comparisons against reference-only guidance (α=1\alpha=1), imitation learning from analytical or heuristic controllers (where feasible), and imitation pretraining followed by PPO would help establish whether reinforcement learning provides additional benefits beyond the analytical prior.

  3. Theorem 5.1 assumes that the analytical reference is directionally aligned with the true cooperative objective. Appendix C establishes alignment for a surrogate load-balancing objective, but its extension to the full RL objective requires an additional compatibility assumption that is not independently verified. Moreover, the proof does not fully justify the transition from state-space alignment to policy-parameter-space alignment, weakening the stationary-point guarantee.

Questions for authors
  1. Could the authors provide comparisons with guidance-only learning (α=1\alpha=1) and imitation learning from analytical or heuristic controllers, where feasible?
  2. The standard policy-gradient component retains an O(N)O(N) variance contribution for fixed α<1\alpha<1, how do the authors reconcile this with the claimed agent-independent variance and sample-complexity guarantees?
3 comments
AuthorsOpenReview ↗

Response to Reviewer kxsS (1/2)

We thank the reviewer for recognizing the promise of leveraging analytical models in structured engineering systems and the clear organization of the paper. DG-PG builds on the analytical models that are widely available in engineered systems, such as optimal power flow and queueing models, and uses them to provide each agent with a learning signal, which makes learning efficient and reliable in large heterogeneous systems. The same approach applies to most resource-allocation and control problems in engineered systems, such as communication networks, voltage control and urban mobility, all of which have such analytical models.

As the reviewer notes, the starting point and goal of this work is CMARL in engineered systems, rather than CMARL in general. The analytical reference is therefore not a limitation but the foundation that gives DG-PG its O(1)O(1) variance. Theorem 5.1 is also stated for these systems, not for arbitrary RL objectives, and we also show in Appendix C how the compatibility condition follows from the analytical model of the system. We regard the general objective raised by the reviewer as a promising future direction that may also inform other CMARL settings such as StarCraft II, but it poses a different question from the one this paper studies.

Regarding the concern that the performance of DG-PG may come from the analytical prior rather than from learning, we would first like to clarify that the analytical models do not provide actions. They only prescribe an efficient state of the system and do not indicate how each agent should act, so imitation learning from them is not possible. Second, the original submission already shows that other methods given the same prior fall short of DG-PG. MAPPO with the reference as a shaped reward does not improve, and the controllers built on the same models, including Best-Fit with global information, fall behind DG-PG at large scales. We have also added guidance-only learning (α=1\alpha = 1) in Appendix M, which becomes unstable at the largest scale while DG-PG remains stable.

Finally, the concern that the O(1)O(1) variance does not hold mainly arises from misreading Theorem 5.2 as a bound at a fixed α\alpha, whereas we bound the minimum variance over α\alpha. Under the parameter-space alignment that we have made explicit as Assumption G.2, we prove that Theorem 5.3 holds at this α\alpha and that the guarantee of Theorem 5.1 holds in parameter space. The experiments also confirm the O(1)O(1) variance, since learning essentially does not slow down even at 1,500 agents. We respond to each concern in detail below.

1. Concern on whether the variance and sample complexity remain independent of NN at a fixed α<1\alpha < 1 (W1, Q2).

We would first like to clarify that Theorem 5.2 states the minimum variance over α\alpha, not the variance at a fixed α\alpha. Section 5.2 explains that the optimal weight α∗\alpha^* tends to 11 as NN grows, while at finite scales α\alpha is a tuning choice. At the tested scales, the fixed α=0.9\alpha = 0.9 already keeps learning from slowing down with NN, since the original submission shows that DG-PG reaches within 10% of its converged reward in fewer than 10 episodes on average and that this number remains bounded up to N=1500N = 1500 (Section 6.4).

Second, Theorem 5.3 holds at the α∗\alpha^* of Theorem 5.2, although α∗\alpha^* depends on NN, so the sample complexity is independent of NN. In Theorem 5.3, a fixed guidance weight means that α\alpha is held constant during training, and the condition σDG2=O(1)\sigma_{\mathrm{DG}}^2 = O(1) from Theorem 5.2 sets it to α∗\alpha^*. We have made this choice explicit in the statement of Theorem 5.3. The bound holds for every α∈(0,1)\alpha \in (0,1), because the proof uses the alignment between ∇J\nabla J and ∇G\nabla \mathcal{G}, which we have made explicit as Assumption G.2, −⟨∇J,∇G⟩≥c ∥∇J∥2-\langle \nabla J, \nabla \mathcal{G} \rangle \ge c\,\|\nabla J\|^2 with cc independent of NN. The coefficient of ∥∇J∥2\|\nabla J\|^2 in the descent inequality is therefore β=(1−α)+αc≥min⁡{1,c}\beta = (1-\alpha) + \alpha c \ge \min\{1, c\}, which does not degrade as α→1\alpha \to 1, and substituting the variance of Theorem 5.2 gives the agent-independent sample complexity.

AuthorsOpenReview ↗

Response to Reviewer kxsS (2/2)

2. Concern on the reliance on analytical references, with a request for comparisons with imitation learning and imitation pretraining (W2).

We would first like to clarify that the analytical reference is the knowledge DG-PG brings into learning, not a limitation. Engineered systems such as cloud computing and power systems already have analytical models of efficient operation, and bringing these models into the policy gradient is what allows DG-PG to outperform MAPPO and IPPO as the number of agents grows. Second, imitation learning from this knowledge is not possible, because the analytical model only prescribes an efficient state of the system and does not indicate how each agent should act. Third, the original submission already establishes the benefit of learning beyond the analytical prior by comparing DG-PG with the controllers built on it. In particular, Best-Fit is a centralized algorithm with global information, whereas DG-PG learns decentralized policies that act on local observations and still outperforms it at large scales. Imitation pretraining would at best start from these controllers.

3. Request for a comparison with guidance-only learning, α=1\alpha = 1 (Q1).

The original submission already defines DG-PG for α∈(0,1)\alpha \in (0,1) only (Section 4) and examines the guidance weight in this range (Section 6.2), where DG-PG is robust across the tested values. Guidance-only learning is excluded by design. At α=1\alpha = 1, the update follows only ∇G\nabla \mathcal{G}, so the policy minimizes the deviation from the reference instead of maximizing the cooperative objective, and Theorem 5.1 no longer applies. DG-PG retains (1−α)g^J(1-\alpha)\hat{g}_J precisely so that every stationary point of DG-PG is a stationary point of JJ for every α∈(0,1)\alpha \in (0,1).

We have nevertheless run guidance-only learning in cloud scheduling at all scales of Section 6.4 with the same seeds, and added the comparison to Appendix M. Up to N=1000N=1000, guidance-only learning and DG-PG reach comparable rewards, with differences within the spread across seeds. At N=1500N=1500, guidance-only learning becomes unstable, and two of the three seeds repeatedly build up long queues during training, giving −14.08±2.52-14.08 \pm 2.52 compared with −10.41±0.02-10.41 \pm 0.02 for DG-PG. The policy-gradient term thus keeps training stable at the largest scale, where the reference alone does not.

4. Concern on the compatibility of the reference with the full RL objective and on the transition from state-space to parameter-space alignment in Theorem 5.1 (W3).

We would first like to clarify that this paper studies engineered systems whose analytical models describe efficient operation, so Theorem 5.1 is stated for these systems, not for arbitrary RL objectives, and Assumption 4.3 expresses exactly this property. Second, the compatibility condition follows from the analytical model of the system and is confirmed by the results, as we show for cloud scheduling and the congestion game. In cloud scheduling, Appendix C shows from queueing theory that queueing delay, energy cost and fairness all improve as utilization is balanced in proportion to capacity. The results confirm this, since load balancing is not part of the reward and enters only through the guidance (Appendix I.3), yet it raises the reward of DG-PG far above that of MAPPO and IPPO. In the congestion game, the condition holds exactly, since C(x)=C(x~)+∑n(xn−x~n)2/cnC(x) = C(\tilde{x}) + \sum_n (x_n - \tilde{x}_n)^2 / c_n for every load xx.

The stationary-point guarantee of Theorem 5.1, that every stationary point of DG-PG is a stationary point of JJ, follows directly from the alignment in parameter space, Assumption G.2. If the DG-PG gradient (1−α)∇J−α∇G(1-\alpha)\nabla J - \alpha \nabla \mathcal{G} vanishes, taking its inner product with ∇J\nabla J gives 0≥β∥∇J∥20 \ge \beta \|\nabla J\|^2, so ∇J=0\nabla J = 0, and every stationary point of DG-PG is a stationary point of JJ. In the congestion game, with the deviation measured by the cost, Assumption G.2 holds exactly with c=1c = 1.

We thank the reviewer again for the constructive comments. If our responses resolve the concerns on the role of the analytical models and on the theoretical guarantees, we would be grateful if the reviewer would reconsider the evaluation, and we would be glad to answer any further questions.

Dear Reviewer, thank you again for your constructive comments. Given your assessment that the soundness concerns are minor or fixable, we would appreciate your thoughts on whether our clarifications and the requested guidance-only comparison address them. If so, we would be grateful if you could reconsider your evaluation. We would be happy to discuss any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.