- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the stability of primal–dual training in deep constrained multi-agent reinforcement learning under tight per-agent cost budgets. It proposes an anchored policy regularizer that penalizes deviations from a slowly updated reference policy, alongside per-agent vector Lagrange multipliers and an optional action-perturbation adversary. The method is instantiated on both transformer-based and off-policy actor–critic backbones. Separately, the paper provides a counterexample to an existing robust constrained Markov game equilibrium-existence claim and establishes existence of a weaker Lagrangian-stationary equilibrium under a robust Slater condition.
Validation centers on cooperative navigation, using budget sweeps, component and stabilizer ablations, and attackers retrained against frozen policies, supplemented by additional cooperative and competitive tasks. The results show substantial improvements in task completion and performance retained under attack on navigation, and reduced multiplier saturation on pursuit–evasion.
Critical strengths
-
The stabilization benefit is supported by substantial controlled evidence. At the tight navigation budget, AC-MAT achieves a clean goal rate of 0.947 across 15 seeds, compared with 0.743 for the unanchored variant trained without warm-up; under retrained attacks, these rates are 0.645 and 0.274, respectively. Controls covering PID updates, warm-up schedules, learning rates, and alternative regularizers strengthen the attribution to slow-reference anchoring. This provides convincing evidence of a practically meaningful improvement within the studied setting, extending beyond recovery from warm-up-induced collapse.
-
The paper makes a valuable correction to the theoretical foundations of robust constrained Markov games. It provides an explicit counterexample to a previously claimed equilibrium-existence result and proves existence of a weaker Lagrangian-stationary equilibrium under robust Slater conditions. The distinction between worst-case feasibility and feasibility at a Lagrangian-minimizing kernel clarifies what the surrogate formulation can actually guarantee. This correction contributes useful knowledge independently of the deep-learning implementation and is an important consideration supporting acceptance.
Critical weaknesses
-
The stabilization benefit has limited demonstrated transfer across tasks and implementations. Table 17 shows clear benefits primarily on navigation and the custom pursuit–evasion environment, with no consistent anchor advantage on discovery, dispersion, tag, or the evaluated MuJoCo tasks. Adding the anchor to the official MAPPO-Lagrangian implementation also fails to reproduce the trainability improvement (Appendix D.13). These results leave unclear which properties of the task or optimization procedure make anchoring effective. This limits the generality of the contribution and makes its usefulness outside the successful experimental settings difficult to predict.
-
Higher task performance comes with less reliable per-agent constraint satisfaction. Across the default navigation budget grid, AC-MAT satisfies every agent’s budgets in 30/45 runs, compared with 44/45 for the unanchored, no-warm-up variant (Table 18). Tightening the training budgets substantially improves feasibility, but the selected margin achieves 41/45 feasible runs on fresh seeds, while the corresponding margin-adjusted baseline achieves 44/45 (Appendix D.9). The authors transparently report this trade-off, and AC-MAT retains higher task performance. Nevertheless, the remaining reliability gap is consequential for applications where individual budget compliance is the primary requirement, limiting the method’s practical advantage as a constrained-control approach.
Questions for authors
-
Clarification of the transfer failures. Appendix D.13 reports that transferring the anchor, deterministic cost signal, and multiplier update into official MAPPO-Lagrangian does not reproduce the improvement. Which consequential differences from AC-MAT remain after these controls, particularly in multiplier initialization and optimization settings? Do the existing logs or ablations help distinguish an implementation-specific interaction from a limitation of the anchoring mechanism? This distinction would materially affect my assessment of generality.
-
Clarification of the optimization target. The policy update uses costs collected under stochastic execution, whereas the multiplier update uses costs from the deterministic mean policy. Do these updates correspond to a common objective or a precisely defined stationarity condition? If not, what mathematical connection remains between the implemented algorithm and Definition 1? This would clarify how much justification the equilibrium results provide for the practical method.
Review
Thank you for the detailed rebuttal and additional experiments. The new PID-Lagrangian results provide encouraging evidence that the anchored regularizer can improve performance beyond the proposed implementation, which partially addresses my concern about portability.
However, the conditions determining when anchoring is beneficial remain unclear. In particular, the results on discovery and navigation without warm-up suggest that training collapse alone does not fully explain its effectiveness. I also appreciate the clarification that the theoretical results do not establish convergence of the deep algorithms.
Regarding safety, the proposed mitigations are useful, but substantial violations under cost-directed attacks remain, and per-agent feasibility is not established under such perturbations.
Overall, the rebuttal strengthens the empirical contribution and clarifies the scope of the claims, although the limitations concerning generalization, stability mechanisms, and robust constraint satisfaction remain. I therefore maintain my original rating.