acceptodds
Under review as a conference paper at ICLR 2027

An Anchored Policy Regularizer for Deep Constrained Multi-Agent Reinforcement Learning under Tight Cost Budgets

Abstract

Reinforcement learning teams must often keep costs like energy or collisions within budget. Standard training gives each budget a Lagrange multiplier that rises under violation and falls otherwise. At binding budgets (below what an unconstrained policy spends), policy and multiplier can cycle and deep training can collapse. It is unknown which cheap stabilizer prevents this, or whether the policy survives an adversary. We propose an anchored policy regularizer that pulls each policy toward a slow copy of itself. In the tabular robust constrained Markov game framework, the existence theorem needs a weaker equilibrium notion; we prove that the one behind our multiplier update exists under robust Slater. On cooperative navigation at tight budgets, unanchored training with our multiplier update collapses after a warm-up holding the multipliers at zero, and trains without it. The anchor removes the collapse and lifts the goal rate above every constrained variant without it, with or without warm-up. Under a bounded action-perturbing adversary retrained against each final policy, anchored variants keep most of their goal rate. Training against such an adversary adds no gain beyond seed noise at that perturbation size. The method meets every budget pair of a navigation grid without per-pair tuning, in seed-pooled mean cost and per agent in two thirds of runs. We enforce the budgets on the deployed deterministic policy.

Primary area:
reinforcement learning
Keywords:
constrained Markov games, multi-agent reinforcement learning, primal-dual learning, Lagrangian relaxation, robust reinforcement learning, policy regularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SP4ZOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the stability of primal–dual training in deep constrained multi-agent reinforcement learning under tight per-agent cost budgets. It proposes an anchored policy regularizer that penalizes deviations from a slowly updated reference policy, alongside per-agent vector Lagrange multipliers and an optional action-perturbation adversary. The method is instantiated on both transformer-based and off-policy actor–critic backbones. Separately, the paper provides a counterexample to an existing robust constrained Markov game equilibrium-existence claim and establishes existence of a weaker Lagrangian-stationary equilibrium under a robust Slater condition.

Validation centers on cooperative navigation, using budget sweeps, component and stabilizer ablations, and attackers retrained against frozen policies, supplemented by additional cooperative and competitive tasks. The results show substantial improvements in task completion and performance retained under attack on navigation, and reduced multiplier saturation on pursuit–evasion.

Critical strengths
  1. The stabilization benefit is supported by substantial controlled evidence. At the tight navigation budget, AC-MAT achieves a clean goal rate of 0.947 across 15 seeds, compared with 0.743 for the unanchored variant trained without warm-up; under retrained attacks, these rates are 0.645 and 0.274, respectively. Controls covering PID updates, warm-up schedules, learning rates, and alternative regularizers strengthen the attribution to slow-reference anchoring. This provides convincing evidence of a practically meaningful improvement within the studied setting, extending beyond recovery from warm-up-induced collapse.

  2. The paper makes a valuable correction to the theoretical foundations of robust constrained Markov games. It provides an explicit counterexample to a previously claimed equilibrium-existence result and proves existence of a weaker Lagrangian-stationary equilibrium under robust Slater conditions. The distinction between worst-case feasibility and feasibility at a Lagrangian-minimizing kernel clarifies what the surrogate formulation can actually guarantee. This correction contributes useful knowledge independently of the deep-learning implementation and is an important consideration supporting acceptance.

Critical weaknesses
  1. The stabilization benefit has limited demonstrated transfer across tasks and implementations. Table 17 shows clear benefits primarily on navigation and the custom pursuit–evasion environment, with no consistent anchor advantage on discovery, dispersion, tag, or the evaluated MuJoCo tasks. Adding the anchor to the official MAPPO-Lagrangian implementation also fails to reproduce the trainability improvement (Appendix D.13). These results leave unclear which properties of the task or optimization procedure make anchoring effective. This limits the generality of the contribution and makes its usefulness outside the successful experimental settings difficult to predict.

  2. Higher task performance comes with less reliable per-agent constraint satisfaction. Across the default navigation budget grid, AC-MAT satisfies every agent’s budgets in 30/45 runs, compared with 44/45 for the unanchored, no-warm-up variant (Table 18). Tightening the training budgets substantially improves feasibility, but the selected margin achieves 41/45 feasible runs on fresh seeds, while the corresponding margin-adjusted baseline achieves 44/45 (Appendix D.9). The authors transparently report this trade-off, and AC-MAT retains higher task performance. Nevertheless, the remaining reliability gap is consequential for applications where individual budget compliance is the primary requirement, limiting the method’s practical advantage as a constrained-control approach.

Questions for authors
  1. Clarification of the transfer failures. Appendix D.13 reports that transferring the anchor, deterministic cost signal, and multiplier update into official MAPPO-Lagrangian does not reproduce the improvement. Which consequential differences from AC-MAT remain after these controls, particularly in multiplier initialization and optimization settings? Do the existing logs or ablations help distinguish an implementation-specific interaction from a limitation of the anchoring mechanism? This distinction would materially affect my assessment of generality.

  2. Clarification of the optimization target. The policy update uses costs collected under stochastic execution, whereas the multiplier update uses costs from the deterministic mean policy. Do these updates correspond to a common objective or a precisely defined stationarity condition? If not, what mathematical connection remains between the implemented algorithm and Definition 1? This would clarify how much justification the equilibrium results provide for the practical method.

Reviewer_Gev4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles an under-reported issue in constrained MARL: when budgets are tight, primal–dual methods can spin their wheels, multipliers ramp up while the policy overspends, then drop while it retreats, before collapsing into a degenerate state. You end up either with multipliers maxed out and the task unsolved, or multipliers near zero because the policy has retreated to cheap, in-budget non-solutions. The fix is a purely primal anchored regulariser: a squared penalty that pulls each agent’s action mean toward a slow copy of itself, added to the policy objective while leaving the reward, multipliers, and dual update untouched. It’s paired with per-agent vector multipliers (one per agent per cost dimension) and tested on MAT and MATD3. Separately, the paper sharpens the tabular RCMG existence theorem: it shows the published Upper-RFNE existence claim needs an extra hypothesis, provides a counterexample with an exact RFNE but no Upper-RFNE, and proves existence of a weaker Lagrangian-stationary notion under robust Slater.

Critical strengths
  • The ablations are honest and well-controlled. The setup (shared backbone, matched sizes and training length, components toggled one at a time) is exactly what this sort of claim needs, and the paper doesn’t oversell the adversary

  • The theory side-contribution is genuine too. Showing the dual objective is a pointwise max of concave functions (not convex), producing a 9-state counterexample with a duality gap of 2, and proving a stationarity notion exists is a real correction to a published result, independent of the deep method.

Critical weaknesses
  • The anchor is a heuristic with a toy-model justification. The bilinear saddle model proves asymptotic stability, but that’s a scalar linear system; the deep-system damping is explicitly labelled a conjecture in Table 2. The phase-lag correlation is suggestive, not causal. I am not sure that whether the anchor is doing what the paper claims, or merely acting as a generic trust region / update-smoother that happens to help on this domain.

  • Budgets hold only for the deterministic deployed policy. Under sampled execution RC-MAT burns through roughly three times its deterministic energy and blows every budget. The paper is upfront that this is a deliberate design choice, but for a paper motivated by "delivery robots with imprecise motors", regulating only the mean policy leaves a fair gap between the motivation and the guarantee.

Questions for authors

Is the anchor distinguishable from a per-iteration trust region pinned to a lagged iterate? The KL-equalised control (to the previous iterate) fails, which is evidence, but a control that anchors to a lagged iterate with a tuned KL budget would sharpen the mechanism claim.

The phase-lag correlation is computed on the first 300 updates after warm-up. Does it persist later in training, and does it show up in the no-warm-up variant?

On the theory side: does the Lagrangian-stationary notion have any bearing on the deep fixed points, or is it purely a tabular existence result? Theorem 2's alignment hypothesis is restrictive; how plausible is it in the domains tested?

What's the wall-clock and parameter cost of the anchor? It's billed as pure primal, but the EMA copy and extra decode aren't free.

Reviewer_ZxRvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an anchored policy regularizer to stabilize primal–dual training in constrained multi-agent reinforcement learning under tight cost budgets. The method penalizes deviations between the current policy and a slowly updated reference policy, aiming to mitigate training instability caused by interactions between policy optimization and Lagrange multiplier updates.

The authors instantiate the approach on both on-policy and off-policy MARL backbones and evaluate feasibility, task performance, and robustness against action-perturbing adversaries. The paper also revisits the equilibrium existence theory of robust constrained Markov games, provides a counterexample to a previous claim, and establishes an existence result for a weaker stationarity notion.

Critical strengths
  1. The anchored regularizer substantially improves training stability and task success under binding cost budgets. The controlled ablations against unanchored, PID-based, and adversarially trained variants provide strong evidence of its practical effectiveness in the primary setting.
  2. The paper identifies a limitation in an existing equilibrium existence result and establishes a weaker stationarity guarantee. The extensive experiments further distinguish policy stability, budget feasibility, and adversarial robustness, while explicitly documenting important negative results.
Critical weaknesses
  1. The anchor provides substantial improvements in navigation and pursuit–evasion but shows limited benefits on several transfer tasks. Moreover, adding the same regularizer to the official MAPPO-Lagrangian implementation does not reproduce the improvement, raising questions about its applicability beyond the proposed training configuration.
  2. The equilibrium existence results concern finite tabular games and do not establish convergence of the deep algorithms. Furthermore, budget feasibility is primarily demonstrated for deterministic policies and aggregated costs, while cost-directed adversarial attacks can substantially violate the constraints.
Questions for authors
  1. Why does the anchored regularizer fail to provide comparable improvements on several transfer tasks and the official MAPPO-Lagrangian implementation?

  2. Can the authors identify conditions under which anchoring reliably stabilizes deep primal–dual MARL beyond the bilinear theoretical model?

  3. How could the approach be extended to maintain per-agent budget feasibility under cost-directed adversarial perturbations?

2 comments
Reviewer_ZxRvOpenReview ↗

Review

Thank you for the detailed rebuttal and additional experiments. The new PID-Lagrangian results provide encouraging evidence that the anchored regularizer can improve performance beyond the proposed implementation, which partially addresses my concern about portability.

However, the conditions determining when anchoring is beneficial remain unclear. In particular, the results on discovery and navigation without warm-up suggest that training collapse alone does not fully explain its effectiveness. I also appreciate the clarification that the theoretical results do not establish convergence of the deep algorithms.

Regarding safety, the proposed mitigations are useful, but substantial violations under cost-directed attacks remain, and per-agent feasibility is not established under such perturbations.

Overall, the rebuttal strengthens the empirical contribution and clarifies the scope of the claims, although the limitations concerning generalization, stability mechanisms, and robust constraint satisfaction remain. I therefore maintain my original rating.

Thank you for considering the PID-Lagrangian runs. We agree that training collapse alone does not explain the anchor's effect.

Without the warm-up the unanchored variant does not collapse, and the anchor still raises the goal rate on navigation from 0.743 to 0.954, with two of five anchored seeds feasible in agent mean against fifteen of fifteen (Table 18). On discovery no tested energy budget is both binding and attainable (Appendix D.5), so that task does not test the anchor under binding budgets. We have no condition that predicts the gain in advance.

We also agree that cost-directed attacks breach the budgets (Section 5.3) and that per-agent feasibility under them is not established.

Thank you again; we would be glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.