acceptodds
Under review as a conference paper at ICLR 2027

Who Bears the Burden? Learning Responsibility for Shared Constraints in Multi-Agent Reinforcement Learning

Abstract

When multiple agents share a cost budget, a common Lagrange multiplier can enforce the aggregate constraint but does not determine how its penalty should be allocated across agents. Uniform penalties ignore heterogeneity in the rewards agents sacrifice, while agent-specific multipliers may still rely on the same aggregate cost signal. We introduce Lagrangian Responsibility Allocation (LiRA), which learns each agent's share of a common multiplier by optimizing social welfare over a finite training horizon. The multiplier enforces the aggregate budget, while responsibility shares redistribute its influence without modifying the original rewards or constraints. For convex games under standard regularity conditions, varying these shares induces a smooth family of normalized generalized Nash equilibria in which active constraints remain at their budgets while welfare varies. To optimize responsibility before convergence, we derive a welfare gradient that accounts for both learning updates and the induced change in data distribution. Across CityLearn, MABIM, Harvest, and MetaDrive, spanning 3 to 400 agents, LiRA improves average social welfare by up to 29% over uniform and agent-specific multiplier baselines. Grid and driving costs remain within budget, inventory violations decrease, and Harvest makes more effective use of available budget.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Safe Reinforcement Learning, Constrained Reinforcement Learning, Equilibrium Selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qjSwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes LiRA, a method that learns agent-specific responsibility shares for shared constraints in multi-agent reinforcement learning. It provides theoretical analysis of equilibrium selection and a finite-horizon meta-gradient estimator. Experiments across four benchmarks show improved social welfare over Lagrangian baselines.

Critical strengths
  1. The theoretical analysis provides a principled connection between responsibility allocation and normalized generalized Nash equilibrium selection.

  2. Experiments across four diverse environments demonstrate consistent improvements in mean social welfare, and the CityLearn case study provides an interpretable explanation of the learned allocations.

  3. The proposed parameterization is simple and can potentially be incorporated into existing Lagrangian MARL algorithms.

Critical weaknesses
  1. The paper does not provide DU-only or SC-only ablations. Consequently, it is unclear whether the sampling correction is necessary or whether the improvements primarily come from learning responsibility shares.

  2. LiRA requires multiple independent lookaheads and additional evaluation rollouts. Without comparisons under equal environment-interaction or computational budgets, its practical advantage over simpler Lagrangian methods remains unclear.

  3. Theorem 1 characterizes equilibrium selection along active constraint surfaces, whereas the experiments involve finite-time nonconvex learning, with some constraints remaining violated or substantially inactive. It is unclear whether the observed improvements reflect equilibrium selection or merely transient learning effects.

Questions for authors
  1. How much does the sampling-correction term contribute to performance? Can the authors provide DU-only and SC-only ablations, ideally together with gradient variance or directional accuracy measurements?

  2. Does LiRA still outperform simpler adaptive responsibility-allocation methods when all approaches are evaluated under the same total environment-interaction budget?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qjSw (1/3)

We thank the reviewer for the review and for the concrete requests. We have added the DU-only ablation on the four benchmarks (Table 1), the DU-only and SC-only ablations in a controlled game with the bias of each estimate and the direction of DU only (Appendix F.1), a comparison at the same total environment interactions (Sec. 4.4) and a result on finite horizons (Appendix E.3).


Weakness 1: The paper does not provide DU-only or SC-only ablations. Consequently, it is unclear whether the sampling correction is necessary or whether the improvements primarily come from learning responsibility shares.

Question 1: How much does the sampling-correction term contribute to performance? Can the authors provide DU-only and SC-only ablations, ideally together with gradient variance or directional accuracy measurements?

We ran the DU-only ablation on the four benchmarks (new last column of Table 1), and both ablations in a three-agent game in which the true gradient can be computed (Appendix F.1).

Rebuttal Table 1. Welfare of LiRA with and without the sampling correction (SC) on the benchmarks, with the protocol and seeds of Table 1 (mean ± s.d. over seeds). The LiRA column repeats Table 1, and the higher mean is in bold.

TaskLiRALiRA without SC
CityLearn (10310^3)−14.51 ± 1.66−16.19 ± 6.10
MABIM (10610^6)−591.36 ± 4.48−588.85 ± 9.51
Harvest50.27 ± 2.5345.93 ± 1.94
MetaDrive64.26 ± 19.9857.75 ± 34.97

Without SC the mean welfare is lower in CityLearn, Harvest and MetaDrive, and the mean cost is over the budget in CityLearn and MetaDrive, where LiRA meets it. With SC the standard deviation of welfare over seeds is also smaller in CityLearn, MABIM and MetaDrive, by a factor of 1.8 to 3.7. In MABIM the two means differ by less than this standard deviation. Both versions have a higher mean welfare than Uniform in all four tasks, so learning the responsibility shares with DU alone already helps. SC adds a further gain in three tasks, and in CityLearn and Harvest this is more than half of LiRA's gain over Uniform. For the gradient variance, we measured the share of the variance of the estimate that comes from the SC term. This share is 0.6% in CityLearn, 7% in MABIM, 4% in Harvest and 44% in MetaDrive.

The directional accuracy can only be measured where the true gradient is known. Rebuttal Table 2 gives the results in the controlled game, which uses the lookahead length and the number of continuations of our CityLearn runs (q=2q=2, M=8M=8).

Rebuttal Table 2. Gradient estimators in the scalar game of Appendix F.1. The game with an externality gives similar numbers (Table 6). The welfare gap is the gap between uniform shares and the optimum, and a negative value means that the learner ends below uniform shares. The best value in each column is in bold.

EstimatorRelative biasWelfare gap closed
Full (DU + SC)0.0295%
DU only0.1397%
SC only1.14−980%
Stopped batch, as in our implementation0.1690%

The sampling correction (SC) is the term that makes the estimate unbiased. The full estimator has a relative bias of 0.02, which is Monte Carlo error, and direct unrolling (DU) alone has a relative bias of 0.13. For the directional accuracy, the DU only estimator has a cosine of at least 0.99 with the true gradient, so its bias is almost all in the magnitude. In this small game a learner that follows either estimate closes about the same share of the welfare gap. SC alone does not give a useful direction, which is expected of a term that corrects DU and is not by itself an estimate of the gradient. For this reason we did not run SC alone on the benchmarks. SC also adds variance, and Figure 3b shows the mean squared error of the two estimators against the number of continuations.

AuthorsOpenReview ↗

Response to Reviewer qjSw (2/3)

Weakness 2: LiRA requires multiple independent lookaheads and additional evaluation rollouts. Without comparisons under equal environment-interaction or computational budgets, its practical advantage over simpler Lagrangian methods remains unclear.

Question 2: Does LiRA still outperform simpler adaptive responsibility-allocation methods when all approaches are evaluated under the same total environment-interaction budget?

Yes, LiRA still outperforms them under the same total budget. With the same total number of environment interactions for every method, LiRA has the highest mean welfare within budget in all four tasks, against Uniform, the per-agent Lagrangian (PAL), successive halving and random search (new Sec. 4.4). Random search (Bergstra & Bengio, 2012) draws candidate shares at random, trains each of them for the same number of interactions and keeps the one with the highest training welfare. Successive halving (Jamieson & Talwalkar, 2016) trains all candidates for a short round, discards the worse half and repeats until one is left. For the new comparison in MetaDrive, MABIM and CityLearn, LiRA runs the lookahead on the live training batches. It differentiates through the last few PPO updates of training and so collects no more data than Uniform (Appendix H.1). In Harvest we gave the other methods as many interactions as LiRA uses with its lookaheads.

Rebuttal Table 3. Table 4 in Sec. 4.4. LiRA's welfare gain over each method at equal environment interactions. MABIM welfare is in units of 10610^6. Successive halving and random search try several fixed shares within the same number of interactions. †^\dagger marks entries where the compared method is over budget.

Taskvs. Uniformvs. PALvs. successive halvingvs. random search
CityLearn+520+520+3,775$^\dagger$+6,337
MABIM+128.9$^\dagger$+128.8$^\dagger$+186.0$^\dagger$+184.2$^\dagger$
Harvest−0.6$^\dagger$+1.1+2.3+0.6
MetaDrive+19.0+8.8+12.1+14.0

For computation, the lookahead on the live batches needs no extra data, and a LiRA run takes 1.0–2.2× as long as a run of Uniform (Rebuttal Table 4).

Rebuttal Table 4. Table 10 in Appendix G. Training time of one run at equal environment interactions. In MABIM and CityLearn each allocation update averages 8 and 10 replays of the stored updates, which are run in parallel with one worker process per replay. The CityLearn times are per training cycle.

TaskUniformLiRARatio
CityLearn0.95 s2.1 s2.2×
MABIM85 s185 s2.2×
Harvest1,499 s1,552 s1.0×
MetaDrive226 s268 s1.2×

In MABIM we also trained Uniform and PAL for eight times as many steps, so that a run takes at least as long as a LiRA run. Their welfare did not improve. It is −603.7 for Uniform and −608.6 for PAL, against −457.4 for LiRA (in units of 10610^6).

References

  • Bergstra and Bengio. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 2012.
  • Jamieson and Talwalkar. Non-stochastic Best Arm Identification and Hyperparameter Optimization. AISTATS, 2016.
AuthorsOpenReview ↗

Response to Reviewer qjSw (3/3)

Weakness 3: Theorem 1 characterizes equilibrium selection along active constraint surfaces, whereas the experiments involve finite-time nonconvex learning … It is unclear whether the observed improvements reflect equilibrium selection or merely transient learning effects.

We agree that Theorem 1 is a statement about equilibria and cannot by itself settle this question for deep policy networks, as we say in Sec. 6 of the submission. We have added a theoretical result and further evidence.

The new theoretical result, Proposition 3, connects the two regimes.

Proposition 3 (Appendix E.3, informal). Let the expected learning map be a contraction with factor ζ<1\zeta<1 near its fixed point x∗(ψ)x^{\ast}(\psi), let W(ψ)\mathcal W(\psi) be the welfare at this fixed point, and let Vˉq(ψ∣x0)\bar V_q(\psi\mid x_0) be the welfare after qq lookahead steps from the current parameters x0x_0. Then

∇ψVˉq(ψ∣x0)=∇W(ψ)−Tq(ψ)+Eq(ψ,x0),\nabla_\psi\bar V_q(\psi\mid x_0)=\nabla\mathcal W(\psi)-\mathcal T_q(\psi)+\mathcal E_q(\psi,x_0),

where the truncation term satisfies ∥Tq∥=O(ζq)\|\mathcal T_q\|=O(\zeta^q) and the transient term satisfies ∥Eq∥=O(qζq−1∥x0−x∗(ψ)∥)\|\mathcal E_q\|=O(q\zeta^{q-1}\|x_0-x^{\ast}(\psi)\|).

In words, the finite-horizon gradient that LiRA follows equals the gradient of welfare at the learner's equilibrium, up to a truncation term that shrinks geometrically with the lookahead length and a transient term that vanishes as training approaches the equilibrium.

We also have evidence that the gain is not transient when the constraint is active. In a game with two outcomes of training, LiRA changes which of them training ends in (Rebuttal Table 5).

Rebuttal Table 5. From Table 7 in Appendix F.2. Outcomes of the two-outcome game over 400 runs per method. H is the outcome with the higher welfare and L the other one. The first three columns give the fraction of runs that end in H, over all runs, over the runs that start in the basin of H, and over those that start in the basin of L. Welfare is the mean ± 95% confidence interval. The better value in each column is in bold.

MethodEnds in HEnds in H, from an H startEnds in H, from an L startWelfare
Uniform / PAL0.7180.9590.0650.922 ± 0.018
LiRA0.7770.9860.2130.946 ± 0.016

In a turn-taking game, LiRA with shares that depend on the state closes 93% of the gap to the optimum with the cost at the budget (Appendix F.3). On the benchmarks, we trained CityLearn for 800 cycles with grid import capped at 4 kWh and evaluated the second half of training. Uniform and LiRA both end at the budget there, and LiRA's welfare is higher (−6,000 against −6,520, Appendix H.2). In the comparison at equal interactions LiRA is within the budget in all four tasks, and its cost is close to the budget in CityLearn (0.299 against 0.30) and Harvest (47.90 against 48.2, Table 11 in Appendix H).


Thank you again for your time and for the concrete suggestions. We would be happy to answer any further questions.

Reviewer_4UeDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies shared constraints in cooperative and general-sum multi-agent reinforcement learning, where uniform penalties can overlook differences in agents’ reward–cost trade-offs. The proposed Lagrangian Responsibility Allocation (LiRA) combines a global dual variable with state-independent, agent-specific responsibility shares while preserving task rewards and safety budgets. Experiments on CityLearn, MABIM, Harvest, and MetaDrive show improved social welfare while generally respecting or improving constraint satisfaction.

Critical strengths
  1. The motivation is clear: uniform penalties for shared costs can be inefficient when agents have different marginal reward–cost trade-offs.

  2. LiRA preserves the original rewards, costs, and budgets, so the reported improvements remain meaningful under the original evaluation criteria.

  3. Experiments span diverse domains with 3–400 agents. The CityLearn case study also illustrates how asymmetric responsibility shares can correct persistent overcooling.

Critical weaknesses
  1. Theorem 1 does not apply to deep MARL. Assumptions (A1)–(A3) require concave rewards, convex costs, and uniform strong monotonicity. As briefly noted in Section 6, neural-network policies (and even tabular policies) violate these conditions. Moreover, strong monotonicity rules out multiple equilibria, which are common in coordination tasks (e.g., whether agent A yields to B or B yields to A). In practice, small updates to responsibility shares can cause policies to jump between different equilibria rather than move smoothly along a single equilibrium curve. Thus, Theorem 1 provides no theoretical guarantee for the actual deep MARL experiments.

  2. Responsibility shares are static across states. Equation 2 (and Section 3.1) assigns each agent a fixed share independent of the current state: $ \rho_i(s) = \rho_i $ This only sets a fixed reward–cost exchange rate for each agent, rather than deciding who should take responsibility in a specific state. The gains in CityLearn likely come from fixing permanent structural imbalances (e.g., buildings that always overcool). It is unclear how state-independent shares can help identical agents that need to take turns bearing costs across different states.

  3. Theorem 2 lacks novelty and mismatches the actual implementation. Splitting the meta-gradient into direct-unrolling and score-function terms is standard in meta-RL, and the leave-one-out baseline is a standard variance-reduction trick. Neither is specific to responsibility allocation. More importantly, Section 3.4 and Appendix C.2 show that the actual PPO implementation uses a "stopped-batch" approximation—fixing behavior log-probabilities, normalized advantages, critic targets, and the dual multiplier during lookahead updates. Therefore, the implemented gradient is only a rough approximation, not the exact gradient claimed in Theorem 2.

  4. High computational cost and unfair sample comparison. In each meta-step, LiRA saves a checkpoint, runs multiple lookahead branches of PPO updates, computes meta-gradients, and then throws away all lookahead trajectories and policy updates. This is very expensive just to tune a few static parameters, and simpler gradient-free search methods are not compared. Also, soft Lagrangian penalties cannot guarantee hard safety during training or deployment, unlike CBFs, MPC, or safety shielding when models are available.

Questions for authors
  1. How does LiRA behave in non-monotone games with multiple local equilibria? Can small responsibility updates trigger equilibrium switching during lookahead and make the meta-gradient noisy or misleading?

  2. In a homogeneous environment requiring agents to alternate responsibilities dynamically, would the optimal static shares become uniform? If not, what does LiRA offer beyond fixed symmetry breaking?

  3. Under the stopped-batch PPO approximation, how much does the sampling-correction term improve performance? Please compare the full estimator with direct unrolling alone and with gradient-free search or bandit methods over the responsibility simplex.

  4. In Table 1 and Appendix D.1, are the comparisons matched by total environment interactions (including all discarded MqMq lookahead trajectories and terminal rollouts per cycle) or only by committed live-learner updates (hh)? Please clarify the exact sample budgets across methods and report wall-clock training costs, since matching only live updates would give LiRA a substantially larger sampling budget.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 4UeD (1/6)

We thank the reviewer for the detailed review. Following the comments, we have added both theoretical results and experiments. For the theory, we give a local version of Theorem 1 that does not need strong monotonicity (Proposition 2, Appendix E.1), and we extend our results to state-dependent shares (Corollary 1, Appendix E.2). For the experiments, we added a game in which either agent can yield (Appendix F.2), the homogeneous turn-taking game that you describe (Appendix F.3), an ablation of the sampling correction on the four benchmarks (Table 1), a comparison with gradient-free search (Appendix F.4), a comparison at equal environment interactions (Sec. 4.4 and Appendix H), and the exact budgets and wall-clock times (Appendix G). We answer each point below.


Weakness 1: Theorem 1 does not apply to deep MARL. … strong monotonicity rules out multiple equilibria, which are common in coordination tasks (e.g., whether agent A yields to B or B yields to A). …

Question 1: How does LiRA behave in non-monotone games with multiple local equilibria? Can small responsibility updates trigger equilibrium switching during lookahead and make the meta-gradient noisy or misleading?

We agree that Theorem 1 gives no guarantee for deep MARL and that strong monotonicity rules out multiple equilibria. We do not claim such a guarantee (Sec. 6 of the submission, and now also Sec. 3.2), and we have added a local version of the theorem that does not need monotonicity.

Proposition 2 (Appendix E.1, informal). Let z0z_0 be a regular rest point of LiRA at the shares ψ0\psi_0, that is, a Karush–Kuhn–Tucker point of the penalized game with strict complementarity and a nonsingular Jacobian. (a) For all shares ψ\psi near ψ0\psi_0 there is exactly one rest point z∗(ψ)z^{\ast}(\psi) near z0z_0, and it varies smoothly with ψ\psi. (b) On this branch every active shared cost stays at its budget, Ck(θ∗(ψ))=dkC_k(\theta^{\ast}(\psi))=d_k. (c) The branches through two distinct regular rest points do not meet, and a branch can end only where regularity fails, for example where an active multiplier reaches zero or an inactive constraint becomes active. Neither strong monotonicity nor a unique equilibrium is assumed.

In your example, "A yields to B" and "B yields to A" are two such rest points, and each lies on a smooth branch of its own. A small change in the shares moves the equilibrium along its branch, and the active cost stays at the budget as in Theorem 1. So the selection result of Theorem 1 holds locally around each equilibrium, without monotonicity.

We also empirically validated it in a two-agent game in which either agent can yield, so that training ends in one of two outcomes, and one of them has higher welfare (Appendix F.2). Over 400 runs, LiRA ends in the better outcome more often than Uniform and the per-agent Lagrangian (PAL). It leaves the better outcome less often and escapes the worse one more often (Rebuttal Table 1).

Rebuttal Table 1. From Table 7 in Appendix F.2. Outcomes of the two-outcome game over 400 runs per method. H is the outcome with the higher welfare and L the other one. The first three columns give the fraction of runs that end in H, over all runs, over the runs that start in the basin of H, and over those that start in the basin of L. Welfare is the mean ± 95% confidence interval. The better value in each column is in bold.

MethodEnds in HEnds in H, from an H startEnds in H, from an L startWelfare
Uniform / PAL0.7180.9590.0650.922 ± 0.018
LiRA0.7770.9860.2130.946 ± 0.016

For Question 1, a small update of the shares can make a lookahead end in the other outcome, but this happens almost only when training is close to the boundary between the two basins. The meta-gradient stays informative there. The agents respond most strongly to the shares near the boundary, and within 0.25 logits of it a single gradient estimate has the sign of the true gradient 94% of the time.

AuthorsOpenReview ↗

Response to Reviewer 4UeD (2/6)

Weakness 2: Responsibility shares are static across states. … It is unclear how state-independent shares can help identical agents that need to take turns bearing costs across different states.

Question 2: In a homogeneous environment requiring agents to alternate responsibilities dynamically, would the optimal static shares become uniform? If not, what does LiRA offer beyond fixed symmetry breaking?

Yes, in such an environment the optimal static shares are uniform, and static shares cannot shift the responsibility with the state. In Sec. 6 of the submission we name shares ρ(s)\rho(s) that depend on the state as the extension for this case. We have now carried it out in theory, in the game that you describe and on the benchmarks.

On the theory side, the shares ρ(s)\rho(s) are learned with the same estimator, and the budget property, the local selection result and the unbiased gradient carry over.

Corollary 1 (Appendix E.2, informal). Let the shares ρik(s)\rho_{ik}(s) depend on the state. (a) Proposition 2 holds with these shares. (b) At every rest point, a shared cost with a positive multiplier is at its budget. (c) Under the conditions of Theorem 2 and for an exact differentiable lookahead, the estimator of Eq. (10) is still an unbiased estimate of the gradient of the lookahead welfare.

In the turn-taking game that you describe, two identical agents share a resource, one of them needs it more in each state, and its use by the other agent also hurts the one that needs it (Appendix F.3). LiRA with ρ(s)\rho(s) learns to shift the responsibility with the state and closes 93% of the gap between uniform shares and the optimum at the same cost (Rebuttal Table 2).

Rebuttal Table 2. From Table 8 in Appendix F.3, the variant with an externality. Welfare and the share ρn\rho_n of the agent that needs the resource less (mean ± 95% confidence interval over 10 seeds). The shared cost is at the budget for all methods. The highest welfare among the methods is in bold.

MethodWelfareShare ρn\rho_n
Uniform0.4506 ± 0.00070.500
PAL0.4506 ± 0.00070.500
LiRA with ρ(s)\rho(s)0.4899 ± 0.00030.889 ± 0.006
Optimum0.49250.900

At benchmark scale, LiRA with state-dependent shares has the highest mean welfare within budget at equal interactions in MABIM and CityLearn (Rebuttal Table 6, in our answer to Weakness 4). So LiRA offers more than fixed symmetry breaking, since its shares can shift the responsibility with the state. We describe this extension at the end of Sec. 3.4.

AuthorsOpenReview ↗

Response to Reviewer 4UeD (3/6)

Weakness 3: Theorem 2 lacks novelty and mismatches the actual implementation. … the actual PPO implementation uses a "stopped-batch" approximation …

We agree that the split of the meta-gradient into a direct term (direct unrolling, DU) and a score-function term (the sampling correction, SC) is standard in meta-reinforcement learning, and so is the leave-one-out baseline. We do not claim these tools as our contribution. The submission already describes our gradient as meta-gradient differentiation combined with the sampling correction of Vuorio et al. (2022) (Sec. 2). We now say this in Sec. 3.3 as well and have added further references (Al-Shedivat et al., 2018; Stadie et al., 2018; Mnih & Rezende, 2016; Kool et al., 2019).

Our contribution is the problem that we apply these tools to. In meta-reinforcement learning these tools are used to adapt a learner's own policy, and in opponent shaping they are used so that one agent can steer the others. We use them to differentiate the social welfare that all agents reach after learning, with respect to how the penalty of a shared constraint is divided among them. Since the rewards, costs and budgets are left unchanged, the only thing that changes is the division of the existing penalty, and by Theorem 1 this division selects which equilibrium the agents play, with the active costs held at their budgets.

On the mismatch with the implementation, Theorem 2 gives the exact gradient, and our implementation uses the stopped-batch approximation of it, as stated in Sec. 3.4 and Appendix C.2 of the submission. To see how much this approximation costs, we compared the two in a three-agent game in which the true gradient can be computed (Appendix F.1), with the lookahead length and number of continuations of our CityLearn runs (q=2q=2, M=8M=8). A learner that follows the stopped-batch estimate closes 90% of the welfare gap, against 95% for the full estimator (Rebuttal Table 3). The exact gradient cannot be computed on the benchmarks, so this comparison is only possible in a controlled game. All results of Table 1 are obtained with the stopped-batch estimate, and so is the ablation of SC in our answer to Question 3.

Rebuttal Table 3. Gradient estimators in the scalar game of Appendix F.1. The game with an externality gives similar numbers (Table 6). The welfare gap is the gap between uniform shares and the optimum, and a negative value means that the learner ends below uniform shares. The best value in each column is in bold.

EstimatorRelative biasWelfare gap closed
Full (DU + SC)0.0295%
DU only0.1397%
SC only1.14−980%
Stopped batch, as in our implementation0.1690%

You list four quantities that are fixed during the lookahead updates. Their effects are as follows.

  • Critic targets. They are the rewards and costs of the batch itself and do not depend on the shares, so fixing them causes no error.
  • Normalized advantages. The normalization statistics do not depend on the shares, so fixing them causes no error either.
  • Behavior log-probabilities. Fixing them is the stopped-batch approximation. It gives the relative bias of 0.16 in the last row of Rebuttal Table 3.
  • Dual multiplier. The lookahead of the controlled game fixes the multiplier in the same way, while the learner keeps updating it, so its effect is already included in the 90%.

References

  • Vuorio, Beck, Whiteson, Foerster and Farquhar. An Investigation of the Bias-Variance Tradeoff in Meta-Gradients. arXiv:2209.11303, 2022.
  • Al-Shedivat, Bansal, Burda, Sutskever, Mordatch and Abbeel. Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments. ICLR, 2018.
  • Stadie, Yang, Houthooft, Chen, Duan, Wu, Abbeel and Sutskever. Some Considerations on Learning to Explore via Meta-Reinforcement Learning. arXiv:1803.01118, 2018.
  • Mnih and Rezende. Variational Inference for Monte Carlo Objectives. ICML, 2016.
  • Kool, van Hoof and Welling. Buy 4 REINFORCE Samples, Get a Baseline for Free! ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction, 2019.
Reviewer_LNoUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LiRA for multi-agent reinforcement learning (MARL) with shared constraints and rewards. A common Lagrange multiplier controls the overall penalty, while learned responsibility determine how strongly each agent responds to that penalty. The allocation is optimized for social welfare by differentiating through short continuations of the agents’ learning process. Under convexity and regularity assumptions, the paper characterizes how allocations can select equilibria with different welfare at the same active constraint budgets. Experiments on CityLearn, MABIM, Harvest, and MetaDrive report higher mean welfare than uniform allocation and per-agent multiplier baselines.

Critical strengths
  • The paper identifies an interesting distinction between enforcing a shared constraint and deciding who should bear its cost. A common budget does not determine which agents should adjust their behavior, particularly when their opportunity costs differ. The responsibility parameterization makes this choice explicit. The equilibrium analysis and exact examples provide a useful explanation of how welfare can change while active shared costs remain fixed.

  • The paper connects its formulation to an implementable learning procedure and evaluates rewards alongside costs. The gradient derivation accounts for both differentiation through learner updates and changes in the data collected by intermediate policies. The use of independent continuations and fresh terminal rollouts is clearly explained. The evaluation spans four different domains, and the CityLearn building-level analysis helps explain how the learned allocation changes behavior. I also appreciate that the paper explicitly reports residual constraint violations in MABIM and distinguishes Harvest’s use of unused budget from improvements at approximately fixed cost.

Critical weaknesses
  • The paragraph in the related works on “Learning through the agents’ response” omits LOLA: https://arxiv.org/abs/1709.04326, The Challenge of Hidden Gifts https://arxiv.org/abs/2505.20579, and Advantage Alignment Algorithms: https://arxiv.org/abs/2406.14662. These papers are all related to learning through another agent’s response and should be discussed there to clarify how LiRA differs to this literature and what its specific learning through other agents is.
  • The benefit of the sampling correction and the quality of the practical lookahead approximation are not empirically established. Theorem 2 provides an exact gradient decomposition under its assumptions, but Appendix C.2 explicitly uses a stopped-batch approximation and notes that the sampling correction does not recover omitted derivatives. The CityLearn probe also uses one functional PPO update per batch and a frozen multiplier, whereas live training uses multiple PPO epochs and subsequent dual updates. These choices may be reasonable in practice, but a direct-unrolling-only ablation and a controlled gradient check are needed to show what the correction contributes and whether the proxy reliably guides a learning agent.
Questions for authors
  • Does LiRA retain its advantage when Uniform, PAL, and a simpler allocation-search method receive the same total environment-interaction and computation budget, including the probe batches and terminal rollouts?
  • What happens when the sampling-correction term is removed?
  • Can the authors compare direct unrolling alone with the full estimator and validate the estimated gradient against finite differences or an exact calculation in a small controlled problem?
  • Appendix C.1 describes separately standardized per-step rewards and costs as the advantages used in CityLearn. How does this normalization affect the interpretation of NλρiN\lambda\rho_i as an agent’s share of the penalty in native reward and cost units?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LNoU (1/4)

We thank the reviewer for the careful review. Following the comments, we have added the related work on opponent shaping (Sec. 2), a gradient check in a controlled game (Appendix F.1), a direct-unrolling-only ablation on the four benchmarks (Table 1) and a comparison at equal environment interactions (Sec. 4.4), and we answer each point below.


Weakness 1: The paragraph in the related works on "Learning through the agents' response" omits LOLA …, The Challenge of Hidden Gifts …, and Advantage Alignment Algorithms …

Thanks for pointing us to these papers. We have added LOLA (Foerster et al., 2018), Hidden Gifts (Malenfant & Richards, 2026) and Advantage Alignment (Duque et al., 2025) to the paragraph "Learning through the agents' response" in Sec. 2.

LiRA differs from these methods in who takes the learning-aware step and in what that step can change. In opponent shaping, one of the agents anticipates how the others will learn and changes its own policy or update to steer them, toward its own return or a collective one. Hidden Gifts uses a learning-aware correction so that agents receive credit for prosocial actions that the others cannot observe. In LiRA no agent shapes another. The step is taken by an allocator outside the game, which differentiates social welfare through the updates of all agents and can change only how the penalty of an existing shared constraint is divided. The allocator leaves the rewards, the costs and the budget as they are.

References

  • Foerster, Chen, Al-Shedivat, Whiteson, Abbeel and Mordatch. Learning with Opponent-Learning Awareness. AAMAS, 2018.
  • Malenfant and Richards. The Challenge of Hidden Gifts in Multi-Agent Reinforcement Learning. Reinforcement Learning Journal, 2026.
  • Duque, Aghajohari, Cooijmans, Ciuca, Zhang, Gidel and Courville. Advantage Alignment Algorithms. ICLR, 2025.
AuthorsOpenReview ↗

Response to Reviewer LNoU (2/4)

Weakness 2: The benefit of the sampling correction and the quality of the practical lookahead approximation are not empirically established. … a direct-unrolling-only ablation and a controlled gradient check are needed …

Question 2: What happens when the sampling-correction term is removed?

Question 3: Can the authors compare direct unrolling alone with the full estimator and validate the estimated gradient against finite differences or an exact calculation in a small controlled problem?

We have added both experiments in the new Appendix F.1. They use a small game in which the true gradient can be computed by finite differences. With the lookahead length and number of continuations of our CityLearn runs (q=2q=2, M=8M=8), we compared the full estimator and direct unrolling (DU) alone with the true gradient, and we trained with each estimate.

Rebuttal Table 1. Gradient estimators in the scalar game of Appendix F.1. The game with an externality gives similar numbers (Table 6). The welfare gap is the gap between uniform shares and the optimum, and a negative value means that the learner ends below uniform shares. The best value in each column is in bold.

EstimatorRelative biasWelfare gap closed
Full (DU + SC)0.0295%
DU only0.1397%
SC only1.14−980%
Stopped batch, as in our implementation0.1690%

The full estimator is unbiased, as stated in Sec. 3.4, and the 0.02 relative bias in the table is Monte Carlo error. The sampling correction (SC) is the term that makes the full estimator unbiased. When SC is removed the estimate has a relative bias of 0.13, almost all of it in the magnitude. The DU only estimator still has a cosine of at least 0.99 with the true gradient, and in this small game the learner reaches about the same welfare. The stopped-batch approximation of Appendix C.2 holds the behavior log-probabilities of a sampled batch fixed, and a learner that follows it still closes 90% of the welfare gap.

DU only does well here because the game has a single step and no state, so that leaving out SC rescales the gradient and hardly rotates it. In the benchmarks the allocation also changes which states the agents visit. We therefore removed the SC term on the four benchmarks to see its effect there (new last column of Table 1).

Rebuttal Table 2. Welfare of LiRA with and without the sampling correction (SC) on the benchmarks, with the protocol and seeds of Table 1 (mean ± s.d. over seeds). The LiRA column repeats Table 1, and the higher mean is in bold.

TaskLiRALiRA without SC
CityLearn (10310^3)−14.51 ± 1.66−16.19 ± 6.10
MABIM (10610^6)−591.36 ± 4.48−588.85 ± 9.51
Harvest50.27 ± 2.5345.93 ± 1.94
MetaDrive64.26 ± 19.9857.75 ± 34.97

Without SC the mean welfare is lower in CityLearn, Harvest and MetaDrive, and the mean cost is over the budget in CityLearn and MetaDrive, where LiRA meets it. With SC the standard deviation of welfare over seeds is also smaller in CityLearn, MABIM and MetaDrive, by a factor of 1.8 to 3.7. In MABIM the two means differ by less than this standard deviation. We therefore keep SC in the experiments of Table 1.

AuthorsOpenReview ↗

Response to Reviewer LNoU (3/4)

In Weakness 2 you also point out two differences between the CityLearn probe and live training. The probe keeps the multiplier frozen, and it makes one PPO update per batch where live training runs several epochs.

The lookahead of the controlled game freezes the multiplier in the same way, while the learner that follows each estimate keeps updating it. The welfare column of Rebuttal Table 1 therefore already includes the effect of freezing the multiplier.

The difference in epochs is absent from the new experiments of Sec. 4.4 in three of the four tasks. There LiRA differentiates through the PPO updates that training actually made, with all their epochs and with the multiplier that was used at each update (Appendix H.1). To verify this gradient, we replayed these updates. The replay reproduces the trained parameters to floating-point precision, and the gradient agrees with a finite difference to a relative error of at most 1.2×10−41.2\times10^{-4}.


Question 1: Does LiRA retain its advantage when Uniform, PAL, and a simpler allocation-search method receive the same total environment-interaction and computation budget, including the probe batches and terminal rollouts?

Yes, LiRA retains its advantage when all methods collect the same number of interactions. It then has the highest mean welfare within budget in all four tasks, against Uniform, PAL, and two allocation-search methods (successive halving and random search), and this comparison is in Sec. 4.4. Random search (Bergstra & Bengio, 2012) draws candidate shares at random, trains each of them for the same number of interactions and keeps the one with the highest training welfare. Successive halving (Jamieson & Talwalkar, 2016) trains all candidates for a short round, discards the worse half and repeats until one is left. For an equal comparison in MetaDrive, MABIM and CityLearn, LiRA runs the lookahead on the live training batches, as in our answer to Weakness 2, and so collects no more data than Uniform. In Harvest we gave the other methods as many interactions as LiRA uses with its lookaheads.

Rebuttal Table 3. Table 4 in Sec. 4.4. LiRA's welfare gain over each method at equal environment interactions. MABIM welfare is in units of 10610^6. Successive halving and random search try several fixed shares within the same number of interactions. †^\dagger marks entries where the compared method is over budget.

Taskvs. Uniformvs. PALvs. successive halvingvs. random search
CityLearn+520+520+3,775$^\dagger$+6,337
MABIM+128.9$^\dagger$+128.8$^\dagger$+186.0$^\dagger$+184.2$^\dagger$
Harvest−0.6$^\dagger$+1.1+2.3+0.6
MetaDrive+19.0+8.8+12.1+14.0

LiRA uses more computation than the other methods at equal interactions, since it also differentiates through the last updates of training. A LiRA run takes 1.0–2.2× as long as a run of Uniform (Rebuttal Table 4).

Rebuttal Table 4. Table 10 in Appendix G. Training time of one run at equal environment interactions. In MABIM and CityLearn each allocation update averages 8 and 10 replays of the stored updates, which are run in parallel with one worker process per replay. The CityLearn times are per training cycle.

TaskUniformLiRARatio
CityLearn0.95 s2.1 s2.2×
MABIM85 s185 s2.2×
Harvest1,499 s1,552 s1.0×
MetaDrive226 s268 s1.2×

You also ask about an equal computation budget. Uniform and PAL have no lookahead, so the only way to give them more computation is to train them for longer. We tried this in MABIM, where we trained Uniform and PAL for eight times as many steps, so that a run takes at least as long as a LiRA run. Their welfare did not improve. It is −603.7 for Uniform and −608.6 for PAL, against −457.4 for LiRA (in units of 10610^6).

References

  • Bergstra and Bengio. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 2012.
  • Jamieson and Talwalkar. Non-stochastic Best Arm Identification and Hyperparameter Optimization. AISTATS, 2016.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (with Summary of Revisions)

We thank the area chair and the three reviewers for their careful reviews. We apologize for the late response. We used the discussion period to run the experiments that the reviewers asked for and to extend the theory, and we wanted these to be complete before replying. The revised PDF is uploaded, with all new text highlighted in blue.

The revision adds three theoretical results (Proposition 2, Corollary 1 and Proposition 3), four controlled experiments in which the optimum or the true gradient is known (Appendix F), and new benchmark experiments on all four tasks (Table 1, Sec. 4.4, Appendices G and H). The reply under each review is self-contained and repeats the statements and tables that it uses, so this comment only summarizes the changes.


Summary of Revisions

  1. LiRA keeps its advantage at equal environment interactions (LNoU Question 1, 4UeD Weakness 4 and Question 4, qjSw Weakness 2 and Question 2). In a new comparison in which all methods collect the same number of interactions, LiRA has the highest mean welfare within budget in all four tasks, against Uniform, the per-agent Lagrangian (PAL), successive halving and random search (new Sec. 4.4 and Appendix H). The submission compares methods at the same number of committed live updates, and the new Appendix G gives the exact interaction budgets and training times of those runs.
  2. The gradient estimate is checked against the true gradient, and the sampling correction is ablated (LNoU Weakness 2 and Questions 2–3, 4UeD Weakness 3 and Question 3, qjSw Weakness 1 and Question 1). In a small game where the true gradient can be computed (Appendix F.1), the full estimator is unbiased, as stated in Sec. 3.4. A learner that follows the full estimator, direct unrolling (DU) alone or the stopped-batch approximation of our implementation closes 90–97% of the welfare gap between uniform shares and the optimum. On the four benchmarks, LiRA without the sampling correction (SC) has a lower mean welfare in three tasks and a larger standard deviation over seeds in three (new last column of Table 1).
  3. The selection result holds without monotonicity (4UeD Weakness 1 and Question 1, qjSw Weakness 3). Proposition 2 is a local version of Theorem 1 that needs neither strong monotonicity nor a unique equilibrium (Appendix E.1), and Proposition 3 relates LiRA's finite-horizon gradient to the gradient of equilibrium welfare (Appendix E.3). In a game in which training can end in either of two outcomes, LiRA reaches the better one in 78% of the runs, against 72% for Uniform and PAL (Appendix F.2).
  4. State-dependent shares for identical agents that take turns (4UeD Weakness 2 and Question 2). The budget property, the local selection result and the unbiased gradient extend to shares that depend on the state (Corollary 1, Appendix E.2). In a turn-taking game in which one agent's use of a shared resource also hurts the other, LiRA with state-dependent shares closes 93% of the gap between uniform shares and the optimum (Appendix F.3).
  5. Normalization, related work and safety (LNoU Weakness 1 and Question 4, 4UeD Weaknesses 3–4). We state how the CityLearn advantages are normalized (Appendix C.1), and we added a discussion of opponent shaping (Sec. 2), further references for the estimation tools (Sec. 3.3) and a note on hard safety (Sec. 6).

We would be glad to run further checks in the remaining days of the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.