- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes LiRA, a method that learns agent-specific responsibility shares for shared constraints in multi-agent reinforcement learning. It provides theoretical analysis of equilibrium selection and a finite-horizon meta-gradient estimator. Experiments across four benchmarks show improved social welfare over Lagrangian baselines.
Critical strengths
-
The theoretical analysis provides a principled connection between responsibility allocation and normalized generalized Nash equilibrium selection.
-
Experiments across four diverse environments demonstrate consistent improvements in mean social welfare, and the CityLearn case study provides an interpretable explanation of the learned allocations.
-
The proposed parameterization is simple and can potentially be incorporated into existing Lagrangian MARL algorithms.
Critical weaknesses
-
The paper does not provide DU-only or SC-only ablations. Consequently, it is unclear whether the sampling correction is necessary or whether the improvements primarily come from learning responsibility shares.
-
LiRA requires multiple independent lookaheads and additional evaluation rollouts. Without comparisons under equal environment-interaction or computational budgets, its practical advantage over simpler Lagrangian methods remains unclear.
-
Theorem 1 characterizes equilibrium selection along active constraint surfaces, whereas the experiments involve finite-time nonconvex learning, with some constraints remaining violated or substantially inactive. It is unclear whether the observed improvements reflect equilibrium selection or merely transient learning effects.
Questions for authors
-
How much does the sampling-correction term contribute to performance? Can the authors provide DU-only and SC-only ablations, ideally together with gradient variance or directional accuracy measurements?
-
Does LiRA still outperform simpler adaptive responsibility-allocation methods when all approaches are evaluated under the same total environment-interaction budget?
5 comments
Response to Reviewer qjSw (2/3)
Weakness 2: LiRA requires multiple independent lookaheads and additional evaluation rollouts. Without comparisons under equal environment-interaction or computational budgets, its practical advantage over simpler Lagrangian methods remains unclear.
Question 2: Does LiRA still outperform simpler adaptive responsibility-allocation methods when all approaches are evaluated under the same total environment-interaction budget?
Yes, LiRA still outperforms them under the same total budget. With the same total number of environment interactions for every method, LiRA has the highest mean welfare within budget in all four tasks, against Uniform, the per-agent Lagrangian (PAL), successive halving and random search (new Sec. 4.4). Random search (Bergstra & Bengio, 2012) draws candidate shares at random, trains each of them for the same number of interactions and keeps the one with the highest training welfare. Successive halving (Jamieson & Talwalkar, 2016) trains all candidates for a short round, discards the worse half and repeats until one is left. For the new comparison in MetaDrive, MABIM and CityLearn, LiRA runs the lookahead on the live training batches. It differentiates through the last few PPO updates of training and so collects no more data than Uniform (Appendix H.1). In Harvest we gave the other methods as many interactions as LiRA uses with its lookaheads.
Rebuttal Table 3. Table 4 in Sec. 4.4. LiRA's welfare gain over each method at equal environment interactions. MABIM welfare is in units of . Successive halving and random search try several fixed shares within the same number of interactions. marks entries where the compared method is over budget.
| Task | vs. Uniform | vs. PAL | vs. successive halving | vs. random search |
|---|---|---|---|---|
| CityLearn | +520 | +520 | +3,775$^\dagger$ | +6,337 |
| MABIM | +128.9$^\dagger$ | +128.8$^\dagger$ | +186.0$^\dagger$ | +184.2$^\dagger$ |
| Harvest | −0.6$^\dagger$ | +1.1 | +2.3 | +0.6 |
| MetaDrive | +19.0 | +8.8 | +12.1 | +14.0 |
For computation, the lookahead on the live batches needs no extra data, and a LiRA run takes 1.0–2.2× as long as a run of Uniform (Rebuttal Table 4).
Rebuttal Table 4. Table 10 in Appendix G. Training time of one run at equal environment interactions. In MABIM and CityLearn each allocation update averages 8 and 10 replays of the stored updates, which are run in parallel with one worker process per replay. The CityLearn times are per training cycle.
| Task | Uniform | LiRA | Ratio |
|---|---|---|---|
| CityLearn | 0.95 s | 2.1 s | 2.2× |
| MABIM | 85 s | 185 s | 2.2× |
| Harvest | 1,499 s | 1,552 s | 1.0× |
| MetaDrive | 226 s | 268 s | 1.2× |
In MABIM we also trained Uniform and PAL for eight times as many steps, so that a run takes at least as long as a LiRA run. Their welfare did not improve. It is −603.7 for Uniform and −608.6 for PAL, against −457.4 for LiRA (in units of ).
References
- Bergstra and Bengio. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 2012.
- Jamieson and Talwalkar. Non-stochastic Best Arm Identification and Hyperparameter Optimization. AISTATS, 2016.
Response to Reviewer qjSw (3/3)
Weakness 3: Theorem 1 characterizes equilibrium selection along active constraint surfaces, whereas the experiments involve finite-time nonconvex learning … It is unclear whether the observed improvements reflect equilibrium selection or merely transient learning effects.
We agree that Theorem 1 is a statement about equilibria and cannot by itself settle this question for deep policy networks, as we say in Sec. 6 of the submission. We have added a theoretical result and further evidence.
The new theoretical result, Proposition 3, connects the two regimes.
Proposition 3 (Appendix E.3, informal). Let the expected learning map be a contraction with factor near its fixed point , let be the welfare at this fixed point, and let be the welfare after lookahead steps from the current parameters . Then
where the truncation term satisfies and the transient term satisfies .
In words, the finite-horizon gradient that LiRA follows equals the gradient of welfare at the learner's equilibrium, up to a truncation term that shrinks geometrically with the lookahead length and a transient term that vanishes as training approaches the equilibrium.
We also have evidence that the gain is not transient when the constraint is active. In a game with two outcomes of training, LiRA changes which of them training ends in (Rebuttal Table 5).
Rebuttal Table 5. From Table 7 in Appendix F.2. Outcomes of the two-outcome game over 400 runs per method. H is the outcome with the higher welfare and L the other one. The first three columns give the fraction of runs that end in H, over all runs, over the runs that start in the basin of H, and over those that start in the basin of L. Welfare is the mean ± 95% confidence interval. The better value in each column is in bold.
| Method | Ends in H | Ends in H, from an H start | Ends in H, from an L start | Welfare |
|---|---|---|---|---|
| Uniform / PAL | 0.718 | 0.959 | 0.065 | 0.922 ± 0.018 |
| LiRA | 0.777 | 0.986 | 0.213 | 0.946 ± 0.016 |
In a turn-taking game, LiRA with shares that depend on the state closes 93% of the gap to the optimum with the cost at the budget (Appendix F.3). On the benchmarks, we trained CityLearn for 800 cycles with grid import capped at 4 kWh and evaluated the second half of training. Uniform and LiRA both end at the budget there, and LiRA's welfare is higher (−6,000 against −6,520, Appendix H.2). In the comparison at equal interactions LiRA is within the budget in all four tasks, and its cost is close to the budget in CityLearn (0.299 against 0.30) and Harvest (47.90 against 48.2, Table 11 in Appendix H).
Thank you again for your time and for the concrete suggestions. We would be happy to answer any further questions.
Response to Reviewer qjSw (1/3)
We thank the reviewer for the review and for the concrete requests. We have added the DU-only ablation on the four benchmarks (Table 1), the DU-only and SC-only ablations in a controlled game with the bias of each estimate and the direction of DU only (Appendix F.1), a comparison at the same total environment interactions (Sec. 4.4) and a result on finite horizons (Appendix E.3).
We ran the DU-only ablation on the four benchmarks (new last column of Table 1), and both ablations in a three-agent game in which the true gradient can be computed (Appendix F.1).
Rebuttal Table 1. Welfare of LiRA with and without the sampling correction (SC) on the benchmarks, with the protocol and seeds of Table 1 (mean ± s.d. over seeds). The LiRA column repeats Table 1, and the higher mean is in bold.
Without SC the mean welfare is lower in CityLearn, Harvest and MetaDrive, and the mean cost is over the budget in CityLearn and MetaDrive, where LiRA meets it. With SC the standard deviation of welfare over seeds is also smaller in CityLearn, MABIM and MetaDrive, by a factor of 1.8 to 3.7. In MABIM the two means differ by less than this standard deviation. Both versions have a higher mean welfare than Uniform in all four tasks, so learning the responsibility shares with DU alone already helps. SC adds a further gain in three tasks, and in CityLearn and Harvest this is more than half of LiRA's gain over Uniform. For the gradient variance, we measured the share of the variance of the estimate that comes from the SC term. This share is 0.6% in CityLearn, 7% in MABIM, 4% in Harvest and 44% in MetaDrive.
The directional accuracy can only be measured where the true gradient is known. Rebuttal Table 2 gives the results in the controlled game, which uses the lookahead length and the number of continuations of our CityLearn runs (q=2, M=8).
Rebuttal Table 2. Gradient estimators in the scalar game of Appendix F.1. The game with an externality gives similar numbers (Table 6). The welfare gap is the gap between uniform shares and the optimum, and a negative value means that the learner ends below uniform shares. The best value in each column is in bold.
The sampling correction (SC) is the term that makes the estimate unbiased. The full estimator has a relative bias of 0.02, which is Monte Carlo error, and direct unrolling (DU) alone has a relative bias of 0.13. For the directional accuracy, the DU only estimator has a cosine of at least 0.99 with the true gradient, so its bias is almost all in the magnitude. In this small game a learner that follows either estimate closes about the same share of the welfare gap. SC alone does not give a useful direction, which is expected of a term that corrects DU and is not by itself an estimate of the gradient. For this reason we did not run SC alone on the benchmarks. SC also adds variance, and Figure 3b shows the mean squared error of the two estimators against the number of continuations.