- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates why wider networks hurt HAPPO in several SMACv2 settings but not shared MAPPO. It varies update order, parameter sharing, and data per update, then measures policy KL and tests learning-rate and batch interventions. Additional environments and reverse interventions examine the limits of the update-scale explanation.
Critical strengths
-
The topology controls answer a concrete question. The four-way comparison and no-prefix control rule out sequential updating, parameter sharing, and the cumulative ratio as sufficient explanations on their own. The actor/critic width split further localizes the capacity effect (Sections 3 and 4).
-
The accumulated-batch experiment is a useful control. Collecting five segments with a frozen policy and critic reproduces the larger-batch result without adding rollout threads. Together with the phase-count control, this strengthens the evidence for the per-network update regime (Appendix D and Table 7).
-
Several independent checks support a qualified mechanism. Exact KL, learning-rate changes, KL stopping, and the HATRPO check provide more than an observational correlation. The paper also reports the failed de-pooling threshold and MPE performance-rescue prediction, rather than claiming a universal KL rule (Appendices D, E, and H).
Critical weaknesses
-
The learning-rate intervention needs a precise specification. The default actor and critic rates are equal, but it is unclear whether both change in the rescue experiments. The actor/critic width control does not resolve this. KL stopping provides direct but partial support, so the role of critic optimization in the learning-rate rescue remains uncertain (Section 5 and Appendix A).
-
Cross-condition scores use different training windows. Last-ten means span five times more environment steps in dose-matched runs than in standard runs. This does not confound h64 versus h256 within a condition, but it complicates comparisons of gap reduction across conditions. Table 9 does not cover the dose and accumulated-batch controls (Appendices A and F).
-
Table 8 mixes reference cohorts. Its caption attributes the reference rows to the two-seed Table 6 runs, but the MAPPO, synchronous-independent, and HAPPO gaps are copied from Table 2 instead. For example, the instrumented MAPPO gap is -0.144, not -0.017. The distinction matters when comparing KL and performance; uncertainty is also substantial in several rescue conditions (Tables 6, 8, and 10).
Questions for authors
-
Were the learning-rate interventions actor-only or joint actor/critic changes? If joint, is an actor-only result available?
-
Do the dose and accumulated-batch conclusions hold when scores use the same final environment-step interval?
-
Please reconcile the reference cohorts in Table 8. Using matched-run KL and performance, which quantitative rescue claims remain supported by the reported uncertainty?