- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether changing the order of agent updates changes their learning results. It proposes an audit that controls training data and randomness, and compares learning states, policies, and returns. The experiments show that order can matter when agents use a shared critic that changes between updates. Keeping the critic fixed during actor updates can remove this dependence under the stated conditions.
Critical strengths
-
This paper addresses a reproducibility issue. The paper shows how update order can affect learning when agents use changing shared state, highlighting an implementation detail that should be specified in MARL comparisons.
-
The audit design and controlled experiments is careful. It tracks optimizer states, controls random inputs, and distinguishes changes in parameters, policies, and returns. These controls help identify genuine order effects, while the paper clearly acknowledges unresolved findings.
Critical weaknesses
-
The contribution is limited. When one update changes shared information used by another update, changing their order can naturally change the result. The paper should explain more clearly what substantial new insight its audit provides and when it is preferable to simply letting all agents use the same fixed critic.
-
The evidence from standard algorithms is limited. Most supporting experiments use simplified training methods. The HAPPO results involve few seeds and substantial variation. More evidence is needed to show that update order regularly causes meaningful performance differences or changes algorithm comparisons.
-
The statistical conclusions need stronger support. The paper’s own tests show that its confidence intervals can be unreliable with few seeds and large differences between runs. This raises concerns about the reliability of the audit’s classifications.
-
The practical value does not clearly justify the cost. Comparing update orders requires additional training and evaluation, but the method does not reliably identify a better order or demonstrate a clear scheduling benefit. Although improved return is not required for an auditing paper, the authors should report the complete overhead and show when the audit provides useful findings worth that expense.
Questions for authors
Please address my concerns raised in "Critical Weaknesses".