acceptodds
Withdrawn from ICLR 2027

Auditing Update-Order Effects in Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Update order can change a multi-agent reinforcement learning (MARL) algorithm even when its objective, data, and optimizer are fixed. We formulate an audit of the complete learning state, including optimizer memory, and characterize when an observable state remains sufficient under common future updates. The audit groups orders that are equivalent under commuting swaps and compares pairs of training trajectories after correcting for evaluation noise, retaining unresolved cases under explicit coverage assumptions. In three controlled PettingZoo/MPE environments, simplified independent-critic actor-critic updates have zero observed protocol range, whereas sequential shared-critic updates have ranges of 5.11-5.95 return points and two sample-mean ranking reversals. A critic-only probe can miss actor differences: actors read different intermediate states even when the final critic states agree. Freezing shared state restores invariance under disjoint actor updates. In HARL-based experiments at 10M steps per branch, reversing the native loop also reassigns random streams; agent-local streams isolate order and yield heterogeneous sensitivity estimates. Projecting finite update differences onto a surrogate utility gradient ranks held-out surrogate sensitivity (mean within-seed Spearman 0.78), but return-based scheduling and detector transfer remain unestablished. The resulting audit supports reproducible comparisons, with distribution-dependent small-seed inference; it does not establish a return-improving optimizer.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, update order, noncommutativity, protocol equivalence, reproducibility, actor-critic

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rBziOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether changing the order of agent updates changes their learning results. It proposes an audit that controls training data and randomness, and compares learning states, policies, and returns. The experiments show that order can matter when agents use a shared critic that changes between updates. Keeping the critic fixed during actor updates can remove this dependence under the stated conditions.

Critical strengths
  1. This paper addresses a reproducibility issue. The paper shows how update order can affect learning when agents use changing shared state, highlighting an implementation detail that should be specified in MARL comparisons.

  2. The audit design and controlled experiments is careful. It tracks optimizer states, controls random inputs, and distinguishes changes in parameters, policies, and returns. These controls help identify genuine order effects, while the paper clearly acknowledges unresolved findings.

Critical weaknesses
  1. The contribution is limited. When one update changes shared information used by another update, changing their order can naturally change the result. The paper should explain more clearly what substantial new insight its audit provides and when it is preferable to simply letting all agents use the same fixed critic.

  2. The evidence from standard algorithms is limited. Most supporting experiments use simplified training methods. The HAPPO results involve few seeds and substantial variation. More evidence is needed to show that update order regularly causes meaningful performance differences or changes algorithm comparisons.

  3. The statistical conclusions need stronger support. The paper’s own tests show that its confidence intervals can be unreliable with few seeds and large differences between runs. This raises concerns about the reliability of the audit’s classifications.

  4. The practical value does not clearly justify the cost. Comparing update orders requires additional training and evaluation, but the method does not reliably identify a better order or demonstrate a clear scheduling benefit. Although improved return is not required for an auditing paper, the authors should report the complete overhead and show when the audit provides useful findings worth that expense.

Questions for authors

Please address my concerns raised in "Critical Weaknesses".

Reviewer_Bmv2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how update order affects training outcomes in multi-agent reinforcement learning and proposes an auditing framework incorporating the full training state, control over randomness, and statistical testing. However, the practical necessity and representativeness of the update procedure used in the core controlled experiments are insufficiently justified.

Critical strengths
  1. The paper highlights implementation details that are easily overlooked. Update order, the timing of shared-state reads, optimizer states, and the allocation of random draws can all affect algorithm behavior and deserve inclusion in reproducibility specifications.
  2. Negative results and limitations are disclosed relatively thoroughly. The authors acknowledge that selecting update orders has not yielded consistent gains, that the detector predicts sensitivity of the surrogate objective, and that transfer across implementations remains unsupported.
Critical weaknesses
  1. The core controlled setting lacks sufficient practical motivation, which is the main concern. S-AC updates the shared critic immediately after each actor update, allowing subsequent actors to read the updated critic and recompute their advantages. This explicitly introduces order dependence. The paper does not adequately explain which practical tasks or algorithms require this specific procedure, or why it is necessary or beneficial compared with a unified update procedure.
  2. The connection to familiar training procedures is insufficiently established. A procedure that pools rollouts, precomputes and freezes advantages, and updates a shared actor using the pooled data does not involve agents alternately reading the critic in this way. Even when actor parameters are not shared, sequential execution can produce identical results across update orders, provided that each actor’s update inputs are fixed and its state is independent. Therefore, updating agents separately does not itself imply order sensitivity; the key factor is the additional read/write dependencies.
  3. The controlled experiments have limited generalizability. They clearly demonstrate that the constructed alternating-update mechanism can produce order effects, but do not establish that conventional MARL training generally faces the same issue. The simplified I-AC/S-AC implementations are also not standard IPPO/MAPPO, so the conclusions should be restricted to the specific implementations studied.
Questions for authors

Overall, I have substantial concerns about the paper’s motivation, particularly whether the update procedure used in the controlled experiments is necessary in practice and whether the findings generalize to broader MARL algorithms. I currently lean toward rejection, but I welcome a discussion with the authors about these motivational concerns and would consider raising my score if they can satisfactorily address them.

Reviewer_ZFTHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether changing the update order of agents will change the training results of MARL. The author describes the update process in mathematical language and proposes an audit method to compare different update sequences. The experiment covers the simplified learning method and the existing MARL method. The author examines how the model part shared by agents affects the results, and also studies the way in which random numbers are allocated to each agent. In addition, some experiments try to change the update process to test whether it can reduce the impact of the update sequence without damaging performance.

Critical strengths

The author formalizes the problem and explains how to study it through experiments. One useful detail is the handling of random numbers. After changing the update order, even if the same random seed is used, each agent may not get the same random input. The paper considers this when comparing different orders. The author also explains why checking only one part of the model may omit changes in other parts. Readers can check these specific problems in their own experiments, so this work has certain practical value.

Critical weaknesses

My main concern is that the theoretical part only gives limited answers under strong assumptions. It does not explain how the differences caused by an update will affect the final training results, so it is difficult to fully help us understand the situation in actual MARL training. I also have doubts about the audit methods. Only five training random seeds were used in some main experiments, and the test of the paper itself also showed that the performance of this method in small samples was worrying. The author discussed this problem, but this method still needs more sufficient evidence to support its practical use, and its advantages are not clear enough. In general, I think the conclusion of the paper is not complete, nor is it convincing that this work has brought enough new understanding.

Questions for authors

Please see the cons part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.