- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper frames decentralized MARL as an agent-centric continual learning problem: peer updates change each agent's effective environment. It defines an invariant core of high-coverage patterns in successful trajectories and relates structural persistence to peer-policy drift, deriving inverse-update-rate lifetime guarantees under explicit conditions. Additional success-mass and execution-calibration assumptions connect structural survival to policy value and transfer guarantees. Corridor experiments test the theory, while continual-control and cue-MNIST studies show that coverage-based monitoring can anticipate failure and guide intervention. An exploratory Level-Based Foraging analysis suggests a similar erosion-failure relationship under learning partners.
Critical strengths
The paper provides a comprehensive theoretical analysis and is clearly organized. The experiments closely match the main claims and provide convincing support for the theoretical results and their practical usefulness.
Critical weaknesses
The LBF results come from a small exploratory study, leaving the generality of the findings in broader MARL settings unclear.
Questions for authors
-
How does the monitor distinguish harmful coverage erosion from a shift toward a different but equally successful strategy?
-
How should the monitored patterns be updated when old patterns become obsolete and new successful patterns emerge?
-
Could the LBF results be confirmed on new partner pairings, with the erosion score and window fixed in advance?
4 comments
Rebuttal revision
We have incorporated the experimental results from your question into the paper.
Thank you again for the feedback. Please let us know if there is anything we can address, clarify or elaborated that might lead you to augment your score.
We wanted to mention that we addressed your concern about broader MARL applicability, we conducted additional experiments in Overcooked Forced Coordination and Multi-Agent Walker2d for another reviewer (see the results here).
Building on your fixed-score, fixed-window experiment, we trained four new Overcooked teams, 12-checkpoint lookback, and three-checkpoint forecast horizon. Two changed-partner continuations fell below 70% cooking success, versus none of four own-partner controls. The prespecified erosion score achieved mean AP 0.794 on the two failing teams.
We also extended the fixed-window prediction design to Multi-Agent Walker2d, selecting a three-checkpoint median-slope score on native-training runs and applying it unchanged under controlled gradual partner change. The score measures whether a fixed gait sequence completes later in successful walks and achieved pooled AP 0.317, compared with 0.192 for the earlier fitted-slope score. Across 41 eligible checkpoints, four preceded one of two incidents in which old-partner walking success fell below 70% within the next three checkpoints.
Thank you again for this experiment. We included these results in Appendix F.9 in another revision. Please don’t hesitate if there is anything else we could address.
Thank you for the response. These clarifications have addressed most of my concerns, and I have decided to maintain my original score.
Thank you for volunteering your time to review and asking for an interesting experiment. Your three questions below.
1. How does the monitor distinguish harmful erosion from a shift toward an equally successful strategy?
The monitor distinguishes harmful erosion from a shift toward an equally successful strategy by checking whether the loss of core-compatible success mass implies a decrease in expected return. Under the paper’s performance-calibration assumptions, harm is certified when the resulting return loss exceeds the calibration error (Theorem 3.3; Corollary E.1).
2. How should patterns be updated when old patterns become obsolete?
Patterns should be updated after discovering new candidates from recent successful trajectories. These are then retained after passing the coverage threshold on independent probe rollouts. Each accepted pattern is monitored from the current peer agent’s checkpoint as a new reference. Its survival bound then uses the new reference’s success mass, coverage margin, and subsequent peer-policy drift.
3. Can the LBF result be confirmed on new partner pairings with the score and window fixed in advance?
20 new independent E11 pairings were ran and confirmed with the erosion score and three-checkpoint window held fixed in advance.
Core erosion achieved mean pairing-level AP 0.894, compared with 0.719 for return slope with a paired advantage of 0.175 and 95% confidence interval of [0.115, 0.234].
This was an interesting experiment to test and will be incorporated into the paper.
This paper relates to broader MARL through its analysis of the focal agent’s induced MDP. Under the stated assumptions, other learning agents in a stationary Markov game generate a sequence of MDPs for that focal agent. The structural-survival bounds can apply to other games if their drift, success mass, and coverage margins satisfy the theorem’s conditions.
The LBF experiments are examples that core erosion can predict failure of a fixed focal policy as its partner learns.