- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MA-JEPA, which introduces JEPA into model-based multi-agent reinforcement learning. Each agent maintains a parameter-shared local stochastic world model. During training, a centralized joint predictor is introduced, which takes the synchronized local states of all agents together with the joint action to predict the embedding of each agent's next local observation. This prediction is then passed through the same local posterior as in real interactions to construct the actor's state, thereby eliminating the inconsistency in the state construction mechanism between training and execution under the CTDE framework. Policy learning adopts PPO in latent-space imagination, while the critic is trained with centralized states. Experiments are conducted on 6 SMAC v1 maps against DMAWM, MAPPO, QMIX, and MAT, but the proposed method does not demonstrate a clear advantage.
Critical strengths
- The problem the paper addresses is practically relevant.
- The method is presented clearly and completely enough that the model design is easy to follow.
Critical weaknesses
-
The abstract claims that the method “provides this learning signal in both single- and multi-agent reinforcement learning,” but the paper includes no single-agent experiments. The Introduction also mentions evaluation on DeepMind Control (DMC), but no such results appear in the paper.
-
In Figure 1, MA-JEPA does not stand out compared with the other methods. Also, why are MAMBA’s results missing from Table 2?
-
The experimental evaluation is rather thin: it is conducted only on SMAC, and more benchmarks are needed to assess the method’s effectiveness and generalization.
-
The ablation study is insufficient. There is only one ablation result (Figure 2), and removing other modules actually improves performance, which casts doubt on the module design.
Questions for authors
- Does “available training seeds” have any special meaning?
- Could you add a comparison that removes the joint predictor and relies only on local world-model imagination?
2 comments
Response to Reviewer MJVx (Part 2/2)
Interpretation and coverage of Figure 2 ablations
"The ablation study is insufficient. There is only one ablation result (Figure 2), and removing other modules actually improves performance, which casts doubt on the module design."
This is an incorrect characterization of our results and it is not clear what ablations the author has in mind. Claiming "ablations are insufficient" without suggesting what they actually want ablated is not particularly helpful in improving the paper. We also disagree that the ablations are insignificant as we vary the one free considerable architectural change. If you want us to perform ablations/work for work's sake, we can do that but we fail to see the point of this.
Figure 2 reports four configurations in a 2×2 ablation. We independently turn off two paths by which the joint model trains the local representations: gradients from predicting the next observation embedding and gradients from predicting reward, continuation, and whether each agent is alive. All four configurations retain the same modules and prediction losses. Turning off a gradient path helps on some maps and hurts on others. In the revised draft's paired-seed results across three maps, keeping both paths gives the highest average win rate (77.0%, compared with 73.9% for the best ablated variant), which is why we did so. So we disagree that ``removing other modules actually improves performance'' and we think this justifies our design decision.
Additionally, doing this before designing our model would have polluted our test results due to an "optimism bias". We tested this after the architecture design.
Meaning of “available training seeds”
"Does “available training seeds” have any special meaning?"
This was vague wording from our side. "Available training seeds" refers to seeds 1302, 2771, and 7636, the actual seeds we used to run the training. We will correct this in the camera-ready version.
Local-only imagination baseline
"Could you add a comparison that removes the joint predictor and relies only on local world-model imagination?"
In principle yes. In practice, we tried this already initially. The performance was quite bad. This makes sense as there is no way in which agents can assign credit/roles to other agents in this way as training is centralized. That is, all weights are identical. Also, observations are local and limited.
We are unsure the value this would bring other than showing that lacking the cross-attention leads to terrible performance. Prior papers (e.g., DMAWM) have already identified similar issues, though, so we do not necessarily think this is necessary.
Response to Reviewer MJVx (Part 1/2)
We thank the reviewer for their comments. We have uploaded a revised version of our paper, aiming to address the points raised in the rebuttal where we felt a paper revision was appropriate. See below for our full rebuttal response going through the reviewer's comments one by one, in two parts.
We believe we have rigorously and comprehensively addressed the reviewer's comments. If so, we would appreciate the reviewer revising their score appropriately.
Thank you for your help in improving the paper!
Missing single-agent and DMC results
"The abstract claims that the method “provides this learning signal in both single- and multi-agent reinforcement learning,” but the paper includes no single-agent experiments. The Introduction also mentions evaluation on DeepMind Control (DMC), but no such results appear in the paper."
You are right about the single-agent and DMC mention. We removed those mentions in the revised draft. The experiments reported here are on just the multi-agent SMAC v1 benchmark. Single-agent performance was strong, but we thought it appropriate to emphasize the multi-agent version as that was the major novel contribution. If the reviewer thinks it would strengthen the paper, we can re-add the single-agent experiments/DMC to the appendix for the camera-ready version.
Performance in Figure 1
"In Figure 1, MA-JEPA does not stand out compared with the other methods."
We disagree with how this response characterizes our performance, particularly in light of the expanded results in Table 1 of the revised draft. At matched environment budgets, MA-JEPA matches or achieves the highest mean win rate on four of eight SMAC maps and outperforms the strongest model-free baseline on seven. It outperforms DMAWM by 16 pp for the winrate on corridor (the hardest task) and 66p for the best model-free baseline. (It doesn't do so significantly for corridor at 1 sd, but this is because of the massive 44pp sd on DMAWM, which should be seen as an indicator of its poor performance rather than a virtue. Of the remainder, all four are ties up to statistical significance (at 1 sd). (We use NIST's Welch test, which is a finite-approximation variant of testing whether the means of our results as a normal distribution is larger than those of DMAWM. On the four remaining tests, where we match 3/4 of our 1 SD ranges contain the mean of DMAWM except for 3s_vs_5z. However, even there, NIST's Welch test reports a p-value of just 0.495.) These are primarily the easier problems where benchmarks are almost fully saturated.) On corridor, MA-JEPA reaches a 66.0% mean win rate at 400k steps, compared with 50.0% for DMAWM and 0% for all three model-free baselines. On 2c_vs_64zg, the corresponding rates at 200k steps are 78.3%, 75.3%, and 0%. The one exception is 3s_vs_4z, where DMAWM remains clearly ahead (95.7% vs. 74.0%), even after accounting for uncertainty. We discuss our understanding as to why this is the case in the paper, and we actually view this as an interesting positive result, rather than negative, as it indicates limitations of JEPA for specific problems (long-horizon tasks) that future work can aim to address. In particular, this result is consistent with other work that finds similar limitations for JEPA-based architectures more broadly in vision tasks (e.g., MotionJEPA: https://arxiv.org/pdf/2609.23881). Taken together, these results show that a world model trained to predict observation embeddings can support competitive decentralized policies on easier tasks and more performant policies on harder tasks, including on tasks where the model-free methods have yet to win at all at the same budget, as well as suggests potential useful avenues of improvement to explore for the JEPA architecture itself for multi-agent RL tasks.
Missing MAMBA results in the performance table
"Also, why are MAMBA’s results missing from Table 2?"
Thanks for pointing out this inconsistency. We've removed MAMBA from both the plots, and the performance table for the revised version. (For what it is worth, it was strictly dominated by DMAWM and our algorithm, MA-JEPA for all environments, so we saw little point in including it.)
Breadth of the benchmark suite
"The experimental evaluation is rather thin: it is conducted only on SMAC, and more benchmarks are needed to assess the method’s effectiveness and generalization."
SMACv1 is a standard classical RL benchmark. In principle, we agree with the reviewer's comment. We will add SMACv2 and Meltingpot benchmark results to the camera-ready version. Early results from this are promising and similar to performance on SMACv1, but were not feasible to fully run within the week of the rebuttals across multiple seeds.