acceptodds
Under review as a conference paper at ICLR 2027

Learning World Models with Joint-Embedding Prediction for Single- and Multi-Agent Reinforcement Learning

Abstract

World models improve sample efficiency by training policies on imagined trajectories, but their usefulness depends on learning representations that capture the information needed for future control. We study whether self-supervised joint-embedding prediction (JEPA) can provide this learning signal in both single- and multi-agent reinforcement learning. We introduce MA-JEPA a stochastic world model that replaces observation reconstruction with prediction of target representations, enabling model-based multi-agent reinforcement learning with centralized training and decentralized execution. A categorical latent state and a causal Transformer are trained with posterior, action-conditioned dynamics, and masked spatial prediction objectives and are then used for actor-critic learning from imagination in the latent space. For multi-agent control, we add a training-only joint predictor that conditions on joint local states and the actions to predict each agent's next local observation representation. These predictions are passed through the same local posterior used during real interaction with a centralized critic that is used only for value learning, with execution remaining decentralized. Our experiments show that this architecture performs strongly on SMAC, matching or exceeding the strongest reported comparator median on three of six evaluated maps.

Primary area:
reinforcement learning
Keywords:
jepa, world model, marl, multi-agent, physical-models, single-agent, reinforcement-learning, latent space, model based reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MJVxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MA-JEPA, which introduces JEPA into model-based multi-agent reinforcement learning. Each agent maintains a parameter-shared local stochastic world model. During training, a centralized joint predictor is introduced, which takes the synchronized local states of all agents together with the joint action to predict the embedding of each agent's next local observation. This prediction is then passed through the same local posterior as in real interactions to construct the actor's state, thereby eliminating the inconsistency in the state construction mechanism between training and execution under the CTDE framework. Policy learning adopts PPO in latent-space imagination, while the critic is trained with centralized states. Experiments are conducted on 6 SMAC v1 maps against DMAWM, MAPPO, QMIX, and MAT, but the proposed method does not demonstrate a clear advantage.

Critical strengths
  1. The problem the paper addresses is practically relevant.
  2. The method is presented clearly and completely enough that the model design is easy to follow.
Critical weaknesses
  1. The abstract claims that the method “provides this learning signal in both single- and multi-agent reinforcement learning,” but the paper includes no single-agent experiments. The Introduction also mentions evaluation on DeepMind Control (DMC), but no such results appear in the paper.

  2. In Figure 1, MA-JEPA does not stand out compared with the other methods. Also, why are MAMBA’s results missing from Table 2?

  3. The experimental evaluation is rather thin: it is conducted only on SMAC, and more benchmarks are needed to assess the method’s effectiveness and generalization.

  4. The ablation study is insufficient. There is only one ablation result (Figure 2), and removing other modules actually improves performance, which casts doubt on the module design.

Questions for authors
  1. Does “available training seeds” have any special meaning?
  2. Could you add a comparison that removes the joint predictor and relies only on local world-model imagination?
2 comments
AuthorsOpenReview ↗

Response to Reviewer MJVx (Part 1/2)

We thank the reviewer for their comments. We have uploaded a revised version of our paper, aiming to address the points raised in the rebuttal where we felt a paper revision was appropriate. See below for our full rebuttal response going through the reviewer's comments one by one, in two parts.

We believe we have rigorously and comprehensively addressed the reviewer's comments. If so, we would appreciate the reviewer revising their score appropriately.

Thank you for your help in improving the paper!

Missing single-agent and DMC results

"The abstract claims that the method “provides this learning signal in both single- and multi-agent reinforcement learning,” but the paper includes no single-agent experiments. The Introduction also mentions evaluation on DeepMind Control (DMC), but no such results appear in the paper."

You are right about the single-agent and DMC mention. We removed those mentions in the revised draft. The experiments reported here are on just the multi-agent SMAC v1 benchmark. Single-agent performance was strong, but we thought it appropriate to emphasize the multi-agent version as that was the major novel contribution. If the reviewer thinks it would strengthen the paper, we can re-add the single-agent experiments/DMC to the appendix for the camera-ready version.

Performance in Figure 1

"In Figure 1, MA-JEPA does not stand out compared with the other methods."

We disagree with how this response characterizes our performance, particularly in light of the expanded results in Table 1 of the revised draft. At matched environment budgets, MA-JEPA matches or achieves the highest mean win rate on four of eight SMAC maps and outperforms the strongest model-free baseline on seven. It outperforms DMAWM by 16 pp for the winrate on corridor (the hardest task) and 66p for the best model-free baseline. (It doesn't do so significantly for corridor at 1 sd, but this is because of the massive 44pp sd on DMAWM, which should be seen as an indicator of its poor performance rather than a virtue. Of the remainder, all four are ties up to statistical significance (at 1 sd). (We use NIST's Welch test, which is a finite-approximation variant of testing whether the means of our results as a normal distribution is larger than those of DMAWM. On the four remaining tests, where we match 3/4 of our 1 SD ranges contain the mean of DMAWM except for 3s_vs_5z. However, even there, NIST's Welch test reports a p-value of just 0.495.) These are primarily the easier problems where benchmarks are almost fully saturated.) On corridor, MA-JEPA reaches a 66.0% mean win rate at 400k steps, compared with 50.0% for DMAWM and 0% for all three model-free baselines. On 2c_vs_64zg, the corresponding rates at 200k steps are 78.3%, 75.3%, and 0%. The one exception is 3s_vs_4z, where DMAWM remains clearly ahead (95.7% vs. 74.0%), even after accounting for uncertainty. We discuss our understanding as to why this is the case in the paper, and we actually view this as an interesting positive result, rather than negative, as it indicates limitations of JEPA for specific problems (long-horizon tasks) that future work can aim to address. In particular, this result is consistent with other work that finds similar limitations for JEPA-based architectures more broadly in vision tasks (e.g., MotionJEPA: https://arxiv.org/pdf/2609.23881). Taken together, these results show that a world model trained to predict observation embeddings can support competitive decentralized policies on easier tasks and more performant policies on harder tasks, including on tasks where the model-free methods have yet to win at all at the same budget, as well as suggests potential useful avenues of improvement to explore for the JEPA architecture itself for multi-agent RL tasks.

Missing MAMBA results in the performance table

"Also, why are MAMBA’s results missing from Table 2?"

Thanks for pointing out this inconsistency. We've removed MAMBA from both the plots, and the performance table for the revised version. (For what it is worth, it was strictly dominated by DMAWM and our algorithm, MA-JEPA for all environments, so we saw little point in including it.)

Breadth of the benchmark suite

"The experimental evaluation is rather thin: it is conducted only on SMAC, and more benchmarks are needed to assess the method’s effectiveness and generalization."

SMACv1 is a standard classical RL benchmark. In principle, we agree with the reviewer's comment. We will add SMACv2 and Meltingpot benchmark results to the camera-ready version. Early results from this are promising and similar to performance on SMACv1, but were not feasible to fully run within the week of the rebuttals across multiple seeds.

AuthorsOpenReview ↗

Response to Reviewer MJVx (Part 2/2)

Interpretation and coverage of Figure 2 ablations

"The ablation study is insufficient. There is only one ablation result (Figure 2), and removing other modules actually improves performance, which casts doubt on the module design."

This is an incorrect characterization of our results and it is not clear what ablations the author has in mind. Claiming "ablations are insufficient" without suggesting what they actually want ablated is not particularly helpful in improving the paper. We also disagree that the ablations are insignificant as we vary the one free considerable architectural change. If you want us to perform ablations/work for work's sake, we can do that but we fail to see the point of this.

Figure 2 reports four configurations in a 2×2 ablation. We independently turn off two paths by which the joint model trains the local representations: gradients from predicting the next observation embedding and gradients from predicting reward, continuation, and whether each agent is alive. All four configurations retain the same modules and prediction losses. Turning off a gradient path helps on some maps and hurts on others. In the revised draft's paired-seed results across three maps, keeping both paths gives the highest average win rate (77.0%, compared with 73.9% for the best ablated variant), which is why we did so. So we disagree that ``removing other modules actually improves performance'' and we think this justifies our design decision.

Additionally, doing this before designing our model would have polluted our test results due to an "optimism bias". We tested this after the architecture design.

Meaning of “available training seeds”

"Does “available training seeds” have any special meaning?"

This was vague wording from our side. "Available training seeds" refers to seeds 1302, 2771, and 7636, the actual seeds we used to run the training. We will correct this in the camera-ready version.

Local-only imagination baseline

"Could you add a comparison that removes the joint predictor and relies only on local world-model imagination?"

In principle yes. In practice, we tried this already initially. The performance was quite bad. This makes sense as there is no way in which agents can assign credit/roles to other agents in this way as training is centralized. That is, all weights are identical. Also, observations are local and limited.

We are unsure the value this would bring other than showing that lacking the cross-attention leads to terrible performance. Prior papers (e.g., DMAWM) have already identified similar issues, though, so we do not necessarily think this is necessary.

Reviewer_XjqxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MA-JEPA, a model-based MARL method that applies joint-embedding predictive architectures (JEPA) to multi-agent settings. The core idea is to predict latent representations rather than reconstruct observations, while using centralized training to learn a world model that remains compatible with decentralized execution. This allows each agent to use the same local mechanism to construct its latent state during both training and execution.

The method adds objectives to ensure that predictions reflect the agents' actions and that predicted and real observations lead to consistent latent states. The paper also provides theoretical guarantees for the learned dynamics and evaluates MA-JEPA on six SMAC v1 tasks against several model-based and model-free MARL baselines.

Critical strengths

Clear and well-motivated idea. The paper identifies that a centralized world model can give the actor access to information that would not be available during decentralized execution. MA-JEPA avoids this by passing predicted observation embeddings through the same local posterior used during real interaction. This keeps the actor's state construction consistent between training and execution while still allowing the model to use joint actions.

Thm. 2 goes beyond adapting prior work. While Thm. 1 extends an identification argument from [1] to the joint-action setting, Thm. 2 provides a finite-sample total-variation bound on the learned joint successor law without assuming linear-Gaussian latent dynamics.

The paper presents the results and limitations transparently. The paper acknowledges that MA-JEPA performs worse, with higher variance, on the two "kiting" maps. Moreover, the authors report that the method matches or exceeds the strongest competitors on only three of the six maps.


Critical weaknesses

The empirical results are modest rather than a clear win. Across the six maps, MA-JEPA matches or outperforms the strongest competitors on only three. This limits the strength of the empirical case for a new algorithm.

Evaluation is limited to SMAC v1. The paper evaluates MA-JEPA on only six SMAC v1 maps, a benchmark where performance is often close to saturation. Strong competing methods, such as DMAWM and MAPPO variants, have also been evaluated on SMAC v2 [2], Google Research Football testbed [3], MPE testbed [4], or Melting Pot [5] in prior work. MA-JEPA has not been tested on these environments. Without evaluation on at least one additional, more complex benchmark, it is difficult to tell whether the benefits of embedding prediction generalize beyond SMAC v1.

Some experimental details are unclear. The results are reported as mean and standard deviation over the "available training seeds from the set {1302, 2771, 7636}" (cf. caption of Table 1), but the paper does not specify whether only these three seeds were used for each method and map, or whether the methods were evaluated across more independent runs. With standard deviations as high as 29.0, it is difficult to judge how robust the reported differences are.

The contribution of the JEPA objective is not isolated. The ablations examine gradient routing into local features, but do not compare MA-JEPA against a reconstruction-based world model with the same architecture, including the joint predictor and shared-posterior interface. As a result, it remains unclear whether any performance gains come from the embedding-prediction objective itself or from the architecture.

Scalability remains unclear. The paper does not examine how the method scales with the number of agents. The joint predictor uses self-attention across agents, which introduces at least quadratic growth in training-time compute and memory with the number of agents NN. The experiments cover only small to moderately sized teams, with no complexity analysis or larger-scale evaluation. This leaves an important limitation of the centralized design underinvestigated. Moreover, the paper does not report any wall-clock training times.

Limited reproducibility. The paper reports the hyperparameters used for MA-JEPA and the baselines, but does not provide source code, making it harder to reproduce the results.

Minor points that did not affect my overall recommendation:

  • Some figures appear to have relatively low resolution and may not be vector graphics.
  • Some figures are not numbered correctly (cf. page 5 and 6).
  • There are line breaks within inline equations, for example in Line 903.

Overall, these concerns currently outweigh the strengths of the paper. However, I am open to revising my assessment based on the authors' clarifications and any changes to the manuscript during the author-reviewer discussion phase.


Questions for authors
  • How many training seeds were actually used for each method and map? Are the reported differences in mean performance robust?

  • Could the authors compare MA-JEPA against a reconstruction-based world model with the same architecture to isolate the contribution of the joint-embedding objective?

  • Do the paper’s key findings generalize beyond SMAC v1?

  • How do the compute and memory requirements of the joint predictor's self-attention scale with the number of agents NN?

  • How large is the residual train vs. execution gap in practice? For example, how much do the posterior states induced by predicted embeddings differ from those induced by real observations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xjqx (Part 1/3)

We thank the reviewer for their comments. We have uploaded a revised version of our paper, aiming to address the points raised in the rebuttal where we felt a paper revision was appropriate. See below for our full rebuttal response going through the reviewer's comments one by one, in three parts.

We believe we have rigorously and comprehensively addressed the reviewer's comments. If so, we would appreciate the reviewer revising their score appropriately.

Thank you for your help in improving the paper!

Size of the empirical advantage

"The empirical results are modest rather than a clear win. Across the six maps, MA-JEPA matches or outperforms the strongest competitors on only three. This limits the strength of the empirical case for a new algorithm."

We disagree with how this response characterizes our performance, particularly in light of the expanded results in Table 1 of the revised draft. At matched environment budgets, MA-JEPA matches or achieves the highest mean win rate on four of eight SMAC maps and outperforms the strongest model-free baseline on seven. It outperforms DMAWM by 16 pp for the winrate on corridor (the hardest task) and 66p for the best model-free baseline. (It doesn't do so significantly for corridor at 1 sd, but this is because of the massive 44pp sd on DMAWM, which should be seen as an indicator of its poor performance rather than a virtue. Of the remainder, all four are ties up to statistical significance (at 1 sd). (We use NIST's Welch test, which is a finite-approximation variant of testing whether the means of our results as a normal distribution is larger than those of DMAWM. On the four remaining tests, where we match 3/4 of our 1 SD ranges contain the mean of DMAWM except for 3s_vs_5z. However, even there, NIST's Welch test reports a p-value of just 0.495.) These are primarily the easier problems where benchmarks are almost fully saturated.) On corridor, MA-JEPA reaches a 66.0% mean win rate at 400k steps, compared with 50.0% for DMAWM and 0% for all three model-free baselines. On 2c_vs_64zg, the corresponding rates at 200k steps are 78.3%, 75.3%, and 0%. The one exception is 3s_vs_4z, where DMAWM remains clearly ahead (95.7% vs. 74.0%), even after accounting for uncertainty. We discuss our understanding as to why this is the case in the paper, and we actually view this as an interesting positive result, rather than negative, as it indicates limitations of JEPA for specific problems (long-horizon tasks) that future work can aim to address. In particular, this result is consistent with other work that finds similar limitations for JEPA-based architectures more broadly in vision tasks (e.g., MotionJEPA: https://arxiv.org/pdf/2609.23881). Taken together, these results show that a world model trained to predict observation embeddings can support competitive decentralized policies on easier tasks and more performant policies on harder tasks, including on tasks where the model-free methods have yet to win at all at the same budget, as well as suggests potential useful avenues of improvement to explore for the JEPA architecture itself for multi-agent RL tasks.

Generalization beyond the six SMAC v1 maps

"Evaluation is limited to SMAC v1. The paper evaluates MA-JEPA on only six SMAC v1 maps, a benchmark where performance is often close to saturation. Strong competing methods, such as DMAWM and MAPPO variants, have also been evaluated on SMAC v2 [2], Google Research Football testbed [3], MPE testbed [4], or Melting Pot [5] in prior work. MA-JEPA has not been tested on these environments. Without evaluation on at least one additional, more complex benchmark, it is difficult to tell whether the benefits of embedding prediction generalize beyond SMAC v1."

We agree with the reviewer's comment. We will add SMACv2 and Meltingpot benchmark results to the camera-ready version. Early results from this are promising and similar to performance on SMACv1, but were not computationally feasible to fully run within the week of the rebuttals across multiple seeds.

AuthorsOpenReview ↗

Response to Reviewer Xjqx (Part 2/3)

Per-cell seed counts and robustness

"Some experimental details are unclear. The results are reported as mean and standard deviation over the "available training seeds from the set {1302, 2771, 7636}" (cf. caption of Table 1), but the paper does not specify whether only these three seeds were used for each method and map, or whether the methods were evaluated across more independent runs. With standard deviations as high as 29.0, it is difficult to judge how robust the reported differences are."

For each method and each map three seeds (1302, 2771, and 7636) were used. We report new standard deviations in the figure. In the revised Table 1, the standard deviations on 2s3z are 14.5 (the highest reported for MA-JEPA anywhere) and 7.0 percentage points for MA-JEPA and DMAWM, respectively. On corridor, they are 3.5 and 44.9. The 29.0-point standard deviation in the submitted version was on 3s_vs_5z (17.7 for DMAWM), which we did not test in the latest version. However, we note the 29-point standard deviation, even if it continued to hold (which, we suspect would be greatly reduced), would not be unusual in comparison to alternative methods such as DMAWM, who report substantially worse standard deviation such as 44.9.

Matched reconstruction-based ablation

"The contribution of the JEPA objective is not isolated. The ablations examine gradient routing into local features, but do not compare MA-JEPA against a reconstruction-based world model with the same architecture, including the joint predictor and shared-posterior interface. As a result, it remains unclear whether any performance gains come from the embedding-prediction objective itself or from the architecture."

A reconstruction-based variant with the same joint predictor and local posterior in theory might be a useful comparison and one we can consider adding to the rebuttal. However, this is not JEPA and clearly not the same "architecture". It would involve substantial degrees of freedom in the design also. For example, what should our objective be? Our current objective is on the latents. How should our policy functions look? Should we still be using SIGREG when it is meaningless in the reconstruction variant.

The entire space of the problem changes so it is not really possible to do an apples-to-apples comparison holding the architecture fixed or something. Even if we did, all it would test is whether our architecture is reasonable in a reconstruction variant. That says nothing about its usefulness in the original variant.

What is more useful is a comparison against an optimized version with a reconstruction objective. We already do this! This is nothing but DMAWM and the other model-based architectures we compare against. See our empirical results for a comparison, but we match or outperform these in 7/8 cases up to 1 sd and in 8/8 at the more standard 2 sd degrees of significance. We clearly beat DMAWM on corridor, however.

Team size and the complexity of agent attention

"Scalability remains unclear. The paper does not examine how the method scales with the number of agents. The joint predictor uses self-attention across agents, which introduces at least quadratic growth in training-time compute and memory with the number of agents NN. The experiments cover only small to moderately sized teams, with no complexity analysis or larger-scale evaluation. This leaves an important limitation of the centralized design underinvestigated."

While the joint predictor incurs quadratic cost in the number of agents due to cross-attention during training, this is not severe, especially at the scale of our experiments. In practice, even for large numbers of agents, beyond memory requirements for truly massive agents, this is not likely to be a major bottleneck. In the latter case, one could still use sharding or other such approaches to decentralize further.

AuthorsOpenReview ↗

Response to Reviewer Xjqx (Part 3/3)

Wall-clock training time

"Moreover, the paper does not report any wall-clock training times."

We agree with the reviewers and will add wall-clock training times to the camera-ready version. Preliminary experiments show that MA-JEPA is almost two times as fast as DMAWM on SMAC using NVIDIA A100 GPUs while maintaining competitive results.

MapModelTimeSpeed-up
2s3zDMAWM3.36 h1.00×
2s3zMA-JEPA1.73 h1.94×
MMMDMAWM3.75 h1.00×
MMMMA-JEPA2.10 h1.79×

Availability of source code

"Limited reproducibility. The paper reports the hyperparameters used for MA-JEPA and the baselines, but does not provide source code, making it harder to reproduce the results."

Thanks for pointing this out. We will provide the source code with the camera-ready version of the paper.

Figure resolution and vector graphics

"Some figures appear to have relatively low resolution and may not be vector graphics."

We agree that the plots were low resolution. We have aimed to correct this for the rebuttal (see latest plots) and will do so for the camera ready version fully.

Missing numbers on the architecture diagrams

"Some figures are not numbered correctly (cf. page 5 and 6)."

Thank you for identifying this. We have fixed this in the revised version of the paper we uploaded.

Inline equation wraps at line 903

"There are line breaks within inline equations, for example in Line 903."

The inline equation around line 903 and the other equations is now fixed.

Residual disagreement between real and imagined posteriors

"How large is the residual train vs. execution gap in practice? For example, how much do the posterior states induced by predicted embeddings differ from those induced by real observations?"

This is an excellent suggestion and we appreciate the reviewer suggesting it. We hold the replay history fixed, pass the real and predicted next-observation embeddings through the same local posterior, and penalize the KL from the real-observation posterior to the predicted-embedding posterior. However, we never actually recorded and plotted this KL distance over time, only as one component of the loss functions (which we do plot.)

We will provide this for the camera ready version.

Reviewer_yXuNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper provides a method that leverages JEPA for world modeling in MARL. The paper combines the JEPA approach with a Dreamer-like approach in learning transition and inference networks. This approach avoids the need for a potentially expensive reconstruction objective.

Critical strengths
  1. The centralized approach of the modeling dureing training leads to an improved transition function, and is connected well with the functions used during rolling out. So the actor seems similar states during training and in test time avoiding a mismatch scenario.
  2. Avoiding the reconstruction cost by using JEPA.
Critical weaknesses
  1. It is not clear why JEPA is better than similar non-reconstruction approaches like Dreamer-Pro and R2 Dreamer. If JEPA is the main contribution, stronger justification and analysis would help, especially since the method doesn't outperform baselines in the experiments.
  2. The use of a local world model for each agent makes the method not scalable in terms of the number of agents
  3. The theory has significant differences to the architecture. Theorem 1 assumes linear-Gaussian dynamics, a squared loss, and an exactly whitened encoder. The method uses cosine losses, approximate regularization (SIGReg), categorical latents, and nonlinear networks
  4. The results show DMAWM is better; is there a reason for this?
Questions for authors

How scalable is the method in the number of agents ? Is there an improvement in the learned latent space by using jepa ?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yXuN (Part 1/2)

We thank the reviewer for their comments. We have uploaded a revised version of our paper, aiming to address the points raised in the rebuttal where we felt a paper revision was appropriate. See below for our full rebuttal response going through the reviewer's comments one by one, in two parts.

We believe we have rigorously and comprehensively addressed the reviewer's comments. If so, we would appreciate the reviewer revising their score appropriately.

Thank you for your help in improving the paper!

Comparison with other reconstruction-free objectives

"It is not clear why JEPA is better than similar non-reconstruction approaches like Dreamer-Pro and R2 Dreamer. If JEPA is the main contribution, stronger justification and analysis would help, especially since the method doesn't outperform baselines in the experiments."

We agree with the reviewers that removing reconstruction alone does not distinguish our method but its use of JEPA does. Both DreamerPro and R2-Dreamer were designed for single-agent control, which limits direct comparison with our multi-agent setting.

If we were forced to compare methods, we would still favor JEPAs for imagined rollouts because it directly predicts the next observation embedding from the agent’s history and action, while DreamerPro and R2-Dreamer apply their representation losses to a latent state derived from that existing observation. Our focus is on jointly predicting the next local observation embeddings of N agents and using SIGReg to prevent collapse of the latent representations. Also, SIGREG is shown to be necessary to prevent latent collapse in JEPA for the reconstruction-free objective and not used in either DreamerPro or R2-Dreamer. We wonder, therefore, if the limited adoption of them compared to encoder-decoder models several years after their release suggests possible instability in their reconstruction-free approaches.

Does a local model per agent prevent scaling?

"The use of a local world model for each agent makes the method not scalable in terms of the number of agents"

The method should be very scalable in the number of agents. The local world model shares parameters across agents due to CTDE, so parameters do not scale with the number of agents. Training is done centrally, so just once (though batched), while execution can be done completely in parallel. The only thing that scales is the joint predictor across agents, which incurs quadratic cost in the number of agents due to cross-attention during training, but this is not severe, especially at the scales we are looking at. In practice, even for large numbers of agents, beyond memory requirements for truly massive agents, this is not likely to be a major bottleneck. In the latter case, one could still use sharding or other such approaches to decentralize further.

Scope of the Gaussian identification theorem

"The theory has significant differences to the architecture. Theorem 1 assumes linear-Gaussian dynamics, a squared loss, and an exactly whitened encoder. The method uses cosine losses, approximate regularization (SIGReg), categorical latents, and nonlinear networks"

There is a major fundamental misunderstanding here. We never claim the original problem is LQG in the theory. That would be absurd.

What we do require is that the underlying latent space is assumed gaussian. This is standard for JEPA and a necessary assumption when using SIGREG. This is because SIGREG regularizes latents towards an isotropic gaussian.

In principle one could consider other regularizers other than SIGREG, but then this is not the original JEPA. This is exactly the same set of assumptions used in the original JEPA paper.

This is not uncommon in theoretical deep RL work versus practical implementation. JEPA with SIGREG requires latent Gaussian variables for the theory to hold. Otherwise, the latent objective is theoretically misspecified. However, note that this is specifically in the latent space, not the original space.

In principle, we could consider alternative regularizers instead of SIGREG. In further theory work we can consider specifically measuring something like the projection of the gaussian onto the true discrete space we consider.

We use the theorem moreso to motivate the objective for an idealized setting. We also consider a categorical matching version.

The original space does not require any of the claimed assumptions, so it does fit into our framework.

Importantly, beyond the gaussian assumption on latents, we also prove a consistency result for categorical latents. For a fixed representation and data-collection policy, the empirical conditional distribution of the next latent state, reward, termination, and action mask converges to the true conditional distribution on observed latentstate-reward-action pairs.

AuthorsOpenReview ↗

Response to Reviewer yXuN (part 2/2)

Where and why does DMAWM do better?

"The results show DMAWM is better; is there a reason for this?"

We disagree with how this response characterizes our performance, particularly in light of the expanded results in Table 1 of the revised draft. At matched environment budgets, MA-JEPA matches or achieves the highest mean win rate on four of eight SMAC maps and outperforms the strongest model-free baseline on seven. It outperforms DMAWM by 16 pp for the winrate on corridor (the hardest task) and 66p for the best model-free baseline. (It doesn't do so significantly for corridor at 1 sd, but this is because of the massive 44pp sd on DMAWM, which should be seen as an indicator of its poor performance rather than a virtue. Of the remainder, all four are ties up to statistical significance (at 1 sd). (We use NIST's Welch test, which is a finite-approximation variant of testing whether the means of our results as a normal distribution is larger than those of DMAWM. On the four remaining tests, where we match 3/4 of our 1 SD ranges contain the mean of DMAWM except for 3s_vs_5z. However, even there, NIST's Welch test reports a p-value of just 0.495.) These are primarily the easier problems where benchmarks are almost fully saturated.) On corridor, MA-JEPA reaches a 66.0% mean win rate at 400k steps, compared with 50.0% for DMAWM and 0% for all three model-free baselines. On 2c_vs_64zg, the corresponding rates at 200k steps are 78.3%, 75.3%, and 0%. The one exception is 3s_vs_4z, where DMAWM remains clearly ahead (95.7% vs. 74.0%), even after accounting for uncertainty. We discuss our understanding as to why this is the case in the paper, and we actually view this as an interesting positive result, rather than negative, as it indicates limitations of JEPA for specific problems (long-horizon tasks) that future work can aim to address. In particular, this result is consistent with other work that finds similar limitations for JEPA-based architectures more broadly in vision tasks (e.g., MotionJEPA: https://arxiv.org/pdf/2609.23881). Taken together, these results show that a world model trained to predict observation embeddings can support competitive decentralized policies on easier tasks and more performant policies on harder tasks, including on tasks where the model-free methods have yet to win at all at the same budget, as well as suggests potential useful avenues of improvement to explore for the JEPA architecture itself for multi-agent RL tasks.

Effect of JEPA on the learned representation

"Is there an improvement in the learned latent space by using jepa ?"

The current results show that the learned representations support strong policies on several SMAC maps. For the camera-ready version we can test whether changing the objective with SIGREG or other similar architectures improves the learned representations and resulting policies. In general, we would expect this to be the case, given the arguments made in the original JEPA literature, where SIGREG is necessary to avoid distributional collapse in the latent space. We can explicitly specify this in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.