- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a theoretical framework for JEPA world models based on a spectral graph. It establishes a connection between JEPA pre-training error and planning error. Finally, it studies the impact of using multiple depths representations for prediction helps to reduce success rate on a toy example and PushT.
Critical strengths
S1: The paper provides a interesting theoretical framework for world models
S2: It shows that applying JEPA using concatenated representations of multiple depths helps to increase planning success rates on a toy example task and PushT.
Critical weaknesses
W1: Limited Contribution
Apart from providing a theoretical framework for JEPA and studying the impact of a Multi-Depth JEPA, the paper does not have major contributions.
W2: Limited Evaluation
The paper proposes a Multi-Depth JEPA but only evaluate its variant on toy tasks. The number of seeds for Figure 2 and table 1 is not provided.
W3: Focus on JEPA while other world model alternates exist
The paper only focuses on JEPA-based world modeling while alternative approaches (Diffusion-based (DIAMOND, EDELINE), Cross entropy (Delta-IRIS), KL Divergence (Dreamer), MCTS (MuZero) objectives) have been proposed in recent years, achieving state-of-the-art performance on various benchmarks.
Questions for authors
Q1: How do you define the success rate in Figure 2?
Q2: How many seeds are used for evaluation experiments ?
3 comments
W3. The paper only focuses on JEPA-based world modeling while alternative approaches (Diffusion-based (DIAMOND, EDELINE), Cross entropy (Delta-IRIS), KL Divergence (Dreamer), MCTS (MuZero) objectives) have been proposed in recent years, achieving state-of-the-art performance on various benchmarks.
A3. We acknowledge the existence and empirical success of alternative world-model approaches. However, our goal is not to establish JEPA as the empirically best world-model architecture, but to provide the first generalization-theoretic framework connecting JEPA pretraining to downstream action planning, a fundamental problem that remains largely unexplored. Specifically, our analysis connects action-conditioned latent prediction to planning regret and finite-sample generalization, revealing an approximation–sample error trade-off governed by latent dimensionality and training-set size. This addresses a distinct theoretical question rather than competing with alternative approaches on benchmark performance.
Our framework also suggests possible extensions to alternative world models. For Dreamer, one could replace the spectral JEPA prediction risk with a divergence-based measure of latent transition error and derive planning-error bounds through KL-to-total-variation inequalities under suitable assumptions. For MuZero, the analysis could incorporate reward/value prediction errors and an additional MCTS search-approximation term, separating model generalization error from planning error. Beyond theoretical extensions, our Multi-Depth representation strategy could also be adapted to other latent world models by combining features from different network depths to enrich their predictive representations. These directions require further theoretical and empirical investigation but illustrate the potential broader applicability of our framework and methodology.
Q1. How do you define the success rate in Figure 2?
A1. As specified in Appendix A.4, a rollout is considered successful when the agent reaches the goal within a ground-truth positional distance of 0.08, i.e., . The success rate is the proportion of successful evaluation rollouts. Both latent- and input-level models use the same evaluation criterion.
Q2. How many seeds are used for evaluation experiments?
A. Three independent training seeds are used. Please also refer to W2 for more details.
Thanks for your comments. We sincerely hope that our explanations and additional experiments solve your concerns.
Dear Reviewer PpA1,
As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We are sincerely grateful for your time and effort devoted to carefully evaluating our manuscript. To address your remaining concerns, we have provided additional explanations and experiments.
We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions and would be delighted to provide additional clarifications.
Best regards, The Authors
We thank the reviewer for recognizing the value of our theoretical framework for world models and the improvements in planning performance. We address your concerns below.
W1. Apart from providing a theoretical framework for JEPA and studying the impact of a Multi-Depth JEPA, the paper does not have major contributions.
A1. We respectfully disagree. To the best of our knowledge, our work provides the first generalization-theoretic framework connecting JEPA pretraining to downstream action planning. As recognized by Reviewer JFVz, establishing a theoretical connection between action-conditioned predictions and downstream planning performance addresses an important gap in the existing JEPA literature. We believe that addressing this fundamental problem is itself a major theoretical contribution.
We re-emphasize our contributions as follows:
Theoretical formulation and generalization insights. We introduce an asymmetric action-conditioned co-occurrence matrix to capture directed state transitions and establish a connection between JEPA pretraining error and downstream planning regret. Our finite-sample analysis characterizes an approximation–sample error trade-off governed by latent dimensionality k and training-set size n, providing a unified theoretical perspective for comparing latent- and input-level prediction.
Theory-inspired Multi-Depth JEPA. Motivated by the approximation–sample error trade-off, we investigate Multi-Depth JEPA, which combines representations from complementary network depths. Our PushT experiments demonstrate improved planning performance, particularly at longer horizons.
Empirical insights into latent versus input prediction. We systematically study how unpredictable observation components and planning horizon affect the relative performance of latent- and input-level prediction. Our results show that latent prediction becomes more advantageous in challenging noisy and long-horizon settings, while input-level prediction can remain competitive or superior in simpler regimes.
Taken together, our contributions extend beyond a theoretical formulation and an architectural extension: we address a fundamental gap in JEPA-based world-model generalization, provide theoretical insights into when latent prediction is preferable, and demonstrate how these insights can guide representation design.
W2. The paper proposes a Multi-Depth JEPA but only evaluate its variant on toy tasks. The number of seeds for Figure 2 and table 1 is not provided.
A2. We first clarify that MD-JEPA is not evaluated solely on toy tasks. Figure 2 uses a controlled 2D environment to isolate the effects of unpredictable observations and planning difficulty, whereas Table 1 evaluates MD-JEPA on PushT, a standard visual robotic manipulation benchmark, using DINO-WM with a pretrained ViT backbone. We evaluate planning performance at three horizons (H = 25, 50, 75). While this provides validation beyond synthetic environments, we acknowledge that additional robotic tasks would further strengthen the empirical evidence.
To address the missing random-seed information, we conducted additional experiments with three independent training seeds, evaluating each seed on 150 fixed queries per planning horizon. Furthermore, we replace the original Layer 12 × 2 baseline with the [11,12] combination, which provides a more meaningful dimension-matched comparison. Unlike duplicating Layer-12 features, [11,12] concatenates representations from two distinct layers, providing additional learned features while maintaining the same representation dimensionality and pretrained backbone as [6,12]. We show that the [6,12] combination outperforms the dimension-matched [11,12] baseline at H = 25, 50, and 75, respectively. These results indicate that selecting complementary representation depths matters beyond merely increasing the concatenated feature dimensionality.
For Figure 2, we replace the original results with a three-seed evaluation in the next version of the manuscript. The updated evaluation preserves the original qualitative conclusions: latent- and input-level prediction perform comparably on simpler tasks, while latent-level prediction exhibits greater robustness under more challenging noisy-observation and long-horizon planning settings.