acceptodds
Under review as a conference paper at ICLR 2027

A Generalization Theory for JEPA-Based World Models

Abstract

Joint Embedding Predictive Architectures (JEPAs) have emerged as a promising paradigm for world modeling by predicting future representations rather than reconstructing observations. Despite their empirical success, the theoretical understanding of JEPA-based world models remains limited. In this paper, we develop the first generalization theory for JEPA-based world models. We formulate JEPA pretraining as a conditional spectral graph learning problem and show that the JEPA objective is equivalent to a low-rank factorization of an action-conditioned co-occurrence matrix. Building on this characterization, we establish a connection between JEPA pretraining error and downstream planning regret, leading to a finite-sample generalization bound for JEPA-based world models. Our analysis reveals an approximation–sample error trade-off governed by latent dimensionality, providing insights into latent- versus input-level prediction. The theoretical trade-off motivates us to investigate a simple Multi-Depth JEPA extension that combines representations from different network depths. In validation experiments, latent-level prediction exhibits greater robustness under unpredictable observations and difficult planning tasks. Multi-depth representations further improve planning success over single-depth representations and a dimensionality-matched control.

Primary area:
learning theory
Keywords:
JEPA, spectral graph theory, generalization theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PpA1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a theoretical framework for JEPA world models based on a spectral graph. It establishes a connection between JEPA pre-training error and planning error. Finally, it studies the impact of using multiple depths representations for prediction helps to reduce success rate on a toy example and PushT.

Critical strengths

S1: The paper provides a interesting theoretical framework for world models

S2: It shows that applying JEPA using concatenated representations of multiple depths helps to increase planning success rates on a toy example task and PushT.

Critical weaknesses

W1: Limited Contribution

Apart from providing a theoretical framework for JEPA and studying the impact of a Multi-Depth JEPA, the paper does not have major contributions.

W2: Limited Evaluation

The paper proposes a Multi-Depth JEPA but only evaluate its variant on toy tasks. The number of seeds for Figure 2 and table 1 is not provided.

W3: Focus on JEPA while other world model alternates exist

The paper only focuses on JEPA-based world modeling while alternative approaches (Diffusion-based (DIAMOND, EDELINE), Cross entropy (Delta-IRIS), KL Divergence (Dreamer), MCTS (MuZero) objectives) have been proposed in recent years, achieving state-of-the-art performance on various benchmarks.

Questions for authors

Q1: How do you define the success rate in Figure 2?

Q2: How many seeds are used for evaluation experiments ?

3 comments

We thank the reviewer for recognizing the value of our theoretical framework for world models and the improvements in planning performance. We address your concerns below.


W1. Apart from providing a theoretical framework for JEPA and studying the impact of a Multi-Depth JEPA, the paper does not have major contributions.

A1. We respectfully disagree. To the best of our knowledge, our work provides the first generalization-theoretic framework connecting JEPA pretraining to downstream action planning. As recognized by Reviewer JFVz, establishing a theoretical connection between action-conditioned predictions and downstream planning performance addresses an important gap in the existing JEPA literature. We believe that addressing this fundamental problem is itself a major theoretical contribution.

We re-emphasize our contributions as follows:

  1. Theoretical formulation and generalization insights. We introduce an asymmetric action-conditioned co-occurrence matrix to capture directed state transitions and establish a connection between JEPA pretraining error and downstream planning regret. Our finite-sample analysis characterizes an approximation–sample error trade-off governed by latent dimensionality kk and training-set size nn, providing a unified theoretical perspective for comparing latent- and input-level prediction.

  2. Theory-inspired Multi-Depth JEPA. Motivated by the approximation–sample error trade-off, we investigate Multi-Depth JEPA, which combines representations from complementary network depths. Our PushT experiments demonstrate improved planning performance, particularly at longer horizons.

  3. Empirical insights into latent versus input prediction. We systematically study how unpredictable observation components and planning horizon affect the relative performance of latent- and input-level prediction. Our results show that latent prediction becomes more advantageous in challenging noisy and long-horizon settings, while input-level prediction can remain competitive or superior in simpler regimes.

Taken together, our contributions extend beyond a theoretical formulation and an architectural extension: we address a fundamental gap in JEPA-based world-model generalization, provide theoretical insights into when latent prediction is preferable, and demonstrate how these insights can guide representation design.


W2. The paper proposes a Multi-Depth JEPA but only evaluate its variant on toy tasks. The number of seeds for Figure 2 and table 1 is not provided.

A2. We first clarify that MD-JEPA is not evaluated solely on toy tasks. Figure 2 uses a controlled 2D environment to isolate the effects of unpredictable observations and planning difficulty, whereas Table 1 evaluates MD-JEPA on PushT, a standard visual robotic manipulation benchmark, using DINO-WM with a pretrained ViT backbone. We evaluate planning performance at three horizons (H = 25, 50, 75). While this provides validation beyond synthetic environments, we acknowledge that additional robotic tasks would further strengthen the empirical evidence.

To address the missing random-seed information, we conducted additional experiments with three independent training seeds, evaluating each seed on 150 fixed queries per planning horizon. Furthermore, we replace the original Layer 12 × 2 baseline with the [11,12] combination, which provides a more meaningful dimension-matched comparison. Unlike duplicating Layer-12 features, [11,12] concatenates representations from two distinct layers, providing additional learned features while maintaining the same representation dimensionality and pretrained backbone as [6,12]. We show that the [6,12] combination outperforms the dimension-matched [11,12] baseline at H = 25, 50, and 75, respectively. These results indicate that selecting complementary representation depths matters beyond merely increasing the concatenated feature dimensionality.

DINO layersH = 25H = 50H = 75
[3,12]76.67 ± 1.1525.11 ± 0.778.89 ± 1.92
[6,12]75.56 ± 2.1431.56 ± 3.0812.89 ± 1.92
[9,12]66.00 ± 0.6626.89 ± 1.029.11 ± 1.54
[11,12]70.22 ± 6.4125.33 ± 1.339.78 ± 0.77

For Figure 2, we replace the original results with a three-seed evaluation in the next version of the manuscript. The updated evaluation preserves the original qualitative conclusions: latent- and input-level prediction perform comparably on simpler tasks, while latent-level prediction exhibits greater robustness under more challenging noisy-observation and long-horizon planning settings.

W3. The paper only focuses on JEPA-based world modeling while alternative approaches (Diffusion-based (DIAMOND, EDELINE), Cross entropy (Delta-IRIS), KL Divergence (Dreamer), MCTS (MuZero) objectives) have been proposed in recent years, achieving state-of-the-art performance on various benchmarks.

A3. We acknowledge the existence and empirical success of alternative world-model approaches. However, our goal is not to establish JEPA as the empirically best world-model architecture, but to provide the first generalization-theoretic framework connecting JEPA pretraining to downstream action planning, a fundamental problem that remains largely unexplored. Specifically, our analysis connects action-conditioned latent prediction to planning regret and finite-sample generalization, revealing an approximation–sample error trade-off governed by latent dimensionality and training-set size. This addresses a distinct theoretical question rather than competing with alternative approaches on benchmark performance.

Our framework also suggests possible extensions to alternative world models. For Dreamer, one could replace the spectral JEPA prediction risk with a divergence-based measure of latent transition error and derive planning-error bounds through KL-to-total-variation inequalities under suitable assumptions. For MuZero, the analysis could incorporate reward/value prediction errors and an additional MCTS search-approximation term, separating model generalization error from planning error. Beyond theoretical extensions, our Multi-Depth representation strategy could also be adapted to other latent world models by combining features from different network depths to enrich their predictive representations. These directions require further theoretical and empirical investigation but illustrate the potential broader applicability of our framework and methodology.


Q1. How do you define the success rate in Figure 2?

A1. As specified in Appendix A.4, a rollout is considered successful when the agent reaches the goal within a ground-truth positional distance of 0.08, i.e., ∥pcurrent−pgoal∥2≤0.08\|p_{\mathrm{current}}-p_{\mathrm{goal}}\|_2 \leq 0.08. The success rate is the proportion of successful evaluation rollouts. Both latent- and input-level models use the same evaluation criterion.


Q2. How many seeds are used for evaluation experiments?

A. Three independent training seeds are used. Please also refer to W2 for more details.


Thanks for your comments. We sincerely hope that our explanations and additional experiments solve your concerns.

Dear Reviewer PpA1,

As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We are sincerely grateful for your time and effort devoted to carefully evaluating our manuscript. To address your remaining concerns, we have provided additional explanations and experiments.

We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions and would be delighted to provide additional clarifications.

Best regards, The Authors

Reviewer_yc9KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a generalization theory for JEPA-based world models and provides insights on the advantages and limitations of latent prediction versus input level prediction. They formulate JEPA pretraining as a conditional spectral graph learning probem using an action-conditioned co-occurrence matrix and establish a connetion between the proposed spectral JEPA objective and low-rank factorization. Additionally, the paper establishes a connection between JEPA pretraining error and downstream planning regret and provide not only sample generalization bounds but also approximation-sample error tradeoffs for JEPA-based world models. Motivated by this analysis, the author also propose MD-JEPA which concatenates intermediate representations from different networks depths. Experiments on a synthetic point-mass environment and PushT benchmark suggest that latent prediction can be more robust to noise as well as multi-depth representations can improve planning success.

Critical strengths

The authors provide an interesting theoretical perspective connecting JEPA pretraining to downstream planning. The connection between pretraining error and planning regret provides an interesting theoretical perspective and the approximation-sample error trade-off provides an explanation of the trade-off between latent-level prediction and input-level reconstruction. The authors also propose an interesting multi-depth JEPA architecture based on their insights that shows preliminary improvements in planning performance.

Critical weaknesses

W1. The theoretical framework requires further justification. In section 3.1 the authors adopt a uniformity term as the regularization for simplicity and justify it by empirically evaluating the effect of the term in Appendix A.3. However, in A.3, they test what happens to LeWM when adding the uniformity term, meaning A.3 tests JEPA + SIGReg + Uniformity instead of JEPA + Uniformity like it should to justify using their adopted uniformity loss over SIGReg in this paper. In essence, A.3 only shows that the term is valid as an additional term but no experiment trains the objective the theory actually analyzes of JEPA + uniformity with unit norm embeddings so it is unclear if the theory supports a working world model. The normalization is also inconsistent across the theory. Theorem 3.1 requires unit-norm f and g but Theorem 4.3 minimizes over all rank matrices and ignores this constraint. Additionally, in Theorem 4.1 the quality of expected goal-state marginals for fixed actions was used to eliminate a term involving state-dependent optimal actions which does not appear to be well justified by the assumption alone. These issues should be clarified because the main finite-sample generalization claims depend on these guarantees.

W2. The experiments do not cleanly support the empirical claims. In figure 2 the latent model improves as observation noise std increase which seems counterintuitive and requires further investigation or explanation. Also in Appendix A.4 it describes varying the reconstruction weight assigned to the noise dimensions rather than the observation noise std. This would favor latent models which should not have a reconstruction loss term yet the input level models do. These discrepancies alongside no reports of how many runs make the results hard to interpret. Additionally, MD-JEPA's evaluation lacks a proper dimensionality-matched baseline. They duplicate layer 12 representation to construct the 12x2 baseline but this doesn't provide any additional representational capacity. A more appropriate comparison would be to increase dimensionality of the JEPA model and without this baseline, it is difficult to say whether the improvements come from multi-depth or simply increasing the effective representation capacity from concatenating intermediate feature representations.

Questions for authors

Q1. Can you train JEPA + uniformity with unit norm embeddings without SIGReg and compare it to LeWM to see if this is a valid alternative? Q2. Does the x-axis change the observation or only the reconstruction weight in figure 2 and why does the latent model perform better as the noise increases? Q3. Can you provide a dimensionality matched baseline with more capacity for MD-JEPA? For example maybe widen the final backbone layer. Q4. Can you explain if Theorem 4.3 is meant to hold under Theorem 3.1's unit-norm constraint? Q5. Could you provide any experiments that directly validate the approximation-sample error trade off by providing an evaluation across different latent dimensionality, k, and different training-set sizes? This would strengthen the connection with the theoretical results.

4 comments

We thank the reviewer for recognizing the value of connecting JEPA pretraining to downstream planning and the insights from our theoretical analysis. We address your concerns below.


W1. The theoretical framework requires further justification.

A1. (1) Uniformity-only ablation. To isolate the contribution of uniformity, we add the uniformity objective to DINO-WM while keeping all other settings unchanged.

MethodH25 Success Rate ↑
DINO-WM64.0% (96/150)
DINO-WM + Uniformity66.0% (99/150)

Adding uniformity improves planning success by 2.0%, providing preliminary evidence of its independent contribution to downstream planning.

(2) Normalization assumption. We have revised Theorems 3.1 and 4.3 by replacing pointwise unit-norm constraints with a fixed expected representation-energy assumption, preserving the JEPA–spectral equivalence and the central approximation–sample error trade-off under the revised conditions. Please refer to our response to Q4 for details.

(3) Action-selection assumption. Our action-selection rule follows the intuitive principle of goal-conditioned latent-space planning: selecting the action whose predicted next-state representation is closest to the goal representation. This provides a natural and interpretable criterion for planning in the learned representation space. We have clarified this motivation in the revised manuscript.


W2. The experiments do not cleanly support the empirical claims.

A2. (1) Clarification of the noise experiment in Figure 2.

We clarify that the observation noise standard deviation in Figure 2 and the reconstruction weight described in Appendix A.4 represent equivalent parameterizations of the effective noise scale in the reconstruction objective, rather than two independent experimental variables. Specifically, independent random noise is introduced only in the noise dimensions, while the task-relevant observation dimensions contain no independently injected noise. Under the squared reconstruction loss, assigning weight λ\lambda to the noise dimensions is equivalent to scaling these dimensions by λ\sqrt{\lambda}. Consequently, increasing the reconstruction weight increases the effective standard deviation of the noise components in the reconstruction objective. We have clarified this relationship in Appendix A.4 to avoid ambiguity.

We further clarify that both input- and latent-level models receive exactly the same noisy observations under identical noise settings, so the setting does not "favor latent models". Specifically, the latent representations are learned directly from these noisy observations, rather than being provided as noise-free or privileged state representations. Both approaches therefore have access to the same input information. Consequently, the ability to suppress task-irrelevant noise is learned by the latent model itself, rather than introduced through a favorable observation setting.

(2) Multi-seed evaluation of MD-JEPA.

To address the lack of variability estimates, we repeat the DINO-based PushT experiments using three independently trained random seeds, evaluating each seed on 150 fixed queries per planning horizon. The results report mean success rate ± standard deviation across seeds (%). The [6,12] combination achieves the highest mean success rates at H50 and H75, while [3,12] performs best at H25. In particular, [6,12] ranks first across all three seeds at H50 and first or tied for first at H75.

DINO layersH25H50H75
[3,12]76.67 ± 1.1525.11 ± 0.778.89 ± 1.92
[6,12]75.56 ± 2.1431.56 ± 3.0812.89 ± 1.92
[9,12]66.00 ± 0.6626.89 ± 1.029.11 ± 1.54
[11,12]70.22 ± 6.4125.33 ± 1.339.78 ± 0.77

(3) Dimension-matched control for MD-JEPA.

We acknowledge that duplicating Layer-12 representations does not increase their information content and is not equivalent to training a wider single-depth model. However, a genuinely doubled-width single-depth backbone would require retraining the pretrained encoder with a modified architecture, introducing changes in parameter count, pretraining dynamics, and computational budget. Simply duplicating features can match the output dimensionality but cannot reproduce the additional features learned by a wider pretrained backbone. As an alternative, we introduce the adjacent-layer combination [11,12], which has the same concatenated dimensionality as [6,12] and uses the same pretrained backbone.

As shown in the above table, [6,12] outperforms [11,12] across all three horizons. These results suggest that combining representations from appropriately separated depths can offer benefits beyond simply concatenating features to increase their dimensionality.

Q1. Can you train JEPA + uniformity with unit norm embeddings without SIGReg and compare it to LeWM to see if this is a valid alternative?

A1. To isolate the contribution of uniformity, we conducted an additional ablation by adding the uniformity objective to DINO-WM while keeping all other settings unchanged.

MethodH25 Success Rate ↑
DINO-WM64.0% (96/150)
DINO-WM + Uniformity66.0% (99/150)

Adding uniformity improves the H25 planning success rate by 2.0%. These results provide preliminary evidence that uniformity can benefit downstream planning.


Q2. Does the x-axis change the observation or only the reconstruction weight in figure 2 and why does the latent model perform better as the noise increases?

A2. Please refer to the reply to W2(1).


Q3. Can you provide a dimensionality matched baseline with more capacity for MD-JEPA?

A3. Please refer to the reply to W2(3).


Q4. Can you explain if Theorem 4.3 is meant to hold under Theorem 3.1's unit-norm constraint?

A4. We apologize for the ambiguity. The original unit-norm condition did not guarantee the attainability of the optimal rank-kk SVD approximation. We have therefore revised Theorems 3.1 and 4.3 and their proofs by replacing pointwise unit-norm constraints with a fixed expected representation-energy assumption:

\mathbb E_{q_a}\|f(x')\|_2^2=\mathrm{const.}$$ Intuitively, this assumption follows the common principle of controlling representation scale through normalization or second-moment regularization (e.g., RMSNorm and VICReg), while allowing state-dependent embedding norms. It can be enforced through global energy normalization. Under deterministic transitions and sufficient encoder–predictor expressivity, the revised proof explicitly constructs a feasible factorization attaining the optimal rank-$k$ SVD approximation. Thus, **Theorem 4.3 holds under the revised fixed-energy assumption, preserving the central insight that increasing representation dimensionality reduces spectral approximation error while potentially increasing sample complexity.** --- **Q5. Could you provide any experiments that directly validate the approximation-sample error trade off by providing an evaluation across different latent dimensionality, k, and different training-set sizes?** **A5.** We conduct additional $k$- and $n$-sweep experiments with results shown in Figures 3, Appendix A.4 in the revised manuscript. - **Training-set size.** As $n$ increases, both latent- and input-level prediction improve. Notably, latent prediction outperforms input-level prediction with 1k–2k samples, whereas input-level prediction overtakes it at 4k samples and achieves substantially higher success rates with more data. These observations are consistent with the approximation–sample error trade-off characterized by Theorems 4.5–4.6. For fixed model capacity, increasing $n$ leaves the approximation error unchanged while reducing the sample-error bound. Input-level prediction may incur greater sample error due to higher model complexity, making latent prediction advantageous with limited data. As $n$ increases, estimation error diminishes, allowing the lower approximation bias of input-level prediction to become more influential. - **Latent dimensionality.** Varying $k$ from 2 to 64 produces a non-monotonic success curve. Planning success increases from 15% at $k=2$ to a peak of 54% at $k=16$, before decreasing to 30% at $k=64$. From a theoretical perspective of Theorems 4.5–4.6, increasing $k$ reduces the spectral approximation term but generally increases model complexity and the sample-error bound. Their competing effects can produce a non-monotonic overall bound, consistent with the observed intermediate optimum. --- Thanks for your insightful and constructive comments. We sincerely hope that our explanations and additional experiments solve your concerns.

Dear Reviewer yc9K,

As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We are sincerely grateful for your time and effort devoted to carefully evaluating our manuscript. To address your remaining concerns, we have provided additional explanations, experiments, and manuscript revision.

We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions and would be delighted to provide additional clarifications.

Best regards, The Authors

Thank you for the response. I don't think A1 addresses my question since DINOWM uses a frozen dino encoder which should not easily collapse and the uniformity term only affects the predictor. The role that it has on learning the encoder, f, of a JEPA model without a regularization term such as SIGReg is not addressed. This experiment again adds uniformity to a pipeline that already works. For A4, thank you for updating and clarifying the proof. However, doesn't the revised proof now rely on deterministic transitions when the formulation in section 3 and experiments in figure 2 are stochastic settings? Could you also explain how the energy constraint is enforced in the actual experiments?

Thank you to the authors for their hard work. It has addressed part of my concerns, but I will be keeping my score.

Reviewer_JFVzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks an important question: how does what a JEPA world model learns in latent space affect subsequent planning? The authors take a spectral approach. They express a prediction objective with a uniformity term as a low-rank factorization of an action-conditioned co-occurrence matrix, then try to relate pretraining error to planning regret in goal-reaching tasks. Their finite-sample analysis suggests a trade-off between approximation error and sample error as latent dimension changes. The experiments have two parts: a comparison of latent prediction and input reconstruction in a noisy 2D point-mass environment, and an evaluation of Multi-Depth JEPA (MD-JEPA), which combines features from different network layers, on PushT.

Critical strengths
  1. I like the question the paper is trying to answer. Much of the JEPA literature focuses on representations themselves; a convincing link from action-conditioned prediction to planning performance would be valuable. An asymmetric co-occurrence matrix also seems a natural way to describe directed state transitions.

  2. The assumptions and full proofs are laid out, so readers can examine the claims closely. The experiments do more than report a single final score: the synthetic study varies both nuisance noise and planning difficulty, while the PushT study includes a dimensionality control that duplicates layer-12 features. In Table 1, the better multi-depth configuration raises success from 64.0% to 77.3% at horizon 25; at horizon 75, the best result rises from 6.7% to 14.0%. These are observations worth following up.

Critical weaknesses
  1. My main concern is the central regret bound. In the proof of Theorem 4.1, the authors assume that E_{x_g}[w(x_g|a)] is equal for each fixed action a, and then drop the term w(x_g|a**) - w(x_g|a_tilde) after taking an expectation. But a** and a_tilde are themselves chosen using the starting state and the goal. Equality of the marginal expectations for fixed actions does not, by itself, imply equality after this goal-dependent action selection. I cannot see why that step goes through as written. The multi-step result and the later finite-sample bounds build on it, so this is a consequential gap.

  2. The deterministic condition in Theorem 4.2 also needs checking. The paper sets w(x,x_plus,a)=1 for the unique successor, although w was previously defined as the joint probability P(x,x_plus|a). With more than one possible current state, those joint probabilities cannot be normalized in that way. I suspect the intended condition is P(x_plus|x,a)=1. If so, the theorem and proof should use that definition consistently.

  3. There is a separate issue in Theorem 4.3. The proof treats the matrices realizable as G(F,a)^T F as though they were all matrices of rank at most k, and then invokes the optimal truncated-SVD error. Yet F comes from a normalized encoder shared across actions, and G is restricted by the predictor's functional form. Unless the authors show that the required rank-k factorization is attainable under these constraints, the spectral tail is at most a lower bound; it cannot simply be equated with the optimal risk.

  4. The experiments support the intuition more than they test the bound itself. There is no measurement of the spectral tail, nor a sweep over sample size or latent dimension that compares predicted and observed trends in planning regret. Figure 2 also contains regimes where input-level prediction is competitive or better, and the discussion could acknowledge that more plainly. The multi-depth PushT table lacks variability estimates across random seeds. Duplicating layer-12 features is a useful control, but it is not a substitute for training a single-depth model with a matched width.

Questions for authors
  1. Can the expectation step involving goal-dependent action selection in Theorem 4.1 be justified rigorously? If a stronger assumption is needed, how would it change Theorems 4.2, 4.5, and 4.6?
  2. Is the deterministic condition in Theorem 4.2 meant to apply to P(x_plus|x,a), rather than the joint probability w(x,x_plus,a)? Could the definitions and proof be made consistent?
  3. Given encoder normalization, parameter sharing across actions, and the restricted predictor class, when can the model attain the best unconstrained rank-k SVD approximation for all actions at once?
  4. Could the authors report seed-level variation for MD-JEPA, add a trainable width-matched single-depth baseline, and vary training-set size and latent dimension to test the proposed trade-off directly?
4 comments

We thank the reviewer for recognizing the value of connecting JEPA representations to planning performance, and the insights from our theoretical and empirical analyses. We address your concerns below.


W1/Q1. Can the expectation step involving goal-dependent action selection in Theorem 4.1 be justified rigorously? If a stronger assumption is needed, how would it change Theorems 4.2, 4.5, and 4.6?

A1. Yes, a little bit stronger assumption is required. We address this by introducing Approximately Action-Invariant Marginals: $ \mathbb E_{x_g}\left[ \sup_{a,a'\in\mathcal A} |w(x_g|a)-w(x_g|a')| \right]\leq\varepsilon. $

This assumption does not require identical transition dynamics across actions. We provide two illustrative examples: (i) a cyclic grid with uniform sampling, where ε=0\varepsilon=0; and (ii) a non-cyclic grid with clipped, lazy transitions, where the discrepancy is confined to boundary states and ε=O(L−3)\varepsilon=O(L^{-3}) under uniformly sampled goals. Please refer to Section 4.1 in our revised manuscript for more details.

Impact on subsequent theorems. Under the original bounded-score and action-ranking conditions, the revised Theorem 4.1 incurs an additive ε\varepsilon term, which propagates to Theorem 4.5. For the multi-step results (Theorems 4.2 and 4.6), the discrepancy can accumulate as TεT\varepsilon under appropriate stepwise conditions.

Overall, the central connection between spectral approximation error and planning performance is preserved under the clarified assumptions.


W2/Q2. Is the deterministic condition in Theorem 4.2 meant to apply to P(x_plus|x,a), rather than the joint probability w(x,x_plus,a)? Could the definitions and proof be made consistent?

A2. We appologize for this typographical error in the original manuscript. The deterministic condition should be imposed on the conditional transition probability, rather than the joint co-occurrence probability. We correct the definition as follows: for every state xx and action aa, there exists a unique successor Ta(x)T_a(x) such that P(x′∣x,a)=1{x′=Ta(x)}P(x'|x,a)=\mathbf 1\{x'=T_a(x)\}. The proof and conclusion of Theorem 4.2 remain unchanged.


W3/Q3. Given encoder normalization, parameter sharing across actions, and the restricted predictor class, when can the model attain the best unconstrained rank-k SVD approximation for all actions at once?

A3. We acknowledge that the original unit-norm condition did not guarantee the attainability of the optimal rank-kk SVD approximation under pointwise unit-norm constraints. We have therefore revised Theorems 3.1 and 4.3 and their proofs by adopting a fixed expected representation-energy assumption: $ \mathbb E_p|g(f(x),a)|_2^2+ \mathbb E_{q_a}|f(x')|_2^2=\mathrm{const.}$

Intuitively, this assumption follows the common principle of controlling representation scale through normalization or second-moment regularization (e.g., RMSNorm and VICReg), but imposes a weaker, distribution-level constraint that allows state-dependent embedding norms. It can be enforced through global energy normalization.

In the revised proof, we first establish the optimal SVD approximation error as a lower bound. Under deterministic transitions and sufficient encoder–predictor expressivity, we then explicitly construct a feasible factorization attaining this bound while satisfying the fixed energy constraint. These revisions preserve the exact JEPA–spectral objective equivalence and our central approximation–sample error trade-off: increasing representation dimensionality reduces spectral approximation error while potentially increasing sample complexity, under the revised assumptions. (Please refer to the theorem and proof sections of our revised manuscript for more details.)

W4/Q4. Could the authors report seed-level variation for MD-JEPA, add a trainable width-matched single-depth baseline, and vary training-set size and latent dimension to test the proposed trade-off directly?

A4. We have expanded our experiments with additional sweeps over training-set size and latent dimensionality, three-seed PushT evaluations, and a dimension-matched control for multi-depth representations.

(1) Additional experiments on training-set size nn and latent dimensionality kk.

The results are shown in Figures 3, Appendix A.4 in the revised manuscript.

  • Training-set size. As nn increases, both latent- and input-level prediction improve. Notably, latent prediction outperforms input-level prediction with 1k–2k samples, whereas input-level prediction overtakes it at 4k samples and achieves substantially higher success rates with more data. These observations are consistent with the approximation–sample error trade-off characterized by Theorems 4.5–4.6. For fixed model capacity, increasing nn leaves the approximation error unchanged while reducing the sample-error bound. Input-level prediction may incur greater sample error due to higher model complexity, making latent prediction advantageous with limited data. As nn increases, estimation error diminishes, allowing the lower approximation bias of input-level prediction to become more influential.
  • Latent dimensionality. Varying kk from 2 to 64 produces a non-monotonic success curve. Planning success increases from 15% at k=2k=2 to a peak of 54% at k=16k=16, before decreasing to 30% at k=64k=64. From a theoretical perspective of Theorems 4.5–4.6, increasing kk reduces the spectral approximation term but generally increases model complexity and the sample-error bound. Their competing effects can produce a non-monotonic overall bound, consistent with the observed intermediate optimum.

(2) Clarification of input-level versus latent-level prediction.

We agree that latent-level prediction is not universally superior. As shown in Figure 2, input-level prediction remains competitive with, and sometimes outperforms, latent-level prediction in certain easier or lower-noise settings. However, as the planning horizon and unpredictable observation noise increase, latent-level prediction becomes more robust, with the most pronounced advantage observed at 25 required planning steps. We have revised the discussion of Figure 2.

(3) Multi-seed evaluation of PushT.

We repeat the PushT experiments using three independently trained random seeds. For DINO, each configuration is evaluated on 150 fixed queries per seed and planning horizon. The results below report mean success rate ± standard deviation across training seeds (in percentage points).

DINO layersH25H50H75
[3,12]76.67 ± 1.1525.11 ± 0.778.89 ± 1.92
[6,12]75.56 ± 2.1431.56 ± 3.0812.89 ± 1.92
[9,12]66.00 ± 0.6626.89 ± 1.029.11 ± 1.54
[11,12]70.22 ± 6.4125.33 ± 1.339.78 ± 0.77

The [6,12] combination achieves the highest mean success rates at H50 and H75, ranking first across all three seeds at H50 and first or tied for first at H75. At H25, [3,12] performs best. These results suggest that the optimal layer combination depends on the planning horizon.

(4) Dimension-matched control for multi-depth representations.

We agree that duplicating Layer-12 features is not equivalent to training a width-matched single-depth model. However, a genuinely doubled-width single-depth backbone would require retraining the pretrained encoder with a modified architecture, introducing changes in parameter count, pretraining dynamics, and computational budget. Simply duplicating features can match the output dimensionality, but cannot reproduce the additional features learned by a wider pretrained backbone. As an alternative, we introduce a dimension-matched control using the adjacent-layer combination [11,12], which has the same concatenated feature dimensionality as [6,12]. Both configurations use features extracted from the same pretrained backbone, avoiding confounding changes in backbone architecture and pretraining. The three-seed evaluation results are summarized below :

DINO layersH25H50H75
[6,12]75.56 ± 2.1431.56 ± 3.0812.89 ± 1.92
[11,12]70.22 ± 6.4125.33 ± 1.339.78 ± 0.77

The [6,12] combination achieves higher mean success rates across all three horizons. These results suggest that combining appropriately separated representation depths can provide benefits beyond simply increasing dimensionality.


Thanks for your insightful and constructive comments. We sincerely hope that our explanations and additional experiments solve your concerns.

Dear Reviewer JFVz,

As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We are sincerely grateful for your time and effort devoted to carefully evaluating our manuscript. To address your remaining concerns, we have provided additional explanations, experiments, and manuscript revision.

We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions and would be delighted to provide additional clarifications.

Best regards, The Authors

Reviewer_JFVzOpenReview ↗

Official Comment by Reviewer JFVz

Thanks for the response and additional experiments. A2 resolves my question about determinism, and the three-seed PushT results and the training-set-size and latent-dimension sweeps strengthen the empirical analysis. The sweep results are consistent with the proposed approximation–sample error trade-off. (1) For W1/Q1, I’d like to see how the revised marginal assumption bounds the expectation when both actions depend on the goal. (2) For W3/Q3, the expected-energy constraint addresses normalization. Can one shared encoder and a predictor in the stated class attain the rank-kk optimum for all actions simultaneously?

As a result, I’m keeping my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.