acceptodds
Under review as a conference paper at ICLR 2027

Nested Latent Representations for Size-Flexible End-to-End World Models

22 views

Abstract

Latent world models for planning typically assume a fixed representation size. While recent work, such as SubJEPA, considered operating in subspaces of smaller dimensionality, they require additional task-specific tuning of the subspace granularity. In this work, we ask whether world models can learn representations that remains useful across multiple post-hoc embedding sizes. As an answer, we introduce ElasticJEPA which jointly trains a world model at multiple nested granularities. ElasticJEPA combines subspace regularization with prediction losses applied to the same prefixes used at inference. The novel prediction loss objective induces a coordinate-supervision profile that is naturally monotonically-ordered. ElasticJEPA also ensures that privileging prefixes are retained under truncation. With such a construction, a single trained model can be evaluated at multiple representation sizes with no need for retraining. Importantly, inference-time evaluation can also include widths not supervised during training. Across four environments, ElasticJEPA is competitive with prior methods at full width, retains most of its performance under substantial truncation, and degrades less than baselines truncated to the same size, most clearly at the smallest widths.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
nested representations, jepa, world models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AHCQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • studies whether JEPA world models can support multiple latent dimensions without retraining
  • proposes ElasticJEPA, combining nested SIGReg with prediction losses on progressively smaller embedding prefixes
  • preserves much of the planning performance down to 6 dimensions (from 192), outperforming truncated LeWM and Sub-JEPA baselines, especially on PushT and Two-Room
Critical strengths
  • simple extension of Sub-JEPA and Matryoshka learning, with no additional trainable parameters!!!
  • clear analysis showing how nested prediction losses induce nonuniform supervision across latent coordinates
  • good factorial ablations separating nested regularization from nested prediction
  • evaluates unseen embedding widths, showing interpolation beyond explicitly trained granularities
  • interesting linear versus nonlinear probing analysis of information retained after truncation
Critical weaknesses
  • the approach largely combines existing Matryoshka learning with Sub-JEPA so overall I would ay the methodological novelty is limited on that angle
  • truncation only affects the planning cost... the predictor still operates at full width, so there is no actual inference speedup
  • full-width performance is below Sub-JEPA on all four environments?
  • factorial ablations cover only two environments but that's not the worst
  • the theoretical result establishes ordered loss weighting, not that earlier dimensions contain more useful information
  • training is up to 33% slower, with limited demonstrated practical benefits beyond smaller stored representations, this is the most painful part for practicing scaling JEPAs
Questions for authors

I have a bunch of very short questions to help me

  • what is the practical benefit of reducing the planning-cost dimension when the encoder and predictor still run at full width?
  • how does ElasticJEPA compare against standard Matryoshka prediction training without nested SIGReg, across all four environments?
  • why does full-width performance consistently trail Sub-JEPA?
  • can you demonstrate actual memory or runtime savings in a realistic deployment, rather than just reduced planning-cost dimensionality?
3 comments
AuthorsOpenReview ↗

Reply to reviewer AHCQ 1/2

We thank the reviewer for the helpful comments and constructive suggestions.

W1. Methodological novelty. As detailed in C1 of our common response, combining Matryoshka learning with JEPA is not straightforward: unlike conventional supervised Matryoshka learning, JEPA learns its own prediction targets. Therefore, naively applying nested prediction with only full-space regularization results in a collapse of early dimensions; on PushT at 6 dimensions, this naive approach achieves 36.0% versus ElasticJEPA's 78.0%. The same holds across all four environments (36.0 / 77.7 / 53.0 / 56.0 against 78.0 / 90.3 / 86.0 / 66.3). In these models, the first 6-d block has total variance below 0.01, against about 6 for a healthy block, and a full-width SIGReg barely registers it. While nested regularization prevents this collapse, it alone doesn't impose dimension ordering. ElasticJEPA resolves these complementary failure modes. The revised manuscript includes this analysis and the completed four-environment factorial ablations (C1, C4; Section 5.2, Tables 8–9).

W2. Inference speedup. As discussed in C1 and C2 of our common response, ElasticJEPA focuses on achieving representation-size flexibility (which is valuable for downstream planning, as remarked by Reviewer d2SW) rather than directly accelerating the existing full-width predictor. Our additional frozen-encoder experiments (Section 5.7, Table 4) demonstrate the practical value of compact representations for downstream planning, alongside speedups obtainable through predictor compression.

W3, Q3. Full-width performance. Regarding your comment "Why does full-width performance consistently trail Sub-JEPA?", please note that Sub-JEPA optimizes a single, task-tuned granularity, whereas ElasticJEPA supports six granularities jointly. We hypothesize that this introduces a specialization–flexibility trade-off. With all three models trained using the same settings, full-width success is 87.0% (LeWM), 91.4% (ElasticJEPA), and 95.0% (Sub-JEPA) on Two-Room (Section 5.4). We also tested weighting the finer granularities more (square-root weights) on all four environments. On Two-Room, this gives 93.3%, but the full-width change is within evaluation noise in every environment (-2.3, +1.3, +0.3, -1.7 points; Appendix C, Table 6). Thus, this test neither confirms nor rules out the trade-off. The Two-Room ordering above is our direct support for it.

W4. Factorial ablations. The revised manuscript extends these experiments from two to all four environments, confirming that nested regularization and nested prediction both contribute to truncation robustness, with Reacher and Cube showing little sensitivity to the latter (C1, C4; Figure 3, Table 8).

AuthorsOpenReview ↗

Reply to reviewer AHCQ 2/2

Continuing our response to the reviewer:

W5. Ordered supervision vs. useful information. The proposition formally establishes ordered loss weighting. We have added two direct empirical tests (Section 5.6, Figure 2(b)) demonstrating that earlier coordinates also retain more task-relevant information:

  • Planning success, first vs. last 6-d block: PushT 78.0% vs. 52.0%; Two-Room 90.3% vs. 69.0%; Reacher 86.0% vs. 78.3%; Cube 66.3% vs. 60.7%.
  • Nonlinear probe (R²), first vs. last block: PushT 0.81 vs. 0.11; Two-Room 0.96 vs. 0.54; Reacher 0.99 vs. 0.72; Cube 0.94 vs. 0.28.

Baselines without nested prediction don't exhibit a comparable ordering. These experiments provide direct empirical evidence of coordinate privilege, most strongly on PushT and Two-Room, complementing the theoretical supervision-ordering result.

W6. Training overhead. The original overhead primarily arose from redundant SIGReg projections across 63 subspaces. Our revised implementation reduces training overhead from 26.1% to 3.1% in time and from 58% to 1.2% in peak memory over LeWM (checked on PushT and Two-Room, one training run each; the main results use the original implementation), while preserving strong small-width performance. At 6 dimensions, the efficient implementation achieves 74.7% versus retrained LeWM's 60.3% on PushT and 88.0% versus 63.3% on Two-Room (C3; Appendix G, Tables 11–12).

Q1. Practical benefit of reduced planning-cost dimensionality. As discussed in points 1 and 2 of our first comment (as well as C1–C2 of our common response), the principal benefit is reusing one pretrained representation model across downstream components with different capacity requirements, rather than accelerating the current CEM cost computation. This application was also highlighted by Reviewer d2SW. The revised manuscript now explicitly motivates this use case and includes supporting frozen-encoder experiments (Introduction; Section 5.7).

Q2. Standard Matryoshka baseline. This corresponds exactly to the (RF+LNP)(R_F + \mathcal{L}_{NP}) condition in our factorial ablation. The revised manuscript evaluates this condition across all four environments (the original manuscript had it for two) (Figure 3, Table 8). Its substantially weaker small-width performance confirms that nested prediction alone is insufficient (please also see C1 and C4 of our common response and W1 of our first comment).

Q4. End-to-end planning speedup. Full-width prediction currently dominates CEM planning runtime. Therefore, latent truncation alone doesn't yield an end-to-end speedup. Although we don't show end-to-end deployment savings in a setting where storage or communication is the bottleneck, ElasticJEPA provides representations up to 32× smaller for storage and transmission. This is consequential, as our frozen-encoder experiments (Section 5.7, Table 4) show that these representations remain useful for smaller downstream predictors (supporting the remark by Reviewer d2SW). Predictor compression independently achieves 2.3× GPU and 5.9× CPU speedups, although these speedups are not attributable to latent truncation alone. The revised implementation also reduces training overhead to near-baseline levels (C2–C3).

We hope our responses address the reviewer's concerns and questions.

AuthorsOpenReview ↗

Official comment by authors

Dear reviewer AHCQ,

We have posted the rebuttal responses to all the issues raised. We are willing to address any further queries and would be happy to do so before the discussion period ends.

In particular, the revised paper now includes the factorial ablation on all four environments, with an analysis of why nested prediction alone fails (Section 5.2, Figure 3, Tables 8–9), a direct test that earlier coordinates hold more information (Section 5.6, Figure 2(b)), smaller predictors on a frozen encoder with measured speedups (Section 5.7, Table 4), and a cheaper nested SIGReg that cuts the training overhead to 3.1% (Appendix G).

If the work and responses have resolved your concerns, we would be grateful if you could reconsider updating your assessment. Thank you for your time.

Reviewer_d2SWOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ElasticJEPA, which trains multi-size representations with no additional compute that can be used for planning at any intermediate sized representation, even if it were untrained at that size. The authors empirically validate their claims and present promising results to this end.

Critical strengths

The paper is well written and generally easy to follow. The methods used are reasonable and sound and are the extension of Matryoshka representations to World Models. The experimental sections are sufficiently thorough and discussions are intuitive and laid out well.

Critical weaknesses

This paper assumes that the reader is fully aware and knowledgeable about SubJEPA. I think that a more in-depth restatement of SubJEPA and its workings would help this be a standalone paper.

I also believe that there should be some text in the introduction as to why such a multi-sized representation is useful in practice to support the motivation to study this problem. Outside of tasks being representable in lower dimensional subspaces, the potential re-use of the same representation model to train multiple fidelities of downstream planners at different compute capacities is an important use case for this work.

Questions for authors

I have no particular questions. I thank the authors for their work.

5 comments
AuthorsOpenReview ↗

Reply to reviewer d2SW

We thank the reviewer for the positive assessment and constructive suggestions.

1. Sub-JEPA background. We have expanded the background to explain LeWM's latent prediction objective and SIGReg anti-collapse regularization, Sub-JEPA's fixed-KK subspace regularization, and ElasticJEPA's extension through a shared nested orthogonal basis and prediction losses applied to retained prefixes (Section 3.1). We hope that the presentation is now more self-contained.

2. Practical motivation. We appreciate the reviewer highlighting multi-fidelity downstream planning as an important use case. This wasn't sufficiently highlighted in the original manuscript, but the revised Introduction now explicitly incorporates this motivation (second paragraph). Our new frozen-encoder experiments directly demonstrate the benefits of ElasticJEPA's compact representations for smaller downstream predictors (C1–C2; Section 5.7, Table 4) .

We hope our response answers your concerns and questions. We will be happy to answer any other questions that you might have.

Reviewer_d2SWOpenReview ↗

Response to Rebuttal

I am inclined to drop my score to 3 post rebuttal since I am confused a by the new results that the authors present.

The Width Read column in Table 10 is also very confusing to read. I read the first two entries as "192 minus d" at first glance. Additionally, if you are going to be using these results as a major defense to the criticisms of my fellow reviewers, relegating this to the appendix with no mention of it in the main paper does not make for a sound choice.

Also, some text on the mismatch between LeWM performance in Table 10 and 14 should be clarified further in text. Why this is expected is not clear.

  • Are both the 6 dim models the same? If yes, why do they differ in performance? If not, what is their architectural difference that can explain the difference in performance?
  • I think both the 192 dim retrained models are identical. Yet, they are reported with wildly different numbers.
AuthorsOpenReview ↗

Response to Reviewer d2SW

First of all, thank you for the prompt reply and careful reading. Below, we clarify a few things briefly that caused the confusion.

The Width Read column in Table 10 is also very confusing to read. I read the first two entries as "192 minus d" at first glance.

That was actually a hyphen (192-d), meaning the full 192 dimensions, but due to LaTeX math mode, it turned into a minus sign. We sincerely regret this inadvertent error. This is fixed in the revision: the column, now in Table 4, reads 'all 192', 'first 24', 'first 6', '24-d projection' and '6-d projection'.

Additionally, if you are going to be using these results as a major defense to the criticisms of my fellow reviewers, relegating this to the appendix with no mention of it in the main paper does not make for a sound choice.

We acknowledge this suggestion and have included the table in the main paper, along with a brief description in Section 5.7, thereby replacing the earlier short reference in Section 6.

Are both the 6 dim models the same?

No. There are three different 6-d setups:

Table 14, LeWM truncated to 6-d: the original encoder and original predictor; only the planning cost uses 6 dimensions. Table 4, 6-d: the same encoder, frozen, with a new small predictor that reads only 6 dimensions. Table 14, LeWM trained at 6-d: a separate LeWM, encoder and predictor, trained from scratch with a 6-d latent.

If not, what is their architectural difference that can explain the difference in performance?

The new predictor sees only those 6 coordinates, so on PushT it cannot model the dynamics (8.0, against 60.3 when the predictor sees all 192); on Two-Room the two are equal (62.7 against 63.3). The model trained at 6-d has an encoder trained to keep the needed information in 6 coordinates.

I think both the 192 dim retrained models are identical.

They share the encoder but not the predictor. This is now mentioned in Table 4. Table 14 uses LeWM’s own predictor (h = 192); Table 4 uses a new one with 1/14 of the multiply-adds (h = 48). The small predictor loses about 17 points on PushT and nothing on Two-Room. ElasticJEPA shows a similar drop (90.7 to 71.0), so the drop is due to the predictor, not the encoder.

Section 5.7 now explains the difference between Table 4 and Table 14.

We hope this addresses all doubts and thank the reviewer again for pointing it out.

If the work and responses have resolved your concerns, we would be grateful if you could reconsider updating your assessment. Thank you for your time.

Reviewer_dFLgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper is a very simple tweak to LeWM/ SubJepa: use the Matroyshka trick to train a latent predictor ||z-z'|| using the first k dimensions of z for different k=1:K (eqn 2), then add add a SIGreg loss to the set of all such embeddings to prevent collapse (eqn 1). On 4 very simple image-based 2d continuous control problems (using CMA-based MPC), they show that their method (whuch trains the WM using 192 dimensions then extracts a smaller subset at runtime ) works better with smaller latent z than LeWM and SubJep trained directly on smaller representations.

Critical strengths

none

Critical weaknesses

They show how to combine Matroshyka embeddings with LeJepa, allowing them to reduce the dimensionality K from 192d to ~24d without incurring much extra loss. However, this is not a very useful thing to do, since 192d is already extremely small (and hence fast to train). Also, as they say in their own limitations section, 'Wall-clock planning cost does not change with K in this implementation as the predictor always runs at full size'. In addition, their results are on very simple domains and they did not do proper factorial experiments or ablations.

Questions for authors

none

2 comments
AuthorsOpenReview ↗

Reply to reviewer dFLg

We thank the reviewer for recognizing the paper's clarity and soundness. We address the specific concerns below.

  1. "This paper is a very simple tweak to LeWM/SubJepa."

We respectfully disagree that ElasticJEPA is a "very simple tweak to LeWM/SubJepa". As also discussed in C1 in our common response, such a characterization does not consider the technical challenge that naively applying Matryoshka prediction losses to a JEPA does not reliably produce useful truncated representations because of the collapse of early dimensions. As evidence, we have included additional experiments in the revised manuscript that demonstrate this failure mode (Section 5.2, Table 9). Moreover, the factorial ablations on all four datasets establish the complementary roles of nested prediction and nested regularization (C1, C4; Table 8). Nested prediction alone shrinks the early dimensions, and nested regularization alone does not order them. These two failure modes, and the fact that a full-width regularizer does not detect the first, are what the paper contributes beyond the combination itself.

  1. "On 4 very simple image-based 2d continuous control problems."

"Their results are on very simple domains."

The 4 environments follow the established LeWM/Sub-JEPA evaluation protocol, ensuring controlled, like-for-like comparisons with the two methods we directly extend. Importantly, OGB-Cube involves 3-D robotic manipulation, rather than 2-D control. These tasks also pose meaningful challenges at small widths: on PushT, a LeWM retrained with our settings drops from 91.0% to 60.3%, whereas ElasticJEPA goes from 89.4% to 74.3% (mean of three training runs; Table 14).

  1. "Than LeWM and SubJepa trained directly on smaller representations."

The original manuscript had truncated and evaluated baselines, we agree that comparison against models trained directly at smaller widths is important. We therefore added experiments training LeWM baselines from scratch at smaller embedding dimensions. At 24-d, the directly trained model matches ElasticJEPA on all four environments. At 6-d, it fails on PushT (23.0% against 74.3%), is level on Two-Room, 3.2 points lower on Reacher and 8.0 points higher on Cube (1.7 SE). A retrained LeWM truncated to 6-d is 14.0 / 26.0 / 7.9 / 6.7 points below ElasticJEPA (Section 5.3, Table 14).

  1. "However, this is not a very useful thing to do, since 192d is already extremely small (and hence fast to train)."

We respectfully disagree that reducing an already-small 192-dimensional embedding has little practical value. We agree that a 192-d model is cheap to train, so we tested the direct alternative: LeWM trained from scratch at the small width (in the submission, the small-width baselines were 192-d models truncated after training). As discussed above, directly trained smaller models do not consistently match ElasticJEPA at 6-d and require one training run per width. A retrained LeWM truncated to 6-d is also substantially worse. Thus, the contribution is not merely compression, but learning once and supporting multiple representation sizes without retraining (a point also acknowledged by Reviewer d2SW), now at only 3.1% extra training time (C3).

  1. "Also, as they say in their own limitations section, 'Wall-clock planning cost does not change with K in this implementation as the predictor always runs at full size'."

This observation is correct for our current CEM implementation, which retains the full-width predictor. Thus, truncating the planning-cost representation alone does not accelerate planning. However, our new frozen-encoder experiments demonstrate the utility of ElasticJEPA's compact representations for smaller downstream predictors (in line with the benefits mentioned by Reviewer d2SW). Predictor compression separately yields substantial CPU/GPU acceleration (C2). The revised manuscript distinguishes these two effects explicitly (Sections 5.7 and 6; Table 4).

  1. "They did not do proper factorial experiments or ablations."

We have now updated the revised manuscript to report the factorial ablation across all four environments (Section 5.2, Figure 3, Table 8) along with additional ablations (C4). Removing nested regularization degrades small-width performance; removing nested prediction also reduces truncation robustness, except on Reacher and Cube, where the difference is marginal. These experiments directly isolate the contributions of both objective components. (C1, C4).

  1. "reduce the dimensionality K from 192d to ~24d "

It might have been unclear, but K denotes nesting granularity, while the retained dimensionality is d=192/Kd=192/K (Section 3.3). The smallest representation is 6-dimensional (32× smaller), rather than approximately 24-dimensional. We also evaluate four previously unseen widths (144, 72, 36, and 18) (Section 5.1, Figure 2(a)).

We hope our responses address the reviewer's concerns and questions.

AuthorsOpenReview ↗

Official comment by authors

Dear reviewer dFLg,

We have posted the rebuttal responses to all the issues raised. We are willing to address any further queries and would be happy to do so before the discussion period ends.

If the work and responses have resolved your concerns, we would be grateful if you could reconsider updating your assessment. Thank you for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response to All reviewers 1/2

We thank all reviewers for their feedback. Here, we first provide a common response to some of the concerns/points raised by the reviewers, and then we respond to each reviewer’s comments separately. The points discussed in our rebuttal, wherever relevant, are also incorporated in the revised manuscript. All changes in the revised PDF are marked in blue.

C1. Methodological contribution and practical significance. We respectfully disagree with the characterization of ElasticJEPA as a straightforward application of Matryoshka learning to existing JEPAs. In this regard, it is pertinent to highlight the following points:

  • Unlike conventional supervised Matryoshka learning where the labels are fixed, JEPA learns its own prediction targets and thus a Matryoshka-style learning of JEPA is not straightforward; a naive attempt at it can reduce prefix prediction losses by shrinking both predicted and target coordinates, potentially causing prefix-level collapse. We have included new experiments to demonstrate the consequences of this failure mode: on PushT at 6 dimensions, nested prediction with full-space regularization achieves only 36.0% success versus ElasticJEPA's 78.0%. The same holds on all four environments (36.0 / 77.7 / 53.0 / 56.0 against 78.0 / 90.3 / 86.0 / 66.3). In these models the first 6-d block has total variance below 0.01, against about 6 for a healthy block, and a full-width SIGReg barely registers it (Section 5.2, Tables 8-9). Conversely, nested regularization alone doesn't induce coordinate ordering (Sections 5.2 and 5.6). ElasticJEPA identifies and resolves these complementary failure modes through jointly nested prediction and regularization.

  • The resulting capability offered by ElasticJEPA has important practical implications. As Reviewer d2SW aptly observed:

    "The potential re-use of the same representation model to train multiple fidelities of downstream planners at different compute capacities is an important use case for this work."

    Indeed, ElasticJEPA enables a single pretrained model to provide useful representations from 192 down to 6 dimensions (32× smaller), including previously unseen widths, without retraining.

    The primary contribution is reusable, size-flexible representations, not merely acceleration of the existing full-width predictor. These representations can support downstream components with different capacity requirements while reducing latent storage and communication costs. In the revised version, we now explicitly discuss these aspects more clearly (Introduction, second paragraph; Sections 5.7 and 6).

C2. Additional downstream-planning experiments. Following the remark by Reviewer d2SW, we freeze pretrained encoders and train smaller downstream predictors. On Two-Room at 6 dimensions, a small predictor on ElasticJEPA’s prefix reaches 89.7% against 62.7% on retrained LeWM’s projection (24-d: 91.7% against 80.3%). On PushT the small predictor itself is the bottleneck: it loses about 20 points even at full width (90.7% to 71.0%) and fails at 6-d for both encoders. Furthermore, reducing predictor hidden width from 192 to 48 yields 2.3× GPU and 5.9× CPU speedups. These speedups arise from reducing predictor capacity, not latent truncation alone; importantly, ElasticJEPA provides substantially more useful compact representations for these downstream predictors. The revised manuscript includes these results (Section 5.7, Table 4; Appendix F).

C3. Efficient training. We identified redundant computation in nested SIGReg and introduced an efficient implementation using 1,024 random projection directions per nesting level (1024/K) rather than per subspace. This reduces training overhead from 26.1% to 3.1% in time and from 58% to 1.2% in peak memory relative to LeWM (checked on PushT and Two-Room, one training run each; the main results use the original implementation), while preserving strong truncation performance. The revised manuscript reports the implementation and results (Section 6; Appendix G, Tables 11–12).

C4. Expanded experimental validation. Following the feedback from the reviewers, in the revised manuscript, we include additional factorial ablations across all four environments (original manuscript had it on two environments) (Figure 3, Table 8), six-seed truncated-baseline evaluations (Table 1), and additional experiments on dimension ordering (Section 5.6), training baseline on smaller embeddings (Section 5.3, Table 14), nonlinear probing (Section 5.6), smaller downstream predictors (Section 5.7, Table 4), and the choice of orthogonal basis (Appendix H). These provide stronger empirical evidence for both the mechanism and the utility of ElasticJEPA.

AuthorsOpenReview ↗

Common Response to All reviewers 2/2

Factorial ablation: Complementary roles of nested prediction and regularization.

Method (6-d)PushTTwo-RoomReacherCube
Nested prediction, full-width regularizer36.0 ± 3.877.7 ± 3.353.0 ± 3.256.0 ± 3.3
Nested regularizer, full-width prediction61.3 ± 3.067.7 ± 2.682.7 ± 1.161.0 ± 2.5
ElasticJEPA (both nested)78.0 ± 2.290.3 ± 1.086.0 ± 2.566.3 ± 3.8

One training run per model; mean ± SE over 6 evaluation seeds. All six widths are reported in Table 8 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.