- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- studies whether JEPA world models can support multiple latent dimensions without retraining
- proposes ElasticJEPA, combining nested SIGReg with prediction losses on progressively smaller embedding prefixes
- preserves much of the planning performance down to 6 dimensions (from 192), outperforming truncated LeWM and Sub-JEPA baselines, especially on PushT and Two-Room
Critical strengths
- simple extension of Sub-JEPA and Matryoshka learning, with no additional trainable parameters!!!
- clear analysis showing how nested prediction losses induce nonuniform supervision across latent coordinates
- good factorial ablations separating nested regularization from nested prediction
- evaluates unseen embedding widths, showing interpolation beyond explicitly trained granularities
- interesting linear versus nonlinear probing analysis of information retained after truncation
Critical weaknesses
- the approach largely combines existing Matryoshka learning with Sub-JEPA so overall I would ay the methodological novelty is limited on that angle
- truncation only affects the planning cost... the predictor still operates at full width, so there is no actual inference speedup
- full-width performance is below Sub-JEPA on all four environments?
- factorial ablations cover only two environments but that's not the worst
- the theoretical result establishes ordered loss weighting, not that earlier dimensions contain more useful information
- training is up to 33% slower, with limited demonstrated practical benefits beyond smaller stored representations, this is the most painful part for practicing scaling JEPAs
Questions for authors
I have a bunch of very short questions to help me
- what is the practical benefit of reducing the planning-cost dimension when the encoder and predictor still run at full width?
- how does ElasticJEPA compare against standard Matryoshka prediction training without nested SIGReg, across all four environments?
- why does full-width performance consistently trail Sub-JEPA?
- can you demonstrate actual memory or runtime savings in a realistic deployment, rather than just reduced planning-cost dimensionality?
3 comments
Reply to reviewer AHCQ 2/2
Continuing our response to the reviewer:
W5. Ordered supervision vs. useful information. The proposition formally establishes ordered loss weighting. We have added two direct empirical tests (Section 5.6, Figure 2(b)) demonstrating that earlier coordinates also retain more task-relevant information:
- Planning success, first vs. last 6-d block: PushT 78.0% vs. 52.0%; Two-Room 90.3% vs. 69.0%; Reacher 86.0% vs. 78.3%; Cube 66.3% vs. 60.7%.
- Nonlinear probe (R²), first vs. last block: PushT 0.81 vs. 0.11; Two-Room 0.96 vs. 0.54; Reacher 0.99 vs. 0.72; Cube 0.94 vs. 0.28.
Baselines without nested prediction don't exhibit a comparable ordering. These experiments provide direct empirical evidence of coordinate privilege, most strongly on PushT and Two-Room, complementing the theoretical supervision-ordering result.
W6. Training overhead. The original overhead primarily arose from redundant SIGReg projections across 63 subspaces. Our revised implementation reduces training overhead from 26.1% to 3.1% in time and from 58% to 1.2% in peak memory over LeWM (checked on PushT and Two-Room, one training run each; the main results use the original implementation), while preserving strong small-width performance. At 6 dimensions, the efficient implementation achieves 74.7% versus retrained LeWM's 60.3% on PushT and 88.0% versus 63.3% on Two-Room (C3; Appendix G, Tables 11–12).
Q1. Practical benefit of reduced planning-cost dimensionality. As discussed in points 1 and 2 of our first comment (as well as C1–C2 of our common response), the principal benefit is reusing one pretrained representation model across downstream components with different capacity requirements, rather than accelerating the current CEM cost computation. This application was also highlighted by Reviewer d2SW. The revised manuscript now explicitly motivates this use case and includes supporting frozen-encoder experiments (Introduction; Section 5.7).
Q2. Standard Matryoshka baseline. This corresponds exactly to the condition in our factorial ablation. The revised manuscript evaluates this condition across all four environments (the original manuscript had it for two) (Figure 3, Table 8). Its substantially weaker small-width performance confirms that nested prediction alone is insufficient (please also see C1 and C4 of our common response and W1 of our first comment).
Q4. End-to-end planning speedup. Full-width prediction currently dominates CEM planning runtime. Therefore, latent truncation alone doesn't yield an end-to-end speedup. Although we don't show end-to-end deployment savings in a setting where storage or communication is the bottleneck, ElasticJEPA provides representations up to 32× smaller for storage and transmission. This is consequential, as our frozen-encoder experiments (Section 5.7, Table 4) show that these representations remain useful for smaller downstream predictors (supporting the remark by Reviewer d2SW). Predictor compression independently achieves 2.3× GPU and 5.9× CPU speedups, although these speedups are not attributable to latent truncation alone. The revised implementation also reduces training overhead to near-baseline levels (C2–C3).
We hope our responses address the reviewer's concerns and questions.
Official comment by authors
Dear reviewer AHCQ,
We have posted the rebuttal responses to all the issues raised. We are willing to address any further queries and would be happy to do so before the discussion period ends.
In particular, the revised paper now includes the factorial ablation on all four environments, with an analysis of why nested prediction alone fails (Section 5.2, Figure 3, Tables 8–9), a direct test that earlier coordinates hold more information (Section 5.6, Figure 2(b)), smaller predictors on a frozen encoder with measured speedups (Section 5.7, Table 4), and a cheaper nested SIGReg that cuts the training overhead to 3.1% (Appendix G).
If the work and responses have resolved your concerns, we would be grateful if you could reconsider updating your assessment. Thank you for your time.
Reply to reviewer AHCQ 1/2
We thank the reviewer for the helpful comments and constructive suggestions.
W1. Methodological novelty. As detailed in C1 of our common response, combining Matryoshka learning with JEPA is not straightforward: unlike conventional supervised Matryoshka learning, JEPA learns its own prediction targets. Therefore, naively applying nested prediction with only full-space regularization results in a collapse of early dimensions; on PushT at 6 dimensions, this naive approach achieves 36.0% versus ElasticJEPA's 78.0%. The same holds across all four environments (36.0 / 77.7 / 53.0 / 56.0 against 78.0 / 90.3 / 86.0 / 66.3). In these models, the first 6-d block has total variance below 0.01, against about 6 for a healthy block, and a full-width SIGReg barely registers it. While nested regularization prevents this collapse, it alone doesn't impose dimension ordering. ElasticJEPA resolves these complementary failure modes. The revised manuscript includes this analysis and the completed four-environment factorial ablations (C1, C4; Section 5.2, Tables 8–9).
W2. Inference speedup. As discussed in C1 and C2 of our common response, ElasticJEPA focuses on achieving representation-size flexibility (which is valuable for downstream planning, as remarked by Reviewer d2SW) rather than directly accelerating the existing full-width predictor. Our additional frozen-encoder experiments (Section 5.7, Table 4) demonstrate the practical value of compact representations for downstream planning, alongside speedups obtainable through predictor compression.
W3, Q3. Full-width performance. Regarding your comment "Why does full-width performance consistently trail Sub-JEPA?", please note that Sub-JEPA optimizes a single, task-tuned granularity, whereas ElasticJEPA supports six granularities jointly. We hypothesize that this introduces a specialization–flexibility trade-off. With all three models trained using the same settings, full-width success is 87.0% (LeWM), 91.4% (ElasticJEPA), and 95.0% (Sub-JEPA) on Two-Room (Section 5.4). We also tested weighting the finer granularities more (square-root weights) on all four environments. On Two-Room, this gives 93.3%, but the full-width change is within evaluation noise in every environment (-2.3, +1.3, +0.3, -1.7 points; Appendix C, Table 6). Thus, this test neither confirms nor rules out the trade-off. The Two-Room ordering above is our direct support for it.
W4. Factorial ablations. The revised manuscript extends these experiments from two to all four environments, confirming that nested regularization and nested prediction both contribute to truncation robustness, with Reacher and Cube showing little sensitivity to the latter (C1, C4; Figure 3, Table 8).