acceptodds
Under review as a conference paper at ICLR 2027

ReWorldModel: Rethinking Visual Representation Learning in Latent World Models

Abstract

Latent world models learn compact visual representations for predicting action-conditioned future states. However, visual information is organized hierarchically across encoder depth: low-level features retain fine-grained spatial and geometric details, favoring representational completeness, whereas high-level features form compact abstractions that favor predictability. Existing world models typically operate only on final-depth representations, potentially discarding details required for precise control. This creates a cross-level trade-off between representational completeness and predictability. To address this, we introduce ReWorldModel (ReWM), a multi-level representation architecture centered on Mixture-of-Depths Attention (MoDA). MoDA enables high-level queries to selectively retrieve low-level features from earlier encoder layers, while a modernized Transformer design improves optimization and spatial modeling. Full ReWM achieves 100% success on four of five tasks and 99.0% mean success on PushT, reaching 95% on PushT with 5× fewer processed training sequences than LeWM under its official recipe. Under a shared training recipe with three paired seeds, a controlled ReWM variant improves Reacher success by 74.4 percentage points at eight encoder updates and reaches 95% with 8× fewer updates on Reacher and TwoRoom. Layer-wise probing further shows that ReWM progressively organizes control-relevant position, orientation, and interaction information across encoder depth, producing final predictive representations with richer control-relevant structure. ReWM thus reshapes the completeness–predictability trade-off toward representations that maintain strong latent predictability while becoming useful for downstream control substantially earlier. These results demonstrate that cross-depth access improves the completeness–predictability frontier of latent world-model representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, predictive representation learning, visual representation learning, self-supervised learning, visuomotor control, robot learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aHEuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReWorldModel, adpating the Mixture-of-Depths attention to a JEPA encoder, to aggreate multi-level representation information. Promising results are reported across five navigation and manipulation environments.

Critical strengths
  1. The problem is well motivated: Instead of using the final-depth information, aggregating multi-level feature is a reasonable direction, where different layers have distinct functions.
  2. Comprehensive ablations: The component ablations in Table 2 are informative, where the proposed components are well ablated.
Critical weaknesses
  1. Lack of baselines: the paper only compared against LeWM, more baselines should be included, for example, PLDM [1], DINO-WM [2]
  2. I acknowledge the application and demonstrated effect of cross-depth attention, but the main contribution is to incorporate MoDA into LeWM.

[1] Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models, https://arxiv.org/abs/2502.14819v4 [2] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, https://arxiv.org/abs/2411.04983

Questions for authors
  1. What are the runtime and memory costs? Can the authors report the analysis?
2 comments
AuthorsOpenReview ↗

Response to Reviewer aHEu (1/2)

We sincerely thank the reviewer for the thoughtful evaluation, and for recognizing that our problem is "well motivated" and that our ablations are "comprehensive" and "informative". We have incorporated your valuable suggestions into the revised manuscript.

Concern 1: Lack of baselines: the paper only compared against LeWM, more baselines should be included, for example, PLDM [1], DINO-WM [2]

Response 1: Thank you for this important suggestion. We evaluated both PLDM [1] and DINO-WM [2] on PushT (300 shared evaluation goals) from two complementary angles:

  1. Representation readout for control (shared GC-IDM protocol fitted on each frozen representation under identical goals and execution budget; Sections 5.4, 6, Appendix Q):
Representation + GC-IDMSuccess
ReWM (5.43M, from scratch, epoch 1)98.00%
LeWM (5.50M, from scratch, epoch 1)82.33%
PLDM (adapted to PushT)97.67%
Frozen DINOv2-S, CLS90.00%
Frozen DINOv2-S, spatial pooling95.00%

Table R10. PushT GC-IDM control comparison (%) against PLDM and frozen DINOv2-S representations.

  1. Planning with each model's learned predictor (CEM under identical planner settings and shared goals):
MethodEpoch 2Epoch 5Epoch 10
ReWM (5.43M)78.33%86.67%82.33%
PLDM (adapted to PushT)61.33%74.67%78.00%
DINO-WM (adapted to PushT)78.33%79.00%63.67%

Table R11. PushT CEM planning success (%) comparing ReWM with PLDM and DINO-WM across training budgets.

On policy readout, ReWM substantially outperforms LeWM and frozen DINOv2-S features while matching PLDM. When planning through each model's learned dynamics, ReWM leads across budgets—exceeding PLDM by +17.0, +12.0, and +4.3 points, matching DINO-WM at epoch 2, and surpassing it by +7.7 and +18.7 points at epochs 5 and 10—using a compact 5.43M encoder trained from scratch rather than the frozen ~21M DINOv2-S backbone in DINO-WM. We have included these comparisons and discussed their complementarity in Sections 2, 5.4, 6 (Discussion), and Appendix Q.

Concern 2: I acknowledge the application and demonstrated effect of cross-depth attention, but the main contribution is to incorporate MoDA into LeWM.

Response 2: We appreciate your recognition of cross-depth attention's demonstrated effect. Rather than treating MoDA as a standalone operator novelty, our contribution follows a focused representational question in latent world models:

  1. Empirical observation & question: Control-relevant state readability is not consistent across encoder depth—e.g., on TwoRoom, ViT position probe R2R^2 drops from 0.8130.813 at block 3 to 0.6190.619 at the final latent zz (Section 5.2)—yet latent world models pass only zz to the predictor. We therefore ask through what path intermediate-layer information should enter the final representation.
  2. Controlled comparison against alternative multi-layer paths: Keeping the predictor, objective, single-latent interface, and parameter scale (5.43M vs. 5.50M) unchanged, we adopt MoDA so deeper layers directly and content-selectively access earlier features during representation formation, and show under matched budgets that generic multi-layer designs do not achieve the same gains:
    • Output-level averaging / concatenation (blocks 3, 6, 9, 12): 7.2%7.2\% / 8.7%8.7\% vs. 36.8% mean early PushT success (Section 5.4).
    • Depth fusion on the ViT backbone (DenseFormer 77.8%, FPN-style top-down fusion 60.4%): score below ReWM on mid-network state readability CC and persistence-relative predictability 1−θ1-\theta (Table R1; Section 5.4, Appendix O) and trail ReWM (82.4%) in epoch-2 CEM planning.
    • Multi-depth supervision (V-JEPA 2.1-style 33.33%, hidden-layer self-distillation 33.44%): trails ReWM (56.56%) at 512 PushT updates (Table R8; Section 5.4, Appendix N).
    • Disabling only historical K/V on the same modern block: lowers early PushT control by 12–16 points (Table R5).
  3. Practical evaluation: Across state readout, shared physical-target rollouts (0.7019 / 0.8349 vs. 0.2060 / 0.1875 early 5-step R2R^2, and 0.9654 vs. 0.9582 at epoch 4; Section 5.5, Appendix P), and CEM planning (Tables R2–R3, R11), the results consistently support full ReWM under the tested conditions.
AuthorsOpenReview ↗

Response to Reviewer aHEu (2/2)

Concern 3: What are the runtime and memory costs? Can the authors report the analysis?

Response 3: Thank you for this practical question. We added a dedicated runtime, memory, and matched-recipe update efficiency analysis in Section 7 (Limitations) and Appendix R.

In a fixed-shape steady-state compute benchmark on one RTX 5090 (BF16, 64 four-frame sequences per update; fused AdamW and CUDA Graphs for both models; one full compute update covering encoder forward, predictor, loss, backward, and optimizer step with all 12 layers, historical K/V cache retained, and FlashAttention for joint attention, excluding data loading and initial compilation):

MetricOptimized ReWMViT (LeWM)
Full compute update59.00 ms56.48 ms
Peak reserved memory19.38 GiB15.71 GiB

Table R12. Steady-state per-update compute and memory benchmark on PushT (single RTX 5090, BF16).

Cross-depth access adds +4.5% per-update compute time (59.00 ms vs. 56.48 ms) and +3.67 GiB of peak reserved memory (19.38 GiB vs. 15.71 GiB for caching earlier-layer K/V), fitting comfortably on a single consumer GPU. Separately, in a matched-recipe paired PushT run (LR 10−410^{-4}, cosine schedule, seed 3072), ReWM reaches ≥95% GC-IDM success in 2,048 updates versus 9,216 for ViT (4.5× fewer updates).

Thank you again for your constructive comments, which have greatly strengthened our paper. We hope our responses have thoroughly addressed your concerns, and we would be sincerely grateful if you would consider raising your score! Please let us know if any further clarification would be helpful.

Reviewer_5ACwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new training architecture for the visual encoder in the existing LeWM backbone in order to improve its representational power while maintaining its predictive capability. The key change compared to LeWM is the use of Mixture-of-Depths Attention which accesses early-layer embeddings to inform later-layer encodings and thus helps it maintain control-relevant features deeper into the encoder network.

The results of this change are promising, offering competitive success rates and predictive performance compared to the standard LeWM baseline, but with much fewer encoder updates.

Critical strengths

The paper is well written, clearly motivated, and easy to follow. The method is detailed sufficiently and the authors provide sufficient justifications and intuition for their design choices. The experimental results are well presented, satisfactorily discussed and are promising in general.

While the modifications to the existing architecture are mild, the results are impressive enough that I am willing to forego my concern over novelty. Overall, I found this to be very nice read and thank the authors for their effort.

Critical weaknesses

I am curious about why the authors chose to present Figure 4 at the "32,768 processed training sequences" snapshot. I would have like to see the evolution of these plots at different points of training. Adding such figures in the appendix would be nice.

I would also more strongly emphasize the value of MoDA by bringing up the ablation numbers earlier in the presentation. The drop in going form MoDA -> vanilla attention is rather drastic.

I will improve my scoring upon a rebuttal.

Questions for authors

I think your work fits into the broader stream of literature which tries to improve predictive performance of world models by improving base representational capabilities of the encoder. There has been work to this end about adapting pre-trained encoders to concentrate on task/dynamic specific portions of the input image rather than task-irrelevant features. These deserves some discussion in the related work in my opinion.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5ACw

We sincerely thank the reviewer for the warm evaluation and generous remarks that the paper is "well written, clearly motivated", "impressive", and "a very nice read". Every suggestion directly improved the clarity and completeness of our manuscript, and we have incorporated all of them into the revised paper.

Concern 1: I am curious about why the authors chose to present Figure 4 at the "32,768 processed training sequences" snapshot. I would have like to see the evolution of these plots at different points of training. Adding such figures in the appendix would be nice.

Response 1: Thank you for this thoughtful suggestion. We chose 32,768 processed sequences in Figure 4 as an early-training snapshot where representation differences across encoder depth are clearest before downstream control saturates. Following your suggestion, we added a multi-stage progression figure in Appendix L (referenced in Section 5.2) tracking position probe R2R^2 across blocks 3, 6, 9, 12, and zz at initialization, 4,096, 32,768, and 262,144 processed sequences on Reacher, TwoRoom, and PushT. Across tasks, the progressive organization across depth appears early and persists throughout training (at 262,144 sequences, final-latent R2R^2 reaches 0.9970.997, 0.9880.988, and 0.8990.899 for ReWM versus 0.4600.460, 0.8170.817, and 0.2800.280 for the official LeWM encoder), confirming that Figure 4 reflects a consistent trend across training.

Concern 2: I would also more strongly emphasize the value of MoDA by bringing up the ablation numbers earlier in the presentation. The drop in going form MoDA -> vanilla attention is rather drastic. I will improve my scoring upon a rebuttal.

Response 2: Thank you for this excellent presentation suggestion—we completely agree. In the revised manuscript, we brought the representative numbers from our MoDA module ablation (Table 2, Section 5.4) directly into the Introduction (Section 1): replacing MoDA with vanilla attention drops mean early control success over the first 64 encoder updates from 79%79\% to 37%37\% on Reacher and from 36.8%36.8\% to 7.5%7.5\% on PushT (and at update 8, from 99%99\% to 1%1\% on Reacher, 100%100\% to 69%69\% on TwoRoom, and 35.0%35.0\% to 9.0%9.0\% on PushT), which output-level averaging or concatenation of summary tokens from blocks 3, 6, 9, and 12 (55%55\% / 52%52\% on Reacher and 7.2%7.2\% / 8.7%8.7\% on PushT) does not recover. In addition, disabling only historical K/V access while keeping RoPE, Q/K-Norm, RMSNorm, and SwiGLU unchanged still drops early PushT control by 12–16 points (see Table R5 in our response to reviewer c9AQ/#1), further underscoring the value of MoDA's cross-depth retrieval.

Concern 3: I think your work fits into the broader stream of literature which tries to improve predictive performance of world models by improving base representational capabilities of the encoder. There has been work to this end about adapting pre-trained encoders to concentrate on task/dynamic specific portions of the input image rather than task-irrelevant features. These deserves some discussion in the related work in my opinion.

Response 3: Thank you for pointing out this important connection. We expanded Related Work (Section 2) and added a dedicated discussion in Discussion (Section 6) covering both streams:

  1. Pretrained visual representations transferred to control and world models (R3M, MVP, VIP, PVR, VC-1, and DINO-WM); and
  2. Task- or dynamics-informed world models that suppress task-irrelevant distractors (DBC, TIA, Denoised MDP, and Iso-Dream).

Whereas those approaches inherit features from large pretrained backbones or add auxiliary objectives to filter distractors, ReWM acts on a complementary architectural axis—reshaping depth-wise information flow inside a compact 5.43M encoder trained end-to-end while keeping the predictive objective unchanged. As discussed in Section 6 and Appendix Q (where ReWM reaches 98.00% PushT GC-IDM success vs. 90.00% / 95.00% for frozen DINOv2-S CLS / spatial pooling), inserting cross-depth retrieval into pretrained backbones or pairing it with distractor-suppressing objectives is a promising direction for future work.

Reading that you found our paper "a very nice read" truly meant a great deal to us, and your thoughtful suggestions have made the narrative and the value of MoDA much sharper. Thank you again for your generosity, time, and constructive engagement—if our revisions have addressed all of your points, we would be sincerely grateful for your kind consideration in raising the score!

AuthorsOpenReview ↗

Thanks for Your Thoughtful Review and Recognition of Our Work

Thank you very much for your thoughtful review, positive comments, and constructive suggestions. We sincerely appreciate your recognition of our work, and we will keep working to make ReWM and our future projects better and better. With 1 day remaining in the discussion period, we hope our responses and revisions have addressed your questions, and we are happy to provide any further clarifications if needed.

Reviewer_5ACwOpenReview ↗

Response to Rebuttal

I thank the authors for the clarifications and the rebuttal. My views since the review have not changed, so I will keep my score.

However, I do note that the changes in related work are not very thorough. I am convinced that your work is sufficiently different than existing works and do not mark you negatively for novelty. However, the need for better encoders that concentrate their embedding on task-relevant information has been observed in works like [1], and there have been follow ups from this observation that seek to mitigate the effort that encoder employs on task-irrelevant parts of the image. I think the authors can, and should, do a better job at contextualizing these in the related work section.

[1] Sobal, V., Zhang, W., Cho, K., Balestriero, R., Rudner, T. G., and LeCun, Y. (2025). Learning from reward-free offline data: A case for planning with latent dynamics models.

Thank you for pointing us to Sobal et al. (2025). We have added a dedicated discussion in Section 2 ("Concentrating the encoder on task-relevant content") and a structured comparison in Appendix S (Table 20).

Sobal et al. (2022) showed that JEPA objectives can gravitate toward slowly varying backgrounds, and PLDM (Sobal et al., 2025) further exposed how spending capacity on such nuisances degrades downstream planning (e.g., on Two-Room). To address this, prior work intervenes primarily at the objective or input level—through auxiliary objectives or supervision (PLDM, Delta-JEPA, Temporal Straightening, physically grounded JEPA, DBC/TIA), pretrained-feature adaptation (JEPA-Bisim, task-conditioned codebooks), or pre-encoding spatial selection (TaskSense).

However, inside the encoder, fine-grained control information available at intermediate layers can become less accessible at the final output. ReWM ensures detail-rich control geometry and high-level predictive abstractions coexist in the final representation.This achieves objective-preserving, reward-free representation learning, yielding drastic early-training sample efficiency while outperforming frozen pretrained backbones.

Reviewer_c9AQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether accessing intermediate encoder features improves representations for latent world models. ReWM adds Mixture-of-Depths Attention (MoDA) to LeWM, allowing each token to attend to current-layer tokens and its own features from earlier layers. The predictive objective and final latent interface remain unchanged. Across five simulated tasks, the method improves early control performance and linear decoding of physical state. Control is evaluated using a separately trained policy on frozen representations, without planning.

Critical strengths
  • Clear and easy to read.
  • Large early-control gains, supported by comparisons across three seeds.
  • Useful ablations against averaging and concatenating intermediate-layer features.
Critical weaknesses
  • The related work misses relevant approaches that exploit multiple encoder depths through supervision, including [1] and [2]. Their mechanisms differ from MoDA, but the paper should explain what cross-depth retrieval contributes beyond these approaches.

  • The methodological novelty is limited. MoDA is an existing operator, and the predictive objective is unchanged. Its application to latent world models produces useful empirical gains, but the broader claims about the completeness–predictability trade-off need stronger support.

  • No planning results are reported. Control uses a separately trained GC-IDM policy that bypasses the learned predictor. On TwoRoom and Puzzle-3×3, this trained policy already reaches 100% success with a frozen, randomly initialized encoder. These results demonstrate a useful architectural bias, but leave the benefit for planning with learned dynamics unclear.

  • The measurements do not establish the proposed mechanism. Better linear probes show that state information is easier to access, which does not necessarily imply that more information is preserved. Prediction R² is also measured against each encoder’s own latent targets, so the scores do not compare predictions of the same quantities.

  • The headline 5× PushT speedup combines architectural changes with changes to learning rate, batch size, and schedule. The controlled experiments support an architectural benefit, but do not establish that specific speedup under matched training settings.

[1] V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning (Mur-Labadia, 2026) [2] Self-Distillation of Hidden Layers for Self-Supervised Representation Learning (Lowe, 2026)

Questions for authors
  • Do the gains carry over to planning with the learned predictor, using the same planner and compute budget?

  • How much does predictive training improve over frozen random features across all five tasks? Could harder goals or longer horizons reveal a difference on TwoRoom and Puzzle-3×3?

  • Can you disable only attention to historical keys and values while retaining RoPE, Q/K normalization, and all other settings? This would isolate the contribution of cross-depth retrieval more precisely.

  • Can you evaluate multi-step predictions against common physical-state targets, since each encoder currently predicts its own latents?

  • Does the 5× PushT advantage hold under matched training recipes? What are the corresponding wall-clock time and peak memory costs?

  • How does MoDA compare with multi-layer predictive supervision inspired by [1] or [2], under the same training budget?

[1] V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning (Mur-Labadia, 2026) [2] Self-Distillation of Hidden Layers for Self-Supervised Representation Learning (Lowe, 2026)

6 comments
AuthorsOpenReview ↗

Response to Reviewer c9AQ (1/5)

We express our sincere gratitude to the reviewer for dedicating the time to provide such a thorough, penetrating, and constructive review. Every question—on multi-depth supervision, methodological positioning and the readability–predictability relationship, predictor-based planning, common physical targets, and matched-recipe efficiency—was genuinely inspiring to us. Your feedback guided us to design a comprehensive suite of new experiments and analyses that have substantially strengthened the paper.

Concern 1: The related work misses relevant approaches that exploit multiple encoder depths through supervision, including [1] and [2]. Their mechanisms differ from MoDA, but the paper should explain what cross-depth retrieval contributes beyond these approaches.

Response 1: Thank you for pointing us to these two highly relevant and insightful works ([1] V-JEPA 2.1 and [2] Self-Distillation of Hidden Layers). We have cited and discussed both in Related Work (Section 2) and further discussed their relationship in Discussion (Section 6).

As you rightly note, [1] and [2] exploit multiple encoder depths through auxiliary supervision on intermediate features. What cross-depth retrieval contributes beyond intermediate supervision is a direct, content-selective forward path inside the encoder while the final representation zz is being formed: deeper queries attend directly to earlier-layer keys and values at the same token position, rather than relying on intermediate losses alone to carry fine control cues (positions, contacts, local geometry) sequentially through every subsequent block to zz. As quantified under an identical training budget in Response 11 (Tables R8 and R9), ReWM reaches 56.56% PushT GC-IDM success at 512 updates versus 33.33% (V-JEPA 2.1-style) and 33.44% (hidden-layer self-distillation), and 53.78% versus 14.11% and 13.89% across 3 seeds at 128 updates. Moreover, because supervision shapes what intermediate layers learn while MoDA determines how earlier features enter zz, combining both is a promising direction for future work (Section 6, Appendix N).

AuthorsOpenReview ↗

Response to Reviewer c9AQ (2/5)

Concern 2: The methodological novelty is limited. MoDA is an existing operator, and the predictive objective is unchanged. Its application to latent world models produces useful empirical gains, but the broader claims about the completeness–predictability trade-off need stronger support.

Response 2: We appreciate the reviewer's insightful feedback on both positioning our contribution and strengthening empirical support.

(1) Positioning our contribution: Rather than proposing a new attention primitive, our study addresses a specific representational question in latent world models:

  • Empirical observation: Readability of control-relevant state is not consistent between intermediate encoder layers and the final representation (e.g., on TwoRoom, ViT position R2R^2 drops from 0.8130.813 at block 3 to 0.6190.619 at zz; Section 5.2), yet latent world models pass only zz to the predictor.
  • Question & design: We ask through what path intermediate-layer information should enter the final representation. Keeping the predictor, objective, parameter scale (5.43M vs. 5.50M), and single-latent interface unchanged, we adopt MoDA so deeper layers directly and content-selectively retrieve earlier features during representation formation.
  • Controlled comparison against alternative depth paths: Under matched budgets, generic multi-layer designs do not obtain the same gains: (i) output-level averaging/concatenation of blocks 3, 6, 9, 12 gives 7.2%7.2\% / 8.7%8.7\% vs. 36.8% mean early PushT success (Section 5.4); (ii) disabling only historical K/V on the same modern block drops early PushT control by 12–16 points (Response 8, Table R5); (iii) multi-depth supervision gives 33.33% / 33.44% vs. 56.56% at 512 updates (Response 11, Table R8); and (iv) in epoch-2 CEM planning with each model's learned predictor (3 seeds × 150 episodes), ViT depth-fusion baselines DenseFormer (77.8%) and FPN-style top-down fusion (60.4%) as well as standard ViT (75.8%) all trail ReWM (82.4%, p=.008p=.008 vs. ViT).

(2) Empirical support for state readability and predictability: Following the state-probing protocol and persistence-relative error diagnostic of H-JEPA (Zhang et al., 2026, arXiv:2610.06805), adapted here to encoder depth, we freeze each PushT model under a shared recipe (3 seeds, epoch 2) and evaluate Block 6 and zz (Sections 5.4, 6, Appendix O):

ModelBlock 6 CCBlock 6 1−θ1-\thetazz CCzz 1−θ1-\theta
ReWM0.8690.8180.8830.947
ViT (LeWM)0.7620.6730.8840.936
+DenseFormer0.7460.6810.8860.934
+FPN-style0.7230.5780.8660.793

Table R1. State readability (CC) and persistence-relative predictability (1−θ1-\theta) on PushT (3 seeds, epoch 2).

Three details clarify the scope of Table R1: (a) CC is the mean MLP-probe R2R^2 over five physical-state groups—an operational measure of state readability rather than total preserved information; (b) Block 6 uses capacity-matched auxiliary predictors while zz uses each model's native predictor, with θ\theta normalizing one-step MSE by a state-persistence baseline; and (c) because 1−θ1-\theta is measured in each model's own feature space, it complements rather than replaces shared physical-target rollouts (Response 9, Table R6) and CEM planning (Response 3, Tables R2–R3), without claiming a universal theoretical frontier.

AuthorsOpenReview ↗

Response to Reviewer c9AQ (3/5)

Concern 3: No planning results are reported. Control uses a separately trained GC-IDM policy that bypasses the learned predictor. On TwoRoom and Puzzle-3×3, this trained policy already reaches 100% success with a frozen, randomly initialized encoder. These results demonstrate a useful architectural bias, but leave the benefit for planning with learned dynamics unclear.

Response 3: Thank you for raising this central concern—we completely agree that evaluating action selection through the learned predictor is essential. Whereas GC-IDM tests policy readout on frozen features, CEM planning rolls candidate action sequences recursively through each model's learned action-conditioned predictor and selects actions from the predicted futures, directly testing whether the representation and learned dynamics jointly support planning.

(a) Uniform-protocol CEM planning at matched early training data (identical planner settings across all five tasks: 300 candidates, 30 iterations, 300 shared goals per training seed; mean over 3 training seeds, success %):

TaskProcessed sequencesReWMLeWM
Reacher262,14463.569.11
TwoRoom262,14494.5628.00
Cube114,68860.6746.67
Puzzle-3×3262,14441.2229.89
PushT262,1448.443.22

Table R2. Uniform-protocol CEM planning success (%) at matched early training data across all five benchmarks.

(b) CEM planning performance as training progresses, alongside our self-trained LeWM baseline (official code and recipe) and the reference scores reported in the LeWM paper:

TaskReWMSelf-trained LeWMLeWM (reported)
Reacher96%, epoch 290%, epoch 2086%
TwoRoom100%, 256 updates94%, epoch 1087%
Cube79%, epoch 1474%, epoch 1574%
Puzzle-3×384%, epoch 872%, epoch 10Not reported
PushT98%, epoch 896%, epoch 1096%

Table R3. CEM planning success (%) at later training checkpoints compared with self-trained and published LeWM.

Tables R2 and R3 show three clear findings: (1) under one uniform early-data protocol, ReWM outperforms LeWM on all five benchmarks; (2) ReWM reaches 79%–100% planning success across all five tasks while our self-trained LeWM matches or exceeds published LeWM references; and (3) ReWM reaches high planning performance at substantially earlier checkpoints (e.g., epoch 2 on Reacher and 256 updates on TwoRoom). In addition, Response 7 (Table R4) shows that longer-horizon goals on TwoRoom and Puzzle-3×3 clearly separate predictive training from frozen random initialization.

Concern 4: The measurements do not establish the proposed mechanism. Better linear probes show that state information is easier to access, which does not necessarily imply that more information is preserved. Prediction R² is also measured against each encoder's own latent targets, so the scores do not compare predictions of the same quantities.

Response 4: We appreciate the reviewer's clear and accurate distinction on both points:

  1. Linear readability vs. total preserved information: We agree that linear probes measure how linearly readable control-relevant state is—the exact property consumed by lightweight downstream controllers (Section 3)—rather than total preserved information. We have explicitly clarified this operational scope in Discussion (Section 6) and Appendix O.
  2. Comparing predictions on identical physical quantities: Because self-target latent R2R^2 is measured in each encoder's own latent space, we added a common physical-state rollout evaluation (detailed in Response 9, Table R6; Section 5.5, Appendix P) that maps multi-step latent rollouts from each model's native predictor onto the same ground-truth future physical states, confirming large early gains (+0.4959 on agent position and +0.6474 on object position 5-step R2R^2) and higher accuracy at epoch 4 (0.9654 vs. 0.9582).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to All Reviewers and Area Chair

We sincerely thank all reviewers and the Area Chair for their time and thoughtful feedback. We are encouraged by reviewer c9AQ's recognition of our "large early-control gains," reviewer 5ACw's description of the paper as "well written, clearly motivated," and reviewer aHEu's appreciation of the "comprehensive ablations."

We believe that learning compact world-model representations that support both prediction and control is an important challenge. ReWM explores how access to earlier-layer features can improve these representations while preserving the predictive objective and single-latent interface. Following the reviewers' suggestions, we added CEM planning with the learned predictor across all five tasks, comparisons against PLDM, DINO-WM, multi-depth supervision (V-JEPA 2.1, self-distillation), and depth fusion (DenseFormer, FPN), multi-step rollouts on shared physical states, isolated historical-K/V ablations, matched-recipe convergence experiments, and runtime measurements. These additions strengthen the empirical support for ReWM, directly address the reviewers' questions, and have been incorporated into the revised manuscript (highlighted in blue). We warmly welcome any further discussion.

AuthorsOpenReview ↗

Appreciation and Follow-up on Reviewer-Author Discussion

We express our sincere gratitude to all reviewers for dedicating your valuable time and effort to review our paper. We have provided detailed responses to all the concerns. With 2 days remaining in the discussion period, we would like to gently check in to see if our responses have adequately addressed your questions. We are more than willing to address any remaining concerns and ensure a comprehensive resolution. Thank you very much for your time and kind consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.