acceptodds
Under review as a conference paper at ICLR 2027

Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learning

Abstract

Recent work on LeWorldModel (LeWM) has shown that the Sketched Isotropic Gaussian Regularizer (SIGReg) enables stable end-to-end world model learning from pixels by regularizing the latent representation toward an isotropic Gaussian. While effective for latent-space planning, the representations learned by Raw LeWM are poorly suited for downstream robot policy learning. In this paper, through Monte Carlo analysis, we show that the Raw LeWM objective biases variance allocation toward the temporally persistent component, thereby suppressing the variance of the temporally centered residual. Consistent with this analysis, trained Raw LeWM representations exhibit suppressed residual variation and reduced ecodability of robot state and dynamics, particularly gripper dynamics, which are crucial for robotic manipulation. To address this issue, we apply SIGReg to rather than to the whole latent representation. This simple change decouples persistent and residual variance allocation while retaining an effective anti-collapse property. On the LIBERO benchmark, our method improves downstream policy success on the Goal suite by 1.66× and raises the average success rate across all suites from 63.6% to 83.8%. Without external pretraining, it also outperforms both Diffusion Policy trained from scratch and the pretrained OpenVLA baseline. These results associate the variance-allocation bias of Raw LeWM with the downstream policy gap, and show that decoupling persistent and residual variation yields representations better suited for downstream robot policy learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, JEPA, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5thiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why representations learned by LeWorldModel may be suboptimal for downstream robot policy learning despite avoiding representation collapse. The authors attribute this limitation to an interaction between latent prediction and SIGReg that favors temporally persistent variation over within-window residual variation. They propose TC-LeWM, which applies SIGReg to temporally centered residuals rather than the full latent representation. Evaluations on LIBERO show substantial improvements over Raw LeWM across single-task, suite-wise, and unified world-model training settings. Representation analyses further associate these improvements with increased residual variation and improved decodability of robot state and dynamics.

Critical strengths

1.A useful observation supported by substantial gains over a controlled baseline. The distinction between preventing collapse and learning control-relevant representations is meaningful. Raw LeWM and TC-LeWM share the architecture, training data, and downstream policy protocol, making their comparison informative. The increase from 63.6% to 83.8% under suite-wise training is substantial, and improvements persist across three training scales and multiple training seeds.

2.A simple modification with a coherent empirical motivation. Temporal centering is straightforward to incorporate into the existing objective. The variance analysis, state/dynamics probes, and stage-wise evaluations provide complementary evidence consistent with the proposed explanation. The appendix also examines window-size sensitivity and shows that reducing the SIGReg weight in Raw LeWM does not fully close the performance gap.

Critical weaknesses

1.The methodological contribution is incremental, and its specific advantage over alternative regularization choices is insufficiently established. The main change is replacing the input to an existing regularizer with temporally centered residuals. Simplicity is attractive, but the current comparisons establish superiority over Raw LeWM more clearly than the necessity or broader value of this particular formulation. Relevant controlled alternatives include applying SIGReg to temporal differences, using a variance/covariance regularizer on the residuals, or combining full-latent and residual regularization. The existing window-size and regularization-weight ablations do not resolve these questions. Moreover, all policy evaluations use LIBERO and the same frozen-encoder flow-matching pipeline. Results from other policy families are quoted from prior work, so they do not substitute for matched comparisons that isolate the representation-learning contribution.

2.The proposed mechanism remains plausible but incompletely validated. In Section 3.2 and Appendix A.2, the Monte Carlo analysis models within-window variation as independent noise and replaces the prediction loss with its variance. This construction makes increased residual variation costly by assumption. In the actual action-conditioned model, however, temporal variation can be predictable; marginal residual variance need not track prediction error. The analysis therefore illustrates a possible failure mode without establishing how accurately it describes the trained system. The empirical diagnostics are consistent with the explanation but remain largely correlational. In particular, the dynamics probes decode temporal differences from latent differences, rather than demonstrating that the corresponding information is accessible from the current observation alone. Additionally, the regularization and variance analysis concern projected CLS representations, whereas the downstream policy also uses patch tokens. These gaps leave the connection between the analyzed variance allocation and the actual policy improvement insufficiently isolated.

Questions for authors

1.Do you have matched comparisons with temporal-difference SIGReg, residual variance/covariance regularization, or combined full-latent and residual regularization? Such comparisons would help establish what is specifically gained by the proposed formulation beyond encouraging temporal variation.

2.What empirical evidence supports using residual variance as a surrogate for action-conditioned prediction error? Does the proposed mechanism persist in a controlled setting where temporal variation is predictable from actions rather than independent noise?

3.Can you clarify how the representation diagnostics relate to the features consumed by the policy? In particular, do the gains persist with CLS-only policy inputs, and do patch-token probes show corresponding improvements? This would help connect the analyzed representation changes to closed-loop performance.

Reviewer_rTB6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores improvements to a prior work LeWM which uses a SIGReg based learning objective. The learning objective is applied on latents which contain a temporally centered and temporally persistent component, and the authors argue an imbalance in their contributions. Authors propose a simple change of adding SIGReg to the temporally centered component only. Results on the LIBERO benchmark are used to validate improvements from their proposed change.

Critical strengths
  1. Simple and well-motivated modification to learning objective
  2. Interesting analysis and motivation for proposed change
Critical weaknesses
  1. Results on a single simulated benchmark only. These results are also weaker than the reported baselines. It is unclear if proposed method would lead to improvements for models with suitable pretraining or generalize to any large scale training.

  2. No real world experiments are provided. This makes is difficult to assess the practicality of the proposed method.

  3. The generality of separate regularization for temporally centered component is not clear. Temporal centering on real world data could lead to different dynamics due to real world effects like camera shake, sensor noise, and background motion. The LIBERO results alone do not establish the generality of this idea. At least testing with more diverse simulations (e.g. LIBERO-Plus which explores robustness) would be helpful.

Questions for authors
  1. How does this work on other simulated benchmarks?
  2. Will this method work for models with stronger pretraining? Could it improve post-training of such models?
  3. How would this model work on real world data?
2 comments

Thank you for the update.

To reiterate, results are still significantly lower than baselines. Pi0.5 (which itself is quite old now) outperforms proposed method by over 10 points in Table 2. All other results comparisons are against weaker baselines. The new results for different sim envs also use weak baselines. No comparison to established strong baselines (such as Pi0.5, GrootN1, recent WAMs).

The real world experiments also skip comparison against strong pretrained model baselines like Pi0.5 or GrootN1. The pretraining experiments also avoid comparison to any such strong baselines. In both cases, it is difficult to understand if the improvements against the weak baseline are actually meaningful and will translate to more powerful models.

Building over an already strong pretrained ckpt and comparing against numbers reported in established prior work on common benchmarks would help bring more credibility to the reported results.

The work seems interesting, but current results do not provide meaningful evidence of usefulness; the paper requires major changes. Voting to retain clear rejection.

Thank you for the information.

To clarify, the reviewer believes that improving the performance of a single method with minimal real world robotic applications is not a sufficient contribution. The request for improved or comparable results against established baselines is because the proposed method only applies to a single method.

If this method was at least more widely applicable to numerous established methods (which it is not), perhaps the authors arguments on a useful general method would have more credit.

For these reasons, i.e.

  1. Proposed method is specific to a single prior work that has not demonstrated strong real world results
  2. Neither baseline nor proposed improved baseline are comparable against any recent strong robot control methods

the reviewer retains the original rating.

Reviewer_zovSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the effect of SIGReg in LeWorldModel (LeWM) on downstream robot policy learning. Through temporal decomposition and Monte Carlo analysis, the authors show that applying SIGReg to the full latent representation in the original LeWM may bias variance allocation toward the temporally persistent component, thereby suppressing temporal residual variation that is important for robot control. Based on this analysis, the authors propose Temporally Centered LeWorldModel (TC-LeWM), which applies SIGReg to the temporally centered residual to reduce the original variance-allocation bias while retaining the anti-collapse property. After world model training, they freeze the encoder and use the flow-matching BC policy to evaluate the learned representations. Then the authors do experiments on LIBERO under single-task, suite-wise, and unified 40-task training settings. Through analyses of variance allocation and the decodability of robot state and dynamics, the authors examine changes in the learned representations and show that TC-LeWM significantly improves the closed-loop success rate of the downstream BC policy compared with the original LeWM.

Critical strengths

S1: Clear Motivation and Analysis The authors address the problem that the original (Raw) LeWM still struggles to learn representations suitable for downstream robot control, even after representation collapse is prevented. They use temporal decomposition and Monte Carlo analysis to study how SIGReg and the prediction loss affect variance allocation. Figs. 2 and 3 further analyze persistent and residual variation in the learned representations, as well as the decodability of robot state and dynamics. The overall analysis follows a clear line of reasoning and provides a reasonable motivation for TC-LeWM.

S2: Simple and Effective Method. TC-LeWM only changes the regularization target of SIGReg from the full latent representation to the temporally centered residual, while keeping the original world model architecture and prediction objective unchanged and retaining the anti-collapse property. This design directly addresses the variance-allocation bias analyzed earlier. The method is simple and has a clear mathematical basis.

S3: Convincing Experimental Validation. The authors evaluate TC-LeWM on LIBERO under single-task, suite-wise, and unified 40-task training settings, where it consistently outperforms Raw LeWM. In particular, both methods use the same world model architecture, training data, and downstream BC policy, allowing the performance improvements from changing the SIGReg regularization target to be evaluated more clearly. The appendix further supports the effectiveness of the method through ablations on the temporal window and SIGReg weight, as well as stage-wise closed-loop analysis.

Critical weaknesses

W1: Limited Evaluation Scope. According to Sec. 5.1 and Appendix A.1, the experiments are mainly based on the ViT-Tiny encoder of LeWM and the LIBERO benchmark. Although TC-SIGReg shows clear performance improvements under single-task, suite-wise 10-task, and unified 40-task training, it has not been further evaluated with other world model architectures or on different robotic manipulation benchmarks. Therefore, the current experiments provide sufficient evidence of the effectiveness of TC-SIGReg in the LeWM setting, but direct experimental evidence of whether this design generalizes to other world models and robotic tasks is still lacking. I suggest to further evaluate the method on more robotic manipulation benchmarks, such as RoboCasa [1].


[1] Nasiriany et al. RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots. (RSS 2024)

Questions for authors

I have two questions that may be worth further discussion.

Q1: Temporal Residual and Task-relevant Information. From a temporal signal perspective, temporal centering can be viewed as separating relatively stable information from short-term variation within a local temporal window. Although Fig. 3(c) shows better decodability of robot state and dynamics, the temporal residual may contain both control-relevant dynamic information and changes caused by visual noise. Have the authors considered analyzing how much of the residual variation at different temporal scales corresponds to actual robot and object motion? Is TC-LeWM still robust to task-irrelevant, high-frequency visual perturbations?

Q2: Role of SIGReg on Temporal Residuals. TC-LeWM achieves significant improvements by changing the SIGReg target from the full latent representation to the temporally centered residual. Have the authors considered further analyzing the respective roles of temporal residual variance preservation and Gaussian distribution matching? For example, could applying variance regularization only to the residual, without requiring its projected distribution to approach a standard Gaussian, also achieve similar improvements in representation quality and control performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.