- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates why representations learned by LeWorldModel may be suboptimal for downstream robot policy learning despite avoiding representation collapse. The authors attribute this limitation to an interaction between latent prediction and SIGReg that favors temporally persistent variation over within-window residual variation. They propose TC-LeWM, which applies SIGReg to temporally centered residuals rather than the full latent representation. Evaluations on LIBERO show substantial improvements over Raw LeWM across single-task, suite-wise, and unified world-model training settings. Representation analyses further associate these improvements with increased residual variation and improved decodability of robot state and dynamics.
Critical strengths
1.A useful observation supported by substantial gains over a controlled baseline. The distinction between preventing collapse and learning control-relevant representations is meaningful. Raw LeWM and TC-LeWM share the architecture, training data, and downstream policy protocol, making their comparison informative. The increase from 63.6% to 83.8% under suite-wise training is substantial, and improvements persist across three training scales and multiple training seeds.
2.A simple modification with a coherent empirical motivation. Temporal centering is straightforward to incorporate into the existing objective. The variance analysis, state/dynamics probes, and stage-wise evaluations provide complementary evidence consistent with the proposed explanation. The appendix also examines window-size sensitivity and shows that reducing the SIGReg weight in Raw LeWM does not fully close the performance gap.
Critical weaknesses
1.The methodological contribution is incremental, and its specific advantage over alternative regularization choices is insufficiently established. The main change is replacing the input to an existing regularizer with temporally centered residuals. Simplicity is attractive, but the current comparisons establish superiority over Raw LeWM more clearly than the necessity or broader value of this particular formulation. Relevant controlled alternatives include applying SIGReg to temporal differences, using a variance/covariance regularizer on the residuals, or combining full-latent and residual regularization. The existing window-size and regularization-weight ablations do not resolve these questions. Moreover, all policy evaluations use LIBERO and the same frozen-encoder flow-matching pipeline. Results from other policy families are quoted from prior work, so they do not substitute for matched comparisons that isolate the representation-learning contribution.
2.The proposed mechanism remains plausible but incompletely validated. In Section 3.2 and Appendix A.2, the Monte Carlo analysis models within-window variation as independent noise and replaces the prediction loss with its variance. This construction makes increased residual variation costly by assumption. In the actual action-conditioned model, however, temporal variation can be predictable; marginal residual variance need not track prediction error. The analysis therefore illustrates a possible failure mode without establishing how accurately it describes the trained system. The empirical diagnostics are consistent with the explanation but remain largely correlational. In particular, the dynamics probes decode temporal differences from latent differences, rather than demonstrating that the corresponding information is accessible from the current observation alone. Additionally, the regularization and variance analysis concern projected CLS representations, whereas the downstream policy also uses patch tokens. These gaps leave the connection between the analyzed variance allocation and the actual policy improvement insufficiently isolated.
Questions for authors
1.Do you have matched comparisons with temporal-difference SIGReg, residual variance/covariance regularization, or combined full-latent and residual regularization? Such comparisons would help establish what is specifically gained by the proposed formulation beyond encouraging temporal variation.
2.What empirical evidence supports using residual variance as a surrogate for action-conditioned prediction error? Does the proposed mechanism persist in a controlled setting where temporal variation is predictable from actions rather than independent noise?
3.Can you clarify how the representation diagnostics relate to the features consumed by the policy? In particular, do the gains persist with CLS-only policy inputs, and do patch-token probes show corresponding improvements? This would help connect the analyzed representation changes to closed-loop performance.
Thank you for the update.
To reiterate, results are still significantly lower than baselines. Pi0.5 (which itself is quite old now) outperforms proposed method by over 10 points in Table 2. All other results comparisons are against weaker baselines. The new results for different sim envs also use weak baselines. No comparison to established strong baselines (such as Pi0.5, GrootN1, recent WAMs).
The real world experiments also skip comparison against strong pretrained model baselines like Pi0.5 or GrootN1. The pretraining experiments also avoid comparison to any such strong baselines. In both cases, it is difficult to understand if the improvements against the weak baseline are actually meaningful and will translate to more powerful models.
Building over an already strong pretrained ckpt and comparing against numbers reported in established prior work on common benchmarks would help bring more credibility to the reported results.
The work seems interesting, but current results do not provide meaningful evidence of usefulness; the paper requires major changes. Voting to retain clear rejection.