- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Paired Rollout Probing (PRP), an evaluation protocol for diagnosing state readability in JEPA-based world models. The proposed framework combines observation-trained probing, feature calibration, and comparisons against persistence baselines. The authors evaluate several existing world models and analyze the relationship between latent prediction error, state readability, and the additional information provided by predicted representations.
Critical strengths
The paper investigates a relevant question regarding the evaluation of latent world models. The experimental analyses are reasonably organized, and the observation that feature calibration can substantially recover readout accuracy is potentially interesting.
Critical weaknesses
- Limited methodological novelty.
The proposed PRP framework largely consists of straightforward combinations of established techniques, including linear probing, feature-statistics calibration, and persistence-based comparisons. In particular, comparing predicted future representations with a frozen initial representation is a natural and elementary baseline rather than a substantial methodological contribution.
While evaluation-oriented papers can be valuable without introducing new model architectures, they should establish novel, non-trivial insights or demonstrate clear advantages over existing evaluation methodologies. I do not find the evidence presented here sufficiently compelling in this regard.
- The persistence-related conclusions may be strongly influenced by limited state changes in the evaluation trajectories.
The paper reports exceptionally strong persistence baselines. For instance, initial-state persistence achieves an R² of 0.980 at prediction step 4 and 0.724 at step 8 in the continuous end-effector evaluation.
These results raise an important question about the magnitude of actual state transitions in the evaluated trajectories. If physical states change only slightly over the prediction horizon, strong persistence performance is expected and does not necessarily reveal a fundamental limitation of latent world models.
The paper does not sufficiently characterize the distribution of physical displacements or systematically evaluate trajectories involving substantial state changes. Experiments stratified by action magnitude, physical displacement, and longer prediction horizons would be necessary to establish the generality of these conclusions.
- Limited generality of the empirical findings.
The most detailed calibration, restart, and adaptation analyses are concentrated on a single DROID-trained V-JEPA 2-AC checkpoint evaluated on Meta-World. The other two models are evaluated primarily under their native training distributions.
Consequently, the observed failures may reflect particular distribution shifts, model-specific characteristics, or evaluation choices rather than general properties of latent world models. The experiments do not adequately disentangle these factors.
- Insufficient evidence of the practical value of the proposed evaluation framework.
Although PRP provides additional diagnostic measurements, the paper does not convincingly demonstrate that these measurements lead to more reliable model selection, better predictions of downstream planning performance, or actionable improvements to world-model learning. The offline action-ranking results are limited and do not establish such practical benefits.
Overall Assessment
I appreciate the motivation behind analyzing latent world models beyond aggregate prediction error. However, the proposed methodology is largely composed of established and relatively straightforward evaluation procedures, while the main empirical findings remain limited to specific experimental conditions.
In my assessment, the paper does not provide sufficient methodological novelty or sufficiently generalizable scientific insights to meet the contribution threshold expected for ICLR.
Questions for authors
Please refer to the Weakness Part.
Dear Reviewers
The author-reviewer discussion period will close soon. The authors put much efforts for their rebuttal. Please read and respond to authors if you have not done yet.
Thank you.
AC