acceptodds
Under review as a conference paper at ICLR 2027

PRP: Diagnosing State Readability Beyond Prediction Error in JEPA World Models

Abstract

World models predict environmental changes from observations and actions to support planning. Latent world models predict learned representations, typically evaluated through aggregate error. We introduce Paired Rollout Probing (PRP), a state-specific evaluation protocol combining three paired representation paths, fixed observation-trained probes, and persistence baselines. Comparing observed and predicted representations measures state readability through probe accuracy at each horizon; comparison with the initial representation measures decoding gains beyond initialization. Two Meta-World-trained models maintain high position readability over 15 rollout steps, whereas a DROID-trained V-JEPA 2-AC model evaluated on Meta-World shows sharp decay within two predictions. In the shifted checkpoint, matched observation restarts improve next-step object decoding by 22 percentage points, with the benefit fading over the next two to three steps. One-step adaptation lowers prediction error while the displayed adapted readout remains below frozen-latent persistence over steps 2–8. PRP thus adds a physical-state criterion to rollout evaluation: whether prediction and adaptation improve access to the required quantities beyond retaining the initial observation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
world models, joint-embedding predictive architectures, representation probing, state readability, rollout evaluation, test-time adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XAhpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Paired Rollout Probing (PRP), an evaluation protocol for diagnosing state readability in JEPA-based world models. The proposed framework combines observation-trained probing, feature calibration, and comparisons against persistence baselines. The authors evaluate several existing world models and analyze the relationship between latent prediction error, state readability, and the additional information provided by predicted representations.

Critical strengths

The paper investigates a relevant question regarding the evaluation of latent world models. The experimental analyses are reasonably organized, and the observation that feature calibration can substantially recover readout accuracy is potentially interesting.

Critical weaknesses
  1. Limited methodological novelty.

The proposed PRP framework largely consists of straightforward combinations of established techniques, including linear probing, feature-statistics calibration, and persistence-based comparisons. In particular, comparing predicted future representations with a frozen initial representation is a natural and elementary baseline rather than a substantial methodological contribution.

While evaluation-oriented papers can be valuable without introducing new model architectures, they should establish novel, non-trivial insights or demonstrate clear advantages over existing evaluation methodologies. I do not find the evidence presented here sufficiently compelling in this regard.

  1. The persistence-related conclusions may be strongly influenced by limited state changes in the evaluation trajectories.

The paper reports exceptionally strong persistence baselines. For instance, initial-state persistence achieves an R² of 0.980 at prediction step 4 and 0.724 at step 8 in the continuous end-effector evaluation.

These results raise an important question about the magnitude of actual state transitions in the evaluated trajectories. If physical states change only slightly over the prediction horizon, strong persistence performance is expected and does not necessarily reveal a fundamental limitation of latent world models.

The paper does not sufficiently characterize the distribution of physical displacements or systematically evaluate trajectories involving substantial state changes. Experiments stratified by action magnitude, physical displacement, and longer prediction horizons would be necessary to establish the generality of these conclusions.

  1. Limited generality of the empirical findings.

The most detailed calibration, restart, and adaptation analyses are concentrated on a single DROID-trained V-JEPA 2-AC checkpoint evaluated on Meta-World. The other two models are evaluated primarily under their native training distributions.

Consequently, the observed failures may reflect particular distribution shifts, model-specific characteristics, or evaluation choices rather than general properties of latent world models. The experiments do not adequately disentangle these factors.

  1. Insufficient evidence of the practical value of the proposed evaluation framework.

Although PRP provides additional diagnostic measurements, the paper does not convincingly demonstrate that these measurements lead to more reliable model selection, better predictions of downstream planning performance, or actionable improvements to world-model learning. The offline action-ranking results are limited and do not establish such practical benefits.

Overall Assessment

I appreciate the motivation behind analyzing latent world models beyond aggregate prediction error. However, the proposed methodology is largely composed of established and relatively straightforward evaluation procedures, while the main empirical findings remain limited to specific experimental conditions.

In my assessment, the paper does not provide sufficient methodological novelty or sufficiently generalizable scientific insights to meet the contribution threshold expected for ICLR.

Questions for authors

Please refer to the Weakness Part.

Reviewer_cTPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper introduces Paired Rollout Probing (PRP), an evaluation framework for examining physical-state readability in JEPA world models. PRP compares predicted representations with observed and frozen initial representations, and investigates the effects of feature calibration, observation refreshes, and model adaptation. Exps on several pretrained world models reveal differences between prediction accuracy and state readability.

Critical strengths
  1. The paper raises a meaningful question about whether accurate latent predictions actually preserve useful physical-state information. The distinction between representation error and state readability is worth investigating.

  2. The evaluation includes several useful diagnostic controls, particularly persistence baselines and feature calibration. These comparisons help clarify the interpretation of latent prediction quality.

Critical weaknesses
  1. The proposed framework largely combines existing probing techniques, persistence comparisons, and feature calibration. It is unclear whether these relatively straightforward analyses constitute a sufficiently substantial contribution.

  2. The experimental scope is rather narrow. Most detailed findings rely on a single distribution-shifted checkpoint, while the native-distribution experiments provide limited diagnostic diversity. Some analyses also involve evaluation-data-dependent model selection or in-sample probing, weakening the generality of the conclusions.

Questions for authors
  1. What are the key methodological differences between PRP and existing latent-state probing approaches?

  2. Can the calibration and adaptation findings be validated on additional world models and environments?

  3. Could the PCA selection and periodic-refresh experiments be repeated using strictly held-out evaluation data?

  4. Do improvements in state readability translate into better downstream planning or action selection?

Reviewer_XZC5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Paired Rollout Probing (PRP), an evaluation protocol for measuring which physical state variables remain readable from world-model predictions. PRP applies the same probe to observed, predicted, and frozen initial representations, using spatial pooling, PCA, and linear classification. Experiments on JEPA-WM, DINO-WM, and V-JEPA 2-AC under distribution shift examine readout transfer across prediction horizons, recovery through affine calibration, and gains over persistence.

Critical strengths
  • Using a shared probe and matched evaluation masks makes the comparisons interpretable. Separating label persistence from frozen-latent persistence also helps distinguish state changes from initial readout error.
  • The calibration experiment is the strongest result. A horizon-specific mean/scale correction fitted only on training data raises step-two end-effector accuracy from 0.64 to 0.97 while keeping the classifier fixed. This demonstrates that much of the early decoding failure comes from recoverable feature misalignment.
  • The observed-versus-predicted restart comparison controls for history truncation. The reported 26.2-percentage-point gain in next-step object decoding therefore provides stronger evidence for the benefit of a fresh observation than a comparison with uninterrupted rollout alone.
Critical weaknesses
  • Recorded proprioception is supplied throughout the rollout. Without an actions-and-proprioception-only baseline, it is unclear how much additional position information the visual dynamics provide.
  • Evidence of useful state prediction beyond persistence remains limited. At step four, repeating the true initial position achieves R2=0.980R^2=0.980, compared with 0.9200.920 for the rollout-trained readout. Trajectories with larger state changes would provide a stronger test of the learned dynamics.
  • Some evaluations lack a clean separation between model selection and testing. The initial PCA dimension selection includes evaluation trajectories, and the periodic-refresh experiment fits and evaluates its probe on the same data.
  • Generalization is not well established. Native-model evaluation uses world-model training data, while the detailed calibration and adaptation experiments cover only one checkpoint under real-to-sim shift.
  • Adaptation results come from single runs. Small gains over persistence need uncertainty estimates across data splits, probe fits, and adaptation runs.
Questions for authors
  • Can the authors add an actions-and-proprioception-only predictor and evaluate position changes relative to initialization?
  • Does calibration remain effective when statistics are shared across horizons or transferred across tasks? Would applying an equivalent calibration procedure to the frozen initial latent help separate feature alignment from useful temporal prediction?
  • During action ranking, are recorded proprioceptive trajectories retained when donor actions are substituted? If so, how is the resulting action–state inconsistency handled? Can donor actions be matched by magnitude and starting state?

Comments on the paper

1 comment

Dear Reviewers

The author-reviewer discussion period will close soon. The authors put much efforts for their rebuttal. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.