- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a three-stage "capture-gated" audit to evaluate whether latent world models (specifically LeWorldModel, PreJEPA, and TD-MPC2) accurately propagate the effects of actions over a 5-step open-loop rollout. Before scoring imagined rollouts (Gate 3), the protocol requires a frozen linear/affine readout to decode the current task query from the root state (Gate 1) and resolve true action-effect differences from executed environment endpoints (Gate 2). Evaluating checkpoints on DMControl tasks (Cheetah, Walker, Finger), the authors report that models often pass Gates 1 and 2 but achieve negative on Gate 3 imagined effects. They also test whether a discrepancy score between model predictions and a fitted quadratic environment response surface correlates with held-out action-selection regret.
Critical strengths
The distinction between static state decodability (capture) and counterfactual transition fidelity (propagation) is conceptually sensible. Requiring a probe to successfully resolve true endpoint differences (Gate 2) before blaming the latent dynamics model for rollout errors (Gate 3) is a reasonable sanity check that avoids conflating probe underfitting with transition failure.
Critical weaknesses
The paper suffers from severe presentation issues and reads like an automated software test log rather than a scientific manuscript. Simple linear probing and multi-step rollout evaluations are obscured behind heavy, non-standard terminology ("environment families", "matched-shell protocols", "closure coordinates"). Both the main text and appendices are padded with exhaustive administrative and logging minutiae—such as file paths, JSON filenames, and ten synthetic unit-test fixtures in Appendix A.1—while lacking deep mechanistic analysis or any constructive algorithmic remedy to improve world-model training.
Questions for authors
- Since a frozen linear readout trained on real states can easily fail under minor off-manifold latent drift during 5-step open-loop rollouts, how can Gate 3 distinguish genuine loss of action-effect information from simple linear-probe covariate shift (especially given that you acknowledge this ambiguity in Section 7)?
- In Section 4.1, Gate 2 fails for LeWorldModel and PreJEPA on environment-random roots and fails for all models on policy-generated roots, yet Section 5.2 notes that the actual regret of these failed-gate models overlaps with passed-gate TD-MPC2 runs in all 48 comparisons. Does this not indicate that your linear gating criteria are overly restrictive and poorly aligned with downstream decision quality?
- Given that the proposed discrepancy score requires oracle environment resets to fit a 495-term quadratic response surface, fails to improve over random features in half the settings, and shows no statistically significant gain over "always selecting" on Finger, what is the actionable utility of this audit for practitioners training or deploying world models?
3 comments
Presentation, where the failure lies, readout shift against genuine loss, and practical use
Thank you for the review. The check you name as a strength is kept: that the readout resolves real end-state differences is established before it is applied to imagined ones, now as a reported number (0.88–0.97) and not as a gate.
Presentation. We agree. The text now describes one design in plain terms. "Environment families", "matched-shell protocols" and "closure coordinates" are gone. File paths, JSON names and the synthetic test fixtures of Appendix A.1 are removed; the appendix keeps the tables behind the reported numbers and the details needed to reproduce them.
No mechanistic analysis or remedy. The added experiments locate the failure. The reward head is much less accurate on differences between actions than on level. Imagined states keep which action was applied, while an affine readout recovers less and less of its consequence with depth, and nonlinear readouts recover no more. The agents plan well with their value and poorly on the model's reward. At the planner's own horizon most of that gap remains when the transitions are the simulator's and only the scoring is learned, and neither a fitted calibration of the head's score nor sharper planner weights close it. From the same starting states, within 25 steps of reward-only control the head orders the candidates poorly (rank correlation 0.10–0.64, against 0.63–0.96 under the full planner at the same steps). We do not propose a training remedy.
Question 1: genuine loss or covariate shift of a frozen linear readout? The revision adds the control for this: the readout is refitted on imagined states at each depth, against a baseline that sees only the starting state and the actions. On TD-MPC2 Cheetah the refitted readout falls from 0.84–0.97 at real end states to 0.31–0.61 after three imagined steps, so the decline is not only shift. On LeWorldModel Finger a readout fitted on predicted states recovers the effect (0.75–0.80), so that failure was shift, and the revision says so. Nor is the decline specific to a linear readout: a two-layer network, kernel ridge and boosted trees, each with a baseline of its own family, show it too, and none recovers more after five imagined steps than the affine readout. The reward-head result uses the model's own head, trained on imagined states. In closed loop, readouts refitted on imagined states do no better than the frozen one (−48 to +21 return points, no consistent sign).
Question 2: failed-gate models overlap passed-gate ones in regret, so are the gates too restrictive? We agree that the pass/fail labels did not track decision quality in that comparison. The revision reports the gates' quantities as numbers and does not gate on them; decision quality is measured directly with the planner.
Question 3: actionable utility of the score. The score is cut. What remains for a practitioner is narrower and, we think, more usable: return and reward accuracy did not show whether these models could be planned with on their reward or reused for a new reward, and tests on matched action sequences did. The conclusion now says what such an evaluation consists of, where a simulator can be reset to a state: compare alternative actions from the same starting state; report conditional prediction and the quality of the selected candidate beside pooled accuracy; repeat both under the controller that will use the prediction; and run the planner with the simulator as its model, and with simulator transitions scored by the learned heads, to see how far transitions and scoring each limit it.
Responses and revised paper now posted
Thank you for checking in, and our apologies: our responses went up very late in the discussion period, after your note. They are now posted, as a general response to all reviewers and an individual response to your review, together with a revised PDF.
The revision keeps the submission's claim and measurement and adds the experiments the reviews asked for. In our response to your review, the refitted-readout control for your Question 1 may be the quickest place to start.
We understand that this leaves you little time. If you are able to, we would be grateful for any reconsideration.
No rebuttal provided; the original score remains unchanged.