- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses latent world-model prediction with a transport function that learns what to copy from the observed encoded features into the future state, together with a learnable gate and a small correction network. This enforces a sensible structure on the predictor, but it requires further validation through scaling curves and downstream tasks.
Critical strengths
- The problem is well motivated: learning what to copy, instead of regenerating it, can improve the performance of latent world models.
- The method can easily be applied as a plug-in to existing world models.
- The experimental section is strong: several baselines are compared on the challenging DROID dataset (external camera views only).
Critical weaknesses
- Scaling curves: All results are in a low-data regime, and no data- or compute-scaling curves are presented. The authors should show that their conclusions remain valid with more data.
- Downstream tasks: The main motivation is robotics, yet no downstream task is demonstrated. What does your approach enable compared to standard V-JEPA 2-AC or DINO-WM?
- Wrist camera: DROID also provides a wrist camera view, which is not used in this work. The authors should report results on the wrist view and discuss whether their conclusions still hold for it.
Questions for authors
Could you provide scaling curves and show what your method enables for downstream robotics tasks that was not possible with existing predictors?
4 comments
Response (2/2): DROID wrist camera related experiments and results
Continuation of Response (1/2): scaling curves and downstream robotics tasks.
W3. Wrist camera. We trained and tested all predictors on DROID's wrist camera with the same episodes, splits and protocol (3 seeds, Section D.8).
| Wrist camera, test error | Persistence | AR | Direct | ShiftWM (7×7) | ShiftWM (11×11) |
|---|---|---|---|---|---|
| Mean over 10 steps | 1.064 | 0.694 | 0.675 | 0.669 | 0.658 |
| Error reduction vs. Direct | 0.9% [0.4, 1.5] | 2.6% [2.1, 3.1] |
- The ordering holds on this view. Averaged over the 10 steps, the unchanged ShiftWM has the lowest error on all three seeds, 0.9% below Direct and 3.7% [2.7, 4.8] below AR. The reduction is largest at short horizons (2.0% at step 1) and reaches parity at step 10; the wider window below restores it at every horizon.
- Why the margin is smaller than on the exterior camera. The wrist camera moves with the gripper, so nearly every patch moves: persistence error is 4.3× that of the exterior view. Optical flow shows that at step 10 the source of 29% of the patches lies outside the 7×7 transport window (exterior camera: 1%).
- We tested this explanation directly. With an 11×11 window and nothing else changed (same parameter count; the only alternative we ran), the error reduction over Direct rises to 2.6% and holds at every horizon (2.8% at step 1, 1.8% [1.3, 2.3] at step 10). A wider window therefore recovers the gain under camera motion; Section D.8 names an ego-motion term as the natural extension.
Changes to the paper. The method, Table 1, the main results and the conclusions are unchanged. All updated parts are highlighted in yellow in the revised PDF.
We hope these experimental details, results and discussion address the reviewer's concerns, and give a stronger motivation and confidence for deployment on complex tasks in real-robot close loop control. We would be grateful if the reviewer would consider the revised manuscript PDF for the final assessment. We are glad to provide further details if required. Thank you again for taking time to carefully review our submission.
Follow-up
Dear Reviewer,
Thank you again for your review. As the discussion period is about to close, we briefly summarise how we addressed your three requests.
- Scaling: ShiftWM's forecast error stays 7.3% to 8.5% below Direct over an eight-fold range of training data, and 7.4% to 8.6% below over a four-fold range of model size.
- Downstream: on real-robot video ShiftWM selects the true actions among 15 same-scene alternatives in 69.0% of windows (Direct 60.9%, AR 39.5%), and the head works as a drop-in with DINO-WM's planner.
- Wrist camera: ShiftWM has the lowest error on this view as well (0.9% below Direct, 2.6% with a wider transport window).
All of this is in the revised PDF (Appendix D.6 to D.8, highlighted in yellow). We would be grateful for any further feedback, and if these updates address your concerns, we kindly ask you to consider them in your final assessment.
Regards.
Followup on Rebuttal
Dear Reviewer;
Thank you for your time and your constructive feedback on our paper. As the rebuttal period will end soon, we would greatly appreciate it if you could revisit our responses and let us know whether any concerns or questions remain.
We hope these answers address the reviewer's concerns, and we would be happy to answer any further questions.
Thank you again for your time.
Regards Authors
Response (1/2): scaling curves and downstream robotics tasks
We thank the reviewer for finding the problem well motivated, the method easy to apply as a plug-in, and the experimental section strong. We ran all three requested experiments (3 seeds each; brackets are 95% bootstrap intervals over recording sessions).
Note: Response (2/2) covers the wrist camera based experiments and results.
W1. Scaling curves. Data scaling on DROID, 3 seeds per point, same test set (Section D.7); subsets are nested and drawn by recording session. Rows give the forecast error (feature MSE on test, mean over the 10 steps; lower is better) and how much lower ShiftWM's error is than Direct's:
Model scaling at full data:
All models were re-trained for these curves; the full-data point agrees with Table 1 of the submission (7.6%), which is unchanged.
W2. Downstream tasks, and what the approach enables. New experiments (Section D.6):
What this enables for real-robot control. In visual model-predictive control, the robot samples candidate action sequences, forecasts each one, and executes the one whose forecast is closest to the goal image. Measured on real-robot recordings, ShiftWM provides what each step of that loop needs, compared with regenerating, recursive predictors such as those of V-JEPA 2-AC and DINO-WM: