acceptodds
Under review as a conference paper at ICLR 2027

ShiftWM: World Models that Move What They Saw

Abstract

World models let an agent evaluate actions before executing them, and forecasting in the patch-feature space of a frozen visual foundation model keeps them compact. Current latent world models, such as DINO-WM and V-JEPA 2-AC, regenerate the whole feature grid at every future step and feed their predictions back as inputs. Yet in manipulation and surgical scenes most content persists and only moves, so regeneration corrupts the static background and lets errors compound. We introduce ShiftWM, a latent world model that forecasts by moving what it has already seen: for every future step it predicts an action-conditioned soft transport of nearby observed patch features, learned without flow supervision, a gate between moved and kept features and a correction for new content, and it decodes all horizons in parallel. As an output head, it also attaches to open-source world models. On held-out real-robot video from DROID, Language-Table and the bimanual IWS tasks, and on seven surgical tasks from Open-H, ShiftWM has the lowest error of all matched output heads, 7.6% below a direct predictor on DROID; as a plug-in, it lowers the error of the 1.3B-parameter V-JEPA 2-AC by 15.7%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, feature forecasting, action-conditioned prediction, DINOv2, robot learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P5RjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses latent world-model prediction with a transport function that learns what to copy from the observed encoded features into the future state, together with a learnable gate and a small correction network. This enforces a sensible structure on the predictor, but it requires further validation through scaling curves and downstream tasks.

Critical strengths
  • The problem is well motivated: learning what to copy, instead of regenerating it, can improve the performance of latent world models.
  • The method can easily be applied as a plug-in to existing world models.
  • The experimental section is strong: several baselines are compared on the challenging DROID dataset (external camera views only).
Critical weaknesses
  • Scaling curves: All results are in a low-data regime, and no data- or compute-scaling curves are presented. The authors should show that their conclusions remain valid with more data.
  • Downstream tasks: The main motivation is robotics, yet no downstream task is demonstrated. What does your approach enable compared to standard V-JEPA 2-AC or DINO-WM?
  • Wrist camera: DROID also provides a wrist camera view, which is not used in this work. The authors should report results on the wrist view and discuss whether their conclusions still hold for it.
Questions for authors

Could you provide scaling curves and show what your method enables for downstream robotics tasks that was not possible with existing predictors?

4 comments
AuthorsOpenReview ↗

Response (1/2): scaling curves and downstream robotics tasks

We thank the reviewer for finding the problem well motivated, the method easy to apply as a plug-in, and the experimental section strong. We ran all three requested experiments (3 seeds each; brackets are 95% bootstrap intervals over recording sessions).

Note: Response (2/2) covers the wrist camera based experiments and results.

W1. Scaling curves. Data scaling on DROID, 3 seeds per point, same test set (Section D.7); subsets are nested and drawn by recording session. Rows give the forecast error (feature MSE on test, mean over the 10 steps; lower is better) and how much lower ShiftWM's error is than Direct's:

Training episodes106 (1/8)208 (1/4)419 (1/2)830 (all)
ShiftWM0.22160.20450.19270.1803
Direct0.23900.22060.21070.1955
ShiftWM error reduction7.3% [6.2, 8.4]7.3% [6.3, 8.3]8.5% [7.3, 9.8]7.8% [7.0, 8.5]

Model scaling at full data:

Width (parameters)256 (9.8M)384 (21.8M)512 (38.4M)
ShiftWM0.18130.18030.1796
Direct0.19830.19550.1940
ShiftWM error reduction8.6% [7.6, 9.6]7.8% [7.0, 8.5]7.4% [6.7, 8.2]

All models were re-trained for these curves; the full-data point agrees with Table 1 of the submission (7.6%), which is unchanged.

  • The conclusion holds across data sizes. Over an eight-fold range up to the full training set, both models improve and ShiftWM's error stays 7.3% to 8.5% below Direct's, with no sign of narrowing.
  • It holds across model sizes. ShiftWM's error stays 7.4% to 8.6% below Direct's, and a four times larger Direct model (38.4M) is still above the smallest ShiftWM (9.8M).
  • Large-scale evidence in the paper. On V-JEPA 2-AC, a 1.3B-parameter model pretrained at large scale, the head lowers the error by 15.7% with 0.16M added parameters.

W2. Downstream tasks, and what the approach enables. New experiments (Section D.6):

  • Action selection on real robot video. Ranking the true actions against 15 other sequences from the same episode, ShiftWM is correct in 69.0% of DROID test windows, Direct in 60.9% and the recursive predictor AR in 39.5% (chance 6.3%). On DROID, ShiftWM is also ahead of Direct under every local position offset, noise level and time shift.
  • Drop-in use with DINO-WM's planner. Added to the official DINO-WM code, the head lowers the one-step forecast error by 6.5% on every seed, and the official CEM planner on PushT runs unchanged on top of it: success is 80.4 ± 0.4% with the head and 82.4 ± 1.7% without (3 training seeds, 3 planning seeds × 50 episodes), within the variation between planning runs (76 to 92%).

What this enables for real-robot control. In visual model-predictive control, the robot samples candidate action sequences, forecasts each one, and executes the one whose forecast is closest to the goal image. Measured on real-robot recordings, ShiftWM provides what each step of that loop needs, compared with regenerating, recursive predictors such as those of V-JEPA 2-AC and DINO-WM:

  • Choosing between nearby actions. The forecast moves observed content according to the commanded actions, so it responds to small action changes: its error rises 7.3% at a 3σ offset, against 1.7% for the recursive predictor, and it selects the true actions in 69.0% vs. 39.5% of DROID windows. This is the signal a controller uses to refine a grasp or a reach.
  • Goal-image costs in cluttered scenes. The identity path keeps static content exact (on DROID, the only learned model at or below persistence on static patches, Table 1), so the distance to a goal reflects what the robot moved, not background drift.
  • Faster re-planning. All K steps come from one pass with no prediction fed back: 1.5× more candidates per second than a recursive rollout (1,155 vs. 778), with no compounding error over the horizon.
  • Tracking the arm without a decoder. The transport field says where each patch's content goes, which places the arm or instrument directly in feature space (Figure 14). A regression head has no such output.
  • Upgrading existing world models. A 0.16M-parameter head lowers the error of the 1.3B-parameter V-JEPA 2-AC by 15.7%, with its training recipe unchanged.
AuthorsOpenReview ↗

Response (2/2): DROID wrist camera related experiments and results

Continuation of Response (1/2): scaling curves and downstream robotics tasks.

W3. Wrist camera. We trained and tested all predictors on DROID's wrist camera with the same episodes, splits and protocol (3 seeds, Section D.8).

Wrist camera, test errorPersistenceARDirectShiftWM (7×7)ShiftWM (11×11)
Mean over 10 steps1.0640.6940.6750.6690.658
Error reduction vs. Direct0.9% [0.4, 1.5]2.6% [2.1, 3.1]
  • The ordering holds on this view. Averaged over the 10 steps, the unchanged ShiftWM has the lowest error on all three seeds, 0.9% below Direct and 3.7% [2.7, 4.8] below AR. The reduction is largest at short horizons (2.0% at step 1) and reaches parity at step 10; the wider window below restores it at every horizon.
  • Why the margin is smaller than on the exterior camera. The wrist camera moves with the gripper, so nearly every patch moves: persistence error is 4.3× that of the exterior view. Optical flow shows that at step 10 the source of 29% of the patches lies outside the 7×7 transport window (exterior camera: 1%).
  • We tested this explanation directly. With an 11×11 window and nothing else changed (same parameter count; the only alternative we ran), the error reduction over Direct rises to 2.6% and holds at every horizon (2.8% at step 1, 1.8% [1.3, 2.3] at step 10). A wider window therefore recovers the gain under camera motion; Section D.8 names an ego-motion term as the natural extension.

Changes to the paper. The method, Table 1, the main results and the conclusions are unchanged. All updated parts are highlighted in yellow in the revised PDF.

We hope these experimental details, results and discussion address the reviewer's concerns, and give a stronger motivation and confidence for deployment on complex tasks in real-robot close loop control. We would be grateful if the reviewer would consider the revised manuscript PDF for the final assessment. We are glad to provide further details if required. Thank you again for taking time to carefully review our submission.

Dear Reviewer,

Thank you again for your review. As the discussion period is about to close, we briefly summarise how we addressed your three requests.

  • Scaling: ShiftWM's forecast error stays 7.3% to 8.5% below Direct over an eight-fold range of training data, and 7.4% to 8.6% below over a four-fold range of model size.
  • Downstream: on real-robot video ShiftWM selects the true actions among 15 same-scene alternatives in 69.0% of windows (Direct 60.9%, AR 39.5%), and the head works as a drop-in with DINO-WM's planner.
  • Wrist camera: ShiftWM has the lowest error on this view as well (0.9% below Direct, 2.6% with a wider transport window).

All of this is in the revised PDF (Appendix D.6 to D.8, highlighted in yellow). We would be grateful for any further feedback, and if these updates address your concerns, we kindly ask you to consider them in your final assessment.

Regards.

AuthorsOpenReview ↗

Followup on Rebuttal

Dear Reviewer;

Thank you for your time and your constructive feedback on our paper. As the rebuttal period will end soon, we would greatly appreciate it if you could revisit our responses and let us know whether any concerns or questions remain.

We hope these answers address the reviewer's concerns, and we would be happy to answer any further questions.

Thank you again for your time.

Regards Authors

Reviewer_zEYpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ShiftWM predicts future visual latents by combining local soft transport of observed features, a content-preservation gate, and a residual correction term across parallel prediction horizons. The paper evaluates feature forecasting across multiple robotic and video datasets and tests the transport head as a plug-in module for existing world models.

Critical strengths
  1. The inductive bias is isolated through rigorous ablations: the parameter-matched Direct baseline shares the exact cross-attention backbone and parallel horizon setup, and additional controls separately test autoregressive rollout training, action conditioning, gating, residual correction, and explicit optical-flow warping.

  2. Forecasting improvements are consistent across datasets—including a 7.6% error reduction over Direct on DROID backed by 3 training seeds and session-level bootstrap intervals—and plugging the transport head into V-JEPA 2-AC demonstrates transferability to an existing pretrained architecture.

Critical weaknesses
  1. The paper's downstream control and semantic readout evaluations do not convincingly demonstrate a practical planning advantage. The action-ranking task only distinguishes the ground-truth action sequence from actions drawn from a completely different episode, rather than ranking plausible local trajectories within the same scene. Similarly, the feature-level segmentation readout in Figure 14 shows a +2.0 IoU gain [1.2,2.7][1.2, 2.7] on DROID but virtually no change on Hamlyn (+0.1+0.1 [−0.2,0.4][-0.2, 0.4]), while Figure 7 showcases qualitative RGB-decoded examples chosen specifically to maximize step-10 advantage (a selection rule buried in Appendix D.5 rather than stated in the figure caption).

  2. The DINO-WM plug-in experiment in Appendix B.3 operates under a different rollout regime than the main model and lacks complete quantitative reporting. Unlike the main ShiftWM model—which transports observed context features in parallel—the DINO-WM head feeds its own predicted features back as transport sources from step 2 onward. The headline 8.0% gain is purely 1-step teacher-forced, while the open-loop rollout curves in Appendix D.2 (Figure 12d) are presented without exact numerical values or multi-seed variance.

Questions for authors
  1. Can you evaluate action ranking using locally perturbed candidate trajectories within the same scene (or report a matched-budget MPC closed-loop success rate)? Also, please state the Appendix D.5 example-selection rule directly in the caption of Figure 7 and clarify the protocol difference between Figures 7 and 14.

  2. For the DINO-WM plug-in experiment, what are the exact numerical errors and seed variances for both teacher-forced and multi-step open-loop rollouts, and how quickly does transporting self-predicted features degrade over longer horizons compared to transporting observed features?

3 comments
AuthorsOpenReview ↗

Response (1/2): Action ranking, segmentation readout, Details about Figures 7 and 14

We thank the reviewer for the detailed review and for recognizing the matched Direct baseline, the separate controls for each component, and the seed and session-level statistics. Every questions are addressed below. Next part of the response (2/2) will cover the DINO-WM plug-in.

Note: Brackets are 95% bootstrap intervals.

Action ranking within the same scene. We agree that another episode's actions are a coarse alternative. Following the suggestion, we rank the true sequence against alternatives from the same scene: 15 sequences from other windows of the same episode, and local perturbations of the true actions (constant position offsets along ±x, ±y, ±z; Gaussian position noise; one-step time shifts; a flipped gripper command). Sizes are in units of σ = 2.3 cm, the standard deviation of the per-step change of the commanded position. All models score identical candidates. Share of DROID test windows (3 seeds) in which the true actions have the lowest forecast error (Section D.6):

Alternatives (number)ShiftWMDirectAR
same episode (15)69.060.939.5
offset 2σ / 8σ (6)21.2 / 78.613.7 / 67.54.7 / 38.6
noise 2σ / 8σ (15)40.3 / 92.232.0 / 89.643.4 / 85.3
time shift (2)48.341.824.0
gripper flip (1)53.653.651.4

The table averages over windows (3,923; 3,447 for the same-episode row). The paired differences below average per episode first and resample recording sessions.

  • ShiftWM vs. Direct: higher for every position offset, noise level and time shift, with every interval excluding zero (same episode +8.6 points [6.6, 10.9]; 8σ offset +11.4 [8.8, 14.0]; time shift +7.3 [5.2, 9.4]).
  • ShiftWM vs. AR: higher on the same-episode test, every offset, the time shift and noise of 4σ and above; at smaller noise the two are statistically tied (2σ: 0.0 [−3.2, +3.5]).
  • Energy landscape (constant x-y offset on a 13×13 grid up to 3σ): all models have their minimum at or within half a step of the true actions; the error rises by 7.3% at 3σ for ShiftWM, 5.0% for Direct and 1.7% for AR.
  • Open-H Hamlyn (3,434 windows; two instruments, so twice the offset, noise and flip alternatives): the same-episode test is saturated (98.8% for ShiftWM and Direct), so local perturbations are the informative ones. ShiftWM is ahead of Direct for offsets of 1 to 8σ (2σ: 55.1% vs. 51.7%, +3.7 [1.9, 5.4]), noise of 0.5 to 2σ and time shifts.
  • Gripper command. ShiftWM's advantage concerns where and when the arm moves, which is what the transport models. A flipped gripper command alone displaces little content: on DROID the three models score alike (53.6%, 53.6% and 51.4%), and on Hamlyn all three are above chance (33%), with AR highest (60.5%) and ShiftWM and Direct at 48.0% and 50.7%. Table 5 in D.6 lists every alternative and size; ShiftWM is the highest of the three at every offset size, from 0.5σ to 8σ, on both datasets.

As the reviewer offered a matched-budget planning success rate as an alternative; we report it as well in response (2/2).

Segmentation readout on Hamlyn. We agree, and Section 5.3 now says that on Hamlyn the three predictors are close. A consistent explanation is that the readout gain grows with mask motion, and Hamlyn instruments move little within ten steps (median mask overlap between t and t+K: 0.88 on Hamlyn, 0.61 on DROID). Splitting windows by mask motion (Section D.4, intervals over episodes):

  • DROID: IoU gain over Direct of +2.0 points [1.1, 2.8] in the third with most motion and +0.5 [−0.1, 1.2] in the third with least.
  • Hamlyn, third with most motion: the mask is placed more accurately than by Direct (centroid error lower by 0.37 px [0.08, 0.70]); IoU is within noise of Direct (+0.2 [−0.3, 0.7]) and above AR (+0.7 [0.1, 1.2]).

In feature error, ShiftWM is below Direct on all seven Open-H tasks (Figure 8f).

Figure 7 selection rule. Thank you; the rule belongs in the caption. The captions of Figures 1, 2 and 7 now state it (held-out windows among the 30% with most motion, with the largest step-10 advantage), and the new Figure 18 shows windows drawn at random with a fixed seed, plus the window at the median advantage.

Protocol of Figures 7 and 14. Both captions now state the difference:

  • Figure 7b is an illustration. Each method's step-10 forecast is decoded to RGB with the same feature-to-RGB decoder, and the decoded image is segmented with SAM 2.1, on the selected windows.
  • Figure 14 is the quantitative evaluation and uses no decoder. Each forecast patch is labelled foreground if it is closer (cosine) to a foreground than to a background patch of the last observed grid. IoU and placement are computed against the SAM 2.1 track of the true future on the held-out windows that pass the tracking filter, with paired intervals.

All quantitative claims rest on Figure 14.

AuthorsOpenReview ↗

Response (2/2): Rollout regime and DINO-WM plug-in, exact numbers with 3 seeds

Continuation after Response (1/2): Action ranking, segmentation readout, Details about Figures 7 and 14.

Rollout regime. The reviewer is right, and Section 5.2 now says so. To leave the official DINO-WM predictor unmodified, the head takes its sources from the predictor's input tokens; from the second rollout step these are DINO-WM's own predictions. ShiftWM itself and the V-JEPA 2-AC head read observed tokens only. The gain in the submission is the teacher-forced one-step error, and it is now labelled as such in the caption of Figure 1, the introduction and Section 5.2.

Exact numbers and seed variance. We trained 3 seeds per arm with the official code and the same budget for both arms (2 epochs, as in the submission). Visual latent error on PushT validation, mean ± std over seeds (Table 3 lists all five open-loop steps, SSIM and LPIPS):

DINO-WM+ ShiftWM headchange (per seed)
teacher-forced0.0630 ± 0.00030.0589 ± 0.0004−6.5% (−5.9, −6.3, −7.5)
open-loop, step 10.0681 ± 0.00390.0654 ± 0.0063−4.0% (−2.3, −0.9, −9.3)
open-loop, step 30.0977 ± 0.00140.0965 ± 0.0084−1.2% (−7.9, +9.2, −4.8)
open-loop, step 50.1382 ± 0.01440.1338 ± 0.0247−3.2% (−14.7, −2.2, +5.7)
  • Teacher-forced: the head is better on every seed, by 6.5% on average, with a gap ten times the seed std. This replaces the single-seed 8.0% of the submission, and Table 3 now reports the three re-trained seeds for both arms.
  • Open-loop: the mean error is 1 to 4% lower with the head at every step, within seed-to-seed variation from step 2. The corresponding sentence in Appendix D.2 now says this.

How quickly does transporting self-predicted features degrade, compared to observed features?

  • Self-predicted sources (DINO-WM plug-in). The gain is clear when the sources are true features (teacher-forced, all seeds). In open-loop rollouts, where the sources are the model's own predictions from step 2, the mean error stays 1 to 3% below DINO-WM through step 5, within seed-to-seed variation.
  • Observed sources. The V-JEPA 2-AC head reads observed tokens at every step, and its gain holds at every horizon (Figure 12c), as does the gain of ShiftWM over Direct (Figure 10).

Also it is important to note that the official DINO-WM predictor exposes only its input tokens to the head, so the observed-source regime is represented by the V-JEPA 2-AC head and by ShiftWM itself. Together, the evidence indicates that the benefit comes from moving observed content, which is why ShiftWM decodes all horizons in parallel from observed features; Section 5.2 now states this.

Matched-budget planning. We use the open-loop CEM planner of the official code on PushT (horizon 5, 300 samples, 30 optimisation steps), with an identical budget for both arms, 3 training seeds and 3 planning seeds × 50 episodes each. The official MPC variant re-plans in the simulator at every step (about 3.4 h per iteration per run), so we use the official CEM planner, which allows 3 training seeds × 3 planning seeds per arm.

mean ± std over training seedsSuccess (%)Final state distance
DINO-WM82.4 ± 1.727.6 ± 1.2
+ ShiftWM head80.4 ± 0.428.7 ± 1.2

Individual planning runs range from 76 to 92%, so the difference of 2.0 points is within planning-seed variation: the head can be added to DINO-WM with its planner running unchanged (Section D.6). This agrees with the open-loop numbers above, since the planner rolls the model out on its own predictions. The same-scene ranking in (1/2) evaluates ShiftWM in its own regime, where it moves observed features.

Changes to the paper. The method, Table 1, the main results and the conclusions are unchanged. All updated parts are highlighted in yellow in the revised manuscript PDF.

We thank the reviewer again. These points made the paper more precise, and we are glad to provide any further detail.

AuthorsOpenReview ↗

Followup on Rebuttal

Dear Reviewer;

Thank you for your time and your constructive feedback on our paper. As the rebuttal period will end soon, we would greatly appreciate it if you could revisit our responses and let us know whether any concerns or questions remain.

We hope these answers address the reviewer's concerns, and we would be happy to answer any further questions.

Thank you again for your time.

Authors

Reviewer_Ebf2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose ShiftWM, a latent world model that forecasts each future patch by moving features it has already observed. This approach can be seen as a post-hoc plugin, and as ShfitWM changes only how the forecast is formed, the authors demonstrate it by plugging it into open-source models without changing their backbone. The provided experiments show benefits across models and setups.

Critical strengths

The key strength of this work is the post-hoc nature and simple and well-motivated idea: The paper offers a seemingly practical and computationally lightweight improvement to latent world-model rollout/forecasting. This strength is also related to the experiments/demonstrations on open-source models.

Critical weaknesses

The weaknesses of the work are:

  1. The overall idea in this work is very closely related to feature-warping and motion-compensation techniques. The primary novelty is adapting the principles to action-conditioned latent world models, which has value. The paper is kind of frank about this.

  2. The experimental evaluation primarily measures latent feature prediction errors rather than improvements in closed-loop planning, control, or success in practical robotic tasks. The chosen evaluation setup is sensible and understandable in the paper, but leaves practical downstream impact of the work open.

Questions for authors

Please, comment on the weaknesses I highlighted above.

3 comments
AuthorsOpenReview ↗

Distinction to feature warping and motion compensation and downstream impact of our proposed method ShiftWM

We thank the reviewer for the positive assessment, and for highlighting the simple, well-motivated idea and its practical, lightweight use with open-source models. Our response to your questions are as follows:

Note: New models in experiments done during the rebuttal use 3 seeds; brackets are 95% bootstrap intervals over recording sessions.

W1. Distinction to feature warping and motion compensation. We agree on the lineage and state it in Section 2. Beyond adapting it to action-conditioned latent world models, the mechanism differs from a classical warp in ways that we can measure:

  • What differs. The transport is conditioned on the commanded actions and learned from the forecasting loss alone (no flow supervision, no flow estimator at test time); it is soft and draws on several observed frames. An exact identity path sits behind a gate; and all horizons are decoded in parallel from observed features, so no prediction is fed back.
  • The ablated differences matter. Removing multi-frame sources raises the forecast error by 4.8%, and removing the ability to move (1×1 window) by 7.4%, as much as removing the transport altogether (7.3%).
  • It is more accurate than flow warping. On the same DROID test windows, ShiftWM has 25.4% lower forecast error than warping with RAFT flow and 18.1% lower than warping with the true future flow, a strong reference for any pure warp of the last frame (Section 5.3).
  • NEW: a learned warping head. We trained the closest classical design under our recipe: an F2MF-style head on the same backbone and data (3 seeds), in which the transport is replaced by a warp of the last frame's features along one displacement per patch, with the same gate and the same learned regression. Its forecast error on the DROID test set (feature MSE, mean over the 10 steps) is 0.1959 with RAFT flow and 0.1947 with a displacement regressed by the network, the level of plain regression (Direct: 0.1955). ShiftWM reaches 0.1803: its forecast error is 7.9% [7.2, 8.7] lower than that of the RAFT-flow head and 7.4% [6.6, 8.1] lower than that of the learned-displacement head, so it predicts the future features more accurately than both. A single-displacement warp of one frame does not give this improvement; the soft, multi-frame, action-conditioned transport does (Section D.4).

Our paper also contributes the diagnosis that motivates the design: on DROID, every regression predictor we train is worse than copying the last frame on static patches (Table 1), and an oracle move of observed features removes 51% of the persistence error on moving content (Section 5.3). Section 2 now states the comparison with warping explicitly.

W2. Downstream impact. We chose held-out feature error as the primary metric because, with a shared backbone, it isolates the output head. We agree that downstream evidence strengthens the paper and added it (Section D.6):

  • Action selection on real video. We rank the true action sequence against plausible alternatives from the same scene. Against 15 other sequences from the same episode, ShiftWM selects the true actions in 69.0% of DROID test windows, Direct in 60.9% and AR in 39.5% (chance 6.3%). Under local perturbations of the true actions, ShiftWM is ahead of Direct for every position offset, noise level and time shift on DROID (for example 78.6% vs. 67.5% for an 8σ offset), and its error rises most steeply away from the true actions (7.3% at 3σ vs. 5.0% for Direct and 1.7% for AR). A sampling-based planner relies on exactly this sensitivity to the candidate actions.
  • Drop-in use with DINO-WM's planner. Added to the official DINO-WM code without changing its predictor or training recipe, the head lowers the one-step forecast error by 6.5% on every seed, and the official CEM planner on PushT runs unchanged on top of it: success is 80.4 ± 0.4% with the head and 82.4 ± 1.7% without (3 training seeds, 3 planning seeds × 50 episodes each), within the variation between individual planning runs (76 to 92%). In this plug-in the planner rolls out on DINO-WM's own predictions; ShiftWM's own design, which moves observed features, is evaluated in the action-selection test above, where it selects the true actions most often.
  • Cost of candidate evaluation. ShiftWM decodes all K steps in one pass: 1,155 candidate action sequences per second on one GPU, against 778 for a recursive predictor of the same size.

Together, these are what a visual control loop on a real robot needs at each step: telling nearby actions apart, a goal cost that reflects what the robot moved, and fast evaluation of many candidates which is very critical for real robot control for complex tasks.

Changes to the paper. The method, Table 1, the main results and the conclusions are unchanged. All updated parts are highlighted in yellow in the revised PDF.

We hope these results answer both points, and we would be grateful if the reviewer would consider them in the final assessment.

Thank you for the rebuttal and clarifications. These indeed clarified things. My assessment continues to lean toward the positive side.

AuthorsOpenReview ↗

Thank you!

Thank you again for the close engagement, which has made the paper sharper and its claims more precise. All the updates now reflect in the revised PDF.

Regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.