- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DreamIF, which augments a VLA policy with auxiliary prediction of future motion-selected visual regions. Offline point tracking identifies moving regions in demonstration videos, which provide masked future-image supervision. The model aggregates intermediate VLM representations, learns parallel multi-frame predictions, and introduces the resulting predictive representation into action decoding through cross-attention. Evaluation includes both in-distribution and out-of-distribution LIBERO-Plus protocols, standard LIBERO, and eight physical-robot tasks. The matched VLA-Adapter comparisons report improvements concentrated on longer tasks, with physical-task average success increasing from 65.0% to 80.6%.
Critical strengths
-
The matched experiments show a meaningful benefit on multi-stage tasks. Under the in-distribution protocol, LIBERO-Plus Long success increases from 76.6% to 85.5%. On the physical platform, DreamIF improves average success by 15.6 percentage points over VLA-Adapter using the same demonstrations and trial protocol.
-
The component comparisons support both selective prediction and explicit action conditioning. Full-frame prediction and removal of the cross-attention conditioning both reduce Long-suite performance. The offline construction of supervision also avoids requiring a tracker during deployment.
Critical weaknesses
-
The headline scaling improvement is confounded by backbone trainability. Appendix D.3 explicitly states that the π0.5 reference freezes its LLM backbone whereas DreamIF+π0.5 does not. The advertised 15.0-point improvement therefore mixes the contribution of DreamIF with that of additional backbone adaptation. A reproduced π0.5 baseline with identical trainability, training budget, and evaluation is necessary to substantiate this central claim.
-
The proposed “intention-associated” target is operationally a motion-saliency target. Region selection depends on displacement magnitude rather than instruction relevance. It can exclude a stationary object needed for the next operation or include unrelated moving regions. The paper does not establish that these masks identify intention-relevant regions reliably, which weakens the claimed distinction between intention reasoning and motion-focused auxiliary learning.
-
The contribution of predictive conditioning is not separated from additional representation capacity. Equations (8)–(10) condition action decoding on a projection of the aggregated VLM representation, rather than explicitly on decoded future predictions. Removing this conditioning path changes the architecture as well as the availability of the auxiliary representation. Consequently, the current ablations do not establish whether the improvement specifically requires future prediction or could arise from layer aggregation and an additional attention pathway.
-
The evidence for long-horizon error correction is under-specified. The open-loop analysis generates 250 action steps from a single initial observation despite an eight-step action horizon, but the continuation procedure is not adequately specified. Moreover, trajectory agreement with a demonstration does not directly establish recovery from execution deviations. This limits the support for the stronger claims concerning persistent predictive constraints and autonomous error correction.
Questions for authors
-
What is the performance of π0.5 without DreamIF when trained with exactly the same backbone trainability, data, optimization schedule, and evaluation pipeline as DreamIF+π0.5? This comparison would materially affect my assessment of the scaling claim.
-
How reliably do motion-selected masks capture the next instruction-relevant interaction? Please provide evidence for cases involving a stationary upcoming target or motion unrelated to the instructed manipulation.
-
Does action decoding consume decoded future predictions, their intermediate features, or only the aggregated VLM representation? What happens when the same aggregation and conditioning architecture is trained without the predictive objective?
-
How are the 250 open-loop actions generated after the first eight-step chunk? Which inputs are updated, and what information is available to the policy during continuation?
-
Were motion thresholds and prediction horizons selected on a separate validation split? Can you provide repeated-run uncertainty for the matched comparisons, including the full model versus the two component ablations?
4 comments
4. The 250-step protocol of Figure 7 and its interpretation (Weakness 4, Question 4)
The policy is queried once per recorded frame, and only the first action of the eight-step chunk is taken each time. At time t, both models receive the third-person image, the wrist image, and the proprioception of that frame from the expert demonstration. At t+1, the recorded input of the next demonstrated frame is used, and the model's predicted state is not used to generate the next observation. The predicted and expert increments are integrated from the same initial state, and the error is computed over six joints. The policy is therefore queried 250 times in total, and 250 single-step actions are retained.
Table 12 distinguishes per-step error from integrated trajectory deviation. Results for the same paired Long episode are as follows:
| Steps | Increment MSE, full / w/o feedback (×10⁻³) | Integrated MSE at segment end, full / w/o feedback |
|---|---|---|
| 1–50 | 4.44 / 14.37 | 4.32 / 11.69 |
| 51–100 | 7.80 / 14.66 | 10.78 / 32.34 |
| 101–150 | 2.71 / 10.75 | 11.41 / 68.25 |
| 151–200 | 3.02 / 14.29 | 13.53 / 126.67 |
| 201–250 | 4.38 / 9.71 | 21.28 / 131.65 |
| 1–250 | 4.47 / 12.76 | n/a |
The full-segment mean increment MSE without feedback is about 2.9× that of the full model, and the integrated deviation at step 250 is about 6.2×; these correspond to different metrics. On this episode, the per-step error is lower overall, and the absolute difference for Joint 5 is small. The evidence supports consistency with the expert trajectory under the same recorded inputs.
5. Choice of τ and K and the scope of a single run (Question 5)
The defaults are based on the current Long sensitivity sweep. Fig. 4(b,c) and Table 6 sweep τ ∈ {0.1, 0.5, 1.0, 5.0, 8.0} and K ∈ {1, 2, 4, 8}. At τ = 0.1, 1.0, and 8.0, the success rates are 79.2%, 85.5%, and 76.4%, and at K = 1, 4, and 8 they are 72.4%, 85.5%, and 84.8%. The defaults are τ = 1.0 and K = 4, at the peak between the two ends; the figure reports the sensitivity sweep on the Long suite.
Each row of Tables 9 and 10 comes from one training run, and the current differences are single-run comparisons under the matched setting.
We hope the responses above answer your questions, and we welcome further discussion on any remaining points.
Finally, we note that W1–W4 and Q1–Q5 concern the completeness of the comparison settings and of the evidence as presented, not the validity of the method itself. The relevant evidence is given in Sections 1–5 and in Tables 8A, 9–10 and 12, and the remaining points can be clarified in revision; we therefore regard them as scope issues that clarification can resolve.
Citations
[1] Black et al. π₀.₅: a Vision-Language-Action Model with Open-World Generalization. CoRL 2025.
[2] Zhong et al. ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Models. CVPR 2026, Table 2.
[3] Karaev et al. CoTracker: It Is Better to Track Together. ECCV 2024.
[4] Fei et al. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv:2510.13626, 2025.
[5] Wang et al. VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model. AAAI 2026.
Thanks for the response
I thank the authors for the clarifications and the matched π₀.₅ baseline, which substantially addresses W1/Q1. My main remaining concerns are W3/Q3 and W4/Q4.
For W3/Q3: The response clarifies that action decoding uses the current quantised latent, shaped by the predictive objective, rather than directly consuming predicted future states. However, it also explicitly acknowledges that the same-architecture control without predictive supervision has not been conducted. Removing dynamic feedback removes the conditioning pathway itself, while replacing masked targets with full-frame targets retains predictive supervision. Neither comparison establishes whether future prediction is necessary for the observed gains. The missing control should retain ALFA, the projection, quantiser, intention token, and residual attention under the same training budget, while setting only the auxiliary intention-loss weight to zero. Without this comparison, the contribution of predictive supervision remains unclear with the additional representation and conditioning capacity.
For W4/Q4: I think the response resolves the procedural ambiguity: the policy is queried 250 times using each recorded expert frame and proprioceptive state, retaining only the first action of each eight-step chunk. However, this clarification changes what the experiment can establish. Since the next input comes from the demonstration regardless of the model’s previous prediction, the policy never encounters the state deviations induced by its own actions. Integrating these predictions can show accumulated discrepancy from the expert trajectory, but cannot demonstrate recovery from execution errors. The reported reductions in per-step and integrated error therefore do not resolve the original concern about autonomous error correction. That claim requires evaluation under policy-induced or controlled execution deviations, or should be narrowed to consistency with expert actions under recorded inputs.
We thank you for reading our response again, and for noting that the π₀.₅ baseline added under matched training settings has gone a long way toward addressing W1/Q1. Your specific suggestion for the ablation control, and your distinction between agreement with the expert trajectory and recovery from execution deviations in Figure 7, have both helped us state the experimental results more clearly. Below we further address these two points in light of the two analyses in the revised manuscript.
1. W3/Q3 — The role of predictive supervision under a fixed action architecture
We thank you for proposing this specific control. Disabling the auxiliary intention loss while holding the action architecture and the training budget fixed would measure the contribution of the prediction objective itself to task success, and this is the piece we hope to add in future work.
Table 9 already compares two prediction targets: the two variants retain the same feature aggregation and action conditioning pathway and both receive future-frame supervision; they differ only in whether the model reconstructs the full frame or the motion-masked regions. With motion-masked targets, LIBERO-Plus Long rises from 79.2% to 85.5%. With the conditioning pathway held fixed, the choice of prediction target still affects action performance, and the difference is largest on Long. A second ablation retains the prediction objective and examines the effect of passing the prediction-supervised representation to the action decoder: enabling dynamic feedback raises Long from 79.5% to 85.5%.
Together, these two results show that the target content and the conditioning pathway each contribute, and they also provide a direct basis for the control you propose. We will add this comparison in future work: under the same training budget, keep ALFA, the projection module, the quantizer, the intention token and the residual attention, set only the weight of the auxiliary intention loss to zero, and report it in the same table as the two existing ablations.
2. W4/Q4 — Trajectory agreement under matched evaluation conditions
We thank you for pointing out that the source of the subsequent observations matters for the interpretation. When designing Figure 7, our aim was to keep the evaluation conditions as consistent as possible, so that we could compare how closely the predicted actions, with and without dynamic feedback, match the expert demonstration. Accordingly, on the same LIBERO-Plus Long demonstration, both policies receive the same third-person image, wrist image and proprioceptive information frame by frame across the 250 calls; each call retains only the first action of the eight-step chunk, and the retained actions are then integrated from the same initial joint state and compared against the same expert trajectory.
In this paired comparison, the full model has lower action-increment MSE in all five 50-step windows: over the full segment it falls from 12.76 × 10⁻³ to 4.47 × 10⁻³, and the integrated joint MSE at step 250 drops from 131.65 to 21.28 (Table 12). As you point out, the subsequent observation at every step comes from the demonstration, so we read the lower errors as evidence that, under these identical observations, both the step-by-step actions and the integrated trajectory of the full model are closer to the expert. The autonomous-recovery question you raise requires observing how the policy responds to state deviations caused by its own actions, and we will address it in future work.
Response to Reviewer Fg3h, brief summary: we add a matched in-distribution π₀.₅ baseline under one recipe, clarify that the motion mask is offline displacement-based supervision with its miss and inclusion cases, separate the conditioning-path ablation from the supervision-target ablation, and fully specify the paired 250-step open-loop protocol and the per-joint data.
We thank the reviewer for the specific questions on the π₀.₅ comparison, the motion mask, and the long-horizon evaluation. Below we respond in three parts with locations: the backbone comparison under identical recipes (Table 1, §5.2; settings in Appendix D.3), the role of the motion mask and action inputs (§4.1, §4.3, §5.3 and limitations; ablations in §5.4.1, Appendix E), and the 250-step prediction analysis (Figure 7 and its caption; pairing rules in Appendix G).
1. Matched π₀.₅ comparison (Weakness 1, Question 1)
Both π₀.₅[1] variants are trained and evaluated under the same in-distribution recipe. The 75.7% figure is the frozen-LLM π₀.₅ SFT reference row in Zhong et al., Table 2[2]. The two rows use the same LIBERO-Plus training corpus, optimisation schedule, eight-step action chunks, frozen vision towers with their LoRA settings, and evaluation pipeline; the experimental variable is whether DreamIF's predictive head and intention conditioning are added. Table 8A reports:
The average improves by 3.1 percentage points and Long by 7.6 percentage points. Along the perturbation dimensions of Table 1, layout improves by 6.5 points and noise by 6.0 points, while language drops by 1.5 points, showing that the gains vary with the condition. The scaling conclusion rests on the in-distribution pair, and the quoted OOD rows provide a cross-source reference.
2. Motion saliency and instruction relevance (Weakness 2, Question 2)
The motion mask is an offline supervision target, and the language instruction is a separate conditioning input to the policy. The mask is constructed from the first and last frames of a four-frame window: the displacement in each view is computed from CoTracker[3] point tracks, and its magnitude is then thresholded and dilated. A target that moves too little within the window may be missed, and an unrelated object that exceeds the threshold may be selected. Sparse future-frame supervision shapes the features used by the instruction-conditioned action branch. The mask itself selects regions by motion.
The current experiments include a stationary-target case and a clutter task. In the pot-and-lid task of Fig. 3, both policies complete the first three stages; DreamIF then picks up the stationary lid and closes the pot, while VLA-Adapter stops at the previous stage. For Task 3 in Table 2, with each method evaluated 20 times, DreamIF reaches 65% and the reactive baseline 50%. For Task 5, which places 3–5 distractors around the pot, DreamIF and VLA-Adapter both reach 80% (Tables 2 and 11). The lid example shows one successful subsequent interaction, and the clutter task does not isolate distractor motion.
3. Action conditioning, cross-layer aggregation, and predictive supervision (Weakness 3, Question 3)
The action head uses an intention token constructed from the current quantised latent and proprioception. The current-image encoding and the projected ALFA aggregate are summed and quantised into z_q,t. The prediction head outputs K future latent grids, and the training decoder reconstructs the corresponding masked targets. The action branch pools the current z_q,t, combines it with the projected proprioception, and obtains H′. The action state x^(n) queries a key/value context formed by concatenating the backbone action tokens, H′, and the layer-n visual tokens. The prediction objective shapes the representation used by the action branch, and the decoded future pixels are used in the training loss.
The two ablations answer different questions. On in-distribution LIBERO-Plus[4] Long, the full model reaches 85.5%, removing dynamic feedback gives 79.5%, full-frame supervision gives 79.2%, and our matched VLA-Adapter[5] baseline reaches 76.6% (Table 9). The 6.0 points compare the entire conditioning path, and the 6.3 points compare the two targets. Table 10 switches the layer input under the same budget; with the LIBERO Long setting noted in the caption, the first, middle, and last layers and all 24 layers give 42.6%, 73.2%, 77.5%, and 85.7%, respectively, which tests the choice of layers.
The revised version does not yet report a control that keeps the same action architecture while disabling the predictive objective. To isolate the contribution of predictive supervision, one should fix ALFA, the projection, the quantiser, the intention token and residual attention, and the training budget, and set only the auxiliary intention-loss weight to zero. The existing ablations directly measure the effect of the feedback path and of the layer input.