acceptodds
Under review as a conference paper at ICLR 2027

DreamIF: Intention-Guided Sparse Dynamics Prediction for Long-Horizon Vision-Language-Action Manipulation

Abstract

Leveraging robust semantic representations of pre-trained VLMs, Vision-Language-Action (VLA) models build end-to-end multimodal perception-to-action mappings to boost semantic alignment and task generalization for embodied manipulation. Nevertheless, existing VLA paradigms lack task-oriented dynamics inference, making them vulnerable to irrelevant background redundancy and cumulative errors in long-horizon manipulation. This work proposes Dream Intention Flow (DreamIF), a novel intention-guided embodied foundation model that enables VLA to focus on intention-relevant core cues and mitigate environmental redundancy and long-horizon errors. We design an intention-driven dynamics inference mechanism: optical-flow-guided motion saliency locates Intention-Associated Dynamic Visual Regions (IADVR) to filter redundant backgrounds, capture physical transition rules, and extract action intentions. The model synchronously predicts multi-frame future IADVR features and embeds these temporal priors into the native action representations, strengthening temporal modeling and shifting control from reactive to predictive. Evaluated on the comprehensive LIBERO-PLUS benchmark under matched training data, DreamIF attains 85.7% on average, and extending its method to a backbone lifts that backbone's average success rate by 15.0 points. Notably, on the long-horizon LIBERO-PLUS Long suite it achieves an 85.5% success rate, 8.9 points above the matched baseline, and on eight real-world robotic tasks involving environmental noise and long-horizon manipulation it raises the average success rate by 15.6 points. Experimental results validate that DreamIF greatly enhances the ability to model task-oriented dynamics and remedies the inherent limitations of conventional VLA models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Intention-driven manipulation, Vision-language-action (VLA)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Fg3hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DreamIF, which augments a VLA policy with auxiliary prediction of future motion-selected visual regions. Offline point tracking identifies moving regions in demonstration videos, which provide masked future-image supervision. The model aggregates intermediate VLM representations, learns parallel multi-frame predictions, and introduces the resulting predictive representation into action decoding through cross-attention. Evaluation includes both in-distribution and out-of-distribution LIBERO-Plus protocols, standard LIBERO, and eight physical-robot tasks. The matched VLA-Adapter comparisons report improvements concentrated on longer tasks, with physical-task average success increasing from 65.0% to 80.6%.

Critical strengths
  1. The matched experiments show a meaningful benefit on multi-stage tasks. Under the in-distribution protocol, LIBERO-Plus Long success increases from 76.6% to 85.5%. On the physical platform, DreamIF improves average success by 15.6 percentage points over VLA-Adapter using the same demonstrations and trial protocol.

  2. The component comparisons support both selective prediction and explicit action conditioning. Full-frame prediction and removal of the cross-attention conditioning both reduce Long-suite performance. The offline construction of supervision also avoids requiring a tracker during deployment.

Critical weaknesses
  1. The headline scaling improvement is confounded by backbone trainability. Appendix D.3 explicitly states that the π0.5 reference freezes its LLM backbone whereas DreamIF+π0.5 does not. The advertised 15.0-point improvement therefore mixes the contribution of DreamIF with that of additional backbone adaptation. A reproduced π0.5 baseline with identical trainability, training budget, and evaluation is necessary to substantiate this central claim.

  2. The proposed “intention-associated” target is operationally a motion-saliency target. Region selection depends on displacement magnitude rather than instruction relevance. It can exclude a stationary object needed for the next operation or include unrelated moving regions. The paper does not establish that these masks identify intention-relevant regions reliably, which weakens the claimed distinction between intention reasoning and motion-focused auxiliary learning.

  3. The contribution of predictive conditioning is not separated from additional representation capacity. Equations (8)–(10) condition action decoding on a projection of the aggregated VLM representation, rather than explicitly on decoded future predictions. Removing this conditioning path changes the architecture as well as the availability of the auxiliary representation. Consequently, the current ablations do not establish whether the improvement specifically requires future prediction or could arise from layer aggregation and an additional attention pathway.

  4. The evidence for long-horizon error correction is under-specified. The open-loop analysis generates 250 action steps from a single initial observation despite an eight-step action horizon, but the continuation procedure is not adequately specified. Moreover, trajectory agreement with a demonstration does not directly establish recovery from execution deviations. This limits the support for the stronger claims concerning persistent predictive constraints and autonomous error correction.

Questions for authors
  1. What is the performance of π0.5 without DreamIF when trained with exactly the same backbone trainability, data, optimization schedule, and evaluation pipeline as DreamIF+π0.5? This comparison would materially affect my assessment of the scaling claim.

  2. How reliably do motion-selected masks capture the next instruction-relevant interaction? Please provide evidence for cases involving a stationary upcoming target or motion unrelated to the instructed manipulation.

  3. Does action decoding consume decoded future predictions, their intermediate features, or only the aggregated VLM representation? What happens when the same aggregation and conditioning architecture is trained without the predictive objective?

  4. How are the 250 open-loop actions generated after the first eight-step chunk? Which inputs are updated, and what information is available to the policy during continuation?

  5. Were motion thresholds and prediction horizons selected on a separate validation split? Can you provide repeated-run uncertainty for the matched comparisons, including the full model versus the two component ablations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fg3h, brief summary: we add a matched in-distribution π₀.₅ baseline under one recipe, clarify that the motion mask is offline displacement-based supervision with its miss and inclusion cases, separate the conditioning-path ablation from the supervision-target ablation, and fully specify the paired 250-step open-loop protocol and the per-joint data.

We thank the reviewer for the specific questions on the π₀.₅ comparison, the motion mask, and the long-horizon evaluation. Below we respond in three parts with locations: the backbone comparison under identical recipes (Table 1, §5.2; settings in Appendix D.3), the role of the motion mask and action inputs (§4.1, §4.3, §5.3 and limitations; ablations in §5.4.1, Appendix E), and the 250-step prediction analysis (Figure 7 and its caption; pairing rules in Appendix G).

1. Matched π₀.₅ comparison (Weakness 1, Question 1)

Both π₀.₅[1] variants are trained and evaluated under the same in-distribution recipe. The 75.7% figure is the frozen-LLM π₀.₅ SFT reference row in Zhong et al., Table 2[2]. The two rows use the same LIBERO-Plus training corpus, optimisation schedule, eight-step action chunks, frozen vision towers with their LoRA settings, and evaluation pipeline; the experimental variable is whether DreamIF's predictive head and intention conditioning are added. Table 8A reports:

MethodReported avg.SpatialObjectGoalLong
π₀.₅, matched recipe87.692.589.685.882.4
DreamIF+π₀.₅90.793.893.186.190.0

The average improves by 3.1 percentage points and Long by 7.6 percentage points. Along the perturbation dimensions of Table 1, layout improves by 6.5 points and noise by 6.0 points, while language drops by 1.5 points, showing that the gains vary with the condition. The scaling conclusion rests on the in-distribution pair, and the quoted OOD rows provide a cross-source reference.

2. Motion saliency and instruction relevance (Weakness 2, Question 2)

The motion mask is an offline supervision target, and the language instruction is a separate conditioning input to the policy. The mask is constructed from the first and last frames of a four-frame window: the displacement in each view is computed from CoTracker[3] point tracks, and its magnitude is then thresholded and dilated. A target that moves too little within the window may be missed, and an unrelated object that exceeds the threshold may be selected. Sparse future-frame supervision shapes the features used by the instruction-conditioned action branch. The mask itself selects regions by motion.

The current experiments include a stationary-target case and a clutter task. In the pot-and-lid task of Fig. 3, both policies complete the first three stages; DreamIF then picks up the stationary lid and closes the pot, while VLA-Adapter stops at the previous stage. For Task 3 in Table 2, with each method evaluated 20 times, DreamIF reaches 65% and the reactive baseline 50%. For Task 5, which places 3–5 distractors around the pot, DreamIF and VLA-Adapter both reach 80% (Tables 2 and 11). The lid example shows one successful subsequent interaction, and the clutter task does not isolate distractor motion.

3. Action conditioning, cross-layer aggregation, and predictive supervision (Weakness 3, Question 3)

The action head uses an intention token constructed from the current quantised latent and proprioception. The current-image encoding and the projected ALFA aggregate are summed and quantised into z_q,t. The prediction head outputs K future latent grids, and the training decoder reconstructs the corresponding masked targets. The action branch pools the current z_q,t, combines it with the projected proprioception, and obtains H′. The action state x^(n) queries a key/value context formed by concatenating the backbone action tokens, H′, and the layer-n visual tokens. The prediction objective shapes the representation used by the action branch, and the decoded future pixels are used in the training loss.

The two ablations answer different questions. On in-distribution LIBERO-Plus[4] Long, the full model reaches 85.5%, removing dynamic feedback gives 79.5%, full-frame supervision gives 79.2%, and our matched VLA-Adapter[5] baseline reaches 76.6% (Table 9). The 6.0 points compare the entire conditioning path, and the 6.3 points compare the two targets. Table 10 switches the layer input under the same budget; with the LIBERO Long setting noted in the caption, the first, middle, and last layers and all 24 layers give 42.6%, 73.2%, 77.5%, and 85.7%, respectively, which tests the choice of layers.

The revised version does not yet report a control that keeps the same action architecture while disabling the predictive objective. To isolate the contribution of predictive supervision, one should fix ALFA, the projection, the quantiser, the intention token and residual attention, and the training budget, and set only the auxiliary intention-loss weight to zero. The existing ablations directly measure the effect of the feedback path and of the layer input.

4. The 250-step protocol of Figure 7 and its interpretation (Weakness 4, Question 4)

The policy is queried once per recorded frame, and only the first action of the eight-step chunk is taken each time. At time t, both models receive the third-person image, the wrist image, and the proprioception of that frame from the expert demonstration. At t+1, the recorded input of the next demonstrated frame is used, and the model's predicted state is not used to generate the next observation. The predicted and expert increments are integrated from the same initial state, and the error is computed over six joints. The policy is therefore queried 250 times in total, and 250 single-step actions are retained.

Table 12 distinguishes per-step error from integrated trajectory deviation. Results for the same paired Long episode are as follows:

StepsIncrement MSE, full / w/o feedback (×10⁻³)Integrated MSE at segment end, full / w/o feedback
1–504.44 / 14.374.32 / 11.69
51–1007.80 / 14.6610.78 / 32.34
101–1502.71 / 10.7511.41 / 68.25
151–2003.02 / 14.2913.53 / 126.67
201–2504.38 / 9.7121.28 / 131.65
1–2504.47 / 12.76n/a

The full-segment mean increment MSE without feedback is about 2.9× that of the full model, and the integrated deviation at step 250 is about 6.2×; these correspond to different metrics. On this episode, the per-step error is lower overall, and the absolute difference for Joint 5 is small. The evidence supports consistency with the expert trajectory under the same recorded inputs.

5. Choice of τ and K and the scope of a single run (Question 5)

The defaults are based on the current Long sensitivity sweep. Fig. 4(b,c) and Table 6 sweep τ ∈ {0.1, 0.5, 1.0, 5.0, 8.0} and K ∈ {1, 2, 4, 8}. At τ = 0.1, 1.0, and 8.0, the success rates are 79.2%, 85.5%, and 76.4%, and at K = 1, 4, and 8 they are 72.4%, 85.5%, and 84.8%. The defaults are τ = 1.0 and K = 4, at the peak between the two ends; the figure reports the sensitivity sweep on the Long suite.

Each row of Tables 9 and 10 comes from one training run, and the current differences are single-run comparisons under the matched setting.


We hope the responses above answer your questions, and we welcome further discussion on any remaining points.

Finally, we note that W1–W4 and Q1–Q5 concern the completeness of the comparison settings and of the evidence as presented, not the validity of the method itself. The relevant evidence is given in Sections 1–5 and in Tables 8A, 9–10 and 12, and the remaining points can be clarified in revision; we therefore regard them as scope issues that clarification can resolve.

Citations

[1] Black et al. π₀.₅: a Vision-Language-Action Model with Open-World Generalization. CoRL 2025.

[2] Zhong et al. ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Models. CVPR 2026, Table 2.

[3] Karaev et al. CoTracker: It Is Better to Track Together. ECCV 2024.

[4] Fei et al. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv:2510.13626, 2025.

[5] Wang et al. VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model. AAAI 2026.

Reviewer_Fg3hOpenReview ↗

Thanks for the response

I thank the authors for the clarifications and the matched π₀.₅ baseline, which substantially addresses W1/Q1. My main remaining concerns are W3/Q3 and W4/Q4.

For W3/Q3: The response clarifies that action decoding uses the current quantised latent, shaped by the predictive objective, rather than directly consuming predicted future states. However, it also explicitly acknowledges that the same-architecture control without predictive supervision has not been conducted. Removing dynamic feedback removes the conditioning pathway itself, while replacing masked targets with full-frame targets retains predictive supervision. Neither comparison establishes whether future prediction is necessary for the observed gains. The missing control should retain ALFA, the projection, quantiser, intention token, and residual attention under the same training budget, while setting only the auxiliary intention-loss weight to zero. Without this comparison, the contribution of predictive supervision remains unclear with the additional representation and conditioning capacity.

For W4/Q4: I think the response resolves the procedural ambiguity: the policy is queried 250 times using each recorded expert frame and proprioceptive state, retaining only the first action of each eight-step chunk. However, this clarification changes what the experiment can establish. Since the next input comes from the demonstration regardless of the model’s previous prediction, the policy never encounters the state deviations induced by its own actions. Integrating these predictions can show accumulated discrepancy from the expert trajectory, but cannot demonstrate recovery from execution errors. The reported reductions in per-step and integrated error therefore do not resolve the original concern about autonomous error correction. That claim requires evaluation under policy-induced or controlled execution deviations, or should be narrowed to consistency with expert actions under recorded inputs.

We thank you for reading our response again, and for noting that the π₀.₅ baseline added under matched training settings has gone a long way toward addressing W1/Q1. Your specific suggestion for the ablation control, and your distinction between agreement with the expert trajectory and recovery from execution deviations in Figure 7, have both helped us state the experimental results more clearly. Below we further address these two points in light of the two analyses in the revised manuscript.

1. W3/Q3 — The role of predictive supervision under a fixed action architecture

We thank you for proposing this specific control. Disabling the auxiliary intention loss while holding the action architecture and the training budget fixed would measure the contribution of the prediction objective itself to task success, and this is the piece we hope to add in future work.

Table 9 already compares two prediction targets: the two variants retain the same feature aggregation and action conditioning pathway and both receive future-frame supervision; they differ only in whether the model reconstructs the full frame or the motion-masked regions. With motion-masked targets, LIBERO-Plus Long rises from 79.2% to 85.5%. With the conditioning pathway held fixed, the choice of prediction target still affects action performance, and the difference is largest on Long. A second ablation retains the prediction objective and examines the effect of passing the prediction-supervised representation to the action decoder: enabling dynamic feedback raises Long from 79.5% to 85.5%.

Together, these two results show that the target content and the conditioning pathway each contribute, and they also provide a direct basis for the control you propose. We will add this comparison in future work: under the same training budget, keep ALFA, the projection module, the quantizer, the intention token and the residual attention, set only the weight of the auxiliary intention loss to zero, and report it in the same table as the two existing ablations.

2. W4/Q4 — Trajectory agreement under matched evaluation conditions

We thank you for pointing out that the source of the subsequent observations matters for the interpretation. When designing Figure 7, our aim was to keep the evaluation conditions as consistent as possible, so that we could compare how closely the predicted actions, with and without dynamic feedback, match the expert demonstration. Accordingly, on the same LIBERO-Plus Long demonstration, both policies receive the same third-person image, wrist image and proprioceptive information frame by frame across the 250 calls; each call retains only the first action of the eight-step chunk, and the retained actions are then integrated from the same initial joint state and compared against the same expert trajectory.

In this paired comparison, the full model has lower action-increment MSE in all five 50-step windows: over the full segment it falls from 12.76 × 10⁻³ to 4.47 × 10⁻³, and the integrated joint MSE at step 250 drops from 131.65 to 21.28 (Table 12). As you point out, the subsequent observation at every step comes from the demonstration, so we read the lower errors as evidence that, under these identical observations, both the step-by-step actions and the integrated trajectory of the full model are closer to the expert. The autonomous-recovery question you raise requires observing how the policy responds to state deviations caused by its own actions, and we will address it in future work.

Reviewer_muFgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DreamIF, a VLA framework that jointly learns action prediction and multi-frame future dynamic-region prediction. It aggregates features from multiple VLM layers and uses cross-attention to incorporate these features into action generation. The method is evaluated on LIBERO-PLUS and eight real-world manipulation tasks.

Critical strengths
  1. The paper is clearly written, and the appendix provides additional experimental and implementation details.

  2. The evaluation includes both simulation and real-world experiments, with ablations on the dynamic-region target, feature fusion, and prediction horizon.

Critical weaknesses
  1. The novelty relative to prior work is unclear. DreamVLA already includes dynamic-region prediction, while Video Prediction Policy uses multi-frame future representations to guide action generation. The paper should clarify its specific advances over these designs and provide direct comparisons or controlled ablations.

  2. DreamIF with the π0.5 backbone achieves 87.3% OOD success rate, compared with 86.6% for ACoT-VLA. Given the differences in evaluation pipelines acknowledged in Appendix D.3, do the authors mean the comparison is unfair and in favor of ACoT-VLA? If so, please explain.

Questions for authors
  1. Since the dynamic-region mask is constructed by thresholding motion magnitude, could it also include moving distractors? Have the authors evaluated cases where task-irrelevant objects are moving? Or authors can write it as limitation.

  2. Could the authors quantitatively evaluate the predicted future images, beyond the visualizations? For example, could they report reconstruction quality within dynamic regions and the accuracy of the predicted region locations?

  3. In Sec. 5.3, the authors state that “intention anticipation constrains the end-effector trajectory and mitigates the execution jitter.” However, Figure 7 reports joint-position errors relative to expert trajectories, which do not directly measure jitter. Could the authors provide a direct measure of trajectory smoothness, such as acceleration or jerk, to support this claim?

  4. Which camera view does the future prediction head predict: the base view, the wrist view, or both? If both, how are the predictions and losses organized?

  5. Do DreamIF and all baselines use the same camera views in both simulation and real-world experiments?

2 comments
AuthorsOpenReview ↗

Response to Reviewer muFg, brief summary: we separate DreamIF from motion-region supervision and video-prediction baselines; we note that current tasks contain only static distractors while a moving one above τ can enter the mask, and we report the trajectory smoothness of Figure 7 together with the two-view and camera setup.

Response to Reviewer muFg

We thank you for recognizing the presentation of the paper and the coverage of the evaluation. Below we respond item by item with locations: the method comparison (§4.1–4.3 and Table 9 of Appendix E), the OOD protocol and π₀.₅ comparison (§5.2 and Appendix D.3), the scope of moving distractors (§4.1 and limitations), the measurement of prediction quality (§4.1 and Fig. 5(b,c)), two-view prediction and loss reduction (§4.2 and Appendices B/C), and the camera setup (Appendix F).

1. Distinction from DreamVLA and VPP (Weakness 1)

DreamIF combines sparse multi-step prediction, learnable cross-layer aggregation, and residual action conditioning within a single VLA policy. ALFA learns softmax weights over the VLM features of each layer. The current-image encoding and the projection of the fused features form a quantised latent, and the temporal prediction head outputs K future latent grids in parallel. During training, the decoder fits the masked target M_(t+k) ⊙ I_(t+k) with a whole-image pixel loss. The action branch applies global average pooling to the current quantised latent, combines it with proprioception to produce an intention token, and the action state queries a context containing that token, the backbone action tokens, and the visual tokens.

Distinction from related work. DreamVLA[1] also uses motion-region supervision, jointly predicting dynamic, depth, and semantic information through dedicated queries and structured attention. DreamIF focuses on pixel-level sparse targets at multiple future times and connects the ALFA-conditioned latent to the action branch through the intention token. Video Prediction Policy (VPP)[2] extracts predictive representations from a video diffusion model adapted to manipulation tasks and then conditions the action policy on them. DreamIF instead learns the predictive representation inside the VLA policy with a masked multi-frame reconstruction objective.

2. ACoT-VLA and the π₀.₅ comparison (Weakness 2)

The matched π₀.₅[5] comparison is an in-distribution setting, and the OOD numbers serve as a cross-source reference. DreamIF+π₀.₅ in Table 1 of this paper is 87.3%, and the 86.6% of ACoT-VLA comes from the zero-shot transfer block of Zhong et al., Table 2[6]. Both results nominally train on LIBERO[7] and test on LIBERO-Plus. Our protocol evaluates the four suites with one checkpoint, covering 10,030 instances, giving 87.27% overall and 87.26% after applying the benchmark's validity filter, a difference of 0.7 percentage points from the quoted 86.6%.

The in-distribution pair directly measures the effect of adding DreamIF. Both groups in Table 1 and Table 8A train and test on LIBERO-Plus, using the same data, optimisation schedule, action-chunk length, frozen vision towers with LoRA settings, and evaluation pipeline. After adding DreamIF, the reported average success rate rises from 87.6% to 90.7% (+3.1) and Long from 82.4% to 90.0% (+7.6). The zero-shot result of the original π₀.₅ quoted in OOD is 85.7%.

3. Moving distractors and the measurement of prediction quality (Questions 1–2)

A moving distractor above the displacement threshold can enter the mask. The mask is constructed from the first and last frames of a four-frame window: CoTracker[8] samples 784 query points on each 224×224 view with stride 8, and the displacement field is pooled, resized, thresholded, and dilated with 3×3 max pooling to obtain the mask. This is offline motion supervision, so targets below the threshold may be missed.

The tasks in the current evaluation are static scenes, and there are no moving distractors yet. The four suites of LIBERO-PLUS Long and the eight real-robot tasks are all static tabletop environments: distractors are placed on the tabletop and do not move themselves, and the 3–5 distractors that Task 5 places around the pot are likewise static, with DreamIF and VLA-Adapter both at 80% and VLA-JEPA at 85% (Tables 2, 11). Such objects have zero displacement and therefore never enter IADVR; the case that would be included is a distractor that moves itself with a displacement above τ, whose boundary is uniquely determined by the criterion. How the same criterion trades off region coverage is given by the threshold sweep: at τ=0.1 noise is taken as the intention region and the success rate is 79.2%, at τ=1.0 the complete trajectories of the end effector and the interaction object are retained and it reaches 85.5%, and at τ=8.0 key motion is clipped away and it drops to 76.4% (Fig. 4(b)). Prediction quality is measured by the training objective itself: the decoder fits the masked future frame, pixels outside the region are zeroed, so the masked pixel loss is computed only inside IADVR; visualisations of region predictions are given in Fig. 5(b,c).

4. Trajectory consistency and smoothness (Question 3)

Figure 7 and Table 12 measure trajectory consistency and execution smoothness on the same Long demonstration. Over 250 steps both models receive the third-person image, wrist image, and proprioception of the same demonstration at each step, and each takes the first action of its eight-step chunk. For the full model and the model without dynamic feedback, the single-step joint-increment MSE over the whole segment is 4.47×10⁻³ vs. 12.76×10⁻³, and over steps 201–250 it is 4.38×10⁻³ vs. 9.71×10⁻³; the MSE of the integrated deviation is 10.15 vs. 60.04, and the mean absolute deviation at the end of step 250 is 3.06 vs. 8.95. The gap comes mainly from the first three joints (single-step MSE 7.11 / 9.04 / 9.78 vs. 25.07 / 24.51 / 24.82); the last three joints stay within 1.1×10⁻³ (0.26 / 0.46 / 0.17 vs. 0.53 / 1.05 / 0.55).

Smoothness is quantified on the same batch of integrated joint trajectories. With the first difference of joint increments as acceleration and the second as jerk, we compute the acceleration sign-reversal rate and the jerk RMS, placing the expert trajectory under the same convention: the acceleration sign-reversal rate is 0.300 for the full model, 0.435 without dynamic feedback, and 0.224 for the expert trajectory, i.e. the acceleration of the full model reverses back and forth less and is closer to the expert. The jerk RMS is 56.4 / 36.4 / 25.5 (×10⁻³, full / w/o feedback / expert); this quantity grows with motion amplitude and sampling discretisation (joint-velocity RMS 222 / 193 / 244×10⁻³), so we take the amplitude-insensitive reversal rate as the main jitter indicator and report jerk alongside. Both runs share the episode, initial condition, and expert trajectory, so the differences reflect dynamic feedback.

5. Two-view prediction, loss reduction, and baseline cameras (Questions 4–5)

Both the fixed base and the wrist view predict the future K frames in parallel. Each prediction is aligned with the masked target of its view and future time; we average over the K frames within a view, then across the two views' reconstruction losses. This keeps the reconstruction weight from growing linearly with the number of views. The main configuration is K=4 with 512 codebook vectors of 64 dimensions, and the intention-loss coefficient is fixed at 0.005 (Tables 4, 6). The pixel decoder is used during training.

The camera inputs are consistent in the matched comparison. In simulation we use the same observation convention, evaluation instances, and success criterion as the matched VLA-Adapter. The four LIBERO-Plus suites contain 2,519, 2,591, 2,518, and 2,402 instances, each run once. On the real robot, DreamIF, VLA-Adapter, and VLA-JEPA all use fixed and wrist-mounted RealSense D415 cameras, with both streams resized to 224×224. The fixed camera sits 60 cm in front of the workspace at a 45° pitch. All methods share the demonstrations; each task is evaluated 20 times and counts as success only if the instructed goal is completed within 60 s without collision (Table 2). Mean success rates over the eight tasks are 80.6% (DreamIF), 65.0% (VLA-Adapter), and 71.3% (VLA-JEPA).


We hope the responses above answer your questions, and we welcome further discussion on any remaining points.

Citations

[1] Zhang et al. DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge. NeurIPS 2025.

[2] Hu et al. Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. ICML 2025.

[3] Fei et al. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv:2510.13626, 2025.

[4] Wang et al. VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model. AAAI 2026.

[5] Black et al. π₀.₅: a Vision-Language-Action Model with Open-World Generalization. CoRL 2025.

[6] Zhong et al. ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Models. CVPR 2026, Table 2.

[7] Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS 2023.

[8] Karaev et al. CoTracker: It Is Better to Track Together. ECCV 2024.

Reviewer_LR3uOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DreamIF proposes intention-guided sparse dynamics prediction for long-horizon VLA manipulation. The stated method combines motion-derived visual regions, adaptive layer-wise feature aggregation (ALFA), simultaneous prediction of multiple future frames, and dynamic feedback into the action representation. It reports simulation, real-robot, and pi0.5-extension results. The central assessment issue is whether the supplied implementation and evaluation evidence substantiate these mechanisms and their reported benefits.

Critical strengths
  1. Concentrating predictive supervision on interaction-relevant regions is a reasonable motivation for reducing irrelevant visual information. Combining spatial and semantic features with action feedback could be useful for long-horizon control, provided that the proposed mechanisms are implemented and evaluated as described.
Critical weaknesses
  1. The released training path does not implement several central mechanisms as described. Equations (6)-(7) specify future-frame targets, with K=4 by default. However, temporal sampling is configured for actions, while reconstruction uses the current observation images and their motion-masked versions. The multiple images are camera views, not future time steps. A motion-derived mask does not turn a current-frame reconstruction target into a future frame. Consequently, the supplied path does not substantiate multi-frame future prediction or identify the implementation underlying the K ablation.
  2. The same problem affects ALFA and dynamic feedback. The actual path uses final-layer VLM features rather than the learnable aggregation across layers in Eq. (5). Feedback uses learned, input-independent query tokens; the attention outputs are pooled into one residual vector and broadcast across all action positions. This is an attention mechanism, but not the observation/action-conditioned residual attention in Eqs. (8)-(9). Moreover, detaching the VLM features before reconstruction blocks that loss from directly shaping the shared VLM through this path. This does not mean the entire model lacks gradients, but it changes the proposed learning mechanism.
  3. Training configurations also differ materially: action L1 versus flow-matching MSE; an intention weight initially 0.5 with decay versus a fixed reconstruction weight of 0.001; LoRA rank 32 versus 16; a frozen versus trainable vision encoder; and weight decay 0.1 versus 1e-10. Some differences could be legitimate pi0.5 adaptations or release-version differences, but no clear mapping connects the released configuration to the main model, extension, and ablation rows. These observations establish a manuscript-release mismatch, not how the authors' unpublished experiments were actually run.
  4. Figure 7 and Appendix G specify a 250-step rollout using only the initial observation, without subsequent observations or replanning. The supplied evaluator instead repeatedly reads later demonstration images and states. With its default parameters, the single-step, chunk-step, and semi-closed-loop modes make 250, 50, and 5 inference calls, respectively, over 250 steps. None implements the stated protocol. Without the actual figure-generating evaluator, checkpoints, and original trajectories, this script cannot validate the claimed long-horizon effect of dynamic feedback. Crucially, there is no established link showing that this script generated Figure 7; the unresolved issue is result provenance, not demonstrated misuse of this script.
Questions for authors
  1. Which exact code revisions, configurations, and checkpoints generated the main model and the K/ALFA/feedback ablations? Please identify future-image sampling, target shapes, layer aggregation, query construction, and gradient paths, and distinguish these from the released pi0.5-style configuration. For the +15.0-point extension comparison, what gain remains with matched backbone freezing and training budgets, given the protocol difference already disclosed in Appendix D.3?
  2. Please provide the evaluator and original trajectories that generated Figure 7, including the actual observation-input times, the 250-step actions and joint positions, and the complete/w/o-feedback checkpoints. Can the initial-observation-only, no-replanning condition be verified from those records?
5 comments
AuthorsOpenReview ↗

Response to Reviewer LR3u, brief summary: the supervision target is the IADVR masked frame at future time steps; we give the K sweep and the layer-aggregation results, confirm that the main implementation applies no detach so the reconstruction loss does shape the shared representation, provide the matched π₀.₅ comparison, and release the 250-step per-joint data.

We thank Reviewer LR3u for recognizing the motivation of this work and for checking the training code, configurations, and evaluation scripts. We state the main model and the π₀.₅ extension as two implementation paths. Locations: targets and sampling windows in §4.2 and Appendix B/C; layer features and the gradient path in §4.2–4.3 and Appendix C; per-item configurations and result rows in Appendix C/D.3; the 250-step rollout in Figure 7 and Appendix G.


1. Future-frame supervision and the implementation of the K ablation (Weakness 1)

Conclusion: the supervision target of the main model is the IADVR masked frame M_{t+k}⊙I_{t+k} at the future K steps (k = 1,…,K); K counts future time steps, not camera views.

The target tensor is built along an explicit future time axis. Preprocessing pairs frames with a sliding window of length L_win = 4 and keeps its first (t = 0) and last (t = 3) frames; 784 query points are set on the 224×224 plane with stride 8, the displacement field ΔF = T₃ − T₀ comes from Co-Tracker, and 2×2 average pooling to 14×14, bilinear projection back to 224×224, thresholding by τ and 3×3 dilation give the mask M_{t+k} of each future step. The predictor emits K future latents in parallel from K learnable time queries in one latent space; after decoding they are aligned pixel-wise with M_{t+k}⊙I_{t+k}, and the loss averages over the K steps. "Multi-frame" is thus the temporal dimension and "two cameras" an independent one; both the horizon K and the frame offset are configuration items, defaulting to K = 4 and k = 1,…,K.

The K ablation on the same implementation shows an inverted U that first rises and then falls. We sweep K ∈ {1, 2, 4, 8} on the main model: K = 1 captures only instantaneous state transitions (72.4%); K = 4 learns a sufficiently long temporal dependency before errors accumulate, peaking at 85.5%; at K = 8 lets long-range error dominate, falling back to 84.8% (Fig. 4(c)). This matches the expectation that multi-frame prediction is necessary but an over-long horizon compounds error, and shows Fig. 4(c) compares the main model, not the extension.

Prediction horizon K14 (default)8
LIBERO-PLUS Long success rate72.485.584.8

The released configuration relates to the main model explicitly. Setting the horizon and frame offset to k = 1,…,K gives the main model's future-frame supervision, which applies no detach, and the reconstruction loss flows back through the projection layer. The two paths differ in these settings, not in architecture.


2. ALFA, dynamic feedback, and the gradient path (Weakness 2)

Conclusion: the original DreamIF aggregates all N = 24 layers with learnable softmax weights and injects the resulting intention features into the action head through a projection and residual cross-attention; the main implementation applies no detach, so the reconstruction loss does flow back through the projection layer and shape the shared representation.

Layer aggregation is not equivalent to the last-layer feature. Fixing every other setting and using a single layer as input to the intention branch, Long gives first 42.6%, middle 73.2%, last 77.5%, and 85.7% for all 24 layers. The "final-layer VLM feature" setting is the 77.5% row, 8.2 points below the full model. Shallow layers keep the spatial configuration of the end effector and objects but lack task semantics; deep layers carry the instruction but lose geometric detail; learnable weighting keeps both. This is the layer-aggregation ablation in Appendix E.

Layer input of the intention branchFirstMiddleLastAll 24 layers (ours)
LIBERO-PLUS Long success rate42.673.277.585.7

Model construction The intention token is H′ = MLP([MLP(P_t); GAP(z_q,t)]), i.e. one vector from proprioception and the pooled features of the quantised latent shaped by the prediction target; in the n-th action block the key/value context is c⁽ⁿ⁾ = [h_a⁽ⁿ⁾; H′; z_t⁽ⁿ⁾], the query is the action state x⁽ⁿ⁾, and a residual FFN updates it. The query is thus observation/action-conditioned and H′ participates in the attention of all action positions as a conditioning term, not an "input-independent query token plus a pooled broadcast".

Gradient path: the main implementation keeps the reconstruction loss shaping the shared representation, and detach belongs only to the extension. The total loss is L_action + λL_intent, where L_intent sums the masked pixel loss and the VQ commitment term; since z_e,t = E(I_t) + P(H_fusion), the reconstruction loss returns through the trainable projection P to the cross-layer aggregated representation and acts as a fixed regulariser on it. The detach in the π₀.₅ extension only stops that extension's reconstruction loss from updating the upstream VLM directly.

3. Training configuration and the matched π₀.₅ comparison (Weakness 3, Question 1)

Conclusion: the main model keeps one complete configuration; the extension comparison applies DreamIF's multi-frame IADVR prediction and intention injection to π₀.₅.

Configuration. Vision towers and VLM body are frozen and LoRA-adapted; projection layers, language-model head, and prediction head are trained in full. Action loss L₁; intention weight a constant λ = 5×10⁻³ (no decay schedule); LoRA rank 32, weight decay 0.1, chunk length 8, AdamW with 500 warm-up steps and cosine decay 10⁻⁴→10⁻⁶. λ stays small because the two losses differ in units (L₁ on normalised actions vs a K-frame pixel loss), so the weighted auxiliary term is a fixed regulariser; hence no "decay from 0.5".

ItemOursReleased π₀.₅
Action objectiveL₁flow-matching MSE
Intention weightλ = 5×10⁻³ constant0.001 fixed
LoRA rank3216
Vision encoderfrozen (LoRA)frozen
Weight decay0.11e-10
Chunk length88

The extension comparison uses one recipe. π₀.₅ without the DreamIF mechanism and DreamIF+π₀.₅ share corpus, schedule, chunk length, frozen vision towers with LoRA, and evaluation pipeline; only multi-frame future-region prediction and intention injection differ. Four-suite in-distribution results:

Method (four suites)AverageSpatialObjectGoalLong
π₀.₅ (same recipe, no DreamIF)87.692.589.685.882.4
DreamIF+π₀.₅90.793.893.186.190.0

Gains concentrate on the long-horizon suite (+7.6), as expected where intention prediction mitigates error accumulation.


4. Evaluation protocol and data of Figure 7 (Weakness 4, Question 2)

Conclusion: each Long-episode frame t queries the policy once with that frame's third-person image, wrist image, and proprioception, keeping the first action of the eight-step chunk; the 250 actions are integrated into a joint trajectory and compared joint by joint with the expert integral.

Pairing: both variants share one condition. Each episode uses one initial state and one expert trajectory with joint-by-joint errors, so differences cannot be attributed to episodes. Eight steps is the chunk length, 250 the evaluation.

The metric measures trajectory fidelity Per-step error stays in a narrow band while integrated deviation keeps growing, so the gap comes from integrating a persistent bias, not from predictor degradation: last-segment per-step error 4.38×10⁻³ vs 9.71×10⁻³, integrated deviation 21.28 vs 131.65 (joint-mean squared deviation at segment end), mean increment per 50 steps 4.2 vs 30.0, and lower per-step error on all six joints.

StepsPer-step oursPer-step w/o DFEnd dev. oursEnd dev. w/o DF
1–504.4414.374.3211.69
51–1007.8014.6610.7832.34
101–1502.7110.7511.4168.25
151–2003.0214.2913.53126.67
201–2504.389.7121.28131.65
1–2504.4712.76n/an/a

All six joints agree. Same episode (Long, ep. 0), same 250-step protocol, 10⁻³ MSE:

JointJ1J2J3J4J5J6Mean
DreamIF7.119.049.780.260.460.174.47
DreamIF w/o DF25.0724.5124.820.531.050.5512.76

Per-step error is lower on all six joints by factors of 2.0–3.5; J4–J6 are small in absolute terms (≤1.05×10⁻³). After integration the mean joint deviation is 10.15 vs 60.04 and the final mean absolute deviation 3.06 vs 8.95, so our trajectory stays closer to the expert throughout.

Data and checkpoints. The 250-step actions, per-joint positions, and error curves are per-episode pred_joint_increment/gt_joint_increment files (one per model: DreamIF_libero_long_ep0_250step.json, DreamIF_wo_DF_libero_long_ep0_250step.json; 6 joints, gripper removed, model field). Checkpoints follow the code release.

Finally, these points concern implementation details and evidence scope (W1–W4, Q1–Q2), addressed above and in accompanying material; they do not amount to a failure of the method, and we do not regard them as grounds for rejection.


Citations

[1] VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model. AAAI 2026.

[2] π₀.₅: A Vision-Language-Action Model with Open-World Generalization. CoRL 2025.

[3] ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Models. CVPR 2026.

[4] CoTracker: It Is Better to Track Together. ECCV 2024.

[5] DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge. NeurIPS 2025.

[6] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. ICML 2025.

Thank you for the clarifications and the matched π0.5 comparison. My concern about Figure 7 remains: the clarified protocol uses demonstration observations at every step, so it does not substantiate the originally claimed initial-observation-only execution. I also share Fg3h’s concern that, without an architecture-matched control disabling only the predictive objective, the benefit of predictive supervision remains confounded with the additional conditioning pathway. While I appreciate the authors’ rebuttal efforts, these evidential gaps remain substantial, and my final rating is 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.