acceptodds
Under review as a conference paper at ICLR 2027

VLA-Δ: Learning Physical State Transition for Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models transfer semantic knowledge acquired through vision-language pretraining to robot control, achieving strong performance across various benchmarks. However, most VLAs directly map observations to demonstrated actions without explicit supervision of intermediate physical state changes, making them prone to overfitting observation–action correspondences. Some methods introduce intermediate reasoning or future-state prediction in language, visual, or latent spaces to guide action generation, but do not explicitly supervise the physical state transition itself. We therefore propose VLA-Δ, a general framework that uses physical changes between current and future states as explicit learning targets. To support this objective, we construct a large-scale pretraining dataset and derive physical state-transition annotations covering task-relevant semantics and visual motion changes, providing more direct supervision for continuous action generation. During inference, the model generates actions directly from current observations without additional inference overhead. VLA-Δ achieves performance competitive with the state of the art on LIBERO and RoboTwin, and outperforms in real-robot experiments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Manipulation, Multimodal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YuV2OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

VLA-Δ proposes to improve vision-language-action policies through explicit supervision of task-relevant state changes. Learnable transition queries are jointly trained with structured state/transition slots, dense optical flow, future-state difference text, and action prediction. The authors curate and annotate approximately 484 hours of demonstrations from DROID, MolmoAct, and RH20T, initializing the VLM from ZR-0base. Downstream fine-tuning uses only action supervision, and auxiliary outputs are not decoded at deployment. Evaluation covers LIBERO, LIBERO-Plus, RoboTwin 2.0, and five real-robot tasks. The paper reports 98.4% on LIBERO, 90.06%/89.74% on RoboTwin Clean/Random, and 68.0% average real-robot success.

Critical strengths
  1. The paper includes a meaningful action-only pretraining control. Table 3 reports an improvement from 96.45% with query-based action-only pretraining to 98.40% with the full supervision package. This is stronger evidence than comparison with unrelated architectures alone, and supports a benefit beyond simply adding pretraining data.
  2. The design is practically motivated and evaluated beyond a single benchmark. Auxiliary supervision is used during pretraining without requiring intermediate text or flow decoding at deployment. Results across simulation and real-robot tasks provide evidence that the overall training recipe has practical value.
Critical weaknesses
  1. The methodological novelty beyond existing predictive-supervision approaches is unclear.

The paper follows an established approach: learn intermediate query representations through auxiliary prediction objectives and use these representations to condition action generation. Relative to approaches such as DreamVLA, the identifiable changes are primarily the selection of supervision targets—structured state/transition slots, difference text, and optical flow—and their organization within the query interface. However, the paper does not clearly establish why these changes constitute a substantive methodological advance. In particular, the distinction between “physical state-transition learning” and future-state representation learning is asserted more strongly than it is demonstrated. The proposed combination may be useful, but its specific contribution beyond an alternative auxiliary-supervision recipe remains insufficiently established.

  1. The ablations do not isolate the claimed advantages of the proposed targets or their integration.

Table 3 shows that adding each supervision type to the query-based action-only baseline improves performance, and that the full combination performs best. However, these experiments do not establish whether all three targets are necessary, since the corresponding leave-one-out combinations are missing. More importantly, they do not distinguish the benefit of transition-specific supervision from current-state or future-feature supervision, nor compare alternative integration mechanisms while holding the supervision fixed. The comparison with DreamVLA does not resolve this issue: the models use different backbone scales, initializations, and training recipes, so the performance gap cannot be attributed to the proposed target design. Consequently, the experiments support the effectiveness of the overall recipe, but provide limited evidence for the particular methodological claims used to motivate it.

Questions for authors
  1. Contribution clarification: What is the precise methodological advance over future-aware query representations trained with auxiliary prediction objectives? Which aspect of the proposed transition formulation cannot be adequately captured by existing future-state or motion-supervised approaches?
  2. Evidence for the claimed contribution: Can the authors provide controlled comparisons using the same backbone, initialization, data, and training budget to distinguish the proposed transition targets from current-state/future-feature targets? Please also clarify whether leave-one-out target ablations or comparisons of alternative integration mechanisms are available.
7 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer YuV2 (W1 & Q1)

[W1 & Q1 - Contributions and Relation to Predictive Supervision]

Thank you for the feedback. We clarify our contribution through continued pretraining, supervision design, and experimental evidence.

1. Methodological contribution: continued pretraining with task-relevant physical changes.

Our core contribution lies in continued pretraining, where we explicitly train VLAs to learn the object motion, contact transitions, and task progress associated with actions and transfer this knowledge to downstream policy learning. Downstream fine-tuning uses only action supervision and still benefits from the change supervision provided during pretraining.

To the best of our knowledge, we are the first to achieve this goal in VLA continued pretraining through joint supervision using difference text, structured physical variables, and visual motion.

2. Future-state and motion supervision: explicitly supervising task-relevant changes.

(1) Physical transitions versus future-state representations.

DreamVLA’s approach to learning action-relevant representations through predictive supervision, as mentioned in your review, has inspired our research. We have cited and discussed this work in the paper. DreamVLA provides future information for action generation by reconstructing future dynamic regions and predicting depth and semantic features. VLA-Δ further uses difference text and structured state-transition fields to explicitly supervise changes from current to future states and their task-related meaning, together with optical flow for dense motion supervision.

(2) What can future-state or motion representations capture?

We acknowledge that future-state or motion representations can, in principle, encode this information. We emphasize the difference in supervision requirements. Endpoint feature matching or motion prediction does not directly specify which task-relevant state changes the model must predict. We explicitly formulate these changes as text and structured fields and include them in the optimization objective. Thus, our claim is that for a given dataset and training budget, directly supervising task-relevant changes can more effectively support downstream action learning.

(3) Controlling confounds in the comparison with DreamVLA.

We report an overall benchmark comparison with DreamVLA.

MethodStandard LIBEROLIBERO-Plus
DreamVLA92.60 (reported in the original paper)51.98 (evaluated in this work)
VLA-Δ (ZR-0)98.476.62

To control for the effects of differences in backbone, initialization, and training procedure, we have added the “DreamVLA supervision targets (VLA-Δ architecture)” control in W2 & Q2, which uses DreamVLA’s auxiliary prediction targets to supervise Query representations within the common architecture.

This variant using DreamVLA’s targets achieves an average LIBERO-Plus success rate of 69.73%, while VLA-Δ achieves 76.62%, an improvement of 6.89 percentage points. These controlled results provide more direct evidence of the benefits of explicit change supervision.

3. Continued pretraining gains: controlled ablations and cross-backbone validation.

(1) Gains beyond additional action training.

Table 3 in the original submission compares no continued pretraining, action-only continued pretraining, and VLA-Δ under the same Query architecture and downstream fine-tuning protocol.

Continued Pretraining SettingDownstream LIBERO Average Success Rate (%)
No continued pretraining96.25
Continued pretraining with action supervision only96.45
Continued pretraining with VLA-Δ98.4

Continued pretraining with action supervision alone improves the success rate by 0.20 percentage points. Using the same continued pretraining data, VLA-Δ achieves a further gain of 1.95 percentage points over the action-only control.

This result shows that explicit change supervision enables the same additional demonstration data to produce greater downstream gains and improves task performance after standard fine-tuning with action supervision.

(2) Applicability across backbones.

We further apply this training scheme to two base policies, ZR-0 and π0.5\pi_{0.5}. The table below reports average success rates. Values in parentheses indicate improvements over the corresponding base policy in percentage points.

Evaluation SettingZR-0VLA-Δ (ZR-0)π0.5\pi_{0.5}VLA-Δ (π0.5\pi_{0.5})
Standard LIBERO (%)97.898.40 (+0.60)96.998.60 (+1.70)
LIBERO-Plus (%)74.3376.62 (+2.29)85.4187.45 (+2.04)

VLA-Δ improves average success on both benchmarks for both backbones, supporting its applicability. See Appendix C.1 for LIBERO-Plus details.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer YuV2 (W2 & Q2)

[W2 & Q2 - Controlled Comparisons of Supervision Targets and Ablations of Auxiliary Objectives]

Thank you for the suggestions. We have added three control conditions using current-state supervision, future-state/feature supervision, and DreamVLA supervision targets within the same architecture, along with three leave-one-out variants.

1. Controlled comparisons: separating target design from model differences.

You noted that the benefits of transition supervision over current-state/future-feature supervision have not been distinguished and that the comparison with DreamVLA is affected by differences in backbone, initialization, and training recipe. To address these concerns, we compare different auxiliary supervision combinations under a common policy architecture and training procedure.

All conditions use the same backbone, initialization before continued pretraining, source demonstration data, and optimization update budget. We also keep the number of Queries, action interface, downstream fine-tuning protocol, and evaluation protocol identical.

Supervision SettingAuxiliary Supervision Targets during Continued PretrainingLIBERO-Plus Average Success Rate (%)
Current-stateCurrent-state descriptions, current-state fields, and frozen visual features E(ot)E(o_t) of the current image73.13
Future-stateFuture-state descriptions, future-state fields, and frozen visual features E(ot+k)E(o_{t+k}) of the future image74.89
DreamVLA supervision targets (VLA-Δ architecture)Reconstruction of future dynamic regions, prediction of future depth, and prediction of future semantic features69.73
VLA-ΔDifference text and structured state-transition fields describing changes from current to future states, together with optical flow76.62

(1) Distinguishing transition supervision from current-state/future-feature supervision.

Compared with current-state supervision and future-state/feature supervision, VLA-Δ improves the average LIBERO-Plus success rate by 3.49 and 1.73 percentage points, respectively. This controlled comparison adds the previously missing evaluation of supervision targets under the same architecture and shows that under the tested architecture and training conditions, explicitly modeling task-relevant changes from current to future states can yield higher downstream task success rates than current-state or future-state/feature supervision.

(2) Controlling for differences in architecture, initialization, and training recipe in the comparison with DreamVLA.

We agree that the performance gap between the two complete models in the original submission cannot, by itself, establish the contribution of the supervision target design. We have therefore added the “DreamVLA supervision targets (VLA-Δ architecture)” control, which uses DreamVLA’s auxiliary prediction targets to supervise Query representations within the common VLA-Δ policy architecture and training procedure.

This variant using DreamVLA’s targets achieves an average success rate of 69.73%, while VLA-Δ achieves 76.62%, an improvement of 6.89 percentage points. This result shows that, with the backbone, initialization, data, and optimization update budget held fixed, the proposed supervision combination still achieves a higher downstream success rate.

2. Leave-one-out ablations: conditional contributions of individual objectives.

To address your concern about the missing experiments that remove individual objectives from their combinations, we have added three leave-one-out variants that individually remove difference text, structured slots, or optical flow supervision from the full model. For each leave-one-out variant, we remove only the corresponding auxiliary training objective and keep the remaining loss weights unchanged.

Pretraining VariantLIBERO Average Success Rate (%)Full Model − Variant (Percentage Points)
Action only96.451.95
Action + Difference Text97.21.2
Action + Structured Slots96.91.5
Action + Optical Flow97.351.05
Action + Structured Slots + Optical Flow (without Difference Text)97.70.7
Action + Difference Text + Optical Flow (without Structured Slots)97.50.9
Action + Difference Text + Structured Slots (without Optical Flow)980.4
Full model98.40

The additional experiments show that individually removing difference text, structured slots, and optical flow from the full model reduces the success rate by 0.70, 0.90, and 0.40 percentage points, respectively. This indicates that all three supervision targets make positive contributions to downstream performance. Even when the other two types of supervision are already present, each provides additional gains, and using all three together achieves the best performance.

AuthorsOpenReview ↗

Follow-Up — Seeking Further Feedback (Discussion Deadline Approaching)

Dear Reviewer YuV2,

Thank you for your careful review of our work and your constructive suggestions. In response to your concerns, we have provided additional experimental results and analyses and further clarified our methodological contributions and experimental settings. We would like to follow up to ask whether these responses have addressed your main concerns and whether any points require further clarification.

We would appreciate it if you could review our responses and share any further feedback before the discussion period ends. If any concerns remain, we would be happy to provide further explanations, continue the discussion, and improve our manuscript accordingly.

Thank you again for the time and effort you have devoted to reviewing and discussing our work. We look forward to hearing from you.

Best regards,

The Authors

Reviewer_Kr8WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VLA-Δ, a vision-language-action framework that explicitly supervises physical state transitions during pretraining. The central idea is to learn representations of task-relevant physical changes, rather than relying solely on direct observation-to-action learning or future-state reconstruction.

The framework introduces Physical State Difference Queries and three complementary supervision signals: Structured Transition Slots, Optical Flow, and Future-State Difference Text. The authors also construct a large-scale robot demonstration corpus with automatically generated transition annotations. These supervision signals are used during pretraining, while downstream fine-tuning and inference rely on action generation without decoding the auxiliary targets.

Experiments on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-robot manipulation tasks demonstrate competitive performance against existing VLA approaches. Ablation experiments further investigate the contributions of the proposed supervision components.

Critical strengths
  1. The paper tries to address a bottleneck of existing VLA models: action supervision alone may not sufficiently encourage policies to capture task-relevant physical changes. The proposed framework provides a meaningful alternative by explicitly supervising state transitions through structured variables, visual motion, and semantic descriptions.

  2. Comprehensive experimental effort is appreciated. The paper evaluates the proposed method across several simulation benchmarks and real-robot manipulation tasks. The results are consistently competitive, with improvements over strong baselines on LIBERO and RoboTwin.

  3. The ablation study is supportive, as it includes action-only pretraining controls, helping distinguish improvements from additional pretraining data versus the proposed auxiliary supervision. The annotation pipeline and quality-control procedures also provide supporting evidence for the feasibility of the approach.

Critical weaknesses
  1. Limited direct quantitative evidence of the learned physical transition representations. The central contribution of the paper is learning physically meaningful state-transition representations. However, the experimental evidence primarily focuses on downstream task success rates, t-SNE visualizations, and qualitative attention maps. Although these results support the usefulness of the proposed supervision, they do not directly establish how accurately the learned representations encode future physical changes, such as object displacement, contact transitions, or task progress.

  2. Robust improvement needs further clarification and additional experimental validation. For instance, while VLA-Δ achieves the highest average success rates among the compared methods, the margins over the strongest baselines are relatively small in some settings. For example, the improvement over ZR-0 is 0.6 percentage points on LIBERO and 1.36/1.76 percentage points on RoboTwin Clean/Random.

Questions for authors

Besides the weakness above, the reviewer has the following questions.

  1. Table 3 demonstrates improvements from introducing Future-State Difference Text, Structured Transition Slots, and Optical Flow individually. Could the authors measure the contributional gain of each element?

  2. Could the authors further clarify the distinction between VLA-Δ and existing approaches that explicitly model changes in world knowledge or physical states, particularly prior-guided world-knowledge variation methods? (novelty related question)

  3. Reporting inference latency and computational overhead relative to the corresponding backbone would better substantiate the efficiency claim.

The reviewer is open to modifying the final rating according to the rebuttal effort for addressing the raised weakness and questions.

8 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer Kr8W (W1)

Thank you for recognizing the motivation for explicit physical state-transition supervision, the breadth of our evaluation, and the importance of the action-only pretraining control. Your comments highlight several aspects that require more direct evidence and a clearer positioning of our work. Below, we address representation quality, robustness, the conditional contributions of individual components, related work, and inference efficiency.

[W1 - Quantitative Evidence for the Learned Physical State-Transition Representations]

Thank you for raising this issue. We have added a slot readout evaluation with frozen policy parameters to directly quantify the decodability of physical state-transition information. We complement this evaluation with real-robot target-displacement experiments that provide behavioral evidence of the policy’s responses to physical state changes during execution.

1. We conducted probing experiments to quantitatively assess the decodability of physical state-transition information.

We compare the model checkpoints listed below. For each checkpoint, we freeze all policy parameters and train a newly initialized set of slot readout heads with the same architecture, comprising nine prediction branches. All checkpoints use the same readout data split, prediction-head architecture, and training budget.

Query features are extracted using only the current images and task instruction. Future observations and their derived annotations are used to construct supervision targets. We report results for the three targets you highlighted, namely object displacement, contact transitions, and changes in task progress.

ModelCheckpoint StageObject Displacement MAE (cm) ↓Contact Transition Macro-F1 (%) ↑Progress Change MAE ↓
ZR-0 + QueriesAfter continued pretraining15.994231.1890.06552
VLA-ΔAfter continued pretraining6.8182443.6070.03808
VLA-ΔAfter LIBERO fine-tuning8.9648739.9840.04559

After continued pretraining with physical state-transition supervision, VLA-Δ reduces object displacement MAE by 57.37%, increases contact-transition Macro-F1 by 39.82%, and reduces task progress change MAE by 41.88%, compared with the baseline model at the same training stage. These results show that, after pretraining with change labels, VLA-Δ’s Query representations allow more accurate readout of physical state-transition information.

Subsequently, after LIBERO fine-tuning using only action supervision, without further physical state-transition labels, readout performance declines somewhat, but the final model still outperforms the baseline on all three metrics. This indicates that the ability to encode physical state-transition information acquired during continued pretraining is partially retained after downstream training with action supervision alone.

2. Complementing representation analysis with real-robot execution results.

Under the same target-displacement protocol, VLA-Δ and ZR0 were each evaluated over 40 valid intervention episodes. VLA-Δ achieves a first-grasp success rate after target displacement of 95.0% and a final task success rate of 70.0%, exceeding ZR0’s 75.0% and 52.5%, respectively. Existing real-robot videos also show the policy re-approaching the target and attempting another grasp after a grasp failure. Further details are provided in Appendix C.3 of the revised manuscript.

These execution results provide behavioral evidence of the policy’s responses to physical state changes.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer Kr8W (W2 & Q1)

[W2 - Magnitude of Improvements and Robustness under Environmental Changes]

Thank you for your comments. We address this concern in three parts, covering the focus of our method, ablation results, and validation across different backbones.

1. Our contribution is a continued pretraining method, rather than a downstream post-training method.

We initialize our models from existing pretrained VLAs and introduce structured slot, optical flow, and difference text supervision during continued pretraining to learn task-relevant physical state changes. We then perform standard downstream fine-tuning (post-training) using the benchmarks’ existing action demonstrations. Our focus is on how continued pretraining can improve representations and thereby enhance task performance after standard post-training.

2. Physical state-transition supervision enables continued pretraining to yield greater gains after post-training.

Table 3 in the original submission compares three settings under the same Query architecture. All variants use the same downstream fine-tuning protocol with action supervision.

Continued Pretraining SettingLIBERO Average Success Rate after Post-Training
No continued pretraining96.25%
Continued pretraining with action supervision only96.45%
Continued pretraining with VLA-Δ98.40%

Continued pretraining with action supervision alone improves the final success rate by 0.20 percentage points. Using the same pretraining data, VLA-Δ achieves 98.40%, exceeding action-only continued pretraining by 1.95 percentage points.

This comparison indicates that translating additional pretraining into downstream gains requires effective supervision design. Explicit physical state-transition supervision enables greater downstream gains from continued pretraining and improves the performance attainable through standard post-training.

3. Average performance improves across different backbones and multiple benchmarks.

We have expanded the LIBERO-Plus evaluation from four to seven perturbation categories and added experiments using the π0.5\pi_{0.5} backbone. On both ZR-0 and π0.5\pi_{0.5}, VLA-Δ improves the average success rates on standard LIBERO and LIBERO-Plus.

BenchmarkZR-0VLA-Δ (ZR-0 Backbone)π0.5\pi_{0.5}VLA-Δ (π0.5\pi_{0.5} Backbone)
Standard LIBERO97.80%98.40% (+0.60)96.90%98.60% (+1.70)
LIBERO-Plus average across seven categories74.33%76.62% (+2.29)85.41%87.45% (+2.04)

On RoboTwin 2.0, VLA-Δ achieves 90.06% under the Clean setting and 89.74% under the Random setting, improving over ZR-0 by 1.36 and 1.76 percentage points, respectively. These results support the effectiveness of the continued pretraining method across different backbones, standard tasks, and perturbed environments.

[Q1 - Conditional Contributions of the Auxiliary Objectives]

Thank you for suggesting that we further quantify the conditional contribution of each auxiliary objective within the full model. We have therefore added the three missing combinations of two auxiliary objectives, with the results reported below.

Pretraining VariantLIBERO Average Success Rate (%)Full Model − Variant (Percentage Points)
Action only96.451.95
Action + Difference Text97.21.2
Action + Structured Transition Slots96.91.5
Action + Optical Flow97.351.05
Action + Structured Transition Slots + Optical Flow (without Difference Text)97.70.7
Action + Difference Text + Optical Flow (without Structured Transition Slots)97.50.9
Action + Difference Text + Structured Transition Slots (without Optical Flow)980.4
Full model98.40

The additional leave-one-out experiments further show that removing difference text, Structured Transition Slots, and optical flow from the full model individually reduces the success rate by 0.70, 0.90, and 0.40 percentage points, respectively. Therefore, under the present experimental settings, each objective exhibits a positive conditional contribution when the other two objectives are already present. These gains are obtained without introducing additional transition annotations downstream and while maintaining the same action-supervised fine-tuning protocol, and therefore measure the benefit of auxiliary supervision during continued pretraining for subsequent policy learning.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer Kr8W (Q2)

[Q2 - Methodological Novelty and Differences from Related Work]

Thank you for the suggestion. We clarify the novelty of our work in terms of its central idea, methodology, and transfer through pretraining, and further explain its specific differences from related work.

1. Conceptual contribution — Learning physical changes to increase the value of continued pretraining for downstream action learning.

Our core contribution lies in continued pretraining, where we explicitly train VLAs to learn the object motion, contact transitions, and task progress associated with actions and transfer this knowledge to downstream policy learning. Downstream fine-tuning uses only action supervision and still benefits from the change supervision provided during pretraining.

To the best of our knowledge, we are the first to achieve this goal in VLA continued pretraining through joint supervision using difference text, structured physical variables, and visual motion.

2. Methodological contribution — Implementing this joint supervision through difference text, structured slots, and optical flow.

Regarding the approaches you mentioned that “explicitly model changes in world knowledge or physical states,” our main distinctions lie in how change targets are defined and how they supervise action representations.

  • Formulating key physical changes in manipulation as quantifiable prediction targets. ECoT / ERVLA support action learning through information such as plans, subtasks, motion descriptions, and localization. ERVLA also includes descriptions of future motion and gripper changes. We further directly supervise object displacement, contact transitions, and changes in task progress, so that Query prediction responsibilities correspond to explicit physical variables. We assess how accurately this information is encoded through readout evaluations using frozen representations.

  • Directly constraining the representations used for action generation with the semantics of changes. DreamVLA predicts dynamic regions, depth, and semantic features, while StaKe supervises manipulation stages and keyframe targets for the next gripper event. We use difference text to explicitly describe how changes relate to task goals and operational constraints. Through the VLM’s native autoregressive language modeling capability, we provide a direct training pathway for connecting task semantics, physical changes, and continuous control.

  • Directly constructing change targets with explicit physical meaning, without learning an additional discrete change codebook. Regarding your specific interest in “prior-guided world knowledge variation modeling,” we compare with ΔVLA, which most closely matches this description. This work is conceptually close to ours and provides useful inspiration. It extracts world knowledge from priors such as regions, semantics, and depth, learns discrete change codes through conditional reconstruction with a VQ-VAE, and organizes attention by knowledge type. We directly specify change targets through difference text and structured slots, assign specific physical prediction responsibilities to Query groups, retain information exchange across groups, and use optical flow to provide additional dense motion supervision.

3. Pretraining contribution — Physical transition supervision yields greater downstream gains from the same pretraining data.

Under the same Query architecture and downstream fine-tuning protocol, Table 3 compares no continued pretraining, action-only continued pretraining, and full VLA-Δ. Their LIBERO success rates are 96.25%, 96.45%, and 98.40%, respectively. Using the same continued pretraining data, VLA-Δ outperforms action-only continued pretraining by 1.95 percentage points. This result shows that physical transition supervision enables greater downstream gains from the same pretraining data.

Therefore, compared with existing world knowledge prediction and prior-guided change modeling approaches, VLA-Δ’s novelty lies in transition targets with explicit physical semantics, the joint constraints imposed by linguistic and physical supervision on action representations, and the value of transferring these representations learned through continued pretraining to downstream learning with action supervision alone.

Reviewer_sjGLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a vision-language-action policy learning framework with additional transition, visual motion, and task semantic supervision. The representation is extracted from the current observation to achieve efficient inference. The dataset is constructed by the structured slot, visual optical flow, and task analysis between the current and future frame. The action expert adapts a Mix-of-Transformer structure, and the vision-language-model backbone predict the free form text for transition description. Query tokens are inserted into the input sequence, and are sent to decoder to decode the future megaflow and structured flow.

The proposed method is trained on multiple datasets including DROID, MolmoAct, and RH20T. The authors report 98.4% average success on LIBERO, 90.06%/89.74% on RoboTwin 2.0 Clean/Random, and 68.0% average success on five real-robot tasks, which surpasses the performance of Pi-0.5. Ablations on LIBERO show the gain from the auxiliary supervisions.

Critical strengths
  1. Reasonable design for auxiliary tasks The combination of structured slots, optical flow, and difference text covers complementary aspects of manipulation. This design covers both high and low level for manipulation tasks, and demonstrate performance gain over downstream evaluation.
  2. Novel free-form language description for transition. The traditional spatial description in embodied chain-of-thought usually translates the action chunk into motion text with a template, and therefore lacks semantic information. The authors propose the natural language transition difference description could alleviate this phenomena.
Critical weaknesses
  1. The precision of the transition labels is not yet clear. Table 5 audits 300 pairs and uses human references for most fields, which is useful. However, the validity criteria have no clear error tolerance. The paper reports no box or displacement errors, contact F1, or agreement between annotators. The two bounding-box references are generated by GPT-5.6 (pp. 18–19). Since these labels are central to the method, the missing error analysis makes it hard to assess how robust the learned transitions are to annotation noise.
  2. The simulation results are weak. The simulation results on both Libero and RoboTwin are not competitive with the current SOTA baselines, and make me doubt the effectiveness of the proposed method.
Questions for authors
  1. The amount of the involved dataset is 650 hours, while the utilized dataset is 484 hours. What exact data were used for pretraining after filtering? And what is the cretirea of data filtering? If we want to scaling the dataset size, could the automatic pipeline satisfy the quality demand?
  2. What are the loss weights, optical-flow weight, number of Queries, optimizer and schedule, training steps, and seeds? Appendix B.3 says the model horizon sets the future endpoint. What is its numerical value, and do slots, text, flow, and actions use aligned intervals?
  3. Can the authors report independent annotation quality metrics, such as box/flow errors, contact-classification F1, and human agreement, rather than only valid-label rates?
  4. Can the authors test recovery after a failed grasp or after an object is moved, to directly test whether transition supervision improves physical-state adaptation?
4 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer sjGL (W1 & Q3)

[W1 & Q3 - Independent Quantitative Evaluation of Annotation Quality]

Thank you for pointing out the limitations of our annotation quality evaluation and suggesting more direct quantitative evidence. We agree that the annotation validity statistics reported in Table 5 of the original submission cannot substitute for quantitative metrics such as localization errors and classification performance. Furthermore, the reference bounding boxes generated by GPT-5.6 cannot be treated as independent human ground truth. We have therefore added an independent accuracy evaluation of the training annotations and an analysis of inter-annotator agreement before adjudication.

1. Independent quantitative evaluation. We compare automatically generated annotations against independent references using 300 current–future image pairs. The sample sizes, numbers of scored units, and results for each evaluation are reported below.

Annotation TargetSample Size$N$Scored Units nnEvaluation MetricResult [95% CI]
Target-object boxes600 boxes582 boxesMean IoU ↑0.931 [0.922, 0.938]
Gripper boxes600 boxes582 boxesMean IoU ↑0.944 [0.941, 0.947]
Contact transitions300 image pairs291 image pairsMacro-F1 (%) ↑95.2 [92.5, 97.6]
Gripper state transitions300 image pairs291 image pairsMacro-F1 (%) ↑96.9 [94.8, 98.7]
Interactable positions600 positions540 positionsPoint-in-admissible-region rate (%) ↑90.2 [87.4, 92.4]
Gripper 3D displacement*300 image pairs270 image pairsMean 3D displacement vector error (cm) ↓1.45 [1.31, 1.59]
Optical flow (all selected points)300 image pairs1,080 pointsMean sparse endpoint error (EPE, px) ↓2.16 [1.99, 2.35]
Optical flow (moving-point subset)300 image pairs540 pointsMean sparse endpoint error (EPE, px) ↓3.72 [3.41, 4.05]

2. Inter-annotator agreement. To assess the reproducibility of the human references, we compute agreement metrics on samples that both annotators could assess before adjudication. We report the mean absolute difference in task progress estimates, agreement rates and Cohen’s κ\kappa for discrete categories, and mean inter-annotator IoU for bounding boxes.

Annotation TargetUnits Assessable by Both AnnotatorsAgreement MetricResult
Task progress582 endpoint valuesMean absolute difference (percentage points) ↓5.4 pp
Task phase582 endpoint labelsAgreement (%) / Cohen’s κ\kappa ↑97.4% / 0.968
Contact transitions291 image pairsAgreement (%) / Cohen’s κ\kappa ↑97.3% / 0.963
Gripper state transitions291 image pairsAgreement (%) / Cohen’s κ\kappa ↑98.3% / 0.977
Target-object boxes582 boxesMean inter-annotator IoU ↑0.928
Gripper boxes582 boxesMean inter-annotator IoU ↑0.943

We have added detailed information about these evaluations to Appendix B.4 of the revised manuscript.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer sjGL (Q1 & Q2)

[Q1 - Training Data, Filtering Criteria, and Scalability]

Thank you for pointing out the ambiguity in our description of the dataset size. The two figures in the original submission refer to different quantities. 650 hours refers to the source data before filtering, while 484 hours refers to the filtered data pool available for Stage 3 training.

1. Data sources and retained dataset size. We use data from DROID, MolmoAct, and RH20T. After filtering, we retain 24,059,815 frames, corresponding to approximately 484 hours based on the sampling rate of each source. We have clarified the distinction between the source data before filtering and the retained data after filtering in Appendix B.1 of the revised manuscript. The reasons for exclusion and the corresponding quantities are reported below.

Data SourceReason for ExclusionExcluded FramesEquivalent Duration (h)
DROIDNo nonempty, reliable task instruction available7,807,335144.58
RH20TTrajectory marked as failed348,5019.681
RH20TMissing valid success-status annotation36,3911.011
RH20TTrajectory flagged for numerical anomalies3,7500.104
RH20TFailed the two-camera validity or synchronization requirements after passing the preceding checks377,58110.488
Total8,573,558165.864

2. Scalability and quality control. The automated pipeline performs keyframe pairing, semantic annotation, and visual annotation in batches. A unified interface, source traceability, and checkpoint-based recovery reduce manual effort per sample and help control annotation costs. In the independent evaluation, target-object and gripper boxes achieve mean IoU values of 0.931 and 0.944, respectively, while contact transitions and gripper state transitions achieve Macro-F1 scores of 95.2% and 96.9%, respectively. These results provide quantitative support for the annotation accuracy of the evaluated fields. As we expand the dataset, we maintain annotation quality through stratified spot checks, error feedback, and targeted reannotation.

[Q2 - Training Configuration and Temporal Alignment of Supervision]

Thank you for suggesting that we provide further training details. We have added the loss weights, Query configuration, optimization settings, and temporal alignment of supervision to Appendix D.1–D.3 of the revised manuscript.

Three-stage training setup. As shown in Appendix D.2, Table 17, training consists of three stages.

  • Stage 1 — Text supervision. We train for 10,000 steps with the loss LText\mathcal{L}_{\mathrm{Text}}.
  • Stage 2 — Auxiliary supervision. We train for 5,000 steps with the loss Lslot+10Lflow\mathcal{L}_{\mathrm{slot}} + 10\mathcal{L}_{\mathrm{flow}}.
  • Stage 3 — Joint training. We train for 90,000 steps with the loss LText+0.1Lslot+Lflow+5Lact\mathcal{L}_{\mathrm{Text}} + 0.1\mathcal{L}_{\mathrm{slot}} + \mathcal{L}_{\mathrm{flow}} + 5\mathcal{L}_{\mathrm{act}}.

Optimization settings. All stages use AdamW with β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95, and ϵ=10−6\epsilon = 10^{-6}. The peak learning rates for the three stages are 10−510^{-5}, 2×10−52 \times 10^{-5}, and 10−510^{-5}, respectively. We use 8% learning-rate warmup, followed by cosine decay. The random seed is 42. The full configuration is provided in Tables 17–18.

Query configuration. The model uses 32 Difference Queries. The allocation of Slot Queries into seven groups and the internal loss weights are provided in Table 13 and Table 16, respectively, in Appendix D.1.

Prediction horizon and temporal alignment of supervision. The action branch predicts 50 action steps. During offline annotation, Qwen3.8 adaptively determines the auxiliary supervision interval based on task events. Shorter intervals are used for operations such as fine alignment, contact, and insertion, while longer intervals are used for operations such as long-distance movement. Once the start and end frames tst_s and tft_f are selected, text, structured slots, and optical flow are all generated from the same frame pair. Further details are provided in Appendix D.3.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer sjGL (W2 & Q4)

[W2 - Weak Simulation Results]

Thank you for your comments. We would like to further clarify our contribution and demonstrate the effectiveness of our method through ablation studies and experiments across different backbones.

1. Our contribution is a continued pretraining method, rather than a post-training method.

We initialize our models from existing pretrained VLAs and introduce structured slot, optical flow, and difference text supervision during continued pretraining to learn physical state changes. We then perform downstream fine-tuning with standard action supervision to evaluate whether these pretrained representations improve task performance after post-training.

2. Continued pretraining with action supervision alone yields limited gains, while our method further improves post-training performance.

The challenge of continued pretraining is to translate additional training into downstream gains. Table 3 compares the following three settings under the same Query architecture. All variants undergo downstream fine-tuning with action supervision.

Pretraining SettingLIBERO Average Success Rate after Post-Training
No continued pretraining96.25%
Continued pretraining with action supervision only96.45%
Continued pretraining with VLA-Δ98.40%

Continued pretraining with action supervision alone improves the success rate by 0.20 percentage points. Using the same pretraining data, our method achieves 98.40%, exceeding action-only continued pretraining by 1.95 percentage points. This result shows that explicitly supervising physical state changes enables greater downstream gains from continued pretraining and improves the performance attainable through standard post-training.

3. Consistent gains across different backbone models.

We have added experiments using the π0.5\pi_{0.5} backbone. On both ZR-0 and π0.5\pi_{0.5}, VLA-Δ improves the average success rates on standard LIBERO and LIBERO-Plus.

BenchmarkZR-0VLA-Δ (based on ZR-0)π0.5\pi_{0.5}VLA-Δ (based on π0.5\pi_{0.5})
LIBERO97.80%98.40% (+0.60)96.90%98.60% (+1.70)
LIBERO-Plus74.33%76.62% (+2.29)85.41%87.45% (+2.04)

Additionally, on RoboTwin 2.0, VLA-Δ achieves 90.06% under the Clean setting and 89.74% under the Random setting, improving over ZR-0 by 1.36 and 1.76 percentage points, respectively. These results support the effectiveness of the continued pretraining method across different backbones, standard tasks, and perturbed environments.

[Q4 - Recovery after Grasp Failure and Object Displacement]

Thank you for suggesting a direct evaluation of how the policy responds to physical state changes during execution. We address this through grasp-failure cases from our existing experiments and additional target-displacement experiments. We have also added the corresponding details to Appendix C.3 of the revised manuscript.

1. Autonomous retries after grasp failure. In our existing real-robot experiments on Long-Horizon Execution and Fine-Grained Manipulation, we observed that the model re-approached the target and attempted another grasp after a grasp failure. Figure 9 shows a sequence of consecutive frames from each task.

2. Quantitative evaluation of recovery after target displacement. Using the existing real-robot task Put the spoon into the cup, we conducted additional experiments comparing VLA-Δ and ZR0. While the robot was autonomously approaching the spoon, before making contact, we moved the spoon to another reachable position. After the operator withdrew their hand, the model continued executing the task using updated observations.

We report two metrics, first-grasp success rate after target displacement and final task success rate.

ModelValid Intervention EpisodesFirst-Grasp Success Rate after Target DisplacementFinal Task Success RateTask Success Rate without Perturbation
ZR04030/40 (75.0%)21/40 (52.5%)67.50%
VLA-Δ4038/40 (95.0%)28/40 (70.0%)72.50%

Under the same target-displacement protocol, VLA-Δ exceeds ZR0 by 20.0 percentage points in first-grasp success rate and 17.5 percentage points in final task success rate. The first-grasp results show that VLA-Δ more frequently achieves a successful grasp on the first attempt after the target position changes. The final task results further show that this recovery performance extends to the subsequent placement stage.

AuthorsOpenReview ↗

Follow-Up — Seeking Further Feedback (Discussion Deadline Approaching)

Dear Reviewer sjGL,

Thank you for your careful review of our work and your constructive suggestions. In response to your concerns, we have provided additional experimental results and analyses and further clarified our methodological contributions and experimental settings. We would like to follow up to ask whether these responses have addressed your main concerns and whether any points require further clarification.

We would appreciate it if you could review our responses and share any further feedback before the discussion period ends. If any concerns remain, we would be happy to provide further explanations, continue the discussion, and improve our manuscript accordingly.

Thank you again for the time and effort you have devoted to reviewing and discussing our work. We look forward to hearing from you.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.