acceptodds
Under review as a conference paper at ICLR 2027

ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning and predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents. However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions. We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene. Specifically, ViDAL learns an action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics through a predictor conditioned on the current visual state. When integrated into downstream robot policies, the proposed Action VAE serves as a plug-in action interface compatible with multiple VLA architectures and enables optional future-video prediction as an additional capability. Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task π0.5 policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Action Representation Learning, Visual Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cFNaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ViDAL learns a continuous action VAE with temporal reconstruction, frequency reconstruction, KL regularization, and future-visual-latent prediction. A downstream VLA predicts the learned action latent and executes it through a frozen decoder; the visual predictor is discarded at deployment. The paper evaluates this interface with regression and flow-matching policies, two simulation benchmarks, and two real-robot platforms.

Critical strengths
  1. The modular action interface is useful and does not require online video generation. Its application to both regression and flow-matching policies supports a contribution beyond one architecture. The sizable RoboTwin improvements and coverage of two physical robot platforms make the practical case stronger than the small LIBERO differences alone would.
  2. Table 3 contains direct component ablations: the vanilla VAE reaches 57.4%/38.3%, removing dynamics yields 62.2%/40.1%, removing frequency reconstruction yields 63.1%/40.4%, and the full model reaches 65.5%/43.1%. Thus, dynamics supervision contributes 3.3/3.0 points within the full configuration, rather than explaining the entire gain over raw actions. Figures 3-4 additionally vary latent design, visual targets, supervision objectives, and policy output branches.
Critical weaknesses
  1. The novelty and meaning of visual grounding require more precise positioning. The already-cited and benchmarked villa-X [1] combines visual latent-action learning with action-related grounding, while the preprint X-Tokenizer [2] includes future visual-feature supervision of action codes. Neither is the same pipeline: ViDAL learns continuous executable latents from action chunks and freezes their decoder. Its broad first-approach statement on p.2 should be restricted accordingly. Moreover, q(z|a) receives actions, not scene context; identical action chunks induce the same posterior across scenes. The representation can be shaped by dynamics, but the scene-specific consequence resides in the predictor conditioned on current vision. The current controls do not fully distinguish correct future dynamics from useful auxiliary regularization.
  2. The strongest raw-action comparison is not yet demonstrably optimization-matched. Appendix B, p.14, specifies an action-expert learning rate ten times the VLM rate for adapting pi0.5 to latent targets, without making equally explicit whether the raw-action baseline and every representation ablation use the same ratio and tuning budget. This matters for attribution of the largest gains. Independent-training uncertainty is also absent from the smaller differences, and ten real-robot trials per task do not establish their training stability. These are material limitations, but the available text does not prove that the baseline was disadvantaged.

[1] Chen, Xiaoyu, et al. "Villa-x: enhancing latent action modeling in vision-language-action models." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Kang, Miracle, et al. "X-tokenizer: A multimodal action tokenizer for vision-language-action pretraining." arXiv preprint arXiv:2606.14752 (2026).

Questions for authors
  1. Please state the initialization, learning-rate ratio, schedule, and tuning budget for the raw-action baseline and every Table 3 variant. Which gains remain under an optimization-matched raw-action comparison?
  2. Can the authors clarify the narrower novelty relative to villa-X and distinguish dynamics-specific supervision from a generic auxiliary target? A matched static-target or shuffled-future diagnostic would be informative, but I do not require a new comparison to X-Tokenizer, for which I verified a preprint but not peer-reviewed publication. The existing component ablations already support a useful, narrower empirical claim.
6 comments
AuthorsOpenReview ↗

Response to Reviewer cFNa (1/3)

W1/Q2: Contribution relative to prior work and the meaning of visual grounding.

Thank you for recognizing the value of our modular action interface and its applicability to both regression and flow-matching action experts. We appreciate your careful distinction between learning an action representation and predicting scene-dependent visual dynamics. Your comments help us state the contribution more precisely.

Relation to villa-X and X-Tokenizer. We agree that the broad first-approach statement should be narrowed. The following comparison clarifies the formulation and downstream role of the representations:

Aspectvilla-X [1]X-Tokenizer [2]ViDAL
Representation source and formLatent actions inferred from visual transitionsDiscrete codes learned from robot actionsContinuous latents learned from robot action chunks
Key representation-learning supervisionVisual prediction and proprioceptive groundingAction reconstruction and multimodal supervision, including next-frame vision-language feature predictionAction reconstruction and future visual-latent prediction conditioned on current vision
Downstream useJoint modeling of latent actions and robot actionsDiscrete-code prediction supervises policy training, while a continuous action expert produces robot actionsAn action expert directly predicts continuous latents, which a frozen decoder maps to control commands

ViDAL studies how conditional future-visual supervision can shape continuous action representations that serve directly as VLA prediction targets and can be decoded into control commands. The action encoder and decoder are frozen after representation learning, providing a common prediction and execution interface for regression and flow-matching action experts. We have narrowed the contribution statement in Section 1 around this formulation and clarified the key distinctions from these related works in Section 2.

Visual grounding with an action-only encoder. As you correctly point out, identical preprocessed action chunks induce the same posterior under qϕ(za∣a)q_\phi(z_a\mid a). During representation learning, the future-visual prediction loss backpropagates through the predictor to the action encoder, encouraging it to retain action information useful for predicting future visual latents. For a particular scene, Pθ(z0,za)P_\theta(z_0,z_a) combines the current visual latent z0z_0 with the action latent zaz_a to predict the future.

Thus, visual grounding refers to future visual supervision shaping the action representation during training, while scene-specific consequences are modeled jointly from the current observation and the action representation. Section 3.2 now clarifies the encoder's action inputs, the predictor's scene conditioning, and the gradient path from the future-prediction loss to the action encoder.

References

[1] X. Chen et al., "villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models," arXiv:2507.23682, 2025.

[2] M. Kang et al., "X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining," arXiv:2606.14752, 2026.

AuthorsOpenReview ↗

Response to Reviewer cFNa (2/3)

W1/Q2: Future visual dynamics versus generic auxiliary supervision.

Thank you for suggesting these targeted controls. We added both static-target and shuffled-future variants to examine whether the content and temporal correspondence of visual supervision matter for downstream policy performance.

Static targets are constructed from the current visual latent. Temporally shuffled future targets randomly reorder the 12 future visual-latent time blocks within each sample, retaining their content. These changes apply to the visual supervision during Action VAE pretraining. All variants retain action reconstruction, frequency-domain reconstruction, and KL regularization, with the other representation-learning and downstream policy-training settings held fixed. The Action VAE remains frozen during downstream policy training.

The table reports success rates (%) averaged over the 50 RoboTwin 2.0 tasks, using the same pi0.5 backbone:

Visual supervision during Action VAE pretrainingCleanRandom
No visual prediction supervision62.240.1
Static visual targets62.339.8
Temporally shuffled future visual targets63.141.0
Correctly ordered future visual targets (ViDAL)65.543.1

Correctly ordered future supervision improves success over static targets by 3.2/3.3 percentage points on Clean/Random, and over shuffled future targets by 2.4/2.1 points. This supports the value of temporally aligned future visual supervision beyond the alternative auxiliary targets tested here.

The shuffled-target variant also performs somewhat better than the static-target variant. A possible explanation is that shuffling retains future visual content from the same sample, which may still encourage the action representation to capture information about action-induced visual changes. Static targets instead describe the current observation, which is already supplied to the predictor. Shuffling nevertheless disrupts the correspondence between actions and future temporal positions, and preserving the correct order gives the strongest downstream results.

These controls and their interpretation are now included in Section 4.4 (Table 4), with target-construction details in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer cFNa (3/3)

W2/Q1: Optimization settings and training stability.

Thank you for asking us to make the optimization settings explicit and to provide evidence across independent training runs. We clarify the settings below and report the additional learning-rate comparisons and real-robot training seeds.

Optimization settings and tuning range. The raw-action baseline and all Table 3 policy variants use the pi0.5 base pretrained checkpoint for initialization. They share a 60,000-step policy-training schedule with 1,000 warmup steps and cosine decay. The VLM peak learning rate is 5×10−55\times10^{-5}, with a decay learning rate of 10−610^{-6}.

The original raw-action baseline uses an action expert/VLM learning-rate ratio of 1. We additionally evaluated ratios of 3 and 10 for this baseline. All Table 3 representation variants, including the vanilla Action VAE, the variants without dynamics or frequency reconstruction, and full ViDAL, use a ratio of 10 without a separate learning-rate search for each variant. Their downstream training settings are identical, and the ablation changes the losses used during Action VAE pretraining.

Raw-action learning-rate comparison. The following table reports success rates (%) averaged over the 50 RoboTwin 2.0 tasks:

MethodAction expert/VLM LR ratioCleanRandom
Raw-action pi0.5154.333.2
Raw-action pi0.5355.733.5
Raw-action pi0.51053.631.9
pi0.5 + ViDAL1065.543.1

Among the evaluated settings, the best raw-action configuration uses a ratio of 3. ViDAL exceeds it by 9.8/9.6 percentage points on Clean/Random. At the shared ratio of 10, the gains are 11.9/11.2 points. Thus, increasing the raw-action expert's learning rate does not reproduce ViDAL's gains under the evaluated settings.

Real-robot results across training seeds. Following your suggestion, we trained two additional pi0.5 + ViDAL models on Franka using seeds 43 and 44, alongside the original seed 42. Both the Action VAE and the pi0.5 policy were trained anew for each seed. Each trained model was evaluated for 10 trials per task across four tasks, giving 120 trials in total. Entries below report successful trials out of total trials:

Training seedMug on PlateStack BowlsBowl in DrawerTwo Mugs in BasketOverall
428/108/106/107/1029/40 (72.5%)
438/109/108/106/1031/40 (77.5%)
447/108/107/106/1028/40 (70.0%)

Across the three independent training seeds, the mean success rate is 73.3%, with a sample standard deviation of 3.8 percentage points, computed over the three overall success rates. The similar performance across these runs supports the stability of ViDAL's full training pipeline on these Franka tasks.

The optimization settings and learning-rate comparisons have been added to Appendix B.

Reviewer_oBFfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a latent-action based method for vision-language-action policy learning. The proposed idea is to leverage an action reconstruction supervision with an additional future visual feature prediction loss for the latent action learning. The full policy adopts an predict-then-decode process which first predicts latent actions then decode it with the frozen latent action decoder for the final action chunk. The full model achieves competitive performance on several existing benchmarks.

Critical strengths
  • The discussion on how latent action facilitates vision-language-action learning is an important problem, especially on how to leverage the visual dynamics information for action representations.

  • The paper is overall clear and easy to follow.

Critical weaknesses
  • The proposed visual dynamics grounding module is in-fact a common practice in latent-action learning where many prior works have leveraged future visual vision feature prediction for latent action representation learning. Though this paper starts from a different point-of-view where actions are tokenized via self-reconstruction, but the overall idea has been tested a lot in previous works.

  • The major concern of latent-action learning via the reconstruction of actions lies in the scaling of the learning. As we gradually scale models with more data (mainly non-teleoped), this learning of action reconstruction gets difficult in such cases with limited robot action annotations. The method or argument is therefore limited to the realm of post-training or task-specific learning. However, this method show limited results on using the learned representation as a post-training technique, the selected benchmarks are also near saturation which makes it hard to tell its actual potential. This makes the technical contribution of this paper somewhat limited.

Questions for authors

See the Weakness section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer oBFf (1/2)

W1: Relation to existing latent-action learning.

Thank you for recognizing the importance of using visual dynamics to improve action representations and for encouraging us to clarify our contribution. We agree that future visual prediction is established in latent-action learning. ViDAL studies how visual-dynamics supervision can shape representations of executable action chunks that serve as prediction targets for downstream VLA policies.

The Action VAE encoder takes a robot action chunk as input and produces a continuous latent, from which the decoder reconstructs control commands. A separate predictor receives the current visual state and the action latent to predict future visual latents. Its loss updates the action encoder during representation learning. The encoder and decoder are then frozen for downstream policy training and execution.

The following comparison clarifies this formulation relative to approaches that infer latent actions from visual transitions [1,2]:

AspectLatent actions inferred from visual transitions [1,2]ViDAL
Representation inputObservation pairs describing a visual transitionRobot action chunks
Representation-learning objectiveLearn implicit action variables that explain observed transitionsReconstruct robot actions while learning latents informative for conditional future visual prediction
Role of visual predictionSupervise inferred latent actions through future-observation predictionShape the action encoder during Action VAE pretraining
Downstream interfaceLatents support policy pretraining or condition action generation, depending on the methodA frozen encoder supplies continuous latent targets for regression or flow-matching action experts, and a frozen decoder maps predicted latents to control commands

In ViDAL, visual supervision acts on the action representation that the policy subsequently learns to predict. Action reconstruction retains a direct mapping to control commands, while conditional future prediction encourages the representation to preserve information useful for predicting action-induced visual changes. This yields a common action interface for regression and flow-matching action experts without requiring a visual predictor at deployment.

We have narrowed the contribution statement in Section 1 to remove the broad first-approach claim. Section 2 now explains the key distinctions from related work, and Section 3.2 clarifies the representation inputs and the role of visual supervision during representation learning.

References

[1] S. Ye et al., "Latent Action Pretraining from Videos," ICLR, 2025.

[2] X. Chen et al., "villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models," arXiv:2507.23682, 2025.

AuthorsOpenReview ↗

Response to Reviewer oBFf (2/2)

W2: Action supervision, scalability, and evidence for post-training.

Thank you for raising the questions of data scalability and practical value in post-training. We agree that action reconstruction requires action sequences, and would like to clarify both their possible sources and the evidence provided by our experiments.

Action supervision and data sources. ViDAL learns from action sequences and synchronized visual observations already available in robot demonstrations. It uses future visual supervision to improve the representation that a downstream VLA learns to predict, without requiring additional action annotations for those demonstrations.

The source of action supervision need not be restricted to robot teleoperation. For example, VideoDex [1] recovers hand and wrist motion from human videos and retargets it to robot trajectories for policy pretraining. This provides a potential route to broaden the data sources for action representation learning.

Post-training on RoboTwin 2.0. Our experiments already evaluate ViDAL in multitask post-training of a pretrained VLA. After representation learning, the Action VAE encoder and decoder remain frozen while the policy is trained to predict the action latents. The original submission reports the following comparison using the same pi0.5 backbone across 50 RoboTwin 2.0 tasks. Success rates are percentages, and gains are percentage-point differences.

Evaluation setting (50 tasks)Raw-action pi0.5pi0.5 + ViDALGain
Clean54.365.5+11.2
Random33.243.1+9.9

These settings retain substantial room for improvement, allowing us to assess the representation's value beyond near-saturated benchmarks. ViDAL improves success on 43 of 50 tasks under Clean and 42 of 50 under Random, showing that the gains extend across a broad range of manipulation tasks. These results support its practical value as an action representation for multitask VLA post-training.

We have clarified the multitask post-training setup in Section 4.1 and the practical value supported by the RoboTwin 2.0 results in Section 4.2.

Reference

[1] K. Shaw, S. Bahl, and D. Pathak, "VideoDex: Learning Dexterity from Internet Videos," CoRL, 2022.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer oBFf,

Thank you for your time and thoughtful feedback on our paper. As the discussion period draws to a close, we would be grateful if you could review our responses and the revisions made to address your concerns. We look forward to your feedback and welcome the opportunity for further discussion.

Thank you again for your time and consideration.

Best regards, The authors

Reviewer_XwfHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViDAL, a visual dynamics-grounded action latent representation for VLA models. It trains an Action VAE using action reconstruction, frequency-domain reconstruction, and future visual latent prediction, encouraging the learned action representation to capture both executable motions and their visual consequences. The trained Action VAE is then frozen and integrated into regression-based and flow-matching VLA policies. Experiments on LIBERO, 50 RoboTwin 2.0 tasks, and two real-robot platforms demonstrate improvements in manipulation success rates.

Critical strengths

Well-motivated and practical design. The paper explores an interesting direction by incorporating future visual dynamics into action representation learning rather than directly adding a video prediction objective to the VLA. The resulting Action VAE provides a modular interface compatible with different action experts.

Strong empirical results with useful ablations. The improvements on RoboTwin 2.0 and real robots are encouraging. In particular, the ablations on action reconstruction, frequency-domain supervision, and visual dynamics help distinguish the contributions of different training objectives.

Critical weaknesses

Insufficient comparison with closely related action representation and visual prediction approaches. Although the paper compares against a vanilla Action VAE and several external VLA methods, it lacks controlled comparisons with alternative action representation methods or downstream VLA training with auxiliary future visual prediction. These comparisons are important for establishing the specific benefit of grounding the action latent during representation pretraining. Moreover, the visual-dynamics objective contributes an additional 3.3/3.0 percentage points on RoboTwin Clean/Random over the reconstruction-plus-frequency baseline, which is smaller than the total improvement over raw-action prediction. A clearer separation of these contributions would strengthen the main claim.

Limited analysis of action-dependent visual grounding and evaluation fairness. The future visual prediction objective does not by itself establish that the learned latent captures the action-dependent changes that matter for manipulation, rather than predictable scene information. The qualitative video examples and latent visualizations provide useful illustrations but limited direct evidence of this mechanism. In addition, the appendix specifies an adjusted action-expert learning rate for π0.5 and an additional 50k-step Action VAE training stage, while the corresponding baseline optimization settings and end-to-end computational costs are not fully clarified. The full RoboTwin results also reveal notable regressions on some tasks despite the overall improvement.

Questions for authors

Can the authors provide controlled comparisons with alternative action latent representations or a VLA trained with auxiliary future visual prediction under the same backbone and data? This would clarify the advantage of incorporating visual dynamics during Action VAE pretraining.

How can the authors verify that the dynamics predictor actually relies on the action latent to model scene changes? For example, comparisons with a state-only predictor or shuffled action latents, together with quantitative evaluation of object motion or contact-related changes, would provide stronger evidence.

Appendix B states that the π0.5 action-expert learning rate is set to 10 times the VLM learning rate. Is the same learning-rate configuration used for the raw-action π0.5 baseline? Can the authors also report the additional training cost of the Action VAE and the inference overhead of latent decoding?

The full RoboTwin results show substantial regressions on some tasks, such as Click Bell and Turn Switch under Random evaluation. Can the authors analyze these cases and explain when the dynamics-grounded representation may be less effective? Also, does the gain at the shorter action horizon (H=16) persist against a raw-action baseline with the same horizon?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XwfH (1/4)

W1/Q1: Visual-dynamics supervision during Action VAE pretraining.

Thank you for suggesting this comparison. To clarify the benefit of visual dynamics during Action VAE pretraining, we compare both the training stage at which visual supervision is applied and the visual target used for representation learning.

All configurations use the same π0.5\pi_{0.5} backbone, demonstrations, action horizon, downstream training steps, and evaluation protocol on RoboTwin 2.0. All Action VAE variants retain action reconstruction, frequency-domain reconstruction, and KL regularization, and remain frozen during downstream VLA training. For the downstream auxiliary baselines, the output features of query tokens processed by the VLM backbone feed a prediction head supervised with future latents from the same frozen Wan VAE.

The table separates visual supervision in the two training stages. "None" means no visual prediction loss at that stage. Static targets are constructed from the current visual latent; shuffled targets randomly reorder the 12 future visual-latent time blocks within each sample, preserving their content. Success rates (%) are averaged over 50 tasks.

Policy action targetVisual target in Action VAE pretrainingAuxiliary visual target in VLA trainingCleanRandom
Raw actionsN/A (no Action VAE)None54.333.2
Raw actionsN/A (no Action VAE)Ordered future latents56.134.7
Action latentsNoneNone62.240.1
Action latentsNoneOrdered future latents63.440.9
Action latentsStatic targetsNone62.339.8
Action latentsTemporally shuffled future latentsNone63.141.0
Action latents (ViDAL)Ordered future latentsNone65.543.1

Supervision during representation pretraining. Downstream auxiliary prediction improves policies predicting either raw actions or action latents. ViDAL nevertheless exceeds the baseline predicting action latents with downstream auxiliary supervision by 2.1/2.2 percentage points on Clean/Random. This supports the benefit of shaping the action representation with future visual supervision during pretraining, compared with applying it only during downstream policy training in these experiments.

Choice of visual supervision. ViDAL exceeds static-target and temporally shuffled-target pretraining by 3.2/3.3 and 2.4/2.1 points, respectively. Shuffled targets retain future visual content and yield some improvement over no visual supervision, but correctly ordered future targets give the strongest results. These comparisons support the value of future visual supervision with its temporal structure intact, beyond the gains obtained from the alternative visual targets tested here.

Regarding attribution, the 11.2/9.9-point gain over raw-action prediction belongs to the complete method. The 3.3/3.0-point gain over the Action VAE without visual supervision is the additional contribution of dynamics supervision on top of action reconstruction, frequency-domain reconstruction, and KL regularization.

These supervision controls are now included in Table 4, with implementation details in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer XwfH (2/4)

W2/Q2: Action-dependent future prediction.

Thank you for suggesting these controls. We evaluate whether action conditioning improves the prediction of scene changes using the same 256 RoboTwin video clips from a sample-level validation split.

Controlled comparisons. The separately trained state-only predictor receives the current visual input with action tokens fixed to zero during both training and evaluation. For inference-time shuffling, we keep the full predictor's weights and current observation unchanged and randomly permute the 12 temporal action-latent tokens within each clip. This changes the action input, while the ground-truth future remains unchanged.

Region-specific evaluation. To assess motion-related visual changes, including object and robot motion during interactions, we separately evaluate prediction errors in dynamic and static regions. We construct masks solely from ground-truth videos and share them across all methods. With RGB values normalized to [0,1], we compute each pixel's channel-averaged absolute change relative to the initial frame and take its maximum over future frames:

  • Dynamic region: pixels with change above 15/255, followed by two dilation iterations.
  • Static region: the complement of pixels with change above 8/255 after three dilation iterations. This is not simply the complement of the dynamic mask.

Dynamic and static MAE measure the mean absolute prediction error over the respective mask, future frames, and RGB channels, averaged across clips. These metrics quantify pixel-level scene changes, including robot and object motion.

ModelAction inputPSNR ↑SSIM ↑Dynamic MAE ↓Static MAE ↓
State-only predictorZero action-latent tokens27.160.94010.027760.00384
Full ViDAL predictorTemporally shuffled action-latent tokens16.750.79970.106570.00502
Full ViDAL predictorCorrectly ordered action-latent tokens31.430.96560.017520.00383

Compared with state-only prediction, the full method improves PSNR by 4.27 dB and reduces dynamic-region MAE by 36.9%, while static-region MAE remains nearly identical (0.00383 vs. 0.00384). Thus, the benefit of action conditioning is concentrated in changing regions. Shuffling the action tokens with the predictor and observation fixed also substantially degrades prediction, demonstrating sensitivity to the ordered action representation. Together, these controls support that the predictor uses action information to model future scene changes.

These diagnostic results and their analysis are now included in Section 4.4 (Table 5), with the evaluation protocol in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer XwfH (3/4)

W2/Q3: Learning-rate settings, training cost, and inference overhead.

Thank you for pointing out the need to clarify these settings and costs.

Learning-rate settings. The original raw-action baseline uses an action expert/VLM learning-rate ratio of 1, whereas ViDAL uses 10. The VLM learning rate is 5×10−55\times10^{-5} in both cases. ViDAL changes the pretrained action expert's prediction target from raw actions to action latents. We use a higher action expert learning rate to facilitate adaptation to this new target space while keeping the VLM learning rate unchanged.

To examine whether increasing the raw-action expert's learning rate can account for the performance difference, we additionally evaluated ratios of 3 and 10. The table reports RoboTwin 2.0 success rates (%):

MethodAction expert/VLM LR ratioAction expert LRCleanRandom
Raw-action pi0.515e-554.333.2
Raw-action pi0.531.5e-455.733.5
Raw-action pi0.5105e-453.631.9
ViDAL105e-465.543.1

Among the evaluated settings, the best raw-action configuration uses a ratio of 3. ViDAL still exceeds it by 9.8/9.6 percentage points on Clean/Random. At the shared ratio of 10, the gains are 11.9/11.2 points. Thus, increasing the raw-action expert's learning rate does not reproduce ViDAL's gains under the evaluated settings.

Training cost. We report visual-latent precomputation using the frozen Wan VAE separately from Action VAE and predictor training:

Training stageHardwareApprox. wall-clock time (h)Approx. GPU-hours
Visual-latent precomputation (frozen Wan VAE)8 H20 GPUs0.252.0
Action VAE + future predictor, 50k steps8 H20 GPUs3.2025.6
Raw-action policy training8 H20 GPUs18.40147.2
ViDAL policy training8 H20 GPUs18.40147.2

The additional representation-learning stage, including precomputation, takes approximately 3.45 hours on 8 H20 GPUs (27.6 GPU-hours), equivalent to about 18.8% of the raw-policy training cost. Downstream policy training takes the same time for both methods. For the same data and action space, this representation-learning cost can be shared across downstream policies that reuse the frozen Action VAE.

Inference overhead. Neither the video encoder nor the future predictor runs at deployment. The additional deployed module is the frozen action decoder. On an L20 GPU, the timings are:

Inference componentLatency (ms)
Raw-action policy96.0
ViDAL policy, excluding action decoding96.0
Action decoder2.5
ViDAL including decoding (sum of components)98.5

Summing the measured policy and decoder times gives approximately 98.5 ms for ViDAL, an additional 2.5 ms (2.6%) relative to the raw-action policy. The visual-prediction computation used to learn the representation is therefore confined to pretraining.

The learning-rate comparison and key training and inference costs have been added to Appendix B.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

We are grateful to all the reviewers for the care and effort they have put into evaluating our work. We appreciate their recognition of ViDAL's practical value, including its modular design, compatibility with different VLA architectures, and encouraging results in simulation and on real robots. Your feedback has helped us improve the presentation and develop a more thorough analysis of the method. We address each comment below with detailed explanations, supplementary experimental results, and further analysis.

Area_Chair_vvVGOpenReview ↗

Please Review the Author Rebuttal and Reassess Your Rating

Dear Reviewers,

Thank you for your time and effort in reviewing the submissions.

The authors have now submitted their rebuttals. I kindly ask you to carefully review the authors' responses and assess whether they have adequately addressed your questions and concerns.

If the rebuttal resolves your concerns, please consider updating your review and adjusting your rating accordingly. If any concerns remain, please feel free to raise them in the discussion.

Your timely feedback and active participation are greatly appreciated, as they are essential to reaching a fair, well-informed decision.

Thank you again for your valuable contributions!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.