- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ViDAL learns a continuous action VAE with temporal reconstruction, frequency reconstruction, KL regularization, and future-visual-latent prediction. A downstream VLA predicts the learned action latent and executes it through a frozen decoder; the visual predictor is discarded at deployment. The paper evaluates this interface with regression and flow-matching policies, two simulation benchmarks, and two real-robot platforms.
Critical strengths
- The modular action interface is useful and does not require online video generation. Its application to both regression and flow-matching policies supports a contribution beyond one architecture. The sizable RoboTwin improvements and coverage of two physical robot platforms make the practical case stronger than the small LIBERO differences alone would.
- Table 3 contains direct component ablations: the vanilla VAE reaches 57.4%/38.3%, removing dynamics yields 62.2%/40.1%, removing frequency reconstruction yields 63.1%/40.4%, and the full model reaches 65.5%/43.1%. Thus, dynamics supervision contributes 3.3/3.0 points within the full configuration, rather than explaining the entire gain over raw actions. Figures 3-4 additionally vary latent design, visual targets, supervision objectives, and policy output branches.
Critical weaknesses
- The novelty and meaning of visual grounding require more precise positioning. The already-cited and benchmarked villa-X [1] combines visual latent-action learning with action-related grounding, while the preprint X-Tokenizer [2] includes future visual-feature supervision of action codes. Neither is the same pipeline: ViDAL learns continuous executable latents from action chunks and freezes their decoder. Its broad first-approach statement on p.2 should be restricted accordingly. Moreover, q(z|a) receives actions, not scene context; identical action chunks induce the same posterior across scenes. The representation can be shaped by dynamics, but the scene-specific consequence resides in the predictor conditioned on current vision. The current controls do not fully distinguish correct future dynamics from useful auxiliary regularization.
- The strongest raw-action comparison is not yet demonstrably optimization-matched. Appendix B, p.14, specifies an action-expert learning rate ten times the VLM rate for adapting pi0.5 to latent targets, without making equally explicit whether the raw-action baseline and every representation ablation use the same ratio and tuning budget. This matters for attribution of the largest gains. Independent-training uncertainty is also absent from the smaller differences, and ten real-robot trials per task do not establish their training stability. These are material limitations, but the available text does not prove that the baseline was disadvantaged.
[1] Chen, Xiaoyu, et al. "Villa-x: enhancing latent action modeling in vision-language-action models." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Kang, Miracle, et al. "X-tokenizer: A multimodal action tokenizer for vision-language-action pretraining." arXiv preprint arXiv:2606.14752 (2026).
Questions for authors
- Please state the initialization, learning-rate ratio, schedule, and tuning budget for the raw-action baseline and every Table 3 variant. Which gains remain under an optimization-matched raw-action comparison?
- Can the authors clarify the narrower novelty relative to villa-X and distinguish dynamics-specific supervision from a generic auxiliary target? A matched static-target or shuffled-future diagnostic would be informative, but I do not require a new comparison to X-Tokenizer, for which I verified a preprint but not peer-reviewed publication. The existing component ablations already support a useful, narrower empirical claim.
6 comments
Response to Reviewer cFNa (2/3)
W1/Q2: Future visual dynamics versus generic auxiliary supervision.
Thank you for suggesting these targeted controls. We added both static-target and shuffled-future variants to examine whether the content and temporal correspondence of visual supervision matter for downstream policy performance.
Static targets are constructed from the current visual latent. Temporally shuffled future targets randomly reorder the 12 future visual-latent time blocks within each sample, retaining their content. These changes apply to the visual supervision during Action VAE pretraining. All variants retain action reconstruction, frequency-domain reconstruction, and KL regularization, with the other representation-learning and downstream policy-training settings held fixed. The Action VAE remains frozen during downstream policy training.
The table reports success rates (%) averaged over the 50 RoboTwin 2.0 tasks, using the same pi0.5 backbone:
| Visual supervision during Action VAE pretraining | Clean | Random |
|---|---|---|
| No visual prediction supervision | 62.2 | 40.1 |
| Static visual targets | 62.3 | 39.8 |
| Temporally shuffled future visual targets | 63.1 | 41.0 |
| Correctly ordered future visual targets (ViDAL) | 65.5 | 43.1 |
Correctly ordered future supervision improves success over static targets by 3.2/3.3 percentage points on Clean/Random, and over shuffled future targets by 2.4/2.1 points. This supports the value of temporally aligned future visual supervision beyond the alternative auxiliary targets tested here.
The shuffled-target variant also performs somewhat better than the static-target variant. A possible explanation is that shuffling retains future visual content from the same sample, which may still encourage the action representation to capture information about action-induced visual changes. Static targets instead describe the current observation, which is already supplied to the predictor. Shuffling nevertheless disrupts the correspondence between actions and future temporal positions, and preserving the correct order gives the strongest downstream results.
These controls and their interpretation are now included in Section 4.4 (Table 4), with target-construction details in Appendix B.
Response to Reviewer cFNa (3/3)
W2/Q1: Optimization settings and training stability.
Thank you for asking us to make the optimization settings explicit and to provide evidence across independent training runs. We clarify the settings below and report the additional learning-rate comparisons and real-robot training seeds.
Optimization settings and tuning range. The raw-action baseline and all Table 3 policy variants use the pi0.5 base pretrained checkpoint for initialization. They share a 60,000-step policy-training schedule with 1,000 warmup steps and cosine decay. The VLM peak learning rate is , with a decay learning rate of .
The original raw-action baseline uses an action expert/VLM learning-rate ratio of 1. We additionally evaluated ratios of 3 and 10 for this baseline. All Table 3 representation variants, including the vanilla Action VAE, the variants without dynamics or frequency reconstruction, and full ViDAL, use a ratio of 10 without a separate learning-rate search for each variant. Their downstream training settings are identical, and the ablation changes the losses used during Action VAE pretraining.
Raw-action learning-rate comparison. The following table reports success rates (%) averaged over the 50 RoboTwin 2.0 tasks:
| Method | Action expert/VLM LR ratio | Clean | Random |
|---|---|---|---|
| Raw-action pi0.5 | 1 | 54.3 | 33.2 |
| Raw-action pi0.5 | 3 | 55.7 | 33.5 |
| Raw-action pi0.5 | 10 | 53.6 | 31.9 |
| pi0.5 + ViDAL | 10 | 65.5 | 43.1 |
Among the evaluated settings, the best raw-action configuration uses a ratio of 3. ViDAL exceeds it by 9.8/9.6 percentage points on Clean/Random. At the shared ratio of 10, the gains are 11.9/11.2 points. Thus, increasing the raw-action expert's learning rate does not reproduce ViDAL's gains under the evaluated settings.
Real-robot results across training seeds. Following your suggestion, we trained two additional pi0.5 + ViDAL models on Franka using seeds 43 and 44, alongside the original seed 42. Both the Action VAE and the pi0.5 policy were trained anew for each seed. Each trained model was evaluated for 10 trials per task across four tasks, giving 120 trials in total. Entries below report successful trials out of total trials:
| Training seed | Mug on Plate | Stack Bowls | Bowl in Drawer | Two Mugs in Basket | Overall |
|---|---|---|---|---|---|
| 42 | 8/10 | 8/10 | 6/10 | 7/10 | 29/40 (72.5%) |
| 43 | 8/10 | 9/10 | 8/10 | 6/10 | 31/40 (77.5%) |
| 44 | 7/10 | 8/10 | 7/10 | 6/10 | 28/40 (70.0%) |
Across the three independent training seeds, the mean success rate is 73.3%, with a sample standard deviation of 3.8 percentage points, computed over the three overall success rates. The similar performance across these runs supports the stability of ViDAL's full training pipeline on these Franka tasks.
The optimization settings and learning-rate comparisons have been added to Appendix B.
Response to Reviewer cFNa (1/3)
W1/Q2: Contribution relative to prior work and the meaning of visual grounding.
Thank you for recognizing the value of our modular action interface and its applicability to both regression and flow-matching action experts. We appreciate your careful distinction between learning an action representation and predicting scene-dependent visual dynamics. Your comments help us state the contribution more precisely.
Relation to villa-X and X-Tokenizer. We agree that the broad first-approach statement should be narrowed. The following comparison clarifies the formulation and downstream role of the representations:
ViDAL studies how conditional future-visual supervision can shape continuous action representations that serve directly as VLA prediction targets and can be decoded into control commands. The action encoder and decoder are frozen after representation learning, providing a common prediction and execution interface for regression and flow-matching action experts. We have narrowed the contribution statement in Section 1 around this formulation and clarified the key distinctions from these related works in Section 2.
Visual grounding with an action-only encoder. As you correctly point out, identical preprocessed action chunks induce the same posterior under qϕ(za∣a). During representation learning, the future-visual prediction loss backpropagates through the predictor to the action encoder, encouraging it to retain action information useful for predicting future visual latents. For a particular scene, Pθ(z0,za) combines the current visual latent z0 with the action latent za to predict the future.
Thus, visual grounding refers to future visual supervision shaping the action representation during training, while scene-specific consequences are modeled jointly from the current observation and the action representation. Section 3.2 now clarifies the encoder's action inputs, the predictor's scene conditioning, and the gradient path from the future-prediction loss to the action encoder.
References
[1] X. Chen et al., "villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models," arXiv:2507.23682, 2025.
[2] M. Kang et al., "X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining," arXiv:2606.14752, 2026.