acceptodds
Under review as a conference paper at ICLR 2027

FreshVLA: Frequency-Regularized Time-Scale Hierarchical Latent Actions for Vision-Language-Action Models

15 views

Abstract

Learning reusable latent actions from action-free videos offers an effective route to improving Vision-Language-Action (VLA) policies. However, reconstruction-oriented latent actions do not necessarily benefit policy learning: as they may encode task-irrelevant static appearance with scene identity, or entangle slowly varying task progress with fine-grained execution dynamics in limited discrete code. We propose FreshVLA, a VLA framework built on a frequency-guided, dynamics-sensitive learning paradigm for acquiring temporally hierarchical continuous latent actions. FreshVLA uses a high-pass prior to guide visual transition encoding toward dynamical regions, thereby reducing reliance on static appearance cues. It further organizes latent actions into coarse-grained task progress and fine-grained execution dynamics through cross-timescale sampling and frequency-domain regularization, enhancing their capacity to represent long-horizon behaviors. During policy training, we freeze the pretrained latent action encoder and transfer its representations to robot control through joint flow-matching of latent and robot actions. FreshVLA achieves 98.2% average success on LIBERO and 98.0% on LIBERO-Long, exceeding GR00T-N1.6 on the latter by 3.6%. It also reaches an average sequence length of 4.5 on CALVIN and strong overall success rate of 80.4% in the zero-shot LIBERO-PLUS evaluation. Furthermore, representation analysis reveals more consistent robot–object interaction attention in FreshVLA and a 51.3% greater relative error reduction in long-horizon action regression than discrete latent actions. Real-world experiments demonstrate that FreshVLA improves the average success rate over by 7.5%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent actions, vision-language-action models, temporal frequency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_m4TWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FreshVLA learns continuous latent actions from video using motion-guided attention, a coarse-to-fine temporal hierarchy, and frequency-domain regularization. The pretrained encoder is then frozen, and a downstream policy jointly generates latent and robot actions. The method achieves strong results on LIBERO, CALVIN, LIBERO, and real-world manipulation tasks. The paper also examines action decodability, frequency structure, and robustness to visual perturbations.

Critical strengths
  1. The temporal hierarchy improves control performance within the continuous representation. The comparison in Figure 4 raises average LIBERO success from 96.0% to 98.2%, with a larger gain on LIBERO-Long, from 93.0% to 98.0%. The CALVIN and real-world results provide further evidence that the approach is practically useful.

  2. The representation analyses complement the policy results. The action-regression, frequency-probing, and appearance-perturbation experiments help examine what the latent actions retain. These analyses provide more insight than benchmark success rates alone.

Critical weaknesses
  1. The gains from the representation and the downstream framework are not sufficiently separated. VILLA-X[1] and ALAM[3] already jointly generate latent and robot actions, while JALA[2] demonstrates the importance of how latent representations are aligned with and incorporated into a policy. FreshVLA combines its representation with joint flow matching, but the current experiments do not establish how much of the improvement comes from this coupling rather than the proposed frequency hierarchy. Removing latent actions changes both factors at once.

  2. The frequency regularizer lacks a direct ablation. Setting the decay coefficient to zero retains the frequency loss with uniform weighting. The 98.1% versus 98.2% comparison therefore does not establish whether frequency regularization itself is useful. A hierarchical model trained without LHL_H is needed to distinguish the architectural contribution from the loss.

  3. The VQ comparison needs tighter controls before supporting a general conclusion about continuous representations. Figure 4 already compares discrete/continuous and flat/hierarchical variants, which is useful. However, the manuscript does not sufficiently document codebook size, latent capacity, quantization settings, or codebook utilization. The large degradation of the hierarchical discrete model could reflect the particular VQ configuration or its optimization, rather than a limitation of discrete representations.

  4. The external comparisons leave the advantage over recent latent-action methods unclear. JALA is cited but not compared experimentally, and the robustness analysis mainly uses LAPA and UniVLA. The paper should clarify its differences from recent alignment-based approaches and specify the camera configurations, pretraining data, and result sources behind the main comparison table. These details matter when interpreting the reported margins.

References

[1]. Chen, X., Wei, H., Zhang, P., Zhang, C., Wang, K., Guo, Y., Yang, R., Wang, Y., Xiao, X., Zhao, L., Chen, J., & Bian, J. (2026). villa-X: Enhancing latent action modeling in vision-language-action models. In The Fourteenth International Conference on Learning Representations.

[2]. Luo, H., Wang, Y., Zhang, W., Yuan, H., Feng, Y., Xu, H., Zheng, S., & Lu, Z. (2026). Joint-aligned latent action: Towards scalable VLA pretraining in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 35047–35058).

[3]. Tang, Z., Liu, H., Chang, X., Wu, C., Huo, D., Yang, Y., Liu, B., Cai, Z., Xiong, F., Xu, M., Luo, J., Ma, D., Ma, Z., & Pan, G. (2026). ALAM: Algebraically consistent latent action model for vision-language-action models [Preprint]. arXiv.

Questions for authors
  1. Can you separate representation quality from policy integration by holding the latent encoder fixed while comparing joint denoising against an alternative use of the same latent targets, and holding the joint-denoising framework fixed while comparing latent representations?
  2. What is the performance of the hierarchical continuous model with LHL_H removed entirely?
  3. Are the VQ and continuous variants matched in latent dimensionality, temporal context, encoder/decoder capacity, data, and training budget? Please report the VQ configuration and codebook utilization, and clarify whether the conclusion holds beyond a single codebook setting.
  4. Please specify the observation configurations and result sources for the main baselines, together with the CALVIN and LIBERO pretraining/evaluation splits.
1 comment

Thank you for the additional experiments, which address my concerns about contribution attribution and frequency regularization. The VQ comparison remains limited to a single codebook configuration, and the baselines’ downstream observation settings and pretraining data still need clarification. At a minimum, the revised related work should discuss the methods mentioned in my review, including VILLA-X, JALA, and ALAM, and explicitly compare their latent representations and policy integration strategies with FreshVLA. Given the substantial progress in addressing my concerns, I raise my rating from 2 to 3.

Reviewer_Ce6ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FreshVLA learns continuous latent actions at multiple temporal scales from videos without action annotations. It uses temporal high-pass responses to guide visual attention toward changing regions, represents visual transitions at coarse and fine timescales, and regularizes their temporal structure using the discrete cosine transform (DCT). During policy training, the pretrained encoder is frozen, and latent and robot actions are jointly predicted through flow matching; future observations are not required at inference. The paper reports an average success rate of 98.2% on LIBERO, 98.0% on LIBERO-Long, an average sequence length of 4.50 on CALVIN, an average success rate of 80.4% on LIBERO-PLUS, and an average success rate of 84.2% across four real-world robotic tasks.

Critical strengths
  1. The paper presents a clear motivation and designs its representation to address the identified challenges. The authors recognize that predicting visual changes does not necessarily lead to action representations useful for control, and addresses static-appearance shortcuts and mixed temporal scales through dynamic-region guidance and a hierarchical temporal structure.
  2. The framework provides a practical way to transfer representations learned from videos without action labels to robot control. A frozen target generator provides latent-action supervision for joint flow-matching training. At inference, the policy jointly generates latent and robot actions without requiring future observations or the target generator used during training.
  3. The paper evaluates the method across standard simulation benchmarks, sequential tasks, visual perturbations, and real-world manipulation, achieving strong overall performance. The ablation in Figure 4 shows that the hierarchical continuous representation improves LIBERO-Long success from 93.0% to 98.0% over its single-scale counterpart, supporting the benefit of temporal hierarchy.
Critical weaknesses
  1. The applicability of the learned latent actions to bimanual manipulation remains unverified. Although pretraining includes data from dual-arm platforms in RoboCOIN and RoboMIND2.0, all downstream evaluations in simulation and on real robots focus on single-arm tasks. Without bimanual experiments, the current results do not establish whether the latent representations capture coordination between the two arms or effectively support bimanual control.
  2. Real-world generalization under distribution shifts lacks quantitative validation. The appendix primarily presents successful examples under changes in objects, backgrounds, positions, and lighting. Success rates and baseline comparisons under these variations are needed to substantiate the generalization claims.
  3. The correspondence between high-frequency visual changes and task-relevant actions is not sufficiently validated. The method uses temporal high-pass responses to guide attention, but prominent visual changes may arise from irrelevant motion or lighting fluctuations, while slow movements and static target locations may contain essential control information. Table 4 shows that adding this guidance improves LIBERO-Long performance but reduces LIBERO-Goal success from 98.0% to 96.3%, suggesting a task-dependent trade-off.
  4. The additional computational cost of jointly predicting latent actions remains unquantified. During inference, the model jointly denoises latent-action and robot-action sequences, but the paper does not report inference latency, memory usage, or control frequency. Comparisons with a baseline that predicts only robot actions would clarify the trade-off between performance gains and computational cost.
Questions for authors

Refer to Weakness

Reviewer_kxwvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to learn better latent action representations for facilitating vision-language-action model learning. The key idea is to leverage the dynamic information in visual dynamics changes, both filter the spatially static information, and model the temporal changes between consecutive chunks to supervise latent action learning which makes it more rich in task related interactions. The overall ablation show improvements over compared baselines.

Critical strengths
  • The modeling of latent actions is an important problem, and the intuition of using dynamic informations and filtering out action-independent background changes catches the problem and provides one potential improvement for latent action learning.
Critical weaknesses
  • The major concern lies in the learning and effectiveness of the proposed latent action learning model. First of all, leveraging DINO features already provide object-oriented features which could already largely remove the static background. Second, in the hierarchical representation learning stage, the authors basically switched from local differences to an interval avg. difference + local relative difference representation, despite the additional parameters and ablation experiments, this still could potentially lead to the same representation (i.e., s + Re(r - s) = r), and since there is no additional constraints on the residual embedding, this still seems problematic.

  • The overall selected experiment benchmarks are, in my opinion, near saturation which makes it hard to distinguish the effect of the design (all above 95% and having 1~2% difference).

  • Given that the latent actions is a relatively large-scale pre-trained one, one may wonder if this only helps for train-from-scratch settings or does it also work for finetuning pre-trained models as well.

  • I found the meaningless (r, s, e, with no clear meanings) notations in the hierarchical latent action construction section hard to follow at first glance, considering the intuitive design, the author might want to revise it to be more clear and easy to follow.

Questions for authors

See the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.