acceptodds
Under review as a conference paper at ICLR 2027

What Matters for Latent Actions in Robot Learning

Abstract

Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent action models, representation learning, embodied foundation models.

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_JA3vOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a systematic empirical study of latent action models for robot learning, evaluating 41 design choices across latent-action modeling, representation regularization, and downstream integration within a unified training pipeline. It further studies proxy metrics for latent-action quality and the benefits of latent-action VLM mid-training through simulation and real-world experiments.

Critical strengths

Comprehensive and well-controlled empirical decomposition of the LAM design space. The paper organizes a fragmented body of latent-action methods into several common design axes and evaluates them within a largely unified model, data, and downstream-policy pipeline. This controlled setup makes the resulting comparisons substantially more informative than comparisons across independently developed LAM systems, and provides a useful empirical picture of which design choices appear to matter in practice.

Useful methodological insights beyond individual model rankings. The study goes beyond benchmarking specific LAM variants and asks broader practical questions about representation regularization, latent dimensionality, integration strategies, and how latent actions should be evaluated. I particularly value the analysis showing that inexpensive proxy metrics can support coarse screening but do not reliably preserve fine-grained downstream rankings, as this provides useful guidance for future LAM development independent of the particular methods compared here.

Critical weaknesses

Limited evidence for the stability of fine-grained design recommendations. Several competitive configurations differ only marginally in downstream performance, while the reported results provide limited information about uncertainty and ranking stability across seeds, tasks, and evaluation conditions. This makes it difficult to assess whether some of the more precise rankings reflect robust differences or are effectively ties. Although the broader empirical trends remain informative, the evidence seems less conclusive for recommending particular configurations over closely performing alternatives.

Sensitivity to experimental selection and evaluation coverage. The robustness of these comparisons is also unclear in two specific respects. First, the regularization-strength study (Q2) selects representative strengths using proxy metrics, whereas the later analysis (Q4) finds these metrics less reliable for fine-grained downstream selection. It is therefore unclear whether nearby regularization strengths would preserve the reported conclusions. Second, Table 1 suggests that LIBERO-Plus and RoboTwin provide greater separation among methods, yet the evaluation covers only the RobotInit setting of LIBERO-Plus and a selected subset of RoboTwin tasks. Additional evidence from these settings would help establish whether the observed rankings are specific to the chosen evaluation conditions or generalize more broadly.

Questions for authors

Stability across evaluation settings. For configurations whose average performance is very close, have the authors examined how stable their relative rankings are across seeds, individual tasks, or additional perturbation settings? I would be interested in whether broader evaluation, for example across more LIBERO-Plus perturbations or RoboTwin tasks, reinforces the same qualitative recommendations or reveals greater task dependence.

Selection of regularization strength. I would be interested in better understanding how the regularization-strength study relates to the later analysis of proxy metrics. Since the representative regularization strengths are selected using proxy measures, while these measures are found to be more reliable for coarse screening than fine-grained downstream selection, have the authors observed whether nearby choices of regularization strength lead to similar downstream conclusions? This would help clarify how robust the reported comparison between regularization types and strengths is to the particular selection procedure.

3 comments

We appreciate the care you took in reviewing our work. As stated in Section 3.1 of the original manuscript, Table 1 reports mean success rates over three random seeds. Given the number of results in Table 1, we omitted standard deviations from the original table, and we apologize for this omission. Table 6 in Appendix C.7 of the revision uses mean ± std. Across all 60 matched method and strategy combinations, the std is highest on LIBERO-Plus and lowest on LIBERO. The mean absolute difference in std is larger across benchmarks than across integration strategies, at 0.44 versus 0.14 percentage points.

We tested how MLP probe width and optimization steps affect the correlation between the composite metric in Eq. 1 and mean closed-loop success. For the 20 LAMs with closed-loop evaluations, the ablation covered six widths and 15 checkpoints from 50 to 5,000 optimization steps, with all four weights fixed at 0.25. At width 128 and 50 optimization steps (W128/50), the composite metric had a Pearson correlation of 0.7335 with mean closed-loop success. Across all 90 settings, this correlation ranged from 0.7301 to 0.7354, although some LAMs changed rank. These results support W128/50 as a robust reference setting for the equal-weight composite metric. Appendix C.3.1 and our response to Reviewer GRPG give the equal-weight analysis and the weight comparison in Table 4.

Seven controlled action categories allowed us to examine where the IDM attends during motion and how LAPO represents similar movements. The IDM spatial attention maps in Figs. 2 and 12 show that attention concentrates around the moving arm and end effector. Under fixed camera configurations, the ratio of mean squared distances within action categories to those between categories was 0.4483 on LIBERO and 0.5907 on RoboTwin2.0. Latent actions from the same category were therefore closer on average in the original latent action space. After excluding neighbors from the same video, mean action category agreement was 99.908% on LIBERO and 99.446% on RoboTwin2.0. Within each benchmark, a sample’s nearest neighbors from other videos thus almost always shared its action category. In the joint projection in Fig. 3, however, the same direction can form separate clusters for LIBERO and for the left and right arms of RoboTwin2.0. These results suggest that a LAM can capture motion while still learning representations specific to the robot or arm. Section 3.2, Appendix C.2.1, and our response to Reviewer 3WU5 present the visualizations and distance analysis.

Following your suggestion, we tested whether the advantage of discrete latent actions under RobotInit extends to other perturbations. We evaluated policies trained on LIBERO under two additional LIBERO-Plus perturbations, Background and Layout. On RoboTwin2.0, we evaluated policies trained on clean demonstrations under randomized conditions in the clean2random setting. At their selected regularization strengths, VQ-VAE outperformed VAE at six of seven policy checkpoints with 64 dimensions and at all seven with 32 dimensions. For VAE, the mean success rates at 10−810^{-8}, 10−710^{-7}, and 10−610^{-6} differed by less than 0.2 percentage points. For Sparsity and SIGReg, a neighboring tested strength gave higher mean success than the strength selected by the open-loop metric. Across four methods, the mean absolute gap was 6.01 percentage points between strengths within a method and 1.41 points between methods at their selected strengths. VQ-VAE was the most sensitive to regularization strength, with a mean absolute gap of 18.37 points. We therefore recommend tuning regularization strength before choosing a method. Table 5 in Appendix C.7 reports the checkpoint results, and our response to Reviewer GRPG details the strength comparison.

Thank you for your detailed response. The new results regarding the statistical stability, the reliability of proxy metrics, and results on more perturbation settings have addressed my concerns. I will maintain my positive assessment.

Thank you very much for your continued support and positive assessment; we sincerely appreciate your time and constructive feedback.

Reviewer_KL3NOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper provides an experimental analysis on the design of latent-action based vision-language-action models with some conclusions made on the choice of designs on latent-action model as well as the full VLA policy.

Critical strengths
  • The experiments and ablations in this paper provides more details in comparing the results of latent-action based VLA models.
  • The paper is overall clear and easy to read.
Critical weaknesses
  • The major concern lies in the effectiveness and timeliness of the conclusions from this paper. Despite the large set of ablations on latent-action and VLA designs, there is still missing a considerably large part of VLA/WAM architectures, (e.g. pi-style mixture-of-transformers, leveraging visual latent features directly as IDM inputs). This makes the experiments and conclusions from this paper less informative given that some of the results have been tested differently given coordination with the latent action model and VLA models.

  • The overall feel of this paper seems more like experiment-first-then-summarize, instead of providing a rationale on how we study the the latent actions and their connections to both visual dynamics and VLA learning. This makes the conclusions very engineering-oriented instead of providing new insights to either better analyze the problem or elicit new design ideas on top of the current conclusions. Such testing is meaningful but provides limited technical contribution in my opinion.

  • The overall experimental scale is at the dataset level and the performance is somewhat close to one-another on these benchmarks. This makes it difficult to analyze if the same conclusions could be drawn in larger-scale pre-training, or steering pre-trained models via post-training.

Questions for authors

See the weakness seciton. Plus:

  • Several ablations that could be added, including pi-style models and their interactions with latent actions using shared branch as well as separate denoising branches; feeding in latent representations similarly as separate images instead of calculating the difference.
1 comment

We sincerely thank you for taking the time to review our work carefully. We apologize that the original manuscript did not clearly explain our motivation or our experimental findings. We investigate which design choices in LAM pre-training most strongly affect latent action quality for downstream robotic manipulation. Despite recent progress in latent action learning, conventional VLA policies remain more common in practice. Most LAM studies evaluate downstream success under their own experimental settings. Using a LAM can require training it on new videos and post-training a policy to measure the gain in manipulation success. Both stages are costly, and the gain can be small. One obstacle is the lack of a widely reusable motion representation. We regard a LAM as a reusable feature extractor for motion between video frames, much like DINO and V-JEPA. We therefore compare LAM pre-training choices in a common pipeline.

In the revised manuscript, we present visualizations and statistical analyses of latent actions in Section 3.2 and Appendix C.2.1. Table 4 in Appendix C.3.1 shows how MLP probe width and optimization steps affect the correlation between the composite metric and mean closed-loop success. Extending the VAE ablation in Fig. 6, we compare 32 and 64 dimensions for VAE and VQ-VAE at their selected strengths in Table 5. Design III in Section 2 and the experiments in Section 3.4 also describe two uses of latent actions. First, a LAM annotates videos without physical action labels, and fine-tuning the VLM backbone on these latent actions provides a stronger initialization for downstream policies using DAP or LAP. Second, the VLM predicts both latent and physical actions from the current observation and instruction, a strategy we call JAP. In this setting, the latent action can provide information about visual changes between the current and future observations. Figs. 2 and 12 show IDM spatial attention around the moving arm and end effector. As for JAP-LAP and JAP-DAP, our original aim was to test whether jointly fine-tuning the VLM on latent and physical actions provides a stronger initialization for downstream policy learning. We therefore compare JAP-LAP with LAP and JAP-DAP with DAP. The small gap between JAP-LAP and JAP-DAP may arise because policy post-training narrows performance differences between the two models.

A WAM can use its VGM backbone’s intermediate features to capture motion between frames without an IDM. Using an IDM-inferred latent action as a policy input requires predicting a future frame at inference, complicating the assessment of LAM pre-training. Combining latent actions with a π-style flow matching action expert or a diffusion policy also affects supervision. A stochastic policy can sample a latent action inconsistent with the recorded physical action used to train the decoder. LAFP studies this mismatch in latent flow policies. GE-Act 2.0 addresses a related mismatch between predicted visual futures and recorded actions. This complicates comparisons of LAM pre-training choices. LAFP · GE-Act 2.0

Our experiments yield five key insights.

  1. In Fig. 4, FDM reconstruction metrics tend to correlate more strongly with closed-loop success than action probe metrics. Policy post-training may narrow success-rate gaps among useful latent action representations, while poor representations can still impair manipulation. This may be an important reason why open-loop metrics support coarse screening more reliably than fine-grained ranking of LAMs.
  2. Regularization strength can have a larger effect on closed-loop success than the choice of method. Across four methods, the mean absolute gap in closed-loop success was 6.01 percentage points between strengths within a method and 1.41 points between methods at their selected strengths. Our response to Reviewer GRPG gives the full comparison.
  3. Once the latent actions are regularized, preserving their original scale often helps downstream policy learning. Unnormalized latent actions in Stage II improved success in 28 of 33 method and benchmark combinations, by 1.15 percentage points on average across all 33. Section 3.5 and Fig. 15 report the results.
  4. Fine-tuning on latent actions gives the VLM a stronger initialization for downstream robotic manipulation tasks. In joint prediction, latent actions also provide supervision about future visual changes. Fig. 8 isolates the effect of initialization on the real robot. Fig. 7 and Table 5 report the Stage-II corpus comparison.
  5. Learning action semantics shared across embodiments may remain difficult under self-supervised LAM pre-training. Under fixed viewpoints, latent actions from the same action category were closer on average. Even on the same RoboTwin2.0 platform, however, the same direction can form separate clusters for the left and right arms in Fig. 3. Appendix C.2.1 and our response to Reviewer 3WU5 give details.
Reviewer_3WU5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of latent action learning for robotic manipulation. It compares 41 design configurations spanning modeling paradigms, regularization, and integration strategies, and examines four proxy metrics against downstream performance. Experiments cover three simulation benchmarks and real-robot tasks, providing practical recommendations for latent-action learning and VLM adaptation.

Critical strengths
  1. The study covers a useful range of interconnected design choices under a common framework, including representation learning, regularization, policy integration, and evaluation metrics.

  2. The proxy-metric analysis addresses an important practical problem: reducing the cost of model selection without assuming that action-recovery accuracy necessarily predicts policy performance.

  3. The simulation experiments and controlled real-robot comparison provide positive evidence that latent-action tuning can improve initialization for downstream policy learning.

Critical weaknesses
  1. The coverage of the study does not fully match its broad framing.

    The paper specifies its experimental setting and extensively studies modeling, regularization, and integration choices. However, under the title “What Matters for Latent Actions in Robot Learning” and the framing of a comprehensive empirical study, the selection of investigated dimensions requires stronger justification. Important questions remain largely unexplored: whether motion representations remain consistent across viewpoints, whether they transfer across held-out embodiments, and whether latent-action tuning affects the VLM’s original visual-language capabilities. Evaluating separately adapted single-arm and bimanual policies does not establish cross-embodiment representation transfer, while improved manipulation performance alone does not characterize potential trade-offs in semantic capability retention. Targeted analyses, or a more restricted framing with a justification for these exclusions, would better align the study’s coverage with its stated ambition.

  2. The study provides limited direct analysis of the semantic structure of latent representations.

    Although the paper extensively measures downstream performance, it provides little visualization or controlled analysis of what the latent representations encode. It remains unclear whether similar motions form consistent groups, whether discrete codes correspond to reusable motion patterns, and how these structures differ across modeling and regularization choices. The discussion offers reusable action primitives and structured motion representations as possible explanations, but these interpretations receive limited direct examination. Visualization paired with motion examples, nearest-neighbor retrieval, or controlled latent interventions would help explain the observed performance trends. Visualization alone would not establish semantic structure, but could support such analysis when combined with behavioral evidence.

  3. Interactions between design choices are insufficiently characterized.

    Section 3.2 compares modeling paradigms at a fixed latent dimensionality and regularization strength, while the regularization and dimensionality studies primarily use LAPO. These experiments establish performance under the selected configurations, but provide limited evidence about whether the rankings persist when other design choices change. This is particularly relevant because Section 3.3 identifies regularization strength as an important factor. Targeted sensitivity tests across selected paradigms would help distinguish stable findings from configuration-dependent recommendations without requiring a complete factorial study.

  4. The reliability of recommendations based on small aggregate differences is not sufficiently quantified.

    Section 3.4 and Appendix C.4 appropriately acknowledge benchmark-specific exceptions. The remaining concern is how strongly the overall means support a preferred strategy. For example, the reported overall difference between JAP-LAP and JAP-DAP is only 0.31 percentage points. Although results are averaged over three seeds, uncertainty around these aggregate differences is not reported. In addition, LIBERO and LIBERO-Plus share an underlying task setting, making the sensitivity of rankings to benchmark weighting relevant. Uncertainty estimates and alternative groupings would clarify whether the results support a reliable preference or indicate broadly comparable alternatives.

  5. The proxy-metric conclusions remain dependent on an insufficiently validated evaluation protocol.

    The authors correctly acknowledge that proxy metrics support coarse screening rather than reliable fine-grained ranking. However, the relative reliability of reconstruction and probe metrics is evaluated using a single probing protocol with 50 optimization steps. Without sensitivity checks, it remains unclear how much of the observed difference reflects representation properties versus ease of decoding under that budget. Furthermore, Appendix C.3 selects regularization strengths by summing two reconstruction gains and two negative probe losses, without explaining their relative scaling or weighting. Since this criterion determines the configurations used in subsequent comparisons, its sensitivity matters for both the metric analysis and the resulting regularization recommendations.

Minor presentation issue. Figures 1 and 3 and Table 1 require frequent cross-referencing between method names, paradigms, and integration acronyms. Consistent grouping and brief annotations of inputs, reconstruction targets, and prediction paths would improve readability.

Questions for authors

Latent action learning is an important and broad topic, and I appreciate the authors’ substantial experimental effort. The paper already describes its experimental setting; my concern is whether the selected scope sufficiently addresses the broad question posed by the title and the framing of a comprehensive study. The semantic structure of latent representations, cross-view and cross-embodiment transfer, and preservation of the VLM’s original capabilities seem directly relevant to understanding what makes latent actions useful for robot learning. I would be open to reconsidering my rating if the authors can provide targeted evidence on these aspects, or justify their exclusion and clarify the intended contribution, while addressing the robustness concerns below.

  1. Latent visualization and semantic structure. Can you visualize the learned representations together with their corresponding motion examples? For continuous representations, do nearest neighbors or interpolations correspond to meaningful motion relationships? For discrete representations, do individual codes exhibit consistent motion semantics? Comparing these patterns across methods would help explain which properties contribute to downstream performance.

  2. Cross-view consistency and cross-embodiment transfer. Do comparable motions retain consistent latent structure across viewpoints or embodiments? Can you provide controlled cross-view analysis or transfer to an embodiment excluded from latent-action training? The latter could allow limited physical-action calibration rather than requiring fully zero-shot control.

  3. VLM capability retention. Does latent-action tuning preserve the backbone’s original visual-language understanding and compositional instruction-following abilities? A before-and-after evaluation would help determine whether improved manipulation performance involves a trade-off with semantic capability retention.

  4. Interactions between design choices. Have you checked whether the main paradigm rankings persist under a few alternative regularization strengths or latent dimensions? Which recommendations have evidence of holding across these changes, and which remain specific to the selected configuration?

  5. Statistical and aggregation sensitivity. Can you quantify uncertainty in the aggregate integration-strategy differences, particularly JAP-LAP versus JAP-DAP? Do the preferred strategies remain unchanged when results are summarized separately by benchmark family or when LIBERO and LIBERO-Plus are grouped together?

  6. Probe sensitivity. Do the relative correlations of reconstruction and probe metrics remain stable with longer probe training or alternative probe capacities? Is the 50-step budget intended to measure rapid decodability, or is there evidence that it sufficiently optimizes the probes across the compared representations?

  7. Hyperparameter selection and screening utility. Are the four components of the composite selection criterion normalized or reweighted before summation, and how sensitive are the selected regularization strengths to these choices? Beyond global correlations, can you test whether the metrics reliably retain promising configurations or discard poor ones on held-out configurations?

3 comments

We sincerely thank the reviewer for the careful assessment and constructive suggestions. We investigate which design choices in LAM pre-training most strongly affect latent action quality for downstream robotic manipulation. We regard a LAM as a reusable feature extractor for motion between video frames, much like DINO and V-JEPA. Following the reviewer’s suggestion, we added visualizations and statistical analyses of latent actions in Section 3.2 and Appendix C.2.1. Fig. 2 covers seven action categories and shows that, in the motion examples, IDM spatial attention concentrates on the moving robot arm and captures information about its displacement. Across the frame intervals in Fig. 12, attention remains localized around the moving arm. Between the 1-frame and 27-frame intervals, the attention centroid shifts downward on LIBERO and upward on RoboTwin2.0, following the end effector. This suggests that, given the current observation, the IDM attends to the position toward which the end effector moves in the future frame. With camera configuration held fixed, the ratio of within-category to between-category mean squared distances was 0.4483 on LIBERO and 0.5907 on RoboTwin2.0. Both ratios are below one, so latent actions from the same category were closer on average. We then pooled views within each benchmark and excluded neighbors from the same video. Nearby latent actions from different videos almost always shared the same action category, with mean agreement of 99.908% on LIBERO and 99.446% on RoboTwin2.0. Yet in Fig. 3, the left and right arms of the same RoboTwin2.0 robot form separate clusters for the same direction. This suggests that learning action semantics shared across embodiments remains a challenge. Cross-view consistency can refer to two third-person views or to a third-person view and a wrist view. A latent action encodes motion between two frames. For the same physical action, changes in the robot arm and the scene look different across views. This makes consistency difficult to learn without additional supervision from physical actions, robot states or depth. This question is closer to 3D flow methods such as PointWorld, where predicting full-scene 3D point flow implicitly captures the scene’s 3D structure. The question of how to use latent actions while preserving the VLM backbone’s original visual-language capabilities, which VLA training itself can weaken, merits a detailed study in its own right, whereas the present study examines how design choices in self-supervised LAM pre-training on 2D videos affect latent action quality for downstream robotic manipulation.

In the revised manuscript, we have clarified how the experiments build on one another. For clarity, we use Linear probe, MLP probe, MSE Gain, and SSIM Gain as open-loop metrics and downstream task success rate as the closed-loop metric. In Design I, we compared seven latent action modeling paradigms on the same raw videos. Although policy post-training may narrow success-rate gaps between representations, the closed-loop comparison in Table 1 can still reveal larger gaps when poorly learned representations impair manipulation. LAPO required no additional preprocessing and achieved the highest mean success rate across the three benchmarks. In Section 3.3 of the revised manuscript, we recommend tuning regularization strength within each method before choosing among methods. Across four regularization methods, the mean absolute success-rate gap was 6.01 percentage points between strengths within a method and 1.41 between methods at their selected strengths. The additional robustness evaluations on LIBERO-Plus and RoboTwin2.0 suggest a modest advantage for VQ-VAE over VAE at their selected strengths. However, VQ-VAE was the most sensitive to regularization strength, with a mean absolute gap of 18.37 percentage points between tested strengths. Our reply to GRPG gives the strength comparison. For the equal-weight composite, Pearson rr was 0.7335 at W128/50 and 0.7301–0.7354 across 90 settings. Three fixed components carry 75% of the weight. At width 128, negative MLP probe loss gave Pearson rr / Spearman ρ\rho of 0.5197/0.6722 at 50 steps, 0.5143/0.6226 at 500 and 0.5251/0.6256 at 5,000. Both reconstruction metrics correlated more strongly with mean closed-loop success at all three budgets on the same 20 LAMs. See Appendix C.3.1. Finally, to test whether the advantage of 32 over 64 dimensions in Fig. 6 extends across regularization methods, we compared these two dimensions for VAE at 10−710^{-7} and VQ-VAE at 11 in Table 5. For each method, 32 dimensions gave higher success at five of seven checkpoints, including the final checkpoint on both benchmarks. This agrees with the closed-loop ablation in Fig. 6 for VAE at λreg=10−6\lambda_{\mathrm{reg}}=10^{-6} and shows that the preference for 32 over 64 dimensions holds at the final checkpoints for both methods at their selected strengths.

Dear Reviewer 3WU5,

We hope this message finds you well. With the deadline of discussion period is approaching, we wanted to make sure we have fully addressed your concerns. If you have any further points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are open to any further suggestions you might have to improve the paper.

Thank you for the time and effort you have dedicated to reviewing our paper.

Best regards,

The Authors

Thank you very much for your positive feedback and for raising your rating; we sincerely appreciate your time and constructive suggestions, which have helped us improve our work.

Reviewer_GRPGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically compares 41 design choices for latent-action learning, covering modeling paradigms, regularization, and policy integration. Experiments on LIBERO, LIBERO-Plus, and RoboTwin2.0 show that simple baselines remain competitive and provide practical training recommendations. The paper also evaluates proxy metrics for model screening and demonstrates that latent-action-based VLM mid-training improves initialization for downstream policy learning.

Critical strengths
  1. The paper makes a substantial empirical contribution by comparing latent-action methods that have largely been studied under different experimental settings.
  2. The comprehensive simulation experiments, complemented by real-robot validation, provide valuable reference for understanding existing approaches and their practical trade-offs.
  3. The paper offers practical guidance supported by detailed comparisons useful for designing and evaluating latent-action systems.
Critical weaknesses
  1. Most design recommendations are derived with other choices held fixed (VLA strucrture, base VLM, etc), leaving their applicability to other architectures and settings unclear.
  2. The conclusions about robustness and zero-shot generalization require stronger evidence. The recommendation to prefer discrete latent actions relies mainly on LIBERO-Plus setting, whose videos are already included in earlier training stages. This demonstrates transfer without target-setting action supervision, but does not establish that the same advantage holds under unseen distribution shifts.
Questions for authors

For probing experiments, I wonder whether only 50 optimization steps is enough?

4 comments

Thank you for your careful review and constructive feedback. We apologize for the confusion caused by our use of “zero-shot generalization” and have replaced it with “robustness” in the revised manuscript. To test whether the VQ-VAE advantage under RobotInit persists in a broader evaluation, we compared discrete VQ-VAE and continuous VAE at matched latent action dimensions and policy checkpoints. For policies trained on LIBERO, we added the Background and Layout perturbations on LIBERO-Plus. On RoboTwin2.0, we added the clean2random setting to evaluate policies trained on clean demonstrations under randomized conditions. In Table 5 of Appendix C.7, VQ-VAE outperformed VAE at six of seven checkpoints with 64 dimensions and at all seven with 32 dimensions. At 64 dimensions, its mean success rate exceeded that of VAE by 0.61 percentage points on LIBERO-Plus and 2.97 points on RoboTwin2.0. These results suggest a modest advantage for VQ-VAE under the tested conditions and at the selected strengths. To assess sensitivity to regularization strength, we tested 10−210^{-2}, 10−410^{-4}, 10−610^{-6}, and 10−810^{-8} as well as the strength selected for each method. The selected strengths were 10−710^{-7} for VAE, 10−510^{-5} for Sparsity, 10−310^{-3} for SIGReg, and 11 for VQ-VAE. We compared mean closed-loop success at the final checkpoints on LIBERO and RoboTwin2.0 mixed, weighting the two benchmarks equally after averaging DAP and LAP within each. VQ-VAE was the most sensitive to regularization strength. Its mean absolute gap in success rate between pairs of strengths was 18.37 percentage points, compared with 3.33 for VAE, 1.31 for Sparsity, and 1.03 for SIGReg. The within-method gap averaged 6.01 points across the four methods, while the gap between methods at their selected strengths was 1.41 points. We therefore recommend tuning regularization strength before choosing a method, as stated in Section 3.3. Fig. 13 shows how the probe and reconstruction metrics vary with regularization strength and marks the selected strength for each method. Appendix C.3.2 discusses these results.

We originally chose 50 probe optimization steps to compare how readily different latent actions map to physical actions under the same optimization budget. To test whether this choice affects the assessment of latent action quality, we varied MLP probe width and optimization steps and measured how the composite metric correlated with mean closed-loop success across 20 LAMs. We froze each LAM and trained separate probes for end-effector delta control and absolute joint position. The LIBERO-90 training set contained 495,168 frame pairs from 80 tasks, and the evaluation set contained 70,160 pairs from the 10 held-out tasks. The composite metric in Eq. 1 adds SSIM Gain and MSE Gain and subtracts the Linear probe and MLP probe losses. After standardizing the four open-loop components, the reference MLP with two hidden layers of width 128 and 50 optimization steps (W128/50) gave Pearson r=0.7335r=0.7335 and Spearman ρ=0.6481\rho=0.6481 with equal weights of 0.25. Across six widths and 15 checkpoints through 5,000 steps, equal-weight Pearson correlations ranged from 0.7301 to 0.7354. Only nine of the other 89 settings exceeded the reference Spearman correlation with equal weights. Optimizing the weights separately for each alternative on the same 20 LAMs gave median correlations of r=0.7396r=0.7396 and ρ=0.6526\rho=0.6526. We retain W128/50 for coarse screening because its association with downstream success is stable across probe budgets and the weights require no fitting to downstream results. Fine-grained rankings can still change. Appendix C.3.1 gives the equal-weight analysis and the weight comparison in Table 4. Fig. 4 compares the correlations of the individual metrics with downstream task success.

To examine what motion information latent actions capture, we analyzed seven controlled action categories on LIBERO and RoboTwin2.0. In Fig. 2, IDM spatial attention concentrates around the moving arm. In the two examples in Fig. 12, attention on the fixed first observation stays near the gripper and wrist. Between intervals of 1 and 27 frames, the attention centroid shifts in the direction of end-effector motion, downward on LIBERO and upward on RoboTwin2.0. This suggests that the IDM attends to the position toward which the end effector moves in the future frame. Under fixed camera configurations, latent actions from the same category were closer on average than those from different categories. Even after excluding neighbors from the same video, mean action category agreement was 99.908% on LIBERO and 99.446% on RoboTwin2.0. Yet the same direction can form separate clusters for LIBERO and for the two RoboTwin2.0 arms in Fig. 3. This suggests that capturing visual motion and learning action semantics shared across embodiments may be distinct challenges. Section 3.2 and Appendix C.2.1 present the analysis, which we discuss further in our response to Reviewer 3WU5.

Dear Reviewer GRPG,

We hope this message finds you well. With the deadline of discussion period is approaching, we wanted to make sure we have fully addressed your concerns. If you have any further points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are open to any further suggestions you might have to improve the paper.

Thank you for the time and effort you have dedicated to reviewing our paper.

Best regards,

The Authors

Thanks for your detailed responses. I will maintain my positive score.

Thank you very much for your continued support and positive assessment; we sincerely appreciate your time and constructive feedback.

Comments on the paper

3 comments

Dear Reviewer KL3N, as the discussion deadline is approaching, we would sincerely appreciate your feedback on whether our responses have addressed your concerns and whether you would consider updating your assessment accordingly.

Dear Reviewer JA3v, as the discussion deadline is approaching, we would sincerely appreciate your feedback on whether our responses have addressed your concerns and whether you would consider updating your assessment accordingly.

AuthorsOpenReview ↗

Summary of Key Findings and Additional Experiments

We thank the Area Chair and reviewers for their careful reading. Latent action models (LAMs) are usually evaluated with different policies and training data, making it difficult to tell which pre-training choices improve the representation itself. We study the LAM as a reusable feature extractor for motion between video frames. We therefore compared 41 designs in a common pipeline on three simulation benchmarks, LIBERO, LIBERO-Plus, and RoboTwin2.0, and tested policy initialization on four real-robot manipulation tasks.

  1. Future-frame reconstruction gains generally correlate more strongly with downstream success than action probe scores. Policy post-training may narrow differences in downstream success among otherwise distinct latent action representations. This makes fine-grained model selection difficult even when open-loop metrics are useful for coarse screening. See Fig. 4 in Section 3.3.
  2. Tuning regularization strength within a method can affect downstream success more than switching methods at their selected strengths. See Appendix C.3.2.
  3. Preserving the scale of regularized latent actions during vision-language model (VLM) fine-tuning improved success in 28 of 33 method and benchmark combinations, with a mean gain of 1.15 percentage points across all 33. See Fig. 15.
  4. Latent actions provide a stronger initialization for downstream policies and can supply supervision about future visual changes when predicted jointly with physical actions. On the four real-robot tasks, latent action fine-tuning raised mean success from 64.75% to 79.25% across five policy checkpoints. See Section 3.4 and Fig. 8.
  5. Within each benchmark, similar latent actions consistently correspond to the same action category across videos, yet learning action semantics shared across embodiments may remain difficult under self-supervised LAM pre-training. See Fig. 3 in Section 3.2 and Appendix C.2.1.

Concern 1: Latent action consistency across embodiments (Reviewer 3WU5).\textcolor{red}{\textbf{Concern 1: Latent action consistency across embodiments (Reviewer 3WU5).}} We added spatial attention maps and analyzed the latent actions of LAPO for seven controlled action categories on LIBERO and RoboTwin2.0. In Fig. 2, the inverse dynamics model attends to the moving arm and end effector. Between the 1-frame and 27-frame intervals in Fig. 12, the attention centroid on the fixed first observation shifts downward on LIBERO and upward on RoboTwin2.0, following the end effector in both images. With the camera configuration held fixed, the ratio of within-category to between-category mean squared distances was 0.4483 on LIBERO and 0.5907 on RoboTwin2.0. Thus, latent actions from the same category were closer on average. In a separate analysis within each benchmark, neighbors from other videos almost always shared the same action category, with mean agreement of 99.908% on LIBERO and 99.446% on RoboTwin2.0. Yet the middle panel of Fig. 3 shows separate clusters for the left and right arms of RoboTwin2.0 even when both arms are the same model and move in the same direction. Learning action semantics shared across embodiments may remain difficult under self-supervised LAM pre-training. Section 3.2 and Appendix C.2.1 give the visualizations and statistical analysis.

Concern 2: Sensitivity to the probe training budget (Reviewers GRPG and 3WU5).\textcolor{red}{\textbf{Concern 2: Sensitivity to the probe training budget (Reviewers GRPG and 3WU5).}} We varied MLP probe width and training duration across 90 settings on the same 20 LAMs with closed-loop evaluations. With four equal weights, the composite metric’s Pearson correlation with mean downstream success was 0.7335 at width 128 and 50 steps and ranged from 0.7301 to 0.7354 across all settings. Because only the MLP component varied, we also compared its score directly with both reconstruction gains at three checkpoints. Both gains correlated more strongly with success at each checkpoint. The reference probe setting gives a stable association with downstream success, although fine-grained rankings can change. See Appendix C.3.1 and Table 4.

Concern 3: Robustness across perturbations, latent dimensions, and regularization strengths (Reviewers GRPG, JA3v, and 3WU5).\textcolor{red}{\textbf{Concern 3: Robustness across perturbations, latent dimensions, and regularization strengths (Reviewers GRPG, JA3v, and 3WU5).}} We evaluated policies trained on LIBERO under two additional LIBERO-Plus perturbations, Background and Layout. The RoboTwin2.0 clean2random setting evaluates policies trained on clean demonstrations under randomized conditions. In Table 5, VQ-VAE outperformed VAE at six of seven policy checkpoints with 64 dimensions and all seven with 32 dimensions, a modest advantage at the selected strengths. A separate sweep on LIBERO and RoboTwin2.0 mixed showed a mean absolute success-rate gap of 6.01 percentage points between strengths within a method, compared with 1.41 points between methods at their selected strengths. VQ-VAE was the most sensitive to strength. We therefore recommend tun

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.