- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a systematic empirical study of latent action models for robot learning, evaluating 41 design choices across latent-action modeling, representation regularization, and downstream integration within a unified training pipeline. It further studies proxy metrics for latent-action quality and the benefits of latent-action VLM mid-training through simulation and real-world experiments.
Critical strengths
Comprehensive and well-controlled empirical decomposition of the LAM design space. The paper organizes a fragmented body of latent-action methods into several common design axes and evaluates them within a largely unified model, data, and downstream-policy pipeline. This controlled setup makes the resulting comparisons substantially more informative than comparisons across independently developed LAM systems, and provides a useful empirical picture of which design choices appear to matter in practice.
Useful methodological insights beyond individual model rankings. The study goes beyond benchmarking specific LAM variants and asks broader practical questions about representation regularization, latent dimensionality, integration strategies, and how latent actions should be evaluated. I particularly value the analysis showing that inexpensive proxy metrics can support coarse screening but do not reliably preserve fine-grained downstream rankings, as this provides useful guidance for future LAM development independent of the particular methods compared here.
Critical weaknesses
Limited evidence for the stability of fine-grained design recommendations. Several competitive configurations differ only marginally in downstream performance, while the reported results provide limited information about uncertainty and ranking stability across seeds, tasks, and evaluation conditions. This makes it difficult to assess whether some of the more precise rankings reflect robust differences or are effectively ties. Although the broader empirical trends remain informative, the evidence seems less conclusive for recommending particular configurations over closely performing alternatives.
Sensitivity to experimental selection and evaluation coverage. The robustness of these comparisons is also unclear in two specific respects. First, the regularization-strength study (Q2) selects representative strengths using proxy metrics, whereas the later analysis (Q4) finds these metrics less reliable for fine-grained downstream selection. It is therefore unclear whether nearby regularization strengths would preserve the reported conclusions. Second, Table 1 suggests that LIBERO-Plus and RoboTwin provide greater separation among methods, yet the evaluation covers only the RobotInit setting of LIBERO-Plus and a selected subset of RoboTwin tasks. Additional evidence from these settings would help establish whether the observed rankings are specific to the chosen evaluation conditions or generalize more broadly.
Questions for authors
Stability across evaluation settings. For configurations whose average performance is very close, have the authors examined how stable their relative rankings are across seeds, individual tasks, or additional perturbation settings? I would be interested in whether broader evaluation, for example across more LIBERO-Plus perturbations or RoboTwin tasks, reinforces the same qualitative recommendations or reveals greater task dependence.
Selection of regularization strength. I would be interested in better understanding how the regularization-strength study relates to the later analysis of proxy metrics. Since the representative regularization strengths are selected using proxy measures, while these measures are found to be more reliable for coarse screening than fine-grained downstream selection, have the authors observed whether nearby choices of regularization strength lead to similar downstream conclusions? This would help clarify how robust the reported comparison between regularization types and strengths is to the particular selection procedure.
3 comments
Thank you for your detailed response. The new results regarding the statistical stability, the reliability of proxy metrics, and results on more perturbation settings have addressed my concerns. I will maintain my positive assessment.
Thank you very much for your continued support and positive assessment; we sincerely appreciate your time and constructive feedback.
We appreciate the care you took in reviewing our work. As stated in Section 3.1 of the original manuscript, Table 1 reports mean success rates over three random seeds. Given the number of results in Table 1, we omitted standard deviations from the original table, and we apologize for this omission. Table 6 in Appendix C.7 of the revision uses mean ± std. Across all 60 matched method and strategy combinations, the std is highest on LIBERO-Plus and lowest on LIBERO. The mean absolute difference in std is larger across benchmarks than across integration strategies, at 0.44 versus 0.14 percentage points.
We tested how MLP probe width and optimization steps affect the correlation between the composite metric in Eq. 1 and mean closed-loop success. For the 20 LAMs with closed-loop evaluations, the ablation covered six widths and 15 checkpoints from 50 to 5,000 optimization steps, with all four weights fixed at 0.25. At width 128 and 50 optimization steps (W128/50), the composite metric had a Pearson correlation of 0.7335 with mean closed-loop success. Across all 90 settings, this correlation ranged from 0.7301 to 0.7354, although some LAMs changed rank. These results support W128/50 as a robust reference setting for the equal-weight composite metric. Appendix C.3.1 and our response to Reviewer GRPG give the equal-weight analysis and the weight comparison in Table 4.
Seven controlled action categories allowed us to examine where the IDM attends during motion and how LAPO represents similar movements. The IDM spatial attention maps in Figs. 2 and 12 show that attention concentrates around the moving arm and end effector. Under fixed camera configurations, the ratio of mean squared distances within action categories to those between categories was 0.4483 on LIBERO and 0.5907 on RoboTwin2.0. Latent actions from the same category were therefore closer on average in the original latent action space. After excluding neighbors from the same video, mean action category agreement was 99.908% on LIBERO and 99.446% on RoboTwin2.0. Within each benchmark, a sample’s nearest neighbors from other videos thus almost always shared its action category. In the joint projection in Fig. 3, however, the same direction can form separate clusters for LIBERO and for the left and right arms of RoboTwin2.0. These results suggest that a LAM can capture motion while still learning representations specific to the robot or arm. Section 3.2, Appendix C.2.1, and our response to Reviewer 3WU5 present the visualizations and distance analysis.
Following your suggestion, we tested whether the advantage of discrete latent actions under
RobotInitextends to other perturbations. We evaluated policies trained on LIBERO under two additional LIBERO-Plus perturbations,BackgroundandLayout. On RoboTwin2.0, we evaluated policies trained on clean demonstrations under randomized conditions in theclean2randomsetting. At their selected regularization strengths, VQ-VAE outperformed VAE at six of seven policy checkpoints with 64 dimensions and at all seven with 32 dimensions. For VAE, the mean success rates at 10−8, 10−7, and 10−6 differed by less than 0.2 percentage points. For Sparsity and SIGReg, a neighboring tested strength gave higher mean success than the strength selected by the open-loop metric. Across four methods, the mean absolute gap was 6.01 percentage points between strengths within a method and 1.41 points between methods at their selected strengths. VQ-VAE was the most sensitive to regularization strength, with a mean absolute gap of 18.37 points. We therefore recommend tuning regularization strength before choosing a method. Table 5 in Appendix C.7 reports the checkpoint results, and our response to Reviewer GRPG details the strength comparison.