- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript studies the performance gap between regression and generative policies in imitation learning. The authors identify this performance gap is caused by the heteroscedastic and heavy-tailed action residuals, which makes the gradient concentrated on high-residual samples. They also propose HT-Policy to adaptively model residual scales and alliveate the effect of large errors. Experiments on simulations, pretrained VLA/WAM models, and real-world robotic tasks show that HT-Policies achieve performance comparable to generative policies while enabling single-step action prediction and faster inference.
Critical strengths
-
The authors proposed an interesting assumption that Flow Matching-based policies outperforms regression-based lies in residual modeling and gradient allocation, rather than in multi-modal modeling and provide quantified analysis.
-
The authors proposed HT-Policy and conducted extensive experiments to demonstrate that employing an appropriate robust regression loss can significantly improve the performance of direct action regression.
Critical weaknesses
-
Although the authors have observed the phenomenon of heavy-tailed action residuals and provides explanations from both statistical and optimization perspectives, the evidence linking these heavy-tailed action residuals to performance disparities remains somewhat insufficient. I would be very interested to see empirical evidence demonstrating, for instance, that in a task where the action residual distribution aligns with the assumptions underlying MSE optimization, an MSE-based approach could perform on par with a Flow-based model. Alternatively, it would be compelling to see whether identifying specific phases within a task, distinguishing between those with high residuals and those with low residuals, and employing separate neural networks to model each, leads to an improvement in performance.
-
The optimization mechanism proposed by the authors could be articulated more clearly. If the discrepancy between MSE and Flow is indeed attributable to gradient allocation, I would expect to see further evidence at the data level—for instance, by employing data sampling strategies or gradient reweighting techniques to align the gradient distribution as closely as possible with that produced by the HT policy, thereby validating the proposed optimization mechanism.
-
The empirical baselines used for comparison can be more sufficient. Although the authors criticize certain drawbacks of MIP in the introduction, it is regrettable that they have not included MIP in their comparative analysis. I would like to see how MIP performs in terms of action residual TALS and gradient allocation.
Questions for authors
Out of curiosity, I am very interested to see what would happen if the author used a "student t loss" to optimize the flow policy—that is, replacing the regression step at each stage of the flow policy with a student t loss. I wonder what the results might be, and I would appreciate an explanation of the potential outcome.
Did the authors try using HT policy in fine-tuning/pretraing regression-based VLAs such as OpenVLA? If HT-Policy can achieve performance comparable to that of flow-based VLAs and considering the inferencing speed-up it brings, it would be another great contribution.
If my raised concerned are properly addressed, I would be happy to raise my score.
3 comments
I appreciate the authors' responses and the provided additional results, which have largely addressed my concerns. I have raised my rating score to reflect it.
We truly appreciate your valuable comments and the positive assessment of our work!
We thank the reviewer for these constructive suggestions. Since we cannot include figures directly in this response, we have presented all the corresponding experimental results and figures in the revised manuscript and summarize the key findings below.
Q1. Can MSE-Policies achieve competitive performance when residuals match MSE's assumption or with phase-specific training?
A1. For tasks where Flow-Policies and HT-Policies outperform MSE-Policies, we find it difficult to obtain datasets whose prediction residuals satisfy MSE's fixed-scale Gaussian assumption, even with smooth actions generated by scripted policies. Alternatively, we investigate phase-specific MSE training and find that it substantially improves performance on challenging tasks. Our newly added phase-specific MSE-Policy experiment (Exp. D, Sec. 5.5, Fig. 12) provides empirical evidence for this. As shown in Figure 12, this approach substantially improves the success rate on Tool-Hang. We also observe more balanced gradient allocation, averaged across the six independently trained policies. These results further suggest that MSE-Policies are able to achieve better performance when the residual scales are more consistent across samples.
Q2. Does gradient reweighting support our optimization explanation?
A2. Yes. We have added a per-sample gradient reweighting experiment (Exp. B, Sec. 5.3) by fine-tuning GR00T N1.7 on RoboCasa-GR1. We establish this experiment by training an MSE-Policy and reweight each sample's gradient by the inverse of its norm. On RoboCasa-GR1, this intervention improves the success rate from 35.55% to 51.75%, achieving comparable performance to HT-Policy (50.01%).
A practical limitation of per-sample gradient reweighting is its high computational cost, as computing individual gradient norms makes training substantially slower. One possible solution is Ghost Clipping, which computes per-sample gradient norms without explicitly materializing individual gradients. For a linear layer, it exploits
= \langle A_i A_i^\top, B_i B_i^\top \rangle_F,$$ where $A_i$ and $B_i$ denote the input activations and back-propagated gradients. We leave more efficient implementations of gradient reweighting for future work. --- **Q3.** How about MIP's residual distribution and gradient allocation? **A3.** We thank the reviewer for pointing this out! We have added a residual and gradient analysis of MIP in Sec. 3 (Fig. 4). Same as Flow-Policy, we find that MIP also exhibits heavy-tailed action residuals. MIP's first step shows imbalanced gradient allocation similar to MSE regression, while the second step shows a more balanced distribution similar to low-noise flow matching. These findings suggest that MIP's denoising objective may also help mitigate the gradient imbalance associated with heavy-tailed action residuals, following a similar principle to Flow-Policies. --- **Q4.** Does Student-$t$ loss work in Flow-Policy training? **A4.** We have added an experiment in Sec. 5.5 (Exp. E, Fig. 13) to investigate whether Student-$t$ loss can also benefit Flow-Policies. As discussed in Sec. 3, at high-noise timesteps, the noisy action provides little information about the target action, leading to heavy-tailed residuals and imbalanced gradient allocation similar to MSE-Policies. We find that applying Student-$t$ loss at all timesteps hurts performance, whereas applying it only at high-noise timesteps ($t<0.5$) improves both success rate and training efficiency. These results suggest that our insights into residual modeling and gradient allocation can also guide the design of generative training objectives. --- **Q5.** Can HT-Policies be applied to pre-train or fine-tune regression-based VLAs? **A5.** Due to the compute and data constraint, we can't establish a regression-based VLA pre-training experiment with HT-Loss. Moreover, relatively few pretrained VLAs use direct action regression as their pretraining objective, making suitable pretrained checkpoints less available. The closest existing setting is OpenVLA-OFT [1], which fine-tunes an autoregressively pretrained VLA (OpenVLA) using a continuous action regression head. However, its reported simulation benchmark results focus on LIBERO, where performance is already near saturation (97.1% average success rate). In terms of applying HT-Loss to VLA training and fine-tuning, our paper already evaluates two settings: training action predictors from scratch and fine-tuning pretrained flow-based VLAs. These experiments cover the main practical scenarios of learning a new action predictor and adapting an existing pretrained action model. We would also appreciate the reviewer's suggestions on any additional evaluation settings they consider important, and would be happy to conduct further experiments where feasible. --- [1] Moo Jin Kim, et al. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. In RSS, 2025.