acceptodds
Under review as a conference paper at ICLR 2027

Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning

Abstract

Learning from demonstration has enabled impressive robot behaviors. A common choice for policy learning is to use diffusion or flow matching (Flow-Policies), which often outperforms direct action regression trained with mean squared error (MSE-Policies). This gap is commonly attributed to multimodal demonstrations. We revisit this gap from the perspective of statistical modeling: how action-prediction residuals shape policy optimization. Our analysis of real-world robot demonstration data reveals substantial state-dependent variation in residual scales and heavier-than-Gaussian tails. While both MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals, their training gradients behave differently: MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization. Motivated by these findings, we introduce heteroscedastic Student- action regression (HT-Policies), which learns input-dependent residual scales and reduces the influence of heavy tails. HT-Policies predict action chunks with a single feed-forward pass and can reuse pretrained flow-matching-based policy networks as the backbone. Across four simulation benchmarks and real-robot evaluations, HT-Policies achieves success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models, despite being faster in training and inference. Together, these findings shed light on the practical advantages of generative objectives in robot learning from demonstrations and offer an efficient direct-regression alternative for a range of architectures and tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot imitation learning, robust regression, heteroscedastic regression, heavy-tailed residuals, flow-matching

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rvs8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript studies the performance gap between regression and generative policies in imitation learning. The authors identify this performance gap is caused by the heteroscedastic and heavy-tailed action residuals, which makes the gradient concentrated on high-residual samples. They also propose HT-Policy to adaptively model residual scales and alliveate the effect of large errors. Experiments on simulations, pretrained VLA/WAM models, and real-world robotic tasks show that HT-Policies achieve performance comparable to generative policies while enabling single-step action prediction and faster inference.

Critical strengths
  • The authors proposed an interesting assumption that Flow Matching-based policies outperforms regression-based lies in residual modeling and gradient allocation, rather than in multi-modal modeling and provide quantified analysis.

  • The authors proposed HT-Policy and conducted extensive experiments to demonstrate that employing an appropriate robust regression loss can significantly improve the performance of direct action regression.

Critical weaknesses
  • Although the authors have observed the phenomenon of heavy-tailed action residuals and provides explanations from both statistical and optimization perspectives, the evidence linking these heavy-tailed action residuals to performance disparities remains somewhat insufficient. I would be very interested to see empirical evidence demonstrating, for instance, that in a task where the action residual distribution aligns with the assumptions underlying MSE optimization, an MSE-based approach could perform on par with a Flow-based model. Alternatively, it would be compelling to see whether identifying specific phases within a task, distinguishing between those with high residuals and those with low residuals, and employing separate neural networks to model each, leads to an improvement in performance.

  • The optimization mechanism proposed by the authors could be articulated more clearly. If the discrepancy between MSE and Flow is indeed attributable to gradient allocation, I would expect to see further evidence at the data level—for instance, by employing data sampling strategies or gradient reweighting techniques to align the gradient distribution as closely as possible with that produced by the HT policy, thereby validating the proposed optimization mechanism.

  • The empirical baselines used for comparison can be more sufficient. Although the authors criticize certain drawbacks of MIP in the introduction, it is regrettable that they have not included MIP in their comparative analysis. I would like to see how MIP performs in terms of action residual TALS and gradient allocation.

Questions for authors

Out of curiosity, I am very interested to see what would happen if the author used a "student t loss" to optimize the flow policy—that is, replacing the regression step at each stage of the flow policy with a student t loss. I wonder what the results might be, and I would appreciate an explanation of the potential outcome.

Did the authors try using HT policy in fine-tuning/pretraing regression-based VLAs such as OpenVLA? If HT-Policy can achieve performance comparable to that of flow-based VLAs and considering the inferencing speed-up it brings, it would be another great contribution.

If my raised concerned are properly addressed, I would be happy to raise my score.

3 comments

We thank the reviewer for these constructive suggestions. Since we cannot include figures directly in this response, we have presented all the corresponding experimental results and figures in the revised manuscript and summarize the key findings below.


Q1. Can MSE-Policies achieve competitive performance when residuals match MSE's assumption or with phase-specific training?

A1. For tasks where Flow-Policies and HT-Policies outperform MSE-Policies, we find it difficult to obtain datasets whose prediction residuals satisfy MSE's fixed-scale Gaussian assumption, even with smooth actions generated by scripted policies. Alternatively, we investigate phase-specific MSE training and find that it substantially improves performance on challenging tasks. Our newly added phase-specific MSE-Policy experiment (Exp. D, Sec. 5.5, Fig. 12) provides empirical evidence for this. As shown in Figure 12, this approach substantially improves the success rate on Tool-Hang. We also observe more balanced gradient allocation, averaged across the six independently trained policies. These results further suggest that MSE-Policies are able to achieve better performance when the residual scales are more consistent across samples.


Q2. Does gradient reweighting support our optimization explanation?

A2. Yes. We have added a per-sample gradient reweighting experiment (Exp. B, Sec. 5.3) by fine-tuning GR00T N1.7 on RoboCasa-GR1. We establish this experiment by training an MSE-Policy and reweight each sample's gradient by the inverse of its norm. On RoboCasa-GR1, this intervention improves the success rate from 35.55% to 51.75%, achieving comparable performance to HT-Policy (50.01%).

A practical limitation of per-sample gradient reweighting is its high computational cost, as computing individual gradient norms makes training substantially slower. One possible solution is Ghost Clipping, which computes per-sample gradient norms without explicitly materializing individual gradients. For a linear layer, it exploits

= \langle A_i A_i^\top, B_i B_i^\top \rangle_F,$$ where $A_i$ and $B_i$ denote the input activations and back-propagated gradients. We leave more efficient implementations of gradient reweighting for future work. --- **Q3.** How about MIP's residual distribution and gradient allocation? **A3.** We thank the reviewer for pointing this out! We have added a residual and gradient analysis of MIP in Sec. 3 (Fig. 4). Same as Flow-Policy, we find that MIP also exhibits heavy-tailed action residuals. MIP's first step shows imbalanced gradient allocation similar to MSE regression, while the second step shows a more balanced distribution similar to low-noise flow matching. These findings suggest that MIP's denoising objective may also help mitigate the gradient imbalance associated with heavy-tailed action residuals, following a similar principle to Flow-Policies. --- **Q4.** Does Student-$t$ loss work in Flow-Policy training? **A4.** We have added an experiment in Sec. 5.5 (Exp. E, Fig. 13) to investigate whether Student-$t$ loss can also benefit Flow-Policies. As discussed in Sec. 3, at high-noise timesteps, the noisy action provides little information about the target action, leading to heavy-tailed residuals and imbalanced gradient allocation similar to MSE-Policies. We find that applying Student-$t$ loss at all timesteps hurts performance, whereas applying it only at high-noise timesteps ($t<0.5$) improves both success rate and training efficiency. These results suggest that our insights into residual modeling and gradient allocation can also guide the design of generative training objectives. --- **Q5.** Can HT-Policies be applied to pre-train or fine-tune regression-based VLAs? **A5.** Due to the compute and data constraint, we can't establish a regression-based VLA pre-training experiment with HT-Loss. Moreover, relatively few pretrained VLAs use direct action regression as their pretraining objective, making suitable pretrained checkpoints less available. The closest existing setting is OpenVLA-OFT [1], which fine-tunes an autoregressively pretrained VLA (OpenVLA) using a continuous action regression head. However, its reported simulation benchmark results focus on LIBERO, where performance is already near saturation (97.1% average success rate). In terms of applying HT-Loss to VLA training and fine-tuning, our paper already evaluates two settings: training action predictors from scratch and fine-tuning pretrained flow-based VLAs. These experiments cover the main practical scenarios of learning a new action predictor and adapting an existing pretrained action model. We would also appreciate the reviewer's suggestions on any additional evaluation settings they consider important, and would be happy to conduct further experiments where feasible. --- [1] Moo Jin Kim, et al. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. In RSS, 2025.

I appreciate the authors' responses and the provided additional results, which have largely addressed my concerns. I have raised my rating score to reflect it.

We truly appreciate your valuable comments and the positive assessment of our work!

Reviewer_fZ7bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates the performance gap between direct regression and generative robot policies, proposing a Student-t regression objective with an observation-dependent scale to model heavy-tailed action residuals. Experiments across simulation and real-robot tasks, including fine-tuning pretrained robot foundation models, show improvements over MSE regression and competitive performance with generative baselines using single-pass action prediction.

Critical strengths
  1. The residual and gradient analyses offer a useful empirical perspective on why standard MSE regression can underperform generative policies.

  2. The method is simple and practical, enabling single-pass action prediction while reusing pretrained flow policies through fine-tuning.

  3. The experiments cover multiple architectures, simulation benchmarks, and real-robot tasks, showing improvements over MSE and generally competitive performance with generative baselines.

Critical weaknesses
  1. The methodological novelty is limited, as the objective combines established Student-t regression with input-dependent scale learning, although the residual diagnostics and adaptation recipe provide useful contributions. The choice to share one scale across the entire action chunk is also not compared with finer-grained alternatives, leaving its modeling trade-off unclear.

  2. The experiments with pretrained robot foundation models demonstrate downstream conversion of flow-pretrained policies, but do not evaluate HT itself as a large-scale pretraining objective or test whether conversion preserves pretrained capabilities beyond the fine-tuning tasks. Its broader value for training generalist policies therefore remains unclear.

  3. The gains over flow are mixed: HT improves substantially on GR1 but drops from 67.83% to 65.25% on Fractal, while the smaller gains on Bridge and Cosmos lack significance tests. Straight Flow (1-NFE) and MIP (two-pass) appear only as borrowed RoboMimic/Push-T results; the robot foundation model latency comparisons use multi-step flow baselines. The performance–latency trade-off against dedicated one-step generative policies remains unclear.

Questions for authors
  1. How does HT relate to residual-dependent loss reweighting in generative modeling, such as MeanFlow (Geng et al., 2025)? Please clarify what is specific to the proposed formulation beyond established robust regression, as distinct from the empirical findings in robotic control.

  2. Table 27 compares fixed-scale Student-t with HT. Would a Student-t policy with one globally learned scale achieve similar gains under the same training setup? This would help isolate the benefit of observation-dependent scales from global scale adaptation during training; the post-hoc calibration in A.7.2 does not test this distinction.

4 comments

We thank the reviewer for their and constructive feedback. We have conducted additional experiments and analyses to address their concerns, as detailed below.


Q1. What is the novelty of this work, and how does it compare with prior work?

A1. [Scientific and Practical Contribution] We recognize that Student-tt regression and input-dependent scale modeling are well-established techniques, and we do not claim novelty in the regression objective itself. Instead, the motivation of this paper is to answer a scientific question: why do direct action-regression policies often underperform generative policies?

As Reviewer iYKh summarized, we find that generative objectives may implicitly handle heavy-tailed residuals through iterative computation and noise injection, while HT-Policy shows that addressing them explicitly can achieve comparable performance. This explanation is supported by our analysis of residual distributions and gradient allocation across different training objectives. To the best of our knowledge, HT-Policy is the first work to propose this explanation through the statistical modeling of prediction residuals. It is also the first work to propose HT-Policy as an efficient and performant method for behavior cloning. Overall, our findings provide valuable scientific insights and useful practice beyond the Student-tt regression objective as a machine learning technique itself.

[Empirical Contribution] HT-Policies provide a simple and efficient approach to fine-tuning pretrained robot foundation models. We argue that effective fine-tuning and post-training methods can have substantial practical impact, as demonstrated by works such as LoRA [1] and DPO [2] and OpenVLA-OFT [3]. Our experiments demonstrate that HT-Policies can achieve competitive performance when fine-tuning pretrained VLA and WAM models with higher training and inference efficiency. These results support our methodological contribution for practitioners.

[Relation to MeanFlow] Thanks for pointing out this connection! We will include these discussions in the updated manuscript. First, MeanFlow and HT-Policies use different training objectives. MeanFlow learns an average velocity field through a self-referential training target, while HT-Policies explicitly model prediction residuals using a heteroscedastic Student-tt likelihood. We argue these two works focus on different research questions: MeanFlow focuses on one-step generative modeling which leverages the reweighting loss for better performance, while our work investigates the role of residual modeling in explaining the performance gap between regression and generative policies.

Second, our findings are also consistent with MeanFlow. Here, we analyze MeanFlow's training residuals and find that they also exhibit heavy-tailed distribution. This provides a statistical motivation for MeanFlow's adaptive reweighting loss. It also suggests that our hypothesis may apply to other domains, such as image generation, when integrated with methods such as MeanFlow.

ThresholdMeanFlow ResidualsGaussian ReferenceRatio
>3σ>3\sigma0.7704%0.2700%2.85×
>4σ>4\sigma0.4653%0.006334%73.45×
>5σ>5\sigma0.3085%0.00005733%5,381×

Third, beyond analyzing residuals in MeanFlow, we investigate whether our insights can improve generative training objectives. Due to computational constraints, we have not conducted large-scale MeanFlow image-generation experiments. However, our newly added experiment (Exp. E, Fig. 13) shows that applying Student-tt loss to Flow-Policies at high-noise timesteps improves both success rates and training efficiency. This further demonstrates that our findings can also guide the design of generative objectives.


Q2. How does scale modeling influence the performance?

A2. We thank the reviewer for raising up this question. We have added a scale modeling ablation (Exp. C, Sec. 5.5, Table 4), comparing four Student-tt regression settings: fixed scale, learnable global scale, observation-dependent chunk-wise scale, and observation-dependent element-wise scale. We find that a global scale only yields a marginal improvement over a fixed scale, whereas a chunk-wise scale achieves the highest success rate. This highlights the importance of allowing the scale to vary across samples. We also find that element-wise scale performs considerably worse than chunk-wise, showing that finer-grained scale modeling does not necessarily improve performance. This further supports our design of predicting a chunk-wise scale for each sample in HT-Policies.

Scale ModelingSuccess Rate (%)
Fixed scale39.49
Learnable global scale40.69
Chunk-wise scale (HT)50.01
Element-wise scale43.82

Q3. Significance Test

A3. Thanks for the suggestion. We have conducted statistical significance tests for both comparisons. The result show that the significance of HT-Policy and Flow-Policy relative to MSE-Policy are largely aligned. They both significant improve on GR00T/RoboCasa-GR1, GR00T/Bridge, and GR00T/Fractal, while not significant on Cosmos/LIBERO. The only exception is π0.5\pi_{0.5}/LIBERO where Flow-Policy shows a small but significant gain over MSE (+0.37), while HT-Policy does not (+0.25).

Model / BenchmarkMSE (%)Flow (%)HT (%)Flow − MSE: Δpp (p-value)HT − MSE: Δpp (p-value)
GR00T / GR135.55 ± 0.6742.34 ± 0.6850.01 ± 0.51+6.79 (<0.001)+14.46 (<0.001)
GR00T / Bridge58.51 ± 0.3762.06 ± 0.4062.31 ± 0.65+3.55 (<0.001)+3.80 (0.0014)
GR00T / Fractal60.60 ± 0.4367.83 ± 0.3965.25 ± 0.44+7.23 (<0.001)+4.65 (<0.001)
π0.5\pi_{0.5} / LIBERO96.72 ± 0.0997.09 ± 0.0496.97 ± 0.07+0.37 (0.0181)+0.25 (0.2128)
Cosmos / LIBERO-1095.10 ± 0.2695.60 ± 0.9096.50 ± 0.45+0.50 (1.0000)+1.40 (0.1028)

We would like to clarify that our goal is not to consistently outperform Flow-Policies, but to close the performance gap between MSE regression and generative policies without iterative computation. Therefore, the key comparisons are between MSE-Policy and Flow-Policy, which establish the original performance gap, and between MSE-Policy and HT-Policy, which evaluate how much of this gap can be closed through HT-Loss.


Q4. Is HT-Loss a good pre-training objective? Does HT-Policy preserve the pre-trained ability?

A4. Due to compute and data constraints, we can not pre-train a large robot model with HT-Loss. To investigate the pre-trained capability preservation, we fine-tune GR00T N1.7 on RoboCasa-GR1 using Flow and HT objectives, and evaluate their action-prediction errors on the DROID test set. Although both fine-tuned models have higher evaluation errors than the pretrained base model, HT achieves substantially lower errors than Flow.

GR00T N1.7 ModelMSEMAE
pre-trained base model0.03860.0986
RoboCasa-GR1 Flow (60k steps)0.69970.6581
RoboCasa-GR1 HT (60k steps)0.14490.2427

Q5. How does HT-Policy compare with 1 NFE baselines? What about the performance/latency tradeoff?

A5. We establish the comparison with Flow-Policy (1 NFE) and MIP using only its first inference step. HT-Policy achieves the best performance among all the 1 NFE methods. In terms of the performance–latency trade-off, two-step MIP is the closest competitor, achieving comparable performance (48.81% vs. 50.01%) but requiring 1.27× the total inference latency of HT-Policies (33.63 ms vs. 26.44 ms).

MethodSR (%)NFEAction Head (ms)Total Latency (ms)
Flow (4-step)42.34429.7147.94
Flow (1-step)37.4218.0925.95
MIP (2-step)48.81215.4633.63
MIP (1-step)40.0518.1026.17
HT (ours)50.0118.1126.44

[1] Edward J. Hu, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.

[2] Rafael Rafailov, et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In NeurIPS, 2023.

[3] Moo Jin Kim, et al. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. In RSS, 2025.

Thank you for the detailed response and revisions. The added scale ablation and inference-time comparisons address my main concerns. I am satisfied with the revisions and now lean toward acceptance.

We sincerely thank you for your constructive suggestions and positive assessment! We are glad that our responses have addressed your main concerns.

Reviewer_iYKhOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Generative models, and in particular diffusion and flow matching models, have become the de facto choice for imitation learning in robotics. Their superiority over MSE-regression model had been attributed to how they can better handle multimodality in data. The authors revisit a more recent result attributing the success of flow models for imitation to iterative refinement, and through the lens of statistical modeling, put forth a non-iterative (single forward pass) direct action regression method exhibiting competitive performance. They evaluate their approach, called HT-Policies, extensively: training from scratch and fine-tuning various models on various task types. The paper makes a meaningful step toward figuring out what makes generative models so effective in learning from demonstrations, and proposes a new action-head modeling approach that allows for lower training and inference costs while preserving performance.

Critical strengths
  • The introduction is clear and complete, giving a good first overview of what is done in the paper.
  • The authors consider a gap left open by Minimal Iterative Policy (MIP) and set out to close it by having a closer look at the residuals of MSE-based policies, with the objective of determining how the residuals affect policy learning. The observations showcased by MIP were telling already; deepening their analyses with statistical modeling is a great contribution.
  • The related work section is compact for seems to cover enough of the neighboring literature.
  • The paper is very easy to read, the argument chain is fluid.
  • The experimental results are great: matching FM-based policies on one benchmark, and surpassing them in another one, while outperforming the MSE-based policies. HT-Policies even match MIP. Comment: I think the authors should write that HT-Policies match MIP in performance in the main paper briefly, before sending the reader to the appendices.
  • The paper makes a valuable step toward furthering the understanding of what makes diffusion and flow-based policies so successful at robotics imitation learning. With the likes of MIP, I believe this to be a great contribution to the community.
  • Evaluation is done on 10 random seeds.
  • The “Residual modeling and policy performance” study is very valuable and scrutinizes various aspect of the HT-Policies’ design.
  • The training and inference efficiency over iterative methods is a valuable practical advantage.
  • The fact that ν=4\nu=4 is the maximum across the three datasets, along with the regularity of the curves, points toward HT-Policies not requiring extensive per-dataset hyper-parameter tuning (for practitioners who want to use the method).
Critical weaknesses
  • L78-79: the authors should be clearer as to what the cited reference makes reference to; the base on which the authors build on? Do the authors resurface a pre-existing method as-is? etc. Clarifying the boundary is important.
  • It is unclear why the sigma learned by HT-Policies constitutes a salient way to conduct exploration (Appendix A.3).
  • L428: “in two training-from-scratch and two fine-tuning settings”. Remind the reader which are which in the caption.
  • (Minor) I think it would be valuable to add, in the conclusion, how the authors’ conclusion conciliates/clashes with the conclusions of MIP.
  • (The authors recognize the limitation that) HT-Policies can’t explicitly represent multiple action modes.
  • L268 and L279–280: Consider using 0\mathbf{0} to explicitly denote the zero vector supplied as the action input.
Questions for authors
  • About the exploration study (Appendix A.3): σθ(o)\sigma_{\theta}(o) is trained to model the scale of demonstration residuals (via the HT loss), not to maximize exploration or information gain. There is no automatic reason to expect the resulting scale to be optimal for exploration. The scale of demonstration residuals is not necessarily the same thing as the amount of exploration that would be beneficial in a particular state. Could the authors clarify what conclusions can be drawn about the usefulness of the learned (\sigma(o)) for exploration specifically?
  • MIP’s conclusion is that 2 things are important to match flow models: stochasticity (noise injection) in combination with iterative computation. Conversely, when it comes to HT-Policies, the authors conclude that the key is to handle network prediction residuals smartly, showing that their non-iterative method matches flow models. In aggregate, is therefore seems that, based on the analyses of the paper, that an iterative refinement/computation objective (not necessarily flows or diffusion) has the desired (or side-) effect of allowing numerically graceful handling of heavy tails; HT-Policies do that purposely and and explicitly, and as a result achieve on-par performances while not suffering the heavy training and inference costs caused by the iterative nature of Flow/Diffusion/MIP models. Is that a fair assessment of the aggregate conclusion?
  • In Section 3, the authors motivate the light-tailed behavior of flow-matching velocity residuals by constructing a predictor whose residual depends only on Gaussian noise. However, the existence of such a predictor does not necessarily imply that minimizing the flow-matching objective will lead to similarly light-tailed residuals. Could the authors elaborate on the mechanism by which flow-matching optimization encourages this behavior, beyond the empirical observations in Appendix A.1.3? Are additional assumptions required for the proposed explanation to hold?
2 comments

We sincerely thank the reviewer for their positive assessment of our work and for the thoughtful comments and suggestions. We have addressed the presentation issues in the revision and we respond to the questions below.


Q1. Does the learned σ(o)\sigma(o) help for exploration?

A1. We agree that the learned σ(o)\sigma(o) models demonstration residual scales rather than optimal exploration noise. Also, we do not claim that the learned σ(o)\sigma(o) provides better exploration than a fixed scale. As shown in Appendix A.3, the two approaches achieve comparable pass@kk performance under moderate noise levels. The main exception is α=2\alpha=2, where the learned σ(o)\sigma(o) achieves higher pass@1, pass@2, and pass@4. One possible explanation is that demonstration residual scales may correlate with action magnitudes, allowing the learned σ(o)\sigma(o) to reduce exploration noise for precise behaviors.

Our primary goal in Appendix A.3 is to demonstrate that HT-Policies can support stochastic exploration with pass@kk performance comparable to Flow-Policies. We believe this property makes HT-Policies promising for RL post-training. Using IL-based policies as initialization for RL is a common practice, but MSE-Policies often struggle to achieve sufficiently high initial success rates on challenging tasks, making subsequent RL exploration difficult. Generative policies generally provide stronger initial performance, but they introduce additional complexity in RL post-training. For example, diffusion-based approaches such as DPPO [1] formulate denoising and environment interaction as a two-layer MDP, extending the effective horizon, while flow-based approaches such as FPO [2] rely on surrogate likelihood ratios derived from the flow-matching loss rather than exact likelihood computation.

In contrast, HT-Policies provide three desirable properties for RL post-training: (1) a simple regression-based formulation with tractable likelihood evaluation, (2) competitive imitation-learning performance, and (3) strong stochastic exploration capabilities. Together, these properties suggest that HT-Policies could serve as an effective starting point for RL post-training. In this work, we only show the stochastic exploration capability of HT-Policies and leave a systematic RL post-training study for future work.


Q2. Main Conclusions of HT-Policy and Relation to MIP.

A2. We thank the reviewer for this insightful summary! This is a fair assessment of our main conclusions. While MIP emphasizes the importance of iterative computation and noise injection, our analysis provides a statistical explanation based on residual modeling and gradient allocation. We have clarified this relationship in the revised conclusion.


Q3. Why flow matching encourages light-tailed residuals?

A3. We recognize that the existence of a predictor with Gaussian residuals does not guarantee that gradient-based optimization will converge to it. Our analysis provides a motivation for why flow matching can mitigate heavy-tailed residuals. We show that in the low-noise regimes of flow-matching, the noisy actions contain enough information about the target action, allowing a simple predictor to achieve residuals that depend only on Gaussian noise.

For now, we can only provide the empirical evidence in Appendix A.1.3 and Appendix A.1.4. We show that the gradient allocations of low-noise regimes for Flow-Policies are more balanced across different benchmarks and training stages. We believe these results provide strong empirical support for our explanation. A rigorous analysis of flow-matching's training dynamics is left for future work, since it requires more work on the analysis of the optimization process of flow-matching networks.


Q4. Reference Clarification and Presentation Revisions.

A4. We thank the reviewer for pointing out these issues. We have addressed them in the revised manuscript.


[1] Allen Z. Ren, et al. Diffusion Policy Policy Optimization. In ICLR, 2025.

[2] David McAllister, et al. Flow Matching Policy Gradients. In ICLR, 2026.

As the rebuttal period comes to an end, we would like to thank you for your insightful summary of our work and constructive suggestions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their time and effort in reading our work and providing valuable comments. In response to their questions and suggestions, we have conducted additional experiments and improved the presentation for clarity in the revised manuscript. All revisions are highlighted in blue.

As Reviewer iYKh summarized, our findings suggest that generative objectives may implicitly handle heavy-tailed prediction residuals, while HT-Policies address them explicitly without iterative computation. We further suggest that differences in residual modeling and gradient allocation provide an intuitive explanation for the performance gap between regression and generative policies. To strengthen our paper and respond to the reviewers' questions, we have added the following experiments to the revised manuscript.

IndexExperimentLocation
AResidual and gradient analysis of MIPSec. 3, Fig. 4
BMSE with per-sample gradient reweightingSec. 5.3
CAblation study of scale modelingSec. 5.5, Table 4
DPhase-specific MSE-PoliciesSec. 5.5, Fig. 12
EStudent-tt loss for Flow-PoliciesSec. 5.5, Fig. 13

These experiments strengthen our paper in three aspects: (1) providing stronger empirical evidence for our proposed explanation (Exps. A, B, and D); (2) offering more detailed ablations to clarify the design choices of HT-Policies (Exp. C); and (3) exploring broader applications of our insights (Exp. E). To address the reviewers' remaining questions, we have also conducted additional experiments that are not yet included in the revised manuscript. We discuss these results in the responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.