- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Denoising-Variance Credit Assignment (DVCA), a method that leverages action-level denoising variance during the Flow Matching inference process to improve credit assignment in online post-training. DVCA uses denoising variance as a proxy for action uncertainty, prioritizing the learning of actions with higher uncertainty by reweighting the advantage function or supervised loss without training additional auxiliary models or incurring extra inference calls. The proposed method can be flexibly integrated into various flow-based robot policies and online post-training frameworks. The authors conduct extensive validations in both simulation and real-world robot environments across multiple policy models and post-training methods, and experimental results demonstrate that DVCA improves task success rates or sample efficiency under the tested settings.
Critical strengths
None
Critical weaknesses
None
Questions for authors
-
The paper uses Denoising Variance as a proxy for action uncertainty and reallocates training weights accordingly. Although Appendix A.5.3 demonstrates the association between high-variance actions and manipulation phases such as grasping and alignment through qualitative visualizations, the instability of denoising predictions does not necessarily imply that the corresponding actions have a greater impact on the final task return. The authors should further validate the relationship between Denoising Variance and the task importance of actions through quantitative experiments, such as analyzing how perturbations to different actions affect the task success rate or Q-value.
-
DVCA measures action uncertainty via the variance of clean action estimates over the final few denoising steps. According to Appendix A.2.2 and A.3.1, the main experiments use 10 denoising steps and a variance estimation window of length 5. Could the authors further analyze whether this uncertainty metric and its resulting post-training gains remain stable across different numbers of denoising steps, variance estimation window lengths, or integration settings?
5 comments
About Sensitivity to Denoising and Integration Settings (Question 2) (1/2)
Thank you for suggesting a closer examination of the denoising settings. The additional experiments in Appendix A.1.7 (Table 12) and A.1.8 (Table 14) cover different tail lengths, direct few-step inference, and distilled policies. DVCA consistently improves GRPO across these settings. We further analyze post-training gains with different solvers and the stability of the denoising variance (DV) signal in Appendix A.1.12 (Tables 22–23 and Figure 13), with DV rank correlations of 0.933–0.992 and post-training gains under both solvers. We provide the details below.
Sensitivity to Denoising and Integration Settings. We examine the effects of the number of denoising steps N, the denoising tail length L, and the numerical solver on both the denoising-variance signal and post-training gains.
Denoising steps and tail length. Table 14 in Appendix A.1.8 compares policies with 10, 5, and 3 denoising steps. For direct 5-step and 3-step inference, DVCA improves the four-task average success rate of π0.5 with GRPO by 3.9 and 3.8 percentage points, respectively. The corresponding gains for distilled policies are 3.4 and 3.3 percentage points. We use . Table 12 in Appendix A.1.7 varies L over {3,5,7} with N=10 and τ=2.5. Success rates on the three held-out tasks vary by at most 1.1 percentage points across these tail lengths, and all settings improve over GRPO. These results show that DVCA retains its post-training benefits across the tested step counts and tail lengths.
Summary of the four-task results in Table 14. Success rates are in percent; gains are relative to GRPO in percentage points.
| Setting | N | GRPO | + DVCA | Gain (pp) | |
|---|---|---|---|---|---|
| Original policy | 10 | 5 | 79.4 | 82.8 | +3.4 |
| Direct few-step inference | 5 | 5 | 72.0 | 75.9 | +3.9 |
| Direct few-step inference | 3 | 3 | 68.8 | 72.6 | +3.8 |
| Few-step distillation | 5 | 5 | 71.4 | 74.8 | +3.4 |
| Few-step distillation | 3 | 3 | 67.7 | 71.0 | +3.3 |
Tail-length comparison from Table 12. N=10 and τ=2.5. Pick Diverse Bottles is the validation task; the remaining three tasks are held out for hyperparameter evaluation.
| Method | L | Pick Diverse Bottles | Turn Switch | Move Stapler Pad | Move Can Pot |
|---|---|---|---|---|---|
| GRPO | — | 91.2 | 78.4 | 86.5 | 91.0 |
| + DVCA | 5 | 95.9 | 82.5 | 89.1 | 94.1 |
| + DVCA | 3 | 95.2 | 81.8 | 88.6 | 93.4 |
| + DVCA | 7 | 95.5 | 81.6 | 88.8 | 93.0 |
Post-training with different solvers. We post-train π0.5 with Filter IL on four RoboTwin 2.0 tasks using Euler and Heun, and compare success rates with and without DVCA. All runs start from the same initial policy and use the corresponding solver for both online rollouts and evaluation, with N=10, L=5, and a fixed time grid. All other settings follow the main experiments. Table 22 shows consistent gains with both Euler and Heun. With Heun, DVCA improves success rates by 3.6, 10.4, 5.0, and 6.0 percentage points across the four tasks, averaging 6.3 percentage points.
Table 22. Success rates (%) of π0.5 with Filter IL using Euler and Heun. Online rollouts and evaluation use the corresponding solver, with N=10 and L=5.
| ODE solver | Method | Turn Switch | Place Mouse Pad | Move Stapler Pad | Move Can Pot |
|---|---|---|---|---|---|
| Euler | π0.5 | 54.9 | 39.0 | 42.8 | 55.7 |
| Euler | + Filter IL | 87.4 | 65.0 | 81.5 | 69.6 |
| Euler | w/ DVCA | 89.3 | 73.3 | 82.9 | 72.4 |
| Heun | π0.5 | 56.6 | 42.1 | 45.7 | 58.1 |
| Heun | + Filter IL | 90.1 | 68.0 | 84.6 | 71.2 |
| Heun | w/ DVCA | 93.7 | 78.4 | 89.6 | 77.2 |
Stability of action-level uncertainty. To assess the sensitivity of DV to denoising settings, we evaluate the same π0.5 checkpoint on a fixed set of observations while varying the step count, tail length, and solver. We reuse the initial noise across configurations. At the start of each integration step, we compute one clean-action estimate from the current state and velocity prediction; the second velocity evaluation in Heun corrects the update. Thus, 10 Heun steps require 20 network evaluations, and DV with L=5 uses estimates from the final five integration steps. We compute Spearman correlations between each configuration and the default configuration over action-level DV scores within each action chunk, then average over paired action chunks. Table 23 reports correlations of 0.960 and 0.956 when varying the step count, 0.987 and 0.992 when varying the tail length, and 0.933 with Heun, indicating largely consistent action-level uncertainty rankings. Figure 13 shows action-level DV under Euler and Heun on RoboTwin. The two sets of scores are highly correlated.
About Sensitivity to Denoising and Integration Settings (Question 2) (2/2)
Table 23. Spearman correlations of action-level DV scores with the default configuration (N=10, L=5, Euler).
| N | L | ODE solver | Spearman correlation with default DV |
|---|---|---|---|
| 10 | 5 | Euler | 1.000 |
| 5 | 5 | Euler | 0.960 |
| 3 | 3 | Euler | 0.956 |
| 10 | 3 | Euler | 0.987 |
| 10 | 7 | Euler | 0.992 |
| 10 | 5 | Heun | 0.933 |
Training objective and extensions. We use Filter IL, adding noise to actions from successful trajectories and predicting the corresponding velocity under the same flow-matching objective. Changing the solver affects the collected actions and DVCA weights, while the form of the training objective remains unchanged. Extending this comparison to GRPO requires coordinating higher-order ODE updates, stochastic denoising steps, and the transition probabilities used in policy optimization, making the implementation more involved. Combining DVCA with hybrid ODE–SDE GRPO using higher-order integration is a promising direction for future work.
We thank the reviewer for prompting this additional analysis of the DV signal and post-training performance across denoising settings.
About Denoising Variance and Action Importance (Question 1)
Thank you for raising this important question. We add two quantitative analyses in Appendix A.1.13 (Tables 24–25). Denoising variance (DV) is positively correlated with resampling disagreement across four tasks (ρ=0.593–0.750). With the same noise magnitude and a target perturbation rate of 20%, perturbing high-DV actions reduces the average success rate by 12.6 percentage points, compared with 5.4 and 2.6 points for random and low-DV perturbations. These results support a link between DV and the impact of actions on task outcomes. We provide the details below.
Denoising Variance and Action Importance. We analyze denoising variance (DV) through resampling disagreement and action perturbations, using a fixed π0.5 policy on four RoboTwin 2.0 tasks: Turn Switch, Place Mouse Pad, Move Stapler Pad, and Move Can Pot. We follow the main experiments with 10 denoising steps and action estimates from the final five steps.
Resampling disagreement. Following prior work that evaluates uncertainty proxies against resampling disagreement, we generate actions from 32 independent initial noise samples for each fixed observation. At each action position, we compute the mean pairwise L2 distance between the final action samples as a measure of resampling disagreement. We align this measure with action-level DV from a single denoising trajectory and compute Spearman rank correlations over observations and action positions within each task. Actions use the policy's original normalization, and the analysis covers the executed action window. [1]
Table 24 reports positive correlations ranging from 0.593 to 0.750 across all four tasks, showing that actions with higher DV tend to exhibit greater resampling disagreement.
Table 24. Spearman correlations between action-level denoising variance (DV) and resampling disagreement for π0.5 on four RoboTwin 2.0 tasks, using 32 initial noise samples per observation.
Sensitivity to action perturbations. For each task, we collect 128 unperturbed rollouts and calibrate low- and high-DV thresholds at the 20th and 80th percentiles of DV over executed actions. We fix these thresholds for evaluation, perturbing actions above the 80th percentile, below the 20th percentile, or selected at random with probability 0.2. All three conditions use additive uniform noise, sampled independently from [−0.01,0.01] for each continuous joint dimension in the policy's normalized action space; gripper commands remain unchanged. We evaluate 1,000 episodes per condition on each task and compare success rates with the unperturbed control. [2]
Table 25. Success rates (%) of π0.5 under action perturbations selected by denoising variance (DV). All perturbation conditions use uniform noise bounded by ±0.01 and a target perturbation rate of 20%. Avg. denotes the four-task mean.
In Table 25, perturbing high-DV actions reduces the four-task average success rate from 48.1% to 35.5%, a drop of 12.6 percentage points, compared with drops of 5.4 and 2.6 points for random and low-DV perturbations, respectively. This ordering holds across all four tasks, indicating that high-DV actions are more sensitive to execution errors and have a greater impact on task success.
Together, these results link denoising variance to both resampling disagreement and the effect of action errors on task success, providing quantitative support for using denoising variance for action-level reweighting during post-training.
We thank the reviewer for suggesting this direct evaluation of the link between denoising variance and task outcomes.
References
[1] Rao et al. The Geometry of Flow-Matching Uncertainty: A Cost-free Uncertainty Proxy and Its Application in Flow-based VLA Failure Detection. 2026.
[2] Ciftci et al. Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning. CoRL 2026.