acceptodds
Under review as a conference paper at ICLR 2027

DVCA: Denoising-Variance Credit Assignment for Flow-Based Robot Policy Optimization

Abstract

Recent Vision-Language-Action (VLA) models and World-Action Models (WAMs) use flow matching to model complex continuous action distributions. Online post-training can further improve these policies, but trajectory-level feedback offers limited guidance for prioritizing actions during training, limiting sample efficiency and performance gains. Process reward models and critics provide finer-grained feedback at the cost of additional training. To address this challenge, we introduce Denoising-Variance Credit Assignment (DVCA), which exploits the policy's denoising process for action-level credit assignment. Denoising variance, defined as the variance of clean-action estimates across denoising steps, serves as an intrinsic proxy for action uncertainty. Our analysis reveals that high denoising variance is concentrated in a small subset of actions associated with key phases of task execution. This observation motivates reweighting advantages or supervised losses to prioritize high-uncertainty actions. DVCA reuses intermediate predictions generated during online rollouts to estimate action uncertainty without additional training or inference. DVCA is a plug-and-play addition to existing post-training methods (e.g., GRPO and RL Token) across flow-based policies (e.g., and Fast-WAM). Simulation and real-world experiments demonstrate consistent gains in sample efficiency or task success rate over baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Manipulation, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YeZSOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Denoising-Variance Credit Assignment (DVCA), a method that leverages action-level denoising variance during the Flow Matching inference process to improve credit assignment in online post-training. DVCA uses denoising variance as a proxy for action uncertainty, prioritizing the learning of actions with higher uncertainty by reweighting the advantage function or supervised loss without training additional auxiliary models or incurring extra inference calls. The proposed method can be flexibly integrated into various flow-based robot policies and online post-training frameworks. The authors conduct extensive validations in both simulation and real-world robot environments across multiple policy models and post-training methods, and experimental results demonstrate that DVCA improves task success rates or sample efficiency under the tested settings.

Critical strengths

None

Critical weaknesses

None

Questions for authors
  1. The paper uses Denoising Variance as a proxy for action uncertainty and reallocates training weights accordingly. Although Appendix A.5.3 demonstrates the association between high-variance actions and manipulation phases such as grasping and alignment through qualitative visualizations, the instability of denoising predictions does not necessarily imply that the corresponding actions have a greater impact on the final task return. The authors should further validate the relationship between Denoising Variance and the task importance of actions through quantitative experiments, such as analyzing how perturbations to different actions affect the task success rate or Q-value.

  2. DVCA measures action uncertainty via the variance of clean action estimates over the final few denoising steps. According to Appendix A.2.2 and A.3.1, the main experiments use 10 denoising steps and a variance estimation window of length 5. Could the authors further analyze whether this uncertainty metric and its resulting post-training gains remain stable across different numbers of denoising steps, variance estimation window lengths, or integration settings?

5 comments
AuthorsOpenReview ↗

About Denoising Variance and Action Importance (Question 1)

Thank you for raising this important question. We add two quantitative analyses in Appendix A.1.13 (Tables 24–25). Denoising variance (DV) is positively correlated with resampling disagreement across four tasks (ρ=0.593–0.750). With the same noise magnitude and a target perturbation rate of 20%, perturbing high-DV actions reduces the average success rate by 12.6 percentage points, compared with 5.4 and 2.6 points for random and low-DV perturbations. These results support a link between DV and the impact of actions on task outcomes. We provide the details below.

Denoising Variance and Action Importance. We analyze denoising variance (DV) through resampling disagreement and action perturbations, using a fixed π0.5 policy on four RoboTwin 2.0 tasks: Turn Switch, Place Mouse Pad, Move Stapler Pad, and Move Can Pot. We follow the main experiments with 10 denoising steps and action estimates from the final five steps.

Resampling disagreement. Following prior work that evaluates uncertainty proxies against resampling disagreement, we generate actions from 32 independent initial noise samples for each fixed observation. At each action position, we compute the mean pairwise L2L_2 distance between the final action samples as a measure of resampling disagreement. We align this measure with action-level DV from a single denoising trajectory and compute Spearman rank correlations over observations and action positions within each task. Actions use the policy's original normalization, and the analysis covers the executed action window. [1]

Table 24 reports positive correlations ranging from 0.593 to 0.750 across all four tasks, showing that actions with higher DV tend to exhibit greater resampling disagreement.

Table 24. Spearman correlations between action-level denoising variance (DV) and resampling disagreement for π0.5 on four RoboTwin 2.0 tasks, using 32 initial noise samples per observation.

MetricTurn SwitchPlace Mouse PadMove Stapler PadMove Can Pot
Spearman ρ0.5930.6720.7500.694

Sensitivity to action perturbations. For each task, we collect 128 unperturbed rollouts and calibrate low- and high-DV thresholds at the 20th and 80th percentiles of DV over executed actions. We fix these thresholds for evaluation, perturbing actions above the 80th percentile, below the 20th percentile, or selected at random with probability 0.2. All three conditions use additive uniform noise, sampled independently from [−0.01,0.01] for each continuous joint dimension in the policy's normalized action space; gripper commands remain unchanged. We evaluate 1,000 episodes per condition on each task and compare success rates with the unperturbed control. [2]

Table 25. Success rates (%) of π0.5 under action perturbations selected by denoising variance (DV). All perturbation conditions use uniform noise bounded by ±0.01 and a target perturbation rate of 20%. Avg. denotes the four-task mean.

Perturbed actionsTurn SwitchPlace Mouse PadMove Stapler PadMove Can PotAvg.
None54.939.042.855.748.1
High DV41.328.530.142.035.5
Random48.337.135.949.442.7
Low DV51.038.140.652.245.5

In Table 25, perturbing high-DV actions reduces the four-task average success rate from 48.1% to 35.5%, a drop of 12.6 percentage points, compared with drops of 5.4 and 2.6 points for random and low-DV perturbations, respectively. This ordering holds across all four tasks, indicating that high-DV actions are more sensitive to execution errors and have a greater impact on task success.

Together, these results link denoising variance to both resampling disagreement and the effect of action errors on task success, providing quantitative support for using denoising variance for action-level reweighting during post-training.

We thank the reviewer for suggesting this direct evaluation of the link between denoising variance and task outcomes.

References

[1] Rao et al. The Geometry of Flow-Matching Uncertainty: A Cost-free Uncertainty Proxy and Its Application in Flow-based VLA Failure Detection. 2026.

[2] Ciftci et al. Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning. CoRL 2026.

AuthorsOpenReview ↗

About Sensitivity to Denoising and Integration Settings (Question 2) (1/2)

Thank you for suggesting a closer examination of the denoising settings. The additional experiments in Appendix A.1.7 (Table 12) and A.1.8 (Table 14) cover different tail lengths, direct few-step inference, and distilled policies. DVCA consistently improves GRPO across these settings. We further analyze post-training gains with different solvers and the stability of the denoising variance (DV) signal in Appendix A.1.12 (Tables 22–23 and Figure 13), with DV rank correlations of 0.933–0.992 and post-training gains under both solvers. We provide the details below.

Sensitivity to Denoising and Integration Settings. We examine the effects of the number of denoising steps N, the denoising tail length L, and the numerical solver on both the denoising-variance signal and post-training gains.

Denoising steps and tail length. Table 14 in Appendix A.1.8 compares policies with 10, 5, and 3 denoising steps. For direct 5-step and 3-step inference, DVCA improves the four-task average success rate of π0.5 with GRPO by 3.9 and 3.8 percentage points, respectively. The corresponding gains for distilled policies are 3.4 and 3.3 percentage points. We use Leff=min⁡(5,N)L_{\mathrm{eff}}=\min(5,N). Table 12 in Appendix A.1.7 varies L over {3,5,7} with N=10 and τ=2.5. Success rates on the three held-out tasks vary by at most 1.1 percentage points across these tail lengths, and all settings improve over GRPO. These results show that DVCA retains its post-training benefits across the tested step counts and tail lengths.

Summary of the four-task results in Table 14. Success rates are in percent; gains are relative to GRPO in percentage points.

SettingNLeffL_{\mathrm{eff}}GRPO+ DVCAGain (pp)
Original policy10579.482.8+3.4
Direct few-step inference5572.075.9+3.9
Direct few-step inference3368.872.6+3.8
Few-step distillation5571.474.8+3.4
Few-step distillation3367.771.0+3.3

Tail-length comparison from Table 12. N=10 and τ=2.5. Pick Diverse Bottles is the validation task; the remaining three tasks are held out for hyperparameter evaluation.

MethodLPick Diverse BottlesTurn SwitchMove Stapler PadMove Can Pot
GRPO—91.278.486.591.0
+ DVCA595.982.589.194.1
+ DVCA395.281.888.693.4
+ DVCA795.581.688.893.0

Post-training with different solvers. We post-train π0.5 with Filter IL on four RoboTwin 2.0 tasks using Euler and Heun, and compare success rates with and without DVCA. All runs start from the same initial policy and use the corresponding solver for both online rollouts and evaluation, with N=10, L=5, and a fixed time grid. All other settings follow the main experiments. Table 22 shows consistent gains with both Euler and Heun. With Heun, DVCA improves success rates by 3.6, 10.4, 5.0, and 6.0 percentage points across the four tasks, averaging 6.3 percentage points.

Table 22. Success rates (%) of π0.5 with Filter IL using Euler and Heun. Online rollouts and evaluation use the corresponding solver, with N=10 and L=5.

ODE solverMethodTurn SwitchPlace Mouse PadMove Stapler PadMove Can Pot
Eulerπ0.554.939.042.855.7
Euler + Filter IL87.465.081.569.6
Euler  w/ DVCA89.373.382.972.4
Heunπ0.556.642.145.758.1
Heun + Filter IL90.168.084.671.2
Heun  w/ DVCA93.778.489.677.2

Stability of action-level uncertainty. To assess the sensitivity of DV to denoising settings, we evaluate the same π0.5 checkpoint on a fixed set of observations while varying the step count, tail length, and solver. We reuse the initial noise across configurations. At the start of each integration step, we compute one clean-action estimate from the current state and velocity prediction; the second velocity evaluation in Heun corrects the update. Thus, 10 Heun steps require 20 network evaluations, and DV with L=5 uses estimates from the final five integration steps. We compute Spearman correlations between each configuration and the default configuration over action-level DV scores within each action chunk, then average over paired action chunks. Table 23 reports correlations of 0.960 and 0.956 when varying the step count, 0.987 and 0.992 when varying the tail length, and 0.933 with Heun, indicating largely consistent action-level uncertainty rankings. Figure 13 shows action-level DV under Euler and Heun on RoboTwin. The two sets of scores are highly correlated.

AuthorsOpenReview ↗

About Sensitivity to Denoising and Integration Settings (Question 2) (2/2)

Table 23. Spearman correlations of action-level DV scores with the default configuration (N=10, L=5, Euler).

NLODE solverSpearman correlation with default DV
105Euler1.000
55Euler0.960
33Euler0.956
103Euler0.987
107Euler0.992
105Heun0.933

Training objective and extensions. We use Filter IL, adding noise to actions from successful trajectories and predicting the corresponding velocity under the same flow-matching objective. Changing the solver affects the collected actions and DVCA weights, while the form of the training objective remains unchanged. Extending this comparison to GRPO requires coordinating higher-order ODE updates, stochastic denoising steps, and the transition probabilities used in policy optimization, making the implementation more involved. Combining DVCA with hybrid ODE–SDE GRPO using higher-order integration is a promising direction for future work.

We thank the reviewer for prompting this additional analysis of the DV signal and post-training performance across denoising settings.

Reviewer_Aw1eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DVCA, a method that exploits the variance of clean-actions estimates during denoising steps as a proxy for action uncertainty during online post-training. This variance signal is then converted into weights that are used to rescale advantages in policy gradients methods or per action losses in BC objectives. The method doesn't require any additional model training or inference, since the method reuses intermediate predictions from the denoising process. The authors show improvements in success rate or sample efficiency with a number of backbones in simulation and real-world manipulation tasks.

Critical strengths
  • DVCA tackes advantage of the intermediate predictions produced during the action generation to construct weights for advantages or supervised losses. This method is a inexpensive way to encourage learning from actions with higher denoising variance and is easy to integrate on existing methods.
  • The experiments show that adding DVCA to a number of backbones improves the success rate of this models in the tested envs. Ablation in A.1.2 show that removing the effect of action position (later action having more variance) maintains the improvement. This suggests that improvement is not explained only by giving more weight to later positions of a chunk
Critical weaknesses
  • The denoising variance estimator is inspired by Feng et al. 2026, and the the two level weighting follows Wang et al 2026 which makes the novelty appear incremental. DVCA seems like combining this ideas for online post-training of flow-based policies.
  • The evaluation does not sufficiently establish the reability of the improvements. The simulations results do not specify independent training-run counts, evaluation episode counts, or uncertainty estimates. This is a problem because some success rates deltas are very small. The real-world protocol is clearer about evaluation trial counts, although variability across independently trained policies is missing.
Questions for authors
  • Could you explain how you choose the checkpoints and show how much the results vary between different runs, especially when the differences in success rate are small?
  • For RL Token, do we still see the improvements if we compare both methods with the same number of interactions? And if we instead set the same target success rate, does DVCA need fewer interactions to reach it? How many initial reference policy trajectories are not counted, and are these initial data the same for the baseline and DVCA?

Comments:

  • Appendix A.1.4 is empty because table overflows to next page, consider adding a sentence to link to it.
9 comments
AuthorsOpenReview ↗

About Contributions and Relation to Prior Work (Weakness 1) (1/2)

Thank you for highlighting the relationship to prior work. We apologize that the original manuscript gave only a brief account of our contribution and its connection to these methods. Denoising-Variance Credit Assignment (DVCA) uses denoising dynamics for action-level credit assignment under episode-level feedback, with a reweighting scheme adapted to action chunks and different online learning objectives. Controlled perturbation experiments show that perturbing actions with high denoising variance causes a larger drop in success at the same perturbation magnitude and rate. Using this signal to allocate training weights improves success rates or sample efficiency. We have added this discussion to Appendix A.3.3, detailing the relationship to prior work, the design, and the empirical evidence below.

Relation to Prior Work

Denoising variance and credit assignment. Episode-level feedback evaluates the outcome of an entire trajectory, while individual actions along the trajectory differ in their impact on task success. Denoising variance (DV) describes prediction stability during action generation; credit assignment concerns the contribution of each action to task outcomes. Denoising-Variance Credit Assignment (DVCA) uses denoising dynamics for action-level credit assignment under episode-level feedback, with a reweighting scheme adapted to action chunks and different online learning objectives. Controlled perturbation experiments show that perturbing actions with high DV causes a larger drop in success at the same perturbation magnitude and rate. Using this signal to allocate training weights improves success rates or sample efficiency.

Relation to prior methods. DVCA adopts the variance estimator from Denoising-Variance Adaptive Chunking (DVAC) and builds on the hierarchical reweighting scheme in Visually-Guided Policy Optimization (VGPO). DVAC uses per-action variance to select the execution horizon at inference time, while VGPO uses visual relevance to reweight token and response advantages in vision-language models. DVCA further investigates the relationship between denoising variance and the impact of actions on task outcomes, and the utility of this signal for online robot learning.

Process reward models also provide finer-grained task signals for policy learning. Reward-Aligned Behavior Cloning (RA-BC) uses a reward model to predict task progress and reweight chunk-level imitation losses. These methods require additional reward-model training, and errors in progress estimation, such as assigning high progress to an incomplete task, affect the resulting training weights. DVCA directly reuses existing denoising predictions to compute per-action weights alongside the underlying learning objective. For example, in RL Token, it weights the reference-action BC term while retaining the original Q-learning objective.

Online post-training. DVCA distinguishes the training contributions of physical actions within each chunk and adjusts the overall weights across chunks. The weights are applied according to the underlying learning objective. In GRPO, they redistribute per-action gradient contributions while retaining the trajectory-advantage sign and the chunk-level likelihood ratio and clipping rule. In Filter IL, they weight per-action flow-matching errors before reduction. In RL Token, denoising variance from the frozen reference policy weights the reference-action BC term on successful trajectories, alongside the original Q-value objective. We normalize the weights to a mean of one and use dropout and annealing to control reweighting strength, emphasizing high-variance actions early and gradually restoring unit weights.

Ablations and policy improvement. DV has Spearman correlations of 0.593–0.750 with resampling disagreement (Table 24). With the same noise magnitude and target perturbation rate, perturbing high-variance, random, and low-variance actions reduces average success by 12.6, 5.4, and 2.6 percentage points, respectively (Table 25). Task success is therefore more sensitive to execution errors in high-variance actions, connecting denoising dynamics to action importance.

Table 8 further examines position-dependent and dynamic information in the signal. With π0.5 and GRPO, using only the fixed position profile yields 80.6% average success; retaining the dynamic variation after removing this profile yields 82.1%. The full signal achieves 82.8%, compared with 79.4% for GRPO. Both prediction position and variance that changes with the current execution provide useful learning signals, with their combination achieving higher success. The dropout and annealing ablations in Table 4 further examine the effect of reweighting strength on learning efficiency.

AuthorsOpenReview ↗

About Contributions and Relation to Prior Work (Weakness 1) (2/2)

Tables 1–3 show the training gains from DVCA with GRPO, RL Token, and Filter IL. Table 3 additionally compares other action-reweighting methods, with DVCA achieving higher ten-task average success rates across all three models. PPO and real-world results extend the evaluation across training methods and execution settings (Table 10; Figures 6–7). For RL Token, the gains primarily concern learning speed: including initialization, the trajectories required to reach 95% success decrease from 988 to 700 (Table 16). These results demonstrate the learning utility of denoising variance and the benefits of action-level reweighting across online post-training methods.

Thank you for helping us clarify the research question and its relationship to prior work.

AuthorsOpenReview ↗

About Statistical Reliability and Checkpoint Selection (Weakness 2, Question 1) (1/2)

Thank you for your suggestions on evaluation reliability and checkpoint selection. We apologize for the incomplete reporting of these details. In Appendix A.1.11 and Tables 17–21, we report means and standard deviations across three independent online post-training runs, 95% confidence intervals and p-values for the gains, and the checkpoint-selection rule and evaluation counts. All methods save and evaluate policies at the same intervals and select the checkpoint with the highest success rate in each run. The standard deviations of four-task average success rates are 0.1–0.5 percentage points in simulation and 0.4–0.7 in the real world. All six simulation gains have 95% confidence intervals above zero and p < 0.01. We also quantify sample efficiency for RL Token.

We provide the details below.

Evaluation protocol and checkpoint selection

We repeat online post-training with three independent random seeds. The seeds vary the initial Gaussian noise used during training and inference, as well as scene initialization in RoboTwin. Other factors are held fixed, including the initial policy, demonstration data, task settings, and hyperparameters. We save and evaluate policies at fixed intervals, select the checkpoint with the highest success rate in each run, and report the mean and standard deviation across the three runs. All methods use the same evaluation frequency and checkpoint-selection rule. Each selected policy is evaluated over 1,000 episodes in simulation. In the real-world experiments, RL Token and Filter IL with π0.5 collect four trajectories per round. We save and evaluate checkpoints every 12 rounds (48 trajectories), with 100 autonomous trials per evaluation checkpoint. All other settings follow the main experiments.

Results across training seeds

Tables 17–19 report the means and standard deviations from the additional runs, corresponding to Tables 1 and 3 and Figure 6 in the main paper. As a result, some means differ slightly from those reported in the main experiments. For Avg. (4), we average the four task scores within each run and then compute the mean and standard deviation across the three runs.

The standard deviations of four-task average success rates across three runs are 0.2–0.5 percentage points for GRPO, 0.1–0.4 for Filter IL, and 0.4–0.7 in the real-world experiments. DVCA improves over the corresponding baselines by 3.6–4.8, 1.8–5.3, and 5.7–9.1 percentage points, respectively. All six simulation comparisons in Table 20 have 95% confidence intervals above zero and p < 0.01, indicating statistically significant average gains.

Table 17. Mean and standard deviation of success rates (%) across three independent GRPO runs on four RoboTwin 2.0 tasks.

MethodTurn SwitchPlace Mouse PadMove Stapler PadMove Can PotAvg. (4)
π042.720.424.648.334.0
 + GRPO85.5 ± 1.071.0 ± 0.359.9 ± 1.384.1 ± 0.775.1 ± 0.5
  w/ DVCA93.2 ± 1.373.4 ± 0.265.0 ± 0.787.9 ± 1.079.9 ± 0.4
π0.554.939.042.855.748.1
 + GRPO77.9 ± 0.761.8 ± 0.786.8 ± 0.290.2 ± 1.279.2 ± 0.3
  w/ DVCA82.5 ± 0.465.7 ± 1.190.2 ± 0.494.1 ± 0.983.1 ± 0.4
Fast-WAM59.189.764.288.475.4
 + GRPO82.3 ± 1.297.2 ± 0.289.8 ± 0.295.0 ± 0.691.1 ± 0.2
  w/ DVCA88.4 ± 0.698.7 ± 0.192.8 ± 0.398.9 ± 0.894.7 ± 0.3
Reviewer_acg4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work proposes DVCA, a plug-in credit assignment module for flow-matching robot policies. It extracts denoising variance from the intermediate multi-step flow solver outputs as an intrinsic uncertainty signal, and re-weights advantages (for GRPO) or supervised losses (for Filter IL / RL-Token) at action level. The method introduces no extra network or forward-pass overhead. Evaluations are performed on RoboTwin2.0, LIBERO and real UR5e single-arm hardware, showing improved success rate and sample efficiency across multiple flow-based policy backbones.

Critical strengths

1.The core insight is elegant: reusing the intermediate denoising trajectory already generated during rollout for flow matching, requiring zero additional inference cost. This is a big advantage compared to learned process reward / critic-based credit assignment which needs extra networks. 2.Breadth of evaluation: three flow-based policies (VLA and WAM), three to four post-training paradigms (on-policy RL, actor-critic with a frozen VLA, filtered IL), two simulation benchmarks and real-robot experiments with OOD conditions. 3.Comprehensive ablations are provided for key hyperparameters. The observation that longer action chunks bring larger DVCA gains is well supported.

Critical weaknesses

1.Real robot validation is limited to single-arm setups. Only one simulated bimanual handover task is tested; no real bimanual or synchronized two-arm manipulation experiments are shown. 2.Hyperparameters including temperature, tail length L, dropout probability and annealing schedule require careful tuning, so it is not fully“out-of-the-box”. 3.No statistical reporting. None of the tables or curves report the number of seeds, standard deviations, confidence intervals or significance tests. Most improvements are small (e.g., +0.6 to +5 points for GRPO in Tables 1/5; +0.0 to +1.5 SR points for RL Token in Table 2; differences of ≤1 point between τ values in Table 4), which is plausibly within seed-to-seed variance of online RL. 4.Hyperparameters are tuned on evaluation tasks. The τ, dropout and annealing ablations (Table 4, Fig. 8) are run on Place Mouse Pad, which is also an evaluation task, and τ = 2.5 is then used everywhere. Please select hyperparameters on held-out tasks or report sensitivity across tasks. 5.Negative advantages. For failed trajectories, upweighting uncertain actions pushes probability mass away from exactly the actions the policy is least sure about. A discussion or analysis of why this is desirable is missing; Table 8 only shows empirically that two-sided weighting works better.

Questions for authors

Q1. How many seeds underlie each number? Can you provide mean ± std and significance tests for Tables 1–3 and the real-world results? Q2. Can τ be chosen on a disjoint set of tasks and then transferred? How sensitive are the results to L? Q3. In the real-world experiments, how were rewards or success determined during online training (human or automatic), and what were the human-intervention and reset protocols? Q4. How does DVCA behave for few-step or distilled flow policies, where the number of denoising steps is smaller than or close to L?

13 comments
AuthorsOpenReview ↗

About Bimanual Manipulation (Weakness 1)

Thank you for highlighting the importance of evaluating coordinated bimanual manipulation. We have added online post-training results on four bimanual tasks (Table 26) and an analysis of action-level uncertainty during transitions between arms (Figure 14) in Appendix A.1.14. DVCA improves GRPO on all four tasks, with an average gain of 5.3 percentage points.

We conduct additional experiments with π0.5 and GRPO on four RoboTwin 2.0 tasks that require close coordination between the two arms, evaluating DVCA for bimanual online post-training. Handover Mic requires transferring an object between the arms; Lift Pot requires lifting a pot jointly; Scan Object requires aligning objects held by the two arms; and Place Bread Skillet requires one arm to hold the pan while the other places bread inside. We compare the initial policy, GRPO, and GRPO with DVCA. All other settings follow the main experiments.

Table 26. Success rates (%) of π0.5 on four RoboTwin 2.0 bimanual tasks. Avg. denotes the four-task mean.

MethodHandover MicLift PotScan ObjectPlace Bread SkilletAvg.
π0.58.335.140.048.232.9
 + GRPO45.278.665.988.469.5
  + DVCA51.783.870.493.574.9

Table 26 shows that DVCA improves GRPO on all four tasks by 6.5, 5.2, 4.5, and 5.1 percentage points, respectively, with an average gain of 5.3 percentage points. These results show that DVCA also improves online post-training for tasks requiring close coordination between the two arms.

Figure 14 shows action-level denoising variance (DV) and execution frames from successful Stack Bowls Two and Hanging Mug episodes. DV increases as one arm finishes placing an object and the other begins reaching or regrasping, then decreases during stable grasping. This pattern reflects action uncertainty during transitions between arms in bimanual manipulation.

Our current hardware consists of a single UR5e arm. We are procuring and assembling a dual-arm UR7e platform. If deployment is completed on schedule, we will include real-world bimanual experiments in the camera-ready version.

Thank you for this suggestion, which prompted us to further evaluate DVCA in coordinated bimanual manipulation.

AuthorsOpenReview ↗

About Hyperparameter Selection, Transfer, and Sensitivity (Weakness 2, Weakness 4, Question 2) (1/2)

Thank you for your comments on hyperparameter selection and sensitivity. In Appendix A.1.7, we add experiments on temperature selection using a separate validation task, cross-task transfer, and sensitivity to all four hyperparameters. The selected temperature transfers directly to the three held-out tasks, and DVCA retains its gains with small changes in success rate across nearby values of the temperature, denoising tail length, dropout probability, and annealing endpoint. These results support shared defaults with less per-task tuning.

Hyperparameter Selection and Sensitivity. We use π0.5\pi_{0.5} with GRPO, select τ=2.5\tau=2.5 on Pick Diverse Bottles, and fix this temperature for Turn Switch, Move Stapler Pad, and Move Can Pot. Table 12 compares τ∈2,2.5,3\tau\in\\{2,2.5,3\\} and L∈3,5,7L\in\\{3,5,7\\}, while Table 13 varies the dropout probability and annealing endpoint separately. DVCA retains gains on all three held-out tasks across these settings, with small changes in success rate. These results indicate a broad effective range around the default configuration and support its reuse across tasks.

Table 12. Success rates (%) of π0.5\pi_{0.5} with GRPO across temperatures and denoising tail lengths. Pick Diverse Bottles is used for hyperparameter selection. ∗^* denotes the selected temperature; (Ours) indicates the full default configuration. Bold indicates the best result within each tail-length group.

MethodLτPick Diverse Bottles (Validation)Turn Switch (Held-out)Move Stapler Pad (Held-out)Move Can Pot (Held-out)
π0.5——71.154.942.855.7
 + GRPO——91.278.486.591.0
  + DVCA5294.582.488.793.4
  + DVCA (Ours)52.5∗2.5^{*}95.982.589.194.1
  + DVCA5395.182.089.193.2
  + DVCA3294.981.588.292.7
  + DVCA32.5∗2.5^{*}95.281.888.693.4
  + DVCA3395.181.788.393.4
  + DVCA7295.081.388.292.5
  + DVCA72.5∗2.5^{*}95.581.688.893.0
  + DVCA7394.681.188.592.7

Temperature. At a fixed LL, varying τ\tau from 2 to 3 changes success rates by at most 0.9 percentage points on the three held-out tasks. A smaller τ\tau emphasizes high-variance actions, while a larger τ\tau moves the weights closer to uniform, making a moderate temperature range effective for balancing reweighting strength. Score normalization before temperature scaling reduces dependence on task-specific variance scales, allowing a shared temperature to have a comparable effect across tasks.

Denoising tail length. LL is the number of final denoising steps used to estimate variance. With τ=2.5\tau=2.5, success rates vary by at most 1.1 percentage points across L∈3,5,7L\in\\{3,5,7\\}. This low sensitivity can be understood from the post-training setting and denoising process. The pretrained π0.5\pi_{0.5} policy already has strong manipulation capabilities, and task-specific post-training uses limited data from a relatively narrow distribution. Earlier denoising stages are more affected by the initial noise, whereas the final stages refine increasingly stable action estimates. Neighboring LL windows cover overlapping late-stage predictions, helping preserve relative fluctuations across actions; subsequent normalization further reduces the effect of changes in the overall variance scale.

Table 13. Sensitivity to reweighting dropout and annealing with L=5L=5 and τ=2.5\tau=2.5. We vary one setting at a time and report success rates (%). Bold indicates the best result within each parameter sweep.

MethodDropout pAnnealing (rounds)Pick Diverse Bottles (Validation)Turn Switch (Held-out)Move Stapler Pad (Held-out)Move Can Pot (Held-out)
π0.5——71.154.942.855.7
 + GRPO——91.278.486.591.0
  + DVCA (Ours)0.21–20095.982.589.194.1
  + DVCA0.11–20095.882.189.093.7
  + DVCA0.31–20095.682.588.694.0
  + DVCA0.21–10095.682.089.193.8
  + DVCA0.21–30095.281.988.893.7
AuthorsOpenReview ↗

About Hyperparameter Selection, Transfer, and Sensitivity (Weakness 2, Weakness 4, Question 2) (2/2)

Reweighting dropout and annealing. Dropout randomly restores unit weights, balancing emphasis on high-variance actions with training contributions from other actions. Annealing gradually reduces reweighting strength, restoring the weighting of the underlying method later in training. Nearby settings moderately change the strength and duration of reweighting while retaining the same learning signal and training procedure. Varying pp from 0.1 to 0.3 and the annealing endpoint from round 100 to round 300 changes success rates by at most 0.5 and 0.6 percentage points, respectively, on the three held-out tasks. These results support shared defaults of L=5L=5, p=0.2p=0.2, and annealing over rounds 1--200 within GRPO, with the temperature selected on the validation task transferred to the other tasks.

Thank you again for your helpful comments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion

We thank the Area Chair for coordinating the review and discussion, and the reviewers for their careful reading and constructive feedback. We have updated the manuscript with additional experiments, quantitative analyses, and methodological clarifications. All section, table, and figure numbers below refer to the revised manuscript.

DVCA uses uncertainty from existing denoising predictions for action-level credit assignment under episode-level feedback. Its three main strengths are reusing existing predictions without additional training or inference, applying action-level reweighting, and integrating with advantage-based and supervised objectives across online post-training methods.

Main revisions

  1. Statistical reliability and sample efficiency. We added three-run statistics for GRPO, Filter IL, and real-world experiments, including means and sample standard deviations, confidence intervals and p-values for simulation gains, and evaluation and checkpoint-selection protocols. All six simulation gains have 95% confidence intervals above zero and p < 0.01. Including initialization, DVCA reduces the trajectories needed for RL Token to reach 95% success on Place Mouse Pad by 29.1%. Appendices A.1.10–A.1.11; Tables 16–21; Figure 12. In response to Aw1e's follow-up questions, we also added per-run success rates in Appendix A.6 (Tables 31–33) and clarified the evaluation protocol.

  2. Hyperparameter selection and denoising settings. We added temperature selection on a separate validation task, cross-task transfer, sensitivity analyses for all four hyperparameters, and experiments with few-step policies, distillation, and different solvers. The selected temperature transfers directly, with small performance changes across nearby hyperparameters. DVCA retains its gains with 5- and 3-step policies, distilled policies, and Euler and Heun solvers. Appendices A.1.7–A.1.8 and A.1.12; Tables 12–14 and 22–23; Figure 13.

  3. Denoising variance and action reweighting. We added resampling-disagreement, action-perturbation, and negative-advantage analyses, and clarified the relation to prior work. At the same perturbation magnitude and target rate, perturbing high-variance actions reduces average success by 12.6 percentage points, versus 5.4 and 2.6 for random and low-variance actions. Failure-segment analysis further explains how deviations and useful shared actions affect negative-advantage weighting. These results connect denoising uncertainty, action importance, and learning gains. Appendices A.1.9, A.1.13, and A.3.3; Tables 15 and 24–25; Figure 11.

  4. Robot experiments and execution protocols. We added real-world training with human interventions and four coordinated bimanual tasks. DVCA improves performance with and without interventions, and improves GRPO by 5.3 percentage points on average across the bimanual tasks. We also clarified human success annotation, the 60 s episode limit, scene resets, and human involvement during training. Appendices A.1.6, A.1.14, and A.2.5; Tables 11 and 26; Figures 10 and 14.

Discussion summary

  • acg4 (rating: 3 → 4; confidence: 3). The reviewer stated that the detailed response resolved their questions and that they were more optimistic about the manuscript.
  • YeZS (rating: 3; confidence: 2 → 3). The reviewer stated that they had no further questions and would maintain their positive rating.
  • Aw1e (rating: 2; confidence: 3). After thanking us for our answers, the reviewer asked about the small across-run standard deviations, per-run results, and evaluation protocol. We provided the results and clarifications summarized above. As of this summary, we have received no further follow-up.

We thank the Area Chair and reviewers again for their time and effort. Detailed responses are available under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.