- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies RL fine-tuning of diffusion and flow policies via value gradients. While prior work has found backpropagating through the sampling chain to be unstable, the authors argue that this instability can be avoided with a suitable choice of sampler. Based on this, they propose VINE, which replaces the Euler update with an update that re-noises the current action with fresh noise at each step. This change is minimal, and updates how value gradients propagate through the sampling chain.
The authors evaluate VINE on three settings: offline RL on 50 OGBench tasks, online fine-tuning of the pi-0.5 VLA on LIBERO, and a real-world plug-insertion task. On OGBench, VINE outperforms 14 baselines (on aggregate score), and on LIBERO it trains more stably than the ODE and SDE samplers. On the real robot, VINE reaches 20/20 success after 20 minutes of fine-tuning, with the least human intervention among the compared methods. The paper also includes ablations on time schedules, noise resampling and gradient dynamics.
Critical strengths
-
The paper is easy to follow. Figures 2 and 3 are particularly helpful, as they show how the samplers differ in intermediate-state coverage and how each behaves under RL fine-tuning, which demonstrates the method nicely.
-
The algorithm itself is simple and easy to implement. It requires only a few lines of changes to the sampler and can be readily applied to existing pretrained models, as the VLA experiments show.
-
The main experiments are thorough and convincing, covering both OGBench and VLA fine-tuning. The results in Table 1 are particularly strong, with VINE outperforming 14 different baselines.
-
Finally, the real-world experiments are a useful addition. Here too, the paper compares against multiple baselines, and VINE achieves a 100% success rate.
Critical weaknesses
-
The paper gives limited insight into why VINE helps. In particular, the sampling process changes the marginal distribution of the denoising path. As a result, the model is evaluated off-distribution (as can also be seen in Figure 2), which could change the performance of the base policy even without RL fine-tuning. A more precise mathematical statement would be useful.
-
The ablation section is relatively weak. In particular, VINE is used in three places in Algorithm 1: critic target actions, actor updates and data collection. It would be valuable to ablate each separately to see what drives the improvement. It would also be informative to compare against other samplers, such as marginal-preserving stochastic samplers.
-
The proposed sampler closely resembles existing samplers, but the paper does not discuss this connection. Multistep sampling in Consistency Models (Song et al., 2023) follows the same procedure of predicting the endpoint and then re-noising with fresh Gaussian noise. The contribution lies in applying this sampler to value-gradient RL, but the paper should position itself relative to prior work.
Questions for authors
-
In general, how does the base policy perform with VINE sampling compared to Euler sampling before any RL fine-tuning (i.e., BC only)? Can the authors characterize how the intermediate and final action distributions differ under these two samplers?
-
Appendix C.4 states that VINE is trained from scratch. Early on, a randomly initialized policy will have a near-zero success rate. Can the authors comment on how interventions are carried out during training, and how this was standardized across methods?
3 comments
Thank you for the response. The base-policy comparison and the distributional analysis are helpful. Could the authors also comment on the remaining points from my review, namely the ablations (applying VINE separately to critic targets, actor updates and data collection) and using marginal-preserving stochastic samplers?
A rebuttal comment for Reviewer ecQ6. Thank you for highlighting these remaining points. We conduct experiment to address the component ablation. Please find our response below.
Component ablations. We vary the samplers used for critic-target construction, actor updates, and data sample. The table reports success rates on antmaze-large after 1M updates, using four seeds and 50 evaluation episodes per seed.
| Method | Critic | Actor | Sample | Success (%) |
|---|---|---|---|---|
| VINE (full) | VINE | VINE | VINE | 87 [83, 92] |
| Collect-Euler | VINE | VINE | Euler | 87 [83, 92] |
| Critic+Collect-Euler | Euler | VINE | Euler | 83 [79, 88] |
| Euler (full) | Euler | Euler | Euler | 20 [7, 34] |
These results identify the actor update as the main source of the observed benefit in this setting. Using VINE only for actor updates retains most of the full method’s performance.
Rebuttal comment for Reviewer ecQ6.
Rebuttal comment for Reviewer ecQ6. Thank you for the detailed review and constructive feedback on this work. We especially appreciate your pointing out some questions. We would like to clarify the base-policy performance, the distributional differences between VINE and Euler, and the human-intervention protocol for you.
Q: How does the base policy perform with VINE sampling compared to Euler sampling before any RL fine-tuning?
A: Thanks for the question. Using the same pretrained checkpoint without RL fine-tuning, VINE and Euler achieve success rates of 45.26% and 44.34%, respectively, on LIBERO (Appendix Figure 8). In the real-robot evaluation reported on website, the corresponding success rates are 61% and 55%. The results show that VINE sampler provides slightly comparable performance with Euler.
Q: Can the authors characterize how the intermediate and final action distributions differ under these two samplers?
A: Thank you for this question. to characterize the distribution, we conduct some analysis below. Generally Euler and VINE induce different intermediate-state distributions and final action distribution. VINE use:
\sim \mathcal N\left(t_{k+1}\hat a_k,\, (1-t_{k+1})^2I\right).$$ Intuitively, at each intermediate point in time, VINE generates a neighborhood around the current predicted endpoint. Euler uses deterministic transitions, $x_k = x_{k-1} + \Delta t v_\theta(x_{k-1},t_{k-1};s)$. The final action distributions generally differ, although their exact forms are unknown for a learned policy. To illustrate this difference, we consider an idealized setting with independent standard Gaussian target actions and initial noise, an exact flow-matching velocity field, and a uniform time grid. As the number of sampling steps $K\to\infty$, $$A_{\mathrm{VINE}}^{(K)} \xrightarrow{d}\mathcal N\left(0,\tfrac12 I_d\right), \qquad A_{\mathrm{Euler}}^{(K)} \xrightarrow{d}\mathcal N(0,I_d).$$ Thus, in this setting, VINE retains the same endpoint support as Euler but produces a more concentrated action distribution. The proof follows three steps. First, Gaussian conditioning gives $v^\star(x,t)=(2t-1)x/[t^2+(1-t)^2]$ and the endpoint estimate $D(x,t)=tx/[t^2+(1-t)^2]$. Second, the corresponding ODE has solution $X_t=\sqrt{t^2+(1-t)^2}X_0$, yielding $X_1=X_0\sim\mathcal N(0,I_d)$. Third, VINE's endpoint estimates remain zero-mean Gaussian with covariance $r_kI_d$, where $r_{k+1}=b_k^2r_k+b_k(1-b_k)$ and $b_k=t_k^2/[t_k^2+(1-t_k)^2]$; this recursion yields $r_K\to\tfrac12$ as the uniform grid is refined. **Q: Can the authors comment on how interventions are carried out during training, and how this was standardized across methods?** **A:** Thanks for the question. Following real-world RL training practices such as RLT [1] and HIL-SERL [2], we allow human interventions during training, particularly early on when policy success rates are low. Humans take over when the policy deviates from the task goal or approaches an unsafe situation. We apply these intervention guidelines as consistently as possible across all baselines. For example, in plug insertion task, operator allows model to explore and go, takes control only when model hit the charger and deviate safe region too much. We also report the human intervention ratio, defined as the fraction of the total training data provided by humans. VINE requires a smaller proportion of human intervention data than the compared methods. References: [1] Xu, Charles, et al. “RL Token: Bootstrapping Online RL with Vision-Language-Action Models.” arXiv preprint arXiv:2604.23073. 2026. [2] Luo, Jianlan, et al. “Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.” Science Robotics 10, no. 105 (2025): eads5033. --- Thank you again for your thoughtful review. We hope these clarifications address your questions and concerns!