acceptodds
Under review as a conference paper at ICLR 2027

Taming Generative Control Policies for Reinforcement Learning

13 views

Abstract

Generative control policies have emerged as an effective policy parameterization for robot learning. They leverage generative models such as diffusion and flow matching to iteratively transform noise into actions, enabling these policies to effectively model complex action distributions. However, prior work has reported that optimizing these policies with reinforcement learning (RL) can lead to training instability. This instability has often been associated with the iterative nature of the generation process. Many existing methods therefore avoid directly backpropagating value gradients through the entire iterative generation process, leading suboptimal performance. Contrary to prior belief, We show that iterative generation can support stable value-gradient RL and identify sampler design as an important factor in optimization. Motivated by this insight, we propose VINE, a new sampling strategy that reconstructs the generation path at each iteration, enabling stable end-to-end value-gradient propagation. VINE requires only a few lines of changes to the vanilla flow matching sampler. VINE retains iterative generation and end-to-end value-gradient optimization through the entire sampling chain. Experiments show that VINE outperforms the evaluated full-BPTT baselines in aggregate on OGBench tasks. VINE enables stable off-policy RL adaptation of the pretrained VLA across LIBERO suites. On real-world manipulation tasks, VINE rapidly improves success rate toward 100% from scratch within 20 minutes of online rollouts.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, vision–language–action model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ecQ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies RL fine-tuning of diffusion and flow policies via value gradients. While prior work has found backpropagating through the sampling chain to be unstable, the authors argue that this instability can be avoided with a suitable choice of sampler. Based on this, they propose VINE, which replaces the Euler update with an update that re-noises the current action with fresh noise at each step. This change is minimal, and updates how value gradients propagate through the sampling chain.

The authors evaluate VINE on three settings: offline RL on 50 OGBench tasks, online fine-tuning of the pi-0.5 VLA on LIBERO, and a real-world plug-insertion task. On OGBench, VINE outperforms 14 baselines (on aggregate score), and on LIBERO it trains more stably than the ODE and SDE samplers. On the real robot, VINE reaches 20/20 success after 20 minutes of fine-tuning, with the least human intervention among the compared methods. The paper also includes ablations on time schedules, noise resampling and gradient dynamics.

Critical strengths
  1. The paper is easy to follow. Figures 2 and 3 are particularly helpful, as they show how the samplers differ in intermediate-state coverage and how each behaves under RL fine-tuning, which demonstrates the method nicely.

  2. The algorithm itself is simple and easy to implement. It requires only a few lines of changes to the sampler and can be readily applied to existing pretrained models, as the VLA experiments show.

  3. The main experiments are thorough and convincing, covering both OGBench and VLA fine-tuning. The results in Table 1 are particularly strong, with VINE outperforming 14 different baselines.

  4. Finally, the real-world experiments are a useful addition. Here too, the paper compares against multiple baselines, and VINE achieves a 100% success rate.

Critical weaknesses
  1. The paper gives limited insight into why VINE helps. In particular, the sampling process changes the marginal distribution of the denoising path. As a result, the model is evaluated off-distribution (as can also be seen in Figure 2), which could change the performance of the base policy even without RL fine-tuning. A more precise mathematical statement would be useful.

  2. The ablation section is relatively weak. In particular, VINE is used in three places in Algorithm 1: critic target actions, actor updates and data collection. It would be valuable to ablate each separately to see what drives the improvement. It would also be informative to compare against other samplers, such as marginal-preserving stochastic samplers.

  3. The proposed sampler closely resembles existing samplers, but the paper does not discuss this connection. Multistep sampling in Consistency Models (Song et al., 2023) follows the same procedure of predicting the endpoint and then re-noising with fresh Gaussian noise. The contribution lies in applying this sampler to value-gradient RL, but the paper should position itself relative to prior work.

Questions for authors
  1. In general, how does the base policy perform with VINE sampling compared to Euler sampling before any RL fine-tuning (i.e., BC only)? Can the authors characterize how the intermediate and final action distributions differ under these two samplers?

  2. Appendix C.4 states that VINE is trained from scratch. Early on, a randomly initialized policy will have a near-zero success rate. Can the authors comment on how interventions are carried out during training, and how this was standardized across methods?

3 comments
AuthorsOpenReview ↗

Rebuttal comment for Reviewer ecQ6.

Rebuttal comment for Reviewer ecQ6. Thank you for the detailed review and constructive feedback on this work. We especially appreciate your pointing out some questions. We would like to clarify the base-policy performance, the distributional differences between VINE and Euler, and the human-intervention protocol for you.


Q: How does the base policy perform with VINE sampling compared to Euler sampling before any RL fine-tuning?

A: Thanks for the question. Using the same pretrained checkpoint without RL fine-tuning, VINE and Euler achieve success rates of 45.26% and 44.34%, respectively, on LIBERO (Appendix Figure 8). In the real-robot evaluation reported on website, the corresponding success rates are 61% and 55%. The results show that VINE sampler provides slightly comparable performance with Euler.

Q: Can the authors characterize how the intermediate and final action distributions differ under these two samplers?

A: Thank you for this question. to characterize the distribution, we conduct some analysis below. Generally Euler and VINE induce different intermediate-state distributions and final action distribution. VINE use:

\sim \mathcal N\left(t_{k+1}\hat a_k,\, (1-t_{k+1})^2I\right).$$ Intuitively, at each intermediate point in time, VINE generates a neighborhood around the current predicted endpoint. Euler uses deterministic transitions, $x_k = x_{k-1} + \Delta t v_\theta(x_{k-1},t_{k-1};s)$. The final action distributions generally differ, although their exact forms are unknown for a learned policy. To illustrate this difference, we consider an idealized setting with independent standard Gaussian target actions and initial noise, an exact flow-matching velocity field, and a uniform time grid. As the number of sampling steps $K\to\infty$, $$A_{\mathrm{VINE}}^{(K)} \xrightarrow{d}\mathcal N\left(0,\tfrac12 I_d\right), \qquad A_{\mathrm{Euler}}^{(K)} \xrightarrow{d}\mathcal N(0,I_d).$$ Thus, in this setting, VINE retains the same endpoint support as Euler but produces a more concentrated action distribution. The proof follows three steps. First, Gaussian conditioning gives $v^\star(x,t)=(2t-1)x/[t^2+(1-t)^2]$ and the endpoint estimate $D(x,t)=tx/[t^2+(1-t)^2]$. Second, the corresponding ODE has solution $X_t=\sqrt{t^2+(1-t)^2}X_0$, yielding $X_1=X_0\sim\mathcal N(0,I_d)$. Third, VINE's endpoint estimates remain zero-mean Gaussian with covariance $r_kI_d$, where $r_{k+1}=b_k^2r_k+b_k(1-b_k)$ and $b_k=t_k^2/[t_k^2+(1-t_k)^2]$; this recursion yields $r_K\to\tfrac12$ as the uniform grid is refined. **Q: Can the authors comment on how interventions are carried out during training, and how this was standardized across methods?** **A:** Thanks for the question. Following real-world RL training practices such as RLT [1] and HIL-SERL [2], we allow human interventions during training, particularly early on when policy success rates are low. Humans take over when the policy deviates from the task goal or approaches an unsafe situation. We apply these intervention guidelines as consistently as possible across all baselines. For example, in plug insertion task, operator allows model to explore and go, takes control only when model hit the charger and deviate safe region too much. We also report the human intervention ratio, defined as the fraction of the total training data provided by humans. VINE requires a smaller proportion of human intervention data than the compared methods. References: [1] Xu, Charles, et al. “RL Token: Bootstrapping Online RL with Vision-Language-Action Models.” arXiv preprint arXiv:2604.23073. 2026. [2] Luo, Jianlan, et al. “Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.” Science Robotics 10, no. 105 (2025): eads5033. --- Thank you again for your thoughtful review. We hope these clarifications address your questions and concerns!

Thank you for the response. The base-policy comparison and the distributional analysis are helpful. Could the authors also comment on the remaining points from my review, namely the ablations (applying VINE separately to critic targets, actor updates and data collection) and using marginal-preserving stochastic samplers?

A rebuttal comment for Reviewer ecQ6. Thank you for highlighting these remaining points. We conduct experiment to address the component ablation. Please find our response below.

Component ablations. We vary the samplers used for critic-target construction, actor updates, and data sample. The table reports success rates on antmaze-large after 1M updates, using four seeds and 50 evaluation episodes per seed.

MethodCriticActorSampleSuccess (%)
VINE (full)VINEVINEVINE87 [83, 92]
Collect-EulerVINEVINEEuler87 [83, 92]
Critic+Collect-EulerEulerVINEEuler83 [79, 88]
Euler (full)EulerEulerEuler20 [7, 34]

These results identify the actor update as the main source of the observed benefit in this setting. Using VINE only for actor updates retains most of the full method’s performance.

Reviewer_favLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies generative policies in reinforcement learning. Due to the need for gradients to backpropagate through multiple steps of the generation process when computing value, it may lead to training instability. To address this, the authors proposed VINE, a new sampling strategy that reconstructs intermediate states by reintroducing high-speed noise at each generation step, thereby achieving stable value gradient propagation. Experimental results show that VINE outperforms existing full-chain BPTT methods on OGBench, achieves more stable online policy optimization on LIBERO, and reaches a 100% success rate in real robot operation tasks after 20 minutes of online training.

Critical strengths
  1. This paper shows that the re-noising trick can effectively alleviate the training instability of generative policy in RL optimization, providing a simple and effective method for stabilizing value gradient optimization.
  2. It is commendable that the experiments in the entire work are quite thorough, including OGBench, large-scale VLA models on LIBERO, as well as real robot operation tasks, which validate the effectiveness of the method in different scenarios.
Critical weaknesses
  1. The sampling mechanism of VINE is essentially a re-noise sampling mechanism, a trick that has already been used in many works, such as gamma-sampling in CTM [1]. The core idea is to first predict the clean target state and then reintroduce Gaussian noise during the iterative generation process. Therefore, although applying this idea to value gradient optimization in stable reinforcement learning has some significance, I have concerns about the novelty of VINE's sampling strategy itself. I hope the authors can further explain the differences between VINE and these existing methods, and clarify the innovative aspects.

  2. I still have some questions about the mechanism of VINE in improving training stability. The paper derives the effect of re-noising on the Jacobian in Eqs. (14)–(15), but the change in the Jacobian itself does not necessarily mean that gradient propagation is more stable. Figure 6(c) compares VINE and Euler FM-BPTT, and the results show that VINE has smaller gradient norm fluctuations and a higher success rate. However, we notice that re-noising actually brings two changes: one is the change in the generated trajectory itself, and the other is the change in gradient propagation. What I want to know is whether this stability is due to changes in the generated trajectory itself or due to gradient propagation (paper's claim). So I think the authors need to clarify the actual effect of re-noisng.

References: [1] Kim, Dongjun, et al. "Consistency trajectory models: Learning probability flow ode trajectory of diffusion." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors
  1. What's the difference between VINE and existing stochastic sampling methods? The trick of re-nosing is quite common in generation, such as gamma-sampling in CTM. Although the specific implementations differ, I'm not quite sure what new designs VINE has in its sampling strategy itself. I hope the author can further explain the differences between the two.
  2. Does the improvement in VINE's training stability mainly come from changes in the sampling trajectory or changes in gradient propagation? Re-noising will affect both at the same time, but the paper mainly emphasizes the latter. Can the authors provide further analysis or ablation experiments?
  3. Could you provide a more thorough comparison between VINE and SDE samplers? The article compares VINE, ODE, and SDE sampling on LIBERO, but I think the analysis is not thorough enough. For example, could the authors further explain the noise settings for the SDE baseline and whether adjusting the noise intensity could achieve stability and performance similar to VINE? Considering that both methods change the generation process by introducing random noise, I am quite curious whether there is a fundamental difference in the mechanisms of VINE's re-noising strategy and the SDE sampler.
15 comments
AuthorsOpenReview ↗

Rebuttal comment for Reviewer favL; Additional experiments and analysis to address your questions about the source of VINE’s improvements and its relation to prior work

Rebuttal comment for Reviewer favL. Thank you for the detailed review. We have conducted additional experiments and analysis to address your questions about the source of VINE's improvements and its relation to prior work. Please find our response below.


Q: Further explain the differences between VINE and these existing methods such as gamma-sampling in CTM, and clarify the innovative aspects.

A: Thank you for highlighting this connection. We agree that CTM's γ=1\gamma=1 sampling shares VINE's high-level structure, and we will explicitly acknowledge this connection and carefully clarify our innovative aspects.

The main difference lies in domain setting and specific implementations. CTM targets for fast, high-quality image generation. We uses re-nosing construction for end-to-end value-gradient RL and toward robot learning. The specific optimization recipe also differs.

The difference also lies in the prediction model. CTM additionally learns flow mappings between time points. VINE constructs estimate directly from a flow-matching velocity field, a^k=xk+(1−tk)vθ(xk,tk,s)\hat a_k=x_k+(1-t_k)v_\theta(x_k,t_k,s), without additional consistency-trajectory constraint.

About the innovative aspects, We would like to clarify our main contribution is the finding of sampler design improve the stability of RL for generative control policy such flow-policy. Prior robot learning works, including FQL, DSRL, EXPO, and QAM, have highlighted difficulties with direct value-gradient optimization. They developed alternative optimization strategies to avoid BPTT. VINE go with change of VINE and focus BPTT. VINE empirically evaluate effectiveness across offline RL, VLA fine-tuning, and real-world robot learning to support the aspects.

To make the scope of our contribution precise, we will revise the paper as follows:

  1. Replace “a new sampling strategy” in the abstract with “an RL-oriented sampler that combines velocity-based endpoint prediction with fresh-noise reconstruction.”
  2. Expand the related-work discussion to acknowledge CTM and distinguish the established re-noising pattern from our construction, analysis, and empirical findings in value-gradient RL.
AuthorsOpenReview ↗

VINE's gains depend on the combination of its sampling dynamics and corresponding gradient propagation both

Q: Does the improvement in VINE's training stability mainly come from changes in the sampling trajectory or changes in gradient propagation?

A: Thank you for the question. Original VINE indeed changes both the forward sampling dynamics and the resulting gradient propagation. To examine whether the backward modification or dynamic alone explains its benefit, our original appendix (lines 898--907) includes an Euler--VINE-SG ablation. Let Eθ,kE_{\theta,k} and Vθ,kV_{\theta,k} denote the Euler and VINE step maps. With sg⁡\operatorname{sg} denoting stop-gradient, we use

ak+1=sg⁡(Eθ,k(ak))+Vθ,k(ak,zk)−sg⁡(Vθ,k(ak,zk)).a_{k+1} = \operatorname{sg}(E_{\theta,k}(a_k)) + V_{\theta,k}(a_k,z_k) - \operatorname{sg}(V_{\theta,k}(a_k,z_k)).

The forward value equals the Euler update, while automatic differentiation of gradient propagation uses VINE derivatives along the Euler trajectory. Also we add the experiment about VINE--Euler-SG. which means the forward value equals the VINE update, while automatic differentiation of gradient propagation uses Euler:

ak+1=sg⁡(Vθ,k(ak,zk))+Eθ,k(ak)−sg⁡(Eθ,k(ak)).a_{k+1} = \operatorname{sg}(V_{\theta,k}(a_k,z_k)) + E_{\theta,k}(a_k) - \operatorname{sg}(E_{\theta,k}(a_k)).

The following results are obtained on AntMaze task 2 after 1M1 M gradient updates, averaged over eight seeds (10001--10008):

AntMazeSuccess rate (%)Note
VINE87  [83,92]87\;[83,92]VINE dynamics, VINE gradient propagation
Euler20  [7,34]20\;[7,34]Euler dynamics, Euler gradient propagation
Euler--VINE-SG3  [0,5]3\;[0,5]Euler dynamics, VINE gradient propagation
VINE--Euler-SG11  [3,21]11\;[3,21]VINE dynamics, Euler gradient propagation

These results suggest that VINE's gains depend on the combination of its sampling dynamics and corresponding gradient propagation in this setting.

AuthorsOpenReview ↗

Comparison between VINE and SDE samplers; fundamental difference

Q: Could you provide a more thorough comparison between VINE and SDE samplers?

A: Thank you for suggesting a more systematic comparison. Empirical comparison are shown in original paper submission. To further compare this, we next provide some algebraic math comparison to make it more clear. We first specify the SDE-based samplers family considered here[1][2][3]. Following the notation of original paper, its Euler SDE sampler update is

xk+1=xk+[vk−σtk22tk(xk+(1−tk)vk)]Δtk+σtkΔtk zk,x_{k+1} = x_k + \left[v_k-\frac{\sigma_{t_k}^{2}}{2t_k}\bigl(x_k+(1-t_k)v_k\bigr)\right]\Delta t_k + \sigma_{t_k}\sqrt{\Delta t_k}\,z_k,

where vk=vθ(xk,tk)v_k=v_\theta(x_k,t_k), Δtk=tk+1−tk\Delta t_k=t_{k+1}-t_k, and zk∼N(0,I)z_k\sim\mathcal{N}(0,I) is independent Gaussian noise. The nonnegative function σtk\sigma_{t_k} is tunable parameters; setting σt=0\sigma_t=0 recovers the Euler ODE update. Motivated by the stochastic-interpolant framework [2], we rewrite these discrete updates in a common interpolation form:

xk+1=tk+1a^k+(1−tk+1)z^k,a^k=xk+(1−tk)vk.x_{k+1} = t_{k+1}\hat a_k+(1-t_{k+1})\hat z_k,\qquad \hat a_k=x_k+(1-t_k)v_k.

Here, a^k\hat a_k is the action-endpoint estimate and z^k\hat z_k is an noise-side endpoint term. For the SDE update above,

z^kSDE=αkxk+βkvk+γkzk,\hat z_k^{\mathrm{SDE}} = \alpha_kx_k+\beta_kv_k+\gamma_kz_k,

where

αk=1−σtk2Δtk2tk(1−tk+1)\alpha_k = 1-\frac{\sigma_{t_k}^{2}\Delta t_k}{2t_k(1-t_{k+1})}; βk=−tk−σtk2(1−tk)Δtk2tk(1−tk+1)\beta_k = -t_k-\frac{\sigma_{t_k}^{2}(1-t_k)\Delta t_k}{2t_k(1-t_{k+1})}; γk=σtkΔtk1−tk+1.\gamma_k = \frac{\sigma_{t_k}\sqrt{\Delta t_k}}{1-t_{k+1}}.

In comparison, VINE predicts the endpoint and reconstructs the next intermediate state using fresh Gaussian noise:

xk+1=tk+1a^k+(1−tk+1)zk,zk∼N(0,I).x_{k+1} = t_{k+1}\hat a_k+(1-t_{k+1})z_k,\qquad z_k\sim\mathcal{N}(0,I).

Thus, the updates have a similar interpolation structure, but construct their effective noise z^k\hat z_k differently: the SDE combines the current state, predicted velocity, and fresh noise, whereas VINE uses fresh noise directly. In particular, we can not got (αk,βk,γk)=(0,0,1)(\alpha_k,\beta_k,\gamma_k)=(0,0,1) by appropriately designed σtk\sigma_{t_k} and tt under the same velocity vθv_\theta to match the dynamic of VINE.

Methodxk+1x_{k+1}a^k\hat a_kz^k\hat z_kNOTE
VINEtk+1a^k+(1−tk+1)z^kt_{k+1}\hat a_k+(1-t_{k+1})\hat z_kxk+(1−tk)vθ(xk,tk)x_k+(1-t_k)v_\theta(x_k,t_k)z∼N(0,Id)z\sim\mathcal N(0,I_d)Fresh noise independent of θ\theta
Euler ODEtk+1a^k+(1−tk+1)z^kt_{k+1}\hat a_k+(1-t_{k+1})\hat z_kxk+(1−tk)vθ(xk,tk)x_k+(1-t_k)v_\theta(x_k,t_k)xk−tkvθ(xk,tk)x_k-t_kv_\theta(x_k,t_k)z^k\hat z_k is deterministic given xkx_k and θ\theta
Euler SDEtk+1a^k+(1−tk+1)z^kt_{k+1}\hat a_k+(1-t_{k+1})\hat z_kxk+(1−tk)vθ(xk,tk)x_k+(1-t_k)v_\theta(x_k,t_k)αkxk+βkvθ(xk,tk)+γkz\alpha_kx_k+\beta_kv_\theta(x_k,t_k)+\gamma_kz(αk,βk,γk)≠(0,0,1)(\alpha_k,\beta_k,\gamma_k)\neq(0,0,1)
DDPMαk+1a^k+σk+1z^k\alpha_{k+1}\hat a_k+\sigma_{k+1}\hat z_kxk−σkϵθ(xk,k)αk\frac{x_k-\sigma_k\epsilon_\theta(x_k,k)}{\alpha_k}Zk(τk)Z_k(\tau_k)τk\tau_k defined below
DDIMαk+1a^k+σk+1z^k\alpha_{k+1}\hat a_k+\sigma_{k+1}\hat z_kxk−σkϵθ(xk,k)αk\frac{x_k-\sigma_k\epsilon_\theta(x_k,k)}{\alpha_k}Zk(ρk)Z_k(\rho_k)σ>0, ρk=ητk\sigma>0,\ \rho_k=\eta\tau_k

For compactness, define

Zk(c)=1−(c/σk+1)2ϵθ(xk,k)+(c/σk+1)z,Z_k(c)=\sqrt{1-(c/\sigma_{k+1})^2}\epsilon_\theta(x_k,k)+(c/\sigma_{k+1})z,

τk=σk+1σk1−αk2αk+12.\tau_k=\frac{\sigma_{k+1}}{\sigma_k}\sqrt{1-\frac{\alpha_k^2}{\alpha_{k+1}^2}}.

The result show that we can't appropriately design such SDE-based samplers to recover VINE's independent-noise reconstruction as an update rule within this family while retaining the same endpoint estimator. For completeness, the table below also compares the corresponding updates of DDPM and DDIM. In both cases, the effective noise term includes a predicted noise component in addition to fresh noise, so the VINE is different from these SDE samplers.

We thank the reviewer for this helpful suggestion and we will incorporate this comparison and detailed analysis into final appendix.

Reference

[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.

[2] Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.

[3] Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861, 2024.

Reviewer_gTsAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VINE, a sampling strategy for stabilizing value-gradient reinforcement learning with generative control policies. By reconstructing intermediate states using fresh Gaussian noise, VINE enables full-chain backpropagation without additional velocity-network evaluations. The method is evaluated on OGBench, LIBERO with a pretrained π₀.₅ VLA, and real-world robotic manipulation.

Critical strengths
  1. Important problem. Stable reinforcement learning with expressive, iterative generative policies is an important problem for robotic control. The paper investigates a complementary direction to methods that simplify the policy or avoid full-chain value-gradient propagation.

  2. Simple and practical method. VINE modifies the sampling procedure without introducing a new policy architecture or additional velocity-network evaluations. This makes the approach potentially easy to integrate into existing flow-based policies.

  3. Broad empirical evaluation. The experiments cover 50 OGBench tasks, online adaptation of a pretrained 3B-parameter VLA, and real-world robotic manipulation. The reported results are encouraging, particularly the 20/20 successful plug-insertion trials after 20 minutes of online fine-tuning.

Critical weaknesses
  1. Unclear source of performance improvements. VINE simultaneously modifies the forward sampling dynamics and backward gradient propagation. It is therefore unclear whether the improvements primarily arise from stochastic exploration, improved gradient propagation, or their combination. The current ablations do not sufficiently disentangle these factors. More controlled experiments are needed to support the central claim.

  2. Insufficient distinction from stochastic sampling methods. VINE introduces fresh Gaussian noise at intermediate sampling steps, which is closely related to stochastic flow sampling. Although the paper includes an SDE baseline, the fundamental differences between VINE and existing SDE-based samplers are not sufficiently analyzed. A more systematic comparison of their sampling dynamics, induced distributions, and gradient properties would help clarify the novelty of the proposed approach.

  3. Incomplete discussion of related work. Several relevant generative-policy RL methods are missing or insufficiently discussed, including DIPO (Yang et al., 2023), CGPO (Ding et al., 2026), GenPO (Ding et al., 2025), GenPO++ (Hu et al., 2026), and OGPO (Patil et al., 2026).

[1] Yang, L., et al. (2023). Policy Representation via Diffusion Probability Model for Reinforcement Learning. arXiv:2305.13122. (DIPO)

[2] Ding, S., et al. (2026). Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance. ICML 2026. (CGPO)

[3] Ding, S., et al. (2025). GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning. NeurIPS 2025.

[4] Hu, K., et al. (2026). GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios. arXiv:2606.06967.

[5] Patil, S., et al. (2026). OGPO: Sample Efficient Full-Finetuning of Generative Control Policies. ICML 2026.

Questions for authors
  1. Can the authors disentangle the contributions of stochastic exploration and gradient propagation through additional controlled ablations?

  2. What fundamentally distinguishes VINE from appropriately designed SDE-based samplers, particularly in terms of sampling distributions and gradient propagation?

  3. How sensitive is VINE to the number of sampling steps and the noise resampling schedule? Does its advantage over standard flow matching persist across different sampling budgets?

  4. Can VINE also benefit on-policy reinforcement learning? Since the current experiments primarily focus on off-policy RL, it would be interesting to investigate whether VINE can improve on-policy generative policy optimization methods such as GenPO and FPO. In particular, can VINE improve learning-from-scratch performance and sample efficiency on IsaacLab or MuJoCo benchmarks?

  5. The real-world experiments are encouraging but focus on a single plug-insertion task. How well does VINE generalize to other manipulation tasks, particularly those requiring different types of contact-rich interactions?

I am willing to raise my score if my questions are well addressed.

7 comments

Thank you for the detailed review. We have conducted additional experiments and analysis to address your questions about the source of VINE's improvements and its relation to prior work. Please find our response below.


Q: Can the authors disentangle the contributions of stochastic exploration and gradient propagation?

A: Thank you for this question. VINE changes both the forward sampling dynamics and the gradients propagated through the sampling chain. To test whether the backward modification alone accounts for its benefit, the original appendix (lines 898--907) presents an Euler--VINE-SG ablation. Let Eθ,kE_{\theta,k} and Vθ,kV_{\theta,k} denote the Euler and VINE step maps, respectively. Using sg⁡\operatorname{sg} to denote stop-gradient, this ablation is defined as

ak+1=sg⁡(Eθ,k(ak))+Vθ,k(ak,zk)−sg⁡(Vθ,k(ak,zk)).a_{k+1} = \operatorname{sg}(E_{\theta,k}(a_k)) + V_{\theta,k}(a_k,z_k) - \operatorname{sg}(V_{\theta,k}(a_k,z_k)).

Its forward pass follows the Euler update, while automatic differentiation uses VINE derivatives along the Euler trajectory. We additionally evaluate the complementary VINE--Euler-SG ablation, whose forward pass follows the VINE update while its gradients use Euler derivatives:

ak+1=sg⁡(Vθ,k(ak,zk))+Eθ,k(ak)−sg⁡(Eθ,k(ak)).a_{k+1} = \operatorname{sg}(V_{\theta,k}(a_k,z_k)) + E_{\theta,k}(a_k) - \operatorname{sg}(E_{\theta,k}(a_k)).

The following results are obtained on AntMaze task 2 after 1M1 M gradient updates, averaged over eight seeds (10001--10008):

OGbenchSuccess rate (%)Note
VINE8787 [83,92][83,92]VINE dynamics, VINE gradient propagation
Euler2020 [7,34][7,34]Euler dynamics, Euler gradient propagation
Euler--VINE-SG33 [0,5][0,5]Euler dynamics, VINE gradient propagation
VINE--Euler-SG1717 [5,29][5,29]VINE dynamics, Euler gradient propagation

These results suggest that, in this setting, VINE's gains depend on the combination of its sampling dynamics and corresponding gradient propagation. We suspect that combining the dynamics of one sampler with the derivatives of the other creates a mismatch during optimization, which may explain the poor performance of the control experiments.

AuthorsOpenReview ↗

Fundamentally distinguishes VINE from SDE-based samplers

Q: What fundamentally distinguishes VINE from appropriately designed SDE-based samplers?

A: Thank you for suggesting a more systematic comparison. To distinguish this, we try to provide some algebraic math comparison to make it more clear. We first specify the SDE-based samplers family considered here[1][2][3]. Following the notation of paper, its Euler SDE sampler update is

xk+1=xk+[vk−σtk22tk(xk+(1−tk)vk)]Δtk+σtkΔtkzk,x_{k+1} = x_k + [v_k-\frac{\sigma_{t_k}^{2}}{2t_k}(x_k+(1-t_k)v_k)]\Delta t_k + \sigma_{t_k}\sqrt{\Delta t_k}z_k,

where vk=vθ(xk,tk)v_k=v_\theta(x_k,t_k), Δtk=tk+1−tk\Delta t_k=t_{k+1}-t_k, and zk∼N(0,I)z_k\sim\mathcal{N}(0,I) is independent Gaussian noise. The nonnegative function σtk\sigma_{t_k} is tunable parameters; setting σt=0\sigma_t=0 recovers the Euler ODE update. Motivated by the stochastic-interpolant framework [2], we rewrite these discrete updates in a common interpolation form:

xk+1=tk+1a^k+(1−tk+1)z^k,a^k=xk+(1−tk)vk.x_{k+1} = t_{k+1}\hat a_k+(1-t_{k+1})\hat z_k, \quad \hat a_k=x_k+(1-t_k)v_k.

Here, a^k\hat a_k is the action-endpoint estimate and z^k\hat z_k is an noise-side endpoint term. For the SDE update above,

z^kSDE=αkxk+βkvk+γkzk,\hat z_k^{\mathrm{SDE}} = \alpha_kx_k+\beta_kv_k+\gamma_kz_k,

where

αk=1−σtk2Δtk2tk(1−tk+1),\alpha_k = 1-\frac{\sigma_{t_k}^{2}\Delta t_k}{2t_k(1-t_{k+1})}, βk=−tk−σtk2(1−tk)Δtk2tk(1−tk+1),\beta_k = -t_k-\frac{\sigma_{t_k}^{2}(1-t_k)\Delta t_k}{2t_k(1-t_{k+1})}, γk=σtkΔtk1−tk+1.\gamma_k = \frac{\sigma_{t_k}\sqrt{\Delta t_k}}{1-t_{k+1}}.

In comparison, VINE predicts the endpoint and reconstructs the next intermediate state using fresh Gaussian noise:

xk+1=tk+1a^k+(1−tk+1)zk,zk∼N(0,I).x_{k+1} = t_{k+1}\hat a_k+(1-t_{k+1})z_k, \quad z_k\sim\mathcal{N}(0,I).

Thus, the updates have a similar interpolation structure, but construct their effective noise z^k\hat z_k differently: the SDE combines the current state, predicted velocity, and fresh noise, whereas VINE uses fresh noise directly. In particular, the SDE-based samplers coefficients satisfy

βk−(1−tk)αk=−1.\beta_k-(1-t_k)\alpha_k=-1.

which means we can not got (αk,βk,γk)=(0,0,1)(\alpha_k,\beta_k,\gamma_k)=(0,0,1) by appropriately designed σtk\sigma_{t_k} and tt under the same velocity vθv_\theta.

The result show that we can't appropriately design such SDE-based samplers to recover VINE's independent-noise reconstruction as an update rule within this family while retaining the same endpoint estimator. For completeness, the table below also compares the corresponding updates of DDPM and DDIM. In both cases, the effective noise term includes a predicted noise component in addition to fresh noise.

MethodGeneral form xk+1x_{k+1}a^k\hat a_kz^k\hat z_kNote
VINEtk+1a^k+(1−tk+1)z^kt_{k+1}\hat a_k+(1-t_{k+1})\hat z_kxk+(1−tk)vθ(xk,tk)x_k+(1-t_k)v_\theta(x_k,t_k)z∼N(0,Id)z\sim\mathcal N(0,I_d)Fresh noise independent of θ\theta
Euler ODEtk+1a^k+(1−tk+1)z^kt_{k+1}\hat a_k+(1-t_{k+1})\hat z_kxk+(1−tk)vθ(xk,tk)x_k+(1-t_k)v_\theta(x_k,t_k)xk−tkvθ(xk,tk)x_k-t_kv_\theta(x_k,t_k)Deterministic given xkx_k and θ\theta
Euler SDEtk+1a^k+(1−tk+1)z^kt_{k+1}\hat a_k+(1-t_{k+1})\hat z_kxk+(1−tk)vθ(xk,tk)x_k+(1-t_k)v_\theta(x_k,t_k)αkxk+βkvθ(xk,tk)+γkz\alpha_kx_k+\beta_kv_\theta(x_k,t_k)+\gamma_kz(αk,βk,γk)≠(0,0,1)(\alpha_k,\beta_k,\gamma_k)\neq(0,0,1)
DDPMαk+1a^k+σk+1z^k\alpha_{k+1}\hat a_k+\sigma_{k+1}\hat z_kxk−σkϵθ(xk,k)αk\frac{x_k-\sigma_k\epsilon_\theta(x_k,k)}{\alpha_k}Zk(τk)Z_k(\tau_k)τk\tau_k defined below
DDIMαk+1a^k+σk+1z^k\alpha_{k+1}\hat a_k+\sigma_{k+1}\hat z_kxk−σkϵθ(xk,k)αk\frac{x_k-\sigma_k\epsilon_\theta(x_k,k)}{\alpha_k}Zk(ρk)Z_k(\rho_k)σk+1>0\sigma_{k+1}>0, ρk=ητk\rho_k=\eta\tau_k

For compactness, define

Zk(c)=1−(c/σk+1)2ϵθ(xk,k)+(c/σk+1)z,Z_k(c)=\sqrt{1-(c/\sigma_{k+1})^2}\epsilon_\theta(x_k,k)+(c/\sigma_{k+1})z,

τk=σk+1σk1−αk2αk+12.\tau_k=\frac{\sigma_{k+1}}{\sigma_k}\sqrt{1-\frac{\alpha_k^2}{\alpha_{k+1}^2}}.


And the gradient dynamic, Let Jv=∂vθ(xk,tk;s)/∂xkJ_v=\partial v_\theta(x_k,t_k;s)/\partial x_k and Jϵ=∂ϵθ(xk,k;s)/∂xkJ_\epsilon=\partial\epsilon_\theta(x_k,k;s)/\partial x_k. The local state-transition Jacobians are:

Method∂xk+1/∂xk\partial x_{k+1}/\partial x_k
VINEtk+1[Id+(1−tk)Jv]t_{k+1}[I_d+(1-t_k)J_v]
Euler ODEId+ΔtkJvI_d+\Delta t_k J_v
Euler SDE[1−σtk2Δtk/(2tk)]Id+Δtk[1−σtk2(1−tk)/(2tk)]Jv[1-\sigma_{t_k}^2\Delta t_k/(2t_k)]I_d+\Delta t_k[1-\sigma_{t_k}^2(1-t_k)/(2t_k)]J_v
DDPM(αk+1/αk)Id+[σk+12−τk2−αk+1σk/αk]Jϵ(\alpha_{k+1}/\alpha_k)I_d+[\sqrt{\sigma_{k+1}^2-\tau_k^2}-\alpha_{k+1}\sigma_k/\alpha_k]J_\epsilon
DDIM(αk+1/αk)Id+[σk+12−ρk2−αk+1σk/αk]Jϵ(\alpha_{k+1}/\alpha_k)I_d+[\sqrt{\sigma_{k+1}^2-\rho_k^2}-\alpha_{k+1}\sigma_k/\alpha_k]J_\epsilon, ρk=ητk\rho_k=\eta\tau_k

We thank the reviewer for this helpful suggestion and we will incorporate this comparison and detailed analysis into final appendix!

Reference

[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.

[2] Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.

[3] Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861, 2024.

AuthorsOpenReview ↗

VINE to on-policy RL

Q: Incomplete discussion of related work about on-policy methods

A: Thank you for pointing out these works. DIPO, CGPO, GenPO, GenPO++, and OGPO study generative-policy RL for on-policy or PPO-style optimization. VINE instead study the stability of generative-policy RL for end-to-end value-gradient RL or off-policy framework. We will add these connections to related work in the final version.


Q: Whether VINE can improve on-policy generative policy optimization methods?

A: Thank for pointing out direction. Our current experiments focus on value-gradient RL. Extending VINE to on-policy generative policy optimization is an interesting direction for future work.

Extending VINE to likelihood-based on-policy optimization requires computing the action likelihood. One possible alternative is to optimize the likelihood of the full sampling path follow OGPO. Since VINE's intermediate transitions are Gaussian, we can compute their likelihoods directly, which is

pθ(xk+1∣xk,s)=N(xk+1;tk+1a^θ(xk,tk,s),(1−tk+1)2Id).p_\theta(x_{k+1}\mid x_k,s) = \mathcal N(x_{k+1}; t_{k+1}\hat a_\theta(x_k,t_k,s), (1-t_{k+1})^2I_d).

By adding a small Gaussian noise to the deterministic final step, the full path likelihood becomes

pθ(τ∣s)=p(x0)∏k=0K−1pθ(xk+1∣xk,s),p_\theta(\tau\mid s) = p(x_0)\prod_{k=0}^{K-1}p_\theta(x_{k+1}\mid x_k,s),

where τ=(x0,…,xK)\tau=(x_0,\ldots,x_K) and xKx_K is the action. We can then use the path ratio pθ(τ∣s)/pold(τ∣s)p_\theta(\tau\mid s)/p_{\mathrm{old}}(\tau\mid s) in a PPO-style objective. We leave its recipe of empirical study to future work and clarify this limitation in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.