- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the exploration–exploitation trade-off for diffusion and flow-matching policies in both online RL and offline-to-online RL post-training. The key idea is to decouple exploration and exploitation into two separate generative policies and compose their score or velocity fields during sampling through classifier-free guidance. The proposed framework is instantiated on top of QSM and QVPO for online RL, showing consistent improvements across the evaluated benchmarks. For RL post-training, the method is evaluated against direct Flow-SDE+PPO fine-tuning and demonstrates consistent performance gains while preserving the pretrained policy via a frozen generative branch.
Critical strengths
The method is conceptually simple and straightforward to implement. Its main appeal lies in providing a unified framework that can be applied to both generative RL and RL post-training. In addition, explicitly decoupling the exploration and exploitation policies offers a natural mechanism for preserving the capability encoded in the frozen branch while optimizing the other branch.
Critical weaknesses
W1. The characterization of Residual RL in the related-work comparison is not fully convincing.
The paper categorizes Residual RL as lacking exploration, but this characterization appears overly restrictive. In practice, residual policies are typically optimized with standard on-policy or off-policy RL algorithms, both of which generally rely on stochasticity or other exploration mechanisms during training. Therefore, it would be helpful for the authors to clarify the criterion used to label Residual RL as not supporting exploration in Table 1.
W2. The claim that Residual RL suffers from limited exploration is not sufficiently validated empirically.
Although the paper motivates CFG-Policy in part by arguing that action-space residual methods provide limited exploration, the experiments do not directly isolate or verify whether this limitation is in fact responsible for the observed performance gap. A controlled comparison against a residual RL baseline under the same reward, interaction budget, and backbone would substantially strengthen this claim.
W3. The implementation details of the RL post-training setting are underspecified, which may hinder reproducibility.
In particular, the paper does not clearly specify the reward function used to train CFG-Policy in the post-training experiments. This is especially important because the method relies on PPO and GAE, whose behavior can depend substantially on whether the reward is sparse, dense, or task-shaped. Moreover, if the trainable branch is optimized directly with the task reward, the terminology of referring to this branch as an “exploration policy” may require further clarification, since its objective appears to be directly aligned with task performance.
W4. The timing and integration of adaptive composition during training are unclear.
The paper introduces an adaptive composition strategy for adjusting the guidance weight, but it is not sufficiently specified when and how this procedure is invoked during training relative to policy and critic updates. Clarifying the update schedule, frequency, and interaction with the base RL optimizer would improve reproducibility.
Minor: There appears to be a citation/reference rendering error in Line 126, where “(?)” is shown in the discussion of classifier-free guidance.
Questions for authors
Q1. Criteria used in Table 1.
Could the authors clarify the operational criteria used to assign ✓, △, and ✗ for Exploration and Adaptive E-E in Table 1? In particular, it is unclear why DPPO is considered to provide only partial exploration, ReinFlow explicit exploration but only partial adaptive E-E, and Flow-SDE/Flow-Noise explicit support for both. I am also not fully convinced by classifying Residual RL as having no exploration, since residual policies trained with standard on-policy or off-policy RL algorithms typically inherit stochastic exploration from the underlying optimizer. A precise definition of what counts as “exploration” and “adaptive exploration–exploitation” would materially affect how I interpret the claimed distinction from prior work.
Q2. Reward specification in RL post-training.
What exact reward function is used for the RoboTwin RL post-training experiments with Flow-SDE+PPO and CFG-Policy? Please clarify whether the reward is sparse success, dense/shaped, or task-specific, and how it is assigned over action chunks. Relatedly, if the trainable branch is directly optimized using the task reward, could the authors clarify in what sense this branch should be interpreted as an “exploration” policy, rather than a task-adaptation or steering policy?
Q3. Adaptive composition schedule.
Could the authors specify exactly how the adaptive composition procedure is integrated into training? In particular, when and how often are and the associated dual variable updated relative to the actor/critic updates, and which reported experiments use adaptive composition versus a fixed or manually tuned guidance weight? The current description makes it difficult to reconstruct the full training procedure.
Q4. Controlled comparison with Residual RL.
The paper motivates CFG-Policy partly by arguing that action-space residual methods provide limited exploration, but the current experiments do not appear to isolate this factor. Could the authors provide, or comment on the feasibility of, a comparison against a residual RL baseline using the same pretrained backbone, reward, interaction budget, and RL optimizer? Such a comparison would directly test whether generative-space composition provides a meaningful advantage over action-space residual adaptation.
5 comments
W3&Q2: Insufficient RL post-training details. The reward function used in RoboTwin RL post-training is unclear, potentially affecting reproducibility. Please specify whether Flow-SDE+PPO and CFG-Policy use sparse, dense, or shaped rewards, and how rewards are assigned across action chunks. Additionally, since the trainable branch is directly optimized for task rewards, why is it termed an “exploration policy” rather than a task-adaptation or steering policy?
A2:
- Reward specification. All RoboTwin RL post-training experiments (Flow-SDE+PPO and CFG-Policy) use the same binary sparse success reward. Each action chunk receives (r=0), except the chunk triggering task success, which receives (r=1) based on RoboTwin's task-specific success checker. Rewards are propagated through GAE, with all actions within a chunk sharing the same advantage. No dense, shaped, or additional task-specific rewards are used.
- Why “exploration policy” rather than “steering” or “task-adaptation policy”?: We clarify that these concepts are not mutually exclusive, but emphasize different aspects of policy learning. Task adaptation typically improves task performance by fine-tuning a pretrained policy (e.g., DPPO[1]), while policy steering redirects pretrained behavior through mechanisms such as latent-noise control (DSRL[2]) or action selection. CFG-Policy differs in its explicit separation of behavioral capabilities. Rather than directly modifying the pretrained policy, we maintain a frozen exploitation branch and independently optimize a second generative policy using PPO. Their velocity fields are composed throughout flow sampling: This design allows the trainable branch to discover alternative task-solving modes while preserving the pretrained behavioral prior. As illustrated in Fig. 6, CFG-Policy achieves broader trajectory-state coverage while retaining behaviors represented by the pretrained policy. Therefore, we term it an exploration policy to emphasize its role in expanding the behavioral distribution beyond the frozen pretrained policy, rather than implying a reward-free exploration objective.
[1] Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Wagenmaker, Andrew, et al. "Steering your diffusion policy with latent space reinforcement learning." arXiv preprint arXiv:2506.15799 (2025).
W4&Q3: Unclear adaptive composition schedule. The integration of adaptive composition into training is insufficiently specified, hindering reproducibility. Clarify when and how frequently the guidance weight (w) and its dual variable are updated relative to actor/critic updates, and which experiments use adaptive composition versus fixed or manually tuned guidance weights.
A3: Thank you for pointing out this ambiguity. We clarify the schedule and integration below.
-
When is adaptive composition activated? It starts after the shared warmup of 10,000 environment steps. We initialize the composition weight to 0.8 and calibrate the auxiliary path-KL target once:
w_0=0.8,\qquad D_{\mathrm{target}}=D_{T_{\mathrm{warm}}}(w_0).$$ The target remains fixed throughout subsequent training. -
How often and in what order are the variables updated? After learning begins, the original actor/critic updates occur once per environment step. After warmup, the adaptive composition controller runs every 10 learner updates, following the actor/critic and target-network updates. The learner counter includes warmup updates. Both and remain unchanged between controller calls.
The controller maximizes action value subject to the lower-bound constraint :
\mathcal L(w,\lambda),\qquad \mathcal L(w,\lambda)=\widehat Q(w) +\lambda\bigl(D(w)-D_{\mathrm{target}}\bigr).$$ Using shared replay states and diffusion noise, it estimates, for $F\in\{\widehat Q,D\}$, $${\partial_w F}≈\frac{F(w_+)-F(w_-)}{w_+-w_-}, \qquad w_\pm=\operatorname{clip}(w\pm0.01,0,1).$$ We parameterize $w=\sigma(u)$ and $\lambda=\exp(z)$, initialized with $u_0=\log(4)$ and $z_0=0$. The descent gradients are $$g_{u,k} =-\left( {\partial_w Q_k} +\lambda_k{\partial_w D_k} \right)w_k(1-w_k), \qquad r_k=D_k(w_k)-D_{\mathrm{target}},$$ $$\widetilde g_{u,k}=\operatorname{clip}(g_{u,k},-10,10), \qquad g_{z,k}=\operatorname{clip}(\lambda_k r_k,-10,10).$$ Separate Adam optimizers update $u$ and $z$, with learning rates $2\times10^{-5}$ and $10^{-4}$, respectively. In particular, the dual update is $$z_{k+1} =\operatorname{clip}\left( z_k-10^{-4} \frac{\widehat m_k}{\sqrt{\widehat v_k}+\epsilon}, -12,12 \right), \qquad \lambda_{k+1}=\exp(z_{k+1}),$$ where $\widehat m_k$ and $\widehat v_k$ are Adam’s bias-corrected moments of $g_{z,k}$. Both scalar updates use the same pre-update measurements. A negative residual gives a dual descent direction that strengthens the constraint penalty. -
How does this interact with the RL optimizer, and which experiments use it?
Controller measurements use the updated networks while holding their parameters fixed. Only and are optimized; the controller does not backpropagate into the actor/critic or alter their optimizer states. The new weight is used in subsequent environment interactions. Adaptive composition is used in online reinforcement learning experiments. Fixed-weight of , or are used in RL post-training experiments.
Adaptive Algorithm
Run the shared 10k-step warmup; retain the learner counter n
Initialize u = log(4), z = 0, w = 0.8, λ = 1
Calibrate D_target = D_Twarm(0.8)
For each subsequent environment step:
Collect and store a transition using the current composed policy
Perform the original actor/critic and target-network updates
Increment n
If joint exploration: update the exploration branch using RND rewards
If n mod 10 == 0:
Sample replay states and shared diffusion noise
Hold network parameters fixed
Measure Q and D at w, w−, and w+
Compute both scalar gradients from these pre-update measurements
Update u and z using their separate Adam optimizers
Clamp z to [−12, 12]; set w = sigmoid(u), λ = exp(z)
Otherwise:
Keep w and λ unchanged
We have added this schedule, the update rules, and the algorithm box to the revised paper.
Thank you for your valuable feedback and constructive comments! We itemize the weaknesses or comments you mentioned and answers to them.
A0: Thank you for pointing it out. We have fixed this issue in the revised paper.
A1: Thank you for the comments. We clarify that Table 1 evaluates dedicated mechanisms for multi-modal exploration, rather than exploratory capability alone. We adopt a star-based scale in the revision: ★★★ for explicit support, ★★ for partial or indirect support, and ★ for no dedicated mechanism. DPPO relies on diffusion stochasticity and noise-schedule adjustments (★★), whereas ReinFlow and Flow-Noise introduce learnable noise injection, and Flow-SDE specifically constructs stochastic dynamics via ODE-to-SDE conversion (★★★). For adaptive E–E, ReinFlow indirectly adjusts exploration through joint noise-policy optimization (★★), while Flow-Noise employs a learnable noise schedule and Flow-SDE explicitly controls stochasticity via its diffusion coefficient and sampling strategy (★★★). Notably, explicit control does not necessarily imply automatic optimization. For residual RL, ★ indicates that residual policy composition lacks a dedicated multi-modal exploration mechanism, although standard RL optimizers may still induce stochastic exploration. Importantly, CFG-Policy uniquely decouples exploration and exploitation into independently preservable and optimizable generative policies.