acceptodds
Under review as a conference paper at ICLR 2027

CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

28 views

Abstract

Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks.

Primary area:
reinforcement learning
Keywords:
Diffusion Reinforcement Learning, Classifier-Free Guidance, Generative Policies, Exploration and Exploitation, Policy Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yrpaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the exploration–exploitation trade-off for diffusion and flow-matching policies in both online RL and offline-to-online RL post-training. The key idea is to decouple exploration and exploitation into two separate generative policies and compose their score or velocity fields during sampling through classifier-free guidance. The proposed framework is instantiated on top of QSM and QVPO for online RL, showing consistent improvements across the evaluated benchmarks. For RL post-training, the method is evaluated against direct Flow-SDE+PPO fine-tuning and demonstrates consistent performance gains while preserving the pretrained policy via a frozen generative branch.

Critical strengths

The method is conceptually simple and straightforward to implement. Its main appeal lies in providing a unified framework that can be applied to both generative RL and RL post-training. In addition, explicitly decoupling the exploration and exploitation policies offers a natural mechanism for preserving the capability encoded in the frozen branch while optimizing the other branch.

Critical weaknesses

W1. The characterization of Residual RL in the related-work comparison is not fully convincing.

The paper categorizes Residual RL as lacking exploration, but this characterization appears overly restrictive. In practice, residual policies are typically optimized with standard on-policy or off-policy RL algorithms, both of which generally rely on stochasticity or other exploration mechanisms during training. Therefore, it would be helpful for the authors to clarify the criterion used to label Residual RL as not supporting exploration in Table 1.

W2. The claim that Residual RL suffers from limited exploration is not sufficiently validated empirically.

Although the paper motivates CFG-Policy in part by arguing that action-space residual methods provide limited exploration, the experiments do not directly isolate or verify whether this limitation is in fact responsible for the observed performance gap. A controlled comparison against a residual RL baseline under the same reward, interaction budget, and backbone would substantially strengthen this claim.

W3. The implementation details of the RL post-training setting are underspecified, which may hinder reproducibility.

In particular, the paper does not clearly specify the reward function used to train CFG-Policy in the post-training experiments. This is especially important because the method relies on PPO and GAE, whose behavior can depend substantially on whether the reward is sparse, dense, or task-shaped. Moreover, if the trainable branch is optimized directly with the task reward, the terminology of referring to this branch as an “exploration policy” may require further clarification, since its objective appears to be directly aligned with task performance.

W4. The timing and integration of adaptive composition during training are unclear.

The paper introduces an adaptive composition strategy for adjusting the guidance weight, but it is not sufficiently specified when and how this procedure is invoked during training relative to policy and critic updates. Clarifying the update schedule, frequency, and interaction with the base RL optimizer would improve reproducibility.

Minor: There appears to be a citation/reference rendering error in Line 126, where “(?)” is shown in the discussion of classifier-free guidance.

Questions for authors

Q1. Criteria used in Table 1.

Could the authors clarify the operational criteria used to assign ✓, △, and ✗ for Exploration and Adaptive E-E in Table 1? In particular, it is unclear why DPPO is considered to provide only partial exploration, ReinFlow explicit exploration but only partial adaptive E-E, and Flow-SDE/Flow-Noise explicit support for both. I am also not fully convinced by classifying Residual RL as having no exploration, since residual policies trained with standard on-policy or off-policy RL algorithms typically inherit stochastic exploration from the underlying optimizer. A precise definition of what counts as “exploration” and “adaptive exploration–exploitation” would materially affect how I interpret the claimed distinction from prior work.

Q2. Reward specification in RL post-training.

What exact reward function is used for the RoboTwin RL post-training experiments with Flow-SDE+PPO and CFG-Policy? Please clarify whether the reward is sparse success, dense/shaped, or task-specific, and how it is assigned over action chunks. Relatedly, if the trainable branch is directly optimized using the task reward, could the authors clarify in what sense this branch should be interpreted as an “exploration” policy, rather than a task-adaptation or steering policy?

Q3. Adaptive composition schedule.

Could the authors specify exactly how the adaptive composition procedure is integrated into training? In particular, when and how often are ww and the associated dual variable updated relative to the actor/critic updates, and which reported experiments use adaptive composition versus a fixed or manually tuned guidance weight? The current description makes it difficult to reconstruct the full training procedure.

Q4. Controlled comparison with Residual RL.

The paper motivates CFG-Policy partly by arguing that action-space residual methods provide limited exploration, but the current experiments do not appear to isolate this factor. Could the authors provide, or comment on the feasibility of, a comparison against a residual RL baseline using the same pretrained π0/π0.5\pi_0/\pi_{0.5} backbone, reward, interaction budget, and RL optimizer? Such a comparison would directly test whether generative-space composition provides a meaningful advantage over action-space residual adaptation.

5 comments

Thank you for your valuable feedback and constructive comments! We itemize the weaknesses or comments you mentioned and answers to them.

Minor: There appears to be a citation/reference rendering error in Line 126, where “(?)” is shown in the discussion of classifier-free guidance.

A0: Thank you for pointing it out. We have fixed this issue in the revised paper.

W1&Q1: Clarification on the Policy-Gradient Tuning method and the residual RL in the Related Work.

A1: Thank you for the comments. We clarify that Table 1 evaluates dedicated mechanisms for multi-modal exploration, rather than exploratory capability alone. We adopt a star-based scale in the revision: ★★★ for explicit support, ★★ for partial or indirect support, and ★ for no dedicated mechanism. DPPO relies on diffusion stochasticity and noise-schedule adjustments (★★), whereas ReinFlow and Flow-Noise introduce learnable noise injection, and Flow-SDE specifically constructs stochastic dynamics via ODE-to-SDE conversion (★★★). For adaptive E–E, ReinFlow indirectly adjusts exploration through joint noise-policy optimization (★★), while Flow-Noise employs a learnable noise schedule and Flow-SDE explicitly controls stochasticity via its diffusion coefficient and sampling strategy (★★★). Notably, explicit control does not necessarily imply automatic optimization. For residual RL, ★ indicates that residual policy composition lacks a dedicated multi-modal exploration mechanism, although standard RL optimizers may still induce stochastic exploration. Importantly, CFG-Policy uniquely decouples exploration and exploitation into independently preservable and optimizable generative policies.

W3&Q2: Insufficient RL post-training details. The reward function used in RoboTwin RL post-training is unclear, potentially affecting reproducibility. Please specify whether Flow-SDE+PPO and CFG-Policy use sparse, dense, or shaped rewards, and how rewards are assigned across action chunks. Additionally, since the trainable branch is directly optimized for task rewards, why is it termed an “exploration policy” rather than a task-adaptation or steering policy?

A2:

  1. Reward specification. All RoboTwin RL post-training experiments (Flow-SDE+PPO and CFG-Policy) use the same binary sparse success reward. Each action chunk receives (r=0), except the chunk triggering task success, which receives (r=1) based on RoboTwin's task-specific success checker. Rewards are propagated through GAE, with all actions within a chunk sharing the same advantage. No dense, shaped, or additional task-specific rewards are used.
  2. Why “exploration policy” rather than “steering” or “task-adaptation policy”?: We clarify that these concepts are not mutually exclusive, but emphasize different aspects of policy learning. Task adaptation typically improves task performance by fine-tuning a pretrained policy (e.g., DPPO[1]), while policy steering redirects pretrained behavior through mechanisms such as latent-noise control (DSRL[2]) or action selection. CFG-Policy differs in its explicit separation of behavioral capabilities. Rather than directly modifying the pretrained policy, we maintain a frozen exploitation branch and independently optimize a second generative policy using PPO. Their velocity fields are composed throughout flow sampling: vCFG=(1−w)vexploit+wvexplore.v_{\mathrm{CFG}}=(1-w)v_{\mathrm{exploit}}+wv_{\mathrm{explore}}. This design allows the trainable branch to discover alternative task-solving modes while preserving the pretrained behavioral prior. As illustrated in Fig. 6, CFG-Policy achieves broader trajectory-state coverage while retaining behaviors represented by the pretrained policy. Therefore, we term it an exploration policy to emphasize its role in expanding the behavioral distribution beyond the frozen pretrained policy, rather than implying a reward-free exploration objective.

[1] Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Wagenmaker, Andrew, et al. "Steering your diffusion policy with latent space reinforcement learning." arXiv preprint arXiv:2506.15799 (2025).

W4&Q3: Unclear adaptive composition schedule. The integration of adaptive composition into training is insufficiently specified, hindering reproducibility. Clarify when and how frequently the guidance weight (w) and its dual variable are updated relative to actor/critic updates, and which experiments use adaptive composition versus fixed or manually tuned guidance weights.

A3: Thank you for pointing out this ambiguity. We clarify the schedule and integration below.

  1. When is adaptive composition activated? It starts after the shared warmup of 10,000 environment steps. We initialize the composition weight to 0.8 and calibrate the auxiliary path-KL target once:

    w_0=0.8,\qquad D_{\mathrm{target}}=D_{T_{\mathrm{warm}}}(w_0).$$ The target remains fixed throughout subsequent training.
  2. How often and in what order are the variables updated? After learning begins, the original actor/critic updates occur once per environment step. After warmup, the adaptive composition controller runs every 10 learner updates, following the actor/critic and target-network updates. The learner counter includes warmup updates. Both ww and λ\lambda remain unchanged between controller calls.

    The controller maximizes action value subject to the lower-bound constraint D(w)≥DtargetD(w)\ge D_{\mathrm{target}}:

    \mathcal L(w,\lambda),\qquad \mathcal L(w,\lambda)=\widehat Q(w) +\lambda\bigl(D(w)-D_{\mathrm{target}}\bigr).$$ Using shared replay states and diffusion noise, it estimates, for $F\in\{\widehat Q,D\}$, $${\partial_w F}≈\frac{F(w_+)-F(w_-)}{w_+-w_-}, \qquad w_\pm=\operatorname{clip}(w\pm0.01,0,1).$$ We parameterize $w=\sigma(u)$ and $\lambda=\exp(z)$, initialized with $u_0=\log(4)$ and $z_0=0$. The descent gradients are $$g_{u,k} =-\left( {\partial_w Q_k} +\lambda_k{\partial_w D_k} \right)w_k(1-w_k), \qquad r_k=D_k(w_k)-D_{\mathrm{target}},$$ $$\widetilde g_{u,k}=\operatorname{clip}(g_{u,k},-10,10), \qquad g_{z,k}=\operatorname{clip}(\lambda_k r_k,-10,10).$$ Separate Adam optimizers update $u$ and $z$, with learning rates $2\times10^{-5}$ and $10^{-4}$, respectively. In particular, the dual update is $$z_{k+1} =\operatorname{clip}\left( z_k-10^{-4} \frac{\widehat m_k}{\sqrt{\widehat v_k}+\epsilon}, -12,12 \right), \qquad \lambda_{k+1}=\exp(z_{k+1}),$$ where $\widehat m_k$ and $\widehat v_k$ are Adam’s bias-corrected moments of $g_{z,k}$. Both scalar updates use the same pre-update measurements. A negative residual gives a dual descent direction that strengthens the constraint penalty.
  3. How does this interact with the RL optimizer, and which experiments use it?
    Controller measurements use the updated networks while holding their parameters fixed. Only uu and zz are optimized; the controller does not backpropagate into the actor/critic or alter their optimizer states. The new weight is used in subsequent environment interactions. Adaptive composition is used in online reinforcement learning experiments. Fixed-weight of w=0.5w=0.5, w=0.8w=0.8 or w=0.98w=0.98 are used in RL post-training experiments.

Adaptive Algorithm

Run the shared 10k-step warmup; retain the learner counter n
Initialize u = log(4), z = 0, w = 0.8, λ = 1
Calibrate D_target = D_Twarm(0.8)

For each subsequent environment step:
    Collect and store a transition using the current composed policy
    Perform the original actor/critic and target-network updates
    Increment n
    If joint exploration: update the exploration branch using RND rewards

    If n mod 10 == 0:
        Sample replay states and shared diffusion noise
        Hold network parameters fixed
        Measure Q and D at w, w−, and w+
        Compute both scalar gradients from these pre-update measurements
        Update u and z using their separate Adam optimizers
        Clamp z to [−12, 12]; set w = sigmoid(u), λ = exp(z)
    Otherwise:
        Keep w and λ unchanged

We have added this schedule, the update rules, and the algorithm box to the revised paper.

Reviewer_ZqbiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed method combines a frozen policy with a trainable policy during diffusion sampling to balance exploration and reward optimization. It reports improved performance across locomotion and manipulation tasks.

The idea is simple and promising, but the experiments needs stronger evidence. Main concerns are excluded exploration-pretraining costs, limited statistical validation, and a lack of direct evidence that the composed policy preserves diversity or pretrained skills.

Critical strengths
  1. The method proposes a simple architecture that is applicable across learning regimes. It combines frozen and trainable generative fields during sampling while retaining the underlying RL objectives. Its application to both online exploration and pretrained-policy adaptation makes it a potentially useful contribution.

  2. The paper shows encouraging empirical gains. The method improves mean returns for both QSM and QVPO across all five MuJoCo tasks and improves or matches success rates across four RoboTwin tasks for both pi0 and pi05. These results support the practical relevance of the method.

Critical weaknesses
  1. A big claim of the paper is that the proposed method preserves exploration and pretrained capabilities, but the experiments do not directly demonstrate such property (e.g., showing how the model better explores in post-training). This is my main concern with the paper.

  2. The paper motivates CFG-policy being more expressive than residual policy, but the experiments do not compare against a learned residual policy

  3. Table 3 is missing the error bars for each reported value

Questions for authors

Are CFG-Policy’s 100,000 RND pretraining steps additional to the baseline’s budget? Do the gains from CFG-Policy still hold if we account for this difference?

2 comments

Thank you for your valuable feedback and constructive comments! We itemize the weaknesses and questions you mentioned and answers to them.

W1: A big claim of the paper is that the proposed method preserves exploration and pretrained capabilities, but the experiments do not directly demonstrate such property (e.g., showing how the model better explores in post-training). This is my main concern with the paper.

A1: To intuitively demonstrate how CFG-Policy enhances the exploration capability of a baseline VLA policy while preserving exploitation, we visualize the end-effector (gripper) trajectories generated by each policy during task execution. As shown in Figure 6 in the revised paper, CFG-Policy covers a broader range of the workspace, indicating enhanced exploration. Meanwhile, it also retains coverage of the trajectories visited by the original VLA policy, suggesting that it preserves the exploitation of task-relevant behaviors while expanding exploration.

W2: The paper motivates CFG-policy being more expressive than residual policy, but the experiments do not compare against a learned residual policy.

A2: Thank you for the comment. We have included a comparative experiment with residual RL, which trains an additional residual Gaussian actor to refine the actions. The residual actor takes the current observation and the action generated by the pretrained policy as inputs, and outputs a residual action that is added to the pretrained action to produce the final action. To ensure a fair comparison, both methods are evaluated using the same pretrained models (π₀/π₀.₅), reward function, training budget, and RL optimizer.

The results show that, under the same sparse reward setting and training budget, Residual RL consistently underperforms CFG-Policy, further demonstrating the advantages of CFG-Policy in RL post-training. The correspinding results are also added in the revison. The experimental results are as follows:

ModelMethodAdjust BottlePick Dual BottlesRotate QR CodeGrab Roller
π₀FlowSDE+PPO82.81% ± 1.56%43.23% ± 3.93%34.38% ± 0%91.14% ± 3.25%
Redisual RL97.39% ± 0.09%34.38% ± 0%26.56% ± 3.13%91.15% ± 0.09%
CFG-Policy98.44% ± 0%57.81% ± 0.16%41.66% ± 0.09%94.27% ± 0.24%
π₀.₅FlowSDE+PPO89.06% ± 3.13%12.50% ± 0%7.81% ± 0%100.00% ± 0%
Redisual RL67.71% ± 2.39%12.00% ± 2.39%6.25% ± 0.09%92.19% ± 4.13%
CFG-Policy92.19% ± 0%16.67% ± 0.09%11.45%± 0.09%100.00% ± 0%

W3: Table 3 is missing the error bars for each reported value.

A3: Thank you for pointing it out, we have added the error bar in the revised paper.

Q1: Are CFG-Policy’s 100,000 RND pretraining steps additional to the baseline’s budget? Do the gains from CFG-Policy still hold if we account for this difference?

A4: Thank you for your comment. The 100K RND pretraining steps are additional and we mention it in the Appendix in Line 1140 and 1141, but if we account for the difference, the gains still holds. On the one hand, we conducted additional experiments by initializing the baseline policy with the policy obtained after the 100k-step RND exploration phase, allowing the baseline to benefit from the same RND-based exploration experience (noted as + RND ). On the other hand, we conducted a compute-matched comparison in which we removed the final 100,000 steps of CFG-Policy's training and evaluated the checkpoint at the same total training-step budget as the baseline (noted as CFG-Policy). Under these two settings, the gains of CFG-Policy becomes:

ConfigHalfCheetahWalker2dAntHumanoidHopper
QSM Original5020 ± 4552041 ± 477−24 ± 35922 ± 2622753 ± 307
QSM + RND5260 ± 952224 ± 520−23 ± 45818 ± 2182477 ± 319
QSM (CFG-Policy)6218 ± 533(+23.9%)2349 ± 583(+15.1%)1425 ± 308 (+6038.5%)6060 ± 185(+2.3%)3101 ± 461(+12.6%)
QVPO Original11166 ± 50765220 ± 6897129 ± 616612 ± 2143727 ± 176
QVPO + RND9159 ± 40795162 ± 4966875 ± 2946099 ± 9742413 ± 245
QVPO (CFG-Policy)12666 ± 1963(+13.4%)5600 ± 362(+7.3%)7230 ± 97(+1.4%)7041 ± 385(+6.4%)3787 ± 200(+1.6%)

We have reported these results and clarified the training budgets in the revised paper.

Dear Reviewer Zqbi,

As the author–reviewer discussion period closes tomorrow, we would like to check whether our response has addressed your concerns. We would be glad to clarify any points that remain open. Thanks!

Reviewer_6tJeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines a frozen generative policy with a trainable branch by mixing their score or velocity fields during action sampling. In online RL, the frozen branch supplies exploratory behavior while the other branch learns from rewards. In post-training, the pretrained policy is frozen and a second branch adapts the composed policy. The framework is evaluated with QSM and QVPO on five MuJoCo tasks and with Flow-SDE + PPO on four RoboTwin tasks.

Critical strengths
  1. The sampler-level composition is a coherent way to retain access to a frozen policy while learning a complementary branch. Applying the same interface to online RL and post-training makes the proposal practically useful beyond one optimizer.

  2. The experiments include substantial gains, such as QSM on Ant improving from approximately −24 to 1425 and π0 Adjust Bottle success increasing from 82.81% to 98.44%. Evaluation across multiple backbones gives the framework enough empirical promise to merit consideration.

Critical weaknesses
  1. The interaction budget does not support the current sample-efficiency claim. Appendix B.4.1 states that the exploration policy is separately pretrained with RND and that these environment steps are excluded from the formal training budget. Table 5 specifies 100k pretraining steps for both backbones, whereas the conclusion claims improved sample efficiency. This additional experience could contribute materially, especially where the original baseline performs poorly. The present comparison establishes the benefit of the complete pretrained-branch system, but does not separate that benefit from the contribution of generative composition. Reporting the existing curves against total environment interactions, and clarifying whether the baselines have access to equivalent exploration experience, would make the comparison much easier to assess.

  2. The post-training definition is inconsistent, and parameter preservation is insufficient to establish behavioral preservation. Equation (9) assigns weight ww to the frozen branch:

    vw=(1−w)vEθ+wvX0.v_w=(1-w)v_E^\theta+w v_X^0.

    Algorithm 2, Algorithm 4, and Eq. (57) instead assign ww to the trainable branch:

    vw=(1−w)vX0+wvEθ.v_w=(1-w)v_X^0+w v_E^\theta.

    At w=0.98w=0.98, these conventions give the trainable branch 2% versus 98% of the field weight. The appendix appears to favor the second convention, but the actual implementation and the settings behind Table 3 should be stated consistently. Separately, freezing the base policy preserves its parameters and allows a fallback to the original policy; it does not ensure that the executed composition retains its capabilities. Table 3 measures success on the adaptation tasks, rather than retention of previously acquired behavior. The preservation claim should distinguish these two meanings.

Questions for authors
  1. Which post-training mixing convention is implemented, and which guidance weights and checkpoint-selection procedure produced Table 3? Does capability preservation refer to the frozen fallback policy or to the executed composition?

  2. How do the learning curves compare when the 100k RND pretraining steps are included, and do the baselines receive equivalent exploration experience?

2 comments

Thank you for your valuable feedback and constructive comments! We itemize the weaknesses and questions you mentioned and answers to them.

W1: The post-training definition is inconsistent, and parameter preservation is insufficient to establish behavioral preservation. Equation (9) assigns weight to the frozen branch: vw=(1−w)vEθ+wvXθv_w = (1-w)v_E^\theta + w v_X^\theta Algorithm 2, Algorithm 4, and Eq. (57) instead assign to the trainable branch: vw=(1−w)vXθ+wvEθv_w = (1-w)v_X^\theta + w v_E^\theta At w=0.98w=0.98, these conventions give the trainable branch 2% versus 98% of the field weight. The appendix appears to favor the second convention, but the actual implementation and the settings behind Table 3 should be stated consistently. Separately, freezing the base policy preserves its parameters and allows a fallback to the original policy; it does not ensure that the executed composition retains its capabilities. Table 3 measures success on the adaptation tasks, rather than retention of previously acquired behavior. The preservation claim should distinguish these two meanings.

Q1 Which post-training mixing convention is implemented, and which guidance weights and checkpoint-selection procedure produced Table 3? Does capability preservation refer to the frozen fallback policy or to the executed composition?

A1 Thank you for your comments. The implemented formulation of RL post-training is vw=(1−w)vX0+wvEθv_w=(1-w)v_X^0+wv_E^\theta, consistent with Algorithms 2 and 4 and Eq. (57). Eq. (9) contains typos, which we have corrected. For Table 3, we use w=0.5w=0.5 for π0\pi_0 and w=0.8w=0.8 for π0.5\pi_{0.5}. All results are obtained from the final checkpoint at the last round. Our intended claim on capability preservation is that the original pretrained policy remains intact and recoverable. Specifically, the exploitation branch is frozen throughout training, and setting w=0w=0 recovers the original pretrained policy without retraining. This prevents overwriting pretrained capabilities during RL adaptation. However, we do not claim that the composed policy necessarily retains all pretrained behaviors. Table 3 demonstrates adaptation performance rather than comprehensive behavioral retention. We have clarified this distinction and refined our claims in the revised paper.

W2: The interaction budget does not support the current sample-efficiency claim. Appendix B.4.1 states that the exploration policy is separately pretrained with RND and that these environment steps are excluded from the formal training budget. Table 5 specifies 100k pretraining steps for both backbones, whereas the conclusion claims improved sample efficiency. This additional experience could contribute materially, especially where the original baseline performs poorly. The present comparison establishes the benefit of the complete pretrained-branch system, but does not separate that benefit from the contribution of generative composition. Reporting the existing curves against total environment interactions, and clarifying whether the baselines have access to equivalent exploration experience, would make the comparison much easier to assess.

Q2 How do the learning curves compare when the 100k RND pretraining steps are included, and do the baselines receive equivalent exploration experience?

A2: Thank you for your comments. To further ensure a fair comparison, we conducted additional experiments by initializing the baseline policy with the policy obtained after the 100k-step RND exploration phase, allowing the baseline to benefit from the same RND-based exploration experience. In addition, we also conducted a compute-matched comparison in which we removed the final 100k steps of CFG-Policy’s training and evaluated the checkpoint at the same total training-step budget as the baseline. As shown in the table below, our CFG-policy still significantly outperforms this RND-enhanced baseline, demonstrating that the performance gains of our method cannot be attributed solely to the additional RND-based exploration training.

ConfigHalfCheetahWalker2dAntHumanoidHopper
QSM Original5020 ± 4552041 ± 477−24 ± 35922 ± 2622753 ± 307
QSM + RND5260 ± 952224 ± 520−23 ± 45818 ± 2182477 ± 319
QSM (CFG-Policy)6218 ± 533(+23.9%)2349 ± 583(+15.1%)1425 ± 308 (+6038.5%)6060 ± 185(+2.3%)3101 ± 461(+12.6%)
QVPO Original11166 ± 50765220 ± 6897129 ± 616612 ± 2143727 ± 176
QVPO + RND9159 ± 40795162 ± 4966875 ± 2946099 ± 9742413 ± 245
QVPO (CFG-Policy)12666 ± 1963(+13.4%)5600 ± 362(+7.3%)7230 ± 97(+1.4%)7041 ± 385(+6.4%)3787 ± 200(+1.6%)

Dear Reviewer 6tJe,

As the author–reviewer discussion period closes tomorrow, we would like to check whether our response has addressed your concerns. We would be glad to clarify any points that remain open. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.