acceptodds
Under review as a conference paper at ICLR 2027

Q-Guided Drifting Policy: Value-Guided Target Construction for One-Step Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) often requires expressive stochastic policies to represent heterogeneous and multimodal behavior distributions. Recently proposed drifting models provide a promising generative modeling framework for offline RL policies, enabling multimodal action distributions with one-step stochastic generation. However, offline RL only provides behavior actions rather than the high-value target actions required to train drifting policies, creating a mismatch between drifting policy learning and offline supervision. To bridge this gap, we propose Q-Guided Drifting (QGD) policy, which uses a learned -function to construct training targets for drifting actors rather than directly optimizing the actor through critic gradients. QGD performs zeroth-order local refinement around multiple behavior samples, retaining high-value refined actions as positive targets and low-value behavior actions as additional negatives. The resulting policy preserves multiple high-value action modes while requiring only a single stochastic forward pass at inference. Experiments on OGBench and D4RL show that QGD achieves at least 95% of the best reported performance on 23 out of 37 tasks while retaining one-step stochastic inference.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Drifting Model, Generative Policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pm52OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Q-Guided Drifting (QGD), which constructs high-value positive samples and low-value negative samples through local Q-guided action refinement to train a one-step generative policy. The method achieves competitive results on OGBench and D4RL, with particularly strong performance on OGBench. However, the proposed approach largely combines existing techniques, and several of its main claims are not sufficiently supported by the theoretical analysis or experiments.

Critical strengths
  1. The method separates value-guided action refinement during training from policy generation at inference, enabling one-step action generation with practical benefits.
  2. QGD achieves strong results on OGBench, substantially outperforming several existing methods.
  3. The paper includes useful ablation studies on multi-target selection, local refinement, and negative-sample guidance.
Critical weaknesses

1.The methodological novelty is limited, and the distinction from existing sampling-based policy optimization methods is unclear. The local refinement procedure in Section 4.1 (Eqs. 7–11) consists of Gaussian sampling, Q-value weighting, and moment-based updates, which closely resemble existing sampling-based optimization methods such as CEM and MPPI. Maintaining separate Gaussian proposals for different initial actions may help cover multiple action regions, but does not appear to introduce a fundamentally new optimization mechanism. Furthermore, selecting high-value actions using a critic and subsequently fitting a generative policy is closely related to existing sampling-based policy extraction methods. The authors should clarify the specific methodological contribution beyond these established techniques and provide direct comparisons with multi-start CEM/MPPI using the same drifting model and computational budget. 2. The theoretical analysis does not adequately support the claimed multimodal preservation property. Proposition 1 establishes a distance bound between the refined distribution and the Boltzmann target distribution under several bounded approximation-error assumptions. However, it does not directly establish that multiple high-value action modes are preserved. More importantly, the proposition concerns the refined mixture distribution, whereas the actual method constructs positive targets through Top-K selection in Eq. 12. This selection may still discard valid action modes. The authors should explain how the theoretical result applies to the actual target-selection procedure and provide stronger justification for the multimodal preservation claim. 3. The ablation results do not convincingly establish the necessity of the proposed local refinement mechanism. Table 1 shows that using 272 raw action proposals under a matched critic-evaluation budget achieves an average score of 80, close to QGD's 83. The raw-proposal variant even outperforms QGD on Puzzle (100 vs. 92). These results suggest that much of the improvement may come from broader candidate sampling and Q-based selection rather than the proposed independent local refinement. The authors should clarify what additional benefit local refinement provides over simply increasing the number of candidate samples and compare the methods under identical computational budgets. 4. The negative-sample guidance mechanism raises a theoretical concern. The equilibrium distribution derived in Eq. 15 is expressed as a linear difference between the positive and negative distributions. However, the resulting expression is not guaranteed to be nonnegative everywhere and therefore may not define a valid probability distribution. The authors should specify the conditions under which this equilibrium is well-defined. In addition, Figure 3 evaluates negative-sample guidance on only two tasks, which is insufficient to establish its general effectiveness. A broader ablation study would strengthen this part of the paper. 5. Several target-selection choices appear heuristic and lack sufficient justification. In Eq. 12, the highest-Q candidate is deliberately excluded before selecting the remaining Top-K positive targets, but the paper does not adequately explain why this is necessary. Table 1 shows that Global Top-K, which retains the highest-Q candidate, performs better than QGD on Scene and Puzzle, suggesting that the benefit of this exclusion is not consistent. Similarly, including the original dataset action substantially affects performance on Cube, but its role is not sufficiently analyzed. The authors should provide a clearer justification for these design choices rather than relying mainly on limited performance differences. 6. The computational efficiency claims need further qualification. One-step inference is primarily a property of the underlying drifting model rather than a new capability introduced by QGD. Meanwhile, the proposed approach requires multiple rounds of candidate sampling and critic evaluation during training, potentially introducing substantial computational overhead. Given that the budget-matched raw-sampling baseline already achieves performance close to QGD, the practical efficiency of local refinement remains unclear. Reporting training time, the number of critic evaluations, and inference latency would help establish the actual efficiency gains.

Questions for authors

The paper reports promising empirical results, but its main contribution appears to be the integration of existing sampling-based optimization, value-guided target construction, and drifting policy learning rather than a substantially new policy optimization method. The theoretical analysis does not fully support the claimed multimodal preservation property, and the budget-matched ablations further weaken the case for the novelty and necessity of local refinement. Overall, the current theoretical and experimental evidence is not sufficient to substantiate the paper's main novelty claims.

2 comments
AuthorsOpenReview ↗

Response to Reviewer pm52 (Part I/II)

Weakness 1: Limited methodological novelty and insufficient comparison with CEM/MPPI

We respectfully disagree that the similarity of individual optimization operations implies limited methodological novelty. The novelty of a method should be assessed by its overall formulation and the resulting capabilities, rather than whether each individual operation has appeared in prior work.

The key distinction of QGD is its multimodal value-guided refinement mechanism. Unlike conventional single-population CEM/MPPI, which updates a single Gaussian and may concentrate on one dominant mode, QGD independently refines multiple behavior-anchored proposals to construct a multimodal set of high-value targets. Crucially, candidates from all independently refined proposals are pooled for joint target selection. The resulting positive targets provide a unified distributional supervision signal for training a single Drifting model. This enables value-guided distributional supervision for one-step policy learning.

Furthermore, we have conducted additional comparisons with multi-start CEM under the same Drifting actor and matched computational budget. The results show that QGD achieves better performance, providing further empirical support for the effectiveness of our proposed target-construction mechanism.

Table R5: Multi-start CEM Ablation

TaskQGDMulti-start CEM
cube-double-task160 ± 140 ± 0
cube-triple-task142 ± 30 ± 0
puzzle-3x3-task1100 ± 045 ± 4
puzzle-4x4-task162 ± 72 ± 0
Average6612

Weakness 2: Theoretical justification for multimodal preservation

We thank the reviewer for this comment. Proposition 1 establishes conditional coverage of high-value regions in the refined proposal mixture. Specifically, the corrected TV bound implies that, for any high-value region AiA_i,

pmix(Ai∣s)≥[π∗(Ai∣s)−ϵ(s)]+.p_{\mathrm{mix}}(A_i\mid s)\geq[\pi^*(A_i\mid s)-\epsilon(s)]_+.

Furthermore, since pmix(Ai∣s)=1M∑m=1MqmT(Ai∣s)p_{\mathrm{mix}}(A_i\mid s)=\frac{1}{M}\sum_{m=1}^{M}q_m^T(A_i\mid s), we obtain

max⁡mqmT(Ai∣s)≥[π∗(Ai∣s)−ϵ(s)]+.\max_m q_m^T(A_i\mid s)\geq[\pi^*(A_i\mid s)-\epsilon(s)]_+.

Thus, every sufficiently supported high-value region retains probability mass in the refined mixture.

Top-KK selection aims to suppress low-value candidates while retaining multiple high-value targets. To illustrate, consider the idealized truncation pτ(a∣s)∝pmix(a∣s)1[Qϕ(s,a)≥τ]p_\tau(a\mid s)\propto p_{\mathrm{mix}}(a\mid s)\mathbf{1}[Q_\phi(s,a)\geq\tau]. For any Ai⊆HτA_i\subseteq H_\tau, where Hτ={a:Qϕ(s,a)≥τ}H_\tau=\left\lbrace a:Q_\phi(s,a)\geq\tau\right\rbrace and pmix(Hτ∣s)>0p_{\mathrm{mix}}(H_\tau\mid s)>0,

pτ(Ai∣s)=pmix(Ai∣s)pmix(Hτ∣s)≥pmix(Ai∣s).p_\tau(A_i\mid s)=\frac{p_{\mathrm{mix}}(A_i\mid s)}{p_{\mathrm{mix}}(H_\tau\mid s)}\geq p_{\mathrm{mix}}(A_i\mid s).

Therefore, value-based truncation preserves and amplifies the relative mass of high-value regions that survive the threshold. While finite-sample Top-KK does not guarantee preservation of every mode, it can retain multiple high-value modes while filtering low-value actions.

Weakness 3: Limited evidence for the benefit of local refinement over raw sampling

To further investigate the benefit of local refinement, we have extended the budget-matched comparison to additional challenging OGBench tasks, as shown in the following table.

The additional results show that raw proposals perform substantially worse on more challenging tasks, even under the same critic-evaluation budget. This suggests that simply increasing the number of behavior proposals and applying Top-K selection is insufficient to reliably identify high-value targets in difficult environments. In contrast, QGD performs directed, Q-guided local refinement, allowing each behavior-anchored proposal to move toward promising high-value action regions rather than relying solely on random sampling.

These findings further demonstrate the advantage of independent local refinement, particularly when high-value actions are difficult to discover through raw behavior sampling.

Table R1: Additional Ablations

TaskQGDRaw Samplingw/o Neg.
cube-double-task160 ± 1434 ± 3--
cube-double-task242 ± 328 ± 925 ± 6
cube-double-task321 ± 122 ± 425 ± 5
cube-double-task45 ± 12 ± 24 ± 2
cube-double-task546 ± 1430 ± 231 ± 6
cube-triple-task142 ± 36 ± 331 ± 7
puzzle-3x3-task1100 ± 098 ± 2--
puzzle-3x3-task2100 ± 098 ± 159 ± 14
puzzle-3x3-task393 ± 189 ± 023 ± 2
puzzle-3x3-task492 ± 6100 ± 033 ± 11
puzzle-3x3-task595 ± 1--44 ± 2
puzzle-4x4-task162 ± 734 ± 317 ± 7
puzzle-4x4-task215 ± 114 ± 67 ± 4
puzzle-4x4-task338 ± 321 ± 210 ± 2
puzzle-4x4-task420 ± 027 ± 125 ± 1
puzzle-4x4-task57 ± 10 ± 05 ± 3
Avg.52.440.222.8
AuthorsOpenReview ↗

Response to Reviewer pm52 (Part II/II)

Weakness 4: Validity of negative-sample guidance and insufficient ablations

Theoretical clarification. We thank the reviewer for raising this point. Eq. (15) follows the established training-time guidance formulation of Drifting Models. We clarify that the formal equilibrium in Eq. (15) defines a valid probability distribution provided that 0≤λ−<10\leq\lambda_-<1 and p+(A∣s)≥λ−qβ−(A∣s)p_+(A|s)\geq\lambda_-q_\beta^-(A|s) for every measurable set AA. If this condition is violated in some regions, the exact equilibrium q~=p+\tilde q=p_+ is not attainable, and Eq. (15) should not be interpreted as a valid probability distribution in those cases.

Importantly, this does not introduce negative probabilities into QGD. The learned policy qθq_\theta is the pushforward of a valid prior distribution, while the effective negative distribution q~=(1−λ−)qθ+λ−qβ−\tilde q=(1-\lambda_-)q_\theta+\lambda_-q_\beta^- remains a valid convex mixture. Following Drifting Models, QGD implements guidance directly through additional low-Q negative references in the drifting objective, without explicitly constructing the potentially signed expression in Eq. (15). Thus, Eq. (15) characterizes a feasible ideal equilibrium rather than an unconditional characterization of the learned policy.

Broader ablation.

We have extended the negative-guidance ablation to additional tasks across different environment families. The results are presented in Table R1, providing a broader evaluation of the effectiveness of low-Q negative references.

Weakness 5: Insufficient justification for target-selection design choices

We thank the reviewer for this comment. We exclude the highest-Q candidate as a precaution against critic overestimation. Since Q-guided refinement already favors high-value actions, retaining the single highest-Q candidate could make the target set more sensitive to critic errors. We agree that this choice is not universally beneficial, as Global Top-K performs better on Scene and Puzzle; however, QGD performs substantially better on Cube and achieves a higher overall average.

Including the original dataset action serves a complementary purpose: it provides an unrefined, behavior-supported positive reference, preventing the target set from relying entirely on critic-optimized candidates. Its substantial benefit on Cube supports this motivation, although its impact varies across tasks.

Weakness 6: Training overhead and computational efficiency

We clarify that QGD inherits one-step inference from Drifting Models; our contribution is improving one-step policy quality through value-guided target construction. Although local refinement incurs additional training-time computation, its cost is comparable to budget-matched raw sampling, as both approaches use approximately the same number of critic evaluations. The additional Gaussian moment updates are lightweight relative to critic evaluation. We report critic-evaluation budgets, training wall-clock time, and inference latency in Table R4 to quantify this trade-off.

Importantly, the benefit of QGD is not simply evaluating more candidate actions, but using a comparable evaluation budget more effectively. While budget-matched raw sampling is competitive on some tasks, structured local refinement can more reliably identify high-value actions on challenging tasks by iteratively improving multiple behavior-anchored proposals. Table R4 reports the critic-evaluation budget and training wall-clock time, together with the inference latency of the resulting policies. These measurements provide a quantitative comparison of training overhead and deployment efficiency.

Table R4: Computational Efficiency on CubeDouble

MethodTrain (h)Infer (ms)
Existing Baselines
BC0.99872.6881
IQL1.04762.8212
ReBRAC0.95092.9589
FBRAC1.07786.6728
IFQL1.34206.6076
QAM2.83566.5896
VGF1.40576.5481
FQL1.33106.7897
Shortcut-FQL1.24126.7921
MeanFlow-FQL1.21726.8213
QGD and Ablation
Raw Sampling2.32693.0252
QGD (ours)2.91433.1521
Reviewer_tGiLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QGD, an offline RL method that constructs value-guided targets for training a one-step stochastic generative policy. The authors claim to be the first to construct behavior-supported, multimodal target distributions for critic-guided drifting learning. They claim several desirable properties, including enabling one-step stochastic inference at deployment time. They evaluate on a suite of OGBench and D4RL tasks and show strong performance on OGBench and competitive on D4RL. They also provide ablation studies including multi-target supervision and sensitivity to refinement hyperparameters.

Critical strengths
  • QGD outperforms baselines on OGBench while preserving one-step stochastic inference.
  • Potentially useful with inference time savings.
Critical weaknesses
  • Missing competitive baselines and results do not tell a clear story. Table 5 results indicate that QGD does not necessarily perform better against missing baselines such as ReFORM [1] and DSRL [2], which undermines method's claimed superiority. In addition, results from Figure 2 only support QGD for OGBench but not D4RL.
  • Method is training intensive and superiority of local refinement procedure over existing approaches is unclear. In Figure 2, In Table 1, QGD only superior on OGBench the budget-matched raw proposals reach avg. score of 80 compared to 83 for QGD. The improvement is concentrated in certain environments, suggesting to me that a significant part of the benefit may simply come from evaluating sufficiently many behavior proposals + retaining multiple high-value targets, rather than iterative local refinement itself. Furthermore, the paper doesn't quantify resulting training-time or overhead from critic evaluations during training. Since computational efficiency is one of the desired properties of the method, having a comparison of performance-vs-training-cost trade-off could be helpful.
  • Claimed contribution is one-step inference vs baselines such as flow and diffusion policies, but no empirical evidence given that this is a crucial bottleneck in practice.
  • I feel I do not understand why this paper's proposed approach would be better than other max-Q offline RL approaches. This sampling procedure doesn't seem to promise good coverage of different modalities still, and the final top-K selection combined with anchor sampling could still collapse to a single mode or provide insufficient coverage.

[1] S. Zhang, O. So, H. M. S. Ahmad, E. Y. Yu, M. Cleaveland, M. Black, and C. Fan. “ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation.” International Conference on Learning Representations (ICLR), 2026.

[2] A. Wagenmaker, Y. Zhang, M. Nakamoto, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning.” Conference on Robot Learning (CoRL), 2025.

Questions for authors
  • Provide [1] and [2] results? (not comprehensively listing all other competitive baselines).
  • In App B, Eq. 53 is equivalent to minimizing DKL(q∣∣q~t+1)D_{KL} (q || \tilde{q}^{t+1}) over the Gaussian family, whereas the subsequent projection on page 18 minimizes DKL(q~t+1∣∣q)D_{KL} (\tilde{q}^{t+1} || q). Since these directions are not generally equivalent, could the authors clarify which objective the proposed weighted-moment refinement approximates, and how it relates to Eq. 11?
  • Report wall clock training time and eval times and compare inference times against baselines (e.g., flow and diffusion policies)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer tGiL (Part I/II)

Weakness 1: Missing competitive baselines and unclear empirical advantage

ReFORM and DSRL employ substantially different mechanisms from QGD and are not direct methodological baselines for our value-guided target construction. Nevertheless, they can serve as strong performance baselines. Our evaluation includes an up-to-date set of strong and competitive baselines, including recent methods such as QAM and VGF (2026).

To further address this concern, we conducted additional experiments with ReFORM on 14 overlapping OGBench tasks. As shown in Table R3, QGD outperforms ReFORM by 10.5 points in average score (43.7 vs. 33.2). These results provide additional evidence of QGD's effectiveness on OGBench.

Table R3: Comparison with ReFORM and DSRL

TaskQGDReFORMDSRL
cube-double-task160 ± 1473 ± 738 ± 11
cube-double-task242 ± 341 ± 353 ± 14
cube-double-task321 ± 167 ± 1114 ± 2
cube-double-task45 ± 17 ± 411 ± 7
cube-double-task546 ± 1457 ± 644 ± 3
cube-triple-task142 ± 31 ± 10 ± 0
cube-triple-task20 ± 00 ± 00 ± 0
cube-triple-task30 ± 00 ± 00 ± 0
cube-triple-task40 ± 00 ± 00 ± 0
cube-triple-task50 ± 00 ± 00 ± 0
puzzle-3x3-task1100 ± 098 ± 198 ± 1
puzzle-3x3-task2100 ± 082 ± 2392 ± 1
puzzle-3x3-task393 ± 129 ± 1392 ± 1
puzzle-3x3-task492 ± 622 ± 70 ± 0
puzzle-3x3-task595 ± 10 ± 04 ± 1
puzzle-4x4-task162 ± 783 ± 848 ± 5
puzzle-4x4-task215 ± 10 ± 00 ± 0
puzzle-4x4-task338 ± 336 ± 1824 ± 4
puzzle-4x4-task420 ± 0--23 ± 11
Avg.43.733.228.5

Weakness 2: Contribution of local refinement and computational overhead

We thank the reviewer for this comment. To further investigate the contribution of local refinement, we have expanded the budget-matched raw-proposal comparison to include additional challenging environments, as shown in Table R1.

Table R1: Additional Ablations

TaskQGDRaw Samplingw/o Neg.
cube-double-task160 ± 1434 ± 3--
cube-double-task242 ± 328 ± 925 ± 6
cube-double-task321 ± 122 ± 425 ± 5
cube-double-task45 ± 12 ± 24 ± 2
cube-double-task546 ± 1430 ± 231 ± 6
cube-triple-task142 ± 36 ± 331 ± 7
puzzle-3x3-task1100 ± 098 ± 2--
puzzle-3x3-task2100 ± 098 ± 159 ± 14
puzzle-3x3-task393 ± 189 ± 023 ± 2
puzzle-3x3-task492 ± 6100 ± 033 ± 11
puzzle-3x3-task595 ± 1--44 ± 2
puzzle-4x4-task162 ± 734 ± 317 ± 7
puzzle-4x4-task215 ± 114 ± 67 ± 4
puzzle-4x4-task338 ± 321 ± 210 ± 2
puzzle-4x4-task420 ± 027 ± 125 ± 1
puzzle-4x4-task57 ± 10 ± 05 ± 3
Avg.52.440.222.8

The new results show that raw proposals perform substantially worse on more challenging tasks, despite using the same critic-evaluation budget. This suggests that simply sampling more behavior actions and applying Top-K selection is insufficient to reliably identify high-value positive targets in difficult environments. In contrast, QGD performs directed, Q-guided local refinement, enabling more effective exploration of promising action regions and construction of high-value training targets.

We additionally report training-time measurements in Table R4 to quantify the computational overhead of refinement. While QGD incurs additional computation during training, it retains one-step action generation at deployment.

Table R4: Computational Efficiency on CubeDouble

MethodTrain (h)Infer (ms)
Existing Baselines
BC0.99872.6881
IQL1.04762.8212
ReBRAC0.95092.9589
FBRAC1.07786.6728
IFQL1.34206.6076
QAM2.83566.5896
VGF1.40576.5481
FQL1.33106.7897
Shortcut-FQL1.24126.7921
MeanFlow-FQL1.21726.8213
QGD and Ablation
Raw Sampling2.32693.0252
QGD (ours)2.91433.1521
AuthorsOpenReview ↗

Response to Reviewer tGiL (Part II/II)

Weakness 3: Practical importance of one-step inference

We would like to clarify that our primary contribution is value-guided target construction for one-step drifting policies, rather than establishing that inference latency is a universal bottleneck in offline RL. One-step inference is a desirable deployment property, particularly for latency-sensitive control, but the practical importance of inference latency varies across applications. Our main objective is to improve the performance of expressive one-step policies while preserving their computationally efficient action generation.

One-step inference is particularly valuable for latency-sensitive robotic control, including high-frequency locomotion, contact-rich manipulation, and dynamic bimanual tasks. For example, real-world legged robots operate at 100 Hz (10 ms per control update) [1]. In state-based manipulation, standard Diffusion Policy requires 816--833 ms per prediction, compared with 25--145 ms for an accelerated variant [2]. This empirical evidence highlights the importance of low-latency action generation, although its practical benefits remain task-dependent.

Weakness 4: Multimodal coverage and potential mode collapse

The key distinction from conventional max-Q methods lies in the distributional learning mechanism of Drifting Models. As shown by [3], drifting benefits from multiple positive and negative samples, which provide richer distributional learning signals.

Motivated by this property, QGD constructs multiple high-value positive targets from behavior-anchored proposals, rather than collapsing supervision into a single max-Q action. While conventional max-Q selection and global Top-K can concentrate targets around one mode, QGD's independently refined anchors encourage broader coverage of high-value regions. This is supported by our ablations: single-best (max-Q method) and advantage-weighted targets achieve 57 and 73, respectively, compared with 83 for QGD, as reported in Table 1 of our original paper.

This is supported by our target-construction ablations. Compressing the selected targets into a single target, as in Single best and Advantage-weighted target, reduces the average score to 57 and 73, respectively, compared with 83 for QGD. Meanwhile, No refinement (Raw8) and budget-matched raw proposals (Raw272) achieve 31 and 80, respectively, showing that a larger candidate budget can recover competitive performance, while local refinement provides additional benefits. Global Top-K also achieves a competitive score of 79, compared with 83 for QGD, as reported in Table 1 of our original paper.

We acknowledge that QGD does not guarantee preservation of every mode, since candidates from a dominant mode may occupy all Top-KK positions. Our theoretical analysis provides conditional coverage guarantees for sufficiently supported high-value regions in the refined proposal mixture before selection. Accordingly, QGD is designed to encourage coverage of multiple high-value modes.

Question 1: Additional ReFORM and DSRL results

Please refer to our response to Weakness1 for a detailed explanation.

Question 2: KL divergence direction and Gaussian refinement

We thank the reviewer for identifying this inconsistency. Our intended derivation follows the two-stage distributional update used in MPPI-style methods. The issue arises because the original Eq. (53) incorrectly restricts the initial KL-regularized improvement to the Gaussian family. We have corrected Eq. (53) to optimize over unrestricted probability distributions, yielding the Boltzmann-reweighted target q~t+1∝qtexp⁡(Qϕ/α)\widetilde q^{t+1}\propto q^t\exp(Q_\phi/\alpha).

This target is then approximated within the diagonal-Gaussian family through a separate forward-KL projection, which yields weighted moment matching in the population setting. In practice, Eq. (11) performs finite-sample weighted Gaussian maximum-likelihood fitting using the standardized Q-value weights in Eq. (10). This correction resolves the inconsistency and aligns the theoretical derivation with the implemented refinement procedure without changing the algorithm. We have incorporated this correction into the revised manuscript, with the corresponding changes highlighted in blue.

Question 3: Training time and inference efficiency

The computational efficiency comparison is presented in Table R4.

[1] J. Alvarez-Padilla, J. Z. Zhang, S. Kwok, J. M. Dolan, and Z. Manchester. "Real-Time Whole-Body Control of Legged Robots with Model-Predictive Path Integral Control." IEEE International Conference on Robotics and Automation (ICRA), 2025.

[2] Y. Duan, H. Yin, and D. Kragic. "Real-Time Iteration Scheme for Diffusion Policy." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025.

[3] M. Deng, H. Li, T. Li, Y. Du, and K. He. "Generative Modeling via Drifting." Advances in Neural Information Processing Systems (NeurIPS), 2026.

Reviewer_m5npOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-Guided Drifting Policy (QGD), a one-step generative policy for offline reinforcement learning. QGD samples multiple behavior-conditioned action anchors, independently refines them using scalar critic evaluations, and constructs positive targets for drifting-policy training. Low-value behavior actions provide additional negative references. The resulting actor requires a single stochastic forward pass at deployment. Experiments on OGBench and D4RL report substantial improvements on several manipulation and puzzle tasks.

Critical strengths
  1. A useful training–deployment trade-off. Moving action search into training while retaining one-step stochastic inference is practically motivated. The construction also avoids differentiating the actor through critic-guided search.
  2. Promising empirical results. The improvements on several OGBench puzzle tasks are substantial and warrant further investigation.
  3. Relevant ablations. The paper includes comparisons against single-target compression, budget-matched raw proposals, alternative positive selection rules, and critic-gradient refinement. These provide useful evidence beyond the main benchmark table.
Critical weaknesses
  1. The central theoretical guarantee is not established. Equation (70) incorrectly assumes that Boltzmann reweighting does not amplify total-variation error. Writing w(a)=exp⁡(Q(s,a)/α)w(a)=\exp(Q(s,a)/\alpha), the direct bound includes a factor ∥w∥∞/Z(s)\|w\|_\infty/Z(s), which can exceed one. The claimed inequality can fail even when the local and global normalization constants coincide, so the existing ϵZ\epsilon_Z term does not resolve the issue. This is consequential because Proposition 1 provides the main theoretical justification that QGD’s constructed distribution approximates the behavior-regularized Boltzmann policy.

  2. A key baseline exhibits unexplained performance discrepancies. Table 5 reports FQL scores of 0±00\pm0, 1±11\pm1, and 2±32\pm3 on pen-human, pen-cloned, and pen-expert, respectively, whereas the original FQL paper reports 53±653\pm6, 74±1174\pm11, and 142±6142\pm6. These discrepancies do not themselves establish an implementation error, but they require reconciliation before the comparison is reliable. In particular, QGD’s pen-expert score of 129±6129\pm6 exceeds the reproduced FQL baseline but falls below FQL’s originally reported result. This materially weakens the evidence for improvement over an important competing one-step policy. reference: FQL, Table 4

Questions for authors
  1. Can Proposition 1 be corrected to account for amplification under Boltzmann weighting? Which implemented target distribution does the corrected result cover?
  2. What explains the FQL pen-task discrepancies, and do the conclusions remain after reproducing the original baseline performance?
  3. Under an identical critic and backup rule, how much improvement does QGD provide over budget-matched raw-proposal target construction?
  4. How does QGD compare with DFP under matched training resources and the same one-step deployment constraint?
2 comments
AuthorsOpenReview ↗

Response to Reviewer m5np (Part I/II)

Weakness 1: Missing Boltzmann amplification factor in the TV bound

We thank the reviewer for pointing out the amplification factor introduced by Boltzmann reweighting. This factor can be directly incorporated into our existing analysis without changing the proof structure or the qualitative conclusion of Proposition 1. Fixing a state ss, let ws(a)=exp⁡(Qϕ(s,a)/α)w_{s}(a) = \exp\left( Q_{\phi}(s,a)/\alpha \right), W(s)=sup⁡aws(a)W(s) = \sup_{a}w_{s}(a) and Z(s)=∫πβ(a∣s)ws(a) daZ(s) = \int\pi_{\beta}(a \mid s)w_{s}(a)\ da. We have derived the corrected weighting step 12∥qˉ0ws/Z(s)−πβws/Z(s)∥1≤[W(s)/Z(s)]ϵβ(s)\frac{1}{2} \parallel {\bar{q}}^{0}w_{s}/Z(s) - \pi_{\beta}w_{s}/Z(s) \parallel_{1} \leq \left[ W(s)/Z(s) \right]\epsilon_{\beta}(s).

Keeping the original assumptions on finite-step refinement and local normalizers, the revised bound becomes DTV(pmix,π∗)≤ϵref(s)+[W(s)/Z(s)]ϵβ(s)+ϵZ(s)/(2Z(s))D_{TV}\left( p_{mix},\pi^{*} \right) \leq \epsilon_{ref}(s) + \left[ W(s)/Z(s) \right]\epsilon_{\beta}(s) + \epsilon_{Z}(s)/\left( 2Z(s) \right). The normalizer term follows by integrating against each qm0wsq_{m}^{0}w_{s}, which cancels Zm(s)Z_{m}(s). The additional factor modifies the coefficient of the existing behavior-approximation error term, while the overall conditional distributional approximation conclusion remains intact.

We have also updated the proof in the revised manuscript, with the corresponding changes highlighted in blue in Appendix B.1.

Weakness 2: Discrepancies between reproduced and published FQL results

We thank the reviewer for highlighting the discrepancy. Our FQL results were obtained using the authors' released codebase and the task-specific hyperparameters specified in the original FQL paper. We also examined the experimental protocols in our original paper and identified several differences: our experiments used 1M gradient steps, four random seeds, and the average of the final three evaluation checkpoints, whereas the original FQL paper reports 500K steps, eight seeds, and the final checkpoint. Our reported results were obtained under the common experimental setup adopted in this study.

We acknowledge that these protocol differences may not fully explain the substantial performance gaps on the Pen tasks. Other factors, such as differences in environment or software versions and evaluation configurations, may also contribute, although we cannot conclusively attribute the discrepancy to any particular factor. Importantly, our reported FQL scores are actual results from running the released implementation in our experimental setup. We therefore distinguish our reproduced FQL results from the originally published results and do not claim that QGD outperforms the published FQL scores on the Pen tasks.

Question 1: Corrected Boltzmann weighting bound and its scope

The mathematical correction is the factor W(s)/Z(s)W(s)/Z(s) in the behavior-mixture term; the corrected complete statement and proof are given in revised Proposition 1. We additionally replace the normalizer-error step with ϵZ(s)/(2Z(s))\epsilon_{Z}(s)/\left( 2Z(s) \right). The theorem covers only pmix=M−1∑mqmTp_{mix} = M^{-1}\sum_{m}^{}q_{m}^{T}, the final local-proposal mixture before candidate-mean extraction and Top-KK. The bound is conditional on the explicit finite-step refinement-error assumption ϵref(s)\epsilon_{\mathrm{ref}}(s) and characterizes this distribution before candidate-mean extraction and Top-KK selection.

Question 2: FQL Pen-task performance discrepancies

As discussed in our response to CW2, our FQL results were obtained using the released codebase and reported hyperparameters, with differences in training and evaluation protocols. We acknowledge that these differences do not fully explain the Pen-task discrepancies.

Using the originally published FQL results, QGD achieves comparable performance on pen-human (53±253\pm2 vs. 53±653\pm6), but lower performance on pen-cloned (52±352\pm3 vs. 74±1174\pm11) and pen-expert (129±6129\pm6 vs. 142±6142\pm6). The corresponding 12-task D4RL averages are 47.647.6 for QGD and 51.651.6 for published FQL. We therefore do not claim overall superiority over the published FQL results on D4RL; our main empirical advantage is demonstrated on OGBench.

AuthorsOpenReview ↗

Response to Reviewer m5np (Part II/II)

Question 3: Improvement over budget-matched raw sampling

We thank the reviewer for this question. We have conducted additional experiments comparing QGD with budget-matched raw-proposal target construction under the same critic and backup rule. Across the 16 evaluated tasks, QGD improves performance by 30% over raw sampling, as reported in Table R1.

Table R1: Additional Ablations

TaskQGDRaw Samplingw/o Neg.
cube-double-task160 ± 1434 ± 3--
cube-double-task242 ± 328 ± 925 ± 6
cube-double-task321 ± 122 ± 425 ± 5
cube-double-task45 ± 12 ± 24 ± 2
cube-double-task546 ± 1430 ± 231 ± 6
cube-triple-task142 ± 36 ± 331 ± 7
puzzle-3x3-task1100 ± 098 ± 2--
puzzle-3x3-task2100 ± 098 ± 159 ± 14
puzzle-3x3-task393 ± 189 ± 023 ± 2
puzzle-3x3-task492 ± 6100 ± 033 ± 11
puzzle-3x3-task595 ± 1--44 ± 2
puzzle-4x4-task162 ± 734 ± 317 ± 7
puzzle-4x4-task215 ± 114 ± 67 ± 4
puzzle-4x4-task338 ± 321 ± 210 ± 2
puzzle-4x4-task420 ± 027 ± 125 ± 1
puzzle-4x4-task57 ± 10 ± 05 ± 3
Avg.52.440.222.8

Question 4: Comparison with DFP under matched training resources

We thank the reviewer for suggesting this comparison. DFP primarily focuses on offline-to-online RL, while QGD is designed for pure offline RL. To provide a more direct comparison under the same one-step deployment constraint, we selected six OGBench tasks covering CubeDouble, Puzzle3x3, and Puzzle4x4. Table~R2 presents the comparison, including the published DFP result on cube-double-task2 and additional evaluation entries for the remaining tasks.

Table R2: Comparison with DFP

TaskQGDDFP
cube-double-task160 ± 1438 ± 5
cube-double-task242 ± 341 ± 4
puzzle-3x3-task1100 ± 058 ± 1
puzzle-3x3-task2100 ± 046 ± 10
puzzle-4x4-task162 ± 732 ± 4
puzzle-4x4-task215 ± 18 ± 2
Average (6 tasks)63.237.2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.