- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Q-Guided Drifting (QGD), which constructs high-value positive samples and low-value negative samples through local Q-guided action refinement to train a one-step generative policy. The method achieves competitive results on OGBench and D4RL, with particularly strong performance on OGBench. However, the proposed approach largely combines existing techniques, and several of its main claims are not sufficiently supported by the theoretical analysis or experiments.
Critical strengths
- The method separates value-guided action refinement during training from policy generation at inference, enabling one-step action generation with practical benefits.
- QGD achieves strong results on OGBench, substantially outperforming several existing methods.
- The paper includes useful ablation studies on multi-target selection, local refinement, and negative-sample guidance.
Critical weaknesses
1.The methodological novelty is limited, and the distinction from existing sampling-based policy optimization methods is unclear. The local refinement procedure in Section 4.1 (Eqs. 7–11) consists of Gaussian sampling, Q-value weighting, and moment-based updates, which closely resemble existing sampling-based optimization methods such as CEM and MPPI. Maintaining separate Gaussian proposals for different initial actions may help cover multiple action regions, but does not appear to introduce a fundamentally new optimization mechanism. Furthermore, selecting high-value actions using a critic and subsequently fitting a generative policy is closely related to existing sampling-based policy extraction methods. The authors should clarify the specific methodological contribution beyond these established techniques and provide direct comparisons with multi-start CEM/MPPI using the same drifting model and computational budget. 2. The theoretical analysis does not adequately support the claimed multimodal preservation property. Proposition 1 establishes a distance bound between the refined distribution and the Boltzmann target distribution under several bounded approximation-error assumptions. However, it does not directly establish that multiple high-value action modes are preserved. More importantly, the proposition concerns the refined mixture distribution, whereas the actual method constructs positive targets through Top-K selection in Eq. 12. This selection may still discard valid action modes. The authors should explain how the theoretical result applies to the actual target-selection procedure and provide stronger justification for the multimodal preservation claim. 3. The ablation results do not convincingly establish the necessity of the proposed local refinement mechanism. Table 1 shows that using 272 raw action proposals under a matched critic-evaluation budget achieves an average score of 80, close to QGD's 83. The raw-proposal variant even outperforms QGD on Puzzle (100 vs. 92). These results suggest that much of the improvement may come from broader candidate sampling and Q-based selection rather than the proposed independent local refinement. The authors should clarify what additional benefit local refinement provides over simply increasing the number of candidate samples and compare the methods under identical computational budgets. 4. The negative-sample guidance mechanism raises a theoretical concern. The equilibrium distribution derived in Eq. 15 is expressed as a linear difference between the positive and negative distributions. However, the resulting expression is not guaranteed to be nonnegative everywhere and therefore may not define a valid probability distribution. The authors should specify the conditions under which this equilibrium is well-defined. In addition, Figure 3 evaluates negative-sample guidance on only two tasks, which is insufficient to establish its general effectiveness. A broader ablation study would strengthen this part of the paper. 5. Several target-selection choices appear heuristic and lack sufficient justification. In Eq. 12, the highest-Q candidate is deliberately excluded before selecting the remaining Top-K positive targets, but the paper does not adequately explain why this is necessary. Table 1 shows that Global Top-K, which retains the highest-Q candidate, performs better than QGD on Scene and Puzzle, suggesting that the benefit of this exclusion is not consistent. Similarly, including the original dataset action substantially affects performance on Cube, but its role is not sufficiently analyzed. The authors should provide a clearer justification for these design choices rather than relying mainly on limited performance differences. 6. The computational efficiency claims need further qualification. One-step inference is primarily a property of the underlying drifting model rather than a new capability introduced by QGD. Meanwhile, the proposed approach requires multiple rounds of candidate sampling and critic evaluation during training, potentially introducing substantial computational overhead. Given that the budget-matched raw-sampling baseline already achieves performance close to QGD, the practical efficiency of local refinement remains unclear. Reporting training time, the number of critic evaluations, and inference latency would help establish the actual efficiency gains.
Questions for authors
The paper reports promising empirical results, but its main contribution appears to be the integration of existing sampling-based optimization, value-guided target construction, and drifting policy learning rather than a substantially new policy optimization method. The theoretical analysis does not fully support the claimed multimodal preservation property, and the budget-matched ablations further weaken the case for the novelty and necessity of local refinement. Overall, the current theoretical and experimental evidence is not sufficient to substantiate the paper's main novelty claims.
2 comments
Response to Reviewer pm52 (Part II/II)
Weakness 4: Validity of negative-sample guidance and insufficient ablations
Theoretical clarification. We thank the reviewer for raising this point. Eq. (15) follows the established training-time guidance formulation of Drifting Models. We clarify that the formal equilibrium in Eq. (15) defines a valid probability distribution provided that and for every measurable set . If this condition is violated in some regions, the exact equilibrium is not attainable, and Eq. (15) should not be interpreted as a valid probability distribution in those cases.
Importantly, this does not introduce negative probabilities into QGD. The learned policy is the pushforward of a valid prior distribution, while the effective negative distribution remains a valid convex mixture. Following Drifting Models, QGD implements guidance directly through additional low-Q negative references in the drifting objective, without explicitly constructing the potentially signed expression in Eq. (15). Thus, Eq. (15) characterizes a feasible ideal equilibrium rather than an unconditional characterization of the learned policy.
Broader ablation.
We have extended the negative-guidance ablation to additional tasks across different environment families. The results are presented in Table R1, providing a broader evaluation of the effectiveness of low-Q negative references.
Weakness 5: Insufficient justification for target-selection design choices
We thank the reviewer for this comment. We exclude the highest-Q candidate as a precaution against critic overestimation. Since Q-guided refinement already favors high-value actions, retaining the single highest-Q candidate could make the target set more sensitive to critic errors. We agree that this choice is not universally beneficial, as Global Top-K performs better on Scene and Puzzle; however, QGD performs substantially better on Cube and achieves a higher overall average.
Including the original dataset action serves a complementary purpose: it provides an unrefined, behavior-supported positive reference, preventing the target set from relying entirely on critic-optimized candidates. Its substantial benefit on Cube supports this motivation, although its impact varies across tasks.
Weakness 6: Training overhead and computational efficiency
We clarify that QGD inherits one-step inference from Drifting Models; our contribution is improving one-step policy quality through value-guided target construction. Although local refinement incurs additional training-time computation, its cost is comparable to budget-matched raw sampling, as both approaches use approximately the same number of critic evaluations. The additional Gaussian moment updates are lightweight relative to critic evaluation. We report critic-evaluation budgets, training wall-clock time, and inference latency in Table R4 to quantify this trade-off.
Importantly, the benefit of QGD is not simply evaluating more candidate actions, but using a comparable evaluation budget more effectively. While budget-matched raw sampling is competitive on some tasks, structured local refinement can more reliably identify high-value actions on challenging tasks by iteratively improving multiple behavior-anchored proposals. Table R4 reports the critic-evaluation budget and training wall-clock time, together with the inference latency of the resulting policies. These measurements provide a quantitative comparison of training overhead and deployment efficiency.
Table R4: Computational Efficiency on CubeDouble
| Method | Train (h) | Infer (ms) |
|---|---|---|
| Existing Baselines | ||
| BC | 0.9987 | 2.6881 |
| IQL | 1.0476 | 2.8212 |
| ReBRAC | 0.9509 | 2.9589 |
| FBRAC | 1.0778 | 6.6728 |
| IFQL | 1.3420 | 6.6076 |
| QAM | 2.8356 | 6.5896 |
| VGF | 1.4057 | 6.5481 |
| FQL | 1.3310 | 6.7897 |
| Shortcut-FQL | 1.2412 | 6.7921 |
| MeanFlow-FQL | 1.2172 | 6.8213 |
| QGD and Ablation | ||
| Raw Sampling | 2.3269 | 3.0252 |
| QGD (ours) | 2.9143 | 3.1521 |
Response to Reviewer pm52 (Part I/II)
Weakness 1: Limited methodological novelty and insufficient comparison with CEM/MPPI
We respectfully disagree that the similarity of individual optimization operations implies limited methodological novelty. The novelty of a method should be assessed by its overall formulation and the resulting capabilities, rather than whether each individual operation has appeared in prior work.
The key distinction of QGD is its multimodal value-guided refinement mechanism. Unlike conventional single-population CEM/MPPI, which updates a single Gaussian and may concentrate on one dominant mode, QGD independently refines multiple behavior-anchored proposals to construct a multimodal set of high-value targets. Crucially, candidates from all independently refined proposals are pooled for joint target selection. The resulting positive targets provide a unified distributional supervision signal for training a single Drifting model. This enables value-guided distributional supervision for one-step policy learning.
Furthermore, we have conducted additional comparisons with multi-start CEM under the same Drifting actor and matched computational budget. The results show that QGD achieves better performance, providing further empirical support for the effectiveness of our proposed target-construction mechanism.
Table R5: Multi-start CEM Ablation
Weakness 2: Theoretical justification for multimodal preservation
We thank the reviewer for this comment. Proposition 1 establishes conditional coverage of high-value regions in the refined proposal mixture. Specifically, the corrected TV bound implies that, for any high-value region Ai,
pmix(Ai∣s)≥[π∗(Ai∣s)−ϵ(s)]+.
Furthermore, since pmix(Ai∣s)=M1∑m=1MqmT(Ai∣s), we obtain
maxmqmT(Ai∣s)≥[π∗(Ai∣s)−ϵ(s)]+.
Thus, every sufficiently supported high-value region retains probability mass in the refined mixture.
Top-K selection aims to suppress low-value candidates while retaining multiple high-value targets. To illustrate, consider the idealized truncation pτ(a∣s)∝pmix(a∣s)1[Qϕ(s,a)≥τ]. For any Ai⊆Hτ, where Hτ={a:Qϕ(s,a)≥τ} and pmix(Hτ∣s)>0,
pτ(Ai∣s)=pmix(Hτ∣s)pmix(Ai∣s)≥pmix(Ai∣s).
Therefore, value-based truncation preserves and amplifies the relative mass of high-value regions that survive the threshold. While finite-sample Top-K does not guarantee preservation of every mode, it can retain multiple high-value modes while filtering low-value actions.
Weakness 3: Limited evidence for the benefit of local refinement over raw sampling
To further investigate the benefit of local refinement, we have extended the budget-matched comparison to additional challenging OGBench tasks, as shown in the following table.
The additional results show that raw proposals perform substantially worse on more challenging tasks, even under the same critic-evaluation budget. This suggests that simply increasing the number of behavior proposals and applying Top-K selection is insufficient to reliably identify high-value targets in difficult environments. In contrast, QGD performs directed, Q-guided local refinement, allowing each behavior-anchored proposal to move toward promising high-value action regions rather than relying solely on random sampling.
These findings further demonstrate the advantage of independent local refinement, particularly when high-value actions are difficult to discover through raw behavior sampling.
Table R1: Additional Ablations