acceptodds
Under review as a conference paper at ICLR 2027

DADC: Dual-Source Anchor Drift Correction for Online Reinforcement Learning

Abstract

Generative policies offer expressive multimodal action representations for online reinforcement learning (RL). However, many existing methods focus on fitting actions favored by the critic, using negative feedback mainly to downweight less promising actions. This provides limited explicit guidance for avoiding low-value actions and can restrict further policy improvement. To address this limitation, we propose Dual-Source Anchor Drift Correction (DADC), an online RL algorithm that combines attractive guidance with explicit repulsion from low-value policy samples. Such guidance requires informative anchors, which are difficult to construct without fixed expert data. DADC therefore constructs attractive and repulsive anchors from distinct sources. For each replay state, we sample initial attractive anchors from an advantage-weighted conditional flow prior trained on replay data and apply several steps of Q-guided Langevin refinement. At the same state, we select current-policy candidates with the lowest critic scores as repulsive anchors, focusing repulsion on relatively low-value actions. The actor update combines direct Q-value maximization with drift guidance toward attractive anchors and away from repulsive anchors. The flow prior and anchor construction are required only during training, while deployment uses a single actor forward pass. Experiments on 14 DMControl and HumanoidBench tasks show that DADC achieves the highest mean return among the evaluated methods on all tasks, delivering substantial gains over the compared generative policy methods.

Primary area:
reinforcement learning
Keywords:
online reinforcement learning, generative policies, drifting models, flow matching, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DDg4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dual-Source Anchor Drift Correction (DADC) for online RL. It builds attractive anchors via an advantage-weighted flow prior plus Q-guided Langevin refinement, and repulsive anchors from the lowest-critic-scored current-policy samples. Beyond direct Q maximization, it adds drift guidance toward attractive and away from repulsive anchors, while keeping single-pass actor inference at deployment.

Critical strengths
  1. The dual-source anchor design directly targets the underutilization of negative feedback, and ablations support the complementarity of the two branches.
  2. The method retains single-pass inference while achieving consistent superiority across 14 tasks.
Critical weaknesses
  1. The paper lacks rigorous theoretical analysis to theoretically justify the effectiveness of the proposed method.
  2. The proposed method introduces a large number of hyperparameters and lacks systematic hyperparameter sensitivity analysis, which poses a substantial obstacle to its practical application.
Questions for authors
  1. In the first paragraph of the Introduction, the sentence "These objectives assign lower weights to low-value actions but ..." Please explain this in detail: what guidance does it provide?
  2. The paper lacks rigorous theoretical analysis to theoretically prove the effectiveness of the proposed method.
  3. The proposed method introduces a large number of hyperparameters, but the paper lacks a systematic hyperparameter sensitivity analysis to analyze the sensitivity of these hyperparameters, such as the Langevin hyperparameters (η, number of steps, noise), the prior update frequency d_target, α, etc. At the same time, the proposed method introducing too many hyperparameters makes the method less practical, because it requires users to tune a large number of hyperparameters when using it.
  4. The DRL community has developed a large number of advanced methods to drive the policy to obtain higher return values, which shares the same ultimate goal as generative RL, while the non-generative RL baselines in the experiments are very few. Therefore, in addition to comparing with SOTA generative RL, 5 additional SOTA non-generative RL methods should also be compared.
  5. The experiments in the paper are almost entirely of the locomotion / whole-body control type, and indeed lack manipulation tasks.
  6. The baselines in the experiments are basically generative RL based on diffusion policies and flow models. Generative RL includes various types of methods based on GAN, Transformer, VAE, etc., and these are also designed to maximize long-term rewards. Therefore, it is suggested to additionally compare with SOTA generative RL methods of other types.
  7. The efficiency comparison is only conducted on a single Dog Run task against a single Flow RL, which is insufficient. It is suggested to compare runtime efficiency with more SOTA algorithms on more tasks to demonstrate the superiority of the proposed method.
Reviewer_MZBROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DADC, a drift-based policy correction method for online RL. It trains a generative drift model to attract actions toward positive anchors from an advantage-weighted regression (AWR) policy and repel actions from negative samples drawn from the current policy. The method also uses critic-guided refinement to construct attractive anchors. The authors evaluate DADC on 14 tasks from DMControl and HumanoidBench and report higher mean returns than the evaluated baselines.

Critical strengths
  1. The method reports the best mean return among the evaluated baselines on all 14 tasks. These results suggest that the approach is promising.
  2. The auxiliary generative model and anchor-generation process are used during training. At deployment, the method uses a single actor forward pass.
Critical weaknesses
  1. Limited methodological novelty. The central idea appears to transfer a drift-based generative policy, previously explored in offline RL, to online RL. The main design choice is to use actions from an AWR policy as positive samples and actions from the current policy as negative samples. This is a reasonable combination, but the paper does not yet establish that it constitutes a substantial methodological advance over existing drift-based policies or critic-guided policy optimization. The authors need to identify precisely what new principle or capability DADC introduces.
  2. Unclear source of the performance gains. DADC combines several elements: an AWR-trained flow prior, critic-guided Langevin refinement, negative samples from the current policy, and drift correction. The reported results do not sufficiently isolate the contribution of the proposed dual-source drift mechanism from the gains provided by the additional components.
  3. Limited ablation coverage. The ablations evaluate the attraction and repulsion components on only two tasks. This evidence is too narrow to establish that both components are consistently useful across the benchmark suite.
  4. Insufficient analysis of robustness and computational cost. The method relies on critic estimates to select and refine anchors. Critic errors could therefore affect both the positive anchors and the learned drift field. The paper also introduces additional training cost (due to more sampling and sorting requirement), but it does not adequately establish whether the performance gains justify this overhead relative to simpler baselines.
Questions for authors

Please see weaknesses; more specifically:

  1. What is the precise novelty over prior drift-based policies in offline RL? Beyond applying the approach in an online setting and choosing AWR actions as positive samples and current-policy actions as negative samples, what new algorithmic insight does DADC provide?
  2. Can the authors isolate the contribution of the drift mechanism? Please compare against variants that use the AWR policy directly, remove Langevin refinement, and remove either the positive or negative anchors. Ideally, these comparisons should control for training compute.
Reviewer_XtybOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper begins by motivating how generative policy methods in online reinforcement learning mostly rely on positive advantage weighting and underutilize negative feedback from low-value actions.

To address this, the paper presents Dual-Source Anchor Drift Correction (DADC), an off-policy RL approach that trains a single-step implicit actor using a kernelized drift field alongside direct critic maximization. During training, attractive anchors are generated from a replay-trained conditional flow-matching prior followed by Q-guided Langevin refinement, while repulsive anchors are selected as the lowest critic-ranked actions sampled from the current actor.

At test time, only the single-pass actor is evaluated without iterative flow or Langevin steps. A number of experiments across 14 tasks from DeepMind Control Suite and HumanoidBench validate the approach and show consistent return improvements over FlowRL and other baselines with fast inference.

Critical strengths

The approach tackles an important latency trade-off in generative control policies by keeping multi-step flow integration and Q-guided Langevin refinement strictly at training time while deploying a fast single-step actor at inference time (achieving a 1.79x inference speedup over 1-step FlowRL in Table 4).

The approach demonstrates consistent performance improvements across all 14 continuous control tasks in DeepMind Control Suite and HumanoidBench, including clear gains over FlowRL within the same underlying codebase.

Critical weaknesses

The paper omits or leaves inconsistent several mathematical details needed for reproducing the approach and results, including the Gaussian kernel bandwidth h in Equation 3, how V(s) and Quantile = 0.9 are computed in Equation 8, and how the Langevin temperature in Equation 6 relates to the separate step size and noise scale in Table 3.

Baseline and ablation isolation: In Table 2 (Dog Run), the Local-Q only variant (α = 0) already outperforms every external baseline in Table 1, yet it is only reported on 2 of the 14 tasks. In addition, the paper does not verify whether the learned policy remains multimodal without entropy regularization, or compare bottom-m critic repulsion against standard unfiltered Drifting Model repulsion (m = M).

Questions for authors

In Table 2 (Dog Run), it seems like Local-Q only outperforms all eight external baselines in Table 1 and is much stronger than Naive Drift. Would it be possible to include Local-Q only across the remaining tasks so we can better disentangle how much of the performance gain comes from the critic backbone versus the drift correction?

In original Drifting Models, repulsive anchors are unfiltered model samples (m = M), which guarantees a density-matching fixed point. What happens in Table 2 if repulsive anchors use all M = 16 actor samples instead of only the bottom m = 4 critic-ranked samples? Also, is there empirical evidence showing that the actor preserves multiple action modes rather than collapsing across latent z?

The reproducibility and clarity of the paper could be improved significantly by specifying the missing details in Equations 3, 6, and 8; specifically how kernel bandwidth h is chosen across 6D to 38D action spaces, how V(s) and Quantile = 0.9 are defined for the advantage weights in Equation 8, and how temperature β reconciles with the separate step size and noise scale in Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.