acceptodds
Withdrawn from ICLR 2027

Advantage-Gated Anchoring for Diffusion Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Advantage-weighted regression enables diffusion models to learn from reward feedback, providing a practical approach to diffusion reinforcement learning. For samples with positive advantages, regression encourages fitting their associated velocity targets. However, negative advantages push predictions away from these targets, without reallocating probability mass to alternative outcomes as negative feedback does in policies that explicitly predict action probabilities. This motivates studying policy anchoring as an additional constraint on velocity updates, acting as a reference for learning from negative feedback. In this work, we analyze how anchor policies shape reward learning and training stability in advantage-weighted regression and reveal that anchoring helps stabilize negative-advantage updates, while applying it to positive-advantage samples can hinder reward learning. Based on these insights, we propose Advantage-Gated Anchoring (AGA), a simple yet effective method that anchors only negative-advantage samples. Experiments demonstrate faster reward improvement, a better balance between reward and image quality, and greater robustness to the choice of anchor policy.

Primary area:
generative models
Keywords:
diffusion models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DX2fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies policy anchoring in advantage-weighted reinforcement learning for diffusion and flow models. The starting observation is that negative advantages behave awkwardly under regression-based policy updates: instead of directly reducing the probability of an undesirable outcome, they encourage the model prediction to move away from the regression target, which can lead to unstable updates. The paper analyzes how an old-policy anchor changes this behavior and argues that positive and negative samples should not be treated symmetrically.
Based on this analysis, the authors propose Advantage-Gated Anchoring (AGA), which leaves positive-advantage regression unchanged and applies anchoring only to negative-advantage samples. The method is intentionally simple and does not introduce a separate anchor-strength hyperparameter. Experiments on SD3.5-Medium cover compositional generation, text rendering, and preference-model optimization, with comparisons against AWM, DiffusionNFT, AdvantageFlow, and RAM. The paper also includes ablations showing that overly strong anchoring can suppress reward learning, while applying anchor-induced repulsion to positive samples can increase reward at the cost of image quality and eventually destabilize training.

Critical strengths
  1. The distinction between fitting positively rewarded samples and handling negatively rewarded samples is well motivated. The ablations are also useful here: applying anchoring only to negative samples gives a better reward-quality tradeoff than anchoring both branches, while making the positive branch repulsive produces exactly the reward/quality divergence that the analysis predicts. This is a good example of the experiments actually testing the proposed mechanism.
  2. The comparison across four rollout/anchor/EMA configurations is valuable because these implementation choices can substantially affect forward RL methods. AGA maintains reward improvement across these settings, whereas DiffusionNFT is more sensitive to the choice of configuration. This makes the method more practically attractive than a result that only works under one carefully chosen anchor update rule.
Critical weaknesses
  1. The incremental contribution over closely related anchoring methods is somewhat narrow. The paper is operating in a crowded space. AdvantageFlow already studies the negative-curvature problem, FlowAWR and RVM introduce old-policy-centered corrections, and DiffusionNFT already treats positive and negative feedback asymmetrically. The paper is aware of these connections, but the final algorithm is essentially a particularly simple choice of where to apply an existing type of anchor.

  2. The experimental evidence is narrower than the general claims suggest. All main experiments use SD3.5-Medium. The three tasks provide useful reward diversity, but they are still one model family and one image-generation setting. The numerical margins are also not uniformly large. On compositional generation, AGA is essentially tied with AWM on GenEval; on OCR it ties DiffusionNFT on the optimized reward; its clearest advantage appears on the PickScore optimization task and in training stability. I would be more confident in the generality of the method with a second base model, a non-image setting, or human evaluation of the preference-alignment results.

Questions for authors
  1. Could the authors explain more concretely how AGA differs from a special case of AdvantageFlow, RVM, FlowAWR, or DiffusionNFT? I understand the motivation for using the anchor only on negative samples, but I would like a sharper statement of what is methodologically new beyond choosing this particular asymmetric coefficient rule.

  2. The “Human Preference Alignment” experiment uses PickScore as the optimization objective. Has the resulting improvement been validated with human raters? This would materially affect how strongly I interpret the preference-alignment claim.

  3. Have the authors tested AGA on another flow/diffusion backbone? Since the proposed argument is about advantage-weighted regression rather than a property specific to SD3.5-Medium, transfer to a second model would significantly strengthen the generality claim.

Reviewer_W1VmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies policy anchoring in advantage-weighted flow matching and proposes Advantage-Gated Anchoring (AGA). AGA preserves ordinary regression for positive-advantage samples and applies an anchor coefficient of �0��0� to negative-advantage samples, replacing their concave quadratic loss with a tangent linear surrogate. Experiments on SD3.5-Medium cover compositional generation, text rendering, and preference alignment, with ablations examining anchor strength, advantage gating, and policy-update configurations. Results show promising OCR and preference improvements, while GenEval performance matches AWM.

Critical strengths
  1. Simple and practical modification. AGA is straightforward to implement and removes an independent anchor-strength parameter.

  2. Relevant controlled ablations. The comparisons of shared, positive-only, and negative-only anchoring provide evidence supporting asymmetric treatment of feedback.

Critical weaknesses
  1. The central argument lacks sufficient precision. The paper moves from canceling negative curvature to claims about training stability without adequately explaining the gap. The resulting negative-sample loss is linear and generally remains unbounded below. Repeated intuitive explanations do not establish what guarantees, if any, apply to neural-network optimization with a moving anchor. This weakens both the presentation and the theoretical justification.

  2. The empirical evidence does not establish a sufficiently reliable comparative advantage. Important implementation and evaluation details are deferred to appendices absent from the supplied PDF. Seed variability is not reported, and shared hyperparameters may disadvantage baselines that require different settings. Quality improvements are also inconsistent: AGA matches AWM’s GenEval reward while scoring lower on all four reported DrawBench metrics. These limitations make the broader reward–quality and robustness claims premature.

Questions for authors
  1. What stability property follows from AGA’s linear negative-sample surrogate, beyond cancellation of prediction-space curvature? What role does the reference penalty play?

  2. Do the comparative gains persist across multiple seeds and individually tuned baseline configurations?

  3. Could you provide the referenced appendices containing baseline equivalences, training configurations, and evaluation details?

Reviewer_HdZfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes the different effects of policy anchoring on positive- and negative-advantage samples in velocity regression. Negative weights produce a concave error term, while anchoring positive samples can slow fitting or push predictions past the target. AGA leaves positive fitting unchanged and uses the old policy to form a tangent correction for negative samples. The experiments test reward learning and image quality on three SD3.5-Medium tasks.

Critical strengths
  1. The positive-only and negative-only anchor ablations isolate the central claim: negative-only anchoring retains more image quality at a similar final OCR reward.
  2. The paper exposes a concrete failure mode in positive-sample repulsion: OCR rises while DrawBench PickScore falls, followed by reward collapse.
Critical weaknesses
  1. The tangent correction controls the local regression objective. It does not establish that a negatively rated image becomes less likely under the resulting generator.
  2. The main evaluation uses one backbone. The analysis would be more convincing if the same reward–quality behavior appeared with another architecture or rollout policy.
  3. The paper reports no training-run variation. This matters for conclusions drawn from curve shape and the timing of collapse.
  4. The gains are uneven: AGA ties AWM on GenEval, and its DrawBench quality scores on that task are lower.
  5. Baselines share a fixed trainer configuration. This isolates the objective change, but does not establish that AGA beats individually tuned alternatives.
Questions for authors

Please refer to weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.