- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies policy anchoring in advantage-weighted reinforcement learning for diffusion and flow models. The starting observation is that negative advantages behave awkwardly under regression-based policy updates: instead of directly reducing the probability of an undesirable outcome, they encourage the model prediction to move away from the regression target, which can lead to unstable updates. The paper analyzes how an old-policy anchor changes this behavior and argues that positive and negative samples should not be treated symmetrically.
Based on this analysis, the authors propose Advantage-Gated Anchoring (AGA), which leaves positive-advantage regression unchanged and applies anchoring only to negative-advantage samples. The method is intentionally simple and does not introduce a separate anchor-strength hyperparameter. Experiments on SD3.5-Medium cover compositional generation, text rendering, and preference-model optimization, with comparisons against AWM, DiffusionNFT, AdvantageFlow, and RAM. The paper also includes ablations showing that overly strong anchoring can suppress reward learning, while applying anchor-induced repulsion to positive samples can increase reward at the cost of image quality and eventually destabilize training.
Critical strengths
- The distinction between fitting positively rewarded samples and handling negatively rewarded samples is well motivated. The ablations are also useful here: applying anchoring only to negative samples gives a better reward-quality tradeoff than anchoring both branches, while making the positive branch repulsive produces exactly the reward/quality divergence that the analysis predicts. This is a good example of the experiments actually testing the proposed mechanism.
- The comparison across four rollout/anchor/EMA configurations is valuable because these implementation choices can substantially affect forward RL methods. AGA maintains reward improvement across these settings, whereas DiffusionNFT is more sensitive to the choice of configuration. This makes the method more practically attractive than a result that only works under one carefully chosen anchor update rule.
Critical weaknesses
-
The incremental contribution over closely related anchoring methods is somewhat narrow. The paper is operating in a crowded space. AdvantageFlow already studies the negative-curvature problem, FlowAWR and RVM introduce old-policy-centered corrections, and DiffusionNFT already treats positive and negative feedback asymmetrically. The paper is aware of these connections, but the final algorithm is essentially a particularly simple choice of where to apply an existing type of anchor.
-
The experimental evidence is narrower than the general claims suggest. All main experiments use SD3.5-Medium. The three tasks provide useful reward diversity, but they are still one model family and one image-generation setting. The numerical margins are also not uniformly large. On compositional generation, AGA is essentially tied with AWM on GenEval; on OCR it ties DiffusionNFT on the optimized reward; its clearest advantage appears on the PickScore optimization task and in training stability. I would be more confident in the generality of the method with a second base model, a non-image setting, or human evaluation of the preference-alignment results.
Questions for authors
-
Could the authors explain more concretely how AGA differs from a special case of AdvantageFlow, RVM, FlowAWR, or DiffusionNFT? I understand the motivation for using the anchor only on negative samples, but I would like a sharper statement of what is methodologically new beyond choosing this particular asymmetric coefficient rule.
-
The “Human Preference Alignment” experiment uses PickScore as the optimization objective. Has the resulting improvement been validated with human raters? This would materially affect how strongly I interpret the preference-alignment claim.
-
Have the authors tested AGA on another flow/diffusion backbone? Since the proposed argument is about advantage-weighted regression rather than a property specific to SD3.5-Medium, transfer to a second model would significantly strengthen the generality claim.