acceptodds
Withdrawn from ICLR 2027

Gradient-Space Evolution Strategies for Diffusion Model Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning aligns diffusion models with learned reward functions, but repeated trajectory evaluation and gradient computation make training expensive. We propose Gradient-Space Evolution Strategies (GS-ES), which organizes alignment around population-based exploration at individual denoising steps. GS-ES perturbs a shared transition mean, evaluates candidate states through gradient-free ODE continuations, and aggregates reward-weighted score-function updates. Selected candidates guide subsequent exploration, while a timestep-dependent budget allocates evaluations according to the remaining rollout length. For Gaussian transitions, we derive an exact identity expressing the local update as a Jacobian pullback of weighted action perturbations. This connects evolution-inspired exploration to gradient-based parameter updates without a reference-policy KL penalty. Experiments with FLUX.1-dev and Wan2.1-T2V-1.3B show higher scores than the evaluated baselines across the listed image and video metrics. On FLUX.1-dev, GS-ES achieves a PickScore of 24.27, compared with 23.33 for DanceGRPO, and reaches the baseline's peak reward in approximately one fifth as many training steps.

Primary area:
generative models
Keywords:
Diffusion Reinforcement Learning, Multimodality, AIGC

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_urhkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GS-ES branches from intermediate denoising states, evaluates candidate ODE continuations with a reward model, and selects states for later exploration. Candidate rewards also weight a local Gaussian score update. The paper uses a timestep-dependent candidate budget and structured perturbations to organize this search. It evaluates the method on FLUX.1-dev and Wan2.1-T2V-1.3B.

Critical strengths
  1. The method shares one transition-mean evaluation across candidates from the same parent, then aggregates their weighted residuals before backpropagation. This is a concrete computational design.
  2. The paper distinguishes its exact pointwise Gaussian-score identity from claims about the expected update under structured sampling.
  3. The ablations test whether later-step budget allocation, selection, and orthogonal probes each contribute to reward gains.
Critical weaknesses
  1. The identity in Proposition 1 is a standard Gaussian score derivative. It does not explain why the full combination of selection, normalized group rewards, and ODE evaluation should optimize the stated trajectory objective.
  2. The main table reports point estimates only. There is no run-to-run variation for the gains over BranchGRPO and DiffusionNFT.
  3. GS-ES replaces gradient-enabled work with many gradient-free continuations. Measured step time is nearly unchanged on images and higher on video, so its efficiency claim is narrower than the forward-count reduction suggests.
  4. The faster PickScore convergence is measured in training steps. A comparison at equal wall-clock cost is needed because the methods perform different amounts of work per step.
Questions for authors

Please refer to weakness.

Reviewer_imw5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GS-ES for diffusion model alignment, combining structured action-space exploration, gradient-free ODE evaluation, reward-based selection, and timestep-dependent sampling budgets. A Gaussian score identity connects weighted perturbations to parameter gradients. Experiments on FLUX and Wan report improved alignment scores, with component ablations and faster convergence in training steps on FLUX PickScore.

Critical strengths
  1. Shared computation across candidates. Candidates reuse a transition mean, allowing reward-weighted perturbations to be aggregated into one backward pass per parent. Gradient-free continuations avoid retaining rollout activations, providing a concrete computational rationale for the proposed local exploration framework.
  2. Empirical coverage. Experiments report improvements across image and video alignment metrics, including held-out reward evaluators for images. Budget, selection, and exploration ablations on FLUX provide evidence supporting the proposed design choices in the tested setting.
Critical weaknesses
  1. Limited justification of the practical update. The Gaussian score identity does not establish ascent for the full trajectory objective under selected-parent sampling, deterministic ODE continuations, and normalized advantages. The combined impact of these choices on optimizing the final generation policy remains insufficiently characterized.
  2. Unvalidated timestep weighting. Each timestep receives equal aggregate weight despite different candidate counts and group-dependent reward normalization. The cost-based budget does not establish statistical efficiency, and the ablations do not examine whether alternative timestep weights improve optimization.
Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_5xpVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GS-ES, which explores multiple latent candidates around shared diffusion transition means, evaluates them through gradient-free ODE continuations, and combines reward-weighted score gradients with reward-based population selection. Experiments on FLUX.1-dev and Wan2.1-T2V-1.3B report improved automatic reward scores and faster convergence in training steps.

Critical strengths
  1. The method provides a clearly described framework for combining local candidate exploration, reward evaluation, score-gradient aggregation, and population selection.

  2. The Jacobian-pullback identity makes the connection between action-space perturbations and parameter gradients explicit.

  3. Experiments cover both text-to-image and text-to-video generation, with ablations on budget allocation, selection, and exploration noise.

Critical weaknesses

1. Limited novelty. Gradient-free ODE continuation is already used in related efficient diffusion RL methods (e.g., MixGRPO. AdaScope) and is not a novel contribution. The paper does not sufficiently establish what fundamental optimization capability is added by combining candidate exploration, reward-based selection, and score-gradient aggregation; the Gaussian score identity is also a direct consequence of the standard score-function formula.

2. Search gains are conflated with optimization gains. Reward-based selection can improve subsequent rollout quality by steering exploration toward high-reward latent states, but this search-time benefit does not necessarily indicate that the generator has learned a policy with higher expected reward. The paper does not establish a clear causal link between selecting high-scoring candidates and obtaining consistently beneficial parameter updates; selecting better states for subsequent rollouts is not equivalent to demonstrating that the resulting score-gradient updates improve the policy. Moreover, selection changes the state distribution from which later candidate groups are formed, while the optimization relies on group-relative advantages that require meaningful reward differences within each group. If selection concentrates exploration on similar high-reward states, the relative reward signal may become weak even when their absolute rewards are high. The authors should disentangle search gains from learning gains, for example by comparing search-only performance with that of the trained model evaluated without candidate search, and analyze how selection affects within-group reward variation and the resulting optimization signal.

3. Insufficient qualitative evaluation. The appendix contains only a small number of visual comparisons, and the advantages of GS-ES are difficult to discern. The ablations also lack visual examples, making it hard to assess how the proposed components affect actual image and video quality beyond automatic reward scores.

4. Incomplete baselines and ablations. The evaluation lacks direct comparisons with relevant efficient diffusion alignment methods, such as MixGRPO (Li et al., 2025) and AdaScope (Yan et al., 2026), which address training efficiency through mixed ODE--SDE sampling and adaptive timestep selection, respectively. More controlled ablations under matched computational budgets are also needed to isolate the contributions of the proposed components.

Questions for authors
  1. Beyond gradient-free ODE continuation and the standard Gaussian score identity, what is the key algorithmic novelty of GS-ES compared with existing branching and efficient diffusion RL methods?
  2. Can the authors disentangle search gains from learning gains by evaluating the trained model without search and comparing it with a search-only setting?
  3. Can the authors provide more representative qualitative results, including visual comparisons for the main ablations?
  4. Why are MixGRPO and AdaScope omitted, and can the authors compare against them under matched computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.