acceptodds
Under review as a conference paper at ICLR 2027

Explicit Critic Guidance for Aligning Diffusion Models

Abstract

Online reinforcement learning is becoming increasingly important for aligning diffusion models with non-differentiable objectives. However, existing methods still face limitations in assigning fine-grained credit along denoising trajectories and in realizing stable value-based optimization. We propose a state-aligned latent actor-critic framework for diffusion post-training, in which the diffusion model serves as its own timestep-conditioned value function and predicts values directly on noisy latent states. This enables trajectory-level PPO training, supports stable actor-critic optimization with simple conditioning and value pretraining strategies, and naturally allows the learned critic to be reused for inference-time steering. We further extend the framework to multi-reward optimization, where joint training with complementary rewards helps alleviate reward hacking. Across both UNet- and DiT-based backbones, our method consistently outperforms prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks, while test-time steering provides additional gains in generation quality.

Primary area:
generative models
Keywords:
Diffusion Models, Reinforcement Learning, Post Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QygtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an actor-critic framework for diffusion model post-training. A separate pretrained diffusion backbone is adapted into a timestep-conditioned critic that predicts expected terminal rewards directly from noisy latent states. The critic is warmed up using rollout returns and subsequently trained alongside the actor, with its predictions providing a state-dependent baseline for PPO. The learned critic also supports inference-time guidance and multiple reward heads. Experiments on SD1.5 and SD3.5M evaluate single-reward optimization, value prediction, multi-reward training, and inference-time steering.

Critical strengths
  1. Learning continuation values directly from noisy latents is a reasonable design. Supervision comes from completed rollouts, and latent-space evaluation avoids repeated intermediate image reconstruction and decoding. This provides a practical way to incorporate state-dependent value estimation into diffusion post-training.

  2. The paper provides useful evidence for its training recipe, including ablations, and results support the importance of calibrating the critic and conditioning its readout.

Critical weaknesses
  1. The evidence does not sufficiently isolate the benefit of the proposed diffusion critic. Section 4.2 states that the compared critics are trained to predict returns, whereas Appendix G.5 describes frozen CLIP encoders with direct cosine-similarity scores and direct use of HPS reward scores, without clearly specifying the return-regression setup. Freezing an encoder is not itself problematic, but it remains unclear whether the alternatives receive comparable value supervision, timestep information, and fitting budgets. Consequently, Tables 4 and 5 cannot be confidently interpreted as demonstrating an advantage of diffusion features over properly trained alternative value functions. This matters because, with gamma = lambda = 1, the actor uses the standard return-minus-baseline estimator. Establishing the value of this particular critic, beyond introducing a learned baseline, is therefore central to the contribution.

  2. The evidence on modern generators is insufficient to establish the method's practical significance and generality. SD1.5 is useful for exploratory ablations, but its gains carry limited weight for current image-generation systems. SD3.5M is included, yet the advantages over DanceGRPO are relatively modest, thus advanced methods like MixGRPO are not carefully considerred. Also, evaluation on another relevant backbone, such as FLUX or BAGEL's image-generation component, would better test whether the critic design transfers beyond the Stable Diffusion family. Moreover, the controlled comparisons match rollout counts rather than total computation, and Appendix G.1 specifies different prompt allocations for DanceGRPO and the proposed method. Although these are legitimate practical configurations, they do not isolate the critic's contribution or establish its benefit under a fixed compute budget.

Questions for authors
  1. For each critic in Tables 4 and 5, what are the trainable modules, regression targets, timestep inputs, and fitting budgets? How should the training description in Section 4.2 be reconciled with Appendix G.5?

  2. With prompt allocation, rollout budget, and PPO settings controlled, how does the full latent-conditioned critic compare with a learned prompt-only baseline? Does it reduce gradient variance beyond accounting for prompt difficulty?

4 comments

We thank the reviewer for the detailed feedback.

Q1: Critic comparison in Tables 4 and 5, and the mismatch between Section 4.2 and Appendix G.5.

We agree that the paper contains a mismatch. In the Section 4.2 experiments, the CLIP encoder is not frozen: the whole model is trainable. In fact, if the critic is not jointly trained, all experiments fail regardless of the reward model type. The actor quickly moves out of distribution, the critic can no longer give effective feedback, and training fails immediately. The frozen-CLIP description in Appendix G.5 belongs to a separate ablation study that we deleted by mistake; we show it below.

Critic training setup. In Tables 4 and 5, all critics are fully fine-tuned with all parameters trainable. The training target is the standard PPO critic loss. In the main paper, the HPS, CLIP, and BLIP critics have no additional timestep conditioning (a timestep-conditioned baseline is added below). The fitting budget is the same for all critics, with 15 steps of value pre-training before PPO training.

Diffusion critic vs. CLIP. First, we include the ablation study mentioned above. To make the effect of the diffusion critic clearer, we add a comparison with LPO [1], which uses a CLIP model as a latent reward model (LRM) and optimizes the generator using DPO with it. This experiment helps us to study the effect of joint training and the value model. LPO first fine-tunes the timestep-conditioned CLIP model on Pick-a-Pic to serve as a human-preference model. We run the optimization on Pick-a-Pic, since LPO officially released a checkpoint for it, and compare four setups:

SetupCritic / reward modelCritic trainingPickScore ↑
SD1.5NoneNone19.68
DPOCLIP-LRMfrozen21.69
PPOCLIP-LRMfrozen16.78 (fail)
PPOCLIP-LRMjoint training22.02
PPO (ours)Diffusion criticjoint training24.58

The results clearly show how both joint training and our diffusion critic improve the policy.

Q2: Timestep conditioning.

Our previous setup did not add timestep conditioning to the alternative critics. We now add a timestep module and train the critic with critic warm-up followed by joint actor–critic training. Timestep conditioning helps the critic distinguish features at different timesteps, but the results remain far below both GRPO and the diffusion critic:

MethodTimestep conditioningHPSv2.1 ↑
PPO + CLIP critic✗0.3323
PPO + CLIP critic✓0.3455
DanceGRPO–0.3779
PPO + diffusion critic (ours)✓0.3912

Q3: Evidence on modern generators and advanced baselines.

On SD3.5M and SD1.5 we use a very large batch size (10x DanceGRPO default setting), so the difference between the two algorithms is relatively small.

We add experiments on FLUX.1-dev using the official DanceGRPO codebase and its well-tuned baseline setup. The only difference is that we train at 512×512 resolution to accelerate the experiments. We train on HPSv2.1 for 200 iterations (this is also the default setting in the DanceGRPO code base):

Method (FLUX.1-dev)HPSv2.1 ↑GenEval ↑
FLUX.1-dev0.3058
DanceGRPO0.3540
MixGRPO0.3638
Ours0.3646
Ours + MixGRPO0.3669

Our method shows a clear gain over the baseline. In the paper, we compared only with DanceGRPO and FlowGRPO because our focus is the source of the learning signal. Methods such as MixGRPO [2] and CPS [3] are orthogonal to our work and can be added to our pipeline as building blocks: as the table shows, the MixGRPO technique also applies to our method and improves both training speed and results. We will include the MixGRPO results in the main paper.

Q4: Prompt allocation.

This is in fact an advantage of PPO: we do not need to form a group for each prompt, because the critic can assign a learning signal to each individual rollout. In the LLM literature, most papers control the total number of rollouts rather than the group sampling. Nevertheless, to test this, we also report results with group sampling, where normalization is done within each prompt group rather than across all images:

SettingUpdateImages / promptAdvantage normalizationHPSv2.1 ↑
S1GRPO12within prompt group0.3540
S2PPO12within prompt group0.3665
S4PPO1across all images0.3646

Adding group sampling to our PPO further improves the results, since it further reduces the variance.

Q5: Rollout budgets.

We agree that the current setup is not fully aligned in total compute. However, we believe that controlling the number of policy update steps is a fair setup. In the LLM domain, including VC-PPO [4], VAPO [5], and recent work [6], the standard protocol is to compare policies at the same number of policy update steps. The policy network is not optimized during critic warm-up, so the comparison in our paper is fair.

Q6: Learned prompt-only baseline.

We build a prompt-only baseline from the CLIP text encoder with an MLP head. During critic warm-up its explained variance reaches only 0.1785 (diffusion critic 0.4377), and training last for 20 steps and then collapses.

[1] Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization. Tao et al.

[2] MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE. Li et al.

[3] Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching. Wang et al.

[4] What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret. Yuan et al.

[5] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. Yue et al.

[6] To Critic or Not to Critic: GRPO vs PPO. Wei et al.

AuthorsOpenReview ↗

Gentle Reminder

Dear reviewer,

As the discussion period is coming to a close. We would love to kindly follow up on our rebuttal. We appreciate it if you could let us know whether your concerns are addressed. We would love to clarify further if there are any remaining questions.

Best regards,

The authors

Thank you for the detailed responses and additional experiments. While the rebuttal has strengthened the empirical evidence, the submission at this stage has many things that require thorough revision. Overall, I consider the paper borderline, and I am leaning toward maintaining my original rating of 2, although I acknowledge that there are reasonable arguments on both sides.

Reviewer_YafZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a diffusion backbone to predict expected terminal reward from noisy latent states. These predictions provide state-dependent advantages for PPO updates and gradients for inference-time guidance. The evaluation tests whether the critic ranks intermediate states and actions accurately, then measures its effect on single-reward and multi-reward alignment.

Critical strengths
  1. The paper tests the premise behind its method. Early-state ranking rises from 58.8% using realized final rewards to 69.0% using the critic.
  2. Warm-up and timestep-conditioning ablations show why the proposed training recipe matters, beyond the choice of backbone.
Critical weaknesses
  1. The state-ranking study uses one frozen SD3.5M policy and HPSv2.1. It does not show whether the early-state advantage persists across policies and reward types.
  2. On SD3.5M, several controlled gains over DanceGRPO are small. The paper reports no run-to-run variation for judging these differences.
  3. The controlled comparisons match rollout budgets, not total compute. The extra critic costs 29% more time per iteration in the reported HPS setting.
  4. The multi-reward result favors GenEval at the expense of HPSv2.1 and CLIP on SD3.5M. The higher reward sum does not establish a better overall trade-off.
  5. Inference guidance improves the optimized scores, but its added latent-gradient cost and independent visual-quality effects are not quantified together.
Questions for authors

Please refer to the weakness.

2 comments

We thank the reviewer for the detailed feedback. We respond to each point below.

Q1: The state-ranking study uses a single policy and reward.

We add SD3.5M on GenEval, and SD1.5 on both HPSv2.1 and GenEval, and FLUX:

PolicyRewardGRPO Ranking acc. (early) ↑Critic Ranking acc. (early) ↑
SD3.5MGenEval56.8%60.6%
SD1.5HPSv2.154.3%59.8%
SD1.5GenEval55.6%62.3%
FLUXHPSv2.157.2%61.2%

Q2: Gains on SD3.5M are small, and no run-to-run variation is reported.

On SD3.5M we use a very large batch size (10x DanceGRPO default setting), so the difference between the two algorithms is relatively small. We therefore add results on FLUX.1-dev, following the default codebase and setup of DanceGRPO, with HPSv2.1 as rewards. Our method clearly improves over the baseline. We also report run-to-run variation (mean ± std over N seeds):

Method (FLUX.1-dev)HPSv2.1 ↑
FLUX.1-dev0.3058
DanceGRPO0.3540
Ours0.3646 ± 0.0005240

Q3: Comparisons match rollout budgets, not total compute.

In the LLM literature, it is widely accepted that a fair comparison between PPO and GRPO matches the number of policy update steps[1][2][3], so we believe the comparisons in our paper are fair. The policy network is not updated during critic warm-up or critic optimization, so the number of policy update steps is controlled, and the gain comes from better training signals.

Q4: The multi-reward result does not establish a better overall trade-off.

The multi-reward setting is an extension of our PPO. Our goal there is to show that training extends to multiple rewards with minimal modification: we do not need a separate critic for each reward model, and a single critic with multiple value heads suffices. We did not tune these experiments heavily, and the multi-reward results depend strongly on the mixture of prompts and rewards.

Q5: The cost and visual-quality effects of inference guidance are not quantified together.

We agree with the reviewer that critic guidance introduces extra cost at inference time. However, the improvement does not come simply from spending more compute. Many prior works have shown that merely increasing the number of denoising steps brings little benefit and can even degrade performance. To verify this in our setting, we sample from the post-trained checkpoint with 50 and 75 denoising steps and no critic guidance, and the improvement is not significant. This indicates that the gain from guidance comes from the effective signal provided by the critic, not from the additional computation. We also evaluate on ImageReward, which is not used as a training reward, as an independent measure of visual quality, and critic guidance improves it as well.

SettingDenoising stepsHPSv2.1 ↑ImageRewardTime / image (s)
Ours250.36461.36792.08
Ours500.36521.36814.13
Ours750.36381.34666.19
Ours + critic guidance250.36891.36886.12

[1] What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret. Yuan et al.

[2] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. Yue et al.

[3] To Critic or Not to Critic: GRPO vs PPO. Wei et al.

AuthorsOpenReview ↗

Gentle Reminder

Dear reviewer,

As the discussion period is coming to a close. We would love to kindly follow up on our rebuttal. We appreciate it if you could let us know whether your concerns are addressed. We would love to clarify further if there are any remaining questions.

Best regards,

The authors

Reviewer_Pij8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper goes back to PPO with a learned critic for diffusion RL, instead of critic-free GRPO. The critic is a separate copy of the pretrained diffusion model (SD1.5 UNet or SD3.5M DiT) with a small head, which predicts the final reward from the noisy latent at each step. It is trained with MSE to the final reward on the policy's own rollouts, with a short warm-up, and the advantage is R−V(sk)R - V(s_k), so each step gets its own advantage instead of the same one as in GRPO. The critic gradient can also be used for guidance at inference time, and the paper extends it to multiple rewards with separate heads. It reports better results than DDPO and DanceGRPO on CLIP, HPS, PickScore, GenEval and OCR.

Critical strengths
  1. The motivation is clear, GRPO gives the same advantage to every denoising step, and the state-ranking experiment shows final reward is a weak signal at early steps.
  2. The paper is clearly written and easy to understand.
Critical weaknesses
  1. The gain from the critic is not isolated. DanceGRPO samples a group per prompt while the proposed method uses one image per prompt and more prompts, so the comparison changes both the baseline and the data. There is no ablation with the same PPO and data using a group-mean baseline instead of the critic.
  2. The critic model is quite expensive as it requires another copy of the diffusion model.
  3. The results didn't demonstrate the effectiveness of the method from the numbers.
Questions for authors

See the weaknesses part.

2 comments

We thank the reviewer for the constructive feedback. To address the concerns, we ran additional experiments on FLUX.1-dev. Since the official DanceGRPO repository provides well-tuned hyperparameters, we follow its setup; the only difference is that we train at 512×512 resolution to accelerate the experiments.

Q1: The gain from the critic is not isolated.

We agree that the current setting also changes the prompt allocation. Our method uses a critic to assign a learning signal to each rollout, so from the policy's perspective the only difference is the source of that signal. To address this concern, we compare four settings:

  • S1: DanceGRPO baseline.
  • S2: PPO update, 12 images per prompt (total rollout is the same as baseline), advantages normalized within each prompt group.
  • S3: PPO update, 1 image per prompt (total rollout is the same as baseline), advantages normalized across all images.
SettingUpdateImages / promptAdvantage normalizationHPSv2.1 ↑
S1GRPO12within prompt group0.3540
S2PPO12within prompt group0.3665
S4PPO1across all images0.3646

As shown in the table, group sampling and group normalization further boost our single-prompt PPO. The critic gives feedback to each individual rollout, and group sampling with normalization further reduces the variance of this feedback.

Q2: The critic is expensive, as it requires another copy of the diffusion model.

A single critic model can indeed be large. However, most current training frameworks support FSDP, which shards the model across GPUs. We use 16 GPUs in our experiments, and the extra overhead is 2.78 GB per GPU, which is tolerable. Peak memory also increases very little, because the actor and critic updates are decoupled: we first optimize the actor and release its computation graph, which leaves enough memory to optimize the critic.

Q3: The numbers do not demonstrate the effectiveness of the method.

In our SD1.5 and SD3.5 experiments, we used a very large batch size (10× the default DanceGRPO setting), so the gap between the methods is relatively small. In the new FLUX experiments, PPO is clearly better than DanceGRPO, and the gain is larger still when group-mean normalization is added (see the table above).

In addition, the critic used in training can be reused at inference time to further improve the results, which is not possible with GRPO. The SD1.5 and SD3.5 results are in Appendices F and K.2. The critic also further improves performance on FLUX:

Method (FLUX.1-dev)HPSv2.1 ↑
Ours0.3646
Ours + inference-time critic guidance0.3689

Finally, we test the generalization of our method by training on HPSv2.1 and evaluating on ImageReward and PickScore:

Method (trained on HPSv2.1)HPSv2.1 ↑ImageReward ↑ (OOD)PickScore ↑ (OOD)
FLUX.1-dev0.30581.12840.2299
DanceGRPO0.35401.32660.2280
Ours0.36461.36790.2301

Our method achieves better OOD results. This is an advantage of PPO: during training, the actor processes more unique prompts, whereas GRPO has to sample a group for each prompt, which reduces the number of prompts seen.

AuthorsOpenReview ↗

Gentle reminder

Dear reviewer,

As the discussion period is coming to a close. We would love to kindly follow up on our rebuttal. We appreciate it if you could let us know whether your concerns are addressed. We would love to clarify further if there are any remaining questions.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.