- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an actor-critic framework for diffusion model post-training. A separate pretrained diffusion backbone is adapted into a timestep-conditioned critic that predicts expected terminal rewards directly from noisy latent states. The critic is warmed up using rollout returns and subsequently trained alongside the actor, with its predictions providing a state-dependent baseline for PPO. The learned critic also supports inference-time guidance and multiple reward heads. Experiments on SD1.5 and SD3.5M evaluate single-reward optimization, value prediction, multi-reward training, and inference-time steering.
Critical strengths
-
Learning continuation values directly from noisy latents is a reasonable design. Supervision comes from completed rollouts, and latent-space evaluation avoids repeated intermediate image reconstruction and decoding. This provides a practical way to incorporate state-dependent value estimation into diffusion post-training.
-
The paper provides useful evidence for its training recipe, including ablations, and results support the importance of calibrating the critic and conditioning its readout.
Critical weaknesses
-
The evidence does not sufficiently isolate the benefit of the proposed diffusion critic. Section 4.2 states that the compared critics are trained to predict returns, whereas Appendix G.5 describes frozen CLIP encoders with direct cosine-similarity scores and direct use of HPS reward scores, without clearly specifying the return-regression setup. Freezing an encoder is not itself problematic, but it remains unclear whether the alternatives receive comparable value supervision, timestep information, and fitting budgets. Consequently, Tables 4 and 5 cannot be confidently interpreted as demonstrating an advantage of diffusion features over properly trained alternative value functions. This matters because, with gamma = lambda = 1, the actor uses the standard return-minus-baseline estimator. Establishing the value of this particular critic, beyond introducing a learned baseline, is therefore central to the contribution.
-
The evidence on modern generators is insufficient to establish the method's practical significance and generality. SD1.5 is useful for exploratory ablations, but its gains carry limited weight for current image-generation systems. SD3.5M is included, yet the advantages over DanceGRPO are relatively modest, thus advanced methods like MixGRPO are not carefully considerred. Also, evaluation on another relevant backbone, such as FLUX or BAGEL's image-generation component, would better test whether the critic design transfers beyond the Stable Diffusion family. Moreover, the controlled comparisons match rollout counts rather than total computation, and Appendix G.1 specifies different prompt allocations for DanceGRPO and the proposed method. Although these are legitimate practical configurations, they do not isolate the critic's contribution or establish its benefit under a fixed compute budget.
Questions for authors
-
For each critic in Tables 4 and 5, what are the trainable modules, regression targets, timestep inputs, and fitting budgets? How should the training description in Section 4.2 be reconciled with Appendix G.5?
-
With prompt allocation, rollout budget, and PPO settings controlled, how does the full latent-conditioned critic compare with a learned prompt-only baseline? Does it reduce gradient variance beyond accounting for prompt difficulty?
4 comments
Q5: Rollout budgets.
We agree that the current setup is not fully aligned in total compute. However, we believe that controlling the number of policy update steps is a fair setup. In the LLM domain, including VC-PPO [4], VAPO [5], and recent work [6], the standard protocol is to compare policies at the same number of policy update steps. The policy network is not optimized during critic warm-up, so the comparison in our paper is fair.
Q6: Learned prompt-only baseline.
We build a prompt-only baseline from the CLIP text encoder with an MLP head. During critic warm-up its explained variance reaches only 0.1785 (diffusion critic 0.4377), and training last for 20 steps and then collapses.
[1] Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization. Tao et al.
[2] MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE. Li et al.
[3] Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching. Wang et al.
[4] What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret. Yuan et al.
[5] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. Yue et al.
[6] To Critic or Not to Critic: GRPO vs PPO. Wei et al.
Gentle Reminder
Dear reviewer,
As the discussion period is coming to a close. We would love to kindly follow up on our rebuttal. We appreciate it if you could let us know whether your concerns are addressed. We would love to clarify further if there are any remaining questions.
Best regards,
The authors
Thank you for the detailed responses and additional experiments. While the rebuttal has strengthened the empirical evidence, the submission at this stage has many things that require thorough revision. Overall, I consider the paper borderline, and I am leaning toward maintaining my original rating of 2, although I acknowledge that there are reasonable arguments on both sides.
We thank the reviewer for the detailed feedback.
Q1: Critic comparison in Tables 4 and 5, and the mismatch between Section 4.2 and Appendix G.5.
We agree that the paper contains a mismatch. In the Section 4.2 experiments, the CLIP encoder is not frozen: the whole model is trainable. In fact, if the critic is not jointly trained, all experiments fail regardless of the reward model type. The actor quickly moves out of distribution, the critic can no longer give effective feedback, and training fails immediately. The frozen-CLIP description in Appendix G.5 belongs to a separate ablation study that we deleted by mistake; we show it below.
Critic training setup. In Tables 4 and 5, all critics are fully fine-tuned with all parameters trainable. The training target is the standard PPO critic loss. In the main paper, the HPS, CLIP, and BLIP critics have no additional timestep conditioning (a timestep-conditioned baseline is added below). The fitting budget is the same for all critics, with 15 steps of value pre-training before PPO training.
Diffusion critic vs. CLIP. First, we include the ablation study mentioned above. To make the effect of the diffusion critic clearer, we add a comparison with LPO [1], which uses a CLIP model as a latent reward model (LRM) and optimizes the generator using DPO with it. This experiment helps us to study the effect of joint training and the value model. LPO first fine-tunes the timestep-conditioned CLIP model on Pick-a-Pic to serve as a human-preference model. We run the optimization on Pick-a-Pic, since LPO officially released a checkpoint for it, and compare four setups:
The results clearly show how both joint training and our diffusion critic improve the policy.
Q2: Timestep conditioning.
Our previous setup did not add timestep conditioning to the alternative critics. We now add a timestep module and train the critic with critic warm-up followed by joint actor–critic training. Timestep conditioning helps the critic distinguish features at different timesteps, but the results remain far below both GRPO and the diffusion critic:
Q3: Evidence on modern generators and advanced baselines.
On SD3.5M and SD1.5 we use a very large batch size (10x DanceGRPO default setting), so the difference between the two algorithms is relatively small.
We add experiments on FLUX.1-dev using the official DanceGRPO codebase and its well-tuned baseline setup. The only difference is that we train at 512×512 resolution to accelerate the experiments. We train on HPSv2.1 for 200 iterations (this is also the default setting in the DanceGRPO code base):
Our method shows a clear gain over the baseline. In the paper, we compared only with DanceGRPO and FlowGRPO because our focus is the source of the learning signal. Methods such as MixGRPO [2] and CPS [3] are orthogonal to our work and can be added to our pipeline as building blocks: as the table shows, the MixGRPO technique also applies to our method and improves both training speed and results. We will include the MixGRPO results in the main paper.
Q4: Prompt allocation.
This is in fact an advantage of PPO: we do not need to form a group for each prompt, because the critic can assign a learning signal to each individual rollout. In the LLM literature, most papers control the total number of rollouts rather than the group sampling. Nevertheless, to test this, we also report results with group sampling, where normalization is done within each prompt group rather than across all images:
Adding group sampling to our PPO further improves the results, since it further reduces the variance.