- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a non-intrusive latent reward model (LRM) for diffusion preference alignment. Learnable register tokens are prepended to a frozen DiT and trained with an attention mask so that they can read internal DiT features without changing the generator’s parameters. Based on this lightweight reward model, the authors introduce Reward-Guided Sampling (RGS) for inference-time alignment and Reward-Gradient On-Policy Distillation (RG-OPD) for training-time alignment.
Critical strengths
- The proposed reward-model method is simple and easy to reproduce. There is no need to retrain the generator or train a separate heavy reward network.
- Preference prediction remains stable at high noise. In Fig. 3, baselines such as DiNa-LRM drop sharply from t=0.2 to t=0.8, while Reward Registers stay flat. At t=0.8, they also outperform the other latent reward models in Table 1.
- The same reward gradients are used for both training and inference. RG-OPD turns them into local on-policy targets, and RGS applies them directly during sampling. On HPSv3, alignment improves without the large MUSIQ / CLIP-IQA drop seen in ReFL and AlignProp. The magnitude-matched CFG control in Table 6 also suggests that the gains come from the reward direction, not just from larger step sizes.
Critical weaknesses
- Turning LoRA off at sampling time leaves the original velocity field unchanged, so a LoRA reward model on the same pretrained DiT can also keep v_ref. But the paper does not show that this is more accurate or cheaper than that alternative.
- Table 7 does not compare against DiNa-LRM, which is the most relevant shared-backbone baseline. It is treated as a main competitor in Table 1 and Fig. 3, but it is missing from Table 7. Peak memory, per-step latency, and reward gains of DiNa-LRM under the same RGS setting should be reported.
Questions for authors
- In Fig. 3, DiNa-LRM is close to Reward Registers at low noise, but drops clearly once t > 0.5. Why do you think this happens, and why can Reward Registers stay stable at high noise?
- Table 1 shows that latent reward models still lag endpoint reward models by a clear margin (for example, HPSv3 at 74.84 versus Reward Registers at 68.32). What do you see as the real advantage of an LRM, and in which settings is each type of reward model more useful?
- Training the registers requires endpoint reward models as teachers, yet RG-OPD and RGS can beat some methods that use those endpoint rewards directly, such as Flow-GRPO. Why do you think that happens?
5 comments
Response to Reviewer MWXa
Q2: latent reward models still lag endpoint ones in Table 1. What is the real advantage of an LRM, and when is each more useful? We compare the two types by input, cost and best use (Table R16; costs from Table 7).
- Table 1 is not like-for-like. Endpoint models score the clean image, while latent models score the noisy latent at t = 0.8, where much of the image is not yet determined.
- The advantage is a reward and gradient at every step, cheaply. Guidance and per-step training need the reward while the image is still noisy. An endpoint model can only be used there by decoding a predicted image and backpropagating through the VAE and the reward network at every step. Table 7 measures this: decode + ImageReward costs 215 ms per guided step and 4.12× CFG per image, versus 50 ms and 1.80× for the registers.
- This gives better results where the reward is used during generation. With the same HPS reward, RGS beats the endpoint-reward guidance baselines DNO and DEMON (+1.61 HPSv3 over DEMON on SD3, Table 3), and RG-OPD beats Flow-GRPO, which trains on the endpoint reward directly (Table R6).
Table R16. When to use each type of reward model.
| Endpoint RM (e.g. ImageReward) | Latent RM (ours) | |
|---|---|---|
| Input | clean image | noisy latent, any step |
| Reward + gradient during generation | needs decoding + backprop through VAE and RM | ✓, from the generator's own pass |
| Cost per guided step (Table 7) | 215 ms (decode + ImageReward) | 50 ms |
| Best use | final evaluation, best-of-N selection, teacher for the LRM | guidance, per-step credit assignment in training |
The two are complementary: the registers are trained from endpoint models as teachers, and they make that endpoint preference available at every denoising step.
Q3: the registers are trained from endpoint reward models, yet RG-OPD and RGS beat methods that use those rewards directly, such as Flow-GRPO. Why? We tested this with the same register reward in GRPO and RG-OPD (Table R17; Table R10 in our response to Reviewer Gbs7). The registers do not know more than their teacher; our methods use the same preference better.
- Dense, differentiable signal instead of one scalar per rollout. Flow-GRPO spreads one endpoint reward over all steps and normalizes it within a group of rollouts, so each step gets a noisy, indirect signal. The registers give each step its own reward gradient. A dense reward alone is not enough (point 2); the gain comes from using its gradient as the per-step target.
- The optimizer matters most. A per-step reward helps GRPO, but it stays far below RG-OPD even at 12.6× the compute. RG-OPD regresses each step onto a reward-shifted reference velocity, a low-variance supervised target, instead of estimating a policy gradient from sampled rollouts.
- The reference anchor limits over-optimization. The student stays close to the pretrained model, and its held-out metrics stay above base throughout training (Table R9).
- RGS vs. endpoint-reward guidance. DNO optimizes through the decoded image, and DEMON searches over decoded candidates; both need a decoded image for each correction, which is costly and limits how often they correct. RGS corrects cheaply at every step, including the early steps that set the layout.
Table R17. Same register reward: GRPO vs. RG-OPD (SD3, Table 2 protocol).
| Same register reward | Training GPU-h | HPSv3 |
|---|---|---|
| GRPO, terminal reward | 6.8 | 8.20 |
| GRPO, per-step reward | 77 | 9.23 |
| RG-OPD | 6.1 | 11.57 |
The gains are not specific to the teacher reward: held-out evaluators and human raters also prefer our methods (Tables R1–R4).
Dear Reviewer MWXa,
Thank you again for your encouraging assessment and thoughtful questions. We added matched DiNa-LRM comparisons covering memory, latency, and reward gains, and clarified high-noise prediction and the complementary roles of latent and endpoint rewards. As the discussion period draws to a close, we would appreciate hearing whether these comparisons and explanations address your questions.
Best regards,
The Authors
Response to Reviewer MWXa
We thank the reviewer for the careful review. We address each point below. Tables R1–R17 are numbered across our responses to all reviewers.
W1: a LoRA reward model on the same DiT also keeps v_ref. We compare the registers with DiNa-LRM, a LoRA reward model on the DiT (rank-64 LoRA on 12 layers), in the same pipelines (Table R14; Table R12 in our response to Reviewer Gbs7).
Table R14. DiNa-LRM (LoRA reward model) vs. reward registers.
W2: DiNa-LRM missing from Table 7. We add DiNa-LRM rows under Table 7's RGS setting (single GPU, bf16, 1024², 42 steps, CFG 4.5, batch 1; Table R15). The registers give a larger reward gain than DiNa-LRM, with less memory and time. DiNa needs more memory because it loads its own backbone and text encoders, while the registers share the generator's.
Table R15. New Table 7 rows: DiNa-LRM vs. reward registers under the same RGS setting. Reward gains are over SD3 + CFG on the Table 3 protocol.
We will add these rows to Table 7, with reward gains for the other guided baselines.
Q1: why does DiNa-LRM drop at t > 0.5 while registers stay stable? We explain the gap with the register design and test it with the Table 4 ablation (t = 0.8).
We will add this analysis to the paper.