acceptodds
Under review as a conference paper at ICLR 2027

Latent Reward Registers for Diffusion Preference Alignment

Abstract

Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, which creates a severe temporal credit-assignment problem across the denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents. Learnable, position-free register tokens are prepended as an auxiliary read path to a frozen Diffusion Transformer (DiT), extracting preference signals without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward field spans the full denoising trajectory and supports two alignment strategies. For training, Reward-Gradient On-Policy Distillation (RG-OPD) converts this dense reward field into per-step targets at states visited by the current generator, replacing rollout-intensive policy gradients with direct on-policy distillation. For inference, Reward-Guided Sampling (RGS) steers trajectories with magnitude-matched reward-gradient corrections and no parameter updates. Empirically, at high noise levels (t = 0.8), the registers reach the highest pairwise accuracy among the evaluated latent reward models. RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x. RGS achieves significant reward improvement with a favorable reward-quality balance against training-free baselines.

Primary area:
reinforcement learning
Keywords:
Diffusion reinforcement learning, Latent reward model.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MWXaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a non-intrusive latent reward model (LRM) for diffusion preference alignment. Learnable register tokens are prepended to a frozen DiT and trained with an attention mask so that they can read internal DiT features without changing the generator’s parameters. Based on this lightweight reward model, the authors introduce Reward-Guided Sampling (RGS) for inference-time alignment and Reward-Gradient On-Policy Distillation (RG-OPD) for training-time alignment.

Critical strengths
  1. The proposed reward-model method is simple and easy to reproduce. There is no need to retrain the generator or train a separate heavy reward network.
  2. Preference prediction remains stable at high noise. In Fig. 3, baselines such as DiNa-LRM drop sharply from t=0.2 to t=0.8, while Reward Registers stay flat. At t=0.8, they also outperform the other latent reward models in Table 1.
  3. The same reward gradients are used for both training and inference. RG-OPD turns them into local on-policy targets, and RGS applies them directly during sampling. On HPSv3, alignment improves without the large MUSIQ / CLIP-IQA drop seen in ReFL and AlignProp. The magnitude-matched CFG control in Table 6 also suggests that the gains come from the reward direction, not just from larger step sizes.
Critical weaknesses
  1. Turning LoRA off at sampling time leaves the original velocity field unchanged, so a LoRA reward model on the same pretrained DiT can also keep v_ref. But the paper does not show that this is more accurate or cheaper than that alternative.
  2. Table 7 does not compare against DiNa-LRM, which is the most relevant shared-backbone baseline. It is treated as a main competitor in Table 1 and Fig. 3, but it is missing from Table 7. Peak memory, per-step latency, and reward gains of DiNa-LRM under the same RGS setting should be reported.
Questions for authors
  1. In Fig. 3, DiNa-LRM is close to Reward Registers at low noise, but drops clearly once t > 0.5. Why do you think this happens, and why can Reward Registers stay stable at high noise?
  2. Table 1 shows that latent reward models still lag endpoint reward models by a clear margin (for example, HPSv3 at 74.84 versus Reward Registers at 68.32). What do you see as the real advantage of an LRM, and in which settings is each type of reward model more useful?
  3. Training the registers requires endpoint reward models as teachers, yet RG-OPD and RGS can beat some methods that use those endpoint rewards directly, such as Flow-GRPO. Why do you think that happens?
5 comments
AuthorsOpenReview ↗

Response to Reviewer MWXa

We thank the reviewer for the careful review. We address each point below. Tables R1–R17 are numbered across our responses to all reviewers.

W1: a LoRA reward model on the same DiT also keeps v_ref. We compare the registers with DiNa-LRM, a LoRA reward model on the DiT (rank-64 LoRA on 12 layers), in the same pipelines (Table R14; Table R12 in our response to Reviewer Gbs7).

  1. One forward pass instead of two. The register tokens run alongside the image tokens in the frozen DiT's pass, so each step returns v_ref and the reward together. A LoRA reward model changes the DiT's weights, so each step needs one pass with the LoRA off (v_ref) and one with it on (reward and gradient).
  2. Better and cheaper than DiNa-LRM. The registers evaluate the reward 1.5× faster and use 41% less memory (as released, DiNa-LRM also loads its own backbone and text encoders, which accounts for most of the memory gap), give higher reward in both pipelines, and are more accurate at high noise (Table 1, Fig. 3).

Table R14. DiNa-LRM (LoRA reward model) vs. reward registers.

DiNa-LRM (LoRA RM)Reward registers (ours)
Forward passes per guided step2 (generator + reward model)1 (shared)
Reward eval / step (ms)76.550 (1.5× faster)
Peak memory (RGS)29.9 GiB17.6 GiB (−41%)
Rewards per pass12 (HPS, ImageReward; at most +4% time)
RGS / RG-OPD HPSv310.57 / 11.4510.70 / 11.57

W2: DiNa-LRM missing from Table 7. We add DiNa-LRM rows under Table 7's RGS setting (single GPU, bf16, 1024², 42 steps, CFG 4.5, batch 1; Table R15). The registers give a larger reward gain than DiNa-LRM, with less memory and time. DiNa needs more memory because it loads its own backbone and text encoders, while the registers share the generator's.

Table R15. New Table 7 rows: DiNa-LRM vs. reward registers under the same RGS setting. Reward gains are over SD3 + CFG on the Table 3 protocol.

Method (SD3, RGS)Trainable params (M)Peak mem (GiB)Reward eval / step (ms)Latency / img (s, ×CFG)ΔHPSv3ΔImageReward
SD3 + CFG—16.9—2.99 (1.00×)——
DiNa-LRM + RGS167.629.976.56.03 (2.02×)+2.02+0.14
Reward registers + RGS (ours)147.617.6505.38 (1.80×)+2.15+0.16

We will add these rows to Table 7, with reward gains for the other guided baselines.

Q1: why does DiNa-LRM drop at t > 0.5 while registers stay stable? We explain the gap with the register design and test it with the Table 4 ablation (t = 0.8).

  1. High-noise evidence is weak and global. Single image tokens carry little visible content; what remains is the emerging layout and prompt alignment, which the frozen DiT still encodes because it must denoise from that state.
  2. Registers accumulate it across blocks. The 32 register tokens attend to all image and text tokens in each of the first 12 blocks and carry their state from block to block (Eq. 14), so weak evidence adds up. DiNa-LRM reads the reward from LoRA-adapted image features, so its signal is limited to what those features retain when the input is mostly noise.
  3. Table 4 isolates this. Without the persistent register state, accuracy is 64.29, below DiNa-LRM (66.47); with it, 68.32, above DiNa-LRM.

We will add this analysis to the paper.

AuthorsOpenReview ↗

Response to Reviewer MWXa

Q2: latent reward models still lag endpoint ones in Table 1. What is the real advantage of an LRM, and when is each more useful? We compare the two types by input, cost and best use (Table R16; costs from Table 7).

  1. Table 1 is not like-for-like. Endpoint models score the clean image, while latent models score the noisy latent at t = 0.8, where much of the image is not yet determined.
  2. The advantage is a reward and gradient at every step, cheaply. Guidance and per-step training need the reward while the image is still noisy. An endpoint model can only be used there by decoding a predicted image and backpropagating through the VAE and the reward network at every step. Table 7 measures this: decode + ImageReward costs 215 ms per guided step and 4.12× CFG per image, versus 50 ms and 1.80× for the registers.
  3. This gives better results where the reward is used during generation. With the same HPS reward, RGS beats the endpoint-reward guidance baselines DNO and DEMON (+1.61 HPSv3 over DEMON on SD3, Table 3), and RG-OPD beats Flow-GRPO, which trains on the endpoint reward directly (Table R6).

Table R16. When to use each type of reward model.

Endpoint RM (e.g. ImageReward)Latent RM (ours)
Inputclean imagenoisy latent, any step
Reward + gradient during generationneeds decoding + backprop through VAE and RM✓, from the generator's own pass
Cost per guided step (Table 7)215 ms (decode + ImageReward)50 ms
Best usefinal evaluation, best-of-N selection, teacher for the LRMguidance, per-step credit assignment in training

The two are complementary: the registers are trained from endpoint models as teachers, and they make that endpoint preference available at every denoising step.

Q3: the registers are trained from endpoint reward models, yet RG-OPD and RGS beat methods that use those rewards directly, such as Flow-GRPO. Why? We tested this with the same register reward in GRPO and RG-OPD (Table R17; Table R10 in our response to Reviewer Gbs7). The registers do not know more than their teacher; our methods use the same preference better.

  1. Dense, differentiable signal instead of one scalar per rollout. Flow-GRPO spreads one endpoint reward over all steps and normalizes it within a group of rollouts, so each step gets a noisy, indirect signal. The registers give each step its own reward gradient. A dense reward alone is not enough (point 2); the gain comes from using its gradient as the per-step target.
  2. The optimizer matters most. A per-step reward helps GRPO, but it stays far below RG-OPD even at 12.6× the compute. RG-OPD regresses each step onto a reward-shifted reference velocity, a low-variance supervised target, instead of estimating a policy gradient from sampled rollouts.
  3. The reference anchor limits over-optimization. The student stays close to the pretrained model, and its held-out metrics stay above base throughout training (Table R9).
  4. RGS vs. endpoint-reward guidance. DNO optimizes through the decoded image, and DEMON searches over decoded candidates; both need a decoded image for each correction, which is costly and limits how often they correct. RGS corrects cheaply at every step, including the early steps that set the layout.

Table R17. Same register reward: GRPO vs. RG-OPD (SD3, Table 2 protocol).

Same register rewardTraining GPU-hHPSv3
GRPO, terminal reward6.88.20
GRPO, per-step reward779.23
RG-OPD6.111.57

The gains are not specific to the teacher reward: held-out evaluators and human raters also prefer our methods (Tables R1–R4).

Dear Reviewer MWXa,

Thank you again for your encouraging assessment and thoughtful questions. We added matched DiNa-LRM comparisons covering memory, latency, and reward gains, and clarified high-noise prediction and the complementary roles of latent and endpoint rewards. As the discussion period draws to a close, we would appreciate hearing whether these comparisons and explanations address your questions.

Best regards,
The Authors

Reviewer_Gbs7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Diffusion RL methods like Flow-GRPO only get a reward at the final image, so every denoising step gets the same advantage and there is no per-step credit. This paper adds learnable register tokens to a frozen SD3-M/FLUX DiT that read its features (without changing the generation) and predict the HPS/ImageReward ranking from noisy latents. The gradient of this latent reward gives a per-step signal, used in two ways: RG-OPD trains a LoRA student to match the reference step plus a small push along the reward gradient, and RGS applies the same push at sampling time without training. The paper reports better HPSv3 than Flow-GRPO and AlignProp, and 14-33x fewer GPU hours than Flow-GRPO.

Critical strengths

The motivation is clear, sparse terminal reward is a real problem in diffusion RL, and the register design is simple. The same latent reward is used for both training and inference, and RG-OPD avoids both backprop through the full chain and high-variance policy gradients.

Critical weaknesses
  1. The evaluation is circular. The registers are trained to imitate HPS/ImageReward, RG-OPD and RGS optimize them, and the main metrics are again HPSv3/ImageReward. There is no independent metric or user study, and Fig. 9 already shows the true HPSv2 peaking around epoch 50 and then dropping while the register reward keeps going up.
  2. The 33x speedup is measured against Flow-GRPO at 9.77 HPSv3, where it is still improving when stopped (and below the 10.26 it gets in Table 2). The register training cost is not counted either.
  3. It is not clear where the gain comes from. Compared to Flow-GRPO the paper changes both the signal (per-step vs. final reward) and the optimizer (gradient regression vs. policy gradient). A simple check would be Flow-GRPO with per-step group-normalized register scores as the advantage. Also missing are comparisons with other step-wise methods (SPO, TempFlow-GRPO) and with using DiNa-LRM/LRM gradients in the same RG-OPD/RGS pipeline. The reward accuracy gain over DiNa-LRM in Table 1 is small and DiNa is better at low noise.
Questions for authors

Does the register ranking on actual student/sampler latents agree with the final reward?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Gbs7

We thank the reviewer for the careful review. We address each point below. Tables R1–R17 are numbered across our responses to all reviewers.

Q1: circular evaluation; Fig. 9 looks like reward hacking. Beyond MUSIQ and CLIP-IQA in the paper, we evaluated every method with held-out evaluators that no method trains on and with a blinded human study (Tables R1–R4, in our response to Reviewer TFAL, Q1), and re-scored every saved RG-OPD checkpoint with them, with ReFL as a known hacking case (Table R9).

  1. Circular evaluation. Ours is best on Q-Align quality and DeQA in all four settings, and raters prefer ours in all eight comparisons (66–97%; seven with CIs above 50%).
  2. No reward hacking. Along RG-OPD training, every held-out metric stays above base, while ReFL falls well below base on MUSIQ, CLIP-IQA, VQAScore and DeQA.

Table R9. Held-out evaluation of every saved checkpoint (SD3, 256 prompts). Each cell is the change from the base model; all metrics are higher-is-better.

RG-OPD +HPS (ours)Baseep30ep60 (paper)ep150ep300
HPSv3 (target)8.00+2.36+2.72+2.88+2.89
HPSv2.10.286+0.021+0.024+0.024+0.024
MUSIQ73.0+1.58+1.64+1.48+1.23
CLIP-IQA0.698+0.039+0.033+0.022+0.011
VQAScore0.886+0.015+0.010+0.015+0.015
Q-Align quality4.762+0.088+0.096+0.103+0.108
DeQA4.414+0.052+0.056+0.050+0.042
ReFL +HPS (known hacking case)Basestep 1000step 2000step 3000step 4000
HPSv38.00+1.64+1.91+1.43+1.56
HPSv2.10.286+0.051+0.066+0.074+0.076
MUSIQ73.0−7.18−6.47−4.74−6.12
CLIP-IQA0.698−0.261−0.238−0.195−0.203
VQAScore0.886−0.036−0.026−0.016−0.039
Q-Align quality4.762−0.151−0.054+0.029+0.031
DeQA4.414−0.229−0.182−0.082−0.091

HPSv2.1 (like Fig. 9's HPSv2) rises, then stays flat. RG-OPD improves overall quality, not just the target.

Q2: the 33× claim. We replace 33× with a lower bound and report the register cost on its own line (Table R5, our response to Reviewer TFAL, Q2). The speedup is at least 12.6× (SD3) / 7.2× (FLUX), and our total cost is below Flow-GRPO alone even with the register included.

  • 9.77 vs. 10.26: the same final Flow-GRPO checkpoint (77 GPU-h), scored on Fig. 6's 256-prompt training-curve set and on Table 2's full set. Speedups compare both methods on the 256-prompt set.
  • Flow-GRPO still improving: it had not reached RG-OPD's 10.72 (at 6.1 GPU-h) when stopped at 77 GPU-h, so further training can only raise the bound.

Q3a: is the gain from the per-step signal or the optimizer? We ran the suggested check: Flow-GRPO with the register reward as one terminal reward or as the per-step group-normalized advantage, with RG-OPD's model, prompts and LoRA, up to Flow-GRPO's Table 2 budget (Table R10). Mostly the optimizer: the per-step reward helps GRPO (terminal GRPO reward-hacks: its register reward rises while HPSv3 falls below base), but even at 12.6× RG-OPD's compute it stays 2.34 HPSv3 below RG-OPD. The gain comes from using the register's gradient as a local regression target, not as a scalar policy-gradient reward.

Table R10. Same register reward: GRPO (terminal / per-step) vs. RG-OPD.

Same register reward, SD3 (Table 2 protocol)Training GPU-hUpdatesHPSv3vs. RG-OPD [95% CI]
Base0—8.55
GRPO, terminal reward6.8128.20
GRPO, per-step reward6.5108.51−3.06 [−3.19, −2.93]
GRPO, per-step reward25408.84
GRPO, per-step reward50809.04
GRPO, per-step reward771249.23−2.34 [−2.47, −2.22]
RG-OPD (ours)6.111.57—

Q3b: other step-wise methods (SPO, TempFlow-GRPO). We ran SPO (official settings, with the register as its step-aware preference model, since none is released for SD3) and TempFlow-GRPO (official recipe, HPSv2.1 reward) to 77 GPU-h (Table R11). Neither reaches RG-OPD at any budget. Both peak at about 25 GPU-h (4× our compute), still 0.89–1.27 HPSv3 below RG-OPD, then decline, and SPO collapses.

Table R11. Other step-wise methods.

Method (SD3, Table 2 protocol)Training GPU-hHPSv3vs. RG-OPD [95% CI]
SPO (official lr, register as step-aware model)6.29.04−2.53 [−2.65, −2.41]
SPO, best checkpoint2510.30−1.27 [−1.38, −1.15]
SPO775.84−5.73 [−6.10, −5.37]
TempFlow-GRPO (official recipe, HPSv2.1)6.210.24−1.33 [−1.45, −1.21]
TempFlow-GRPO, best checkpoint2510.68−0.89 [−1.02, −0.76]
TempFlow-GRPO7710.12−1.45 [−1.59, −1.30]
Flow-GRPO +HPS (Table 2)7710.26−1.31 [−1.42, −1.20]
RG-OPD (ours)6.111.57—
AuthorsOpenReview ↗

Response to Reviewer Gbs7

Q3c: DiNa-LRM in our pipeline; small Table 1 gain. We plugged the official DiNa-LRM into the identical RGS and RG-OPD pipelines (Table R12).

  1. Higher reward. The register beats DiNa-LRM on HPSv3 in both pipelines (+0.13 / +0.12), and is more accurate at high noise (Table 1, Fig. 3); DiNa is slightly better on Q-Align quality and DeQA.
  2. Lower cost. The register reads the generator's own forward pass, so its reward evaluation is 1.5× faster, with 41% less memory.
  3. The pipelines are also our contribution. RGS and RG-OPD work with either reward model and lift both above Flow-GRPO (11.57 / 11.45 vs. 10.26).

Table R12. Register vs. DiNa-LRM in the same RGS / RG-OPD pipelines.

SD3, register − DiNa (paired, 95% CI)RGSRG-OPD
HPSv3 (register / DiNa)10.70 / 10.5711.57 / 11.45
HPSv3 difference+0.13 [0.06, 0.20]+0.12 [0.03, 0.21]
ImageReward+0.03 [0.00, 0.05]+0.04 [0.02, 0.07]
MUSIQbetterbetter
Q-Align quality−0.03−0.03
DeQA−0.04−0.03
Reward eval / step, ms (register / DiNa)50 / 76.5
Peak memory (register / DiNa)17.6 / 29.9 GiB

Q4: does the register's ranking on real trajectories agree with the final reward? We scored 800 images (100 prompts × 8 seeds, all from one model) along real CFG, RG-OPD and RGS trajectories, and compared the register's ranking with the final HPSv3 ranking, using forward-noised versions of the same images as reference (Table R13).

  1. No train/test shift. On every sampler, on-trajectory and forward-noised agreement match.
  2. Consistent on CFG and RG-OPD samples, which do not use the register at sampling time.
  3. Lower on RGS because RGS optimizes the register itself per image during sampling, so the register is no longer an independent judge of those images. The true reward still improves: HPSv3 rises for 96% of RGS images, and raters prefer RGS (Table R4).

Table R13. Spearman ρ between the register (preference head, t = 0.5) and final HPSv3, over 800 images.

SamplerOn sampling trajectoryForward-noised (same images)
CFG0.470.46
RG-OPD0.420.42
RGS (register is the guidance signal)0.160.19

Q5: ethics box. Answered by Q4 (on-trajectory agreement) and Q3a (per-step Flow-GRPO).

Dear Reviewer Gbs7,

Thank you again for your constructive feedback, which motivated several additional experiments. Following your suggestions, we added per-step GRPO and DiNa-LRM comparisons, examined reward rankings on actual denoising trajectories, and supplemented our evaluation and cost analysis. With the discussion period nearing its end, we would be grateful for your thoughts on these results and any remaining concerns.

Best regards,
The Authors

Dear Reviewer,

As the discussion period is coming to a close, we would appreciate any feedback on our responses above, particularly whether any concerns remain. We would be happy to provide further clarification if needed.

Thank you for your time and consideration.

Reviewer_TFALOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Reward Registers, an auxiliary read-only register stream attached to a frozen diffusion transformer. The registers predict preference proxies from intermediate noisy latents and provide differentiable latent-space reward gradients. These gradients support: (1) Reward-Gradient On-Policy Distillation (RG-OPD), which distills locally reward-shifted reference transitions into a trainable generator; and (2) Reward-Guided Sampling (RGS), which applies the gradients at inference without updating the generator. Experiments on SD3-Medium and FLUX.1-dev report high-noise pairwise reward-prediction accuracy, improved target reward metrics for both training-time and training-free alignment, and claimed GPU-hour savings versus online RL.

Critical strengths
  1. The method is well motivated and technically clean. The auxiliary register stream preserves the frozen DiT’s native hidden states and velocity field while creating a differentiable interface to intermediate latents. This makes one learned component usable for both parameter updates (RG-OPD) and test-time steering (RGS), which is a useful unification.

  2. The empirical study is broad for this setting. It covers two backbones, high-noise latent reward prediction, training-time alignment, inference-time guidance, register ablations, several solvers, and cost measurements. The ablation showing a substantial accuracy reduction without persistent registers supports the central architectural choice.

Critical weaknesses
  1. The evidence for improved human-preference alignment is incomplete. HPSv3 and ImageReward are both optimization targets and primary reported outcome metrics, so the improvements may partly reflect optimization of the chosen learned proxies. MUSIQ and CLIP-IQA are useful but are also automated no-reference metrics. The held-out pairwise tests validate the register predictor, but they do not establish that the final aligned generators are preferred by humans. A blinded human study, or at least a robust evaluation with held-out evaluators not used for optimization, is important for the paper’s central claim.

  2. The “up to 33x” training-efficiency claim is not yet demonstrably end-to-end. RG-OPD depends on a separately trained reward-register model, but the reported GPU-hour comparison does not appear to include its training cost, target-specific reward-head preparation, or tuning/selection cost. In addition, Tables 2–3 provide no uncertainty estimates or multi-run variability. These omissions make it hard to judge whether the reported superiority and efficiency gap would persist under a complete matched budget.

  3. Novelty is not sufficiently isolated empirically. The core ingredients, latent reward prediction from noisy diffusion states, learned readouts over frozen diffusion features, reward-gradient guidance, and on-policy distillation, are all closely related to existing latent-reward and diffusion-alignment work. The paper’s distinctive element is the persistent, non-intrusive register side stream, but the experiments do not compare it to a parameter- and compute-matched learned-query/pooling/probe readout over the same frozen DiT features. Consequently, it remains unclear whether the gains are specifically attributable to the proposed register mechanism, rather than added reward-model capacity or the overall training recipe. This weakens the claimed methodological novelty.

Questions for authors
  1. Can you provide a blinded human-preference evaluation of final samples, or an evaluation using genuinely held-out reward/preference evaluators that were not optimization targets? Please report how the conclusion changes relative to HPSv3/ImageReward.

  2. What are the GPU hours for training the reward registers, and what is the cumulative end-to-end cost of register training plus RG-OPD to reach each reported target level? Please compare this against the full cost of each baseline under matched hardware and selection/tuning budgets.

  3. Please report paired confidence intervals or multi-seed variability for Tables 2 and 3, especially for quality metrics and the smaller cross-method differences. Were baseline hyperparameters tuned comparably to the proposed method?

  4. Please clarify the precise conceptual distinction between Latent Reward Registers and prior latent reward/probe models that read intermediate diffusion features. Can the authors include a controlled comparison against parameter-matched alternatives, such as learned query tokens without persistent register states, attention pooling, or a lightweight probe using the same frozen DiT blocks and reward supervision? In particular, does the proposed register side stream retain a meaningful advantage under matched parameter count, latency, and training data?

5 comments
AuthorsOpenReview ↗

Response to Reviewer TFAL

We thank the reviewer for the careful review. We address each point below. Tables R1–R17 are numbered across our responses to all reviewers.

Q1: circular evaluation, no human study. We re-scored every method in Tables 2/3 with evaluators that no method trains on (Tables R1–R3), and ran a blinded human study (Table R4).

  1. Held-out evaluators. Ours is best on Q-Align quality and DeQA in all four settings and best or tied-best on VQAScore; baselines lead only on Q-Align aesthetic in training-time alignment.
  2. Human study. Raters prefer ours in all eight comparisons (66–97%; seven with CIs above 50%).

Table R1. Held-out evaluators, SD3 training-time alignment (Table 2 protocol).

MethodVQAScoreQ-Align qualityQ-Align aestheticDeQA
Base + CFG0.9114.7663.4624.408
ReFL +HPS0.8894.8714.4364.376
AlignProp +HPS0.8974.9134.5294.438
Flow-GRPO +HPS0.9124.8453.8574.475
Diffusion-NFT +HPS0.8834.8853.8764.499
RG-OPD +HPS0.9084.9003.8664.489
RG-OPD +MH0.9124.9203.9954.501

Table R2. Held-out evaluators, FLUX training-time alignment (Table 2 protocol).

MethodVQAScoreQ-Align qualityQ-Align aestheticDeQA
Base + CFG0.8784.8234.0214.398
ReFL +HPS0.8494.3143.5913.932
AlignProp +HPS0.8544.5083.7054.009
Flow-GRPO +HPS0.8614.9434.5414.467
Diffusion-NFT +HPS0.8374.8964.4804.398
RG-OPD +HPS0.8804.9684.3034.558
RG-OPD +MH0.8884.9714.3934.560

Table R3. Held-out evaluators, inference-time guidance (Table 3 protocol).

SD3VQAScoreQ-Align qualityQ-Align aestheticDeQA
CFG0.9114.7663.4624.408
DNO +HPS0.9134.7703.5754.411
DEMON +HPS0.9094.7273.5634.377
RGS +HPS0.9144.8943.8184.490
FLUXVQAScoreQ-Align qualityQ-Align aestheticDeQA
CFG0.8784.8234.0214.398
DNO +HPS0.8674.7704.0034.327
DEMON +HPS0.8624.6213.7704.209
RGS +HPS0.8894.9234.2064.492

Table R4. Blinded human study (10 raters; 80 image pairs, 10 per comparison, each rated by 8–10 raters; 720 votes). Win rate of ours, ties excluded [95% bootstrap CI over pairs].

ComparisonSD3FLUX
RGS vs. CFG79.8% [71, 88]75.0% [61, 87]
RGS vs. DEMON82.1% [67, 95]79.3% [60, 95]
RG-OPD vs. AlignProp96.5% [93, 100]81.0% [69, 92]
RG-OPD vs. Flow-GRPO66.7% [56, 79]65.9% [48, 81]

Q2: end-to-end cost and the 33× claim. We replace 33× with a lower bound on RL training cost, and report the register's training cost on its own line (Table R5). RG-OPD needs at least 12.6× (SD3) / 7.2× (FLUX) less RL training than Flow-GRPO, and is still cheaper with register training included.

Table R5. Training cost and speedup lower bound (HPSv3 on the 256-prompt training-curve set of Fig. 6).

SD3FLUX
Register training (once per backbone)37.8 GPU-h98.6 GPU-h
Flow-GRPO endpoint (HPSv3 @ GPU-h)9.77 @ 7711.43 @ 148
RG-OPD paper checkpoint (HPSv3 @ GPU-h)10.72 @ 6.111.44 @ 20.5
RL training speedup to reach our level (lower bound)≥ 12.6×≥ 7.2×
Register + RG-OPD vs. Flow-GRPO alone43.9 vs. 77 GPU-h119 vs. 148 GPU-h

Flow-GRPO had not reached our level when stopped, so further training can only raise the bound. The register is trained once per backbone and reused by every RG-OPD run and by RGS.

Q3: confidence intervals, seeds, tuning. Tables 2/3 average seeds 42/43; we added seed 44 for every main row except DNO/DEMON, and paired bootstrap 95% CIs over prompts (Table R6). Every CI is above zero, and the sign is the same on every seed. All baselines use their published training recipes; full per-seed tables will be in the appendix.

Table R6. Paired HPSv3 differences: the paper's Tables 2/3 (seeds 42/43), the new seed 44, and all three seeds with 95% CIs over prompts.

ComparisonPaper Δ (seeds 42/43)Seed 443-seed Δ [95% CI]
SD3 RG-OPD vs. Flow-GRPO+1.31+1.29+1.31 [1.20, 1.42]
SD3 RG-OPD vs. AlignProp+0.37+0.34+0.36 [0.24, 0.49]
SD3 RG-OPD vs. ReFL+0.73+0.66+0.71 [0.56, 0.85]
SD3 RGS vs. CFG+2.15+1.89+2.07 [1.96, 2.17]
SD3 RGS vs. DEMON+1.61–[1.52, 1.71]*
FLUX RG-OPD vs. Flow-GRPO+0.32+0.36+0.33 [0.22, 0.45]
FLUX RG-OPD vs. ReFL+0.38+0.40+0.39 [0.28, 0.50]
FLUX RG-OPD vs. AlignProp+0.55+0.56+0.55 [0.43, 0.68]
FLUX RGS vs. CFG+1.14+1.19+1.16 [1.06, 1.25]
FLUX RGS vs. DNO / DEMON+1.39 / +1.47––

*Seeds 42/43 only.

AuthorsOpenReview ↗

Response to Reviewer TFAL

Q4: conceptual distinction, and registers vs. matched readouts. We compare the registers with Table 4's matched control (Table R7) and with two lightweight probes from Table 1 (Table R8).

  1. Distinction. Prior latent reward models read the DiT's features with a separate probe, or adapt the backbone with a LoRA. Reward registers are extra tokens processed by the first L blocks of the frozen DiT, through its own attention and timestep modulation. The image tokens do not attend to them, so the generator is unchanged and the reward comes from the generator's own forward pass.
  2. Matched control. Table 4's "w/o reward register" row keeps the same learned tokens, frozen blocks, data and training at 97% of the parameters, and removes only the persistent register stream. This costs 4.03 points.
  3. Lightweight probes. Table 1's two probes, trained on the same backbone and data, one 10× smaller and one 4.7× larger, are 3–4 points lower. This points to the register mechanism, not capacity or the training recipe, as the source of the gain.

Table R7. Matched comparison (Table 4 ablation).

VariantTrainable paramsImageRewardHPDv2HPDv3GenAI-BenchAvg
Reward registers (ours)147.6M63.1278.1069.1062.9568.32
w/o reward register142.9M56.8674.1265.4260.7664.29

Table R8. Registers vs. lightweight probes (Table 1 protocol).

Table 1 protocol, SD3-M, same dataTrainable paramsImageRewardHPDv2HPDv3GenAI-BenchAvgPeak memory (RGS)
Reward registers +HPS (ours)147.6M61.8478.4669.9661.4167.9217.6 GiB
Diffusion Probe15.3M58.6775.2968.6857.1464.9534.3 GiB
PAVRM691.4M56.7673.5268.1857.2063.9131.3 GiB

We will add a parameter column to Table 4.

Dear Reviewer TFAL,

Thank you again for your thoughtful feedback and constructive suggestions. Our responses provide held-out and human evaluations, costs including register training, uncertainty estimates, and matched readout comparisons. As the discussion period draws to a close, we would appreciate hearing whether these additions address your concerns and would be happy to clarify any remaining points.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript: Rebuttal Experiments Incorporated

Dear Area Chair and Reviewers,

We have uploaded the revised manuscript, incorporating all additional experiments discussed during the rebuttal, including held-out evaluations, a blinded human study, architectural comparisons, and training-cost analyses. The new content is highlighted in blue.

With the discussion period closing in two days, we would appreciate your feedback on whether these updates address your concerns. We remain available to clarify any outstanding questions before the deadline.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.