acceptodds
Under review as a conference paper at ICLR 2027

Personalizing Few-Step Diffusion Distillation via Stepwise Reward Augmentation

Abstract

Personalizing a step-distilled diffusion model requires learning a new subject or style while retaining few-step inference. Reference-conditioned on-policy self-distillation (OPSD) provides teacher supervision along the student’s sampling trajectory, but encourages reference fidelity and image quality only indirectly. We propose Stepwise Reward Augmentation (SRA), which combines this supervision with local targets constructed from normalized reward gradients at intermediate denoising states. Schedule-derived weights allocate reward supervision across states, while aggregate gradient-norm balancing controls its strength relative to distillation. Training uses detached rollout states without backpropagating through sampling transitions. A gradient interpretation and matched controls clarify the roles of these design choices and inform practical recommendations for allocating reward feedback across sampling states and balancing it with teacher supervision. Experiments on object and style personalization across two backbones demonstrate improved reward-related reference similarity and image quality over teacher-only distillation and uniform-weight reward baselines. Additional reward recipes, new personalization cases, and training-cost comparisons further support the method’s applicability and competitive efficiency.

Primary area:
generative models
Keywords:
Diffusion Models, Few-Step Diffusion Distillation, On-Policy Self-Distillation, Image Personalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XckWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a personalization method for few-step diffusion models to learn user-specified attributes. The authors argue that prior work, which relies on indirect supervision from a teacher model, lacks sufficient signals for image quality. To address this limitation, the authors introduce SRA, which incorporates rewards using DINOv3 and TOPIQ during personalization, applies them at each timestep, and adjusts their weights according to the sampling schedule. The method is evaluated on Z-Image-Turbo and FLUX.2-klein. Further ablation studies examine the effects of individual components of the proposed method.

Critical strengths

The paper is well-organized, and both the writing and figures are easy to follow. The experiments aim to examine the individual components of the proposed method in detail. The authors show that explicitly introducing rewards leads to better image quality metrics than learning generation quality indirectly through the teacher model in D-OPSD.

Critical weaknesses

The authors argue that the designed rewards, based on DINOv3 and TOPIQ, improve the personalization of few-step diffusion models. However, while the method improves image quality metrics, the text-image alignment score (CLIP-S) is lower than that of D-OPSD. Table 4 also reports only image metrics such as DINO and Quality-S. These results make it difficult to assess whether the proposed method improves key aspects of personalization, including prompt alignment.

The experimental setting differs from that of D-OPSD, the main baseline used in this paper. The original D-OPSD paper evaluates 30 subjects and 10 styles in its personalization experiments, while SRA uses 10 subjects and 10 styles in Table 1. This difference also makes the results more difficult to interpret. Since the proposed method extends D-OPSD by introducing additional rewards, the overall evaluation should also include the original D-OPSD setting to establish its improvements.

Questions for authors

The paper presents the proposed method in a logical structure, but further validation is needed to establish its effectiveness. The following questions address the weaknesses discussed above: Can the proposed rewards improve overall personalization performance, including prompt adherence? How does the proposed method perform under the same experimental settings as the other baselines? What are the overall trends across evaluation metrics in Tables 3–6, beyond the selected metrics reported?

5 comments
AuthorsOpenReview ↗

Response to Reviewer XckW. Part [1/3]

We sincerely thank the reviewer for the time and patience given to this review, and for the constructive questions on prompt alignment and the evaluation setting.


W1. The authors argue that the designed rewards, based on DINOv3 and TOPIQ, improve the personalization of few-step diffusion models. However, while the method improves image quality metrics, the text-image alignment score (CLIP-S) is lower than that of D-OPSD. Table 4 also reports only image metrics such as DINO and Quality-S. These results make it difficult to assess whether the proposed method improves key aspects of personalization, including prompt alignment.

Q1. Can the proposed rewards improve overall personalization performance, including prompt adherence?

Thank you for raising prompt alignment. Our reward targets reference fidelity and image quality and contains no prompt term, and its cost on CLIP-S grows with the reward strength κ. At κ = 0.1, SRA's CLIP-S ↑ is 0.3210 and 0.3347 against D-OPSD's 0.3227 and 0.3354 on the two base models, while SRA still leads D-OPSD [1] on all three reward-related metrics (General Response #5). The new human study also asks which image better matches the prompt. On this question, participants do not prefer D-OPSD on either base model. SRA's rate is 62.5% on Z-Image-Turbo and 58.1% on FLUX.2-klein-4B, and about half of the judgments are ties. They prefer SRA on reference fidelity, image quality and overall (General Response #3). As suggested, we have added CLIP-S and VLM-J to Tables 3 and 4. On the eight test cases SRA's CLIP-S ↑ is 0.3078 against D-OPSD's 0.3099.

AuthorsOpenReview ↗

Response to Reviewer XckW. Part [2/3]

Q3. What are the overall trends across evaluation metrics in Tables 3–6, beyond the selected metrics reported?

Thank you for this suggestion on the overall trends. We have scored every configuration of Tables 3 to 5 on eleven metrics, the paper's seven plus PickScore [2], ImageReward [3], CLIP-I [4] and SigLIP-I [5]. We report them in full in Appendix C.1, Appendix E.2 and Appendix E.4. On the validation and test cases of Tables 3 and 4, the metrics fall into two groups relative to D-OPSD: those that read the reference or image quality (DINO-D, DINOv3 [6], Quality-S, Aesthetic-S, VLM-J, CLIP-I and SigLIP-I) improve with SRA, and the prompt-based metrics (CLIP-S, PickScore and ImageReward) decline, more so as κ grows. LPIPS-D ↓ [7] is 0.7216 to 0.7253 along Table 3 and 0.7736 against 0.7662 for D-OPSD on the test cases. Table 6 shows the same pattern over three seeds for the seven paper metrics, and in Table 5 SRA scores highest on the metric each recipe rewards.

Over the thirty DreamBooth subjects, VLM-J ↑ does not rise on Z-Image-Turbo (1.446 against 1.575, General Response #4). The metrics that Tables 3 and 4 do not show, on Z-Image-Turbo, one training run per configuration:

ConfigurationVLM-J ↑CLIP-S ↑Aesthetic-S ↑PickScore ↑ImageReward ↑CLIP-I ↑SigLIP-I ↑
Table 3, D-OPSD (κ = 0)1.4530.32275.30322.361.0020.64350.6958
Table 3, SRA, κ = 0.11.5660.32105.61822.200.9480.65540.7070
Table 3, SRA, κ = 0.31.7620.31915.62821.940.7940.66450.7180
Table 3, SRA, κ = 0.61.9220.31365.55621.530.4570.67850.7300
Table 4, D-OPSD1.5700.30995.31522.291.0150.67670.7380
Table 4, SRA1.7810.30785.68821.820.7540.69670.7668
AuthorsOpenReview ↗

Response to Reviewer XckW. Part [3/3]

W2. The experimental setting differs from that of D-OPSD, the main baseline used in this paper. The original D-OPSD paper evaluates 30 subjects and 10 styles in its personalization experiments, while SRA uses 10 subjects and 10 styles in Table 1. This difference also makes the results more difficult to interpret. Since the proposed method extends D-OPSD by introducing additional rewards, the overall evaluation should also include the original D-OPSD setting to establish its improvements.

Q2. How does the proposed method perform under the same experimental settings as the other baselines?

Thank you for this suggestion on the original D-OPSD setting. Every method in our tables, D-OPSD included, is retrained by us under one protocol and scored with D-OPSD's four metrics (DINO-D, LPIPS-D, CLIP-S and VLM-J) plus ours, so the comparisons are like for like. D-OPSD's thirty subjects are the public DreamBooth [8] set. The submission used fourteen of them (the ten of Table 1 and four test subjects), and we have now trained all thirty on both base models. Over the thirty subjects SRA has the best mean on all three reward-related metrics on both base models, for example DINO-D ↓ 0.1198 against D-OPSD's 0.1699 on Z-Image-Turbo (General Response #4). We have added these results to Appendix E.2 (New test cases). D-OPSD's ten styles were not released as a set, and its Quality-S and Aesthetic-S come from an internal reward model that is not public, so for styles we use ten public UnlearnCanvas [9] styles.


We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.


References

[1] D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models.

[2] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.

[3] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.

[4] Data Filtering Networks.

[5] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.

[6] DINOv3.

[7] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.

[8] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.

[9] UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models.

Reviewer_hxaqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies personalization of step-distilled diffusion models. It augments Diffusion-OPSD with local targets constructed from normalized reward gradients at detached clean predictions. Euler coefficients weight the sampling states, while aggregate gradient-norm balancing controls reward strength relative to teacher supervision. The main reward combines DINOv3 reference similarity and TOPIQ image quality. Experiments cover object and style personalization on two backbones, with ablations, training-seed replication, additional cases, and a human preference study. The paper reports improved reward-related metrics and favorable human preferences over teacher-only distillation and uniform-weight reward baselines, alongside lower prompt-alignment scores.

Critical strengths
  1. The method addresses an interesting problem that how to personalize already step-distilled diffusion model. On Z-Image-Turbo, SRA improves all three reward-related metrics even against Euler-weighted ReFL, which controls for state weighting and aggregate reward strength.

  2. The ablations are extensive. They examine state coverage, weighting, local gradient normalization, prediction-norm scaling, and reward strength. The gradient interpretation explains the update clearly, and the paper explicitly reports metric overlap and unfavorable results.

  3. The evaluation includes two backbones, three training seeds for the main comparisons, and eight additional cases evaluated after fixing the method. The human study provides complementary evidence, with 68-86% case-weighted preference for SRA over the tested baselines.

Critical weaknesses
  1. Evidence against the stronger Euler-weighted ReFL baseline is limited. It is competitive on FLUX, but is not included in the multi-seed, new-case, human preference, or efficiency studies. Extending these comparisons would help establish SRA's practical advantage.

  2. The main metrics overlap with the training reward, while separate prompt-conditioned preference scorers favor the baselines. The positive human results are useful, but the single overall-preference question does not separate reference fidelity, prompt adherence, and image quality.

  3. The benefit of reward supervision at every state is not fully established. Final-state-only supervision performs better on some new-prompt metrics, while SRA better preserves reference rewrites. A comparison of human preference and training cost would clarify this trade-off.

Questions for authors
  1. How does this approach relate to PSO and other methods for fine-tuning step-distilled models? Please expand this discussion in Related Work and summarize the baseline adaptations in the main experimental setup.

  2. What causes the reward-gradient variation at identical inputs? Does it affect the comparison between SRA and Euler-weighted ReFL?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hxaq. Part [1/2]

We sincerely thank the reviewer for the time and patience given to this review, and for engaging with our ablations and appendix analyses in detail.


W1. Evidence against the stronger Euler-weighted ReFL baseline is limited. It is competitive on FLUX, but is not included in the multi-seed, new-case, human preference, or efficiency studies. Extending these comparisons would help establish SRA's practical advantage.

Thank you for this suggestion. Euler-weighted ReFL is part of our own study. We built it for the ablations of the paper by adding SRA's Euler weighting to ReFL [1], to give SRA the strongest comparison we could build, and it equals SRA without the normalized target. We have added it to all four studies. Over three seeds SRA has the better mean on the three reward-related metrics on both base models (DINO-D ↓ 0.1504 against 0.1570 on Z-Image-Turbo and 0.1372 against 0.1387 on FLUX.2-klein-4B, General Response #1), and on Z-Image-Turbo the lead holds on the eight test cases. Participants prefer SRA over it on all four questions on Z-Image-Turbo, at 57.8% to 67.2%. On FLUX.2-klein-4B the rates are 55.6% to 58.1%, and only the prompt question has its interval above 50 (General Response #3). On Z-Image-Turbo SRA reaches the 1,000-update means of Euler-weighted ReFL in Quality-S, DINO-D and DINOv3 [2] after 550, 650 and 750 of its own updates, at 2.12 against 2.05 GPU hours per 1,000 updates. Means on the eight Z-Image-Turbo test cases, one training run per configuration:

MethodDINO-D ↓DINOv3 ↑Quality-S ↑
Euler-weighted ReFL0.14080.74330.8202
SRA0.13130.74990.8386

We have added these results to Appendix E.1, Appendix E.2, Figure 3, Appendix G.1 and Appendix H.


W2. The main metrics overlap with the training reward, while separate prompt-conditioned preference scorers favor the baselines. The positive human results are useful, but the single overall-preference question does not separate reference fidelity, prompt adherence, and image quality.

Thank you for pointing out the metric overlap and the single-question human study. Two reference-similarity metrics that share no backbone with the reward, CLIP-I [3] and SigLIP-I [4], confirm SRA's fidelity gain over D-OPSD [5] on both base models, for example CLIP-I ↑ 0.6645 against 0.6435 on Z-Image-Turbo (General Response #2). PickScore [6] and ImageReward [1] score an image against its prompt without access to the reference, their drop grows with the reward strength κ, and it depends on the reward recipe: with PickScore in place of TOPIQ [7] in the reward, SRA's PickScore ↑ rises to 23.20, above the 22.98 of the unadapted base model (General Response #5). As suggested, the new human study asks four separate questions, and participants prefer SRA over D-OPSD on reference fidelity, image quality and overall on both base models, at 67.2% to 70.6%, and do not prefer D-OPSD on prompt match (General Response #3).


W3. The benefit of reward supervision at every state is not fully established. Final-state-only supervision performs better on some new-prompt metrics, while SRA better preserves reference rewrites. A comparison of human preference and training cost would clarify this trade-off.

Thank you for suggesting a comparison of human preference and training cost. We have now run both comparisons. At equal κ the trade-off is as the reviewer describes. Final-state-only supervision differs from SRA only in its state weights. In our implementation it evaluates the reward at the same four states, so it has the same compute per update. In the new human study, participants prefer SRA over it on all four questions on Z-Image-Turbo, at 57.5% to 66.9% (General Response #3). At the same PickScore ↑ (21.79 against 21.80), SRA at κ = 0.4 is ahead of final-state-only on DINO-D, LPIPS-D [8], DINOv3 and Quality-S. Means over the twenty Z-Image-Turbo validation cases, one training run per configuration:

MethodPickScore ↑ImageReward ↑DINO-D ↓LPIPS-D ↓DINOv3 ↑Quality-S ↑
Final-state-only, κ = 0.321.800.7190.15540.72960.73360.8479
SRA, κ = 0.421.790.6770.14180.72250.73810.8576

We have added both comparisons to Appendix E.4 (Reward-state coverage) and Appendix G.1.

AuthorsOpenReview ↗

Response to Reviewer hxaq. Part [2/2]

Q1. How does this approach relate to PSO and other methods for fine-tuning step-distilled models? Please expand this discussion in Related Work and summarize the baseline adaptations in the main experimental setup.

Thank you for this suggestion on PSO and the baseline adaptations. We have made both revisions, in Section 2.1 (Related work) and Section 4 (Experimental setup). PSO [9], Uni-DAD [10] and OPAD [11] change the objective used to fine-tune a step-distilled model. PSO uses a pairwise preference for noised reference images over the student's own samples, and Uni-DAD and OPAD adapt during distillation from a many-step teacher. D-OPSD and SRA instead change the source of supervision. D-OPSD uses a reference-conditioned teacher at the states the student visits, and SRA adds an explicit reward target at the same states. The two choices are therefore orthogonal, and PSO is a baseline in Table 1 on both base models. We have summarized the adaptations in Section 4 (Experimental setup): ReFL maximizes the reward at every detached state, DRaFT [12] backpropagates the final-image reward through the whole chain, both keep D-OPSD's teacher and the same κ balancing, and PSO and DreamBooth [13] keep their prior-preservation terms with class-noun prompts (Appendix E).


Q2. What causes the reward-gradient variation at identical inputs? Does it affect the comparison between SRA and Euler-weighted ReFL?

Thank you for this question on the reward-gradient variation. We traced it with a new determinism probe. We have added the probe to Appendix E.4 (Fixed-checkpoint gradient probes). The source is the DINOv3 reward's attention backward pass, a non-deterministic GPU kernel that changes the image gradient by about 1e-4 in relative terms, and the bf16 backward pass through the Z-Image-Turbo decoder amplifies this. At the final state the DINOv3 and TOPIQ gradients nearly cancel after this backward pass, so the small change becomes large in their residual. On FLUX.2-klein-4B the reward gradient is reproducible to a cosine of 1.000 at every state. The variation is a property of reward evaluation on Z-Image-Turbo and enters every reward-gradient method through the same path. It does not explain SRA's lead over Euler-weighted ReFL. The lead holds over three seeds on Z-Image-Turbo, and SRA still has the better three-seed mean on all three reward-related metrics on FLUX.2-klein-4B, where the reward gradient is reproducible (General Response #1).


We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.


References

[1] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.

[2] DINOv3.

[3] Data Filtering Networks.

[4] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.

[5] D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models.

[6] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.

[7] TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment.

[8] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.

[9] Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization.

[10] Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation.

[11] Adversarial Concept Distillation for One-Step Diffusion Personalization.

[12] Directly Fine-Tuning Diffusion Models on Differentiable Rewards.

[13] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.

AuthorsOpenReview ↗

Kind reminder (follow-up to Reviewer hxaq)

Dear Reviewer hxaq,

Thank you again for your careful review and for engaging with our ablations and appendix analyses in detail. During the rebuttal we ran additional experiments and revised the paper to address your concerns. The experiments and revisions are summarized below.

Additional experiments

  • Euler-weighted ReFL in all four studies (W1). We added it to the multi-seed study on both base models, to the eight test cases, to the human study, and to the efficiency comparison.
  • A four-question human study (W2, W3). Four participants each rated 200 pairs on reference fidelity, prompt match, image quality and overall preference, against D-OPSD, Euler-weighted ReFL and final-state-only supervision.
  • Reference similarity outside the reward family (W2). We added CLIP-I and SigLIP-I, two metrics that share no backbone with the reward.
  • Final-state-only supervision at matched preference score (W3). We trained SRA at κ = 0.4, which matches the PickScore of final-state-only supervision, and compared the two on fidelity and quality.
  • A determinism probe for the reward gradient (Q2). It traces the variation at identical inputs to the DINOv3 attention backward pass and the Z-Image-Turbo decoder.
  • κ sweeps of SRA and Euler-weighted ReFL on both base models. Each method now has five κ values on Z-Image-Turbo and on FLUX.2-klein-4B.

Revisions to the paper (all marked in blue)

  • Section 2.1 (Related work) now discusses PSO, Uni-DAD and OPAD, and Section 4 (Experimental setup) summarizes the baseline adaptations (Q1).
  • Table 6 reports all seven metrics over three seeds, and Figure 3 adds the Euler-weighted ReFL and DRaFT curves.
  • Appendix E.1 reports Euler-weighted ReFL and DRaFT over three seeds. Appendix E.2 reports them on the test cases and adds all thirty DreamBooth subjects.
  • Appendix E.4 adds the reward-state coverage comparison, the κ sweeps on both base models, and the determinism probe.
  • Appendix E.5 is new and reports CLIP-I and SigLIP-I.
  • Appendix G.1 reports the four-question human study, and Appendix H reports the efficiency comparison with Euler-weighted ReFL.

We would be glad to know whether our responses have addressed your concerns, and we would be grateful if you would take them into account in your final evaluation. We remain available for any further questions.

Best regards,

Authors of Submission 10940

Reviewer_99jvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Stepwise Reward Augmentation (SRA), a reward-guided on-policy self-distillation method for personalizing few-step diffusion models. Building upon D-OPSD, which trains a text-conditioned student using predictions from a reference-conditioned EMA teacher, SRA introduces explicit reward supervision at intermediate denoising states to improve reference fidelity and image quality.

Specifically, SRA constructs local regression targets by taking bounded steps along normalized reward gradients of the student's clean-latent predictions. The reward supervision is weighted across denoising steps using Euler sampling coefficients, while gradient-norm balancing controls its relative contribution to teacher distillation. The method operates on detached rollout states, avoiding backpropagation through the full sampling trajectory.

Experiments on Z-Image-Turbo and FLUX.2-klein-4B, covering object and style personalization, show improvements in reward-related reference similarity and image-quality metrics over D-OPSD and reward-augmented baselines, including ReFL and DRaFT. The authors further provide ablation studies on reward target construction, timestep weighting, and gradient balancing, along with additional reward settings, human preference evaluations, and computational cost comparisons. The results also reveal trade-offs on certain prompt-alignment and reference-free preference metrics.

Critical strengths
  • Robust and consistent empirical gains. SRA gives the best reward-related metrics on both backbones, holds its lead across five reward recipes, has very small variance over three seeds (std ≤ 0.004, smallest lead 0.011), and still leads on eight held-out cases evaluated after hyperparameters were fixed.
  • Simple, practical, and efficient method. The motivation is clear, D-OPSD optimizes reference fidelity only indirectly. The fix is lightweight: one reward-gradient step per visited state, with no backpropagation through sampling and no change to few-step inference. Per-update cost is close to ReFL and well below DRaFT.
  • Thorough ablations that give actionable guidance. Table 2 and Table 3 systematically vary target construction, state weighting, teacher supervision, and reward strength.
Critical weaknesses
  • The main improvements are measured largely by the training reward itself, and independent evidence is mixed.

    • LPIPS-D is essentially unchanged relative to D-OPSD.
    • CLIP-S is consistently lower. On FLUX.2-klein-4B it is lower than all three baselines at every seed.
    • PickScore and ImageReward, evaluated without reference images, favor D-OPSD and the uniform-weight baselines (Appendix F).
    • The human study has only seven participants and asks a single overall question. It cannot tell whether the preference comes from reference fidelity, prompt adherence, or image quality.
  • The gains over simple baselines are small, and the main table uses weaker baselines. Main-table ReFL and DRaFT use uniform state weights. Adding Euler weighting closes much of the gap

    • DINO-D: ReFL uniform 0.1825, ReFL with Euler weights 0.1591, SRA 0.1503.
    • Euler-weighted DRaFT nearly matches SRA on DINOv3 and Quality-S.
    • Ad-hoc last-heavy weights perform comparably to Euler weights.
    • The stronger baselines appear only in single-seed ablations on one backbone.
Questions for authors
  1. Can you report reference similarity with scorers outside the training reward's families, e.g., CLIP-I or other non-DINO features? This would help separate genuine fidelity gains from reward over-optimization.

  2. Can you report Euler-weighted ReFL and DRaFT on FLUX.2-klein-4B with multiple seeds? Does SRA's lead over them remain significant?

  3. Since aggregate norm balancing cancels δ\delta (Appendix B), what does the per-state rescaling contribute beyond retuning κ\kappa or the learning rate? Is there evidence that it matters beyond a better-tuned ReFL?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 99jv. Part [1/2]

We sincerely thank the reviewer for the time and patience given to this review, including the close reading of our tables and appendices.


W1. The main improvements are measured largely by the training reward itself, and independent evidence is mixed.

  • LPIPS-D is essentially unchanged relative to D-OPSD.
  • CLIP-S is consistently lower. On FLUX.2-klein-4B it is lower than all three baselines at every seed.
  • PickScore and ImageReward, evaluated without reference images, favor D-OPSD and the uniform-weight baselines (Appendix F).

Thank you for pointing out the dependence on the training reward and the need for independent evidence. We agree with all three observations. LPIPS-D [1] is a spatially aligned perceptual distance between each reference image and an image generated to reproduce it, computed on VGG features, so it is sensitive to layout and color. Our reward (DINOv3 [2] similarity and TOPIQ [3] quality) does not target these. SFT and DreamBooth [4] lower LPIPS-D ↓ to 0.6772 and 0.6947 against 0.7216 for D-OPSD [5] on Z-Image-Turbo, but they fit the reference images directly and lose image quality (Quality-S ↑ 0.3328 and 0.3746 against 0.6408). CLIP-S, PickScore [6] and ImageReward [7] score an image against its prompt without access to the reference, and their drop grows with the reward strength κ. At κ = 0.1, SRA's CLIP-S ↑ is 0.3210 and 0.3347 against D-OPSD's 0.3227 and 0.3354 on the two base models, and SRA still leads D-OPSD on all three reward-related metrics (General Response #5). The new human study also asks which image better matches the prompt. On this question, participants do not prefer D-OPSD on either base model. SRA's rate is 62.5% on Z-Image-Turbo and 58.1% on FLUX.2-klein-4B, and about half of the judgments are ties (General Response #3).


W1.4. The human study has only seven participants and asks a single overall question. It cannot tell whether the preference comes from reference fidelity, prompt adherence, or image quality.

Thank you for pointing out the limits of a single overall question. We have adopted this suggestion. The new study has four participants instead of seven because of the time available during the discussion, but each participant rated all 200 pairs, and the study asks four questions per pair instead of one: which image is closer to the reference, which better matches the prompt, which has better image quality, and which is preferred overall. Against D-OPSD, participants prefer SRA on reference fidelity, image quality and overall on both base models, at 67.2% to 70.6%, and do not prefer D-OPSD on prompt match (General Response #3, figure).


Q1. Can you report reference similarity with scorers outside the training reward's families, e.g., CLIP-I or other non-DINO features? This would help separate genuine fidelity gains from reward over-optimization.

Thank you for suggesting scorers outside the reward families. We have added two reference-similarity metrics that share no backbone with the reward, CLIP-I [8] and SigLIP-I [9]. Among D-OPSD, the uniform-weight ReFL [7] and DRaFT [10], and SRA, SRA has the highest similarity under both metrics on both base models, for example CLIP-I ↑ 0.6645 against D-OPSD's 0.6435 on Z-Image-Turbo (General Response #2).

AuthorsOpenReview ↗

Response to Reviewer 99jv. Part [2/2]

W2. The gains over simple baselines are small, and the main table uses weaker baselines. Main-table ReFL and DRaFT use uniform state weights. Adding Euler weighting closes much of the gap

  • DINO-D: ReFL uniform 0.1825, ReFL with Euler weights 0.1591, SRA 0.1503.
  • Euler-weighted DRaFT nearly matches SRA on DINOv3 and Quality-S.
  • Ad-hoc last-heavy weights perform comparably to Euler weights.
  • The stronger baselines appear only in single-seed ablations on one backbone.

Thank you for raising the role of Euler weighting. Euler weighting is the state allocation of SRA. It is the second contribution of the paper. Euler-weighted ReFL and DRaFT are part of our own study. We built them for the ablations of the paper by adding this allocation to ReFL and DRaFT, to give SRA the strongest comparison we could build rather than weak baselines. The last-heavy weights are our own variant of this allocation. Euler-weighted ReFL equals SRA without the normalized target, so its gap to SRA isolates the normalized target. Against uniform-weight ReFL and DRaFT, SRA's gain over D-OPSD is 1.3 to 2.1 times theirs over three seeds on both base models (Table 6). Multi-seed results for both Euler-weighted baselines on both base models are in General Response #1.


Q2. Can you report Euler-weighted ReFL and DRaFT on FLUX.2-klein-4B with multiple seeds? Does SRA's lead over them remain significant?

Thank you for this suggestion. Yes, we trained both Euler-weighted baselines under two more seeds on both base models. SRA has the best three-seed mean on every reward-related metric on both base models, and ten of its twelve paired differences from the two baselines are statistically significant, all except DINOv3 against Euler-weighted ReFL on FLUX.2-klein-4B and against Euler-weighted DRaFT on Z-Image-Turbo (General Response #1).


Q3. Since aggregate norm balancing cancels δ (Appendix B), what does the per-state rescaling contribute beyond retuning κ or the learning rate? Is there evidence that it matters beyond a better-tuned ReFL?

Thank you for this question on the per-state rescaling. The per-state rescaling is the normalized target of SRA. The cancellation covers positive scalars shared by all four states, such as δ. κ and the learning rate likewise act on all states alike, while SRA's per-state factor differs across states, so it changes the direction of the reward update. Retuning κ or the learning rate cannot reproduce this. With the Euler weights and κ fixed, adding this rescaling improves all three reward-related metrics on Z-Image-Turbo (Appendix E). Euler-weighted ReFL lacks only this rescaling, and over three seeds SRA leads it on both base models (General Response #1). Euler-weighted ReFL retuned to κ = 0.45 reaches SRA's reward level but scores lower on the three prompt-based metrics. Means over the twenty Z-Image-Turbo validation cases, one training run per configuration:

MethodDINO-D ↓DINOv3 ↑Quality-S ↑CLIP-S ↑PickScore ↑ImageReward ↑
Euler-weighted ReFL, κ = 0.450.14710.73170.83310.315121.830.668
SRA, κ = 0.30.15030.72830.83820.319121.940.794

We have stated which scalars the balancing cancels in Appendix B (Euler weights and common scales) and added the κ = 0.45 row to Appendix E.4 (Teacher supervision and aggregate strength).


We thank you again for your valuable comments. If you have any further questions, please let us know, and we will do our best to respond. We hope our responses address your concerns, and we would be grateful if you would take them into account in your final evaluation.


References

[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.

[2] DINOv3.

[3] TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment.

[4] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.

[5] D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models.

[6] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.

[7] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.

[8] Data Filtering Networks.

[9] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.

[10] Directly Fine-Tuning Diffusion Models on Differentiable Rewards.

Thank authors for the detailed rebuttal and the substantial additional experiments.

A few points remain open for me:

  • I agree that Euler weighting is part of your contribution. However, the remaining gain attributable to the normalized target is very small on FLUX.2-klein-4B (DINO-D 0.1372 vs 0.1387, DINOv3 0.7471 vs 0.7467, Quality-S 0.7791 vs 0.7768), and the human preference there is not significant on the overall question (58.1 [50.0, 66.2]). The significance tests bootstrap over cases. Could you also report the standard deviation across the three seeds, so the differences can be compared with seed-to-seed variation?
  • In the $ \kappa = 0.45$ comparison, Euler-weighted ReFL is slightly better on DINO-D and DINOv3, and SRA’s advantage is on the prompt-based metrics. The claim therefore becomes that SRA gives a better fidelity–prompt trade-off at matched reward level. This is a reasonable claim, but a single κ\kappa point from a single run on one backbone is thin evidence for it. A κ\kappa sweep for both methods (ideally also on FLUX.2-klein-4B) showing the two trade-off curves would be much more convincing.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response. Part [3/3]

#5 Prompt-based metrics and the reward strength κ

We thank the reviewers for raising the prompt-based metrics. CLIP-S, PickScore and ImageReward score an image against its prompt without access to the reference. Along the reward strength κ on Z-Image-Turbo (one training run per configuration, κ = 0 is D-OPSD, and κ = 0.3 is SRA's setting):

κDINO-D ↓DINOv3 ↑Quality-S ↑CLIP-S ↑PickScore ↑ImageReward ↑
00.22780.67920.64080.322722.361.002
0.10.17830.70580.77380.321022.200.948
0.30.15030.72830.83820.319121.940.794
0.60.13510.74550.87190.313621.530.457

The prompt-based metrics fall as κ grows, and at κ = 0.1 SRA still improves all three reward-related metrics over D-OPSD, with FLUX.2-klein-4B following the same pattern (Appendix E). On Z-Image-Turbo, PickScore rates the reference images below the images of the unadapted base model for the same training prompts (Appendix F), and with PickScore in place of TOPIQ in the reward, SRA's PickScore ↑ rises to 23.20, above the 22.98 of the unadapted base model (Table 5, Appendix F). The new human study also asks which image better matches the prompt, and on this question participants do not prefer D-OPSD on either base model (#3). We have added the prompt-based metrics along κ to Appendix E.4 (Teacher supervision and aggregate strength) and Appendix F.


We have revised the paper accordingly, with all changes marked in blue.

AuthorsOpenReview ↗

General Response. Part [2/3]

#3 A new human study with four questions

We thank the reviewers for suggesting a human study that separates reference fidelity, prompt adherence and image quality. We ran a new human study. Four participants outside this work each rated all 200 pairs, 40 per comparison and two per case over the twenty validation cases. Each pair showed the reference image, the prompt and two images generated with the same case, prompt and seed, with the methods unnamed and the sides balanced, and the participants answered four questions in order: which image is closer to the reference, which better matches the prompt, which has better image quality, and which they prefer overall. Each cell is SRA's case-weighted preference rate in percent (win 1, tie 0.5, loss 0), and 50 means no preference.

SRA against (160 judgments each)Closer to the referenceMatches the promptImage qualityOverall
D-OPSD, Z-Image-Turbo70.6 [57.5, 83.1]62.5 [55.6, 69.1]70.3 [56.9, 82.5]70.3 [56.9, 82.5]
D-OPSD, FLUX.2-klein-4B67.5 [53.1, 81.9]58.1 [51.9, 64.7]67.2 [53.4, 80.9]67.5 [53.1, 81.9]
Final-state-only, Z-Image-Turbo66.6 [55.6, 76.6]57.5 [51.6, 63.4]66.9 [55.9, 76.9]66.6 [55.6, 76.6]
Euler-weighted ReFL, Z-Image-Turbo66.6 [58.8, 74.1]57.8 [53.8, 62.5]65.0 [57.2, 72.8]67.2 [59.1, 75.0]
Euler-weighted ReFL, FLUX.2-klein-4B58.1 [50.0, 66.2]55.6 [50.6, 61.3]56.9 [49.7, 64.1]58.1 [50.0, 66.2]

Brackets are 95% case-bootstrap intervals. Participants prefer SRA over D-OPSD and over final-state-only supervision on reference fidelity, image quality and overall, and do not prefer the baseline on prompt match. Each of the four participants prefers SRA overall in all three of these comparisons (60.0% to 73.8%). Against Euler-weighted ReFL, the preference for SRA is 57.8% to 67.2% on Z-Image-Turbo and 55.6% to 58.1% on FLUX.2-klein-4B, where only the prompt question has its interval above 50. The study has four participants instead of seven because of the time available during the discussion, but each participant rated every pair on four questions instead of one, so each comparison has 160 judgments per question. We have replaced the earlier study with this one in Section 4 (Human preference) and detailed it in Appendix G.1.

The results are also shown in this figure.


#4 All thirty DreamBooth subjects

We thank the reviewers for suggesting the original D-OPSD setting. We trained all thirty DreamBooth subjects of D-OPSD's object evaluation on both base models, of which the submission used fourteen (the ten of Table 1 and four test subjects), with every method retrained under one protocol. Means over the thirty subjects, one training run per configuration:

MethodDINO-D ↓DINOv3 ↑Quality-S ↑Aesthetic-S ↑
Z-Image-Turbo, D-OPSD0.16990.76610.65075.107
Z-Image-Turbo, ReFL (uniform)0.14090.79830.77095.468
Z-Image-Turbo, DRaFT (uniform)0.14390.79190.77085.493
Z-Image-Turbo, SRA0.11980.81580.85725.602
FLUX.2-klein-4B, D-OPSD0.14490.82650.51705.010
FLUX.2-klein-4B, ReFL (uniform)0.12110.84330.76025.432
FLUX.2-klein-4B, DRaFT (uniform)0.12210.83800.74405.407
FLUX.2-klein-4B, SRA0.11100.84710.82035.476

SRA has the best mean on the three reward-related metrics and Aesthetic-S on both base models. D-OPSD also reports LPIPS-D ↓, CLIP-S ↑ and VLM-J ↑. On these three metrics, SRA scores 0.6899, 0.2957 and 1.446 against D-OPSD's 0.6861, 0.2973 and 1.575 on Z-Image-Turbo, and 0.6857, 0.3154 and 2.402 against 0.6885, 0.3208 and 2.323 on FLUX.2-klein-4B. We have added the thirty-subject results to Appendix E.2 (New test cases).

AuthorsOpenReview ↗

General Response. Part [1/3]

Dear reviewers, we sincerely thank you for your valuable feedback. Your insightful suggestions have greatly improved our paper. We appreciate the recognition of our strengths, particularly the simple and practical method [Reviewer 99jv], the benefit of explicit rewards over the indirect teacher supervision of D-OPSD [Reviewer XckW], the consistent empirical gains [Reviewer 99jv and hxaq], and the thorough ablations [Reviewer 99jv, hxaq and XckW].

Following your suggestions, we have conducted additional experiments and clarified the shared concerns below, and we respond to each of your points in the individual threads, in the hope of resolving your concerns.


SRA is a further optimization of D-OPSD for personalizing step-distilled models. The main shared concern is whether its gains go beyond the training reward, and the new results show that they do. CLIP-I and SigLIP-I are not part of the reward, and they show the same gain in reference fidelity over D-OPSD and the uniform-weight baselines (#2). In a new four-question human study, participants prefer SRA to D-OPSD on reference fidelity, image quality and overall (#3). The gains also hold against the strongest baselines we could build: over three seeds, SRA leads Euler-weighted ReFL and DRaFT on both base models (#1).


#1 Multi-seed results for the Euler-weighted baselines

We thank the reviewers for suggesting multi-seed results for these baselines. We trained Euler-weighted ReFL and Euler-weighted DRaFT under two more seeds on both base models. Both baselines are part of our own study. We built them for the ablations of the paper to give SRA the strongest comparison we could. Euler-weighted ReFL equals SRA without the normalized target, so its gap to SRA isolates the normalized target. Means over three seeds and the twenty validation cases:

MethodDINO-D ↓DINOv3 ↑Quality-S ↑
Z-Image-Turbo, Euler-weighted ReFL0.15700.72170.8145
Z-Image-Turbo, Euler-weighted DRaFT0.15700.72580.8288
Z-Image-Turbo, SRA0.15040.72840.8368
FLUX.2-klein-4B, Euler-weighted ReFL0.13870.74670.7768
FLUX.2-klein-4B, Euler-weighted DRaFT0.13930.74550.7715
FLUX.2-klein-4B, SRA0.13720.74710.7791

SRA has the best mean on all three metrics on both base models. Of its twelve paired differences from the two baselines, ten are statistically significant (95% case bootstrap), all except DINOv3 against Euler-weighted ReFL on FLUX.2-klein-4B and against Euler-weighted DRaFT on Z-Image-Turbo. We have added these results to Appendix E.1 (Training-seed replication).


#2 Reference similarity outside the reward family

We thank the reviewers for suggesting reference-similarity metrics outside the reward family. We added two reference-similarity metrics that share no backbone with the reward, CLIP-I and SigLIP-I. Means over the twenty validation cases, one training run per configuration:

MethodCLIP-I ↑, Z-Image-TurboSigLIP-I ↑, Z-Image-TurboCLIP-I ↑, FLUX.2-klein-4BSigLIP-I ↑, FLUX.2-klein-4B
D-OPSD0.64350.69580.67870.7230
ReFL (uniform)0.65550.71030.68020.7279
DRaFT (uniform)0.65550.71060.67860.7269
SRA0.66450.71800.68720.7369

SRA has the highest similarity of the four methods under both metrics on both base models. We have added both metrics in a new Appendix E.5.

AuthorsOpenReview ↗

Summary of the discussion

We sincerely thank all the reviewers, the Area Chair, the Senior Area Chairs and the Program Chairs for their valuable feedback and time. We also deeply value the thoughtful evaluations, which highlight the strengths of our work:

  • A relevant problem with a clear motivation: personalizing models that are already step-distilled, where D-OPSD optimizes reference fidelity only indirectly. [Reviewers 99jv and hxaq]
  • A simple, practical and efficient method: one reward-gradient step per visited state, with no backpropagation through sampling and no change to few-step inference. [Reviewer 99jv]
  • Robust and consistent empirical gains over D-OPSD on two base models, over three seeds and on additional cases. [Reviewers 99jv, hxaq and XckW]
  • Extensive ablations that examine each component of the method in detail. [Reviewers 99jv, hxaq and XckW]
  • A well-organized and easy-to-follow presentation that reports metric overlap and unfavorable results openly. [Reviewers hxaq and XckW]

A major concern shared by the reviewers is that the main metrics overlap with the training reward, while the prompt-based metrics (CLIP-S, PickScore and ImageReward) are lower. Another shared concern is that the human study asked only one question. The reviewers also asked for more evidence against the stronger Euler-weighted baselines, for the original D-OPSD evaluation setting, and for a clearer picture of what the per-state design contributes.

In our rebuttal, we ran additional experiments and gave clarifications to respond to these concerns, including:

  • Added two reference-similarity metrics outside the reward family, CLIP-I and SigLIP-I. [99jv, hxaq]
  • Ran a new human study with four separate questions on reference fidelity, prompt match, image quality and overall preference. [99jv, hxaq, XckW]
  • Added Euler-weighted ReFL and DRaFT to the three-seed study on both base models, to the test cases, to the human study and to the efficiency comparison. [99jv, hxaq]
  • Trained κ sweeps of SRA and Euler-weighted ReFL with five κ values per method on both base models, and reported the standard deviation across seeds. [99jv]
  • Reported the prompt-based metrics over the κ sweep and under other reward recipes, with eleven metrics per configuration. [99jv, hxaq, XckW]
  • Trained all thirty DreamBooth subjects of the D-OPSD setting on both base models. [XckW]
  • Compared SRA with final-state-only supervision at a matched preference score. [hxaq]
  • Traced the reward-gradient variation at identical inputs with a new determinism probe. [hxaq]
  • Clarified which scalars the norm balancing cancels and what the per-state rescaling adds. [99jv]
  • Extended the related work on PSO, Uni-DAD and OPAD, and summarized the baseline adaptations in the experimental setup. [hxaq]

These results support the conclusions of the paper. Metrics outside the reward family confirm the fidelity gain, and participants prefer SRA over D-OPSD on reference fidelity, image quality and overall on both base models. SRA has the best three-seed mean on all three reward-related metrics on both base models. Its margin over Euler-weighted ReFL is clear on Z-Image-Turbo and small on FLUX.2-klein-4B, and we report both openly.

During the discussion period, all three reviewers responded to our rebuttal, and we are grateful for their engagement. Reviewer hxaq wrote that the concerns "have been addressed well". Reviewer XckW thanked us for the additional experiments. Reviewer 99jv asked two follow-up questions, on the variation across seeds and on a κ sweep for both methods, and we answered both with new experiments.

We have revised the paper accordingly, with all changes marked in blue. Finally, we greatly appreciate the reviewers' recognition and their constructive suggestions, which have made the paper stronger. We hope our work can provide useful insights for the personalization of few-step diffusion models.

Best regards,

Authors of Submission 10940

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.