acceptodds
Under review as a conference paper at ICLR 2027

Gradient-based Timestep-Partitioned LoRA for Reward Fine-Tuning of Diffusion Models

Abstract

Reward-based fine-tuning enables diffusion models to optimize human preferences and task-specific objectives, but its optimization signals can be highly imbalanced across denoising timesteps. Prior work has identified this issue in policy-gradient-based alignment and addressed it through objective- or scheduler-level correction. We show that concentrated timestep-wise gradients also arise in direct-gradient optimization and that their temporal profile can differ from on-policy reward sensitivity. When a single LoRA adapter is shared across the entire denoising trajectory, concentrated gradients can bias its adaptation toward particular timestep regimes. We propose Gradient-based Timestep-Partitioned LoRA (GTP-LoRA), which partitions the trajectory into contiguous segments and assigns a separate LoRA expert to each segment. Segment boundaries are determined from a reference denoiser-output gradient profile such that each expert receives a comparable cumulative upstream gradient magnitude. This timestep-partitioned design confines gradient updates to their assigned denoising regimes, limiting the disproportionate influence of high-gradient timesteps on a single shared adapter. Experiments across multiple diffusion models, reward objectives, and optimization algorithms show that GTP-LoRA consistently improves reward performance while maintaining competitive sample diversity. The source code is included in the supplementary materials.

Primary area:
generative models
Keywords:
Diffusion, Reinforcement Learning, Generative Model

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ry2yOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines timestep-wise reward sensitivity and gradient magnitude in reward-based diffusion fine-tuning. The authors find that reward sensitivity is greater during early denoising steps, while optimization gradients are concentrated in later steps. Based on this finding, they propose GTP-LoRA, which groups denoising timesteps by gradient magnitude and assigns a separate LoRA expert to each group. The method improves reward performance while preserving competitive sample diversity.

Critical strengths
  1. Clear and well-written paper. The paper is easy to follow, with a clear motivation, analysis, and methodology.

  2. Insightful empirical analysis. The contrasting timestep-wise profiles of reward sensitivity and optimization gradients are noteworthy and are supported by empirical analysis.

  3. Simple and effective method. GTP-LoRA can be integrated into existing reward fine-tuning algorithms without modifying their objectives or sampling schedulers, while adding minimal inference overhead.

  4. Promising empirical results. The method consistently improves performance across multiple optimization algorithms while maintaining competitive sample diversity.

Critical weaknesses
  1. Limited evaluation across reward objectives. Although the paper evaluates multi-reward optimization on SDXL, all single-reward experiments optimize only the CLIP reward on SD1.4. It remains unclear whether the observed improvements generalize to other reward objectives and larger models in the single-reward setting.

  2. Limited analysis compared to existing timestep-wise expert baselines. While the paper provides a thorough evaluation of the benefits of applying GTP-LoRA to existing reward-alignment methods, its comparisons with prior timestep-wise expert approaches, such as TSM and TD-LoRA, remain limited and are presented only in the appendix. Given the methodological similarities, more extensive comparisons and a clearer explanation of GTP-LoRA’s novelty and advantages over existing approaches would strengthen the paper’s contribution.

Questions for authors
  1. Could the authors provide additional single-reward experiments using other objectives, such as Aesthetic, PickScore, or HPS? It would also be valuable to evaluate CLIP-only optimization on SDXL to determine whether the improvements generalize to larger models.

  2. Could the authors clarify the methodological differences between GTP-LoRA and existing timestep-wise expert methods, such as TSM and TD-LoRA, and provide a more detailed comparative analysis of these baselines?

3 comments

We thank the reviewer for the positive assessment and constructive suggestions. We address each concern below, with additional details provided in our shared Official Comment.

1. Limited Evaluation Across Reward Objectives

As detailed in 3. Generalization Beyond Animal Activity and CLIP Reward of our shared Official Comment, we conducted additional single-reward experiments using HPSv2.1 instead of CLIP on the HPS Benchmark. GTP-LoRA improves HPSv2.1 and both diversity metrics over DDPO. Notably, ImageReward, an independent human-preference evaluator not used during training, also improves. These results provide additional evidence that GTP-LoRA's effectiveness is not restricted to CLIP reward optimization.

For larger models, our original SDXL experiments already demonstrate improvements across all five reward components under 5-Mix optimization. Due to computational constraints, we may not be able to complete SDXL CLIP-only experiments within the rebuttal period. Instead, we are currently conducting single-reward fine-tuning experiments on FLUX.1-dev to further assess generalization to flow-matching architectures. These experiments are ongoing, and we will update our response as the results become available.

2. Limited Analysis Compared to Existing Timestep-Wise Expert Baselines

Unlike TSM and TD-LoRA, which employ learned expert routing, GTP-LoRA uses fixed expert boundaries determined by cumulative reward-optimization gradient magnitudes, requiring no additional router training. Although these methods share the idea of timestep-specific expert adaptation, GTP-LoRA is specifically motivated by the gradient concentration observed in reward fine-tuning across both policy-gradient and direct-gradient optimization (Sec. 4). Furthermore, our additional analysis (Fig. 12, Appendix K) reveals weak timestep-wise gradient affinity without a pronounced clustering structure, providing further motivation for using gradient magnitude rather than affinity as the partitioning criterion.

In terms of empirical performance, as shown in Table 8, GTP-LoRA achieves higher CLIP and diversity scores than TD-LoRA, with lower adapter computation (1.1× vs. 2.0×). Although TSM achieves a slightly higher CLIP score, GTP-LoRA maintains substantially higher diversity with much lower adapter computation (1.1× vs. 4.0×).

As further discussed in 2. Novelty & Gradient-Based Partitioning of our shared Official Comment, our contribution extends beyond timestep-wise expert specialization to identifying and characterizing gradient imbalance in diffusion reward fine-tuning and introducing a gradient-informed partitioning strategy to address it.

Thank you for addressing my concerns. I will maintain my original positive score.

We sincerely thank the reviewer for the thoughtful feedback and for maintaining your positive assessment of our work.

We would also like to share the completed FLUX.1-Dev experiments mentioned in our previous response. We evaluated DDPO and DDPO + GTP-LoRA on FLUX.1-dev using CLIP reward, with results averaged over three random seeds.

\hline \text{Model} & \text{CLIP}\uparrow & \text{DINOv2}\uparrow & \text{DreamSim}\uparrow \\\\ \hline \text{FLUX.1-dev} & 0.379 & 0.248 & 0.207 \\\\ \hline \text{DDPO} & \underline{0.406} & \underline{0.229} & \underline{0.218} \\\\ \text{DDPO + GTP-LoRA} & \mathbf{0.424} & \mathbf{0.250} & \mathbf{0.238} \\\\ \hline \end{array}$$ GTP-LoRA improves **CLIP (0.406 → 0.424), DINOv2 (0.229 → 0.250), and DreamSim (0.218 → 0.238)** over DDPO. These additional results further support the generalizability of GTP-LoRA to modern flow-matching architectures, demonstrating improvements in both independent evaluation performance and sample diversity. We greatly appreciate your support and constructive suggestions, which have helped us strengthen the paper.
Reviewer_pYjQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GTP-LoRA to address timestep-wise gradient imbalance in reward-based diffusion fine-tuning by assigning separate LoRA experts to gradient-balanced timestep segments. Experiments across multiple diffusion models, reward functions, and optimization methods show improved reward performance with competitive sample diversity, supported by additional ablation studies.

Critical strengths

The paper provides a clear motivation and a simple, intuitive method supported by theoretical analysis. Experiments across multiple models and optimization methods demonstrate consistent reward improvements, with useful ablation studies.

Critical weaknesses

1. Limited novelty and incomplete positioning relative to prior work

The architectural motivation of GTP-LoRA substantially overlaps with existing timestep-specialized diffusion fine-tuning methods. In particular, TimeStep Master (Zhuang et al., ICML 2025) already introduces timestep-specific LoRA experts, while Decouple-Then-Merge (Ma et al., CVPR 2025) addresses cross-timestep gradient interference through specialized fine-tuning. Moreover, Addressing Negative Transfer in Diffusion Models (Go et al., NeurIPS 2023) already investigates gradient-affinity-based clustering of denoising timesteps into contiguous intervals.

Although these works are cited and some are experimentally compared, the paper could more clearly distinguish its contribution from these earlier approaches. The primary methodological novelty appears to be the use of cumulative reward-gradient magnitude to determine expert boundaries, rather than timestep-specific expert adaptation itself.

In particular, it remains unclear why gradient-magnitude-based partitioning should be preferred over existing gradient-affinity-based segmentation methods. A comparison with alternative partitioning criteria would help establish the novelty and practical significance of the proposed design.

Furthermore, related studies such as GradSPO (Tee et al., NeurIPS 2025) and CoCA (Liao et al., 2025) investigate temporal gradient behavior and timestep-level reward attribution in diffusion alignment. Discussing these works would better contextualize the proposed approach relative to existing methods for addressing timestep-dependent optimization and credit assignment.

2. Insufficient causal justification of the proposed mechanism

The paper motivates GTP-LoRA by observing a mismatch between timestep-wise reward sensitivity and optimization gradient magnitude. However, the presence of such a mismatch does not necessarily imply inefficient optimization or harmful interference between timestep-specific gradients.

Although the authors provide empirical evidence of gradient concentration, including parameter-gradient measurements in Appendix D, they do not directly demonstrate that the observed imbalance causes detrimental interference when using a shared LoRA adapter. Consequently, it remains unclear whether the performance improvements of GTP-LoRA result from resolving this particular issue or simply from increased model capacity and timestep specialization.

I would encourage the authors to provide more direct evidence of the claimed mechanism, for example by analyzing gradient conflicts across timesteps or measuring how timestep-specific learning and reward improvements change under the proposed method.

3. Limited empirical evidence for the effectiveness of gradient-informed partitioning

The uniform-expert ablation in Appendix F.2 is helpful, but it raises questions about the incremental benefit of the proposed gradient-informed partitioning.

For example, uniform timestep experts already improve the CLIP score from 0.404 to 0.417, while GTP-LoRA further improves it to only 0.420. Although GTP-LoRA achieves additional improvements in the diversity metrics, the reward improvement attributable specifically to gradient-informed boundary selection is relatively small, and the results are based on only three training seeds.

Moreover, this comparison is limited to the SD1.4/DDPO setting with CLIP reward. It is unclear whether the proposed partitioning strategy provides consistent benefits under other optimization algorithms and reward objectives.

I would encourage the authors to compare against alternative boundary-selection strategies, such as random or gradient-affinity-based partitions, using the same expert architecture and matched computational budgets. Extending the uniform-expert ablation to direct-gradient optimization and multi-reward fine-tuning would also help establish the generality of the proposed method.

Questions for authors

see above.

2 comments

We thank the reviewer for the constructive feedback and thoughtful suggestions. We address each concern below, with additional details provided in our shared Official Comment.

1. Limited Novelty and Incomplete Positioning Relative to Prior Work

As detailed in 2. Novelty & Gradient-Based Partitioning of our shared Official Comment, our key contribution lies not only in introducing timestep-wise LoRA experts into diffusion reward fine-tuning, but also in identifying and characterizing the optimization issue that motivates their use.

While TSM, DeMe, and ANT investigate timestep specialization or gradient interference in conventional diffusion training, GTP-LoRA specifically targets gradient concentration arising from reward optimization across both policy-gradient and direct-gradient methods. Unlike ANT, which considers gradient affinity between denoising tasks for timestep clustering, GTP-LoRA determines expert boundaries by balancing cumulative reward-optimization gradient magnitudes.

To further motivate this choice, we additionally analyzed pairwise cosine similarities between timestep-wise parameter gradients under both policy-gradient (PG) and direct-gradient (DG). As shown in Fig. 12 of Appendix K in the revised manuscript, most off-diagonal similarities are close to zero, with no pronounced block-wise affinity structure. Although some neighboring timesteps exhibit weak positive similarities under DG, the overall directional affinity provides limited guidance for timestep grouping. As discussed in Appendix K, this may stem from the different gradient structures of reward optimization, where PG relies on reward-weighted transition noise and DG propagates reward gradients through timestep-dependent Jacobians. In contrast, our analysis consistently identifies pronounced gradient magnitude imbalance under both optimization families (Sec. 4). Thus, we adopt magnitude-based segmentation to directly address this observed imbalance rather than relying on weak timestep-wise affinity.

We also acknowledge the relevance of GradSPO and CoCA. GradSPO investigates biased timestep weighting and noisy gradient estimates in stepwise preference optimization, while CoCA addresses sparse terminal rewards through timestep-wise credit assignment. In contrast, GTP-LoRA addresses the disproportionate influence of concentrated gradients on shared parameters through expert specialization, without modifying the underlying reward objective or sampling scheduler. We have clarified these distinctions in Section 2 (Related Work) of the revised manuscript.

2. Insufficient Causal Justification of the Proposed Mechanism

As detailed in 1. Validation of Gradient Interference of our shared Official Comment, we conducted a one-step transfer analysis to directly examine whether late-stage gradients adversely affect earlier denoising stages. By comparing virtual updates with and without late-stage gradient contributions, we observe negative transfer on earlier-stage optimization objectives (Fig. 11, Appendix J of the revised manuscript). This provides direct evidence of harmful cross-timestep interactions beyond gradient magnitude imbalance alone.

3. Limited Empirical Evidence for the Effectiveness of Gradient-Informed Partitioning

As detailed in 2. Novelty & Gradient-Based Partitioning of our shared Official Comment, we additionally compared GTP-LoRA with Uniform 3-Experts under identical boundary mixing, expert architecture, and training settings. GTP-LoRA achieves higher CLIP and diversity scores under both 20- and 50-step settings. Although the CLIP gains are modest, these controlled comparisons support the independent contribution of gradient-informed boundary selection beyond timestep specialization and boundary mixing.

Furthermore, our original experiments demonstrate improvements across both policy-gradient and direct-gradient methods and multi-reward optimization. As detailed in 3. Generalization Beyond Animal Activity and CLIP Reward of our shared Official Comment, additional HPSv2.1 experiments also show improvements beyond CLIP optimization.

Thank you again for your constructive feedback. We would appreciate your thoughts on whether the additional experiments and clarifications in our rebuttal address your main concerns. Please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_mHuWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GTP-LoRA to address timestep-wise gradient imbalance in reward-based diffusion fine-tuning. It partitions denoising timesteps based on gradient magnitudes and assigns separate LoRA experts to each segment. Experiments show improved reward performance across different optimization algorithms.

Critical strengths
  1. Solid theoretical analysis. The authors derive timestep-wise gradients for both policy-gradient and direct-gradient optimization, offering useful insights into the sources of gradient imbalance.

  2. Simple and flexible design. GTP-LoRA adopts a simple yet effective design and can be integrated into different reward optimization algorithms.

Critical weaknesses
  1. Limited novelty and utdated model backbones. The use of timestep-specific LoRA experts is not new and has been explored in prior works such as TSM [1] and T-LoRA [2]. The method appears to be an incremental extension of existing timestep-wise expert methods. The experiments are restricted to U-Net-based SD1.4 and SDXL, while more recent models adopt diffusion transformers and flow matching.
  2. Limited evaluation diversity. All experiments use the narrow Animal Activity dataset with only 135 prompts. Such a narrow task distribution is insufficient to demonstrate generalizability to more diverse text-to-image generation tasks

[1] Zhuang, S., et al. (2025). TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision. ICML 2025.

[2] Soboleva, V., et al. (2026). T-LoRA: Single Image Diffusion Model Customization Without Overfitting. AAAI 2026.

Questions for authors
  1. In Sec. 4, the authors observe that early denoising steps exhibit higher reward sensitivity, while the optimization gradients are concentrated toward later steps.However, higher sensitivity does not necessarily imply greater optimization benefits, especially given the high uncertainty at early denoising stages. Could the authors provide direct empirical evidence about the connection between early stage optimization and its effectiveness? For example, an ablation that independently fine-tunes early, middle, and late stages to quantify their respective contributions.
  2. The experiments are limited to SD1.4 and SDXL, whereas recent generation models more favor flow-matching formulations, such as FLUX. Since the gradient analysis in this paper relies on DDIM, it remains unclear whether the same imbalance patterns and the proposed partitioning strategy can extend to these more advanced models. If not, what are the limitations?
  3. As described in Sec 6.1 (L367–372) and Appendix H, all experiments are conducted on the Animal Activity dataset, which consists of only 135 prompts. This is a a narrow evaluation. Could the authors evaluate GTP-LoRA on more diverse benchmarks to better demonstrate its generalizability?
  4. The authors use 1,024 reference trajectories to estimate the timestep-wise gradient profile and determine fixed expert boundaries. However, the data source is not clearly specified. More importantly, would the method require recalculating the boundaries when the training prompt distribution changes?
4 comments

We thank the reviewer for the constructive feedback and valuable suggestions. We address each concern below, with additional details provided in our shared Official Comment.

1. Novelty & Gradient-Based Partitioning

As detailed in 2. Novelty & Gradient-Based Partitioning of our shared Official Comment, our key contribution lies not only in introducing timestep-wise LoRA experts into diffusion reward fine-tuning, but also in identifying and characterizing the optimization issue that motivates their use.

2. Reward Sensitivity and Optimization Benefits

We agree that higher reward sensitivity does not necessarily imply greater optimization benefits from fine-tuning early denoising stages. Our analysis does not assume such a relationship. Instead, the mismatch between reward sensitivity and gradient magnitude is an empirical observation that motivates further investigation, rather than direct evidence of harmful interference. We therefore examine whether late-stage gradient concentration adversely affects earlier-stage optimization through the additional one-step transfer analysis. Our method aims to mitigate this imbalance through timestep-wise expert separation, rather than assuming that early-stage optimization inherently yields greater reward improvements.

As detailed in 1. Validation of Gradient Interference of our shared Official Comment, our additional one-step transfer analysis provides direct empirical evidence that late-stage gradient contributions can adversely affect earlier-stage optimization (Fig. 11, Appendix J of the revised manuscript).

3. Extension to Flow-Matching Models

To investigate whether timestep-wise gradient imbalance also arises in flow-matching models, we conducted an additional analysis using FLUX.1-dev. As shown in Figure 13 of Appendix L in the revised manuscript, both the scheduling factor and the corresponding denoiser-output gradient norm exhibit pronounced concentration toward late reverse-generation steps. This suggests that timestep-wise gradient imbalance is not restricted to DDIM-based diffusion models but can also arise in flow-matching models. We are also conducting experiments with DDPO and DDPO + GTP-LoRA on FLUX.1-dev to evaluate the effectiveness of our method in this setting. These experiments are ongoing, and we will update our response as the results become available.

4. Generalization Beyond Animal Activity

As detailed in 3. Generalization Beyond Animal Activity and CLIP Reward of our shared Official Comment, we conducted additional experiments on the HPS Benchmark (3,152 training / 48 unseen prompts) using HPSv2.1 as the training reward. GTP-LoRA improves both reward performance and diversity over DDPO, while also improving ImageReward, an independent evaluator not used during training. These results provide additional evidence of generalization beyond Animal Activity and CLIP reward optimization.

5. Reference Trajectories and Boundary Recalculation

The 1,024 reference trajectories are generated from the pretrained base model using prompts sampled from the training distribution during the pre-initialization stage, before reward fine-tuning. When the training prompt distribution changes, the gradient profile and expert boundaries should be recalculated to reflect the new distribution. However, this requires only a small one-time computational cost (Table 6 of Appendix B), which is minor relative to the full training cost.

We would like to share the completed FLUX.1-dev experiments mentioned in our previous response, directly addressing your concern regarding the applicability of GTP-LoRA to modern flow-matching architectures.

We evaluated DDPO and DDPO + GTP-LoRA on FLUX.1-dev, a flow-matching Diffusion Transformer, using CLIP as the training reward. All results are averaged over three random seeds.

\hline \text{Model} & \text{CLIP}\uparrow & \text{DINOv2}\uparrow & \text{DreamSim}\uparrow \\\\ \hline \text{FLUX.1-dev} & 0.379 & 0.248 & 0.207 \\\\ \hline \text{DDPO} & \underline{0.406} & \underline{0.229} & \underline{0.218} \\\\ \text{DDPO + GTP-LoRA} & \mathbf{0.424} & \mathbf{0.250} & \mathbf{0.238} \\\\ \hline \end{array}$$ Compared with DDPO, GTP-LoRA improves **CLIP (0.406 → 0.424) and both diversity metrics (DINOv2: 0.229 → 0.250; DreamSim: 0.218 → 0.238)**. Together with the gradient imbalance observed in FLUX.1-dev (Fig. 13, Appendix L), these results provide further empirical evidence that the benefits of GTP-LoRA extend beyond U-Net-based diffusion models to modern flow-matching architectures. We hope these additional results help address your concerns regarding the generalizability of our approach.

Thank you again for your constructive feedback. We would appreciate your thoughts on whether the additional experiments and clarifications in our rebuttal address your main concerns. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for the response. My concerns have been partially addressed. I decide to maintain my original score.

Reviewer_TbhbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GTP-LoRA: based on the timestep-wise denoiser-output gradient profile estimated prior to training, the denoising process is partitioned into segments whose cumulative gradient magnitudes are approximately balanced, a separate LoRA expert is assigned to each segment, and adjacent experts are mixed at segment boundaries. The method aims to mitigate timestep-wise optimization interference within a shared adapter and yields reward improvements across multiple reward fine-tuning algorithms.

Critical strengths
  1. Simple design with good compatibility. The method requires no modification to the original reward objective and can be integrated with both policy-gradient and direct-gradient algorithms, with only a single expert activated at most timesteps.
  2. Relatively detailed analysis. The appendix supplements the noise correction for the reward-sensitivity estimate, the parameter-space gradient analysis, and the stability of fixed boundaries, providing reasonable motivation for the method.
  3. Reasonably thorough basic validation. The study covers two base models, multiple optimization algorithms, three random seeds, and several ablations; relative to shared LoRA, the reward gains are reasonably consistent.
Critical weaknesses
  1. The independent contribution of the core innovation remains unclear. Timestep-wise LoRA experts have been explored in prior work; in Table 8, the uniform-segmentation variant already reaches a CLIP score of 0.417, versus 0.420 for this method. The two also differ in their boundary-mixing scheme, so the benefit attributable to gradient-based segmentation itself cannot be adequately isolated.
  2. The mechanistic explanation lacks direct validation. Imbalanced gradient norms do not necessarily imply harmful interference, and balanced cumulative output gradients do not equate to balanced actual parameter updates. The current analysis mainly provides correlational support.
  3. The scope of the experimental conclusions is limited. Evaluation is concentrated on Animal Activity, and the reward evaluator also participates in training; DreamSim degrades in some settings, and diversity is not reported for SDXL. Evidence for broad generalization and quality preservation is therefore still lacking.
Questions for authors
  1. What is the independent gain of gradient-based segmentation over uniform segmentation? Please add an experiment with "uniform three experts + identical boundary mixing," keeping the parameter count, optimization configuration, and training budget identical, and report per-seed differences, so as to disentangle the contributions of the segmentation criterion and the mixing mechanism.
  2. Is there direct evidence that updates from late reverse-denoising timesteps adversely affect earlier denoising stages? A cross-timestep gradient-direction analysis on the shared LoRA parameters, or a controlled intervention measuring the effect of updates from one timestep range on another, would help assess this explanation. For an intervention, please clarify how the effect on another stage is measured.
  3. Do the gains extend to prompts beyond Animal Activity and translate into quality improvements under an evaluator not used for reward optimization or a human preference assessment? Reporting SDXL diversity would also help clarify whether the reward–diversity trade-off carries over to the larger model.
3 comments

We thank the reviewer for the constructive feedback and detailed suggestions. We address each concern below, with additional details provided in our shared Official Comment.

1. Independent Contribution of Gradient-Based Partitioning

As detailed in 2. Novelty & Gradient-Based Partitioning of our shared Official Comment, we additionally evaluated Uniform 3-Experts with identical boundary mixing, expert architecture, and training settings. GTP-LoRA achieves higher CLIP and diversity scores under both 20- and 50-step settings. Although the CLIP improvements are modest, these controlled comparisons support the independent benefit of gradient-informed partitioning beyond timestep-wise expert separation and boundary mixing.

2. Direct Validation of Gradient Interference

As detailed in 1. Validation of Gradient Interference of our shared Official Comment, we conducted a one-step transfer analysis to examine whether late-stage gradients adversely affect earlier denoising stages. We compared virtual updates with and without late-stage gradient contributions and separately evaluated their effects on the optimization objectives of earlier stages (Fig. 11, Appendix J of the revised manuscript).

3. Generalization and SDXL Diversity

As detailed in 3. Generalization Beyond Animal Activity and CLIP Reward of our shared Official Comment, additional experiments on the HPS Benchmark using HPSv2.1 reward demonstrate improvements beyond Animal Activity prompt and CLIP optimization. Notably, GTP-LoRA also improves ImageReward, an independent human-preference evaluator not used during training.

We additionally evaluated SDXL diversity under 5-Mix reward optimization, addressing the missing evaluation in the original submission. The results are summarized below and have also been incorporated into Table 3 of the revised manuscript.

\hline \text{Model} & \text{DINOv2}\uparrow & \text{DreamSim}\uparrow \\\\ \hline\hline \text{SDXL} & 0.261 & 0.294 \\\\ \hline\hline \text{DDPO} & 0.188 & \underline{0.176} \\\\ \text{PCPO-base} & 0.191 & 0.164 \\\\ \text{PCPO} & \underline{0.223} & 0.164 \\\\ \text{DDPO + GTP-LoRA} & \mathbf{0.230} & \mathbf{0.189} \\\\ \hline \end{array}$$ GTP-LoRA achieves the highest diversity among the fine-tuned methods on both DINOv2 and DreamSim. These results provide additional evidence that GTP-LoRA can improve the reward–diversity trade-off on larger diffusion models.

Thank you again for your constructive feedback. We would appreciate your thoughts on whether the additional experiments and clarifications in our rebuttal address your main concerns. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for addressing all my concerns. I’ll update my score accordingly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared Official Comment (1/2)

We thank all reviewers for their constructive feedback and valuable suggestions. Below, we address the key concerns raised across multiple reviews. All additional experimental results are reported as averages over the same three random seeds used in our original experiments.

1. Validation of Gradient Interference

To examine whether late-stage gradient concentration leads to harmful interference with earlier denoising stages, we conducted a one-step transfer analysis. We divided the 20-step denoising trajectory into three stages (A/B/C: 8/9/3 steps, from early to late denoising). Here, JAJ_A and JBJ_B denote the stage-specific DDPO surrogate objectives, defined as the negatives of the mean DDPO losses over timesteps belonging to stages A and B, respectively, evaluated on the same training batch.

Every four training updates, starting from identical model parameters and the same batch, we performed two virtual one-step updates: one using the A/B losses and another using the A/B/C losses, yielding θAB\theta_{AB} and θABC\theta_{ABC}, respectively. We then re-evaluated both models on the same batch and computed the transfer scores JA/B(θABC)−JA/B(θAB)J_{A/B}(\theta_{ABC})-J_{A/B}(\theta_{AB}). Since these objectives are maximized, a negative score indicates that including stage C gradients reduces the earlier-stage objective relative to the AB-only update. As shown in Figure 11 of Appendix J in the revised manuscript, including late-stage gradients leads to negative transfer on earlier stages, providing direct evidence of cross-timestep interference beyond gradient magnitude imbalance alone.

2. Novelty & Gradient-Based Partitioning

Our contribution is not limited to gradient-based segmentation itself, nor do we claim novelty in timestep-wise LoRA experts alone. Rather, we identify and characterize an optimization issue specific to diffusion reward fine-tuning: gradients from both policy-gradient and direct-gradient optimization are disproportionately concentrated at late denoising steps (Sec. 4). Based on this analysis, we introduce timestep-wise LoRA experts into diffusion reward fine-tuning, with boundaries determined by gradient profiles to prevent high-gradient timesteps from disproportionately dominating a shared adapter. To our knowledge, this is the first work to introduce timestep-wise LoRA experts into diffusion reward fine-tuning, supported by a gradient-based analysis explaining why such specialization can benefit reward optimization. This analysis explains why even Uniform 3-Experts improves over vanilla LoRA (0.404 → 0.417, Table 8): uniform partitioning also separates gradient updates across timestep regimes.

To further isolate the effect of gradient-based segmentation, we compared GTP-LoRA with Uniform 3-Experts under identical boundary mixing and training settings. At 20 denoising steps, Uniform with identical boundary mixing achieves a similar CLIP score but lower diversity. The small CLIP gap may be due to the limited number of denoising steps, as the two partitions differ by only a few effective steps per expert (7.87/8.70/3.43 vs. 6.67/6.67/6.67). At 50 steps (18.32/21.30/10.38 vs. 16.67/16.67/16.67), GTP-LoRA shows a slightly larger CLIP gain with higher diversity. These results suggest a consistent benefit from gradient-based segmentation beyond expert separation and boundary mixing, with only a small one-time pre-initialization cost (Table 6).

\hline \text{Model} & \text{CLIP}\uparrow & \text{DINOv2}\uparrow & \text{DreamSim}\uparrow \\\\ \hline\hline \text{SD1.4} & 0.352 & 0.454 & 0.451 \\\\ \hline\hline \text{DDPO (20 steps)} & 0.404 & 0.358 & 0.366 \\\\ \text{+ Uniform} & 0.417 & \underline{0.380} & 0.363 \\\\ \text{+ Uniform w/ Boundary Mixing} & \underline{0.418} & 0.378 & \underline{0.377} \\\\ \text{+ GTP-LoRA} & \mathbf{0.420} & \mathbf{0.413} & \mathbf{0.384} \\\\ \hline \text{DDPO (50 steps)} & 0.399 & \underline{0.373} & \underline{0.377} \\\\ \text{+ Uniform w/ Boundary Mixing} & \underline{0.420} & 0.372 & 0.369 \\\\ \text{+ GTP-LoRA} & \mathbf{0.424} & \mathbf{0.433} & \mathbf{0.413} \\\\ \hline \end{array}$$ (page 1/2)
AuthorsOpenReview ↗

Shared Official Comment (2/2)

3. Generalization Beyond Animal Activity and CLIP Reward

To assess generalization beyond the original prompt set (Animal Activity) and CLIP reward optimization, we fine-tuned SD1.4 on the HPS Benchmark (3,152 training prompts / 48 unseen evaluation prompts) using HPSv2.1 reward. We evaluated HPSv2.1, two diversity metrics (DINOv2 and DreamSim), and ImageReward, an independent human-preference metric not used during training:

\hline \text{Model} & \text{HPSv2.1}\uparrow & \text{DINOv2}\uparrow & \text{DreamSim}\uparrow & \text{ImageReward}\uparrow \\\\ \hline\hline \text{SD1.4} & 0.220 & 0.437 & 0.409 & -0.069 \\\\ \hline\hline \text{DDPO} & \underline{0.260} & \underline{0.422} & \underline{0.366} & \underline{0.305} \\\\ \text{DDPO + GTP-LoRA} & \mathbf{0.285} & \mathbf{0.459} & \mathbf{0.369} & \mathbf{0.426} \\\\ \hline \end{array}$$ Compared with DDPO, GTP-LoRA consistently improves HPSv2.1 and both diversity metrics. Importantly, ImageReward also improves without being optimized, supporting generalization beyond the training reward. These results demonstrate GTP-LoRA's **generalization across different prompts and reward objectives** while improving the reward–diversity trade-off. We plan to extend the evaluation to additional optimization algorithms in the camera-ready version. (page 2/2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.