- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines timestep-wise reward sensitivity and gradient magnitude in reward-based diffusion fine-tuning. The authors find that reward sensitivity is greater during early denoising steps, while optimization gradients are concentrated in later steps. Based on this finding, they propose GTP-LoRA, which groups denoising timesteps by gradient magnitude and assigns a separate LoRA expert to each group. The method improves reward performance while preserving competitive sample diversity.
Critical strengths
-
Clear and well-written paper. The paper is easy to follow, with a clear motivation, analysis, and methodology.
-
Insightful empirical analysis. The contrasting timestep-wise profiles of reward sensitivity and optimization gradients are noteworthy and are supported by empirical analysis.
-
Simple and effective method. GTP-LoRA can be integrated into existing reward fine-tuning algorithms without modifying their objectives or sampling schedulers, while adding minimal inference overhead.
-
Promising empirical results. The method consistently improves performance across multiple optimization algorithms while maintaining competitive sample diversity.
Critical weaknesses
-
Limited evaluation across reward objectives. Although the paper evaluates multi-reward optimization on SDXL, all single-reward experiments optimize only the CLIP reward on SD1.4. It remains unclear whether the observed improvements generalize to other reward objectives and larger models in the single-reward setting.
-
Limited analysis compared to existing timestep-wise expert baselines. While the paper provides a thorough evaluation of the benefits of applying GTP-LoRA to existing reward-alignment methods, its comparisons with prior timestep-wise expert approaches, such as TSM and TD-LoRA, remain limited and are presented only in the appendix. Given the methodological similarities, more extensive comparisons and a clearer explanation of GTP-LoRA’s novelty and advantages over existing approaches would strengthen the paper’s contribution.
Questions for authors
-
Could the authors provide additional single-reward experiments using other objectives, such as Aesthetic, PickScore, or HPS? It would also be valuable to evaluate CLIP-only optimization on SDXL to determine whether the improvements generalize to larger models.
-
Could the authors clarify the methodological differences between GTP-LoRA and existing timestep-wise expert methods, such as TSM and TD-LoRA, and provide a more detailed comparative analysis of these baselines?
3 comments
Thank you for addressing my concerns. I will maintain my original positive score.
We sincerely thank the reviewer for the thoughtful feedback and for maintaining your positive assessment of our work.
We would also like to share the completed FLUX.1-Dev experiments mentioned in our previous response. We evaluated DDPO and DDPO + GTP-LoRA on FLUX.1-dev using CLIP reward, with results averaged over three random seeds.
\hline \text{Model} & \text{CLIP}\uparrow & \text{DINOv2}\uparrow & \text{DreamSim}\uparrow \\\\ \hline \text{FLUX.1-dev} & 0.379 & 0.248 & 0.207 \\\\ \hline \text{DDPO} & \underline{0.406} & \underline{0.229} & \underline{0.218} \\\\ \text{DDPO + GTP-LoRA} & \mathbf{0.424} & \mathbf{0.250} & \mathbf{0.238} \\\\ \hline \end{array}$$ GTP-LoRA improves **CLIP (0.406 → 0.424), DINOv2 (0.229 → 0.250), and DreamSim (0.218 → 0.238)** over DDPO. These additional results further support the generalizability of GTP-LoRA to modern flow-matching architectures, demonstrating improvements in both independent evaluation performance and sample diversity. We greatly appreciate your support and constructive suggestions, which have helped us strengthen the paper.
We thank the reviewer for the positive assessment and constructive suggestions. We address each concern below, with additional details provided in our shared Official Comment.
1. Limited Evaluation Across Reward Objectives
As detailed in 3. Generalization Beyond Animal Activity and CLIP Reward of our shared Official Comment, we conducted additional single-reward experiments using HPSv2.1 instead of CLIP on the HPS Benchmark. GTP-LoRA improves HPSv2.1 and both diversity metrics over DDPO. Notably, ImageReward, an independent human-preference evaluator not used during training, also improves. These results provide additional evidence that GTP-LoRA's effectiveness is not restricted to CLIP reward optimization.
For larger models, our original SDXL experiments already demonstrate improvements across all five reward components under 5-Mix optimization. Due to computational constraints, we may not be able to complete SDXL CLIP-only experiments within the rebuttal period. Instead, we are currently conducting single-reward fine-tuning experiments on FLUX.1-dev to further assess generalization to flow-matching architectures. These experiments are ongoing, and we will update our response as the results become available.
2. Limited Analysis Compared to Existing Timestep-Wise Expert Baselines
Unlike TSM and TD-LoRA, which employ learned expert routing, GTP-LoRA uses fixed expert boundaries determined by cumulative reward-optimization gradient magnitudes, requiring no additional router training. Although these methods share the idea of timestep-specific expert adaptation, GTP-LoRA is specifically motivated by the gradient concentration observed in reward fine-tuning across both policy-gradient and direct-gradient optimization (Sec. 4). Furthermore, our additional analysis (Fig. 12, Appendix K) reveals weak timestep-wise gradient affinity without a pronounced clustering structure, providing further motivation for using gradient magnitude rather than affinity as the partitioning criterion.
In terms of empirical performance, as shown in Table 8, GTP-LoRA achieves higher CLIP and diversity scores than TD-LoRA, with lower adapter computation (1.1× vs. 2.0×). Although TSM achieves a slightly higher CLIP score, GTP-LoRA maintains substantially higher diversity with much lower adapter computation (1.1× vs. 4.0×).
As further discussed in 2. Novelty & Gradient-Based Partitioning of our shared Official Comment, our contribution extends beyond timestep-wise expert specialization to identifying and characterizing gradient imbalance in diffusion reward fine-tuning and introducing a gradient-informed partitioning strategy to address it.