- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates reward weights across denoising steps in multi-reward fine-tuning of diffusion models. The authors propose ReCAST, which uses a reward-by-timestep weight matrix to adjust each reward’s weight across denoising steps while preserving its total weight budget. The allocation is guided by a proxy for per-step Rényi discriminability gains. A Sinkhorn projection then enforces the specified per-reward budgets and a uniform total weight per timestep. Experiments are conducted on SD3.5-Medium in two multi-reward settings incorporating OCR and GenEval, respectively. The authors report improvements in the OCR setting's aggregate objective and comparable performance in the GenEval setting, along with higher average scores from held-out reward models in both settings.
Critical strengths
Clear Problem Formulation: The paper separates how much each reward matters overall from when it should have greater influence during denoising. This distinction is well motivated, and the weight matrix provides a clear mathematical formulation that preserves per-reward budgets while allowing reward-specific temporal allocation.
Evaluation Beyond Training Rewards: The experiments cover five reward budgets and three training seeds and include multiple held-out reward models. In the GenEval setting, ReCAST achieves higher average scores from all four held-out models despite comparable training-reward scores to the static baseline. This provides evidence that the benefits extend beyond the rewards directly optimized during training.
Critical weaknesses
Gap Between the Theoretical Divergence and the Implemented Surrogate: The temporal allocation is motivated by Rényi divergence gains, but the implemented estimator uses a common external generator in place of the reward-specific positive policies. Although Section 3.3 acknowledges this approximation and Appendix A.3 provides an error bound under a small-KL-mismatch assumption, the paper does not establish that this assumption holds in the experimental setting. Nevertheless, Appendix B.6 interprets the reduction in the surrogate quantity as a reduction in the corresponding Rényi divergence and invokes the Rényi-to-KL inequality to claim that the KL gap is nearly closed. This conclusion is not justified for the surrogate without additional assumptions. Moreover, clipping negative estimated increments breaks the telescoping identity in Eq. (7). While mean-normalization remains well defined when the total clipped gain is positive, its denominator can no longer generally be identified with the data-level divergence. These issues weaken the claimed theoretical interpretation of the implemented allocation.
Insufficient Controls for the Temporal Profiles and Their Computational Cost: The main comparison is against static reward weighting, while the no-Sinkhorn ablation tests the column constraint rather than the importance of the estimated temporal profiles. Comparisons with simple reward-specific schedules and shuffled or reversed profiles would help establish whether the gains depend on the proposed reward-specific temporal structure. This distinction matters because constructing the profiles requires substantial additional computation: Appendix E.2 implies approximately 90 GPU-hours for a 25-step curve, compared with approximately 37 GPU-hours for an OCR training-stage run. Although this one-time cost can be amortized across runs, a compute-matched comparison and an explicit account of the reuse assumptions are needed to assess the practical benefit.
Questions for authors
Interpretation of the surrogate and clipping: What assumptions justify applying the Rényi-to-KL inequality to the external-generator surrogate in Appendix B.6, and what evidence supports these assumptions in the experimental setting? Please also clarify how clipping negative estimated increments affects the telescoping identity in Eq. (7), the interpretation of the normalization denominator in Eq. (13), and the resulting allocation weights.
Contribution of the estimated gain curves: How does ReCAST compare with simple reward-specific temporal schedules or with shuffled/reversed gain curves under the same marginal constraints? Such controls would help determine whether the improvements depend on the estimated reward-specific temporal structure.
Computational cost: When gain-curve construction is included in the total computational budget, how does ReCAST compare with the static baseline given an equivalent budget? Please clarify how many downstream runs must reuse a curve for its construction cost to be amortized.
3 comments
Thank you for the detailed rebuttal. Since your additional experiments, theoretical corrections, and amortization analysis have addressed my concerns, I will raise my score to 3.
Thank you for your constructive feedback, which helped us strengthen the experiments and clarify the theoretical analysis and computational costs. We are glad that the revisions have addressed your concerns.
Thank you for your feedback. We address each weakness and question below.
Weakness 1 — Gap between the theoretical divergence and implemented surrogate. Common response 3 defines the reward-moment correction and includes the surrogate correction and endpoint error in the KL bound. Common response 4 separates the signed telescoping sum from the clipped sum and quantifies the effect of clipping on the allocation weights.
Weakness 2 — Temporal-profile controls and computational cost. Common response 1 reports reversed and time-shuffled controls, including their training rewards and held-out evaluations. Common response 2 reports construction and projection costs, amortization across budgets and seeds.
Question 1 — Interpretation of the surrogate and clipping. Common response 3 gives the corrected KL bound and empirical correction estimates. For Eq. (7), the signed increments telescope, while clipping adds the negative mass Mi−. The denominator in Eq. (13) is the mean clipped gain, equal to the signed mean plus Mi−/T. Common response 4 reports that removing clipping changes the applied profiles by mean TV 0.005–0.009, compared with finite-sample variability of 0.059.
Question 2 — Contribution of the estimated gain curves. Common response 1 provides the requested reversed and time-shuffled comparisons. The controls preserve both applied marginals. ReCAST has the highest mean aggregate and individual reward. The same response discusses why a prespecified schedule requires reward-specific timing choices that are not known in advance.
Question 3 — Computational cost and reuse. Common response 2 reports construction costs of approximately 2 GPU-hours for a 10-step curve and 12 GPU-hours for a 25-step curve. Reusing each curve across 5 reward budgets and 3 training seeds gives 15 downstream runs, amortizing these costs to approximately 0.13 and 0.8 GPU-hours per run.
Comparison with a larger training budget for static. On the held-out OCR test set at the balanced reward budget λ=(1,1,1,1), we compare static at step 180 (60 OCR-stage updates) with ReCAST at step 170 (50 OCR-stage updates), giving static ten additional training updates. Across three matched seeds, ReCAST achieves a higher mean aggregate reward of 2.443±0.048 versus 2.387±0.058 for static (mean ± SD). Under the 15-run curve-reuse setting above, static's ten extra updates exceed the one-to-two-update equivalent of ReCAST's amortized curve-construction cost. ReCAST therefore retains the higher mean aggregate reward with a lower estimated total GPU compute budget in this comparison.