- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes LEEGS, a method for learning time-dependent guidance schedules in loss-guided diffusion. By optimizing guidance weights through the sampling trajectory with an efficient gradient approximation, the method improves generation quality across image inverse problems, face-ID guidance, and PDE applications.
Critical strengths
-
Meaningful extension to a broad application domain. Extending learned guidance schedules to loss-guided diffusion is valuable, as this framework supports diverse conditional generation and inverse problems. The method is general and does not require modifying the pretrained diffusion model.
-
Practical and coherent methodology. The proposed framework is straightforward and well-motivated. The trajectory-level objective and efficient gradient approximation make end-to-end schedule optimization computationally feasible.
-
Comprehensive experimental validation. The paper evaluates diverse tasks and sampling budgets, supported by extensive ablation studies. The results generally demonstrate improvements over fixed or heuristic guidance schedules and competitive performance against established methods.
Critical weaknesses
-
Limited methodological novelty. The underlying idea of optimizing time-dependent parameters through differentiable diffusion sampling has been explored in prior work. For example, Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models (CVPR 2023) optimizes per-step text-embedding mixing coefficients, while DRaFT (ICLR 2024) also explores learning per-step guidance weights. Therefore, the main contribution lies in extending an established optimization paradigm to loss-guided diffusion rather than introducing a fundamentally new framework. While this extension is meaningful, its relationship to these earlier works deserves clearer discussion.
-
Insufficient justification for unconstrained per-step schedule optimization. LEEGS independently optimizes the guidance weight at every sampling step, but the experiments primarily compare against fixed or manually parameterized schedules. A comparison with low-dimensional schedules trained using the same optimization procedure would better establish the necessity of full per-step optimization. Moreover, Appendix C.4 shows that matching the overall guidance magnitude substantially reduces performance differences between schedules on some tasks, although task-specific shapes remain important for others. This further motivates investigating whether simpler learnable parameterizations could achieve comparable performance.
Questions for authors
Q1. How does LEEGS compare with low-dimensional learnable schedules, such as polynomial or spline schedules, optimized using the same objective and training budget?
2 comments
Response
Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.
1. Methodological novelty and positioning.
I appreciate the clarification of the relationship between LEEGS and prior work on optimizing sampling parameters. I agree that extending end-to-end guidance schedule learning to loss-guided diffusion is a meaningful contribution, particularly given the additional computational challenges introduced by differentiating through guidance gradients. The revised positioning appropriately clarifies the contribution of this work.
2. Low-dimensional learnable schedules.
The new B-spline experiments directly address my main experimental concern. The results show that low-dimensional parameterizations can achieve performance comparable to full per-step optimization, which is an interesting finding.
I also find the authors' explanation for retaining per-step parameterization convincing. Since the dominant computational cost comes from backpropagation through the diffusion sampler rather than the number of schedule parameters, reducing the parameter count offers limited computational benefits. Meanwhile, per-step optimization provides greater flexibility without requiring assumptions about the schedule shape.
Therefore, I consider the choice of per-step parameterization sufficiently justified, although the results also demonstrate that simpler parameterizations can be effective.
Overall, I am satisfied with the rebuttal. The authors have adequately addressed my primary concerns, and I appreciate the additional experimental evidence and clarification of the paper's contributions.
I will maintain my original rating of 3 (Weak Acceptance).
We thank the reviewer for their positive assessment, recognising the extension to loss-guided diffusion as meaningful, the methodology as practical, and the experimental validation as comprehensive. We are thankful also for their suggestions on related work and on the parametrisation of the schedule. We address each point below. All changes to the manuscript are highlighted: changes to the text in orange, and new table rows in red. In the list of contents of the appendix, new appendix sections are marked in orange, and existing sections with new content in red. Section, figure, table and line numbers below refer to the revised manuscript.
W1 (relation to Wu et al. and DRaFT). We thank the reviewer for these references, which we have added to the related work (Section 5) together with other methods that learn sampling hyperparameters by differentiating through the sampler. LEEGS does belong to the family of methods that optimize per-step sampler quantities. As the reviewer notes, its contribution is to learn guidance schedules for loss-guided diffusion, a setting that neither mentioned work considers and in which the sampler itself contains the guidance gradient (see our response to Reviewer
cPQR, W1a). Wu et al. (2023) learn per-step mixing weights between two text embeddings for text-driven image editing. DRaFT (Clark et al., 2024) fine-tunes model weights on a differentiable reward, and learns per-step weights for classifier-free guidance in an exploratory experiment (their App. B.12). It is therefore also closely related to the works on learned CFG weights already discussed in our related work, and we have added the reference in that section too. Interestingly, DRaFT reports that learning these sampler inputs "did not improve results much at large scale", whereas in loss-guided diffusion a learned schedule beats constant guidance at a matched budget (Tables 1 and 2).W2/Q1 (low-dimensional learnable schedules). We have run the comparisons requested by the reviewer (new App. H, Table 22-23 and Fig. 28): cubic B-spline schedules with 4, 5, 7 and 11 parameters, trained with the same objective, training images and learning rate as LEEGS on FFHQ box inpainting at 50 NFEs. With 5 or more parameters, the splines reproduce the shape of the learned LEEGS schedule, and the 11-parameter spline is nearly identical to it. In preliminary results on 300 held-out images (disjoint from the training images of all LEEGS draws), the 11-parameter spline reaches LPIPS 0.173 and the 5-parameter spline 0.185, both comparable to LEEGS (0.180 ± 0.004, 3 draws). The 7-parameter spline reaches 0.192, while the 4-parameter spline (a single cubic segment) is clearly worse at 0.305, which we attribute to both a limited modeling capacity, and worse training dynamics (see App. H, "Further analysis"). This shows that LEEGS also works reliably for low-dimensional parametrizations. While using low dimensional schedules does not save computational time (which is dominated by the repeated backpropagation through the diffusion model), they may be useful in cases where prior knowledge is available, or there is a need to enforce smooth schedules. In other settings where more complex schedules arise (such as sampling with renoising; App. D), or in the absence of prior knowledge, per-step parametrization remains preferable. In addition to App. H, we have added a paragraph in Sec. 4.2 summarising this study.