acceptodds
Under review as a conference paper at ICLR 2027

Learned End-to-End Guidance Schedules for Diffusion Models

Abstract

Diffusion models are a powerful generative paradigm used across multimedia and scientific applications. Guided diffusion methods impose requirements on the generation by adding the gradient of a differentiable loss (the *guidance function*) as a drift term during inference. The weight of this drift (the *guidance scale*) is critical for the trade-off between data quality and requirement satisfaction. To achieve both of these goals, guided diffusion must resort to small guidance scales and lengthy sampling, incurring high computational costs. This work proposes *learned end-to-end guidance schedules* (LEEGS) to achieve these objectives with *fewer sampling steps*. LEEGS trains a time-dependent schedule by minimizing the guidance function over a small set of examples using stochastic gradient descent. Backpropagating through guided sampling is computationally expensive, so LEEGS uses an approximation of the gradient that cuts training time by a factor of 4. We evaluate LEEGS on diverse guidance tasks, including (a) image inpainting, (b) noisy image inverse problems, (c) face-ID-guided generation, and (d) forward and inverse PDE problems, outperforming baselines at equal budget (50 or 100 NFEs), or matching constant guidance with only 10% of the steps.

Primary area:
generative models
Keywords:
diffusion, guidance, image inverse problems, inpainting, generative models, steering, constrained diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_prNLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LEEGS, a method for learning time-dependent guidance schedules in loss-guided diffusion. By optimizing guidance weights through the sampling trajectory with an efficient gradient approximation, the method improves generation quality across image inverse problems, face-ID guidance, and PDE applications.

Critical strengths
  • Meaningful extension to a broad application domain. Extending learned guidance schedules to loss-guided diffusion is valuable, as this framework supports diverse conditional generation and inverse problems. The method is general and does not require modifying the pretrained diffusion model.

  • Practical and coherent methodology. The proposed framework is straightforward and well-motivated. The trajectory-level objective and efficient gradient approximation make end-to-end schedule optimization computationally feasible.

  • Comprehensive experimental validation. The paper evaluates diverse tasks and sampling budgets, supported by extensive ablation studies. The results generally demonstrate improvements over fixed or heuristic guidance schedules and competitive performance against established methods.

Critical weaknesses
  • Limited methodological novelty. The underlying idea of optimizing time-dependent parameters through differentiable diffusion sampling has been explored in prior work. For example, Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models (CVPR 2023) optimizes per-step text-embedding mixing coefficients, while DRaFT (ICLR 2024) also explores learning per-step guidance weights. Therefore, the main contribution lies in extending an established optimization paradigm to loss-guided diffusion rather than introducing a fundamentally new framework. While this extension is meaningful, its relationship to these earlier works deserves clearer discussion.

  • Insufficient justification for unconstrained per-step schedule optimization. LEEGS independently optimizes the guidance weight at every sampling step, but the experiments primarily compare against fixed or manually parameterized schedules. A comparison with low-dimensional schedules trained using the same optimization procedure would better establish the necessity of full per-step optimization. Moreover, Appendix C.4 shows that matching the overall guidance magnitude substantially reduces performance differences between schedules on some tasks, although task-specific shapes remain important for others. This further motivates investigating whether simpler learnable parameterizations could achieve comparable performance.

Questions for authors

Q1. How does LEEGS compare with low-dimensional learnable schedules, such as polynomial or spline schedules, optimized using the same objective and training budget?

2 comments

We thank the reviewer for their positive assessment, recognising the extension to loss-guided diffusion as meaningful, the methodology as practical, and the experimental validation as comprehensive. We are thankful also for their suggestions on related work and on the parametrisation of the schedule. We address each point below. All changes to the manuscript are highlighted: changes to the text in orange, and new table rows in red. In the list of contents of the appendix, new appendix sections are marked in orange, and existing sections with new content in red. Section, figure, table and line numbers below refer to the revised manuscript.

W1 (relation to Wu et al. and DRaFT). We thank the reviewer for these references, which we have added to the related work (Section 5) together with other methods that learn sampling hyperparameters by differentiating through the sampler. LEEGS does belong to the family of methods that optimize per-step sampler quantities. As the reviewer notes, its contribution is to learn guidance schedules for loss-guided diffusion, a setting that neither mentioned work considers and in which the sampler itself contains the guidance gradient (see our response to Reviewer cPQR, W1a). Wu et al. (2023) learn per-step mixing weights between two text embeddings for text-driven image editing. DRaFT (Clark et al., 2024) fine-tunes model weights on a differentiable reward, and learns per-step weights for classifier-free guidance in an exploratory experiment (their App. B.12). It is therefore also closely related to the works on learned CFG weights already discussed in our related work, and we have added the reference in that section too. Interestingly, DRaFT reports that learning these sampler inputs "did not improve results much at large scale", whereas in loss-guided diffusion a learned schedule beats constant guidance at a matched budget (Tables 1 and 2).

W2/Q1 (low-dimensional learnable schedules). We have run the comparisons requested by the reviewer (new App. H, Table 22-23 and Fig. 28): cubic B-spline schedules with 4, 5, 7 and 11 parameters, trained with the same objective, training images and learning rate as LEEGS on FFHQ box inpainting at 50 NFEs. With 5 or more parameters, the splines reproduce the shape of the learned LEEGS schedule, and the 11-parameter spline is nearly identical to it. In preliminary results on 300 held-out images (disjoint from the training images of all LEEGS draws), the 11-parameter spline reaches LPIPS 0.173 and the 5-parameter spline 0.185, both comparable to LEEGS (0.180 ± 0.004, 3 draws). The 7-parameter spline reaches 0.192, while the 4-parameter spline (a single cubic segment) is clearly worse at 0.305, which we attribute to both a limited modeling capacity, and worse training dynamics (see App. H, "Further analysis"). This shows that LEEGS also works reliably for low-dimensional parametrizations. While using low dimensional schedules does not save computational time (which is dominated by the repeated backpropagation through the diffusion model), they may be useful in cases where prior knowledge is available, or there is a need to enforce smooth schedules. In other settings where more complex schedules arise (such as sampling with renoising; App. D), or in the absence of prior knowledge, per-step parametrization remains preferable. In addition to App. H, we have added a paragraph in Sec. 4.2 summarising this study.

Reviewer_prNLOpenReview ↗

Response

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.

1. Methodological novelty and positioning.

I appreciate the clarification of the relationship between LEEGS and prior work on optimizing sampling parameters. I agree that extending end-to-end guidance schedule learning to loss-guided diffusion is a meaningful contribution, particularly given the additional computational challenges introduced by differentiating through guidance gradients. The revised positioning appropriately clarifies the contribution of this work.

2. Low-dimensional learnable schedules.

The new B-spline experiments directly address my main experimental concern. The results show that low-dimensional parameterizations can achieve performance comparable to full per-step optimization, which is an interesting finding.

I also find the authors' explanation for retaining per-step parameterization convincing. Since the dominant computational cost comes from backpropagation through the diffusion sampler rather than the number of schedule parameters, reducing the parameter count offers limited computational benefits. Meanwhile, per-step optimization provides greater flexibility without requiring assumptions about the schedule shape.

Therefore, I consider the choice of per-step parameterization sufficiently justified, although the results also demonstrate that simpler parameterizations can be effective.

Overall, I am satisfied with the rebuttal. The authors have adequately addressed my primary concerns, and I appreciate the additional experimental evidence and clarification of the paper's contributions.

I will maintain my original rating of 3 (Weak Acceptance).

Reviewer_pfHFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LEEGS to address the imbalance between generation quality and condition satisfaction in loss-guided diffusion models. LEEGS generalizes conventional fixed or manually designed guidance weights into a learnable, timestep-dependent schedule. The schedule is trained end-to-end by minimizing the constraint loss over the entire sampling trajectory. To make this optimization computationally and memory efficient, the authors employ a gradient approximation that ignores second-order terms, together with gradient checkpointing. The experiments cover image inpainting, noisy inverse problems, Face-ID-conditioned generation, and both forward and inverse PDE problems. The results show that, under the same number of function evaluations (NFE), LEEGS generally outperforms fixed and heuristic schedules and improves the trade-off between generation quality and constraint satisfaction.

Critical strengths

1.The paper addresses an important and practically relevant problem. The work focuses on the trade-off between generation quality and constraint satisfaction in loss-guided diffusion. If the guidance is too weak, the generated samples may fail to satisfy the specified constraints; if it is too strong, the samples may deviate from the data distribution and exhibit visible artifacts. This issue arises broadly in image inverse problems, controllable generation, and scientific computing, making the problem both practically meaningful and worthy of investigation. 2.The proposed timestep-dependent guidance-schedule learning method is simple, intuitive, and easy to integrate. LEEGS balances sample quality and constraint satisfaction by adapting the strength of the loss-gradient drift term across diffusion timesteps. Importantly, it does not require retraining the pretrained diffusion model. The overall idea is clearly motivated and can be readily incorporated into existing loss-guided sampling frameworks. 3.The experimental evaluation is relatively comprehensive. The experiments cover four categories of tasks, multiple datasets, and different NFE budgets. The paper also includes extensive comparisons with baseline methods and several ablation studies. LEEGS achieves competitive results in most comparisons under matched computational budgets, providing substantial empirical support for the effectiveness and applicability of the proposed method.

Critical weaknesses

1.The motivation in the abstract is not sufficiently clear or well connected. The transitions between several statements in the abstract feel abrupt. In particular, it is unclear whether the primary objective of the paper is to address the imbalance between generation quality and user-specified constraints, or to reduce the computational cost of learning the guidance schedule. The authors should clarify the central research question and better explain how the computational techniques support, rather than compete with, the main motivation. 2.The pretrained diffusion models used in the experiments are not described clearly enough. It is unclear whether different tasks use diffusion models pretrained on different datasets. If so, the paper should explicitly specify which pretrained model and dataset are used for each task. The authors should also provide links to the corresponding pretrained checkpoints and release the relevant code and configuration files to improve reproducibility. 3.The claimed trade-off between generation quality and constraint satisfaction is not directly validated in most experiments. A central claim of the paper is that LEEGS achieves a better balance between generation quality and constraint satisfaction. However, except for the Face-ID experiment, which reports both FID and an identity-constraint distance, the remaining experiments do not report the final constraint loss (h(x_0,c)). Metrics such as LPIPS and FID primarily measure reconstruction or generation quality and do not directly reveal whether the constraints are better satisfied. I recommend reporting constraint-related metrics in the corresponding tables or presenting two-dimensional quality-versus-constraint scatter plots or Pareto curves. Such results would provide more direct evidence that LEEGS improves the intended trade-off. 4.The evaluation of model-level generalizability is limited. The current experiments appear to rely on a relatively small set of pretrained diffusion models. It therefore remains unclear whether the proposed scheduling strategy generalizes across different generative-model architectures. Cross-model experiments using multiple diffusion backbones or related generative architectures would strengthen the claim of broad applicability. 5.The presentation and organization should be revised to better follow the ICLR formatting conventions. The table captions are currently placed below the tables, and the related-work section appears at the end of the paper. The authors should revise the formatting and organization in accordance with the ICLR guidelines to improve the paper’s rigor, readability, and overall presentation.

Questions for authors

see weakness

3 comments

We thank the reviewer for their careful reading and constructive suggestions on the clarity, reproducibility and evaluation of our work. We respond to each point below. All changes to the manuscript are highlighted: changes to the text in orange, and new table rows in red. In the list of contents of the appendix, new appendix sections are marked in orange, and existing sections with new content in red. Section, figure, table and line numbers below refer to the revised manuscript.

W1 (motivation in the abstract). The central goal of LEEGS is to improve the trade-off between generation quality and user-specified constraints at a small inference budget. To do so, LEEGS learns task-specific guidance schedules. Gradient approximation and rematerialisation serve this goal by making it computationally tractable to learn this schedule (i.e., to backpropagate through the sampler). We have revised the abstract to state this research question first and to present the computational techniques as the means to achieve it; the introduction already stated this aim (l. 53–54).

W2 (pretrained models, code and configurations). Different tasks use different pretrained models and App. B specifies which models are used in the "Models" paragraph of each subsection: the FFHQ model of Chung et al. (2023) and the ImageNet model of Dhariwal & Nichol (2021) for noisy inverse problems (App. B.1); the CelebA-HQ model of Lugmayr et al. (2022) and the ImageNet model from Dhariwal & Nichol (2021) for inpainting (App. B.2); the CelebA-HQ model of Lugmayr et al. (2022) for Face-ID guidance (App. B.3); and the models from Huang et al. (2024) for the PDE problems (App. B.4); the new latent experiment uses Stable Diffusion v1.5 (W4). Notice that checkpoints for these models are all readily available: we do not train any diffusion model. To make this easier to find, we have added a table (new App. B.5, Table 11) that lists, for each task, the pretrained model, its training dataset, resolution and architecture, and a download link. We will release our code and configuration files on GitHub with the camera-ready version.

W3 (quality–constraint trade-off). The reviewer has a point that constraint satisfaction should be reported directly and we have added a new appendix with the corresponding results (App. F: final constraint values in Table 21 and quality-versus-constraint curves in Fig. 25). For the noisy inverse problems, the constraint is the residual h(x0)=∣A(x0)−y∣2h(x_0) = |A(x_0) - y|_2; since yy is noisy, even the ground truth has the residual approximately equal to the noise level, and lower values would mean fitting the noise. This is why our main tables report LPIPS against the ground truth. We now report the final h(x0)h(x_0) of LEEGS and the linear baselines on all noisy inverse tasks: at 50 and 100 NFEs. LEEGS is within 20% of the noise level on every task except super-resolution. We also show LPIPS-versus-hh curves (Fig. 25) on three FFHQ tasks (evaluated on 300 test images), sweeping the scale of constant and linear-increasing guidance. At 10 NFEs, LEEGS dominates both families (e.g. box inpainting: LPIPS 0.324 vs. 0.502); at 50 NFEs, it lies on or slightly beyond the linear-increasing frontier, while constant guidance never comes within 2.6× of the noise level. On super-resolution, LEEGS and linear-increasing guidance fit the noise (hh below the noise level), consistent with its susceptibility to reward hacking (Fig. 2). We will add the constraint values of the remaining baselines in the camera-ready version. In addition to App. F, we have added a paragraph in Sec. 4.2 summarising this study.

W4 (model-level generalisability). We agree that our original experiments use pixel-space U-Net models only. We have added an experiment on Stable Diffusion v1.5, a latent text-to-image model, which differs both in architecture (a VAE with a text-conditioned latent U-Net) and in the space where sampling happens (new App. G, Figs. 26–27; see also our response to Reviewer cPQR, W6). On a style-transfer task, where the guidance loss compares CLIP Gram matrices of the decoded sample and of a reference image, we train LEEGS exactly as in pixel space, backpropagating the guidance loss through the VAE decoder and the latent sampler. On 256 held-out samples, the learned schedule improves the style score over constant guidance at an equal CLIP score (2.45 vs. 3.08). While these results are preliminary (a single task and training run), they indicate that LEEGS is not tied to a particular backbone: it only requires a differentiable sampler and guidance loss. In addition to App. G, we have added a paragraph in Sec. 4.2 summarising this study.

W5 (formatting). We thank the reviewer for pointing this out. We have moved all table captions above the tables, as the ICLR style guide requires, and rechecked the remaining formatting conventions. As for the related work section, we chose to place it at the end of the paper to be able to compare works once the objective and gradient approximation of LEEGS are formalized. We find this makes for a clearer comparison to, e.g., learned CFG weights and methods that differentiate through the sampler. If the reviewer sees a better way of introducing these concepts, we would be glad to consider their suggestions.

Reviewer_pfHFOpenReview ↗

The authors have addressed my concerns.

Thank you for the detailed response and substantial revisions. The authors have adequately addressed my main concerns by clarifying the motivation, providing additional model and reproducibility information, adding an analysis of the quality–constraint trade-off, and including experiments on Stable Diffusion v1.5. The revised paper is clearer, more complete, and more convincing; therefore, I support acceptance.

Reviewer_cPQROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LEEGS, a method that learns a non-negative guidance scale for each sampling step in loss-guided (DPS-style) diffusion. It does this by backpropagating a guidance loss, summed over the whole trajectory, through the unrolled sampler. A stop-gradient on the guidance gradient removes the second-order terms, and gradient checkpointing keeps memory bounded. Experiments cover noisy inverse problems on FFHQ and ImageNet, noiseless inpainting, Face-ID guidance and DiffusionPDE problems, at 50 to 200 NFEs.

Critical strengths

1.The method is simple, works with any sampler and adds no inference overhead.

Critical weaknesses
  1. Limited novelty. Prior work already tunes sampling parameters by differentiating through unrolled samplers (Watson et al., 2022; Tong et al., 2025; Tang et al., 2024). LEEGS applies this idea to S scalars with a standard first-order truncation.
    • The claim to be the first such method for loss-guided diffusion is weakened by Galashov et al. (2025), who explicitly extend learned guidance weights to reward-tilted targets.
    • The statement in Section 5 that these methods "do not use rewards" is therefore inaccurate.
  2. Missing baselines. Several peer-reviewed solvers are absent: DiffPIR [1], ΠGDM [2], RED-diff [3], ReSample [4], and DDNM, which is cited but not compared. The most direct competitor is TFG [5], which already searches time-dependent guidance hyperparameters per task. FreeDoM and MPGD [6] are natural Face-ID baselines.
  3. Unequal tuning budgets. LEEGS uses 1.5 to 5 GPU hours per subtask and budget. The baselines only get a one-parameter grid search on 20 images. A comparably expensive search over parametric schedules is needed. In addition, the DPS reference rows come from a different test split.
  4. Objective design. Minimizing ||A(x) − y||² on noisy measurements rewards fitting the noise. The resulting reward hacking is handled with task-specific weight decay, but the protocol for choosing it is unclear. The intermediate terms h(x̂₀|s) are a heuristic proxy, and the bias they introduce is not analyzed.
  5. Gradient approximation barely tested. It is checked against the true gradient only at S = 10 on motion deblur. Its bias at large S, or with strongly nonlinear guidance functions such as ArcFace, is not studied.
  6. Narrow scope. Only small pixel-space models are evaluated. Latent and text-to-image models, where sampling cost matters most, are not tested.

[1] Zhu et al., Denoising Diffusion Models for Plug-and-Play Image Restoration, CVPRW 2023. [2] Song et al., Pseudoinverse-Guided Diffusion Models for Inverse Problems, ICLR 2023. [3] Mardani et al., A Variational Perspective on Solving Inverse Problems with Diffusion Models, ICLR 2024. [4] Song et al., Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency, ICLR 2024. [5] Ye et al., TFG: Unified Training-Free Guidance for Diffusion Models, NeurIPS 2024. [6] He et al., Manifold Preserving Guided Diffusion, ICLR 2024.

Questions for authors

See weaknesses.

8 comments

We thank the reviewer for their insightful comments. We address each of their concerns below point by point. In addition, we have updated parts of our manuscript with the changes. Changes in the body of the text are marked in orange, and new rows in the tables are marked in red. In the list of contents of the appendix, new and modified appendix sections are marked in orange and red. Section, figure, line numbers etc below refer to the revised manuscript.

W1a (methods that differentiate through the sampler).

As we mentioned in Sec. 1 (l. 99-101), LEEGS indeed shares the computational technique of differentiating through the sampling trajectory with Watson et al. (2022), Tong et al. (2025), and Tang et al. (2024), however, with a different goal. Watson et al. (2022) and Tong et al. (2025) seek to learn global sampling parameters to improve the quality of conditional and unconditional generation. On the other hand, Tang et al. aim to learn DDPM noise vectors per sample for conditional generation to optimize an image reward. LEEGS learns global guidance weights for loss-guided diffusion, which biases the sampling with the guidance gradient gsg_s, which is not considered by the above works. Due to the different goals, all four works also have different optimization objectives. Watson et al. optimize KID of the samples against real data, Tong et al. match the outputs of a teacher ODE solver at fewer steps, and Tang et al. (2024) optimize rewards evaluated at the final sample. In contrast, LEEGS optimizes the same loss functions which guide the diffusion model (with their gradient), summed along the entire trajectory.

Moreover, since loss-guided sampling involves the guidance gradient gsg_s, differentiating through the sampling trajectory involves expensive second derivatives of the diffusion model. This challenge does not appear in the other three works. LEEGS solves it by dropping the second derivative term of the sampling step, making training roughly 4× cheaper than with the true gradient (App. C.2). To make these distinctions explicit, we have added a paragraph to the related work (Sec. 5) on methods that differentiate through the sampling trajectory.

W1b/W1c (Galashov et al., 2025). Our setting is loss-guided diffusion, which biases an unconditional (or conditional) diffusion process by adding the gradient of an off-the-shelf loss function to the output of the diffusion model. Galashov et al. (2025) instead learn a model that predicts classifier-free guidance (CFG) weights, which combines the conditional and unconditional outputs of the same diffusion model. These are two distinct problems. In particular, loss-guided diffusion has much broader applicability since it allows adapting pretrained diffusion models to novel tasks, while conditional diffusion models need to be trained from scratch. Thus, to the best of our knowledge, our claim that LEEGS is the first to learn guidance schedules for loss-guided diffusion still stands.

However, while the reviewer is right that our statement that they "do not use rewards and instead rely on labeled data" is inaccurate, the underlying argument still holds. While Galashov et al. do use a reward (a CLIP score), this is only used in conjunction with a self-consistency loss that does require clean data labeled with the condition (e.g. a class label for class-conditional generation). In contrast, we only use guidance losses (sometimes also called rewards) in our objective which require only the noisy degraded measurements that are the input to the guided sampling algorithm. However, we have removed the phrase “do not use rewards” from Section 5 and edited Section 5 to improve the clarity of the comparison with Galashov et al.

W2a (TFG). We thank the reviewer for pointing us to TFG (Ye et al., 2024), which we have added to the related work (Sec. 5) and as a baseline (W2b). However, we disagree that TFG "already searches time-dependent guidance hyperparameters per task". TFG unifies several training-free guidance methods in one sampler and searches three scalar hyperparameters per task. Two of them are made time-dependent by one of three fixed shapes, two of which are derived from the noise schedule ("increase" ∝αt\propto \alpha_t, "decrease" ∝1−αt\propto 1-\alpha_t, or "constant"; their Eq. 8). It is worth noting that in their experiments the shape is set to "increase" for all tasks (their Sec. 5.1). TFG's schedules are therefore of the same kind as our constant and linear baselines (with three hyperparameters instead of one), in contrast to LEEGS that learns one weight per sampling step. This matters because good schedules are task-specific. On the PDE tasks, the learned schedules apply most guidance early in sampling, the opposite of TFG's "increase" (Fig. 3; Table 5), and even among the FFHQ tasks each learned schedule is nearly always best on its own task (App. C.4).

W2b (missing baselines). Our image inverse benchmark already compares LEEGS against three recent inverse-problem solvers (DAPS, CDIM and DSG), in addition to the three heuristic schedules (constant, linear increase and linear decrease), along with the 1000-NFE DPS reference. Following the reviewer's suggestion, we have added TFG, the baseline most relevant to our setting, on FFHQ and ImageNet at 50 and 100 NFEs: it is the closest method to ours, since it tunes the magnitudes of a fixed-shape guidance schedule (W2a). We tune it with the authors' beam search, followed by a local refinement of the best configuration until it lies inside the searched range (App. B.1). We have added these results to Tables 1 and 2 of the revised manuscript; the rows for TFG and LEEGS are extracted below (LPIPS / FID, lower is better; better of the two in bold).

FFHQ

MethodNFEsSR ×4Box inpaintingRandom inpaintingGaussian deblurMotion deblur
TFG500.257 / 39.270.205 / 30.720.272 / 37.980.248 / 37.610.217 / 31.91
LEEGS (ours)500.222 / 35.520.175 / 33.110.197 / 33.630.243 / 42.550.211 / 39.24
TFG1000.234 / 32.420.173 / 25.690.214 / 29.100.234 / 33.680.199 / 27.21
LEEGS (ours)1000.217 / 32.400.141 / 25.390.176 / 26.440.228 / 34.050.181 / 30.29

ImageNet

MethodNFEsSR ×4Box inpaintingRandom inpaintingGaussian deblurMotion deblur
TFG500.383 / 52.050.255 / 47.070.374 / 52.120.366 / 50.550.325 / 41.70
LEEGS (ours)500.333 / 46.860.286 / 55.070.272 / 31.540.395 / 58.870.338 / 50.76
TFG1000.347 / 43.790.226 / 40.890.303 / 35.920.353 / 47.580.298 / 34.17
LEEGS (ours)1000.319 / 39.620.234 / 47.730.235 / 23.690.368 / 50.380.310 / 42.33

TFG is a strong baseline, which is expected, since it combines several guidance mechanisms whose strengths are tuned per task. LEEGS nevertheless remains the strongest method overall: across the four full tables (FFHQ and ImageNet, 50 and 100 NFEs, all baselines), it has the best LPIPS in 13 of the 20 columns and a top-three LPIPS in all 20, more than any other method (next: TFG with 3 best and 15 top-three), and it also leads when FID is included (20 first and 39 top-three places out of 40).

W3a (tuning budget for baselines). While training LEEGS costs several GPU-hours per task, LEEGS learns one guidance weight per sampling step, i.e., up to 200 parameters in our experiments (App. B.6, Table 12). In comparison, for Imagenet at 100 NFEs, TFG's search over 3 hyperparameters (see W2a) using a combination of beam search and hill climbing heuristics (5.9–7.4 A40 GPU-hours per task) costs about as much as training LEEGS (8.6 A40 GPU-hours per task, or 4.7 A100 GPU-hours as already reported in Table 12). What is more, it is restricted to a fixed grid decided a priori. Thus, while the tuning budget of the new baseline (TFG) is comparable to LEEGS (as the reviewer requested), LEEGS remains more effective.

Meanwhile, methods which depend on a single hyperparameter (e.g., the constant and heuristic schedules, or DSG) are better suited to a grid search. Our current procedure selects it on 20 validation images, and expands the grid whenever the optimum lies at its boundary (App. B.1). We also report the full search curves (App. K, Figs. 35–42), so the quality of the tuning can be checked directly. We believe this protocol is sufficient to find good guidance scales and don’t think that additional compute would lead to substantially better results. That being said, if the reviewer has a protocol in mind that they believe would enable substantial performance improvements (e.g., using more images or coarse-to-fine search), we are happy to consider it in the experiments.

W3b (test split for the DPS rows). As stated in App. B.1 (l. 1005–1012 and l. 1050), the DPS rows are taken from Chung et al. (2023), and our test sets follow theirs as closely as possible. On FFHQ, our test set consists of 950 of their 1000 test images; the other 50 are used to train and validate LEEGS and the baselines. On ImageNet, the DPS test split is not published, so we sampled our own test set of 1000 images from the ILSVRC 2012 validation set, one image per class. Re-running the DPS results at S=1000 would take a considerable amount of time (10x the sampling time of the S=100 rows), not including time taken to tune the hyperparameters (the paper's and the codebase's values differ). Thus we chose to report the results from the original paper, which were tuned by the authors themselves. The difference in test images is unlikely to affect the comparison with LEEGS: with about 1000 test images, the standard error of the mean LPIPS for LEEGS is about 0.001 on FFHQ and 0.002–0.003 on ImageNet. To test this directly, we evaluated LEEGS at 100 NFEs on a second, disjoint ImageNet test set of 1000 images (one per class) for super-resolution, the subtask where LEEGS at S=100 has the smallest advantage over DPS at S=1000. With the same trained schedule (draw 1 from Table 2), the LPIPS/FID is 0.319/39.17 on the new set versus 0.320/39.49 on the original set. This difference is far smaller than the gap to DPS at 1000 NFEs which scores 0.337/50.66.

W4a (reward-hacking prevention protocol). We address the reward hacking the reviewer describes in Section 3 ("Regularization", l. 203–211), Fig. 2 and App. C.3 (Fig. 12). When ground truth is available, as in our image tasks, it is easy to detect: the LPIPS of the training examples shows a clear elbow, after which the guidance loss keeps decreasing but reconstruction gets worse (Fig. 2). We prevent it with weight decay on the schedule, γ∣W∣22\gamma|W|_2^2. We choose γ\gamma by inspecting this training curve and increasing γ\gamma until the elbow disappears. App. C.3 shows this effect and, as an ablation, how held-out performance depends on γ\gamma across step budgets SS. We now describe this protocol explicitly in App. C.3 ("Protocol for choosing γ\gamma").

W4b (intermediate terms h(x^0∣s)h(\hat x_0|s)). While we agree that the inclusion of the intermediate terms are a heuristic, we have already studied their effect in App. C.6, which compares our objective with the terminal-only objective on FFHQ box inpainting for SS from 10 to 100. Our study shows that intermediate terms improve held-out LPIPS, our metric of interest, at every SS, and the gap widens with SS. The training LPIPS curves show the same: terminal-only training is consistently worse, with a gap that widens with SS. Both observations support the motivation we provided in Section 3 (l. 151–154), namely that a terminal-only objective leaves long trajectories with little training signal.

To further improve the comparison, and analyse the “bias” of the intermediate terms, we have now included a comparison of the learned guidance schedules in Appendix C.6 (Figure 18). At S=10, both schedules (terminal-only and full trajectory) are comparable (but not exactly the same). However, the correspondence for the overall schedule sharply degrades from S=20, while the guidance values learned at the last step have similar values. This further reinforces the idea that the problem with the terminal-only objective is propagating the learning signal back to earlier steps. Thus the addition of intermediate terms is a necessary heuristic.

Should our reply not fully address W4b, we would appreciate it if the reviewer could clarify which kind of "bias" they have in mind, so that we can analyse it specifically.

W5 (gradient approximation). In addition to the comparison at S=10S=10 in App. C.2, we have run a new comparison of the LEEGS gradient with the true gradient at S=50S=50 on FFHQ noisy motion deblurring, with 3 training draws. On the 950 test images, the true gradient reaches LPIPS 0.227 ± 0.013 (FID 39.0–40.5), against 0.211 ± 0.002 (FID 38.7–39.8) for LEEGS (mean ± std over 3 training draws), while costing about 6× more per training iteration (73 s vs 12 s on an A40). Even though we trained it with a lower learning rate, the true gradient also learns much noisier schedules than LEEGS, although they follow the same general shape (App. C.2). The approximation therefore does not cost accuracy at S=50S=50, while being substantially cheaper. We have added this comparison to App. C.2 (Table 14 and Fig. 11). Moreover, our choice is validated by the good performance of the learned schedules in our experiments (Tables 1-5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.