acceptodds
Withdrawn from ICLR 2027

Residual-Guided Sampling: Structured Acceleration and Timestep Scheduling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient sampling from diffusion and flow models depends jointly on numerical updates and timestep schedules, yet these components are often designed separately. We introduce residual-guided sampling to unify them through a structural residual. For smooth linear data-noise interpolants, pathwise forward acceleration lies in the state-velocity span under a nondegeneracy condition. For the variance-preserving cosine interpolant, augmenting Euler with a second-order acceleration term whose state and velocity coefficients are derived from the forward interpolant recovers deterministic DDIM through second order. However, reverse-trajectory acceleration need not lie in this span, so we approximate it as a weighted sum of state and velocity. With the pretrained velocity field fixed, a lightweight head predicts the two coefficients from time, conditioning, and guidance scale. The resulting acceleration mismatch defines a computable residual. Our analysis shows that it characterizes the leading discretization defect and enters a global error bound; it also guides coefficient fitting and a propagation-aware timestep schedule. The sampler uses the predicted acceleration without extra velocity-field evaluations, while the schedule transfers to other integrators. A single head supports unconditional and conditional sampling with or without classifier-free guidance. Across tested settings, our method improves upon matched Euler, Heun, AB2, and RF-Solver baselines, with the largest gains at low-to-intermediate evaluation budgets.

Primary area:
generative models
Keywords:
Residual-guided sampling, structured acceleration, timestep scheduling, diffusion models, flow models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_nx38OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a residual-guided framework for accelerating diffusion and flow sampling. It approximates the frozen velocity field’s acceleration using a learned linear combination of the current state and velocity, and uses the approximation residual to guide timestep selection through a propagation-weighted minimax objective. The resulting schedules can also be transferred to existing solvers. Experiments on CIFAR-10 and ImageNet-256 show improvements over uniform-grid baselines, particularly at low evaluation budgets, but the lack of comparisons with directly relevant methods such as LD3 and AMED leaves the incremental benefits over existing learned scheduling and solver approaches insufficiently established.

Critical strengths
  1. A coherent and computationally lightweight sampling construction. The paper connects acceleration approximation, local discretization error, and timestep allocation through a common residual. Learning only two scalar coefficients keeps the base model frozen and introduces no additional velocity-field evaluations during sampling. The distinction between the exact forward-interpolant identity and the approximate reverse-flow representation is stated clearly.
  2. Useful empirical improvements with informative component analysis. The CIFAR-10 experiments separate several effects of the correction and scheduling, and the transfer to existing solvers is practically relevant. For example, on ImageNet-256 without CFG at NFE 16, AB2-DPA improves FID from 11.4 to 9.46. At guidance 1.375 and NFE 30, it achieves FID 2.17 versus the reported DSS result of 2.24. The paper also reports recall, residual diagnostics, parameter sensitivity, and substantial implementation details, while acknowledging settings where improvements diminish or reverse.
Critical weaknesses
  1. Missing comparisons with directly relevant learned sampling methods. The paper learns acceleration coefficients and optimizes timestep placement, but does not compare with LD3 or AMED, despite citing both. LD3 directly targets learned timestep scheduling, while AMED and subsequent methods such as Consistency Solver and AdaSDE have demonstrated strong few-step sampling performance. Without comparisons against these relevant baselines, the practical advantage of the proposed framework remains unclear.
  2. The incremental benefit of the learned correction is not sufficiently established relative to its offline cost. The competitive AB2-DPA results do not require the learned coefficient head. On ImageNet-256, Table 2 omits the SAS-U, SAS-DPA, EUL-DPA, and HEU-DPA controls needed to separate the head’s contribution from scheduling. The CIFAR-10 results also show mixed benefits: for 2-RF at NFE 4, SAS-U worsens FID from 4.63 to 9.02, although recall improves. Appendix A.10 transparently reports approximately 185 GPU-hours for target collection, 4 hours for head training, and 160 GPU-hours for separate profiling in the SiT setup. However, it remains unclear when the full learned pipeline offers a favorable quality–cost tradeoff over the simpler scheduling-only variants.
Questions for authors

See weaknesses, especially weakness 1.

Reviewer_2EszOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes residual-guided sampling for a fixed learned velocity field. The method approximates the field's acceleration as a linear combination of the current state and velocity, with two coefficients predicted by a lightweight head conditioned on time, conditioning information, and guidance scale, and adds this approximation as a second-order correction to the Euler update at one network evaluation per step. The resulting approximation error (the structural residual) serves both as the training objective for the head and as the cost of a dynamic-programming timestep schedule, and it enters a global error bound. Experiments on CIFAR-10 (rectified-flow) and ImageNet-256 (SiT) apply the correction and the schedule to the Euler and Heun's predictor and the schedule alone to AB2 and RF-Solver, reporting gains at low NFE and comparing with DSS.

Critical strengths
  • Trajectory-free fitting and analysis. Bespoke-style solvers are trained on ground-truth trajectories. Here the local defect is instead estimated from forward-interpolant samples via JVPs, so for w∈{0,1}w \in \{0,1\} the head-dependent term of the error bound can be computed without simulating reverse trajectories. The derivation in A.4 appears correct.
  • Reusable, low-overhead coefficients. The coefficients are continuous in tt and conditioned on (s,w)(s, w), so a single head serves all NFE budgets and the trained guidance scales without refitting. The correction adds no network evaluations at sampling time, and CFG targets for any guidance scale are assembled from four branch-level JVPs computed once offline.
Critical weaknesses

W1. Limited novelty relative to Bespoke solvers

  • For a fixed (s,w)(s,w) and a fixed grid, Eq. (8) is a per-step affine update yn=(1+Δn22Aϕ) yn+1−Δn(1−Δn2Bϕ) vθy_n = (1 + \tfrac{\Delta_n^2}{2}A_\phi)\,y_{n+1} - \Delta_n(1 - \tfrac{\Delta_n}{2}B_\phi)\,v_\theta. Updates of this form are a subset of the Bespoke RK1 family (Shaul et al., 2024a, Eq. 17), which also learns the time grid. BNS (Shaul et al., 2024b, Thm. 3.2) subsumes that family.
  • The error analysis follows the same structure as Bespoke’s training objective (Shaul et al., 2024a, Eqs. 24–27). Bespoke bounds the global error by one-step local defects weighted by products of later-step Lipschitz constants, and also treats the Lipschitz constant as a hyperparameter. This parallels Jθ\mathcal{J}_\theta in Theorem 2 and the Γ\Gamma-weighted DP cost with a tuned LL.
  • The genuine differences are that the local defect is estimated from forward-interpolant samples without ground-truth trajectories, and that the coefficients are continuous in tt and conditioned on (s,w)(s,w), so they need not be re-fitted for each NFE. Sec. 3.6 (l.285–288) mentions only that Bespoke uses trajectory-level objectives; it does not discuss the shared update form or bound structure. The contribution therefore rests on these differences, whose benefit over Bespoke/BNS is not tested empirically (W3).

W2. Estimated quantities do not match the theory in key settings

  • For CFG (w∉{0,1}w \notin \{0,1\}), Jθ,w\mathcal{J}_{\theta,w} is defined with expectations over the guided-flow marginals (Sec. 3.4). The fitting and the profiles are instead computed on forward-interpolant marginals as a proxy (l.231–234), so the scheduling objective is not an estimate of the quantity bounded in Theorem 2.
  • Many of the strong results (e.g., AB2-DPA in Tables 1–3) come from transferring the Euler-residual schedule to second-order solvers, whose leading local error is not a Δ2\Delta^2-weighted acceleration.
  • Both limitations are acknowledged in A.10, but they mean that the error analysis covers neither the CFG setting nor the transferred schedules.

W3. Experimental evidence is limited relative to the claims

  • Schedule comparison is only against uniform grids (Tables 1–3, apart from the reported DSS results), while LL is selected by FID sweeps: per NFE on CIFAR-10 (A.9.2) and per sampler, guidance scale, and NFE on ImageNet (Table 5). Comparisons with established non-uniform schedules (e.g., EDM ρ\rho, time shifting, AYS/GITS/LD3), tuned under the same protocol, are needed.
  • The strongest results come from the proposed schedule rather than the structured-acceleration update.
    • AB2-DPA, i.e., the schedule applied to an unmodified AB2, is among the best variants in Tables 1–3.
    • The structured-acceleration update is often weaker than AB2 at the same NFE, even though both use one evaluation per step. On ImageNet it trails even AB2-U (15.9 vs. 11.4 at w=1w=1, NFE 16; 3.06 vs. 2.24 at w=1.5w=1.5, NFE 32), and on CIFAR-10 (1-RF) AB2-DPA beats SAS-DPR at every budget.
    • SAS is not included in Table 3.
    • The evidence therefore supports the schedule, and only against uniform grids, rather than the structured acceleration that is the paper’s main component.
  • Improvements are marginal beyond the lowest budgets.
    • On ImageNet at w=1w=1, NFE 64, gains are at most 0.11 FID (Heun 9.76 → 9.65; AB2 9.54 → 9.47; RF-Solver 9.60 → 9.49), and SAS-DPR is worse than EUL-U (10.7 vs. 10.5).
    • At w=1.5w=1.5, NFE 128, no proposed variant improves on its uniform baseline: each is tied or 0.01–0.04 FID worse.
    • Without variance estimates, it is unclear whether these differences are meaningful.
  • Gains are inconsistent, and the residual does not predict them.
    • SAS-U is worse than Euler on 2-RF at NFE 4–32 (e.g., 9.02 vs. 4.63 at NFE 4, with higher recall) and on 3-RF at NFE 4–16 (Table 4).
    • In the Heun block, all four proposed variants are worse than HEU-U on 3-RF at NFE 4 and on 2-RF at NFE 64 (Table 4).
    • On 1-RF, the matched SAS-DPR is worse than SAS-U at NFE 8 and 16 (9.07 vs. 7.88; 5.07 vs. 4.68).
    • The authors acknowledge that a smaller residual does not guarantee better FID (A.10), so the method offers no way to tell when the correction or the matched schedule will help.
  • Missing solver baselines.
    • Learned solvers in the same category (Bespoke/BNS, LD3, PAS, AMED) are not compared, although they are the natural reference for a method that fits solver coefficients offline.
    • Stronger training-free multistep solvers beyond AB2 (e.g., DPM-Solver++, UniPC) are also absent. Since AB2 already outperforms SAS on ImageNet, these comparisons are needed to justify the offline cost (roughly 345 GPU-hours on ImageNet, A.10).
Questions for authors

(W2, W3) How do DPA/DPR compare with established non-uniform schedules (e.g., EDM ρ\rho, time shifting, and at least one of AYS/GITS/LD3), with each baseline tuned under the same FID-sweep protocol as LL? For w∉{0,1}w \notin \{0,1\}, does the schedule change when the profile is estimated on actual guided trajectories instead of the forward-interpolant proxy?

(W1, W3) Why does SAS underperform AB2-U on ImageNet, and why does SAS-DPR underperform AB2-DPA at every 1-RF budget? Why is SAS omitted from Table 3? How does SAS compare with Bespoke/BNS and with stronger training-free solvers (e.g., DPM-Solver++, UniPC) at matched NFE?

(W3) Could the authors report the mean and standard deviation over multiple sampling seeds for the main tables? Were LL selection and the final evaluation run on different noise seeds?

Reviewer_nwzQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces residual-guided sampling in order to join numerical updates and timestep schedules into a single framework. Their contributions are are as follows. They give a theoretical and empirical framework that adopts and controls the residual term, that is dependent on neural networks. Also, they arrange the timestep schedules that help reduce the error of the numerical updates. Their approach improves Euler, Heun, and some other numerical methods in their experiments done on CIFAR-10 and ImageNet-256. Their experimental domains work as testbeds of their methods.

Critical strengths

Controlling the residual term by parameterizing some variables is intuitively understandable. The way that they develop their theory seems reasonable.

Critical weaknesses

The main weakness is their experiment. At a glance, one might say that their experiment deals with many baselines. However, they should have compared with other methods that construct the numerical update and timestep together. Moreover, experiments dealt with only toy testbeds like CIFAR-10 and ImageNet-256 are not enough to show the practicality of this method, considering the image generation trend nowadays. Moreover, the most critical issue of this paper is that there can be many other approaches that can be alternatively selected to substitute the proposed method. If someone asked me "Would you choose this paper's method to update your diffusion model?", then I would say that I would find better approaches instead, because I believe there are plenty.

Questions for authors

Would you expose the competitor methods of the proposed method frankly, and show me why I should choose this paper's method instead of them through experiments on larger image sets?

Reviewer_PyrZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a residual-guided sampler for diffusion and flow models, combining structured acceleration correction with time-step scheduling. With the pretrained velocity field frozen, a lightweight head predicts two scalar coefficients to approximate acceleration as a linear combination of the state and velocity, using offline JVP supervision. The remaining residual defines a minimax scheduling objective that accounts for error propagation, optimized through dynamic programming.

The paper provides local and global error analyses, connects the VP-cosine special case to DDIM, and evaluates the method on CIFAR-10 and ImageNet-256, including schedule transfer to Euler, Heun, AB2, and RF-Solver. Gains are concentrated at low to moderate NFE budgets.

Critical strengths

1、The method exploits higher-order information with little inference overhead. Approximating acceleration with two scalar coefficients requires no changes to the pretrained model or additional velocity field evaluations during sampling. The analysis links the approximation residual to local discretization error and establishes a global error bound, providing a clear rationale for the structured update.

2、Several improvements are substantial. On CIFAR-10 with 1-RF, SAS-U reduces FID from 16.6 to 7.88 relative to Euler-U at NFE = 8. On ImageNet-256 at (w = 1) and NFE = 16, AB2-DPA reduces FID from 11.4 to 9.46 relative to AB2-U (Tables 1 and 2). These gains span image-space U-Net and latent-space SiT models. The CIFAR-10 ablations and joint reporting of FID and recall also help clarify the contributions and trade-offs of correction and scheduling.

Critical weaknesses

1、The ImageNet experiments do not sufficiently establish the added value of learning an acceleration correction and scheduling from its remaining residual. The most competitive results come from AB2-DPA, which uses acceleration statistics for scheduling without a learned prediction head. Missing Euler-DPA, SAS-U, and SAS-DPA comparisons make it difficult to separate the contributions of correction and residual-based scheduling. Moreover, SAS-DPR yields worse FID than SAS-U on CIFAR-10 with 1-RF at NFE = 8 and 16 (Table 1). The evidence thus supports the scheduling component more clearly than the incremental value of the unified framework.

2、The advantage over existing methods for optimizing time steps remains unclear. Most experiments compare against uniform grids, without closely related baselines such as SharpEuler. The comparison with reported DSS results is mixed: AB2-DPA yields worse FID at NFE budgets of 12–20 and slightly better FID at NFE = 30, while achieving higher IS at all matched budgets (Table 3). These results support the effectiveness of scheduling but leave the additional benefit of the minimax objective unresolved. Since scheduling accounts for several of the paper’s strongest results, this gap matters for assessing the contribution.

Questions for authors

1、Appendix A.9.1 states that CIFAR-10 FID uses 1,024-dimensional Inception-v1 features. Is this a typographical error or a custom evaluation protocol? Please provide the evaluator configuration, identify which tables and figures use it, and clarify whether the results are comparable to standard FID scores.

2、Could you provide Euler-DPA, SAS-U, and SAS-DPA results using the same ImageNet checkpoint, guidance strengths, and NFE budgets? These would clarify whether learned correction and its corresponding residual-based schedule improve upon scheduling based solely on acceleration statistics. Also, why was SAS-DPR omitted from the DSS comparison?

3、How does the proposed schedule compare with SharpEuler or other directly relevant methods for optimizing the time grid, using the same checkpoint and total NFE budget? Comparable offline calibration and hyperparameter search budgets would help assess the contribution of the scheduling objective itself.

4、Under classifier-free guidance (CFG), the paper uses forward-interpolated samples to approximate the marginal distributions of the guided flow. In a representative setting, do the estimated residual curves agree with those measured along actual guided trajectories? Does the resulting schedule reduce endpoint numerical error relative to a sufficiently accurate reference solution? This would help establish whether the residual criterion reflects the errors encountered during actual sampling.

Reviewer_sz2LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes residual-guided sampling for diffusion and flow models. With the pretrained velocity field frozen, a small head predicts two scalar coefficients that approximate acceleration as a linear combination of the current state and velocity. The remaining acceleration residual guides both head fitting and propagation-aware timestep allocation. The correction requires no additional velocity-field evaluations during sampling. The paper provides local and global error analyses and evaluates the sampler and transferable schedules on CIFAR-10 and ImageNet-256, with the clearest gains at low-to-intermediate NFE.

Critical strengths
  1. The paper connects acceleration approximation and timestep allocation through a shared residual. The local defect analysis gives a clear reason to study this quantity, and the two-coefficient parameterization keeps the sampling overhead small while leaving the pretrained model unchanged.
  2. The experiments show useful improvements across several solver families. The CIFAR-10 ablations separate the effects of the update and schedule, and the ImageNet results support transfer beyond the smaller image-space models. For example, AB2-DPA improves FID from 11.4 to 9.46 at NFE 16 without CFG. The paper also reports cases where gains diminish or reverse, which helps establish the practical scope.
Critical weaknesses
  1. The scheduling contribution needs stronger comparisons and ablations. Table 3 includes reported DSS results, but there are no direct comparisons with AYS, GITS, or SharpEuler, which are discussed in Section 3.6. The advantage over existing nonuniform schedules is therefore unclear. The head improves sampling on CIFAR-10, but its added value on ImageNet is harder to isolate without the SAS-U/DPA/DPR ablation. AB2-DPA, which uses no head, also achieves lower FID than SAS-DPR at the low-budget settings in Table 2: 9.46 versus 15.9 without CFG, and 2.15 versus 3.06 with guidance 1.5. These results leave the settings that justify the extra head-fitting work insufficiently established.
  2. The scheduling objective lacks direct validation against global integration error. Theorem 2 motivates the propagation-weighted residual objective, and Appendix A.9.4 studies the propagation parameter. However, the experiments do not show whether a lower objective leads to a final state closer to high-accuracy integration of the same frozen field. Comparing grids with a fixed sampler, head, and paired initial noise would strengthen the numerical explanation beyond residual profiles and generation metrics.
Questions for authors
  1. Beyond the reported DSS comparison, have you compared DPA/DPR with existing fixed-grid optimization methods under matched checkpoints, guidance, and NFE, keeping the solver fixed where applicable? Such results would help assess the scheduling contribution beyond improvement over uniform grids.
  2. Can you provide the SAS-U, SAS-DPA, and SAS-DPR comparison on ImageNet, and clarify the settings in which the learned correction is preferable to using AB2-DPA? This would help establish its independent benefit relative to the extra offline preparation.
  3. Have you measured paired endpoint error against high-accuracy integration of the frozen model across different grids? Does a lower scheduling objective correspond to lower global integration error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.