acceptodds
Withdrawn from ICLR 2027

TraceSampler: Making Sampling Errors Predictable and Correctable for Accelerated Generative Sampling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurate sampling from continuous-time generative models can require many model evaluations. Low-step sampling reduces this cost, but coarse updates introduce endpoint errors that can accumulate along the sampling trajectory and degrade generation quality. We introduce TraceSampler, a post-training method that corrects these errors without modifying the pretrained model. Rather than directly optimising the low-step update itself, TraceSampler models its endpoint error as a sample-specific prediction problem. A higher-step reference integration over the same interval provides the supervised target, while a compact representation makes the high-dimensional error predictable from the current sampling trace. This representation combines shared residual structure, directions derived from the trace, and a low-rank remainder, so shallow per-step MLP heads need only predict sample-dependent coefficients. These coefficients drive a two-stage learned residual correction: the first stage predicts the full endpoint residual, while a backbone evaluation at the intermediate state supplies new trace information for predicting the residual that remains; this evaluation is reused by the following step. TraceSampler is evaluated with Rectified Flow on CIFAR-10, SiT-XL/2 on ImageNet-256, and SD3.5 Medium for text-to-image generation. At 10 steps, it reduces trajectory MSE relative to matched-step Euler by 90.5% on CIFAR-10 and 85.6% on ImageNet-256, while achieving FID scores of 2.692 and 2.189, close to 120-step Heun references of 2.601 and 2.056 with 24 fewer backbone evaluations. On SD3.5 Medium, TraceSampler improves image quality and prompt following; at 10 steps, ImageReward rises from 0.752 to 0.906 while trajectory MSE falls by 70.7%. These gains preserve Euler's backbone-evaluation budget, add less than 0.4% inference FLOPs, and incur negligible wall-clock overhead, showing that coarse-step sampling error can be made predictable from the sampling trace and efficiently corrected. Code is provided in the supplementary file.

Primary area:
generative models
Keywords:
diffusion models, flow matching, continuous-time generative models, accelerated sampling, few-step sampling, ODE solvers, post-training acceleration, numerical integration error, residual correction, sample-dependent correction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QVvxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TraceSampler to improve low-step sampling. Using a higher-step integration over the same interval as the reference, the method decomposes the endpoint error into three components:

  • A shared residual: all errors at a timestep t share this term.
  • Directions: a linear combination of different directions (2 for predictor and 4 for corrector), where the directions are computed from the trajectory so far and the coefficients are outputs of a learned network taking different features computed from the trajectory.
  • A low-rank remainder: to model the remaining error, where the low-rank basis is computed offline and the coefficients are output by the same network as the direction coefficients.

The paper also proposes a way to optimize the timestep schedule. The method is evaluated with Rectified Flow on CIFAR-10, SiT-XL/2 on ImageNet-256, and SD3.5 Medium for text-to-image generation, and compared against Euler, IIA-Euler, and UniPC (Euler only on SD3.5), and outperforms the baselines on most metrics.

Critical strengths
  • The experiments show the improvement over baselines across three generative models, different numbers of steps, and different baseline solvers. The inference overhead is small. The training overhead is a few hours, but it's a one-time cost per step budget.
  • The ablation study shows the contribution of each component and confirms that using all components with the optimized schedule has the best performance.
Critical weaknesses
  • The motivation for several design choices remains unclear. The directions and features (Sec. 3.2, App. A.1) are quite heuristic, for example, the target blocks, how many target blocks, etc. It is unclear whether other feature engineering would work equally well or whether, as long as the features are sufficiently rich, the method would still work. The writing is also hard to follow and not consistent. For example, since each column of PkP_k and $ \tilde {C} _ k $ has the same dimensionality as the sampling state, readers need to assume that $ \hat {r} _ {p,k} $ and $ \hat {r} _ {c,k} $ in Eq. (1) and (2) are vectorized. The notation ss in rs,k,Ds,kr_{s,k} , D_{s,k} (line 182, 183) is used before its definition in line 243. In equation (3), the paper introduces Esched(Δt)E_{sched}(\Delta _ t), while in Appendix B it does not appear, but E^\hat{E}.
  • Positioning against related works: Compared to IIA, TraceSampler uses a model with more capacity to model the error, where multiple directions are considered, and the coefficients are not learnable scalars but outputs of a learnable neural network. AMED-Solver[1] also predicts the parameters (intermediate timestep sks_k, scaling factor ckc_k) with a neural network from timesteps tk+1,tkt_{k+1}, t_k, and the bottleneck feature hh. Furthermore, methods such as LD3[2] and GITS[3] are also relevant to optimize the timestep schedule. These need discussion to clarify the contribution.

[1] Zhou, Z., Chen, D., Wang, C., & Chen, C. (2024). Fast ODE-based Sampling for Diffusion Models in Around 5 Steps. CVPR 2024, 7777–7786.

[2] Tong, V., Hoang, T.-D., Liu, A., Van den Broeck, G., & Niepert, M. (2025). Learning to Discretize Denoising Diffusion ODEs. ICLR 2025.

[3] Chen, D., Zhou, Z., Wang, C., Shen, C., & Lyu, S. (2024). On the Trajectory Regularity of ODE-based Diffusion Sampling. ICML 2024, PMLR 235, 7905–7934.

Questions for authors
  • Can you elaborate on the design of the predictor and corrector directions, for example, why is the initial noise involved in the corrector directions but not in the predictor directions?
  • How does TraceSampler differ from AMED-Solver?
  • Optional: Ablation studies to show which directions are the most important and how the coefficients vary among samples.
Reviewer_WJQfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TraceSampler, a post-training method for few-step diffusion sampling. The residual is represented using a step-specific mean, handcrafted trajectory directions, and a low-rank PCA remainder. Separate per-step MLP heads predict sample-dependent coefficients for a two-stage predictor–corrector update. Experiments on CIFAR-10, ImageNet-256, and SD3.5 Medium show strong improvements over Euler, IIA, and UniPC. The empirical results are promising. However, the paper omits several directly related learned-solver methods, requires a large number of setting-specific parameters, and relies on numerous heuristic design choices without sufficient validation. These issues make it difficult to establish either a distinct methodological contribution or a practical advantage over existing solver-fitting and distillation approaches.

Critical strengths
  1. The paper is clearly written and the overall pipeline is easy to follow. The decomposition of endpoint residuals and the sequential fitting procedure are presented in an organized manner.

  2. TraceSampler achieves strong few-step results across three backbones. In particular, it substantially reduces trajectory error and approaches the 120-step baselines on CIFAR-10 and ImageNet-256 while introducing small additional overhead.

Critical weaknesses
  1. The related work and comparisons are incomplete. Several directly relevant learned or fitted samplers are omitted, including AMED-Solver [1], BES [2], LD3 [3], S4S [4], and GAS [5]. The experiments compare only with IIA and UniPC on CIFAR-10 and ImageNet, and only with Euler on SD3.5. Consequently, the novelty and comparative advantage of TraceSampler are unclear.

  2. The method has substantial parameter and configuration costs. A TT-step sampler requires 2T−12T-1 independent MLP heads. Based on Table 15, the 8-step configurations add approximately 309M, 380M, and 1.33B parameters for Rectified Flow, SiT, and SD3.5, while 10-step SD3.5 adds approximately 1.69B parameters. Each new step budget or schedule requires refitting. Although the additional FLOPs are small, the storage, memory, preparation, and maintenance costs are considerable. The paper does not demonstrate a practical advantage over recent efficient distillation methods such as ECM [6] and TDM [7].

  3. The method relies on many insufficiently justified design choices. These include six handcrafted trajectory directions, a fixed Gram–Schmidt order, selected backbone features, PCA remainders, wide per-step heads, and a separate schedule optimizer. The ablations add all corrector directions together and do not isolate their individual contributions. A shared timestep-conditioned head and smaller head architectures are also not evaluated. Therefore, the necessity of the proposed components and parameterization remains unclear.

Reference

[1] Zhou Z, Chen D, Wang C, et al. Fast ode-based sampling for diffusion models in around 5 steps[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 7777-7786.

[2] Shaul N, Perez J, Chen R T Q, et al. Bespoke solvers for generative flow models[C]//International Conference on Learning Representations. 2024, 2024: 12510-12546.

[3] Tong V, Hoang T D, Liu A, et al. Learning to discretize denoising diffusion odes[C]//International Conference on Learning Representations. 2025, 2025: 47244-47282.

[4] Frankel E, Chen S, Li J, et al. S4s: Solving for a diffusion model solver[C]. International Conference on Machine Learning (ICML 2025), 2025.

[5] Oganov A, Bykov I, Neudachina E, et al. GAS: Improving Discretization of Diffusion ODEs via Generalized Adversarial Solver[C]//International Conference on Learning Representations. 2026, 2026: 39711-39747.

[6] Geng Z, Pokle A, Luo W, et al. Consistency models made easy[C]//International Conference on Learning Representations. 2025, 2025: 75158-75185.

[7] Luo Y, Hu T, Sun J, et al. Learning few-step diffusion models by trajectory distribution matching[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 17719-17728.

Questions for authors
  1. How does TraceSampler differ technically from AMED-Solver, BES, LD3, S4S, and GAS [1–5], particularly in its use of reference trajectories, sample-dependent coefficients, and learned solver components?

  2. Can a fitted configuration transfer across step budgets, schedules, CFG scales, or resolutions, or is separate fitting always required?

  3. Ablations on the individual predictor and corrector directions, the Gram–Schmidt order, backbone hidden features, PCA components, and MLP width are needed. Can a smaller or shared timestep-conditioned head achieve similar results? These experiments would clarify which components and how much model capacity are actually necessary.

  4. The main CIFAR-10 and ImageNet results use model- and budget-specific optimized schedules for TraceSampler. How do Euler, IIA, UniPC, and the learned-solver baselines perform under the same schedule, or when their schedules are optimized using comparable resources?

  5. The SD3.5 fitting prompts are drawn from 30,000 COCO validation captions, while the COCO evaluation also uses 30,000 captions. Are these the same captions?

Reviewer_ikoJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TraceSampler, a post-training method for accelerating generative sampling by predicting and correcting the endpoint errors of coarse numerical integration steps. The method combines trajectory-dependent directions, a low-rank residual representation, and step-specific MLPs to estimate sampling errors without modifying the pretrained generative model. Experiments on CIFAR-10, ImageNet-256, and SD3.5 Medium demonstrate improved generation quality with limited inference-time overhead.

Critical strengths
  1. Relevant and important problem. Accelerating generative sampling while maintaining generation quality is an important and practically relevant research direction.

  2. Clear presentation. The proposed method is well organized and easy to follow. The experimental setup and results are also presented clearly.

  3. Promising generation quality and efficiency. The method achieves strong generation quality across multiple benchmarks while introducing only a small reduction in inference throughput compared with Euler sampling.

Critical weaknesses
  1. Insufficient motivation for systematic error structure. The central hypothesis that sampling errors exhibit systematic, predictable structure is not sufficiently justified. The authors could provide a preliminary empirical analysis, such as examining residual alignment with trajectory directions or the explained variance of the PCA components, to better support this motivation.

  2. Heuristic design of trajectory directions. The selection of directions used to construct the predictor and corrector representations appears largely heuristic, with limited theoretical justification. An ablation study examining the contribution of each direction, or comparing against simpler alternative direction sets, would help establish the necessity of the proposed design.

  3. Limited generalization across sampling configurations. The method requires separately fitted components for different numbers of sampling steps and timestep schedules. This limits its flexibility compared with conventional numerical solvers and introduces additional offline training costs.

  4. Non-negligible overhead from additional MLPs. Although the reported inference FLOPs and throughput overheads are small, the proposed MLPs are not necessarily lightweight in terms of parameter count. Separate predictor and corrector networks are required for each sampling step, resulting in substantial additional parameters and memory consumption.

  5. Unfair evaluation. The paper primarily compares TraceSampler against conventional numerical solvers, such as Euler and UniPC, which do not require additional training data, while TraceSampler leverages thousands of high-step reference trajectories to train sample-dependent correction networks.

Questions for authors
  1. Error analysis of intermediate-state backbone evaluation. In TraceSampler, the backbone velocity used for the next Euler update is evaluated at the intermediate state (\tilde{x}{k+1}), rather than the final corrected state (x{k+1}). This introduces a discrepancy between the state used for the next update and the state at which its velocity was evaluated. Could the authors provide a theoretical or empirical analysis of the error introduced by this design choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.