acceptodds
Under review as a conference paper at ICLR 2027

Iterating Consistency Models: Stability, Error Bounds and Noise Schedules

Abstract

Consistency models (CMs) have become a leading approach for generating high-quality samples in few steps. However, adding steps can improve or degrade sample quality in ways that are highly sensitive to the schedule and that existing theory does not fully explain. To provide accuracy guarantees and guide CM sampler design, we analyze multistep CM sampling as a composition of noising and approximate denoising operators. Under explicit, verifiable stability assumptions, we derive a non-asymptotic error bound that separates contraction of the initialization error from accumulation of approximation error. The bound assigns distinct roles to the schedule: large early noise levels drive contraction, while small late noise levels control the residual bias. As a corollary, we obtain explicit constants for strongly log-concave and semi-log-concave targets. We further establish a complementary guarantee whose assumptions, one-step accuracy and stability, can be estimated for a given trained model. Experiments show that the contraction and approximation profiles entering our bounds can be reliably measured and closely match the predicted functional forms. Together, these results provide a meaningful convergence theory for multi-step CMs and a practical route to sampler design.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Consistency Models, Diffusion Models, Multistep Sampling, Convergence Guarantees, Error Bounds

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YUChOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes iterative consistency model (CM) sampling through a perturbation analysis that bounds the Wasserstein error after repeated noising and denoising steps. The bounds separate contraction of the initialization error from accumulation of the model approximation error. Furthermore, the paper provides stability estimates from target distribution curvature, covering strongly log-concave distributions and a specific class of non-convex potentials. Gaussian examples concretely illustrate possible attainment of the error bounds.

Critical strengths
  1. Extremely clear and didactic presentation. As someone who does not work directly on this topic, I am certainly impressed by the authors' effort in explaining and interpreting their results, as well as providing the scope of their contributions. For example, the roles of the assumptions for exact-denoiser stability and learned-map stability are thoroughly discussed (Sections 3.1-3.3). The Gaussian examples very clearly illustrate the trade-off between contraction and new approximation error. I also appreciate the careful distinction between improving an upper bound error and improving the actual sampling error. In all, the nice writing makes the contribution very easy to understand and assess (at least at a high level).
Critical weaknesses
  1. (Minor) Limited methodological novelty: The paper provides an error-propagation framework for repeated CM sampling. However, in my view, the contribution remains limited because the approach and its' ingredients are mostly standard, as they follow existing CM stability analysis. To be fair, the paper does acknowledge that it builds on the error recursion of Lyu et al, and mentions the two main high-level consequences of this recursion (Introduction, Remark 3.7). It is thus my perspective that the contribution in this paper mostly explores the estimates and conditions for which the recursion of Lyu et al applies. Subjectively, I think this is a bit modest (of course, the value of a paper does not need to depend primarily on the difficulty of its' mathematical argument).

  2. (Major) Limited explanatory power of the results relative to the motivating question: As mentioned above, the motivating question of the paper is why additional sampling steps can improve or degrade sampling quality, and how this depends on the noise schedule (Introduction). To that end, the paper seems to take a robustness perspective, and treats the worst-case triangle bounds as a guideline for schedule design based on one-step error budgets and stability factors (Propositions 3.1, 3.9). In principle, this is not an invalid approach by itself, especially if we only have access to these quantities. However, it is my opinion that it is precisely the alignment and cancellation between inherited and new approximation errors that make this question (and the answer to it) interesting. It seems to me that the results the authors provide are still a bit far away from a meaningful first step towards understanding this question. Even in the authors' Gaussian analysis (Section 3.4), the actual Wasserstein distance, thus the tightness of the upper bound, behaves quite differently depending on whether the discrepancy lies in the mean or variance, and whether inherited and newly introduced variance errors superposition or interfere with one another. It is my sense that there can be information regarding the learned denoiser's error structure and its' relationship with the target distribution that is yet to be found. This is a tricky technical challenge for sure, but certainly important.

Moreover, on the practical aspect, the Gaussian example shows that minimizing the certificate upper bound can introduce an extra sampling step that actually increases sampling error (Appendix E.3, Figure 5). This is already acknowledged by the authors in Section 3.1. However, under the stated goal of the paper, I am a bit concerned that there is no no-harm guarantee with respect to sampling quality.

Lastly, as a minor detail, the comparison with Chen et al. is quantitatively correct but is not methodologically convincing. Chen et al. only propagates KL errors and has a lossy step of converting to a Wasserstein bound, but does not require extra assumptions used in this paper (Remark 3.8). Having superior numerical certificates with a more informative setting is not surprising in my view.

I want to clarify that the authors very attentively scope the contribution of this paper, and I have no issue regarding over-claiming. However, as stated I am not confident there is sufficient novelty or significance.

Questions for authors

I do not have questions. I can change my assessment based on the authors' rebuttal.

7 comments
AuthorsOpenReview ↗

W1: Novelty relative to Lyu et al.

We thank the reviewer for a thoughtful report, and for recognising the care taken over the scope of the results.

(Minor) Limited methodological novelty: [...] the contribution in this paper mostly explores the estimates and conditions for which the recursion of Lyu et al applies. Subjectively, I think this is a bit modest.

The error recursion is indeed due to Lyu et al., as the introduction and Remark 3.7 state. Our contribution is to establish its ingredients under different assumptions, which can be verified in closed form for classes of targets and estimated for trained models, and most of the work lies there:

  • Stability from the target. We place the stability condition on the exact denoiser, as Lyu et al. suggest in their Section 4, and derive it from the target's curvature. Strong log-concavity gives explicit factors qt=(1+mγt2)−1/2q_t=(1+m\gamma_t^2)^{-1/2} (Theorem 3.3, Corollary 3.4). For semi-log-concave targets we give the exact noise threshold above which contraction is certified (Theorem 3.6).
  • A weaker error model in a general setting. The score error may blow up as M1/rM_1/r when the noise vanishes, instead of being uniformly small; this changes the shape of the error profile (the arcsinh term). We allow general Gaussian noising, including VE and TrigFlow, and arbitrary schedules.
  • Trained models. A learned-map route whose assumptions can be estimated (Proposition 3.9), and a transfer from consistency losses to the target error (Theorem 3.11).
  • Tightness. An exact Gaussian analysis of when the certificate is attained and when it is loose (§3.4, Appendix E).

As the reviewer notes, the value of a result need not lie in the difficulty of its argument; here it lies in making each ingredient of the recursion explicit, so that for a given target and trained model one can assess empirically whether the assumptions are plausible, as we do for two latent consistency models.

AuthorsOpenReview ↗

W2: Error cancellation and explanatory power

(Major) Limited explanatory power of the results relative to the motivating question: [...] it is precisely the alignment and cancellation between inherited and new approximation errors that make this question (and the answer to it) interesting. [\ldots] Even in the authors' Gaussian analysis (Section 3.4), the actual Wasserstein distance, thus the tightness of the upper bound, behaves quite differently depending on whether the discrepancy lies in the mean or variance, and whether inherited and newly introduced variance errors superposition or interfere with one another.

We agree that cancellation can affect sampling error and make the bound conservative. However, it is not needed to explain why an additional step can help or harm. Our Gaussian example with errors in the mean only (§3.4) shows this directly: there the bound is exact for every schedule, yet along a decreasing schedule the error first decreases and then increases. The balance between reducing existing error and introducing new error is therefore sufficient to produce this behavior.

The same example shows that a sharper bound requires more information about the errors, including how they align and cancel. Our Gaussian analysis illustrates these effects, and the framework can accommodate them by tracking different sources of error separately.

We also examine cancellation empirically on SD 1.5. We observe partial cancellation at the first step, but little evidence of it at later steps along the three schedules tested. In particular, the steps that increase the error reinforce existing errors rather than cancel them. These observations support the relevance of the mechanism described by our analysis.

Changes in the revision. Section 3.4 (p. 8, ll. 408–415) clarifies what the bound captures and what additional information is needed to account for cancellation. Appendix F reports the empirical findings, and the conclusion identifies this refinement as future work (p. 10, ll. 524–526).

AuthorsOpenReview ↗

W2: No-harm guarantee for an extra step

Moreover, on the practical aspect, the Gaussian example shows that minimizing the certificate upper bound can introduce an extra sampling step that actually increases sampling error (Appendix E.3, Figure 5). This is already acknowledged by the authors in Section 3.1. However, under the stated goal of the paper, I am a bit concerned that there is no no-harm guarantee with respect to sampling quality.

No upper bound can guarantee that an extra step helps: that depends on the current error, which the sampler does not know (§3.1 notes that its statements compare certificates, not actual errors, and Fig. 5 shows a step that lowers the certificate but raises the error). An upper bound still gives insight into when a step can harm and by how much: a necessary condition for harm and a cap on it. By Proposition 3.9, a step at level tt takes the error δ\delta to at most q~tδ+βt+Etπ\widetilde q_t\delta+\beta_t+E^\pi_t, a weighted average of δ\delta and the plateau (Etπ+βt)/(1−q~t)(E^\pi_t+\beta_t)/(1-\widetilde q_t), so

W2(L~tθ(ρ),π)≤max⁡{W2(ρ,π), Etπ+βt1−q~t}:W_2\big(\widetilde{\mathcal L}^\theta_t(\rho),\pi\big)\le\max\Big\lbrace W_2(\rho,\pi),\ \frac{E^\pi_t+\beta_t}{1-\widetilde q_t}\Big\rbrace:

a step can increase the error only if it is already below the plateau of that level, and never beyond it; the same holds with (qt,Et)(q_t,E_t). Fig. 5 is an instance: after the first step the variance errors nearly cancel, the error is far below every plateau, and the harmful step raises it, but not above the plateau. Trained models behave the same way: along an equispaced SD 1.5 schedule, steps at high noise raise the error by 60%, below their plateaus (our reply to Reviewer nkjH, W3).

Changes in the revision. §3.1 (p. 4, ll. 204–211) states this bound on the harm of a step, after the comparison of certificates, and that whether a step harms must be checked empirically.

Reviewer_nkjHOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multistep consistency-model sampling through a Wasserstein error recursion that separates contraction of initialization error from accumulated approximation error. The authors derive explicit stability profiles under curvature assumptions and provide a complementary analysis for trained models. Gaussian examples and latent consistency-model experiments illustrate the resulting bounds.

Critical strengths
  1. The paper is well written and easy to follow. The decomposition into contraction and approximation error provides a clear explanation of the trade-off involved in multistep consistency-model sampling.
  2. The theoretical analysis goes beyond the general error recursion. The authors derive explicit stability profiles under different curvature conditions and connect consistency training errors to the accuracy of trained models.
  3. The Gaussian examples are informative. They illustrate when the bounds are tight and explain how variance errors and error cancellation can make the guarantees conservative, which helps clarify the scope of the analysis.
Critical weaknesses
  1. The empirical stability measurements do not establish the global assumptions required by the theory. Measurements from sampled inputs may miss expanding directions, while covariance-based estimates do not establish strong log-concavity of the latent distribution.
  2. The practical applicability of the guarantees remains somewhat unclear. The exact-map analysis requires uniform approximation control, while the learned-map analysis still depends on global network stability. The fitted experimental profiles do not directly verify these conditions.
  3. The experiments provide limited validation of the main multistep implications. The real-model evaluation focuses on the contraction and approximation profiles, without showing whether schedules selected using the proposed analysis improve multistep sampling quality.
Questions for authors

Could the authors use the estimated profiles to select multistep noise schedules and compare them with standard schedules at the same evaluation budget? This would help establish the practical value of the proposed analysis.

6 comments
AuthorsOpenReview ↗

W1: Empirical measurements and the global assumptions

We thank the reviewer for a careful reading, and for the positive assessment of the decomposition, the stability profiles and the Gaussian examples.

The empirical stability measurements do not establish the global assumptions required by the theory. Measurements from sampled inputs may miss expanding directions, while covariance-based estimates do not establish strong log-concavity of the latent distribution.

We agree that finite measurements cannot verify the assumptions over all input distributions. We now emphasize the difficulty of checking these conditions in the manuscript and present our experiments as empirical support for the theory. Likewise, covariance does not establish strong log-concavity; the revised manuscript uses the covariance-based profile only as a Gaussian reference (our reply to Reviewer Esog, Q2).

This difficulty also motivates extending our theory to local versions of the original conditions, required only near the target distribution. The proofs remain essentially unchanged, since they use the conditions only at the distributions visited by the sampler. This extension provides a basis for estimating the relevant quantities near the target, as discussed in our reply to Reviewer Esog, Q1.

The revision develops statistical estimators of these quantities in Appendices D.2–D.3 (pp. 23–25). We assess them empirically: the estimates are stable across independent repeats, and the measured contraction profiles and multistep errors are consistent with our theoretical predictions. These findings support our claims empirically. A full theoretical analysis of the statistical properties of the estimators is, we believe, beyond the scope of this paper. We clarify this distinction between empirical evidence and verification of the assumptions in the abstract, introduction, §4 and conclusion.

AuthorsOpenReview ↗

W2: Practical applicability of the guarantees

The practical applicability of the guarantees remains somewhat unclear. The exact-map analysis requires uniform approximation control, while the learned-map analysis still depends on global network stability. The fitted experimental profiles do not directly verify these conditions.

The two analyses offer alternative sets of assumptions. The first relies on properties of the target distribution and accuracy of the learned model, without requiring network stability. The second requires stability of the learned model, but imposes a weaker accuracy condition. This gives two ways to apply the theory, depending on the information available.

In the revision, we further weaken these conditions by requiring them only near the target distribution. The proofs remain essentially unchanged, and we explain why the sampler's iterates stay in this neighborhood. This extension broadens the applicability of the results and provides a basis for the local estimates used in our experiments.

As discussed in our previous reply, checking these conditions remains difficult. We therefore develop statistical estimates of the relevant quantities and assess them empirically. The results support our theoretical predictions, while a full theoretical analysis of the estimators is beyond the scope of this paper. The revised manuscript makes this distinction explicit.

AuthorsOpenReview ↗

W3: Validation of the multistep implications

The experiments provide limited validation of the main multistep implications. The real-model evaluation focuses on the contraction and approximation profiles, without showing whether schedules selected using the proposed analysis improve multistep sampling quality.

The revision adds a multistep evaluation on the SD 1.5 model (Appendix F, pp. 29–30, and Fig. 7). We start from 2048 samples of N(0,I)\mathcal N(0,\mathrm I) and run three standard schedules: the four-step LCM default, and geometric and equispaced eight-step schedules from σ=14.6\sigma=14.6 to 0.130.13. At each step we compare the error of the iterate, its distance to held-out target latents, with the certificate Bk=qtkBk−1+βtk+EtkπB_k=q_{t_k}B_{k-1}+\beta_{t_k}+E^\pi_{t_k}, where B0B_0 is the initial distance and the one-step quantities are those measured as in our reply to Reviewer nkjH, W1. Errors and certificate use the W2W_2 distance between Gaussian fits in the top 100 principal components: it bounds W2W_2 from below, satisfies the triangle inequality, so the recursion applies in it, and, unlike exact optimal transport, is not dominated by sampling bias in d=16384d=16384. Over three repeats:

  • the certificate holds at every step of every schedule;
  • it follows the shape of the error, including its growth at high noise along the equispaced schedule, and orders the three schedules as their errors do;
  • it exceeds the error by a nearly constant factor of 2.7–3.2, mostly set at the first step: the initial error lies in the variance, spread over directions that contract faster than the least contracted one, the two effects that §3.4 identifies as making certificates conservative.

The exact Gaussian analysis of §3.4 complements this with cases where the certificate is attained. Our aim is to understand the design factors behind multistep phenomena, why a step helps or harms and how the schedule sets the balance, rather than to optimize steps from the bounds; we elaborate on this in our reply to Reviewer nkjH, Q1.

Changes in the revision. Appendix F reports the schedules, the one-step bound for both models in this distance, and the triangle ratios of our reply to Reviewer YUCh, W2. The results are summarized in §4 of the main text (p. 10, ll. 507–511).

Reviewer_EsogOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The topic of this paper is about the generation task with consistency models. The authors analyzes the competition between contraction of initialization error and accumulation of approximation error in multistep consistency-model sampling. They extablish a complementary guarantee with assumptions one-step accuracy and stability for a given trained model. The experiments are conducted on CelebA dataset with a latent consistency model (LCM) distilled from Stable Diffusion 1.5.

Critical strengths
  • The interesting idea and the consistency model-based generation task. The substantive theoretical contribution is establishing geometric conditions for the recursion.
  • The idea has been evaluated on popular datasets and achieves good performance.
Critical weaknesses
  • The robustness of the key idea on different latent consistency models(LCM). In the experiments, only a LCM distilled from SD 1.5 has been utilized. To better show the robustness of the idea, provide more numerical results on other kinds of LCM (e.g., with different networks). For example, distill a diffusion transformer for different latent consitency models.
  • The implementation details are insufficient. For instance, the empirical evidence for multistep schedule selection should be analyzed.
Questions for authors

Q1: How stable are the estimated contraction factors under independent sampling, increasing particle/pair counts, and inputs drawn from intermediate sampler distributions?

Q2: What additional distributional assumptions justify inferring a lower bound on mm from covariance?

6 comments
AuthorsOpenReview ↗

W1: Results on other latent consistency models

We thank the reviewer for the positive assessment and for the questions on the estimators, which led to the clarifications below.

The robustness of the key idea on different latent consistency models(LCM). In the experiments, only a LCM distilled from SD 1.5 has been utilized. To better show the robustness of the idea, provide more numerical results on other kinds of LCM (e.g., with different networks). For example, distill a diffusion transformer for different latent consitency models.

The paper reports two different models, not one; we apologise if this was not clear in the original manuscript. The first is a variance-exploding LCM distilled from SD 1.5 and evaluated on AFHQ-cat latents (d=16384d=16384). The second is a TrigFlow LCM trained on CelebA in the latent space of a separate KL-regularized VAE (d=32d=32). They differ in noising process (VE vs. TrigFlow), training (distillation vs. consistency training), dimension and data, and both show the same qualitative profiles (Figs. 2 and 3). We agree that a transformer backbone would strengthen the evidence further.

Changes in the revision. Appendix D.1 (p. 23, ll. 1190 and 1219–1227) gives the architecture and training of both models (our reply to Reviewer Esog, W2), and both are evaluated with the new estimators of Appendices D.2–D.3 (pp. 23–25) (our reply to Reviewer Esog, Q1).

AuthorsOpenReview ↗

W2: Implementation details

The implementation details are insufficient. For instance, the empirical evidence for multistep schedule selection should be analyzed.

The revision expands Appendix D (pp. 22–25) with the estimators, sample sizes, repeats and noise levels behind every figure, and with the models: the CelebA LCM, a convolutional U-Net with 0.83M parameters trained from scratch by continuous-time consistency training in TrigFlow form, and its VAE (3.3M parameters, KL weight 2.5×10−42.5\times10^{-4}); and the SD 1.5 LCM, distilled with the diffusers consistency-distillation script (https://github.com/huggingface/diffusers/tree/main/examples/consistency_distillation) with its default hyperparameters, for 10,000 optimizer steps on Conceptual 12M, whose EMA U-Net we evaluate with a fixed prompt and no guidance. The seeds of every sample and noise draw are recorded, and an anonymized implementation of all the experiments, including those added for this rebuttal, will be released upon acceptance. Empirical evidence on multistep schedule selection is Reviewer nkjH's question (Q1), answered there; the multistep evaluation in the revision (Appendix F) is described in our reply to Reviewer nkjH, W3.

AuthorsOpenReview ↗

Q1: Stability of the estimated contraction factors

How stable are the estimated contraction factors under independent sampling, increasing particle/pair counts, and inputs drawn from intermediate sampler distributions?

We agree that checking the stability condition (A7, old A6) for a trained model is difficult. It must hold over a class of input distributions, and finite measurements can miss expanding directions or regions that are rarely sampled. We now emphasize this difficulty in the manuscript and clarify that our experiments provide empirical evidence for the condition.

This difficulty also motivates extending our theory to local versions of the original conditions, required only for distributions in a neighborhood of the target. The proofs remain essentially unchanged, since they use the conditions only at the distributions visited by the sampler. This extension provides a theoretical basis for focusing our estimates on perturbations near the target, at scales relevant to the sampler's errors.

In the revised version, we develop statistical estimators of the relevant quantities in Appendix D.2 (pp. 23–24, ll. 1230–1289). These estimates need no pairs, are stable across independent repeats, and yield contraction profiles consistent with our theoretical predictions (Fig. 2, p. 9), and multistep error bounds that hold on the measured errors (Appendix F, pp. 29–30, and Fig. 7), providing empirical support for our claims. The SD 1.5 step expands some small perturbations, which the revised condition (A7, old A6) covers with a measured offset (Remark 3.10, p. 7, ll. 328–329 and 343–347). A full theoretical analysis of the statistical properties of these estimators is, we believe, beyond the scope of this paper; in particular, we do not claim that these measurements verify the stability condition over all admissible input distributions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the main changes in uploaded revision

We thank the reviewers for their careful reports. The reviewers find the theoretical contribution substantive (Reviewer Esog), the paper "well written and easy to follow" (Reviewer nkjH), and the presentation "extremely clear and didactic" (Reviewer YUCh). The main questions concern what the experiments show, how the analysis informs schedules, and how worst-case bounds relate to error cancellation. The main changes are:

  1. Weaker assumptions. Reviewer nkjH asked about the practical applicability of our assumptions (A4) and (A7, old A6), which originally had to hold for every distribution. We now require them to hold only in a neighborhood of the target distribution, and a new assumption (A5, p. 4, ll. 185–189) keeps the sampler's iterates there.
  2. When a step can harm. Reviewer YUCh noted that the paper does not provide a no-harm guarantee. As seen in our Gaussian example (Appendix E.3, Fig. 5), a step chosen to lower the certificate can increase the actual error. We agree that this is a valid concern, and the revision makes it explicit that the recursion gives only an upper bound on the error, not a guarantee that a step helps. Nevertheless, it may still provide insight into the conditions under which a step can increase the error, and how the trade-off between contraction and new error affects the overall error profile.
  3. Sharper experiments. Reviewer Esog and Reviewer nkjH asked how stable the contraction estimates are and whether they support the assumptions. We now emphasize the importance of this question in the manuscript and provide estimates of the relevant quantities. Although we do not fully characterize their theoretical properties, these estimates appear consistent and provide empirical support for our claims.

Each reply below lists the changes in the revised manuscript that concern it, with their sections there; changes are in blue. An anonymized implementation of all the experiments will be released upon acceptance.

Dear reviewers,

Thank you again for your thoughtful feedback and for taking the time to review our paper. If you have a chance to read our responses, we’d appreciate hearing whether they address your main concerns or whether any points remain open. A brief note on any point that would benefit from further clarification would be very helpful, and we’d be happy to follow up here.

Best, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.