acceptodds
Under review as a conference paper at ICLR 2027

Small Loss, Poor Generator: Where Consistency Tuning Should Train for One-Step Generation

Abstract

Training a one-step image generator is expensive. *Easy Consistency Tuning* (ECT) turns a pretrained diffusion model into a one-step consistency model by fine-tuning, and its smallest setting on CIFAR-10 uses about one million images. We study which distribution of training noise levels (which we call the training path) gives the best one-step generator for a fixed budget, and why. We introduce a novel theoretical framework with the following results. (i) The averaged consistency loss can tend to zero whilst the generator stays wrong. (ii) Adapting a known telescoping bound to the level at which generation starts, we bound the generator's error by quantities that involve only noise levels . (iii) In a simple chain model of training, the least-trained pairs of neighbouring noise levels below limit how fast the generator improves. (iv) A uniform draw of the noise levels gives every such pair nearly the same weight in the averaged loss. From this framework we derive the resistance of a training path, which is computed before any training. We apply these theoretical results to propose *Uniform Consistency Tuning* (**UCT**). Experiments on CIFAR-10 and ImageNet-64 show that **UCT** improves the one-step generation of ECT at small training budgets.

Primary area:
generative models
Keywords:
consistency models, one-step generation, consistency tuning, diffusion models, noise level sampling, flow maps

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fM8HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the distribution of training noise levels affects one-step consistency tuning under limited training budgets. It shows that a small average consistency loss need not imply low generation error, develops an error bound along noise levels below the generation start, and introduces a simplified chain model linking training allocation to convergence speed. Motivated by this analysis, the authors propose Uniform Consistency Tuning (UCT), which replaces 25% of ECT’s noise-level samples with uniform draws over a specified interval. Experiments on CIFAR-10 and ImageNet-64 demonstrate improvements over ECT at small training budgets, although the ImageNet-64 advantage disappears with longer training.

Critical strengths
  • Simple and inexpensive modification: UCT is easy to incorporate into ECT and adds negligible per-update computational overhead.
  • Relevant theoretical question: The paper examines why low average consistency loss may fail to predict generation quality and offers an interpretable account of how training allocation affects convergence.
  • Consistent small-budget improvements: UCT improves CIFAR-10 results across six paired seeds, under both data-matched and pure-noise initialization, and shows gains on ImageNet-64 at smaller budgets.
  • Useful controlled experiments: The study compares alternative sampling distributions, mixture proportions, and interval endpoints, alongside longer-training runs and an additional DINOv2-based evaluation metric.
Critical weaknesses

Overall, I find UCT’s contribution over prior work limited for the following reasons:

  • Limited theoretical support for the proposed modification: The methodological contribution consists of adding a uniform component to ECT’s noise-level sampling distribution. Although this simple change improves performance at small training budgets, its theoretical justification remains indirect. The convergence analysis assumes independently parameterized outputs across noise levels, and the resistance criterion provides an error bound within this simplified model without establishing improved convergence for shared-parameter neural networks. Moreover, the rate-monotonicity result does not directly establish UCT’s advantage, since the mixture decreases sampling rates at some noise levels. The theory therefore motivates the sampling heuristic but does not establish its superiority over ECT.

  • Diminishing gains with increased training budget: On CIFAR-10, increasing training from 8K to 100K updates reduces UCT’s FID improvement over ECT from 0.41 to 0.13 under data-matched initialization (5.23→4.82 versus 4.13→4.00), and from 1.01 to 0.20 under pure-noise initialization (7.35→6.34 versus 4.50→4.30). The 100K-update results use only one paired seed, leaving the robustness of these smaller gains uncertain. On ImageNet-64, the advantage disappears at 100K updates. Although these findings are consistent with the paper’s focus on limited training budgets, they leave unclear whether UCT provides substantial benefits at budgets needed to achieve stronger generation quality.

  • Limited evidence of scalability and generality: Evaluation is restricted to CIFAR-10 and ImageNet-64, whereas prior work, including ADCM [1] and CMT [2], already reports ECT/ECM baselines on ImageNet-512. A matched higher-resolution comparison would help establish whether UCT retains its efficiency benefits in more demanding settings. Its applicability to text-to-image and video generation also remains an open question: conditioning and temporal consistency introduce additional requirements that the current experiments and theoretical analysis do not address. While evaluating all these tasks may exceed the paper’s scope, the present evidence supports a narrow conclusion about UCT’s practical applicability.

[1] Bai et al. “Adaptive Discretization for Consistency Models.” NeurIPS, 2025.

[2] Hu et al. “CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Models.” ICLR, 2026.

Questions for authors
  • How was the mixture weight π = 0.25 selected, and does this choice remain effective across datasets and longer training budgets?

  • Why is the uniform sampling interval fixed to [0.002, 5] when generation starts from pure noise at level 80? How should this interval be selected for a new dataset?

Reviewer_67wYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Uniform Consistency Tuning (UCT), which replaces a quarter of ECT's lognormal training noise levels with uniform.

Critical strengths
  1. The presentation of the paper was clear, and the key aspects of the methodology were well articulated.
Critical weaknesses
  1. The claimed "novel theoretical framework" mostly restates known results. Theorem 2 adapts the telescoping bound of [1-2]. Theorem 1 perturbs a single, measure-zero noise level under a point-mass data distribution, a mechanism already noted by TCM [1], and it holds equally for ECT and UCT.

  2. The method is a hyperparameter change that the theory does not determine. Re-weighting the training noise distribution is a long-studied design axis [4-7], and the diagnosis that ECT's lognormal under-trains high noise levels was already made in [8].

  3. The baseline is not tuned. ECT's lognormal parameters are never searched, and simply shifting its mean already beats UCT in the standard pure-noise setting (FID 6.01 vs. 6.38, Table 2), with a lower DINOv2 distance in all three seeds (Table 7).

  4. The gains vanish at larger budgets. On ImageNet-64 at 100k updates, UCT is slightly worse (8.53 vs. 8.50). On CIFAR-10 at 100k updates, the gain is 0.13. The headline numbers also rely on a data-matched start taken from prior work [9, 10], which by itself improves ECT more (7.35→5.23) than UCT does (5.23→4.82).

  5. The experiments are limited to small-scale benchmarks. Only CIFAR-10 and ImageNet-64 are used, and on ImageNet-64 the main comparison is between two models that are far from usable (FID 49.75 vs. 48.39 after 8k updates). It is therefore unclear whether the effect carries over to higher-resolution or latent-space settings, where one-step generation matters in practice.

  6. Missing citations: [11–14].

[1] Consistency Models Made Easy. ICLR 2025.

[2] ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models. CVPR 2024.

[3] Truncated Consistency Models. ICLR 2025.

[4] Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022.

[5] Perception Prioritized Training of Diffusion Models. CVPR 2022.

[6] Efficient Diffusion Training via Min-SNR Weighting Strategy. ICCV 2023.

[7] Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation. NeurIPS 2023.

[8] High Noise Scheduling is a Must. arXiv 2024.

[9] Spontaneous Symmetry Breaking in Generative Diffusion Models. NeurIPS 2023.

[10] The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications. TMLR 2024.

[11] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024.

[12] Adaptive Non-uniform Timestep Sampling for Accelerating Diffusion Model Training. CVPR 2025.

[13] See Further When Clear: Curriculum Consistency Model. arXiv 2024.

[14] Improved Denoising Diffusion Probabilistic Models. ICML 2021.

Questions for authors
  1. Please report two-step FID for both methods (ECT's main metric at this budget) and one-step FID from pure noise on ImageNet-64.
Reviewer_gEW8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the distribution of training noise levels affects low-budget consistency tuning for one-step generation. The authors argue that a small averaged consistency loss does not necessarily imply a good one-step generator, because poorly trained links near the generation start can have little contribution to the average loss but still affect the final sample.

The paper develops a theoretical framework based on a telescoping error bound and a simplified chain model of consistency training. From this model, it defines the resistance of a training path, which measures how strongly the least-trained links limit convergence. Motivated by this analysis, the authors propose Uniform Consistency Tuning (UCT), which replaces 25% of ECT's lognormal noise-level samples with uniform samples between the minimum noise level and the generation start (T').

On CIFAR-10, UCT consistently improves ECT at small training budgets, including in all six paired seeds at 8k updates. Smaller improvements are also demonstrated on ImageNet-64.

Critical strengths
  • On CIFAR-10 at 8k updates, UCT improves one-step FID in all six paired seeds, from 5.23±0.08 to 4.82±0.07 from the data-matched start and from 7.35±0.12 to 6.34±0.21 from pure noise. The paired design and consistent per-seed improvements make the result convincing.

  • The paper clearly shows that the averaged consistency loss alone need not certify one-step generation and develops a concrete notion of training-path resistance. The resulting prediction—that poorly trained links near the start can bottleneck low-budget tuning—is intuitive and practically useful.

Critical weaknesses
  • The chain assumes separate outputs at every noise level, no parameter sharing, gradient-flow dynamics, an exact lowest level, and ordered initial errors. These assumptions differ substantially from an actual neural consistency model, where all levels share parameters and updates interact globally. Accordingly, resistance is not theoretically guaranteed to predict the optimization of the real network. Its empirical Spearman correlation with FID is encouraging but moderate (0.68), and the “range alone” configuration is an important counterexample: it has low resistance but substantially worse FID.

  • The strongest evidence uses six seeds on CIFAR-10, whereas ImageNet-64 uses only two seeds at 8k updates and one seed for longer training. The ImageNet gain is relatively small and essentially disappears after 100k updates (8.50 vs. 8.53). Moreover, ImageNet is evaluated only from the data-matched start rather than the standard pure-noise start. Thus, the evidence strongly supports UCT as a low-budget improvement to ECT on CIFAR-10, but is less conclusive about broader scalability and generality.

Questions for authors
  • How well does resistance predict performance after averaging FID over seeds for each training path, rather than treating the 36 checkpoints as separate observations? A held-out-path analysis would also help establish whether resistance predicts previously unseen sampling distributions rather than explaining the paths used to develop it.

  • Which assumptions of the chain model are most important in practice? In particular, is there evidence that the “least-trained link” behavior persists when outputs at different noise levels share parameters, as they do in the actual consistency network?

Comments on the paper

1 comment
Reviewer_67wYOpenReview ↗

Response to Rebuttal

Thank you for the response.

My main concerns remain, however. The theoretical results are largely adaptations of known arguments, and the lognormal baseline is still not tuned. The new results also suggest that the benefit of UCT is limited to one-step generation at small budgets: it does not improve two-step FID, nor one-step FID from pure noise on ImageNet-64. I will therefore keep my score.

I understand that adding new datasets during the discussion period is difficult, and I hope the following is useful for a future version. The paper is motivated by the cost of training one-step generators, and that cost matters most for high-resolution and latent-space models, where a better training path would have the most practical value; recent consistency model work also routinely reports results at that scale. Evidence in such a setting would make the contribution much more convincing. If the claims are instead meant to cover only pixel-space settings at small budgets, it would help to state this explicitly as a limitation. I encourage the authors to strengthen the evaluation before resubmitting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.