acceptodds
Under review as a conference paper at ICLR 2027

When Does Loss Normalization Preserve Stationarity in Consistency Learning?

Abstract

Consistency models are commonly trained with pseudo-Huber losses, whose gradients normalize each sampled correction before averaging. Normalization can move a model away from its intended flow map even when the expected unnormalized gradient vanishes there. We characterize when it preserves this stationary state. For a fixed joint law with integrable Jacobian-weighted corrections, stationarity holds at every positive floor, or smoothing constant, exactly when the Jacobian-weighted correction has zero mean conditional on the correction magnitude. This radial-balance condition requires cancellation within each group of equal correction magnitude, so it is stronger than cancellation of the overall mean. A smooth Gaussian mixture violates it at the exact map, and one of three networks trained to approximate this map meets prespecified criteria for accuracy and for separating the normalized field from the linear one. For fixed features and quadratic error, we derive sufficient conditions under which one step reduces the error, while a step of the same size from the same state under the reversed correction law increases it; on 16 fresh feature seeds, every resolved forecast agrees with the computed outcome. Across six paired tuning seeds from one pretrained diffusion model, two-step Fr\'echet inception distance favors zero-floor normalization over a squared-loss control with matched update lengths; three earlier continuation studies give mixed results that depend on the update policy.

Primary area:
learning theory
Keywords:
generative models, consistency models, flow maps, pseudo-Huber loss, loss normalization, stationarity, optimization theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_t9G5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when normalized consistency-learning updates preserve a stationary flow map. For a fixed joint distribution of corrections and Jacobians, it proves that stationarity at every positive normalization floor is equivalent to a conditional radial-balance condition. It also develops effective-floor identities, smooth counterexamples, parameter-projection criteria, and restricted conditions for one-step improvement or harm.

Critical strengths
  • A precise and useful theoretical characterization. Theorem 2 provides a necessary-and-sufficient condition. Conditioning Jacobian-weighted corrections on their unprojected magnitude correctly distinguishes overall mean cancellation from cancellation preserved under normalization.

  • Careful treatment of the executed update. The analysis distinguishes stop-gradient algebra, conditional fields, parameter projection, optimizer processing, and teacher lag. These distinctions prevent conclusions about one object from being incorrectly transferred to another.

  • Supporting analysis is good. The smooth Gaussian-mixture witness establishes the phenomenon concretely.

Critical weaknesses
  • The connection to actual training trajectories remains limited. The central theorem fixes the correction–Jacobian law while varying the floor. During training, this law, the teacher, and the optimizer all change. The paper provides extensions and sufficient conditions, but does not verify the relevant trajectory premises in the image experiments. The result is therefore a stationarity characterization rather than a convergence guarantee.

  • The strongest image experiment does not directly test the theorem’s positive-floor regime. The six-seed comparison uses zero-floor normalization, explicitly outside the positive-floor premise of Theorem 2. A positive-floor sweep would provide a more direct bridge between the main mathematical result and image-generation behavior.

Questions for authors

See weaknesses.

2 comments

Thank you for your careful review. We agree with both limitations. The final paragraph of the introduction now states that Theorem 2 concerns one training state and that the six-seed image comparison uses a zero floor, outside the theorem. We also clarified the trajectory assumptions in Section 9 and the interpretation of Table 8 in Section 8 and Appendix D.5.1.

1. Connection to training trajectories. Theorem 2 fixes the joint law of the correction and the network Jacobian and varies only the normalization floor. During training, the model, teacher, and optimizer state evolve, and the correction–Jacobian law can change. The theorem therefore characterizes stationarity at a given state; it gives no convergence guarantee.

Proposition 6 (Appendix A.1) gives a narrower result along one trajectory. Let D(ξ)D(\xi) be the difference between the expected PH and linear gradients at the same training state ξ\xi, with the two scores matched in slope at zero. Suppose its projection onto a fixed unit direction uu starts at u⊤D(ξ0)≥m>0u^\top D(\xi_0)\ge m>0 and is LL-Lipschitz in the state metric dd over the trajectory segment. Then

u⊤D(ξk)≥m−L∑j<kd(ξj+1,ξj).u^\top D(\xi_k)\ge m-L\sum_{j<k}d(\xi_{j+1},\xi_j).

The initial sign persists while the total change in state is small enough. Both gradients are evaluated at each state on the same trajectory; the bound does not predict how two runs trained with different losses will separate. As Section 9 now states explicitly, we did not verify the Lipschitz or displacement conditions in the image runs.

2. Positive floors in the image experiments. The six-seed comparison used c=0c=0. We chose this setting after finding that 82% of the measured residual norms at a trained ECM checkpoint were below 0.10.1, so a floor of 0.10.1 would weaken normalization for many examples (Appendix D.5.1). Starting from one pretrained EDM checkpoint, zero-floor normalization gave lower two-step FID-50k than squared loss in all six pairs, by 2.33 points on average (Table 8). The realized update lengths were matched, but each arm kept its own optimizer moments. This measures image quality under that training protocol; it does not test Theorem 2 or identify loss of stationarity as the reason for the FID difference.

The existing positive-floor results are more limited. Earlier continuations from a pretrained consistency model used c=0.03c=0.03 or 0.10.1 and gave mixed rankings under different protocols (Appendix D.5.2). At one frozen checkpoint with c=0.03c=0.03, we could not distinguish PH from its slope-matched linear control in the three-dimensional subspace measured (Table 10).

We agree that a positive-floor sweep would be informative. Varying the floor at a frozen checkpoint, with the teacher and sampling law fixed, would measure how the expected gradient changes under the theorem’s assumptions. Repeating the paired training comparison at several positive floors would address the separate question of image quality. We did not run a controlled positive-floor sweep of the six-seed experiment, and the revised Section 8 distinguishes this missing comparison from the positive-floor results already in the paper.

Thank you for the response. I appreciate the authors' explanation and acknowledging the limitation.

From the theoretical perspective, I feel the studied problem is interesting. Though I am not an expert directly in this area, I have explored consistency learning on image tasks. The use of pseudo-Huber loss or adaptive normalization would largely improve the performance, but their mechanism remain unclear. This paper provides a good theoretical study on this point.

That said, though acknowledging the limitation in the rebuttal, i) the connect to real training trajectory and ii) the transition between image experiments and the main math remains, which I think is also important in practice.

Considering the above, I would like to keep the rating unchanged.

Reviewer_HwoJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a theoretical analysis of the normalization effect of Pseudo-Huber losses in consistency models. They provide results showing in which conditions Pseudo-Huber losses preserve the target flow map or not.

Critical strengths

Some theoretical results on the normalization effect of Pseudo-Huber losses.

Critical weaknesses
  1. The main and critical issue is that the paper is poorly written.

Beginning of the section 3: niche jargon; the authors do not explain the context and model studied. What is the generative model? How is it trained? What are the quantities that you study in this generative model? Without a proper introduction giving the framework, a scientific paper is hardly readable.

For example, the very first sentence of section 3 is: "The update field is determined by the backward graph, including which targets are detached". What is an update field in the context of consistency models? What is the backward graph? What are the targets? This is understandable with prior knowledge of consistency models but is very hard to read. The whole paper is full of such poorly specified writing, which makes it impossible (to me) to assess correctly. Everything, in the paper, is supposed known by the reader. This is not how it should be: a paper should be mainly self-contained. Even though I have knowledge on consistency models (already published on the topic), there is too much undefined quantities and terms in the paper for me to read it.

Even for the experimental part, it is not clear what is done, and what insights do the authors gain from the proposed experiments.

B) Pseudo-Huber losses are used in first consistency models or in ECT, that the authors used for their experiments. However they are not used in continuous consistency models, which are state-of-the-art since 2025. There is no discussion around this in the paper.

Questions for authors

No

2 comments

Thank you for your comments. We agree that the original paper was difficult to follow. We rewrote the opening of Section 3 to introduce the model before the training rule, and revised Sections 5–8 to explain what each experiment tests and what it shows. We also expanded the comparison with continuous-time consistency models in Section 2 and Appendix B.

Model, training rule, and experiments. Section 3 now starts with the generation task. Time runs from noise at t=0t=0 to data at t=1t=1. Given xx at time tt, the network fθ(x,t)f_\theta(x,t) predicts the data endpoint of the flow through that point. To generate a sample in one evaluation, we draw noise ZZ and return fθ(Z,0)f_\theta(Z,0). We also distinguish the sampled path from the flow: for example, Section 5 uses Xt=(1−t)Z+tYX_t=(1-t)Z+tY, with sampled velocity V=Y−ZV=Y-Z. The conditional mean velocity E[V∣Xt=x,t]\mathbb E[V\mid X_t=x,t] defines the flow whose endpoint the network predicts.

Training compares predictions at nearby times. In the basic pair in Eq. (2), the current prediction is fθ(x,t)f_\theta(x,t) and its target is fθ(x−δV,t−δ)f_\theta(x-\delta V,t-\delta). We recompute the target at the next model state but hold it fixed when differentiating the loss. This is what we mean by a detached target. The backward graph specifies which predictions receive derivatives; here only the current prediction does. We added this definition at the start of Appendix B.

Writing rr for the current prediction minus its target and J=∂θfθ(x,t)J=\partial_\theta f_\theta(x,t), the two expected gradients, with unit outer weight, are

G_{\rm PH}=\mathbb E\left[\frac{J^\top r}{\sqrt{\lVert r\rVert^2+c^2}}\right].$$ Here $c$ is the PH smoothing constant. By *update field* we mean the expected parameter gradient at a fixed training state, before the optimizer acts. Stationarity means that this gradient is zero. The denominator shows why cancellation under squared loss need not carry over to PH: small and large residuals receive different weights. Table 2 now summarizes what was measured in each experiment. In particular: - Section 5 constructs a Gaussian mixture with a nonzero expected PH gradient at the exact map. Section 6.1 then measures gradients at three networks fitted to that map. In all three, the lower bound on a PH gradient projection exceeded the upper bound on the full linear-gradient norm, although only one fit met all four prespecified cutoffs (Table 1). - Section 7 measures map error after one step. At the original error scale, reversing the velocity fluctuations did not reverse the comparison between PH and the control. The revised text explains that the sufficient condition in Eq. (10) did not hold there. When it does hold in this model, reflection reverses the comparison at the same initial map and step size. - Section 8 separates gradient direction from image quality. We detected non-collinear gradients at 234 of 240 squared-loss-trained states (Table 7). In a separate CIFAR-10 experiment, zero-floor normalization gave lower two-step FID in all six matched pairs (Table 8). The FID result does not establish that loss of stationarity caused the difference. **Continuous-time consistency models.** We agree that citing Lu and Song without explaining their objective left an important gap. Section 2 and Table 3 in Appendix B now give the two formulas side by side in the discussion. In continuous-time sCM, Lu and Song normalize a tangent as $g/(\lVert g\rVert+c)$ and use it to form the detached target of an adaptively weighted squared loss. In the finite-gap PH rule, it is the prediction residual that is scored as $r/\sqrt{\lVert r\rVert^2+c^2}$. Both the quantity being normalized and the denominator differ. Theorem 2 applies to the PH rule. A stationarity result for the full sCM objective would require a separate derivation accounting for its tangent and weighting. Our six-seed image experiment used Easy Consistency Tuning (ECT).

I acknowledge author's response. While some changes have been made in the good direction, my main criticisms remain. The paper still requires a large revision to become a self-contained, well-written paper that reaches the level expected at ICLR.

Reviewer_5q3yOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper derives conditions under which the Psuedo Huber loss introduces a nonzero expected training gradient at the exact flow map, so the exact flow map is no longer necessarily a stationary point of consistency training. The paper then derives the necessary and sufficient conditions for which stationarity of the PH loss for every positive normalization floor is equivalent to the radial balance of the parameter gradients for almost every correction magnitude. They then proceed to demonstrate this non-stationarity of the Psuedo Huber loss on the exact flow map in Gaussian mixture models and provide additional experiments examining the effects of normalization on neural-network parameter gradients and image-generation performance.

Critical strengths

The paper derives radial balance as the necessary and sufficient condition for expected pseudo-Huber gradient stationarity across all positive normalization floors. The experiments are comprehensive, examining near-oracle neural networks where PH loss can have nonzero expected gradients even when the MSE loss is small. They also produce controlled experiments that symmetrize or reflect the conditional velocity fluctuations while preserving the marginal velocity, examining how the resulting PH gradient affects flow-map approximation error.

Critical weaknesses

The paper's central motivation seems unclear. The paper's central observation follows from the fact that nonlinear losses generally do not preserve the conditional-mean solution obtained by squared loss. Although the authors provide a necessary-and-sufficient characterization of stationarity across all positive normalization floors, the practical significance of this stronger condition remains unclear. In general, training only needs stationarity at a single normalization floor, which does not require the radial balance. In particular, the experiments do not establish that the resulting non-stationarity for settings in which the PH-loss is useful might hurt performance such as in image generation. The authors also do not provide any suggestions to which their observations might yield better objective functions for consistency training.

Questions for authors
  1. Since radial balance depends on the conditional correction distribution induced by the stochastic interpolant, and is not generally guaranteed for data distributions, how can the characterization inform the design of practical consistency-training objectives?

  2. Can you help to clarify why the stronger all-floor property is important for learning accurate flow maps?

  3. What does the experiment on 240 trained neural-network states establish beyond the fact that PH and squared loss have different gradients?

2 comments

Thank you for your comments. We revised the abstract and introduction to clarify the motivation and contribution. We also significantly improved the paper's overall writing style. We agree that a nonlinear loss can shift a conditional mean. What this observation alone does not tell us is whether the shift survives multiplication by the network Jacobian and averaging over inputs. Proposition 3 gives a smooth Gaussian mixture and a parameterization for which it does: the exact flow map has zero expected squared-loss gradient but a nonzero expected pseudo-Huber (PH) gradient.

1. If the aim is to preserve the exact map, one option is to give corrections at the same input and time z=(x,t)z=(x,t) the same weight. At the exact map, the differential correction CC satisfies E[C∣z]=0\mathbb E[C\mid z]=0. Ordinary PH weights each correction by its own size, which can break this cancellation.

Sections 7 and 9 now explain a control that preserves it. When two independent corrections C1,C2C_1,C_2 can be sampled at the same zz, we use the detached PH weight from one to weight the other. Conditional independence gives

=\mathbb E[w(C_1)\mid z]\,\mathbb E[C_2\mid z].$$ At the exact map the last factor is zero. For a deterministic network, the Jacobian is fixed at $z$, so the expected parameter gradient is also zero in the differential limit. In the 100-update toy experiment in Appendix D.1, both ordinary PH and this control reduced the measured risk on average, with a slightly larger reduction for the control. We have not tested this objective for image generation. **2.** We agree that training at one chosen floor does not require radial balance. Section 4 now gives the single-floor condition immediately before Theorem 2. At a common floor $\tau$, with $J$ the network Jacobian, the condition is simply $$G_\tau=\mathbb E\left[\frac{J^\top C}{\sqrt{\lVert C\rVert^2+\tau^2}}\right]=0.$$ The reason to consider other floors is that the effective floor is $\tau=c/(a\delta)$. Even with the loss floor $c$ and target coefficient $a$ fixed, halving the time gap $\delta$ doubles $\tau$. Changing the gap can also change the corrections. Theorem 2 isolates the effect of reweighting: it keeps the joint law of $(C,J)$ fixed and varies only $\tau$. Under its integrability assumption, a zero persists throughout a neighborhood of the chosen floor exactly when $\mathbb E[J^\top C\mid\lVert C\rVert]=0$ almost surely. Otherwise the positive floors at which cancellation occurs are isolated. This is a statement about stationarity, not a requirement for learning an accurate map. In the fixed-feature model of Section 7, Eq. (10) gives sufficient conditions for one PH step to reduce squared map error; reflecting the velocity fluctuations can then make a step from the same map, at the same step size, increase that error. The image experiment does not establish that loss of stationarity hurts image quality. In fact, zero-floor normalization gave lower two-step FID in all six matched pairs (Table 8). Section 8 now explains why the theorem does not predict that comparison. **3.** The distinction here is between a change in gradient length and a change in direction. At 240 states trained with squared loss, we evaluated both gradients on held-out batches at $\tau=0.1$. Their difference was detectable at all 240. At 234, projections onto an independently fitted plane also established non-collinearity; at the other six, the direction test was inconclusive (Appendix D.2, Table 7). Section 8 now states the implication: at those 234 states, changing the scalar learning rate in plain gradient descent cannot reproduce the expected PH step from the squared-loss gradient.

I thank the review for their careful response. However, my concerns about practical training objectives for flow maps still remain unresolved and thus maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.