- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when normalized consistency-learning updates preserve a stationary flow map. For a fixed joint distribution of corrections and Jacobians, it proves that stationarity at every positive normalization floor is equivalent to a conditional radial-balance condition. It also develops effective-floor identities, smooth counterexamples, parameter-projection criteria, and restricted conditions for one-step improvement or harm.
Critical strengths
-
A precise and useful theoretical characterization. Theorem 2 provides a necessary-and-sufficient condition. Conditioning Jacobian-weighted corrections on their unprojected magnitude correctly distinguishes overall mean cancellation from cancellation preserved under normalization.
-
Careful treatment of the executed update. The analysis distinguishes stop-gradient algebra, conditional fields, parameter projection, optimizer processing, and teacher lag. These distinctions prevent conclusions about one object from being incorrectly transferred to another.
-
Supporting analysis is good. The smooth Gaussian-mixture witness establishes the phenomenon concretely.
Critical weaknesses
-
The connection to actual training trajectories remains limited. The central theorem fixes the correction–Jacobian law while varying the floor. During training, this law, the teacher, and the optimizer all change. The paper provides extensions and sufficient conditions, but does not verify the relevant trajectory premises in the image experiments. The result is therefore a stationarity characterization rather than a convergence guarantee.
-
The strongest image experiment does not directly test the theorem’s positive-floor regime. The six-seed comparison uses zero-floor normalization, explicitly outside the positive-floor premise of Theorem 2. A positive-floor sweep would provide a more direct bridge between the main mathematical result and image-generation behavior.
Questions for authors
See weaknesses.
2 comments
Thank you for the response. I appreciate the authors' explanation and acknowledging the limitation.
From the theoretical perspective, I feel the studied problem is interesting. Though I am not an expert directly in this area, I have explored consistency learning on image tasks. The use of pseudo-Huber loss or adaptive normalization would largely improve the performance, but their mechanism remain unclear. This paper provides a good theoretical study on this point.
That said, though acknowledging the limitation in the rebuttal, i) the connect to real training trajectory and ii) the transition between image experiments and the main math remains, which I think is also important in practice.
Considering the above, I would like to keep the rating unchanged.
Thank you for your careful review. We agree with both limitations. The final paragraph of the introduction now states that Theorem 2 concerns one training state and that the six-seed image comparison uses a zero floor, outside the theorem. We also clarified the trajectory assumptions in Section 9 and the interpretation of Table 8 in Section 8 and Appendix D.5.1.
1. Connection to training trajectories. Theorem 2 fixes the joint law of the correction and the network Jacobian and varies only the normalization floor. During training, the model, teacher, and optimizer state evolve, and the correction–Jacobian law can change. The theorem therefore characterizes stationarity at a given state; it gives no convergence guarantee.
Proposition 6 (Appendix A.1) gives a narrower result along one trajectory. Let D(ξ) be the difference between the expected PH and linear gradients at the same training state ξ, with the two scores matched in slope at zero. Suppose its projection onto a fixed unit direction u starts at u⊤D(ξ0)≥m>0 and is L-Lipschitz in the state metric d over the trajectory segment. Then
u⊤D(ξk)≥m−L∑j<kd(ξj+1,ξj).
The initial sign persists while the total change in state is small enough. Both gradients are evaluated at each state on the same trajectory; the bound does not predict how two runs trained with different losses will separate. As Section 9 now states explicitly, we did not verify the Lipschitz or displacement conditions in the image runs.
2. Positive floors in the image experiments. The six-seed comparison used c=0. We chose this setting after finding that 82% of the measured residual norms at a trained ECM checkpoint were below 0.1, so a floor of 0.1 would weaken normalization for many examples (Appendix D.5.1). Starting from one pretrained EDM checkpoint, zero-floor normalization gave lower two-step FID-50k than squared loss in all six pairs, by 2.33 points on average (Table 8). The realized update lengths were matched, but each arm kept its own optimizer moments. This measures image quality under that training protocol; it does not test Theorem 2 or identify loss of stationarity as the reason for the FID difference.
The existing positive-floor results are more limited. Earlier continuations from a pretrained consistency model used c=0.03 or 0.1 and gave mixed rankings under different protocols (Appendix D.5.2). At one frozen checkpoint with c=0.03, we could not distinguish PH from its slope-matched linear control in the three-dimensional subspace measured (Table 10).
We agree that a positive-floor sweep would be informative. Varying the floor at a frozen checkpoint, with the teacher and sampling law fixed, would measure how the expected gradient changes under the theorem’s assumptions. Repeating the paired training comparison at several positive floors would address the separate question of image quality. We did not run a controlled positive-floor sweep of the six-seed experiment, and the revised Section 8 distinguishes this missing comparison from the positive-floor results already in the paper.