- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CAT-OT uses finite differences of previously evaluated velocities, while CAT-OV uses exponential moving averages of the first and second moments of the scaled velocity. Neither requires additional neural function evaluations (NFEs). The methods are motivated by a connection between conditional straight-line trajectories and gradient flow.
Critical strengths
The proposed methods are simple, lightweight, and require no additional training. The experiment shows performace gain.
Critical weaknesses
-
CAT-OV does not consistently improve CLIP/AES on Krea, most Schnell differences are small, and CAT-OT can underperform on Krea. In appendix, The O(λ2) needs to be clearly identified as a local-error statement: the displayed global bound scales linearly with λ for fixed γ.
-
the proof uses the unclipped step rule and does not directly cover the implemented fixed minimum step size. More importantly, the strictly positive lower-bound assumption in Eq. (13) cannot hold as written. Since Ak contains all positive convex combinations, a weight can approach one, making the variance arbitrarily small even for fixed, distinct, finite observations. The infimum is therefore zero. Identical support points, also permitted by the definition, give zero variance directly. The analysis needs a different condition, potentially restricted to the actual EMA trajectory.
Questions for authors
-
How should Eq. (12) account for unnormalized EMA weights, and how can the infimum in Eq. (13) be strictly positive? What corrected guarantees apply to the clipped algorithms?
-
What conditions support Eq. (8) for the marginal sampling field? Does removing (1−t) preserve the benefits, and how does the CAT-OV statistic relate to actual local error or acceleration?
2 comments
Additional response
We would like to respond to the following observation raised as a “critical weakness”:
CAT-OV does not consistently improve CLIP/AES on Krea, most Schnell differences are small, and CAT-OT can underperform on Krea.
We would like to clarify that, as stated in lines 342-343 from our paper, our main focus is on efficiency performance rather than performance on absolute score values. We understand the quoted observation is only about performance on score values, and so we will focus our following discussion on that.
First, we highlight that CAT-OV shows remarkable outperformance consistency under the CLIP/AES metrics on FLUX-1-dev, SD-3.5-large and FLUX-1-Schnell (with smaller differences on the last one, as pointed out by the reviewer), which showcases the overall outperformance of CAT-OV. Moreover, there is still noticeable improvement on FLUX-1-Krea, though less consistent as pointed out by the reviewer. We also point out that a great strength of our work is that the outperformance of CAT-OV is consistent under the HPSv3 metric across all models.
Having shown the good performance of CAT-OV across models and metrics, we now explain some caveats when it comes to using CLIP/AES for image quality assessment.
CLIP embeddings majorly capture high-level semantics instead of visual features [1,2]. Thus, both CLIP and AES metrics, which are based on CLIP embeddings, are not strictly evaluating image quality. For this reason, we included the HPSv3 metric and gave more emphasis to it. Moreover, [3,4] argue that the semantic content of a generated image is established very early in the sampling trajectory, which means that semantic improvement will be small unless external changes (e.g., guidance) are made during sampling. Consequently, our methods, which only focus on sampling efficiency and image quality improvement as reflected by HPSv3, are not guaranteed to generate large improvements on CLIP and AES metrics. Thus, we do not see this as a “weakness”, but rather as a consequence of CLIP/AES not strictly measuring image quality improvement. Nonetheless, as we mentioned, CAT-OV still has noticeable gains in the CLIP/AES metrics anyways.
Regarding the observation about FLUX-1-Schnell, this is a model specially tuned to be a “few-step model” (produce images of higher quality in 1-4 steps). Thus, it is natural that our methods, which seek to reduce the number of steps needed for general flow matching models to generate high quality images, will have a smaller quality improvement on a model which is already optimized to be “few-step”. Remarkably, CAT-OV still shows better performance under this model anyway.
The underperformance of CAT-OT for FLUX-1-Krea may be due to the sensitivity to the classifier-free guidance scale, as explained in Appendix J.2. However and most importantly, CAT-OV does not have such limitations.
References:
We are grateful to the reviewer for their positive feedback and recommendation on our work. We now respond to the reviewer’s detailed questions below.
Eq. (12) handles the unnormalized EMA weights by treating the initial value v0=g0=0 as a virtual sample, which carries the remaining weight βk, so that all weights sum to one. In addition, the norm of the EMA variance along the actual sampling trajectory is at least above 15 in our experiments with FLUX-1-dev and SD-3.5-large. A zero variance would indicate a flat path, which then would mean one could, in theory, take a large step to produce samples since there is no truncation error.
Eq. (13) is an assumption, not a derived fact, and we need it only for the states actually visited by the sampler. We find this assumption reasonable for two reasons. First, pretrained models do not have purely straight flows and using CAT-OV with larger step-sizes inexorably introduces more variance along the discretized path (Appendix N, Figure 7). Second, as mentioned earlier, we empirically found that the sampled trajectories for FLUX-1-dev and SD-3.5-large never present a norm of the EMA variance below a positive value of 15.
In short, more change of velocity in recent history leads to a larger CAT-OV statistic, which then leads to smaller step-sizes: this leads to smaller local error but a larger amount of total steps during sampling, i.e., a slower sampling. Below we provide a more detailed explanation.
The CAT-OV statistic is a proxy for the change of velocity X¨t=(d/dt)u(Xt,t;θ) across the discretization path, not a direct measurement of it. For a perfectly straight flow the velocity is constant and so X¨=0, thus, any change of velocity along the path indicates curvature. Since the CAT-OV statistic measures the discounted spread of the past values of (1−t)u, it is large when the velocity has recently been changing, which is what a large X¨t produces (u(Xt2,t2;θ)−u(Xt1,t1;θ)=∫t1t2X¨(τ)dτ for times t2<t1). Now, the Euler local error over a step-size Δt is bounded by Δt2Mk/2, where Mk bounds ∥X¨t∥ (Appendix E). Therefore, a large CAT-OV statistic gives both a smaller step-size Δt (Eq. (6)) and a smaller error bound; and a small statistic allows a larger step. The CAT-OV statistic controls this bound under the assumption in Eq. (13).
We empirically observed that the clipping is seldom activated across the sampling path (i.e., for most of the sampling path we have that Δt∈(Δmin,1−t) in Algorithms 1 and 2), with the only exceptions observed on the final step of the adaptive algorithm, so that the total integration time does not go beyond 1.
The learned vector field is the conditional vector field E[X1−X0∣Xt]. If we assume concentration of measure as a condition, e.g., by assuming that given any Xt=x there is only one learned path X1−X0 so that x=tX1+(1−t)X0, then the conditional distribution is well-represented by this representative path (there is one large mode around this representative path). This would effectively allow the conditional sampling field to be well-approximated by the marginal sampling field corresponding to Xt=x at time t. This is an extension of our summarized explanation in lines 226-229, and we will include it in the paper.
Some of our experiments while writing the paper showed that removing (1−t) leads to a small loss of performance. We did not include them in the paper since removing (1−t) does not correspond to our theoretical motivation from equation (8), and so to the formulation of our methods.