- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates one-step generative distillation. Existing distribution-matching and adversarial distillation methods only align output distributions. Even with perfectly matched output distributions, the student’s Jacobian response to noise inputs can still differ drastically from the teacher, which breaks downstream capabilities such as noise-based editing and seed reproducibility.
The authors propose TangentFlow, a spectral tangent distillation scheme. It offline precomputes and caches the top-k singular directions from the noise-to-image Jacobian of the teacher. During training, the student is constrained to reproduce the teacher’s Jacobian-vector product responses along these dominant singular directions via a whitening-weighted loss. No extra overhead is introduced at inference.
The paper provides theoretical guarantees: the leading singular directions preserve most of the teacher’s response energy; the whitened loss corresponds to mean squared relative response error. It also notes that whitening is only suitable for high-fidelity cached dominant directions, as it amplifies approximation errors for weaker singular directions, explaining the coupling between direction selection and weighting.
Experiments are conducted on CIFAR-10 and ImageNet-64 against baselines including DM+GAN and GAD.
Critical strengths
-
This work provides incisive mechanistic insight: output distribution alignment does not guarantee aligned input–output mappings, a critical oversight in many one-step distillation works. Jacobian singular-spectrum analysis explains the inefficiency of GAD’s random-direction sampling, with solid, inspiring theory.
-
Controlled stepwise ablations validate each design choice (direction sampling, whitening, teacher responses). All theoretical predictions are experimentally reproduced, and the separate effects of direction selection and loss weighting are clearly disentangled.
-
There is no inference overhead. SVD and caching are done offline; the student runs as fast as a vanilla one-step model. It improves both FID and geometric consistency on CIFAR-10.
Critical weaknesses
-
Evaluations are limited to small-resolution datasets. Validation on high-resolution (256/512) flow-matching or diffusion models is missing. ImageNet-64 only receives a lightweight adaptation rather than full distillation experiments, leaving its transferability to real industrial models unproven.
-
The method relies on offline latent caching: it requires prior data collection and randomized SVD for teacher Jacobians, bringing storage costs. There is no systematic analysis of how cache size and sampling distribution affect generalization, nor OOD latent robustness tests.
-
Evaluation leans heavily on automatic metrics without human studies. It remains unverified whether improved geometric metrics and lower LPIPS translate to better perceptual quality for humans.
-
Strategies to bypass offline caching, such as online estimation of the dominant Jacobian subspace, are not discussed to reduce upfront deployment cost. The scaling behavior of storage and compute overhead on larger models is also unanalyzed.
Questions for authors
-
The method is only validated on small-resolution CIFAR-10 and ImageNet-64. Could the authors add experiments to verify TangentFlow on models with resolution 256 or higher? If computational resources are constrained, please analyze and speculate on potential risks when transferring this method to large models.
-
The approach relies on offline caching over a finite latent set. How does cache set size affect geometric error on held-out samples? How does geometric consistency degrade when test latents deviate substantially from the cached distribution? Have the authors conducted tests on out-of-distribution latents?
-
While extensive metrics demonstrate geometric improvements, human evaluation is absent. Do gains in geometric consistency translate to perceptible benefits for humans? Could a small-scale pairwise human A/B test be added to compare TangentFlow against baselines on seed reuse and latent editing scenarios?
-
Offline precomputed caching becomes prohibitively expensive for high-dimensional large models. Are there ideas to eliminate offline caching by online estimation of the teacher’s dominant Jacobian singular subspace during training? Please discuss the feasibility and obstacles of such a scheme.
-
The ImageNet-64 result is only a lightweight adaptation from an existing consistency-training checkpoint. How does TangentFlow perform under full distillation from scratch? Can conclusions drawn from this lightweight adaptation generalize to full distillation?
Dear authors and dear AC:
Thank you for all the effort during the rebuttal. I can truly see the authors ran quite a lot of new experiments, and I appreciate it. I want to be very honest here about my feelings after reading everything.
W2 is well addressed. I appreciate the honesty and the effort of tuning the GAD baseline with exact JVPs, and being upfront that Proposition 1 is about geometry rather than sample quality (so the FID gains are empirical), is exactly what I hoped to see.
For W1, the ImageNet-256/512 experiments are done, but the gains look quite marginal to me: at 512, FID basically ties with the endpoint control at the best checkpoints (2.626 vs. 2.627), and the fidelity improvements are small. I also appreciate the text-to-image results added at the end of the discussion, but they come with very little explanation: how is the cache built when the Jacobian depends on the prompt, how many latents / prompts are cached, how long is the training, and what is the compute? All the gains over GAD are quite marginal. And authors did not put any images (I suppose they could be very hard to tell as well), neither for T2I nor for ImageNet-256/512, while editing and seed reuse, the most direct applications motivated in the introduction, are exactly something one would want to see.
So in the end, the strongest point of this paper seems to me to be doing the same thing with less compute / lower cost. But the paper is not positioned this way, and I did not find a clear compute comparison against GAD either.
My honest feeling is that this paper seems to me very very unfortunately just one step short.... Every point is fine, but they are just fine, every point leaks a little bit. As a reviewer I very rarely use novelty as an argument, (mainly due to as authors I hate people writing novelty problems to my own paper), but for this paper I do have a bit of this feeling. The theory is solid and I don’t see a problem with it, but solid theory does not necessarily mean that pushing this problem forward in this way is valuable. To me, the novelty of what is solved here is somehow limited, and the positioning does not quite fit what the results actually show.
So I will keep my score of 2 on paper, but I want to let AC and authors to know that : my 2 as something infinitely close to borderline. In my mind this is a completely borderline paper (there is just no borderline option to choose). I cannot push it up, but I cannot push it down either.