acceptodds
Under review as a conference paper at ICLR 2027

TangentFlow: Spectral Tangent Distillation for One-Step Generative Models

Abstract

One-step generative distillation trains a student to replace a multi-step teacher with a single network evaluation. We study whether explicitly matching selected teacher Jacobian responses can improve the student's geometric fidelity while maintaining image quality. We introduce TangentFlow, a spectral tangent objective that caches the leading singular directions of the teacher's noise-to-image Jacobian and trains the student to match the corresponding responses with spectral weighting. Across CIFAR-10 and ImageNet-64, TangentFlow achieves the lowest geometric error among the evaluated methods on all six measures, including a 40% reduction in retained-subspace response error on held-out CIFAR-10 latents relative to the matched distribution-matching and adversarial baseline (three seeds) and an 8% reduction on held-out ImageNet-64 latents. It reduces Fréchet Inception Distance (FID) by 21% relative to the matched CIFAR-10 baseline and by 3.8% relative to distribution matching on ImageNet-64. Our analysis motivates direction selection through teacher-response energy retention, establishes relative-error bounds on the selected subspace, shows that the whitened loss equals the average relative response error at every error level, and shows that cached-target error makes whitening reliable only on leading directions, which explains the observed interaction between direction selection and weighting. These results support spectral tangent supervision as a way to improve geometric fidelity while maintaining image quality.

Primary area:
generative models
Keywords:
One-step generative distillation, Diffusion models, Jacobian Matching, Spectral Direction Selection, Teacher–Student Learning, Geometric Fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LdozOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates one-step generative distillation. Existing distribution-matching and adversarial distillation methods only align output distributions. Even with perfectly matched output distributions, the student’s Jacobian response to noise inputs can still differ drastically from the teacher, which breaks downstream capabilities such as noise-based editing and seed reproducibility.

The authors propose TangentFlow, a spectral tangent distillation scheme. It offline precomputes and caches the top-k singular directions from the noise-to-image Jacobian of the teacher. During training, the student is constrained to reproduce the teacher’s Jacobian-vector product responses along these dominant singular directions via a whitening-weighted loss. No extra overhead is introduced at inference.

The paper provides theoretical guarantees: the leading singular directions preserve most of the teacher’s response energy; the whitened loss corresponds to mean squared relative response error. It also notes that whitening is only suitable for high-fidelity cached dominant directions, as it amplifies approximation errors for weaker singular directions, explaining the coupling between direction selection and weighting.

Experiments are conducted on CIFAR-10 and ImageNet-64 against baselines including DM+GAN and GAD.

Critical strengths
  1. This work provides incisive mechanistic insight: output distribution alignment does not guarantee aligned input–output mappings, a critical oversight in many one-step distillation works. Jacobian singular-spectrum analysis explains the inefficiency of GAD’s random-direction sampling, with solid, inspiring theory.

  2. Controlled stepwise ablations validate each design choice (direction sampling, whitening, teacher responses). All theoretical predictions are experimentally reproduced, and the separate effects of direction selection and loss weighting are clearly disentangled.

  3. There is no inference overhead. SVD and caching are done offline; the student runs as fast as a vanilla one-step model. It improves both FID and geometric consistency on CIFAR-10.

Critical weaknesses
  1. Evaluations are limited to small-resolution datasets. Validation on high-resolution (256/512) flow-matching or diffusion models is missing. ImageNet-64 only receives a lightweight adaptation rather than full distillation experiments, leaving its transferability to real industrial models unproven.

  2. The method relies on offline latent caching: it requires prior data collection and randomized SVD for teacher Jacobians, bringing storage costs. There is no systematic analysis of how cache size and sampling distribution affect generalization, nor OOD latent robustness tests.

  3. Evaluation leans heavily on automatic metrics without human studies. It remains unverified whether improved geometric metrics and lower LPIPS translate to better perceptual quality for humans.

  4. Strategies to bypass offline caching, such as online estimation of the dominant Jacobian subspace, are not discussed to reduce upfront deployment cost. The scaling behavior of storage and compute overhead on larger models is also unanalyzed.

Questions for authors
  1. The method is only validated on small-resolution CIFAR-10 and ImageNet-64. Could the authors add experiments to verify TangentFlow on models with resolution 256 or higher? If computational resources are constrained, please analyze and speculate on potential risks when transferring this method to large models.

  2. The approach relies on offline caching over a finite latent set. How does cache set size affect geometric error on held-out samples? How does geometric consistency degrade when test latents deviate substantially from the cached distribution? Have the authors conducted tests on out-of-distribution latents?

  3. While extensive metrics demonstrate geometric improvements, human evaluation is absent. Do gains in geometric consistency translate to perceptible benefits for humans? Could a small-scale pairwise human A/B test be added to compare TangentFlow against baselines on seed reuse and latent editing scenarios?

  4. Offline precomputed caching becomes prohibitively expensive for high-dimensional large models. Are there ideas to eliminate offline caching by online estimation of the teacher’s dominant Jacobian singular subspace during training? Please discuss the feasibility and obstacles of such a scheme.

  5. The ImageNet-64 result is only a lightweight adaptation from an existing consistency-training checkpoint. How does TangentFlow perform under full distillation from scratch? Can conclusions drawn from this lightweight adaptation generalize to full distillation?

Reviewer_woa2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TangentFlow, a one-step image generation method based on spectral tangent distillation, aiming to address the difficulty of conventional generative distillation methods in preserving local geometric consistency between teacher and student models. Specifically, the proposed method employs randomized singular value decomposition (SVD) to extract and cache the leading singular directions of the teacher's Jacobian, and introduces a spectrally whitened loss to supervise the student's responses along these key directions. In this way, TangentFlow improves generation quality and geometric fidelity between the teacher and student models while maintaining the efficiency of one-step generation.

Critical strengths

The paper addresses an important limitation of existing one-step generative distillation methods by explicitly preserving teacher-student geometric consistency. The combination of spectral direction selection and Jacobian response matching is technically meaningful. The theoretical analysis is relatively comprehensive, and the experiments demonstrate promising improvements in both FID and geometric fidelity.

Critical weaknesses

The novelty of the proposed method is somewhat limited, as it mainly combines existing Jacobian matching and spectral decomposition techniques. The justification for selecting the top four singular directions remains insufficient, and systematic sensitivity analyses of the cache rank and supervision budget are lacking. Moreover, the relationship between geometric consistency improvements and generation quality is not fully explained, particularly on ImageNet-64.

Questions for authors
  1. Why does the manuscript select the top four singular directions for supervision rather than adopting a sampling strategy based on cumulative spectral energy?
  2. In the ImageNet experiments, why does the error along the leading singular directions show only marginal improvement, while the overall FID still decreases? The discrepancy between the improvements in geometric consistency and generation quality has not been sufficiently explained.
  3. The experimental section lacks a comprehensive sensitivity analysis of the cache rank (K) and the number of supervised directions (q).
  4. The manuscript selects the directions corresponding to the largest singular values of the Jacobian for distillation. However, directions associated with smaller singular values may also significantly influence image structure, semantic content, and other important attributes. This issue has not been sufficiently investigated or validated.
Reviewer_D81JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TangentFlow, a one-step distillation method that aims to make the student’s noise-to-image mapping faithful to the teacher’s, rather than only matching its output distribution. GAD matches teacher and student JVP along random directions. TangentFlow instead precomputes the leading singular directions of the teacher’s Jacobian offline and trains the student to match the teacher’s responses along them, normalizing each error by the size of the teacher’s response. The authors provide theoretical analysis motivating both the choice of directions and the normalization. On CIFAR-10 and ImageNet-64, the method improves FID over distribution-matching baselines (e.g., 2.69 → 2.12 on CIFAR-10) and achieves the lowest teacher–student geometric discrepancy among the compared methods. Ablations attribute the gain mainly to the choice of directions.

Critical strengths

Well-motivated problem and an elegant method. Matching the student’s local noise-to-image behavior, not just its outputs, is a meaningful goal for one-step distillation (especially for some editing tasks). Supervising the teacher’s leading singular directions is a principled way to spend a small tangent budget, and estimating them offline with a randomized SVD and caching them for reuse makes this practical.

Theory and experiments support each other. The analysis makes concrete predictions, notably that normalization helps on the leading directions but hurts on weak ones. Well-controlled ablations across several studies confirm these predictions.

Good results on CIFAR-10. TangentFlow clearly improves FID over the matched baseline (2.69 → 2.12), and the gain is well beyond the variation across seeds.

Critical weaknesses

1 Limited experimental scale. Full training is done only on CIFAR-10. The ImageNet-64 experiment is a short 100-update adaptation with a DM-only base, and its gains are quite small.... I strongly encourage the authors to evaluate at a larger scale, such as higher resolutions or text-to-image generation, where GAD's paper was evaluated. This matters for two reasons. First, the uses that motivate the paper, such as previewing with the student and rendering the same seeds with the teacher, or editing by perturbing the noise, are most relevant in such settings, and the paper does not demonstrate them. Second, the cost and accuracy of the cached spectral targets may change considerably with dimension, so whether the design still works there needs to be shown empirically.

2 TangentFlow is essentially a cheaper, more targeted form of GAD’s Jacobian matching, so its better FID needs an explanation. Table 13 shows that each step from GAD to TangentFlow lowers FID. However, the paper gives no clear mechanism for why matching the leading directions improves image quality, and Proposition 1 concerns retained Jacobian energy, not sample quality. The GAD baseline also appears weak in this reproduction. In every study its FID is no better than that of its own base (e.g., 3.12 vs. 2.69 on CIFAR-10), even though it reduces to the base as its weight goes to zero. It also lowers recall, contrary to the diversity gains reported by GAD. Since no method’s tangent weight is tuned, a tuned comparison, ideally in GAD’s original setting, would make the conclusion more convincing.

3 Most of the geometry measures are computed with the same approximate spectral factors used for training, so they are closely related to the objective that TangentFlow optimizes. The one measure independent of the cache, the endpoint distance to the teacher, improves more modestly: about 12–14% on CIFAR-10 and 3% on ImageNet-64 (it decrease when the diemsion increase?). Additional measures that do not depend on the cache, such as paired perceptual distance to the teacher averaged over many seeds, would help confirm that the student is more faithful to the teacher.

Minor:

In Table 1, many entries in the control rows end in zero at the displayed precision (e.g., 4.620, 1.310, 82.60), unlike the base and TangentFlow rows. This appears to me more than just random, could the authors clarify how these numbers were obtained?

Questions for authors

See my major concerns.

2 comments

Dear authors and dear AC:

Thank you for all the effort during the rebuttal. I can truly see the authors ran quite a lot of new experiments, and I appreciate it. I want to be very honest here about my feelings after reading everything.

W2 is well addressed. I appreciate the honesty and the effort of tuning the GAD baseline with exact JVPs, and being upfront that Proposition 1 is about geometry rather than sample quality (so the FID gains are empirical), is exactly what I hoped to see.

For W1, the ImageNet-256/512 experiments are done, but the gains look quite marginal to me: at 512, FID basically ties with the endpoint control at the best checkpoints (2.626 vs. 2.627), and the fidelity improvements are small. I also appreciate the text-to-image results added at the end of the discussion, but they come with very little explanation: how is the cache built when the Jacobian depends on the prompt, how many latents / prompts are cached, how long is the training, and what is the compute? All the gains over GAD are quite marginal. And authors did not put any images (I suppose they could be very hard to tell as well), neither for T2I nor for ImageNet-256/512, while editing and seed reuse, the most direct applications motivated in the introduction, are exactly something one would want to see.

So in the end, the strongest point of this paper seems to me to be doing the same thing with less compute / lower cost. But the paper is not positioned this way, and I did not find a clear compute comparison against GAD either.

My honest feeling is that this paper seems to me very very unfortunately just one step short.... Every point is fine, but they are just fine, every point leaks a little bit. As a reviewer I very rarely use novelty as an argument, (mainly due to as authors I hate people writing novelty problems to my own paper), but for this paper I do have a bit of this feeling. The theory is solid and I don’t see a problem with it, but solid theory does not necessarily mean that pushing this problem forward in this way is valuable. To me, the novelty of what is solved here is somehow limited, and the positioning does not quite fit what the results actually show.

So I will keep my score of 2 on paper, but I want to let AC and authors to know that : my 2 as something infinitely close to borderline. In my mind this is a completely borderline paper (there is just no borderline option to choose). I cannot push it up, but I cannot push it down either.

We sincerely thank you for your thoughtful and honest final assessment, and for the many suggestions throughout the discussion. Your requests for higher resolution, a tuned GAD baseline, cache-free measures and text-to-image results substantially strengthened the paper. We address the remaining points briefly.

The geometric improvements are substantial. We feel the geometric gain is substantial, as both the numbers and our human respondents show. TangentFlow aims at a student that is more faithful to its teacher at the same image quality, and three features of the results make the gains substantial rather than marginal.

1. They are large where they matter most. At ImageNet-512, with FID tied with Endpoint (2.626 vs 2.627):

  • edit cosine to the teacher is 27% higher (0.288 vs 0.226; GAD 0.238), so editing the noise moves the image far more like the teacher does, the seed-reuse and editing property motivated in the introduction;
  • Jacobian error on untrained directions is 20% lower (4.02 vs 5.05), and endpoint error 10% lower (66.5 vs 73.5).

At the same training length, untrained-direction error at ImageNet-512 is 26–27% lower than both baselines (2.78 vs 3.78–3.82), and on held-out CIFAR-10 latents the supervised-direction error falls by 39% (1.615 → 0.981).

2. They generalise beyond what is trained. The largest gains are on directions that TangentFlow never supervises, and they hold against the teacher's exact Jacobian, with no cache or approximation. TangentFlow is closer on relative error, cosine, angle and endpoint. On 1,000 out-of-distribution latents it wins all 12 paired comparisons, on 85–98% of latents.

3. They are consistent and significant everywhere.

  • Higher resolution: at ImageNet-256, Jacobian errors are 9–19% lower than both baselines.
  • Long training: after long ImageNet-64 training, untrained-direction error is 3.001 vs 3.126–3.226.
  • From scratch: TangentFlow beats GAD on all ten geometric measures, each with a paired 95% confidence interval excluding zero (for example, untrained-direction error 5.07 vs 5.52, exact-Jacobian cosine 0.459 vs 0.434).
  • Text-to-image: it adds a further 63% to GAD's own gain over the base in seed teacher alignment, 29% in self-identifiability and 22% in layout AP, and lowers FID to the teacher by 2.1–2.2%, with image quality unchanged.
  • Paired comparisons: all 15 paired edit and interpolation comparisons favour TangentFlow.
  • Human study: in our pairwise study, on average 23% more respondents preferred TangentFlow's outputs over the baselines'.

Compute. At ImageNet-256, GAD's exact JVPs cost about 33 s of teacher solves per batch, about 46 GPU-hours over a training run. The same TangentFlow run, including its one-time cache, takes 4.5 GPU-hours: roughly a tenth of the compute, with no cost at inference.

Text-to-image details. We use GAD's published pipeline unchanged, including its hardware and evaluation protocol (MS-COCO and DrawBench):

  • PixArt-α, four-step TDM: 2K iterations, batch 64 per GPU, learning rate 4×10⁻⁶, FP16, on RTX 4090 GPUs.
  • SD 2.1, one-step LADD: 100K iterations, batch 64 per GPU, learning rate 1×10⁻⁶, BF16, on A800 GPUs.

Since the Jacobian depends on the prompt, the cache is built per (prompt, latent) pair. Re-running the base methods and GAD with this pipeline reproduces their published results, so the comparison is like-for-like. We will add the cache size and the full run details for both models to the paper, with the run details reported in the same format as the GAD paper.

In summary, at the same or better image quality (FID 2.626 vs 2.627–2.734 at ImageNet-512, 2.091 vs 2.222–2.245 at ImageNet-256, 4.19 vs 4.36 from scratch), TangentFlow gives a substantial geometric improvement (27% higher edit transfer and 20% lower untrained-direction error at ImageNet-512), which is the central claim our analysis targets.

If you feel these points address your remaining concerns, we would be grateful if you would consider revisiting your assessment.

Regardless of the outcome, it was a pleasure to hear your opinion, and we sincerely appreciate your time and thoughtful engagement throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.