acceptodds
Under review as a conference paper at ICLR 2027

Future Discrepancy Knowledge Distillation for Diffusion Model Compression

Abstract

Diffusion model capacity distillation trains a compact Student to imitate a high-capacity Teacher, reducing the computation required for each network evaluation during generation. Classic knowledge distillation (KD) typically matches their current predictions at states independently constructed by noising real data. During generation, however, the current state is produced recursively through preceding model predictions and state updates. Different Teacher and Student sampling histories can therefore yield different states for local supervision. Under a common local objective, we compare Teacher- and Student-rollout states and find that their relative performance depends on the Teacher–Student configuration. Even with the training-state source fixed, local prediction matching measures only the immediate discrepancy without accounting for its evolution during subsequent sampling. Local errors of similar magnitude can produce different future state deviations under shared reference dynamics. To address this gap, we introduce Future Discrepancy (FD). Starting from a common state, the Teacher and Student each perform one state update. We then propagate the two next states with the same frozen Teacher and use their discrepancy at a selected future position to supervise the current Student state update. FD applies to either Teacher- or Student-rollout states. We use Temporal Sampling to reduce Teacher continuation queries during training and combine FD with complementary classic KD on independently noised data states to form Future-Discrepancy Knowledge Distillation (\method). Experiments across Student construction strategies, Teacher–Student configurations, and image generation tasks demonstrate the gains of FD over local supervision using the same state source and the effectiveness of the complete FD-KD method.

Primary area:
generative models
Keywords:
Diffusion Models, Knowledge Distillation, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_54UZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a previously unexplored question in knowledge distillation: whether all points should be weighted equally when supervising the discrepancy between the student and teacher models. It argues that weights should instead be assigned according to how much the current prediction error will affect the future generation process. Experiments on datasets including ImageNet and COCO demonstrate the effectiveness of the proposed method compared with other baselines.

Critical strengths
  • The question studied in this paper is relatively novel and worth investigating: when supervising the discrepancy between the student and teacher models, greater weight should be assigned to points that will have a larger future impact. The proposed FD loss provides an intuitive answer to this question.
  • The paper provides a mathematical interpretation of the FD loss and theoretically discusses the essence of the method: it is equivalent to replacing the Euclidean distance used in standard KD with a local quadratic metric determined by the teacher dynamics.
Critical weaknesses
  • Computing the future rollout of the teacher model requires gradient computation, which results in a very large computational graph and substantially slows down training. Moreover, because the teacher is itself a relatively large model, multi-step sampling is already slow. This may limit the scalability of the method.
Questions for authors

See “Critical Weaknesses.”

3 comments
AuthorsOpenReview ↗

Author Response

Dear Reviewer 54UZ,

We thank the reviewer for recognizing the value of evaluating current errors through their future effects and the interpretation of FD as a local quadratic metric induced by Teacher dynamics.

Q1. Computing the future rollout of the teacher model requires gradient computation, which results in a very large computational graph and substantially slows down training. Moreover, because the teacher is itself a relatively large model, multi-step sampling is already slow. This may limit the scalability of the method.

Teacher continuation adds training computation. We address it by clarifying its gradient scope, then examining training cost and quality gains across Teacher sizes.

1. Gradients pass through the current Student update and short Teacher continuation. The preceding sampling history is excluded from backpropagation, and the Teacher reference branch provides a fixed target. On the Student branch, the frozen Teacher retains input-state derivatives to transmit the future discrepancy back to the current Student update. Thus, each query spans one Student update and hh Teacher updates. Query gradients are computed separately and accumulated before one Student parameter update. With the default h=1h=1, R=3R=3, this requires 1-step continuation at three separately processed positions.

2. Temporal sampling reduces training cost while retaining FD gains. Alongside the short horizon, temporal sampling (TS) reduces supervised positions and also applies to TPD and OPD. Table 15 separates these effects for SiT-B to SiT-B-D6 at the same update budget, using h=1h=1 for FD and stratified sampling with R=3R=3 for TS. Checkmarks denote enabled components; Cost measures a complete iteration in s/iter. Parentheses show changes from each group’s dense local baseline, and bold marks group-wise best results.

MethodFDTSFID ↓IS ↑Cost ↓
TPD16.09109.491.18
FD(T)✓15.07 (−1.02)112.94 (+3.45)1.60 (+0.42)
TPD✓16.62 (+0.53)107.44 (−2.05)0.63 (−0.55)
FD(T)✓✓15.73 (−0.36)111.62 (+2.13)0.79 (−0.39)
OPD15.64119.071.36
FD(S)✓14.29 (−1.35)125.57 (+6.50)1.79 (+0.43)
OPD✓16.09 (+0.45)118.08 (−0.99)0.55 (−0.81)
FD(S)✓✓14.89 (−0.75)124.92 (+5.85)0.72 (−0.64)

At matched sampling settings, FD improves FID and IS over both local baselines, with similar gains under dense supervision and TS. Meanwhile, TS lowers Cost for all four methods with modest quality changes. Combining FD and TS therefore preserves these gains and outperforms the dense local baselines in FID, IS, and per-iteration Cost at the same update budget.

3. Quality gains extend across Teacher sizes. To examine scalability, Figure 5 and Table 16 fix the SiT-S Student and vary the Teacher from SiT-B to SiT-XL. All configurations use 25K updates, the same global rollout batch, and TS with R=3R=3; FD uses h=1h=1 and shared Teacher continuation. Parentheses and bold indicate changes and best results within each local/FD pair. Cost is reported in s/iter for two RTX 4090 GPUs.

TeacherMethodFID ↓IS ↑Cost ↓
SiT-BTPD14.12114.310.560
SiT-BFD(T)13.87 (−0.25)115.26 (+0.95)0.787 (+0.227)
SiT-BOPD13.34126.310.654
SiT-BFD(S)12.84 (−0.50)128.53 (+2.22)0.774 (+0.120)
SiT-LTPD12.21130.341.302
SiT-LFD(T)11.90 (−0.31)130.58 (+0.24)2.511 (+1.209)
SiT-LOPD12.10144.011.114
SiT-LFD(S)11.38 (−0.72)149.40 (+5.39)1.954 (+0.840)
SiT-XLTPD15.65112.521.850
SiT-XLFD(T)13.78 (−1.87)119.37 (+6.85)3.860 (+2.010)
SiT-XLOPD21.88109.431.903
SiT-XLFD(S)17.51 (−4.37)125.35 (+15.92)2.926 (+1.023)

FD improves FID and IS at every tested Teacher size, with the largest FID reductions over local baselines for SiT-XL: 1.87 for FD(T) and 4.37 for FD(S). Their measured per-GPU Mem is 17.25 and 17.05 GB, respectively. These results demonstrate training feasibility and quantify the quality–Cost trade-off as Teacher size increases.

4. The additional computation of FD is confined to training. FD-KD’s additional supervision and Teacher continuation leave inference unchanged: the compact Student retains its original architecture and sampling procedure, with no additional parameters or network evaluations.

Revision in paper. Sections 3.5 and 4.2.5; Appendices B.2 and B.4; Appendices D.4 and D.5, Tables 15 and 16 (gradient scope, inference, and quality–Cost comparisons).

Thank you for the response. Most of my concerns have been addressed, so I will maintain my positive score.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for your follow-up and positive assessment. We appreciate your careful review and constructive feedback, which helped us clarify the method and strengthen the manuscript.

Reviewer_aFVfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Future-Discrepancy Knowledge Distillation (FD-KD) for diffusion model capacity distillation. Conventional knowledge distillation directly matches the predictions of the Teacher and Student models, potentially leading to a distribution mismatch during the Student's iterative sampling process. To address this limitation, FD-KD introduces future-state discrepancy supervision, which measures how the current prediction error propagates through subsequent sampling steps under shared Teacher dynamics. To reduce the additional computational overhead, the authors propose a temporal sampling strategy for more efficient training. Experiments on image generation tasks demonstrate the effectiveness of FD-KD.

Critical strengths
  • The proposed method is conceptually simple, intuitive, and easy to follow.
  • The paper presents an interesting attempt to incorporate Teacher-guided future trajectory supervision into diffusion model distillation.
Critical weaknesses
  • The performance gap between the compressed Student and the original Teacher remains noticeable. In Table 6, the reduced Recall suggests a potential loss of generation diversity, while Table 7 shows that the CLIP scores of the distilled models remain below those of the Teacher. These results raise concerns about how well the Student preserves the Teacher's generation capabilities.
  • Future-state supervision introduces additional computational overhead. Although temporal sampling reduces the training cost, it introduces a trade-off between computational efficiency and generation quality.
Questions for authors
  • Could the authors include additional evaluation metrics commonly used in text-to-image generation, such as ImageReward and HPSv3, to provide a more comprehensive assessment of perceptual quality and human preference alignment?
  • In Table 3, why does increasing the future supervision horizon not consistently improve generation quality? In particular, what explains the performance degradation when the continuation horizon increases to h=3h=3?
5 comments
AuthorsOpenReview ↗

Author Response 1

Dear Reviewer aFVf,

We thank the reviewer for the thoughtful feedback and for recognizing the simplicity and intuition of FD-KD, as well as the value of Teacher-guided future supervision. Your questions help sharpen the assessment of capability preservation, training cost, and the role of the continuation horizon. We address both weaknesses and both questions separately below.

Q1. The performance gap between the compressed Student and the original Teacher remains noticeable. Results in Table 6 and 7 raise concerns about how well the Student preserves the Teacher's generation capabilities.

Thank you for highlighting the preservation of the Teacher's generation capabilities. We agree that Recall and CLIP scores indicate a remaining gap between the compressed Students and the Teacher. However, capacity distillation aims to preserve as much of a larger Teacher's generation capability as possible in a smaller Student. The Teacher's performance provides a reference for capability preservation, while comparisons under the same Student architecture reveal differences in the effectiveness of distillation methods.

Specifically, in Table 6, FD-KD(T) achieves Recall 0.55, matching TinyFusion and LIFT+PLACE, while improving FID from their respective 2.86 and 2.79 to 2.61. This demonstrates improved generation quality while maintaining generation coverage. FD-KD(S) further achieves an FID of 2.59 while maintaining comparable Recall to the other Student methods (0.53 versus 0.55). For text alignment, Table 7 shows that both Base and Tiny FD-KD configurations achieve higher CLIP scores than BK-SDM and LIFT+PLACE with the same architectures, indicating better preservation of the Teacher's text-alignment capability.

The additional evaluations in Table 7 further show that these FD-KD configurations outperform their architecture-matched BK-SDM baselines on ImageReward, HPSv3, DeQA, VQAScore, and ImageDoctor Overall; detailed results and metric implementations are provided in our response to Q3 below. Together, these results support better preservation of the Teacher's generation capabilities with FD-KD.

Revision in paper. Section 4.3, Tables 6 and 7 (benchmark and additional evaluation results), and Appendix E (metric details).


Q2. Future-state supervision introduces additional computational overhead. Although temporal sampling reduces the training cost, it introduces a trade-off between computational efficiency and generation quality.

Thank you for highlighting training efficiency. Teacher continuation introduces additional training computation in FD. FD and temporal sampling (TS) serve distinct roles: FD provides supervision through future-state discrepancies, whereas TS reduces the number of temporal positions at which supervision is evaluated along each trajectory. TS also applies to local supervision such as TPD and OPD. We therefore examine the quality gains from FD separately from the quality–cost trade-off introduced by TS.

The joint ablation below, from Table 15, uses SiT-B to SiT-B-D6 with the same number of training updates. FD uses h=1h=1, and TS uses stratified sampling with R=3R=3. Checkmarks indicate that FD or TS is enabled. Cost covers a complete training iteration in s/iter. Parentheses report changes from the dense TPD/OPD baseline in the first row of each group.

MethodFDTSFID ↓IS ↑Cost ↓
TPD16.09109.491.18
FD(T)✓15.07 (−1.02)112.94 (+3.45)1.60 (+0.42)
TPD✓16.62 (+0.53)107.44 (−2.05)0.63 (−0.55)
FD(T)✓✓15.73 (−0.36)111.62 (+2.13)0.79 (−0.39)
OPD15.64119.071.36
FD(S)✓14.29 (−1.35)125.57 (+6.50)1.79 (+0.43)
OPD✓16.09 (+0.45)118.08 (−0.99)0.55 (−0.81)
FD(S)✓✓14.89 (−0.75)124.92 (+5.85)0.72 (−0.64)

With the sampling setting held fixed, FD improves FID and IS over the corresponding local baseline for both trajectory sources. These gains remain comparable under dense supervision and TS: the FID improvements are 1.02 and 0.89 for Teacher-trajectory states, and 1.35 and 1.20 for Student-trajectory states, respectively. Thus, FD retains its main quality gains even when fewer temporal positions are evaluated.

With the supervision objective held fixed, TS substantially reduces cost for TPD, OPD, and both FD configurations, with modest changes in generation quality. Combining FD with TS also yields better FID and IS at lower per-iteration Cost than the corresponding dense local baseline at the same update budget.

Additionally, FD and TS are used only during training. Inference retains the original compact Student and sampling procedure, with no additional parameters or network evaluations.

Revision in paper. Section 4.2.4 and Appendix D.4, Table 15 (joint FD/TS comparison).

AuthorsOpenReview ↗

Author Response 2

Q3. Could the authors include additional evaluation metrics commonly used in text-to-image generation, such as ImageReward and HPSv3, to provide a more comprehensive assessment of perceptual quality and human preference alignment?

Thank you for this suggestion. We have added ImageReward and HPSv3 evaluations on MS-COCO to assess alignment with human preferences. To broaden the dimensions covered by the evaluation, we additionally use DeQA for perceptual image quality, VQAScore for semantic alignment, and ImageDoctor for overall impression.

We conduct these evaluations using the official released evaluation code and documented model options, with implementation details provided in Appendix E. Each of the five additional metrics is averaged over the 30,000 images generated by the corresponding model. The table below presents the Teacher, BK-SDM, and FD-KD results from Table 7. Students are grouped by architecture. Bold marks the best result within each Student architecture, and ImageDoctor reports its Overall score.

MethodFID ↓IS ↑CLIP ↑ImageReward ↑HPSv3 ↑DeQA ↑VQAScore ↑ImageDoctor ↑
SD2.1 Teacher13.9335.930.30750.34466.74143.93660.57980.5715
BK-Base-v2 Student
BK-SDM15.8531.700.2868-0.15062.11473.81320.41890.5407
FD-KD(T)14.6332.010.2902-0.10782.19873.84460.43280.5482
FD-KD(S)14.5632.010.2902-0.10112.20163.84110.43080.5496
BK-Tiny-v2 Student
BK-SDM15.6831.640.2897-0.15601.74163.77240.43750.5367
FD-KD(T)13.7932.860.2937-0.10852.10873.80640.45250.5504

Both Base FD-KD configurations and Tiny FD-KD(T) outperform their architecture-matched BK-SDM baselines on ImageReward and HPSv3, showing consistent gains on the two requested learned human-preference metrics. DeQA, VQAScore, and ImageDoctor Overall also improve across these configurations, providing complementary evidence of better perceptual quality, semantic alignment, and overall impression. The results across both Student sizes thus extend the evidence for FD-KD beyond the original generation metrics.

Complementing these automatic metrics, the blinded user study in Section 4.4 and Table 8 involves 25 participants, each assigned 128 formal image-pair comparisons, one per randomly sampled MS-COCO prompt. Within the BK-Base-v2 architecture, FD-KD(S) achieves win rates of 53.40% for visual quality and 53.71% for text alignment against BK-SDM, showing a modest human preference advantage consistent with the automatic evaluation.

Revision in paper. Section 4.3, Table 7 and Appendix E (automatic evaluation); Section 4.4, Table 8 and Appendix G (user study).

AuthorsOpenReview ↗

Author Response 3

Q4. In Table 3, why does increasing the future supervision horizon not consistently improve generation quality? In particular, what explains the performance degradation when the continuation horizon increases to (h=3)?

Thank you for the question. A longer horizon provides more future information, but its usefulness depends on the accuracy of the Teacher continuation. Prediction errors may propagate and accumulate over successive Teacher updates, weakening the resulting supervision. This offers a possible explanation for why extending the horizon does not consistently improve generation quality.

To examine supervision effectiveness, we added the local-correction diagnostic in Appendix C.5. We compare OPD (h=0h=0) and FD (h=1,2,3h=1,2,3) on the same Student at common valid positions. Each objective applies a small correction to the Student next state along its negative gradient. The Student then completes sampling without weight updates, and we evaluate latent endpoint MSE against the same fine-step Teacher reference. We define UhU_h as the percentage reduction in mean endpoint MSE relative to uncorrected sampling; larger values indicate more effective local corrections. The table below combines the generation metrics and training cost from Table 3 with the diagnostic results from Appendix C.5, Table 10. Cost is measured in s/iter, and bold marks the best result in each column.

hhFID ↓IS ↑Prec. ↑Rec. ↑Cost ↓UhU_h (%) ↑
0 (OPD)15.64119.070.740.311.360.5252
114.29125.570.760.301.790.6409
214.10126.480.770.302.220.6414
315.42118.470.740.312.610.5889

In the local-correction diagnostic, short-horizon FD improves endpoint matching over OPD. The improvement is similar for h=1h=1 and h=2h=2, with only a slight increase at h=2h=2, and decreases at h=3h=3. These results show that extending the horizon does not necessarily yield more effective local supervision, complementing the non-monotonic generation results in Table 3. In the training comparison, h=2h=2 offers modest quality gains over h=1h=1 while increasing Cost from 1.79 to 2.22 s/iter. Extending to h=3h=3 further increases Cost and worsens generation quality. We therefore use h=1h=1 in subsequent experiments as a quality–cost trade-off.

Revision in paper. Section 4.2.3 and Table 3; Appendix C.5 and Table 10 (local-correction diagnostic).

Reviewer_udcWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for distillation of diffusion models following the diffusion compression approach (i.e., distill a pre-trained teacher model to a lightweight student model). The proposed method takes into account the discrepancy of future states from both the teacher and student models during the diffusion trajectory. Temporal sampling is adopted to select future positions, reducing the computational cost from teacher continuation queries (i.e., the teacher model is queried to continue the generation from student-visited states). The distillation is driven by the combination of the classical knowledge distillation (KD) and proposed Future Discrepancy (FD) approaches. The proposed method was evaluated across various teacher–student configurations and on several benchmarks including ImageNet and MS-COCO.

Critical strengths
  • The paper studies an important and fundamental research problem
  • The proposed FD use is reasonable and clearly motivated.
  • Computational costs are reported in detail and comprehension.
Critical weaknesses
  • Although the introduction of Future Discrepancy (FD) sounds reasonable, there is no theory supporting its importance and success.

  • It is not clear how OPD/TPD performs compared with the same sparse budget (R = 3). A comparison between OPD/TPD with the same sparse sampling should help to answer this.

  • Writing needs improvement. Some word phrases are not defined/explained, making them difficult to understand. For instance, what is "historical measurement protocol"? The paper, in its current form, does not appear to be "ready". For instance, line 821 writes "pending entries are marked TBD" and the caption of Table 9 writes "TBD marks pending standard deviations", which appears incomplete.

  • Some technical details are missing from the manuscript, e.g., batch size, memory footprint, training time.

  • A user study should be conducted to further assess the generation quality of a student model.

Questions for authors

Please address questions/issues raised in the weaknesses section.

3 comments
AuthorsOpenReview ↗

Author Response 1

Dear Reviewer udcW,

We thank the reviewer for the careful assessment and constructive suggestions. We appreciate your recognition of the importance of diffusion model compression, the motivation for Future Discrepancy (FD), and our effort to report computational costs. Your comments identify several ways to strengthen both the evidence and its presentation. We address each concern below.

Q1. Although the introduction of Future Discrepancy (FD) sounds reasonable, there is no theory supporting its importance and success.

Thank you for suggesting stronger theoretical support for FD. FD uses these future deviations to supervise the current Student state update. We provide two complementary theoretical explanations: the first-order analysis explains how FD reflects the different effects of local errors under Teacher dynamics, while the final-state analysis connects these future discrepancies to the final Student–Teacher state discrepancy. The analyses are presented in Section 3.3, with detailed derivations in Appendices A.2 and A.3.

To understand why future effects matter, we first examine how Teacher continuation transforms local errors through the first-order analysis. When the Teacher continuation is locally smooth and the current discrepancy is small, future discrepancy weights local error directions through Ji→j⊤Ji→jJ_{i\to j}^{\top}J_{i\to j}, where Ji→jJ_{i\to j} is the Jacobian of the Teacher continuation map. Consequently, local errors of similar magnitude can produce different future deviations, which FD captures in its supervision target.

We then connect FD to the final Student–Teacher state discrepancy. Consider deterministic sampling with NN updates and shared initial noise, conditioning, and time grid. At each Student-visited state, let Gi→N−1G_{i\to N-1} denote the difference between the final states obtained by taking either a Student or Teacher update and then completing sampling with the Teacher. Following the successive policy-replacement argument in AggreVaTe, Lemma 4.3 [R1], we replace Teacher updates with Student updates in chronological order. Each replacement changes the final state by the corresponding full-continuation FD, so summing these differences yields

xNS−xNT=∑i=0N−1Gi→N−1,x_N^{\mathcal S}-x_N^{\mathcal T}=\sum_{i=0}^{N-1}G_{i\to N-1},

where xNSx_N^{\mathcal S} and xNTx_N^{\mathcal T} are the final states of the complete Student and Teacher sampling processes. Let mm be the state dimension and use the paper’s mean squared discrepancy, ℓi→jFD=∣Gi→j∣22/m\ell_{i\to j}^{\mathrm{FD}}=\\|G_{i\to j}\\|_2^2/m. Cauchy–Schwarz then gives

\le N\sum_{i=0}^{N-1}\ell_{i\to N-1}^{\mathrm{FD}}.$$ The left-hand side measures the final-state MSE between the Student and Teacher, while the right-hand side sums FD losses across positions, scaled by $N$. In this setting, the FD vectors capture the per-update effects that compose the final state difference, and **reducing the sum of their squared losses tightens its MSE upper bound**. Together with the first-order analysis, this connection provides a theoretical rationale for supervising current updates through their future effects. The following experiments examine how this principle translates into effective supervision with the shorter Teacher continuation used in training. **Figure 3(b,c) provides empirical support for this mechanism.** Figure 3(b) shows that smaller local errors can produce larger future discrepancies, while Figure 3(c) shows increasingly different future effects for sample pairs with similar local errors as continuation length increases. These findings support evaluating future effects beyond local error magnitude. **Table 1 evaluates the generation benefits of FD**, comparing TPD with FD(T) and OPD with FD(S) under the same trajectory-state source. Selected results are shown below. **FD improves FID across both Student constructions and both trajectory-state sources**, supporting the effectiveness of future-state supervision. | Student | Trajectory states | local | FID ↓ | FD | FID ↓ | | --- | --- | --- | ---: | --- | ---: | | SiT-B-D6 | Teacher | TPD | 16.09 | FD(T) | **15.07** | | SiT-B-D6 | Student | OPD | 15.64 | FD(S) | **14.29** | | SiT-S | Teacher | TPD | 14.12 | FD(T) | **13.56** | | SiT-S | Student | OPD | 13.34 | FD(S) | **12.58** | [R1] Ross, Stephane, and J. Andrew Bagnell. “Reinforcement and Imitation Learning via Interactive No-Regret Learning.” *arXiv*, 2014. `Revision in paper.` *We have incorporated the final-state analysis into **Section 3.3 and Appendix A.3**. Related analysis and experiments appear in Appendix A.2, Section 4.2.2, Figure 3(b,c), Table 1, and Appendices C.2 and C.5.*
AuthorsOpenReview ↗

Author Response 2

Q2. A comparison between OPD/TPD with the same sparse sampling should help to answer this.

Thank you for suggesting this comparison. We compare local supervision and FD under dense and sparse sampling for SiT-B to SiT-B-D6 at the same number of training updates. A checkmark in FD denotes future-discrepancy supervision, and a checkmark in TS denotes stratified temporal sampling with R=3R=3. FD uses h=1h=1. Parentheses give changes from the dense TPD/OPD baseline in the first row of each trajectory-source group. Cost is measured in seconds per complete training iteration. The results are shown in the table below.

MethodFDTSFID ↓IS ↑Precision ↑Recall ↑Cost ↓
TPD16.09109.490.720.361.18
FD(T)✓15.07 (−1.02)112.94 (+3.45)0.74 (+0.02)0.35 (−0.01)1.60 (+0.42)
TPD✓16.62 (+0.53)107.44 (−2.05)0.72 (0.00)0.35 (−0.01)0.63 (−0.55)
FD(T)✓✓15.73 (−0.36)111.62 (+2.13)0.74 (+0.02)0.34 (−0.02)0.79 (−0.39)
OPD15.64119.070.740.311.36
FD(S)✓14.29 (−1.35)125.57 (+6.50)0.76 (+0.02)0.30 (−0.01)1.79 (+0.43)
OPD✓16.09 (+0.45)118.08 (−0.99)0.74 (0.00)0.30 (−0.01)0.55 (−0.81)
FD(S)✓✓14.89 (−0.75)124.92 (+5.85)0.76 (+0.02)0.29 (−0.02)0.72 (−0.64)

With the TS setting held fixed, FD improves FID and IS over the corresponding local baseline for both trajectory-state sources, with sparse-setting gains close to those under dense supervision. The small Precision gains and Recall decreases relative to local baselines also persist in both settings. Temporal sampling thus retains FD's quality benefits and its quality–coverage trade-off.

Comparing each method with and without TS, temporal sampling substantially reduces Cost with modest changes in generation performance. Under the same sparse sampling, FD retains a small additional Cost over its local baseline. TS therefore offers a performance–Cost trade-off for both local and future supervision, while FD retains its quality advantage at the same sampling setting.

Revision in paper. Section 4.2.4 and Appendix D.4, Table 15 (joint FD/TS comparison).


Q3. Writing needs improvement.

Thank you for identifying these issues with the manuscript's clarity and completeness. We recognize that the leftover statements and insufficient explanations made the manuscript appear unfinished, and we apologize for this. We have made the following revisions.

Statistical reporting (Appendix D.1 and Table 11, previously Table 9). The submitted table included the means and sample standard deviations over three training runs for all reported methods and metrics. We overlooked the obsolete TBD statements in the text and caption during manuscript preparation. We have removed these statements, and the statistical results in the table remain unchanged.

Figure 6 and its measurement protocol (Section 4.2.5 and Appendix C.4). The term “historical measurement protocol” referred to measuring the initial trajectory gap using paired 10-step Euler trajectories from the Teacher and initial Student before distillation. We have replaced this ambiguous wording with an explicit description of the measurement procedure and the figure. The horizontal axis gives the state gap normalized by the Teacher-state magnitude and averaged across paired trajectories and update positions; the vertical axis gives the Student/Teacher parameter ratio. We also clarify that horizontal bars show paired-trajectory bootstrap 95% intervals, point colors identify the lower-FID local method, and the background regions summarize an empirical partition of the tested configurations. Appendix C.4 gives the state-gap definition, partition cutoffs, and how they were determined.

We have also checked the manuscript for leftover placeholder statements and reviewed the terminology and consistency between the text and captions in the affected sections.

Revision in paper. Section 4.2.5 and Appendix C.4 (Figure 6); Appendix D.1, Table 11 (statistical reporting).

AuthorsOpenReview ↗

Author Response 3

Q4. Some technical details are missing from the manuscript, e.g., batch size, memory footprint, training time.

Thank you for pointing out that the training settings and resource reporting were insufficiently detailed in the submitted manuscript. We have expanded these details in Appendices B.2–B.4. The key settings of our training are summarized below, with resources reported for two RTX 4090 GPUs.

  • Ablations and method analysis: The default batch contains 64 rollout trajectories, with 25K training updates. For SiT-B → SiT-B-D6 with h=1h=1 and R=3R=3, FD(T) and FD(S) have per-GPU Mem of 6.81 and 8.67 GB and Cost of 0.79 and 0.72 s/iter, respectively.
  • ImageNet comparisons with state-of-the-art methods: The effective global image batch is 256, with 32 rollout trajectories globally and 500K training updates. For TinyDiT-D14, Mem is approximately 20 GB per GPU and Cost is 1.281 s/iter.
  • MS-COCO text-to-image experiments: The effective global image batch is 128 with two gradient accumulation steps, and each accumulation step uses 16 rollout trajectories across both GPUs. Training runs for 50K updates. Mem is approximately 18 GB per GPU and Cost is 6 s/iter.

The optimizer, learning rate, EMA, precision, training sampler, and evaluation settings for these experiments are detailed in Appendices B.2 and B.3. Memory usage and iteration Cost are summarized in Appendix B.4.

Revision in paper. Section 4.1 and Appendices B.2–B.4 (training and evaluation settings, Mem, and Cost).


Q5. A user study should be conducted to further assess the generation quality of a student model.

Thank you for suggesting a user study. We conducted blinded A/B comparisons of Student image quality and text alignment, following DMD2 [R2] in evaluating these dimensions separately and BK-SDM [R3] in comparing images generated from the same prompt.

Experimental setup. We randomly sampled 128 prompts from our existing MS-COCO 30K evaluation set and used each model's corresponding generated images to compare FD-KD(S\mathcal S) and BK-SDM within the BK-Base-v2 architecture, together with the SD2.1 Teacher. Each of the 25 participants was assigned 128 formal comparisons, one per prompt, balanced across the three model pairs. Trial order was randomized, left/right positions were counterbalanced, and model identities were hidden throughout. Each trial required two A/B choices: visual quality with only the images shown, then text alignment after revealing the prompt. The table reports the win rate for the first model in each pair, computed within each prompt and then averaged equally across prompts.

Comparison (first vs. second model)Visual quality win rateText alignment win rate
FD-KD(S\mathcal S) Base vs. BK-SDM Base53.40%53.71%
FD-KD(S\mathcal S) Base vs. SD2.1 Teacher32.95%33.64%
BK-SDM Base vs. SD2.1 Teacher30.93%31.45%

A rate of 50% denotes equal preference. In the direct Student comparison, FD-KD has preference rates of 53.40% for visual quality and 53.71% for text alignment, compared with 46.60% and 46.29% for BK-SDM. These results show a modest preference advantage with the same Student architecture and training budget. Rates below 50% against SD2.1 indicate that both Students remain less preferred than the Teacher.

Complementary automatic evaluation. We also evaluated human preference, image quality, and semantic alignment using automatic metrics. ImageReward and HPSv3 estimate human preference, ImageDoctor assesses overall impression, and DeQA and VQAScore assess image quality and semantic alignment, respectively. The results are shown below. Both Students use the BK-Base-v2 architecture, and bold marks the better Student result. The ImageDoctor column reports its Overall score.

Model / methodImageReward ↑HPSv3 ↑DeQA ↑VQAScore ↑ImageDoctor ↑
SD2.1 Teacher0.34466.74143.93660.57980.5715
Base BK-SDM-0.15062.11473.81320.41890.5407
Base FD-KD(S\mathcal S)-0.10112.20163.84110.43080.5496

FD-KD(S\mathcal S) outperforms BK-SDM on all five metrics, including the human-preference scores from ImageReward and HPSv3 and the overall-impression score from ImageDoctor. These results are consistent with the human preference advantage.

[R2] Yin, Tianwei, et al. “Improved Distribution Matching Distillation for Fast Image Synthesis.” Advances in Neural Information Processing Systems, vol. 37, 2024.

[R3] Kim, Bo-Kyeong, et al. “BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion.” European Conference on Computer Vision, 2024.

Revision in paper. Section 4.4, Table 8 and Appendix G, Figure 9 (user study and evaluation interface); Section 4.3, Table 7 and Appendix E (automatic evaluation).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.