acceptodds
Under review as a conference paper at ICLR 2027

Flow-Guided Wasserstein Compensation Distillation

Abstract

Few-step flow distillation constructs training directions from model predictions at student-generated states, but these directions need not realize a prescribed evolution toward a reference distribution. We introduce Flow-Guided Wasserstein Compensation Distillation (FWCD), a framework that combines existing flow guidance with distributional information from unpaired reference samples. At each noise level, FWCD defines the minimum-energy correction to a scaled flow prior that completes a prescribed marginal response. A weak primal–dual formulation learns a scalar potential from unpaired samples; its input gradient supplies the correction for magnitude-controlled student updates. Under suitable function-space assumptions, we characterize the optimal field and its residual decomposition, relate population dual suboptimality to field approximation error, and establish Wasserstein contraction for the corresponding ideal distribution dynamics. Experiments on ImageNet-256 and SANA-1.6B demonstrate improvements over Mean Flow Distillation with four-step inference at matched student and auxiliary update counts. The flow-prior variant reduces ImageNet FID from 4.551 to 4.222 and SANA teacher-reference FID from 27.048 to 24.755. Ablations show task-dependent objective preferences: additive compensation achieves lower mean FID and KID on ImageNet at the default fitting budget, while flow-guided compensation performs better on SANA under a shared cap. FWCD preserves the student architecture and requires no additional model evaluations at inference.

Primary area:
generative models
Keywords:
flow matching, flow distillation, generative model distillation, optimal transport, Wasserstein geometry, distribution matching, few-step generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZC4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Flow-Guided Wasserstein Compensation Distillation (FWCD), which introduces a distribution-aware correction to existing flow distillation guidance using unpaired reference samples. The method learns a scalar potential to estimate the correction field and incorporates it into student training. The paper provides theoretical analysis and evaluates the approach on ImageNet-256 and SANA-1.6B, reporting improvements over Mean Flow Distillation (MFD) under four-step inference.

Critical strengths

S1. Empirical improvements over MFD. The reported results indicate some improvements in generation quality on ImageNet-256 and SANA-1.6B, suggesting the potential usefulness of distributional compensation for few-step distillation.

Critical weaknesses

Overall, the paper lacks sufficient clarity and coherence in its presentation. The motivation, significance, and intended contributions of the proposed approach are not clearly established, and the roles and connections of individual sections are difficult to follow.

Furthermore, the purposes of different baselines and experimental settings, as well as the interpretations of the reported results, are insufficiently explained.

As a result, understanding the paper often requires readers to infer connections and assumptions that should be explicitly communicated. These issues substantially affect the overall readability and completeness of the manuscript.


W1. Insufficient motivation and problem formulation. The paper does not clearly explain the practical limitations of existing methods, why the prescribed density evolution is necessary, or how the proposed formulation addresses these limitations.

W2. Unclear notation and terminology. Several quantities (e.g., vj,gtv_j, g_t, and student particles) are introduced without sufficient definitions. The use of multiple related notations further complicates understanding.

W3. Lack of coherent methodological explanations. The transitions between flow guidance, density constraints, potential learning, and student optimization are insufficiently motivated. Readers must infer why individual components are introduced and how they relate to one another.

W4. Insufficient experimental setup and evaluation scope. The main paper does not clearly establish all baseline configurations, experimental protocols, or the meaning of R1–R5. Furthermore, evaluation is restricted to NFE=4, leaving generalization across inference budgets unexplored.

W5. Limited interpretation of experimental results and figures. The paper provides limited explanations of the purpose and findings of individual ablations. In particular, Figure 3 lacks meaningful qualitative analysis, and the implications of the differing performance between FWCD and Additive D0 are not sufficiently discussed.

Questions for authors

Please refer to the weaknesses outlined above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZC4J

We thank the reviewer. We clarified the motivation, notation, training workflow, and experiments.

W1. Motivation

MFD supplies guidance at student-generated queries, whose distribution can differ from the reference early in training. FWCD prescribes linear relaxation toward the reference and learns the response still required after the scaled MFD field. This choice permits estimation from unpaired sample expectations and explicit ideal dynamics with squared-Wasserstein contraction.

Revision: Sections 1 and 3.1–3.2.

W2. Notation

vj(x,t,c)v_j(x,t,c) is model jj's instantaneous velocity at query xx, noise time tt, and condition cc. With flow map Φr←tj\Phi^j_{r\leftarrow t}, mean velocity is

\begin{cases} [x-\Phi^j_{r\leftarrow t}(x,c)]/(t-r),&r<t,\\ v_j(x,t,c),&r=t. \end{cases}$$ The base is $b=u_{\rm aux}-u_{\rm teacher}$; $g_t$ is total guidance. A student particle is a sample from the student query law. Noise time $t$, ideal evolution time $\tau$, and discrete optimizer updates are distinct. *Revision: Section 2; Appendix A, Table 4.* ### W3. How the method fits together 1. **Base:** MFD supplies direction $b$. 2. **Target:** For student/reference laws $\mu,\nu$, the constraint $-\nabla\cdot(\mu g)=\lambda(\nu-\mu)$ prescribes their distributional response. Its weak form, $\mathbb E_\mu\langle g,\nabla f\rangle=\lambda(\mathbb E_\nu f-\mathbb E_\mu f)$, uses unpaired samples. 3. **Potential:** Minimum-energy fitting gives $g_\beta[h]=(\beta b+\nabla h)/(1+\beta)$. The potential fits the target response minus that already supplied by the scaled base. 4. **Student:** We read the EMA potential's correction, apply magnitude control, and hold guidance fixed during backpropagation through the student query. ImageNet separately generates fitting/update queries. SANA fits on 16 queries; the first is detached during fitting and reused with its retained graph for the student update. *Revision: Sections 3.2–3.4 and 4, Figure 2; Appendix E.5.* ### W4. Experimental settings and inference steps R1–R3 are the main ImageNet runs; R4–R5 are matched-relative-magnitude replay runs. Seeds are listed in Appendix F. Original ImageNet comparisons use SiT-XL/2, qkv LoRA 64/64, 560 student updates, and four-step inference, with method-specific auxiliary/discriminator schedules and samplers. Original SANA training uses 272 student and 2,728 auxiliary updates. One- and two-step SANA models are trained separately for 3,000 global updates with three seeds. Evaluation shares 5,000 prompts, noises, and teacher references; FWCD uses natural magnitude and $\beta=9$. | Inference steps | MFD FID ↓ | FWCD FID ↓ | |---|---:|---:| | 1 | **257.948 ± 12.996** | 304.223 ± 8.839 | | 2 | 383.635 ± 21.679 | **86.592 ± 42.250** | Values are means ± sample SD. MFD has lower one-step FID in all pairs; FWCD has lower two-step FID and higher scores on all four image–text metrics in all pairs. A single-seed check changes only the prompt prefix and preserves both rankings. *Manuscript: Section 7.1; Appendices F and G.1/G.5. NFE and prefix-check results are supplied in this response.* ### W5. Ablations and Figure 3 Attenuation tests base scaling: ImageNet FID changes from **4.551 to 4.549**. D0/D9 test objectives on the same $0.9b$ base and 10% cap: FIDs are **4.059/4.222** at $M=5$ and **4.134/4.061** at $M=1$. Shuffle tests assignment, raising D9 FID from **4.222 to 4.442**. Appendix I.4 quantifies objective/readout and correspondence contributions separately; objective preference depends on the setting. Figure 3 uses shared classes/noise at eight evenly spaced evaluation indices. Its caption identifies the unresolved small foreground face in column (e), enlarged in **Figure 10(a)**. Quantitative results use the full evaluation set; selected images do not estimate failure rates. *Revision: Figure 3 caption in Section 7.2; Sections 7.3–7.4, Tables 1/3; Appendix I.4 (Table 16, Figure 8); J.4 (Figure 10(a)).*

Dear Reviewer,

The discussion deadline is approaching. Please review the authors’ responses and clearly indicate whether they adequately address your concerns. Kindly update your score if appropriate.

Best regards,

AC

Reviewer_ZC4JOpenReview ↗

Response to Authors

We thank the authors for the additional comments and experiments. However, the following points do not yet appear to be fully addressed.

1.The implication of the gap between FWCD and Additive D0 remains unclear. (W5)

  • In Tables 1, 3, 10, and 16, Additive D0 often appears competitive with the proposed β-objective on ImageNet. The manuscript does not explain why, or when each objective should be preferred.

2. Generalization across inference budgets is still not established. (W4)

  • Evaluation remains limited to 4 steps. 1 and 2 step scores appear to be at a level where the models themselves are not properly trained, rather than being directly comparable. The response also does not explain why the ranking reverses between 1 and 2 steps, or why the variance across seeds is so large.

For these reasons, I will maintain my score.

I hope this feedback is helpful for the next version of the manuscript.

Reviewer_e6jBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FWCD, which adds a learned potential gradient to the guidance used in flow distillation. The correction comes from a minimum-energy formulation with a prescribed marginal relaxation, and is learned from unpaired student and reference samples through a dual objective. Results on ImageNet-256 and SANA improve over MFD with four-step inference and no extra inference cost.

Critical strengths

I find the mathematical formulation useful. The paper gives a precise meaning to the correction and derives a way to learn it from samples without evaluating densities or computing transport matchings. It also clearly states where the theoretical guarantees apply and where the implementation introduces approximations.

Critical weaknesses
  1. My main concern is the training cost. The comparisons match student and auxiliary updates, but FWCD adds potential training, differentiation through its input gradients, and reference construction. The paper acknowledges this, but does not report training time, peak memory, or quality versus training time. I am therefore unsure whether the gains would remain if the same compute were spent on training MFD longer.

  2. I am not fully convinced that the paper explains why the proposed correction improves on MFD. There is little direct analysis of which limitation of MFD guidance is being addressed. The theory characterizes an optimal field for the chosen relaxation, without establishing why this dynamics should be preferable to that induced by MFD.

Questions for authors
  1. Could you report training time and peak GPU memory for MFD and FWCD, including all potential-fitting and reference-construction costs? In particular, could you show generation quality versus GPU-hours under matched hardware, allowing MFD to train for the same total budget?

  2. Could you provide an ablation over the flow-prior weight (\beta), including values below and above the default (\beta=9)? This would help clarify how sensitive the method is to this choice and how strongly it depends on MFD guidance.

  3. Could you clarify which limitation of MFD guidance the prescribed relaxation is intended to address? A direct comparison or diagnostic showing how the correction addresses this limitation would help explain the observed gains in generation quality.

I am open to revising my rating upward if the authors provide convincing explanations and supporting evidence that adequately address these concerns.

5 comments
AuthorsOpenReview ↗

Response to Reviewer e6jB

We thank the reviewer for the constructive questions. We added matched-time experiments, flow-prior ablations, and direct response measurements.

1. Compute budget and memory

At matched SANA training-time budgets, FWCD lowers terminal teacher-reference FID in all three pairs: 20.777 ± 3.193 versus 27.650 ± 0.106 for MFD.

Each four-step pair ran on the same A100-40GB. Natural-magnitude FWCD (β=9\beta=9) completed 9,000 global updates (818 student updates); MFD continued to the same 8.85–9.08 GPU-hour budget, completing 3,915–4,120 student updates. Timing starts after loading and includes warm-up/fitting, reference construction, optimization, saving, and export. Evaluations share 5,000 held-out prompts, initial noise, and teacher references. Values below are means ± sample SD over three seeds.

Elapsed timeMFD FID ↓FWCD FID ↓
30 min25.224 ± 1.68568.351 ± 3.728
60 min26.152 ± 2.61749.792 ± 6.077
120 min23.708 ± 1.54228.927 ± 2.018
180 min28.238 ± 1.56524.041 ± 2.103
Matched endpoint27.650 ± 0.10620.777 ± 3.193

The lowest evaluated within-budget FIDs also favor FWCD in all pairs (means 20.062 vs. 23.437), selected retrospectively from ten checkpoints per method/seed. MFD leads at the 30-, 60-, and 120-minute checkpoints; FWCD leads at 180 minutes and the endpoints.

At 3,000 matched updates, FWCD takes 5.17 ± 0.12× the MFD time. Peak PyTorch allocated memory is 21.394 vs. 20.798 GiB; reserved memory is 22.469–22.480 vs. 21.635 GiB.

Three ImageNet pairs resume their own 560-update checkpoints for 4,628 seconds on eight A100s, including startup/loading, with separate 50k-image, four-step evaluation. Mean strict within-budget minimum FIDs are 2.791 (FWCD) vs. 2.771 (MFD); endpoint means are 2.684 vs. 3.082. Appendix G.4 separates endpoint saves exceeding budget by up to 5.02 seconds. Thus ImageNet within-budget minima are comparable.

Revision: Sections 7.1–7.2; Appendix G.4, Table R7; Appendix H.1, Figure R1 and Tables R3–R5.

2. Sensitivity to the flow-prior weight

Natural-magnitude SANA runs use three seeds, 3,000 global updates, and shared four-step evaluation inputs:

β\betaTeacher-reference FID ↓
323.185 ± 2.996
924.567 ± 2.008
2723.462 ± 0.307

All three means are below MFD's 26.529 ± 1.626. The field is gβ=β1+βb+∇hβ1+βg_\beta=\frac{\beta}{1+\beta}b+\frac{\nabla h_\beta}{1+\beta}: changing β\beta changes the retained MFD coefficient, potential target, and readout magnitude.

For complete removal of MFD guidance, ImageNet R1 compares natural-magnitude g0=∇h0g_0=\nabla h_0 with g9=0.9b+∇h9/10g_9=0.9b+\nabla h_9/10. At 560 student updates, FIDs are 100.064 vs. 5.597. This single-seed test compares complete configurations under the same fitting budget; the base, target, and magnitude change together. Table 1 uses the separate 10% cap protocol.

Revision: Section 7.4 gives the role of β\beta; Appendix I.5, Tables R6 and R9 give the results and protocols.

3. What does compensation correct?

MFD supplies a direction bb at student-generated queries. FWCD additionally prescribes their distributional rate, ∂τμτ=λ(ν−μτ)\partial_\tau\mu_\tau=\lambda(\nu-\mu_\tau). Its weak form uses unpaired sample expectations; the ideal relaxation has an explicit solution and squared-Wasserstein contraction.

At fixed student/reference laws μ,ν\mu,\nu, define Bμ(v)=−∇⋅(μv)\mathcal B_\mu(v)=-\nabla\cdot(\mu v) and cβ=β/(1+β)c_\beta=\beta/(1+\beta). Substituting gβ⋆=cβb+uβ⋆g_\beta^\star=c_\beta b+u_\beta^\star into the target constraint and using linearity gives Bμ(cβb)+Bμ(uβ⋆)=λ(ν−μ),\mathcal B_\mu(c_\beta b)+\mathcal B_\mu(u_\beta^\star)=\lambda(\nu-\mu), hence Bμ(uβ⋆)=λ(ν−μ)−Bμ(cβb).\mathcal B_\mu(u_\beta^\star)=\lambda(\nu-\mu)-\mathcal B_\mu(c_\beta b). The correction supplies the difference between the target rate and the scaled base's rate, including counteracting excess response. Equation (4) selects the minimum-energy correction among fields satisfying this constraint. Appendix C.3 shows how exact residual fitting avoids duplicating representable base responses in a fixed closed linear gradient class.

Held-out ImageNet diagnostics restore the same student weights and AdamW state for each direction. At global update 500, adding uncapped u9u_9 to 0.9b0.9b changes the normalized field residual by −0.0395 ± 0.0144, the post-AdamW response residual norm by +1.412 ± 0.753, and one-step MMD change by −(4.362 ± 2.911) × 10⁻⁶. These distinguish field-level improvement from the realized student response. Contraction concerns exact population dynamics before finite fitting, capping, and parameter updates.

Revision: Sections 3.2–3.4, 5, and 7.3; Appendices C.3, E.1, and G.3 (Tables R1–R2).

Thank you for the additional experiments and analysis. However, I still have some concerns about the explanation in your third response.

MFD [1] already aims to align the student distribution with the teacher. Is there a reason to prefer the prescribed linear relaxation over mean velocity matching?

If a theoretical justification is difficult to provide, could you at least clarify which practical limitation of MFD is being addressed and how FWCD mitigates it? Clarifying this motivation would strengthen the paper.


[1] Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Models, ICML 2026

AuthorsOpenReview ↗

Response to Reviewer e6jB

Thank you for the follow-up. Our starting point is distribution matching from a Wasserstein gradient-flow viewpoint. We recognize MFD's distribution-matching guarantee under its assumptions (Theorem 4.3).

1. Why prescribe linear relaxation when MFD already targets distribution matching?

We seek velocities that move the student law toward the reference. We examined correction directions from explicit minibatch OT pairings. In the tested high-dimensional image setting, these did not improve endpoint FID evaluated on 50k images and were sensitive to displacement magnitude (Appendix J.3). These observations motivate a weak formulation using unpaired student/reference samples to learn a minimum-energy correction in Wasserstein kinetic geometry, retaining MFD as a flow prior.

At fixed noise, μ,ν\mu,\nu denote the student/reference query laws:

∂τμτ=λ(ν−μτ).\partial_\tau\mu_\tau=\lambda(\nu-\mu_\tau).

The reasons are (i) its weak target λ(Eνf−Eμf)\lambda(\mathbb E_\nu f-\mathbb E_\mu f) is estimable from unpaired samples; (ii) ideal evolution has an explicit mixture solution and Wasserstein contraction; and (iii) its minimum-energy velocity has a local OT interpretation.

OT maps a student sample xx to Tε(x)T_\varepsilon(x); Tε(x)−xT_\varepsilon(x)-x is its displacement. Let g0,ε⋆g_{0,\varepsilon}^\star be the minimum-energy velocity for linear relaxation. Near the reference, under the conditions below,

=\lambda(T_\varepsilon-\mathrm{Id})+o(\varepsilon).$$ The error is negligible relative to the density perturbation $\varepsilon$. With the MFD prior, Proposition 2 gives $g_\beta^\star=g_{0,\varepsilon}^\star+c_\beta b_\perp$, $c_\beta=\beta/(1+\beta)$, $\nabla\cdot(\mu_\varepsilon b_\perp)=0$: the extra term has zero instantaneous density response. Conditions: at fixed noise and conditioning, work on a flat torus with $\nu=\rho dx$, $\mu_\varepsilon=p_\varepsilon dx$, $p_\varepsilon=\rho+\varepsilon\sigma$, smooth $\rho,\sigma$, $\rho\ge\rho_{\min}>0$, $\int\rho dx=1$, $\int\sigma dx=0$, fixed $\lambda>0$, and $p_\varepsilon\ge\rho/2$. Assume the quadratic-cost OT maps $T_\varepsilon:\mu_\varepsilon\to\nu$ form a differentiable regular branch through $\mathrm{Id}$ at $\varepsilon=0$, with $b\in L^2(\mu_\varepsilon)$ and fixed finite $\beta\ge0$. The result follows by linearizing OT and the minimum-energy Poisson equation $-\nabla\cdot(p_\varepsilon g_{0,\varepsilon}^\star)=-\lambda\varepsilon\sigma$. Remainders use $L^2(\rho dx)$. It concerns ideal fields before finite fitting and magnitude control; $\tau$ differs from generation/noise time. **2. Which practical limitation is targeted, and how does FWCD address it?** At finite budgets, auxiliary tracking of a changing student law and numerical mean-velocity estimates can leave guidance errors. FWCD adds student/reference sample supervision to fit the response still required after scaled MFD guidance. Write $g_\beta^\star=c_\beta b+u_\beta^\star$. For admissible test functions $f$, $$\mathbb E_\mu\langle u_\beta^\star,\nabla f\rangle =\lambda(\mathbb E_\nu f-\mathbb E_\mu f) -c_\beta\mathbb E_\mu\langle b,\nabla f\rangle.$$ The correction fills the target response minus that supplied by the scaled prior. Minimum correction energy selects among feasible fields. Appendix C.3 separates approximation/fitting errors in a fixed closed linear gradient class; Appendix D defines the reference laws. | ImageNet control | FID ↓ | |---|---:| | MFD / attenuation (Table 1) | 4.551 / 4.549 | | Reference contrast removed, relative-magnitude replay (Appendix I.1) | 4.000 → 4.389 | | Corrections shuffled, main 10% cap (Table 1) | 4.222 → 4.442 | | MFD / Exact-5% OT / 10%-cap OT (Appendix J.3, R6) | 4.117 / 4.125 / 4.236 | Table 1 averages R1–R3; replay averages R4/R5 at a shared correction/base RMS sequence; removing contrast also changes target dynamics. OT is R6, a single run with 16 candidates per class. Its raw displacement RMS averaged 3.95 times the base; magnitude control strongly affected quality (Appendix J.3). This may reflect limited minibatch coverage in high dimensions and coupling errors accumulating during training. These controls support marginal supervision and query-specific correction. Same-A100 SANA comparisons show lower terminal FID at matched time. FWCD plugs into MFD on SiT and SANA, retaining student architecture and the auxiliary objective. Inference uses only the student; reference construction and potential fitting are included in matched-time costs.
Reviewer_VZwUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flow-Guided Wasserstein Compensation Distillation (FWCD), which extends MFT by introducing an additional scalar-potential network to correct the student training gradients. The additional network is trained using student and reference samples through a distributional contrast objective with a gradient penalty incorporating the original MFD direction, which encourages the corrective gradient to compensate MFD instead of repeating the MFD direction. Theoretical analysis is presented, and experiments evaluate the method on 4-step ImageNet-256 and SANA-1.6B generation.

Critical strengths

Experiments appear to show improvements over the MFD baseline, and the theoretical analysis appears self-consistent in general.

Critical weaknesses
  • My major concerns are about the experiments.
    • The main ImageNet experiments show weak FIDs compared to state-of-the-art few-step distillation methods, which can typically yield a 1-step FID of 2~3. The baselines are the authors' own adaptations, using the same 560 student update schedule, there is no evidence that the baselines are optimally tuned and have converged under this very limited schedule.
    • On SANA, the paper reports only 272 student updates for its MFD and FWCD experiments, but it is unclear how this schedule compares with those of the other methods listed in Table 2. Also, the experiments generally use short training schedules, leaving its practical benefits under longer training unclear.
    • The advantage of the proposed FWCD formulation over the simpler additive baseline is inconsistent. In Table 1, the simple additive baseline yields the best FID, beating the proposed FWCD. The same trend shows in Table 3 with M=5, while M=1 makes FWCD better than the additive baseline.
  • The method appears expensive to train, yet training cost is not reported. MFD itself is already expensive, and yet FWCD adds 5 critic steps per student update, with second-order differentiation for the additional scalar network. This also makes many comparisons unfair, since the control is student update steps, not the wall time or FLOP count.
  • The authors argue that MFD guidance does not guarantee the desired distributional evolution, which motivates the proposed method. Yet, I do not see a clear connection between FWCD's theoretical analysis and a guaranteed distribution evolution either, as the implemented method uses approximate potential fitting, magnitude control, with complex student network dynamics that cannot be fully justified by the math.
Questions for authors

Please clarify the training cost, such as wall time, or time per student update, and the baseline setups.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VZwU

We thank the reviewer. We added matched-time, longer-training, and mechanism experiments.

Q1. Computational cost and fairness

In three paired SANA runs on the same A100-40GB per pair, FWCD lowers endpoint FID to 20.777 ± 3.193 from MFD's 27.650 ± 0.106, with lower FID in every pair.

Each 8.85–9.08 GPU-hour budget starts after loading and includes fitting, reference construction, optimization, saving, and export. FWCD completes 818 student updates; MFD 3,915–4,120. Four-step evaluation shares 5,000 prompts, noises, and teacher references. MFD leads at 30/60/120 minutes; FWCD leads at 180 minutes and endpoints. Retrospective within-budget minima also favor FWCD in all pairs: means 20.062 vs. 23.437, ten points per method/seed.

At 3,000 matched global updates, natural-magnitude FWCD takes 5.17 ± 0.12× MFD's time. In the separate capped protocol, MFD/D9/D0 take 38.49/190.79/187.34 minutes, with peak allocated memory 20.798/21.394/21.394 GiB.

ImageNet pairs resume their own 560-update checkpoints for 4,628 seconds on eight A100s, including startup/loading; evaluation is separate. Endpoint means are 2.684 (FWCD) vs. 3.082 (MFD). Table R7 separates endpoint overshoots up to 5.02 seconds and defines checkpoint selection.

Revision: Sections 7.1–7.2; Appendix G.4 (Table R7), H.1 (Figure R1, Tables R3–R5).

Q2. Longer training and baseline convergence

We extended SiT-XL/2, qkv LoRA 64/64 to 2,240 student updates, using 50,000 images, CFG 1, and each method's original four-step sampler. Student updates are matched; auxiliary/discriminator costs differ.

With one parameter variation per adapted baseline, the lowest evaluated R1 FIDs within each student-update budget are:

R1 configuration≤560≤1,120≤2,240
CD, EMA 0.99 after 56024.73423.98423.984
VSD, student LR 5×10−55\times10^{-5}5.1064.2663.821
DMD2, original GAN weight8.7158.7155.104
FWCD, original setting4.2423.1282.676

Minima are retrospective on the same evaluation set. DMD2's 8.715 occurs at 280 updates; its 560-update FID is 8.760. FWCD improves by 1.565 and leads these tested configurations. Table R8 gives all branches, checkpoint choices, and evaluation counts; Figure R2 gives curves. Some baselines still improve at the final checkpoint; these single-seed parameter checks leave convergence and full tuning unresolved.

On SANA, 9,000 global updates (818 student updates) yield 20.777 ± 3.193 for FWCD vs. 24.877 ± 0.840 for MFD, favoring FWCD in all three pairs. Table R10 gives the fixed-update trajectory.

Revision: Sections 7.2 and 8; Appendix G.5 (Table R8, Figure R2), H.1 (Table R10).

Q3. External SANA baselines

Table 2's five external rows come from Zhao et al., Table 3. Their Table 4 reports about 22/4/30 GPU-hours for Diff-Instruct/CD/DMD2 on RTX PRO 6000; other schedules/costs are incomplete. These published budgets supplement this reply. Controlled cost evidence uses our same-A100 MFD/FWCD pairs.

Manuscript: Section 7.2, Table 2; Appendix H.1 gives our controlled costs.

Q4. Additive D0 versus FWCD D9

With the same 0.9b0.9b base and 10% RMS cap, ImageNet D0/D9 FIDs are 4.059 ± 0.163 / 4.222 ± 0.105 at M=5M=5, and 4.134 ± 0.282 / 4.061 ± 0.065 at M=1M=1 (Table 3). Objective ranking changes with fitting budget.

Under Appendix C.3's assumptions, exact residual fitting has no larger H˙−1(μ)\dot H^{-1}(\mu) response residual than additive fitting at fixed laws, in the same closed linear gradient class and on the same scaled base before magnitude processing. Finite CNN fitting, EMA, capping, and student updates require empirical evaluation.

The 2×22\times2 Shapley study fixes M=5M=5, base, and cap. ImageNet FID contributions (improvement-positive) are −0.144 ± 0.168 for D0→D9 objective/readout and +0.238 ± 0.204 for Shuffle→Aligned correspondence; single-seed SANA gives +3.94/+1.14. The M=1M=1 reversal's optimization mechanism remains open.

Revision: Sections 7.3–7.4, Table 3; Appendices C.3, I.3–I.4 (Table 16, Figure 8).

Q5. Theory versus actual updates

The contraction result concerns an exact population field satisfying −∇⋅(μg)=λ(ν−μ)-\nabla\cdot(\mu g)=\lambda(\nu-\mu): W22(μτ,ν)≤e−λτW22(μ0,ν).W_2^2(\mu_\tau,\nu)\le e^{-\lambda\tau}W_2^2(\mu_0,\nu). Finite fitting, magnitude processing, the student Jacobian, and AdamW change realized dynamics.

At ImageNet global update 500, adding uncapped u9u_9 to 0.9b0.9b changes normalized field residual by −0.0395 ± 0.0144, post-AdamW response residual norm by +1.412 ± 0.753, and one-step MMD change by −(4.362 ± 2.911) × 10⁻⁶. These local measurements distinguish field agreement from actual updates and delimit the ideal-dynamics guarantee.

Revision: Sections 5 and 7.3; Appendices E.1 and G.3, Tables R1–R2.

Thank you for the additional experiments. These are now more informative. The FID vs GPU hours chart shows that MFD improves faster while FWCD's FID improves slower. MFD's FID degrades with more training while FWCD still improves, eventually beating MFD on SANA, though the gain on ImageNet is still unclear.

Could authorw explain why not warm up the model with MFD first and then tune it with FWCD, which I think could make it faster and better overall?

AuthorsOpenReview ↗

Response to Reviewer VZwU

Thank you for suggesting this schedule. We tested it on SANA-1.6B (seed 42) by switching from the saved MFD checkpoint at approximately 120 minutes to FWCD. All three training trajectories used the same physical A100-40GB. The two-stage run had the same 8.85 GPU-hour budget, including the MFD phase, checkpoint loading, potential warm-up, and FWCD continuation. We used four inference steps, the same 5,000 held-out prompts and noise seeds, and the same teacher reference as in Appendix H.1.

Cumulative training timeMFD throughoutFWCD throughoutMFD → FWCD
Approximately 180 min26.698225.194625.2029
Matched-time endpoint27.533617.108322.2408

The table reports teacher-reference FID (lower is better). Switching improved FID over continuing MFD at both measured times. The two-stage schedule and FWCD throughout had nearly identical FID at 180 minutes. FWCD throughout had the lowest FID at the matched endpoint. This single-seed experiment evaluates the 120-minute switch point.

One motivation for applying FWCD from the start is to supplement the guidance when student and reference queries may differ early in distillation. In a separate ImageNet control (Table 8), applying FWCD only for the first 140 of 560 student updates lowered mean final FID from 4.551 with MFD throughout to 4.313 across three runs.

Thank you again for taking the time to discuss our work and share your feedback.

Best regards,

The Authors

Comments on the paper

1 comment
Area_Chair_TYfDOpenReview ↗

Reminder: Discussion Deadline — November 18

Dear Reviewers,

Thank you for your valuable time and contributions to ICLR. This is a friendly reminder to review the authors' responses and engage in discussions with them. The discussion deadline is November 18.

Thank you for your continued efforts!

ACs

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.