acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Guided Temporal Allocation for Flow Matching Distillation

Abstract

Flow matching and diffusion models achieve state-of-the-art image synthesis but require slow, iterative inference. While few-step distillation reduces this cost, these methods conventionally distribute training updates uniformly across time. We propose Complexity-Weighted Rectified Distillation (CWRD), a plug-and-play training wrapper that uses a frozen teacher-geometry profile to reweight timestep sampling. The profile is based on Local Complexity (LC), which measures how the dominant endpoint-sensitivity subspace varies across neighboring latent states. CWRD increases the relative sampling probability of high-LC timesteps, reallocating temporal capacity allocation while strictly preserving the host method's per-example loss, student architecture, and inference procedure. We evaluate CWRD on -Flow, LADD, TDM, and MFD using high-resolution Stable Diffusion 3.5 and SANA teachers. Across the evaluated configurations, CWRD reduces teacher-referenced FID and improves semantic prompt alignment, with absolute data-referenced FID improvements dependent on the specific host architecture. Complementary trajectory diagnostics show that following a teacher-trajectory prefix before a single suffix update can preserve the teacher output better than uniform sampling at a matched evaluation budget. These findings suggest that endpoint-sensitivity variation provides a useful empirical signal for temporal allocation in few-step generative models.

Primary area:
generative models
Keywords:
Distillation, Flow Diffusion model, Generative model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hAwMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates timestep training resource allocation for flow matching distillation. Existing few-step distillation mostly uses fixed or analytic time-sampling weights and overlooks the intrinsic geometry of the flow ODE trajectory. The authors adapt Local Complexity (LC) into a temporal analysis tool to quantify how the endpoint-sensitive subspace varies across nearby latents at different timesteps. They observe an LC peak along the trajectory: latent geometry changes rapidly before this peak, while the trajectory becomes highly compressible afterward.

Based on this finding, they propose CWRD (Complexity-Weighted Rectified Distillation), a plug-and-play training wrapper. It offline computes the LC temporal profile from the teacher and reweights timestep sampling probabilities to allocate more training budget to high-LC timesteps. It requires no changes to the student network, inference pipeline or per-sample loss function.

Evaluations are conducted for 4/8-NFE distillation across four distillation frameworks (π-Flow, LADD, TDM, MFD) and three large teacher models (SD3.5, SANA, Flux.2). Compared with vanilla sampling and the analytic weighting from SenseFlow, CWRD improves FID and several CLIP / VQAScore metrics. Interpolation experiments, single-step suffix recovery, time-shift ablations and cross-attention entropy correlation analysis support the significance of the LC peak.

Critical strengths
  1. This work examines timestep resource allocation for distillation from the geometric perspective of flow ODEs, departing from conventional heuristic weighting.
  2. This work is a plug-and-play module compatible with multiple flow-matching distillation frameworks.
Critical weaknesses
  1. The LC computation pipeline involves numerous hyperparameters (differentiation radius, subspace dimension k, thresholds, truncation strategy). Systematic full hyperparameter sweeps are missing, raising concerns about the robustness of performance gains to tuning.

  2. The LC peak and optimal performance only demonstrate statistical correlation and do not establish that the peak corresponds to the global optimum. Time-shift ablations merely show performance degradation under misalignment and cannot validate optimality at the peak location.

  3. Offline precomputation of the teacher LC profile incurs upfront computational cost.

Questions for authors
  1. Could the authors conduct a full suite of ablation studies over all LC hyperparameters to verify the robustness of performance gains against each hyperparameter?

  2. A grid sweep over hinge positions across the full timeline is expected to verify whether the LC peak indeed marks the optimal training point, so as to disentangle correlation from causation.

  3. Please discuss potential strategies to mitigate the computational overhead incurred by offline precomputation of the teacher LC profile.

1 comment

We thank Reviewer hAwM for the thoughtful review and constructive questions on hyperparameter robustness, the hinge-shift sweep, and offline profiling cost. All new sweeps and analyses are in the revised PDF.

1. Robustness to LC Estimator Hyperparameters (W1, Q1)

Structural Insensitivity. CWRD maps raw C(ti)C(t_i) via min-max normalization onto [0,1][0,1] (Eq. 7), followed by flooring (γ=0.2\gamma=0.2) and grid-mean normalization (Eq. 9). Because positive affine scalings cancel out, training weights wiw_i depend only on the profile's ordinal shape and hinge t∗t^*. All teachers and hosts use the exact same default hyperparameters without per-model tuning. Profile & Truncation Sweeps (App. A Table 5; App. C.2 Table 11). Across 64 runs on SD3.5-Medium (12-step, CFG g=4.5g=4.5, native bf16), sweeping every estimator parameter over 10×10\times-100×100\times ranges preserves high Spearman rank correlation ρ\rho with the default profile and shifts E[t∗]\mathbb{E}[t^*] by ≤0.05\le 0.05:

ParameterDefaultAblated RangeMean-Profile Spearman ρ\rho∣ΔE[t∗]∣\lvert\Delta\mathbb{E}[t^{\ast}]\rvert
Probe dimension r^\hat{r}6464{8,16,32,64}\{8, 16, 32, 64\}≥0.90\ge 0.90 (most ≥0.96\ge 0.96)≤0.05\le 0.05
Subspace rank kk11{1,2,4}\{1, 2, 4\}≥0.96\ge 0.96≤0.05\le 0.05
FD step η0\eta_0 (RMS)0.020.02{0.002,0.02,0.2}\{0.002, 0.02, 0.2\}≥0.90\ge 0.90≤0.05\le 0.05
Neighbor radius ρ0\rho_0 (RMS)0.020.02{0.002,0.02,0.2}\{0.002, 0.02, 0.2\}≥0.90\ge 0.90≤0.05\le 0.05
Eigengap threshold τ\tau0.050.05[0.01,0.20][0.01, 0.20]≥0.96\ge 0.96≤0.05\le 0.05
Calibration runs NprofN_{\mathrm{prof}}64641616 random runs0.970.97 med. (0.880.88 5th pct)≤0.05\le 0.05

We also ablated the post-peak truncation/plateau strategy end-to-end on SD3.5 π\pi-Flow (Table 11):

  • 4 NFE FID: Native (21.58021.580) →\to Unmodified no-plateau (19.23019.230), Plateau at Peak (19.01019.010), Default Plateau at Peak+1 (18.72418.724).
  • 8 NFE FID: Native (20.12220.122) →\to Default (19.30519.305), Unmodified (19.29019.290), Plateau at Peak (18.99018.990).

2. Full-Timeline Hinge Grid Sweep: Correlation vs. Optimality (W2, Q2)

We agree an empirical profile cannot prove theoretical global optimality; rather, the frozen LC profile automatically locates the high-performing transition region without an expensive per-teacher training grid search. App. C.1 (Table 10) sweeps 7 hinge positions sr=clip⁡(σ∗+r,0.05,0.95)s_r = \operatorname{clip}(\sigma^* + r, 0.05, 0.95) across [0.05,0.95][0.05, 0.95] around σ∗≈0.633\sigma^* \approx 0.633 on SD3.5 π\pi-Flow (4 NFE; native FID 21.58021.580):

Shift rrΔ\DeltaFID ↓\downarrowΔ\DeltaIS ↑\uparrowΔ\DeltaPrec ↑\uparrowΔ\DeltaRec ↑\uparrow
−0.50-0.50+0.265+0.265−0.282-0.282−0.001-0.001−0.005-0.005
−0.25-0.25+0.149+0.149−0.156-0.156−0.001-0.001−0.002-0.002
−0.15-0.15+0.089+0.089−0.193-0.193−0.001-0.0010.0000.000
+0.15+0.15+0.371+0.371−0.267-0.2670.0000.000−0.003-0.003
+0.25+0.25+1.721+1.721−0.863-0.863−0.017-0.017−0.030-0.030
+0.50+0.50+2.509+2.509−1.415-1.415−0.044-0.044−0.043-0.043
  • Empirical Optimum on the Grid: Aligned r=0r=0 (s0=0.633s_0=0.633) achieves the best FID, IS, Precision, and Recall across all 7 positions (and beats per-step random shifts σ∗±R\sigma^*\pm\mathcal{R}, which degrade FID 18.724→20.31718.724\to 20.317 and VQA 0.875→0.7020.875\to 0.702, Table 4).
  • Forgiving Basin vs. High-Noise Cliff: Moderate shifts (r∈[−0.50,+0.15]r\in[-0.50,+0.15]) incur minor drops (ΔFID≤+0.371\Delta\mathrm{FID}\le +0.371, still >2.1>2.1 FID below native 21.58021.580), whereas shifting into high noise (r≥+0.25r\ge +0.25) degrades FID by +1.721+1.721 to +2.509+2.509 by over-weighting pure noise before endpoint sensitivity emerges. LC lands directly in this low-FID basin without a sweep.

3. Offline Profiling Cost & Mitigation Strategies (W3, Q3)

Exact Cost via Prefix Truncation (Sec. 3.2). Because deterministic ODE integration yields all MM anchors from one rollout Φ1→0ϕ(z,c)\Phi^\phi_{1\to 0}(z,c) (0.00.0 diff vs. truncated rollouts), profiling z,z′z,z' costs only 2(r^+1)=1302(\hat{r}+1)=130 rollouts/run vs. 1,5841,584 for suffix loops (6.6×6.6\times fewer solver steps). Computed once per teacher, it is reused across all hosts (π\pi-Flow, LADD, TDM-ft, SenseFlow-ft) and NFE budgets with zero training or inference overhead.

Four Cost-Reduction Strategies (Table 5; 16×16\times--32×32\times Faster):

  • Fewer Runs (Nprof:64→16N_{\mathrm{prof}}: 64\to 16, 4×4\times speedup): Keeps median ρ=0.97\rho=0.97 (5th pct 0.880.88).
  • Smaller Probe (r^:64→8 or 16\hat{r}: 64\to 8\text{ or }16, 4×4\times-7×7\times speedup): Keeps ρ≥0.90\rho\ge 0.90-0.960.96 and ∣ΔE[t∗]∣≤0.05|\Delta\mathbb{E}[t^*]|\le 0.05; at Nprof=16,r^=16N_{\mathrm{prof}}=16, \hat{r}=16, profiling needs 544544 rollouts total (∼10\sim 10-1515 min on one RTX 5090).
  • Coarse Anchor Interpolation (Used in CWRD): Profiles M=12M=12 steps and interpolates onto 1,0001,000 bins.
  • Batched Forward-Mode AD: WW computes [JW,t,c(z)][J_{W,t,c}(z)] in one forward pass.
Reviewer_gNC6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly investigates whether reallocating training weights can improve flow model distillation. Specifically, it explores whether geometric properties of the teacher’s generation process, particularly Local Complexity (LC), can guide this allocation to substantially improve performance such as FID and IS. Its main contribution is an empirically validated method that uses LC information to improve performance across multiple distillation frameworks, together with some insights into the underlying mechanism.

Critical strengths

The method delivers improvements across a broad range of metrics in the evaluated settings, while remaining relatively simple and easy to integrate. The experiments control for training budget and cover multiple distillation frameworks and teacher models. The paper also offers a valuable research perspective by connecting Local Complexity to temporal loss weighting in distillation, supported by diagnostic experiments.

Critical weaknesses

1.The abstract states that “CWRD increases the relative sampling probability of high-LC timesteps,” whereas Section 3.3, Eq. (11), and the supplied code apply loss reweighting while leaving the timestep sampling distribution unchanged. It’s better to revise the abstract to accurately describe the implementation. If actual resampling is intended, then the code and experimental description should be updated accordingly.

  1. The rationale for holding the LC profile constant after the peak is insufficiently explained in the main text, and no direct comparison with the unmodified LC profile is reported. Please provide further justification and include results without this plateau modification, at least in the appendix.

  2. The current experiments do not distinguish whether CWRD primarily improves computational efficiency under a limited training budget or also improves the final attainable performance when both models are trained sufficiently close to convergence. Please report and plot evaluation metrics against training iterations or training time for both the baseline and CWRD, including longer training runs.

Questions for authors

My questions are related to the concerns raised in the Weaknesses section: 1.Is the post-peak plateau modification necessary? This design choice does not seem particularly natural and would benefit from further justification. 2.Do the performance gains primarily reflect more efficient use of computation or better use of the available data? Do these gains persist with a larger training budget? 3.The LC computation described in the main text appears to differ from the implementation in the supplementary code. The former measures how the sensitivity of the mapping from the current state to the final generated output varies locally, whereas the latter appears to measure this for the mapping from the initial noise to the current state. Am I misunderstanding the implementation, or is there a discrepancy?

4 comments

We thank Reviewer gNC6 for thoroughly reviewing our paper and code. All alignments and new ablations are in the revised PDF.

1. Prefix vs. Suffix Map Alignment (Q3)

Thank you for catching this discrepancy. Our script perturbs source noise z=x1∼N(0,I)z=x_1\sim\mathcal{N}(0,I) (along basis WW, step η1\eta_1, shift ρ1Wξ\rho_1 W\xi) and integrates forward to xtiϕ=Φ1→tiϕ(z,c)x_{t_i}^\phi=\Phi^\phi_{1\to t_i}(z,c) for ti∈Tproft_i \in \mathcal{T}_{\mathrm{prof}}, differentiating the prefix flow map Gϕ,t,c(z):=Φ1→tϕ(z,c)G_{\phi,t,c}(z):=\Phi^\phi_{1\to t}(z,c).

  • Chain-Rule Coupling: Since Φ1→0ϕ=Φt→0ϕ∘Φ1→tϕ\Phi^\phi_{1\to 0}=\Phi^\phi_{t\to 0}\circ\Phi^\phi_{1\to t}, JΦ1→0(z)=JΦt→0(xtϕ)JΦ1→t(z)J_{\Phi_{1\to 0}}(z)=J_{\Phi_{t\to 0}}(x_t^\phi)J_{\Phi_{1\to t}}(z). Near t→1t\to 1, Φ1→tϕ\Phi^\phi_{1\to t} is near-identity (raw LC 1.7501.750 at t=0.982t=0.982, Table 12); integrating toward t=0t=0 accumulates nonlinear routing, peaking at t=0.448t=0.448 (LC=20.490\mathrm{LC}=20.490) at the cross-attention entropy minimum.
  • Prefix Truncation (6.6×6.6\times Faster): Anchor states from one rollout Φ1→0ϕ(z,c)\Phi^\phi_{1\to 0}(z,c) are bit-identical (0.00.0 diff) to truncated rollouts, so all MM anchors for z,z′z,z' cost only 2(r^+1)=1302(\hat{r}+1)=130 rollouts vs. 1,5841,584 for suffix loops.
  • Revision: Updated Abstract, Sec. 1 & 3.2 (Eqs. 4-5) to Gϕ,t,c(z):=Φ1→tϕ(z,c)G_{\phi,t,c}(z):=\Phi^\phi_{1\to t}(z,c), matching our code and Table 12.

2. Loss Reweighting vs. Timestep Resampling Wording (W1)

We agree and revised this throughout. CWRD keeps each host's native sampler T∼qT \sim q and weights per-timestep loss by w(T)w(T), optimizing $ \mathbb{E}_q[w(T)\ell_{\mathrm{base}}(\Theta;T)]=\bar{w}_q\mathbb{E}_{p_w}[\ell_{\mathrm{base}}(\Theta;T)] $ with pw(ti)=qiwi/wˉq∝qiwip_w(t_i)=q_i w_i/\bar{w}_q\propto q_i w_i and constant wˉq=∑iqiwi>0\bar{w}_q=\sum_i q_i w_i>0 (Eqs. 11, 17). Updated Abstract, Intro, Sec. 3.3, 4, Conclusion, Reproducibility, and App. A.

3. Post-Peak Plateau Justification & Ablation (W2, Q1)

App. C.2 (Table 11) compares unmodified LC against plateau variants on SD3.5 π\pi-Flow (50k samples):

Profile Shape (π\pi-Flow, SD3.5)4-NFE FID ↓\downarrow8-NFE FID ↓\downarrow
Native Baseline (wi=1w_i=1)21.58020.122
CWRD Unmodified (no plateau)19.23019.290
CWRD Plateau at Peak+1 (default)18.72419.305
CWRD Plateau at Peak19.01018.990
  • Unmodified LC Beats Native: Raw LC alone drops FID by 2.3502.350 at 4 NFE (21.580→19.23021.580\to 19.230) and 0.8320.832 at 8 NFE (20.122→19.29020.122\to 19.290). Prefix LC rises steeply from t=0.982t=0.982 (0.0060.006) to t=0.448t=0.448 (0.9210.921) and decays gently to t=0.009t=0.009 (0.7970.797, Table 12), so both shapes suppress pure-noise draws (t∈[0.85,1.0]t\in[0.85,1.0]).

  • Why Plateau Helps at 4 NFE (19.230→18.72419.230\to 18.724): At 4 NFE, one student policy must both resolve the pre-peak transition and take large substeps to t→0t\to 0. Holding post-peak weights constant prevents suffix decay, adding 0.5060.506 FID at 4 NFE (and 18.99018.990 at 8 NFE).

4. Compute Efficiency vs. Convergence Performance (W3, Q2)

Tables 2-3 span two regimes separating fixed-budget speed from post-convergence allocation:

RegimeTeacher / HostNFENative FIDCWRD FIDΔ\DeltaFIDNative FIDt\mathrm{FID}_tCWRD FIDt\mathrm{FID}_t
1. From-Scratch (4k)SD3.5 / π\pi-Flow421.58018.724−2.856-2.85611.6157.976
SD3.5 / π\pi-Flow820.12219.305−0.817-0.8174.2093.502
SD3.5 / LADD422.81318.138−4.675-4.6754.1574.077
SANA / MFD433.35330.584−2.769-2.76913.39813.286
SANA / MFD832.04428.973−3.071-3.0718.2457.977
Flux.2 / π\pi-Flow412.65012.600−0.050-0.0502.8352.734
2. Post-Conv. FTSD3.5 / TDM-ft424.04021.950−2.090-2.0909.1609.090
SD3.5 / SenseFlow-ft424.69024.540−0.150-0.150--
  • Matched Compute & Data: All pairs share COCO-10k prompts, batch 44, and 4,0004,000 steps, adding zero extra data or compute.

  • Gains Persist Post-Convergence: TDM-ft and SenseFlow-ft start from official checkpoints already trained to convergence. Fine-tuning TDM-ft for 4k steps natively gives 24.0424.04 FID (FIDt=9.16\mathrm{FID}_t=9.16), whereas CWRD drops FID by 2.092.09 to 21.9521.95 (FIDt=9.09\mathrm{FID}_t=9.09, CLIP 26.39→26.6426.39\to 26.64, HPS 0.258→0.2680.258\to 0.268).

  • Stationary Optimum: If CWRD only sped up early warmup, fine-tuning a converged student would not help. Because a single network Θ\Theta shares finite capacity across t∈(0,1)t\in(0,1), Θ∗(w)=arg⁡min⁡Θ∑iqiwiℓbase(Θ;ti)\Theta^*(w)=\arg\min_\Theta\sum_i q_i w_i\ell_{\mathrm{base}}(\Theta;t_i) depends on qiwiq_i w_i: down-weighting pure noise shifts the stationary optimum toward higher transition and suffix fidelity.

Thank you for the detailed response and the additional experiments. The clarification on loss reweighting resolves W1. However, the revision of the LC definition from the suffix map to the prefix map changes a core component of the method, and the motivation for the prefix-based LC is not yet sufficiently justified. In addition, the effect of the plateau modification is small and inconsistent across NFE settings, and training curves for longer runs are not provided. I therefore maintain my score.

We thank Reviewer gNC6 for engaging with our rebuttal and raising these three focused points.

1. Theoretical Motivation & Role of Prefix-Based LC (Φ1→tϕ\Phi^\phi_{1 \to t})

First, we clarify that CWRD's algorithm, code, and empirical results did not change: every diagnostic (Table 1 interpolation, Tables 8--9 one-step suffix recovery, Tables 12--13 cross-attention) and every distillation run (Tables 2--4) in the original submission used this exact prefix-map profile; our revision corrected only the text/subscript in Sec. 3.2 where Φt→0ϕ\Phi^\phi_{t \to 0} was miswritten instead of Φ1→tϕ\Phi^\phi_{1 \to t}.

Why Prefix-Based LC Directly Guides Distillation Allocation: Along the guided teacher ODE dxtdt=vϕg(xt,t,c)\frac{\mathrm{d}x_t}{\mathrm{d}t} = v_\phi^g(x_t, t, c) starting at x1=zx_1 = z, the prefix Jacobian Jt(z)=∇zΦ1→tϕ(z,c)J_t(z) = \nabla_z \Phi^\phi_{1 \to t}(z, c) obeys the variational equation: ddtJt(z)=∇xvϕg(xtϕ,t,c) Jt(z),J1(z)=I.\frac{\mathrm{d}}{\mathrm{d}t} J_t(z) = \nabla_x v_\phi^g(x_t^\phi, t, c)\, J_t(z), \qquad J_1(z) = I. Tracking neighboring trajectories zz and z′=z+ρ1Wξz' = z + \rho_1 W\xi shows why prefix subspace variation ℓkLC(t)\ell_k^{\mathrm{LC}}(t) isolates the spatial nonlinearity of the teacher vector field:

  1. High-Noise Regime (t∈[0.85,1.0]t \in [0.85, 1.0], LCnorm∈[0.006,0.246]\mathrm{LC}_{\mathrm{norm}} \in [0.006, 0.246]): Near Gaussian noise, vϕgv_\phi^g averages over data modes and is spatially smooth/nearly affine (∇xvϕg(xtϕ,t,c)≈∇xvϕg(xt′,t,c)\nabla_x v_\phi^g(x_t^\phi, t, c) \approx \nabla_x v_\phi^g(x_t', t, c), i.e., ∇x2vϕg≈0\nabla_x^2 v_\phi^g \approx 0). Neighboring states undergo nearly identical linear transforms, so Jt(z)≈Jt(z′)J_t(z) \approx J_t(z') and LC(t)≈0\mathrm{LC}(t) \approx 0. Heavy updates here waste student capacity on a nearly linear field.
  2. Pre-Peak Mode-Splitting Transition (t∈[0.448,0.85]t \in [0.448, 0.85], LCnorm:0.246→0.921\mathrm{LC}_{\mathrm{norm}}: 0.246 \to 0.921): Steep growth in ℓkLC(t)\ell_k^{\mathrm{LC}}(t) occurs only when ∇xvϕg\nabla_x v_\phi^g varies rapidly across space (∇x2vϕg≫0\nabla_x^2 v_\phi^g \gg 0), rotating dominant transport subspaces differently as nearby states split toward distinct semantic layouts (matching the pre-peak drop in cross-attention entropy, r=−0.70r = -0.70, Table 12).
  3. Post-Peak Locally Affine Basin (t∈(0,0.448)t \in (0, 0.448), LCnorm:0.921→0.797\mathrm{LC}_{\mathrm{norm}}: 0.921 \to 0.797): Once trajectories enter committed mode basins past tpeakt_{\mathrm{peak}}, the remaining flow Φtpeak→tϕ\Phi^\phi_{t_{\mathrm{peak}} \to t} becomes locally laminar/affine (∇x2vϕg≈0\nabla_x^2 v_\phi^g \approx 0). Composing with a locally linear suffix creates little additional subspace divergence, so prefix LC stops growing and stabilizes, explaining why linear interpolation at $t^* $ is smooth (Table 1) and one-step Euler jumps from t∗t^* recover x0x_0 (Table 8).

Weighting by prefix LC thus suppresses the pre-bifurcation Gaussian regime (t→1t \to 1) while concentrating student capacity across the nonlinear mode-splitting ascent and committed trajectory states (t≤tpeakt \le t_{\mathrm{peak}}).

2. Why the Small Plateau vs. No-Plateau Gap Validates Our Core Thesis & Saves Compute

The fact that w/ plateau and w/o plateau perform very similarly, while both strongly beat the native baseline, supports CWRD in two complementary ways:

  1. Validates That the Pre-Peak Transition & Peak t∗t^* Drive Distillation Gains: In Table 11, both unmodified CWRD (19.230 at 4 NFE; 19.290 at 8 NFE) and plateau CWRD (18.724 / 19.010 at 4 NFE; 19.305 / 18.990 at 8 NFE) yield substantial gains over Native (21.580 / 20.122), with only a 0.015--0.506 FID difference between them. Contrast this with Table 10, where shifting the pre-peak hinge into high noise degrades FID by +1.721 to +2.509. Together, Tables 10--11 confirm our central argument: accurately locating the pre-peak ascent and LC peak t∗t^* is the primary driver of distillation improvement, whereas fine-grained LC fluctuations in the post-peak regular suffix have minimal impact.
  2. Computational Savings via Post-Peak Truncation: Because prefix profiling integrates forward from t=1t=1 toward t=0t=0, freezing C^i=C~+\hat{C}_{i} = \widetilde{C}_{+} after $t_+ $ (Eq. 8) means CWRD does not need to update LC values after the peak. Once the profile turns over past tpeakt_{\mathrm{peak}}, the 2(r^+1)2(\hat{r}+1) perturbed probe rollouts and Gram-matrix eigendecompositions can terminate early instead of integrating down to t=0t=0, saving ∼35%\sim 35\%--45%45\% of solver evaluations and eigendecompositions while preserving student quality.

3. Longer Training Runs (3×3\times) To directly test whether CWRD's gains persist near convergence, we trained both Native and CWRD on SD3.5 π\pi-Flow (4 NFE) out to 12k steps and evaluated FID (25k) across training iterations:

Step1k4k (Default)8k12k
Baseline FID ↓\downarrow44.1521.8221.2621.52
CWRD FID ↓\downarrow37.9918.7819.0418.84
Δ\Delta−6.16-6.16-3.05-2.22-2.68
  1. Both Faster Convergence and Higher Asymptotic Ceiling: CWRD accelerates early learning (ΔFID=−6.16\Delta\text{FID} = -6.16 at 1k steps), and CWRD at 4k (18.78) already surpasses Native at 8k (21.26) and 12k (21.52) FID.
  2. Persistent Advantage at Convergence: Native saturates around 21.26--21.52 FID between 8k and 12k steps, whereas CWRD converges to 18.78--19.04, maintaining a 2.22--2.68 FID lead. Consistent with our TDM-ft result (24.04→21.9524.04 \to 21.95), reallocating finite student capacity away from pure noise shifts the stationary optimum itself, not just early warmup speed.
Reviewer_GKkuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CWRD, a timestep reweighting strategy for few-step distillation of diffusion and flow models. CWRD runs offline probing with the newly introduced local complexity (LC) metric, which measures how dominant directions of teacher's trajectory Jacobian vary across nearby latent states. The averaged LC profile is then modified with a post-peak plateau and a floor to construct a fixed loss weighting schedule. Experiments across multiple T2I models appear to show consistent FID improvements over the baselines.

Critical strengths
  • The quantitative experiments appear to show consistent FID improvements over multiple distillation baselines (pi-Flow, LADD, TDM, MFD) across multiple base models (SD3.5, FLUX.2). However, the details of these baselines are not well-documented, thus these gains are not convincing enough for acceptance.
Critical weaknesses
  • The paper proposes a very elaborated method for a very simple and still mostly heuristic weighting strategy. The LC profile introduces a lot of math and needs expensive offline procedure, which ends up becoming just a static time-dependent weighting. The profile is further modified heuristically with the plateau and the floor, and the final weighting (from Appendix D) is nearly a simple increasing function of the denoising progress. This raises the question of whether the complexity is necessary.
  • The optimality of the weighting appears not sufficiently validated, neither theoretically nor empirically.
    • Theoretically, the relationship between the variation of sensitive directions and few-step distillation is not clearly established. The connections appear weak, and other straightforward alternatives like endpoint sensitivity magnitude (norm of the Jacobian) are not discussed.
    • Empirically, the effective time weighting also depends on time distribution, yet the paper does not compare with baselines with manually tuned time sampling distribution.
  • Qualitatively, the baseline results look weak. It appears that at least some of the baselines are the paper's reproduction (because some distillation methods don't have official checkpoints on the teachers used in this paper). Yet, the reproduction details are not given, raising concerns about whether the baseline themselves are solid enough.
  • LC estimator details are not given in the PDF, e.g., the number of sampled directions, the rank k, etc. These choices are also not ablated.
Questions for authors

Please clarify the details of the baselines: what baselines are from the official checkpoint, what are reproduced, what are the hyperparameters. Appendix A states that the paper follows official implementations, but many hyperparameters may not be directly transferable to a new teacher, thus a full report of the details would be better to clarify it.

In addition, please explain why a more direct probe (like an estimation of the norm of the Jacobian) or other alternatives are not considered, as well as the details such as the number of sampled directions, the rank k, etc.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.