- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates timestep training resource allocation for flow matching distillation. Existing few-step distillation mostly uses fixed or analytic time-sampling weights and overlooks the intrinsic geometry of the flow ODE trajectory. The authors adapt Local Complexity (LC) into a temporal analysis tool to quantify how the endpoint-sensitive subspace varies across nearby latents at different timesteps. They observe an LC peak along the trajectory: latent geometry changes rapidly before this peak, while the trajectory becomes highly compressible afterward.
Based on this finding, they propose CWRD (Complexity-Weighted Rectified Distillation), a plug-and-play training wrapper. It offline computes the LC temporal profile from the teacher and reweights timestep sampling probabilities to allocate more training budget to high-LC timesteps. It requires no changes to the student network, inference pipeline or per-sample loss function.
Evaluations are conducted for 4/8-NFE distillation across four distillation frameworks (π-Flow, LADD, TDM, MFD) and three large teacher models (SD3.5, SANA, Flux.2). Compared with vanilla sampling and the analytic weighting from SenseFlow, CWRD improves FID and several CLIP / VQAScore metrics. Interpolation experiments, single-step suffix recovery, time-shift ablations and cross-attention entropy correlation analysis support the significance of the LC peak.
Critical strengths
- This work examines timestep resource allocation for distillation from the geometric perspective of flow ODEs, departing from conventional heuristic weighting.
- This work is a plug-and-play module compatible with multiple flow-matching distillation frameworks.
Critical weaknesses
-
The LC computation pipeline involves numerous hyperparameters (differentiation radius, subspace dimension k, thresholds, truncation strategy). Systematic full hyperparameter sweeps are missing, raising concerns about the robustness of performance gains to tuning.
-
The LC peak and optimal performance only demonstrate statistical correlation and do not establish that the peak corresponds to the global optimum. Time-shift ablations merely show performance degradation under misalignment and cannot validate optimality at the peak location.
-
Offline precomputation of the teacher LC profile incurs upfront computational cost.
Questions for authors
-
Could the authors conduct a full suite of ablation studies over all LC hyperparameters to verify the robustness of performance gains against each hyperparameter?
-
A grid sweep over hinge positions across the full timeline is expected to verify whether the LC peak indeed marks the optimal training point, so as to disentangle correlation from causation.
-
Please discuss potential strategies to mitigate the computational overhead incurred by offline precomputation of the teacher LC profile.
We thank Reviewer hAwM for the thoughtful review and constructive questions on hyperparameter robustness, the hinge-shift sweep, and offline profiling cost. All new sweeps and analyses are in the revised PDF.
1. Robustness to LC Estimator Hyperparameters (W1, Q1)
Structural Insensitivity. CWRD maps raw C(ti) via min-max normalization onto [0,1] (Eq. 7), followed by flooring (γ=0.2) and grid-mean normalization (Eq. 9). Because positive affine scalings cancel out, training weights wi depend only on the profile's ordinal shape and hinge t∗. All teachers and hosts use the exact same default hyperparameters without per-model tuning. Profile & Truncation Sweeps (App. A Table 5; App. C.2 Table 11). Across 64 runs on SD3.5-Medium (12-step, CFG g=4.5, native
bf16), sweeping every estimator parameter over 10×-100× ranges preserves high Spearman rank correlation ρ with the default profile and shifts E[t∗] by ≤0.05:We also ablated the post-peak truncation/plateau strategy end-to-end on SD3.5 π-Flow (Table 11):
2. Full-Timeline Hinge Grid Sweep: Correlation vs. Optimality (W2, Q2)
We agree an empirical profile cannot prove theoretical global optimality; rather, the frozen LC profile automatically locates the high-performing transition region without an expensive per-teacher training grid search. App. C.1 (Table 10) sweeps 7 hinge positions sr=clip(σ∗+r,0.05,0.95) across [0.05,0.95] around σ∗≈0.633 on SD3.5 π-Flow (4 NFE; native FID 21.580):
3. Offline Profiling Cost & Mitigation Strategies (W3, Q3)
Exact Cost via Prefix Truncation (Sec. 3.2). Because deterministic ODE integration yields all M anchors from one rollout Φ1→0ϕ(z,c) (0.0 diff vs. truncated rollouts), profiling z,z′ costs only 2(r^+1)=130 rollouts/run vs. 1,584 for suffix loops (6.6× fewer solver steps). Computed once per teacher, it is reused across all hosts (π-Flow, LADD, TDM-ft, SenseFlow-ft) and NFE budgets with zero training or inference overhead.
Four Cost-Reduction Strategies (Table 5; 16×--32× Faster):