- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper suggests a post-training method (TSS) to better align training with the sampling trajectories. For JiT (x-prediction), TSS-X shifts the noisy training input slightly toward the EMA model's clean-image prediction, while keeping the true clean image as target. For velocity models, the path is bent and the model is trained on its exact derivative (full TSS), or on a pure time warp (TSS-T). The authors argue that this improves sample quality at low NFE and produces better (noise, sample) pairs for distillation, which train reflow and flow-map students in fewer updates.
Critical strengths
-
The gain in FID is real for low NFE
-
Experiments cover three model sizes, two solvers and three NFE budgets, although L/16 and H/16 are single runs
Critical weaknesses
-
Paper seems to be about exposure bias, which addresses the misalignment problem between training and sampling. It is a common problem in diffusion models in general, but the paper never mentions it or cites paper related to that. In addition, the main method (TSS-X) is only shown to work on JiT. The velocity version (TSS-T / full TSS) is a different method and only helps at 10 NFE on LightningDiT (Table 10). To be thoroughly tested, it should be applied to various diffusion models and parameterisation. At best it would work on different parameterisation, or then it would expose a failure mode of x-prediction that would then require an explanation on why their method works here and not else where. The field would then gain deeper understanding of JiT like that. Results on other models seem unlikely given that the TSS-X transfer to LightningDiT was negative (A.4.6), but that limitation should be made clear upfront. Also, TSS-T, a pure time warp, gives most of the latent gain, so schedule baselines (e.g. time-shift sampler, vanilla on the warped sampling grid) are necessary. The gains are modest, which is ok in my opinion if the paper is able to explain better its motivation and the failure modes, so that we have a better understanding of the mechanism.
-
A gain of 1 FID over 10-20 NFE is real but quite modest. Table 2 has no error bars and does not say which rows are multi-seed (L/16 and H/16 are single runs), so it is hard to really quantify how substantial the contribution is. Also the slight degradation at higher NFE is slightly worrying (within noise for Heun-50, but positive in all three seeds for DPM++2M at 50 NFE, Table 8). The method should in the worst case not degrade at higher NFE. I am willing to buy that there is no degradation if error bounds are provided.
-
The distillation gain can come from better targets or better couplings. Better targets is a weak argument: generating the pairs is a one-time cost, so saving a few NFE is not really worthwhile. In addition, vanilla at 14 NFE already matches TSS-X at 10 NFE (FID 8.11 vs 8.22, Table 26), so the whole procedure does not even buy 4 sampling steps. The natural baseline is vanilla at a much higher budget (e.g. 50-100 NFE), which is missing. Vanilla19 is used for the 47.4% cost saving claim, but its endpoint FID is never reported.
-
Better coupling is a solid hypothesis, and it is in the title, but it is never tested. The only evidence is the transport cost and cosine between noise and endpoint (Table 26), and the two do not agree on the ordering (Vanilla14). A cosine of 0.06-0.09 is close to orthogonal; a reference point is needed to know if it is meaningful (e.g. shuffled pairs vs. minibatch OT pairs, vanilla at higher NFE).
Questions for authors
-
How does TSS-X compare to exposure-bias methods (e.g. DDPM-IP, epsilon scaling, time-shift sampler) ?
-
How do students trained on vanilla pairs generated at 50-100 NFE compare?
-
Can you test the coupling hypothesis directly (e.g. the straightness measure of Liu et al. 2023, or cosine and transport cost compared to shuffled and minibatch OT pairs)?
-
Why does TSS-X transfer negatively to LightningDiT?
2 comments
I acknowledge and thank the authors for their response and will raise my score to 2.
Thank you for the exposure-bias references and the questions about downstream data value.
1. Exposure-bias methods and scope. We will expand the related work to include DDPM-IP, epsilon scaling, Time-Shift Sampler, and MixFlow. The training/sampling mismatch is established prior motivation. TSS-X's particular intervention uses a detached EMA clean-image estimate to choose a sample-dependent displacement, concentrated around an anchor, while retaining the clean-image target. This differs from perturbing inputs with additional noise, scaling predictions at inference, or changing inference query times.
We completed a common-environment FID50K evaluation of existing single-seed JiT MixFlow adaptations. All four arms use a clean-image head and 100-epoch continuations from the same epoch-600 source. DPM++2M results are:
TSS-X improves on both adaptations; vanilla remains best at NFE50. This tests two JiT adaptations, not official MixFlow reproduction or exhaustive tuning.
2. Effect sizes and high-NFE regression. We will identify B/16 as three paired seeds and L/H as single pairs. Paired Student-t intervals from the original FID50K files give:
All three differences favor TSS-X at 10/20 NFE and vanilla at 50 NFE. We retain the high-NFE regression. These three-seed intervals are not multiplicity corrected. H/16 improves from 7.6652 to 6.6524 at 10 NFE and 4.9841 to 4.2417 at 20 NFE (13.2% and 14.9%), with a +0.0584 regression at 50 NFE. The practical claim is low-budget sampling improvement.
3. Higher-quality vanilla pools and costs. The 19-call vanilla endpoint pool's measured FID is 7.0070, compared with 9.4154 for vanilla at 10 calls and 8.2179 for TSS-X at 10 calls. We will make this value explicit next to the student table.
Despite its better endpoint FID, the 19-call pool gives one-call reflow-student EMA FIDs 32.3649/31.8548, versus 29.7561/29.6124 from TSS-X in the two matched student streams. At 15 student calls, vanilla-19 instead wins: 13.2008/13.2715 versus 13.5930/13.5491. The 14-call, endpoint-FID-matched control also produces worse students than TSS-X in both streams. This establishes that endpoint FID alone does not predict downstream performance. We have not trained the proposed 50/100-call vanilla-pool students, so these controls do not establish superiority over such a source.
The efficiency measurements concern student updates and pool size. Teacher post-training and pool generation cost extra; fewer generation calls alone do not establish full-pipeline savings.
4. Coupling geometry and straightness. Your distinction is important. Endpoint distance and cosine do not establish straighter solver trajectories or isolate pairing from image quality. We subsequently measured 8,192 trajectories per model for each of three matched JiT seeds. At 50 DPM++2M calls, mean path-length/chord ratios are 1.0096 for vanilla and 1.0127 for TSS-X; midpoint normalized chord residuals are 0.0421 and 0.0500. Thus these diagnostics do not support straightening as the explanation. We will replace that interpretation with the measured downstream data-source result.
A within-class permutation or matching on the exact same endpoint/noise sets would isolate coupling more directly. Such a student comparison is still missing; neither the small endpoint cosines nor the scalar FID match substitutes for it.
5. Latent transfer. The negative TSS-X transfer jointly changes the prediction head, MSE-plus-cosine loss, denominator floor, time distribution and shaping profile; it does not isolate a cause. A new three-seed Euler-NFE10 control gives 13.9057 for frozen vanilla with exact inference-time reparameterization versus 13.9658 for TSS-T training. There is no demonstrated training advantage over that baseline at this point. We will distinguish these recipes and retain their high-NFE limitations.