- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper aims to improve few-step generation of temporal data using flow matching. The intuition is starting from a noise whose covariance is close to the target data, instead of pure white noise that is independent across dimensions. The paper first provide theoretical intuitions on Gaussian data, where a matched covariance minimizes the distance when using Euler sampler. They then implement this idea using length-dependent temporal covariance estimates for sequence generation, without modifying the model architecture or training objective. Experiments suggest improvements over several alternative noise sources.
Critical strengths
- This paper provide a simple but useful perspective on few-step generation, which can improve the performance without changing the training and sampling algorithm.
- The authors compare multiple source distributions, and these experiments provide preliminary evidence that source design can affect practical generation quality.
Critical weaknesses
- The presentation. First, the storyline is not so clear, and it is difficult for me to find i) what are the most important novelties compared to existing methods, since the intuition is very simple, and ii) the implementations useful for understanding the method, because there are too many references to the appendix. Second, the theoretical intuition uses a simplified setting and is much different to what the paper actually implements (e.g., block-restricted covariance). Since this paper mainly focuses on the experiments, such a theoretial intuition might be unnecessary to appear in its position.
Questions for authors
- Your intuition suggests a full covariance matrix, but the experiments use covariances restricted to each block. How does the size of the blocks affect the performance in practice? Is there any theoretical intuitions for it?
4 comments
Response to Reviewer UBzP (2/2)
2. W1(i): Storyline and novelty relative to existing methods
A natural starting point is indeed to choose a source distribution that is globally close to the target data. However, our counterexample in Section 2.5 (Table 11) shows that generic distributional closeness does not govern few-step generation accuracy (at , the periodic GP source is closer to real motion windows than OU and white noise in empirical and energy distance, yet has much worse 24-step FID: vs. and ). Our paper therefore asks: at a fixed sampling step budget, which source covariance minimizes the endpoint error? The revised Section 1 opens directly from this question.
Whereas prior methods use fixed stationary kernels (TSFlow), one-parameter video noise heuristics (PYoCo), or additional learned condition-dependent prior networks (PriorGrad, CW-Gen), Theorem 1 derives the closed-form optimum of the finite-step Euler endpoint problem for arbitrary covariances, and TSCM realizes it via offline uncentered empirical second moments with no extra network overhead.
3. W1(ii): Implementation details in the main text
Sections 3.1–3.2 present the three steps directly: (1) estimate temporal second moments from training windows without subtracting each window's own mean; (2) regularize with Ledoit–Wolf shrinkage and precompute Cholesky factors; and (3) multiply ordinary white noise by the factor at the sequence's length in both training and sampling. Equations (8) and (9) give the exact constructions of TSCM and TSCM-Toeplitz. Thank you for pointing out the excessive appendix pointers in Section 3: we have moved the implementation details from Appendices A.9 and B.5 into Sections 3.1–3.2 (while keeping the 16-source evaluation in Section 2.5), so the method implementation is self-contained in the main text.
Thanks for your response. I will keep my score.
Response to Reviewer UBzP's Follow-up Comment
Thank you for your continued support and for taking the time to consider our response! We sincerely appreciate your constructive feedback, which has helped us improve our paper.
Response to Reviewer UBzP (1/2)
Thank you for your constructive review, and for recognizing that our paper provides a simple and practical perspective on few-step generation that improves performance without changing the training or sampling algorithm, supported by controlled multi-source comparisons.
1. Q1 and W2: How block size affects performance in practice, and the theoretical intuition behind it
Larger blocks retain more correlations but require more data for accurate estimation. Per-channel blocks retain dependencies across time within each channel; joint channel–time blocks also retain dependencies between channels. Choosing the block size therefore requires considering both the correlations present in the data and whether the available samples suffice to estimate them.
Both the theoretical explanation for block size and our practical selection criterion follow directly from the training-free oracle risk SN(A;Bref) derived in Section 2.2 (Eq. 5). This is also why the theory in Section 2 is essential to the main text: it provides a quantitative criterion that tells us, before training, which block size will work best.
Specifically, Theorem 1 shows that omitting correlated off-diagonal blocks—using blocks that are too small—causes generalized eigenvalues rk=1, increasing SN. Meanwhile, Corollary A.1, now stated in Section 2.2, shows that finite-sample estimation error δ in overly large blocks incurs O(δ2) excess risk, which also increases SN when evaluated against a held-out covariance Bref.
We therefore assess candidate block sizes before training using the held-out oracle risk S24 together with the estimation error scale d/n. When d/n approaches or exceeds 1, we avoid the joint block.
1. Why cannot the joint block be reliably estimated on HumanML3D?
2. Why can the joint block be estimated for video?
To test this prediction in practice, we added a UCF-101 experiment, fine-tuning Wan2.1-T2V-1.3B for 6 epochs across three random seeds (Section 5, Table 3). The actual FVD results under both 16-step Euler and 25-step UniPC validate the block-size ordering predicted by S24, with the 144×144 joint block achieving the best results on every seed (visual comparisons are provided in Appendix B.20, Figure 14):