acceptodds
Under review as a conference paper at ICLR 2027

Temporal Source Covariance Matching

Abstract

Which source covariance should a few-step flow-matching model use? We formulate source selection as a finite-step numerical design problem. For Gaussian population fields under independent coupling and linear interpolation, we prove that covariance matching uniquely minimizes endpoint Wasserstein error for every uniform Euler steps, including non-commuting source covariances. The resulting covariance-only oracle has second-order excess risk near the optimum. Temporal Source Covariance Matching (TSCM) implements this principle with an offline, length-indexed Gaussian source estimated from within-channel temporal second moments and used in both training and sampling, without changing the architecture or objective. On HumanML3D, a controlled comparison of 16 sources with three training seeds each gives mean 24-step FIDs of for TSCM, for uncentered Toeplitz, and for white noise, with source-specific constant guidance selected on validation. Oracle risk is associated with these guided test FIDs (log–log Pearson ). Frozen-field controls also reveal learned-flow differences that persist under reference integration. Further analyses characterize solver-dependent calibration, conditional shared-source regret, and local covariance-deficit minimaxity beyond Gaussian targets. A separate guidance analysis identifies an early covariance bias; matched source replacements in other sequence generators assess empirical transfer.

Primary area:
generative models
Keywords:
flow matching, diffusion models, source covariance, temporal covariance matching, few-step sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UBzPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to improve few-step generation of temporal data using flow matching. The intuition is starting from a noise whose covariance is close to the target data, instead of pure white noise that is independent across dimensions. The paper first provide theoretical intuitions on Gaussian data, where a matched covariance minimizes the W2W_2 distance when using Euler sampler. They then implement this idea using length-dependent temporal covariance estimates for sequence generation, without modifying the model architecture or training objective. Experiments suggest improvements over several alternative noise sources.

Critical strengths
  • This paper provide a simple but useful perspective on few-step generation, which can improve the performance without changing the training and sampling algorithm.
  • The authors compare multiple source distributions, and these experiments provide preliminary evidence that source design can affect practical generation quality.
Critical weaknesses
  • The presentation. First, the storyline is not so clear, and it is difficult for me to find i) what are the most important novelties compared to existing methods, since the intuition is very simple, and ii) the implementations useful for understanding the method, because there are too many references to the appendix. Second, the theoretical intuition uses a simplified setting and is much different to what the paper actually implements (e.g., block-restricted covariance). Since this paper mainly focuses on the experiments, such a theoretial intuition might be unnecessary to appear in its position.
Questions for authors
  • Your intuition suggests a full covariance matrix, but the experiments use covariances restricted to each block. How does the size of the blocks affect the performance in practice? Is there any theoretical intuitions for it?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UBzP (1/2)

Thank you for your constructive review, and for recognizing that our paper provides a simple and practical perspective on few-step generation that improves performance without changing the training or sampling algorithm, supported by controlled multi-source comparisons.

1. Q1 and W2: How block size affects performance in practice, and the theoretical intuition behind it

Larger blocks retain more correlations but require more data for accurate estimation. Per-channel blocks retain dependencies across time within each channel; joint channel–time blocks also retain dependencies between channels. Choosing the block size therefore requires considering both the correlations present in the data and whether the available samples suffice to estimate them.

Both the theoretical explanation for block size and our practical selection criterion follow directly from the training-free oracle risk SN(A^;B^ref)S_N(\widehat A;\widehat B_{\mathrm{ref}}) derived in Section 2.2 (Eq. 5). This is also why the theory in Section 2 is essential to the main text: it provides a quantitative criterion that tells us, before training, which block size will work best.

Specifically, Theorem 1 shows that omitting correlated off-diagonal blocks—using blocks that are too small—causes generalized eigenvalues rk≠1r_k\neq1, increasing SNS_N. Meanwhile, Corollary A.1, now stated in Section 2.2, shows that finite-sample estimation error δ\delta in overly large blocks incurs O(δ2)O(\delta^2) excess risk, which also increases SNS_N when evaluated against a held-out covariance B^ref\widehat B_{\mathrm{ref}}.

We therefore assess candidate block sizes before training using the held-out oracle risk S24S_{24} together with the estimation error scale d/n\sqrt{d/n}. When d/n\sqrt{d/n} approaches or exceeds 11, we avoid the joint block.

1. Why cannot the joint block be reliably estimated on HumanML3D?

  • The joint block has dimension 263 channels×196 frames=51,548263\text{ channels}\times196\text{ frames}=51{,}548.
  • Taking one window from each training motion of at least 196 frames provides only 8,620 samples.
  • A matrix estimated from 8,620 samples has rank at most 8,620, leaving more than 40,000 directions unrepresented. The estimation error scale is d/n≈2.45\sqrt{d/n}\approx2.45, exceeding the scale of the matrix itself.
  • Each per-channel block, by contrast, has only 196 dimensions. With the same 8,620 samples, the estimation error scale drops to d/n≈0.15\sqrt{d/n}\approx0.15, favoring per-channel estimation.

2. Why can the joint block be estimated for video?

  • Wan’s VAE compresses each video clip into 16 channels and 9 latent frames, giving a joint dimension of only 144.
  • Each of the 32×3232\times32 spatial locations contributes a sample, yielding approximately 15,927 clips×1,024 locations≈16.315{,}927\text{ clips}\times1{,}024\text{ locations}\approx16.3 million samples.
  • The estimation error scale is d/n=144/(1.63×107)≈0.003\sqrt{d/n}=\sqrt{144/(1.63\times10^7)}\approx0.003, favoring joint-block estimation.

To test this prediction in practice, we added a UCF-101 experiment, fine-tuning Wan2.1-T2V-1.3B for 6 epochs across three random seeds (Section 5, Table 3). The actual FVD results under both 16-step Euler and 25-step UniPC validate the block-size ordering predicted by S24S_{24}, with the 144×144144\times144 joint block achieving the best results on every seed (visual comparisons are provided in Appendix B.20, Figure 14):

Noise source / block sizePre-training S24↓S_{24}\downarrow16-step Euler FVD$\downarrow$ (3 seeds)25-step UniPC FVD$\downarrow$
White noise (1×11\times1)0.252399.6±13.8399.6\pm13.8416.5
PYoCo mixed (α=1\alpha=1)0.175363.3±18.8363.3\pm18.8348.5
PYoCo progressive (α=2\alpha=2)0.167349.7±10.7349.7\pm10.7327.1
TSCM (per-channel blocks, 9×99\times9)0.164348.3±16.8348.3\pm16.8315.4
TSCM (joint channel–time block, 144×144144\times144)0.131327.3±11.2327.3\pm11.2294.8
AuthorsOpenReview ↗

Response to Reviewer UBzP (2/2)

2. W1(i): Storyline and novelty relative to existing methods

A natural starting point is indeed to choose a source distribution that is globally close to the target data. However, our counterexample in Section 2.5 (Table 11) shows that generic distributional closeness does not govern few-step generation accuracy (at L=64L = 64, the periodic GP source is closer to real motion windows than OU and white noise in empirical W2W_2 and energy distance, yet has much worse 24-step FID: 1.8671.867 vs. 0.2070.207 and 0.4970.497). Our paper therefore asks: at a fixed sampling step budget, which source covariance minimizes the endpoint error? The revised Section 1 opens directly from this question.

Whereas prior methods use fixed stationary kernels (TSFlow), one-parameter video noise heuristics (PYoCo), or additional learned condition-dependent prior networks (PriorGrad, CW-Gen), Theorem 1 derives the closed-form optimum of the finite-step Euler endpoint problem for arbitrary covariances, and TSCM realizes it via offline uncentered empirical second moments with no extra network overhead.

3. W1(ii): Implementation details in the main text

Sections 3.1–3.2 present the three steps directly: (1) estimate temporal second moments from training windows without subtracting each window's own mean; (2) regularize with Ledoit–Wolf shrinkage and precompute Cholesky factors; and (3) multiply ordinary white noise by the factor at the sequence's length in both training and sampling. Equations (8) and (9) give the exact constructions of TSCM and TSCM-Toeplitz. Thank you for pointing out the excessive appendix pointers in Section 3: we have moved the implementation details from Appendices A.9 and B.5 into Sections 3.1–3.2 (while keeping the 16-source evaluation in Section 2.5), so the method implementation is self-contained in the main text.

Thanks for your response. I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer UBzP's Follow-up Comment

Thank you for your continued support and for taking the time to consider our response! We sincerely appreciate your constructive feedback, which has helped us improve our paper.

Reviewer_zsBcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper starts with a well defined theoretical problem. Using linear interpolants to generate gaussian data with covariance matrix B, starting from colored noise with covariance matrix A, and applying a Euler discretization of the generative process, it derives the endpoint error. The result is simple, it depends only on the eigenvalue of B^{-1/2} A B^{-1/2}, as the modes decouple under appropriate rotation. Then one can optimize the source covariance in order to reduce the error made in the generation with a finite number of discretization steps. Relying on this result, the authors propose a practical optimization of the source that they apply to video-type data, temporal-source covariance matching (TSCM). This is an improvement with respect to previous attempts at adapting the source covariance to the data. It is a rather simple scheme and experiments show that it provides some improvement.

Critical strengths

There is a clear theoretical result giving the error made in generation using any number of discrete Euler steps. This suggests a practical choice of source covariance, TSCM Its effectiveness is tested in substantial numerical experiments

Critical weaknesses
  • A first main weakness is the presentation. The main paper is written in a way which is very dense, not self-contained. It seems that the authors are writing only for the group of experts of this subfield, without paying any attention to readability by a somewhat broader audience. For instance, clear statement about the role of physical time should be explained. The theory parts applies to Gaussian distributions in general. But the motivation, and the experiments, deal with a specific situation in which data has a temporal structure, without ever introducing it clearly (see the first paragraph for instance) . This is not well explained, and one has to discover it gradually with the introduction of block-diagonal covariance. A clearer separation between the general general Gaussian theory and its implementation in terms of temporally structured data, together with a clear distinction between the physical time and the generative time, would substantially improve accessibility.

  • The practical covariance used in experiments uses within channel temporal blocks and does not necessarily give the optimal covariance when the target has cross-channel correlations. Therefore the theorem gives a motivation for the TSCM but does not give quality guarantee

Questions for authors
  • As explained above, the presentation should be substantially improved. Maybe introducing form the beginning temporal sequence data, their representation, the structure and meaning of their covariance.
  • The paper needs a sharper distinction between what is proved, what motivates TSCM, and what the experiments demonstrate.
  • It should be clearer about the novelty TSCM with respect to previous attempts at optimizing the covariance of the source.
2 comments
AuthorsOpenReview ↗

Response to Reviewer zsBc (1/2)

Thank you for your thorough review, and for recognizing that our paper provides a clear theoretical result characterizing finite-step Euler generation error at any number of steps, which naturally motivates TSCM and is validated across substantial numerical experiments.

We agree with your suggestions for improving the presentation—introducing temporal sequence structure from the very beginning and distinguishing physical time from generative time, separating the general Gaussian theory from its temporal block-diagonal instantiation, and clarifying the boundary between what is mathematically proved and what motivates TSCM. Following your advice, we have reorganized Sections 1–3 in the revised PDF and moved the key stability bound and implementation details from the appendix into the main text.

1. W1 and Q1: Temporal sequence setting, physical time vs. generative time, and separating the general theory from the temporal block implementation

Following your suggestion, Section 1 now opens directly with multivariate time series. Sections 2.1–2.4 analyze a general vector x∈Rdx \in \mathbb{R}^d along generative time t∈[0,1]t \in [0,1], without temporal block notation. Section 3 then introduces the temporal block structure, with self-contained formulas for TSCM and TSCM-Toeplitz in Sections 3.1–3.3.

2. Q3: Novelty relative to existing approaches to source covariance optimization

Making the source distribution closer to the target data is a natural starting point and the idea behind many existing methods. This was also our initial intuition, from which TSCM-Toeplitz emerged: estimating temporal correlations from data while retaining differences in the overall level across sequences, so that the noise more closely matches these data statistics.

However, a source being closer to the data does not necessarily make it better for few-step generation. Section 2.5 provides a counterexample: the periodic Gaussian-process source is closer to real motion windows than OU noise and white noise in both empirical W2W_2 and energy distance, yet produces worse 24-step FID. We therefore sharpen the question: for a given number of sampling steps, how should we choose the source covariance to minimize the generation error at the endpoint?

Our core contribution is a computable error criterion and an optimality result for this question. Under an exact Gaussian velocity field, independent coupling, linear interpolation, and uniform Euler sampling, we derive the finite-step endpoint error, prove that A=BA=B is the unique optimum for every N≥2N \ge 2, and obtain a training-free score SNS_N for comparing candidate sources. This result provides the basis for moving beyond the structured approximation of TSCM-Toeplitz to directly estimating the full within-block temporal second moment at each sequence length.

Thus, compared with TSFlow’s fixed GP kernels, PYoCo’s single-parameter noise designs, and the adaptive or conditional priors of PriorGrad and CW-Gen, we provide both a data-driven noise construction and an explicit finite-step error objective with its theoretical optimum. TSCM implements this design through offline statistics, without adding a prior network or changing the generator’s architecture or training objective.

AuthorsOpenReview ↗

Response to Reviewer zsBc (2/2)

3. W2 and Q2: What is proved, what motivates TSCM (cross-channel correlation), and what the experiments demonstrate

I. What is mathematically proved (Section 2).

  1. The exact endpoint error of each generalized mode under uniform-step Euler integration (Proposition 2, Section 2.2).
  2. When AA and BB commute, the weighted sum SN(A;B)S_N(A;B) of these modal errors equals the exact endpoint W2W_2 distance; when they do not commute, SN(A;B)S_N(A;B) upper-bounds the endpoint W2W_2 distance (Proposition 3, Section 2.2).
  3. Under the exact Gaussian population field, independent coupling, and linear interpolation, A=BA = B is the unique optimum for every finite step count N≥2N \ge 2, including non-commuting AA (Theorem 1, Section 2.2).
  4. The objective is flat near the optimum: when the source covariance is within relative spectral error δ\delta of the target covariance, the excess risk is second order, O(δ2)O(\delta^2) (Corollary A.1, stated in Section 2.2).

II. How these theoretical results motivate TSCM's design (Section 3).

  1. From Result 1 (Proposition 2): the noise should retain the distinct types of variation present in the data. In the theoretical model, if the noise has too much or too little variance along a direction, the generative process must contract or expand it, increasing few-step Euler error. This motivates retaining both types of variation: sequence-to-sequence level differences and within-sequence temporal fluctuations. Accordingly, when estimating TSCM statistics (Eq. 8), we do not subtract each sequence's own temporal mean, so as not to discard the level variation.
  2. From Result 2 (Proposition 3): compute a training-free score to compare noise designs beforehand. Proposition 3 combines the directional errors into a single score SNS_N that requires only the data and candidate noise covariances, with no model training. We can therefore compare candidate noise designs before training and, whenever a reference covariance can be estimated reliably, check whether including cross-channel correlations lowers this score.
  3. From Result 3 (Theorem 1): construct the noise directly from the data covariance. Theorem 1 identifies the optimal target in the idealized setting: matching the data covariance. TSCM therefore estimates these statistics offline from training data to construct a Gaussian source, using the same source in both training and sampling.
  4. From Result 4 (Corollary A.1): a sufficiently close covariance estimate stays close to the theoretical optimum. Because small covariance estimation errors incur only second-order excess risk O(δ2)O(\delta^2), approximate matching from finite samples is well-founded. In choosing block size, larger blocks retain more correlations but need more data to estimate accurately (d/n\sqrt{d/n}), whereas smaller blocks are easier to estimate but omit inter-block correlations. On HumanML3D, the full joint dimension (d=51,548d = 51{,}548) far exceeds the sample size (n=8,620n = 8{,}620, d/n≈2.45\sqrt{d/n} \approx 2.45), so we use per-channel temporal blocks (d/n≈0.15\sqrt{d/n} \approx 0.15; synthetic cross-channel tests in Section 3 and Appendix A.6, Table 5 show that up to ρ=0.95\rho = 0.95, the per-channel source stays within 6.6%6.6\% of the best channel-independent source). On Wan2.1 video latents, the joint dimension is only d=144d = 144 with ample pooled samples (d/n≈0.003\sqrt{d/n} \approx 0.003), so we directly estimate the 144×144144\times 144 joint channel–time block, and verify via Result 2 that held-out S24S_{24} drops from 0.1640.164 to 0.1310.131.

As you noted, matching per-channel blocks is not identical to matching the full covariance, and a learned neural network is not the exact population velocity field. Thus, the results above explain what motivates TSCM's design (stated explicitly in Sections 1 and 3); whether actual generation quality improves is answered by the experiments below.

III. What the experiments demonstrate (Sections 2.5 and 5).

  1. On HumanML3D (16 sources ×\times 3 seeds), pre-training oracle risk S24S_{24} correlates strongly with trained-model FID (r=0.93r = 0.93), and TSCM lowers 24-step FID from 0.4970.497 (white noise) to 0.1080.108, while reference-integration diagnostics (Figure 3) show that covariance matching both reduces finite-step discretization error and makes the continuous velocity field easier to learn.
  2. On eight standard time-series benchmarks (Table 1, three seeds per dataset), TSCM achieves the lowest 4-step Euler W2W_2 on all eight datasets.
  3. To directly test cross-channel covariance matching on video, we added a UCF-101 experiment, fine-tuning Wan2.1-T2V-1.3B for 6 epochs across three seeds (Section 5, Table 3; visual comparisons in Appendix B.20, Figure 14); the mean FVD under both 16-step Euler and 25-step UniPC follows the S24S_{24} ordering, with the joint 144×144144\times 144 block best on every seed (Table 21).
Reviewer_DjGmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For Gaussian targets under independent coupling, the paper derives the Euler endpoint error in closed form and shows that matching the source covariance to the data (A = B) minimizes endpoint W2 for any N ≥ 2. TSCM uses length-indexed temporal second moments as the source. Validation includes a 16-source, three-seed study on HumanML3D (FID 0.108 vs 0.497 for white noise), transfer to several motion generators, time series, and a Wan2.1 fine-tune.

Critical strengths
  1. The derivation of finite-step Euler error is careful and fairly rigorous. It turns "match the data covariance" into a precise statement within its setting. The cosine-product view of the Euler multiplier and the early CFG term t²w(w−1)V are also useful on their own.
  2. The controlled study (same architecture, budget and guidance search, three seeds per source) seems well designed, and gains appear to transfer to strong generators. The swap sources that isolate the level component help attribute the effect to specific covariance properties rather than to tuning.
Critical weaknesses
  1. Evidence beyond motion is limited. The headline "log–log Pearson r = 0.93" (Abstract, p. 1, l. 021) seems driven by a few outlying sources, and Sec. 2.5 notes that "the FID correlation is negative at w = 1 and w = 10" (p. 5, l. 247–248). The Sky Time-lapse result (Table 2, p. 8, l. 424–431: "228.7 ... 215.2 ... 201.9") improves with fewer steps, uses a single "seed 3407" and "UniPC" (App. B.17, p. 60, l. 3193–3196), and the white_ft3 panels in the supplementary video (around 5:45–6:00) look undertrained.
  2. The paper frames the problem as numerical (Abstract, p. 1, l. 010: "formulate source selection as a finite-step numerical design problem") and separates itself from prior work whose analyses "address ... training conditioning and data-to-prior divergence" (Sec. 1, p. 2, l. 089). Yet Sec. 5 reports "RK4-192 FIDs of 2.36 ± 0.20 for white noise and 0.91 ± 0.13 for TSCM" (p. 9, l. 432–433), so much of the gain may come from an easier-to-learn field rather than reduced discretization error. The framing may overstate what the theory explains.
Questions for authors
  1. Sec. 5 (p. 9, l. 432–433) and Sec. 2.5 (p. 5, l. 243, "unguided Euler-24–RK4-192 endpoint residual"): what fraction of the white-to-TSCM Euler-24 FID gap remains at RK4-192?
  2. Sec. 2.5 (p. 5, l. 237) and Table 8 (p. 45, l. 2383): what is the rank correlation of S24 with FID among the near-matched sources only?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DjGm (1/2)

Thank you for your insightful review, and for recognizing the rigor of our finite-step Euler error derivation, the cosine-product perspective, the early CFG bias term t2w(w−1)Vt^2w(w-1)V, and our controlled 16-source, three-seed experimental design.

1. Q2 and W1: Rank correlation within near-matched sources and outlier checks

For Q2, if we look only at the 11 near-matched sources (the 10 Toeplitz variants plus TSCM), the correlation between S24S_{24} and the validation-selected guided test FID is Spearman ρ=0.84\rho = 0.84 and log–log Pearson r=0.96r = 0.96. Across all 16 sources, the r=0.93r = 0.93 correlation is not driven by a few poor outlying sources. The minimum leave-one-source-out Pearson rr is 0.860.86, and even when the three worst sources—periodic kernel, time-permuted source, and white noise—are removed simultaneously, the correlation remains r=0.96r = 0.96 and ρ=0.87\rho = 0.87. Regarding W1, the two negative correlations concern FID comparisons among these 11 sources at specific guidance strengths, w=1w=1 and w=10w=10. At w=1w=1, S24S_{24} remains positively correlated with the actual Euler-to-RK4 integration residual across the same 11 sources (r=0.78r=0.78, Figure 9), showing that the predictive relationship with integration error remains present. Strong CFG additionally changes the covariance of the generated distribution (Proposition 4), which can further affect the FID ranking. We have moved both sets of results from Appendices B.6 and B.10 into Section 2.5.

2. Q1 and W2: RK4-192 vs. Euler-24 FID comparison and velocity-field learnability

For Q1, on fixed final EMA weights, 65% of the unguided Euler-24 FID gap remains under RK4-192 (at w=1w = 1 across three seeds, white noise vs. TSCM is 4.14±0.254.14 \pm 0.25 vs. 1.91±0.101.91 \pm 0.10 under Euler-24, a gap of 2.232.23, and 2.36±0.202.36 \pm 0.20 vs. 0.91±0.130.91 \pm 0.13 under RK4-192, a gap of 1.451.45, i.e., 1.45/2.23≈65%1.45 / 2.23 \approx 65\%). We have moved this paired comparison from Appendix B.10 into Section 2.5.

Regarding W2, we fully agree with your observation that a large part of the gain comes from the velocity field itself being easier to learn. In fact, Figure 5 in our initial submission was designed specifically to verify this. In the original manuscript, placing Figure 5 at the very end of Section 5 with only a two-line caption made it easy to overlook. We have moved it forward to Section 2.5 as Figure 3, right after the sampling-step curves, and updated the Abstract and Introduction to explicitly state both effects. Proposition 1 shows that when a mode has a variance mismatch rk≠1r_k \neq 1, the population velocity field must make a sharp turn near t=0t = 0 or t=1t = 1 to expand or contract that mode. Matching the covariance sets rk=1r_k = 1 and eliminates these sharp turns, which brings two benefits in the trained models, as shown in Figure 3 (formerly Figure 5):

  1. In Figure 3a, even under converged RK4-192 integration, TSCM recovers the true data power spectrum in the low- and mid-frequency bands much more accurately than white noise, showing that the continuous velocity field itself is learned more accurately.
  2. In Figure 3b, the paired residual between 24-step Euler and RK4-192 is also lower across all frequencies, showing a substantial reduction in finite-step discretization error.
AuthorsOpenReview ↗

Response to Reviewer DjGm (2/2)

That the neural network is easier to fit is supported by Figure 3 (formerly Figure 5) and the RK4-192 experiments, and can also be seen from a concrete linear-network example. Whiten the target data so that its covariance is II, and denote the source covariance by CC. Fix a generative time tt and fit the velocity field with a linear network vW(x)=Wxv_W(x) = Wx. The input covariance is Σt=(1−t)2C+t2I.\Sigma_t = (1-t)^2 C + t^2 I. Running gradient descent on the population squared loss with the optimal fixed step size, the per-step contraction factor of the parameter error is q=κ(Σt)−1κ(Σt)+1,q = \frac{\kappa(\Sigma_t) - 1}{\kappa(\Sigma_t) + 1}, where κ\kappa is the condition number; a smaller qq yields faster training convergence. Under covariance matching, C=IC = I, so Σt\Sigma_t is a scalar multiple of the identity, giving κ=1\kappa = 1 and q=0q = 0. Thus, in this setting, one can strictly prove that covariance matching achieves the fastest optimization convergence. Establishing a general theorem for deep networks would further require analyzing nonlinear approximation capacity and nonconvex optimization dynamics; we therefore focus our formal proofs on the tractable finite-step objective and demonstrate the learnability benefit through our controlled experiments.

3. W1: Evidence beyond motion and new UCF-101 video experiments

For time-series generation, Table 1 in Section 5 re-runs the official TSFlow code across three random seeds per dataset, changing only the noise source. TSCM achieves the lowest mean 4-step Euler W2W_2 on all eight benchmark datasets, and the lowest 16-step W2W_2 on seven of the eight (Table 22).

For video generation, the Sky Time-lapse experiment (Table 2; Appendix B.17) is a preliminary single-seed (seed 3407) adaptation check with a short 3-epoch budget (which is why white_ft3 in the supplementary video remains underfitted) and Wan2.1's default UniPC sampler (because sky time-lapse clips are nearly static, the temporal level component carries 93% of the latent second-moment energy, so TSCM starts close to the dominant mode, which may explain why FVD does not degrade at fewer steps). To provide multi-seed video evidence at a larger scale, we added a UCF-101 experiment (101 classes, 15,927 training clips, and 3,781 test clips; Section 5, Table 3), fine-tuning Wan2.1-T2V-1.3B for 6 epochs across three random seeds. Before training, the held-out oracle risk S24S_{24} directly tells us whether to use per-channel blocks or a joint channel–time block:

  • On HumanML3D, the held-out oracle risk of per-channel TSCM is already within 0.3%0.3\% of the theoretical floor (Figure 4c), whereas the joint covariance is unusable because its dimension far exceeds the sample size (d=51,548≫n=8,620d = 51{,}548 \gg n = 8{,}620, which worsens the estimation error rate d/n\sqrt{d/n} from 0.150.15 to 2.452.45 and makes the sample matrix rank-deficient); therefore HumanML3D uses per-channel blocks.
  • On UCF-101 video latents, the held-out oracle risk drops from S24=0.164S_{24} = 0.164 for per-channel 9×99\times 9 blocks (close to PYoCo's 0.167–0.1750.167\text{--}0.175) to S24=0.131S_{24} = 0.131 for the joint 144×144144\times 144 channel–time block. This is because the joint dimension is small (d=16×9=144d = 16\times 9 = 144) and pooling across video clips and 32×3232\times 32 spatial locations provides ample samples (d/n≈0.003\sqrt{d/n}\approx 0.003; even counting one sample per clip gives 144/15,927≈0.095\sqrt{144/15{,}927}\approx 0.095), while capturing strong cross-channel correlations that account for 81%81\% of the off-diagonal energy; therefore video uses the joint block.

The results after 6 fine-tuning epochs support the oracle risk predictions under both 16-step Euler and 25-step UniPC, with joint TSCM achieving the best FVD on every seed (visual frame comparisons across the five noise sources are in Appendix B.20, Figure 14):

Noise sourcePre-training S24↓S_{24}\downarrow16-step Euler FVD$\downarrow$ (3 seeds)25-step UniPC FVD$\downarrow$ (seed 3407)
White noise0.252399.6±13.8399.6 \pm 13.8416.5
PYoCo mixed (α=1\alpha=1)0.175363.3±18.8363.3 \pm 18.8348.5
PYoCo progressive (α=2\alpha=2)0.167349.7±10.7349.7 \pm 10.7327.1
TSCM (per-channel, 9×99\times 9)0.164348.3±16.8348.3 \pm 16.8315.4
TSCM (joint channel–time, 144×144144\times 144)0.131327.3±11.2327.3 \pm 11.2294.8

Thank you for the clarification. I will maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer DjGm's Follow-up Comment

Thank you for taking the time to consider our clarifications and for your continued support! We sincerely appreciate your constructive feedback, which has helped us improve our paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response: summary of the revision

We thank the three reviewers. All three found the finite-step analysis careful and the controlled source comparison informative, and all three asked for clearer presentation or a clearer scope of claims. We have uploaded a revised PDF (main text 10 pages, changes in blue). The main changes are listed below, with the reviewers they answer.

Presentation

1. Section 1 opens with multivariate sequences X∈RC×LX\in\mathbb{R}^{C\times L} and separates physical time (evolution within a sequence) from generative time t∈[0,1]t\in[0,1] (transformation from noise to data). (zsBc)

2. Section 2 treats a general vector x∈Rdx\in\mathbb{R}^d along generative time t∈[0,1]t\in[0,1] without temporal block notation; Section 3 introduces the temporal block structure and self-contained formulas for TSCM and TSCM-Toeplitz. (zsBc, UBzP)

3. Section 3 gives the estimation recipe and the block-size criterion (d/n\sqrt{d/n} and held-out SNS_N) in the main text, moving the implementation details of Appendices A.9 and B.5 into Sections 3.1–3.2. (UBzP)

4. Corollary A.1 (second-order excess risk O(δ2)O(\delta^2) near the match) is stated in Section 2.2. (zsBc, UBzP)

5. Section 1 ("Closest prior work") compares TSCM with TSFlow, PYoCo, PriorGrad, and CW-Gen in both the fitted source and the finite-step objective. (zsBc, UBzP)


Scope of claims and diagnostics

6. Sections 1 and 3 state explicitly that Theorem 1 holds for the exact Gaussian population field and motivates TSCM rather than guaranteeing FID optimality under cross-channel correlations or learned neural fields. (zsBc, UBzP)

7. Section 2.5 moves Figure 3 (formerly Figure 5) and the paired RK4-192 vs. Euler-24 comparison from Appendix B.10 into the main text, and the Abstract and Introduction now explicitly state both effects: reducing finite-step discretization error and easing velocity-field learning (65% of the unguided Euler-24 FID gap remains at RK4-192). (DjGm)

8. Section 2.5 incorporates the near-matched rank correlation (ρ=0.84,r=0.96\rho=0.84, r=0.96) and leave-one-source-out checks (r≥0.86r\ge 0.86) from Appendix B.6, as well as the w=1w=1 (r=0.78r=0.78 with integration residual EintE_{\mathrm{int}}) and w=10w=10 scope analysis from Appendix B.10. (DjGm)


New experiment

9. UCF-101 text-to-video generation on Wan2.1-T2V-1.3B: five noise sources (white noise, PYoCo mixed α=1\alpha=1, PYoCo progressive α=2\alpha=2, per-channel TSCM 9×99\times 9, and joint channel–time TSCM 144×144144\times 144), fine-tuned for 6 epochs across three training seeds with paired draws (Section 5, Table 3; Appendix B.18, Table 21; Appendix B.20, Figure 14).

Pre-training held-out S24S_{24} selects the joint 144×144144\times 144 block (0.1310.131 vs. 0.1640.164), and mean test FVD under both 16-step Euler and 25-step UniPC follows the S24S_{24} ordering, with joint TSCM best on every seed. (DjGm, UBzP, zsBc)


Individual responses follow under each review.

Thanks for changing the organisation of the presentation. I find the paper much more readable now

AuthorsOpenReview ↗

Response to Reviewer zsBc's Follow-up Comment

Thank you for taking the time to read the revised manuscript. We are glad that the reorganization has improved its readability. We would be happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.