acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Manifold Hypothesis: Hybrid Spectral Parameterizations for Flow Matching

Abstract

Flow matching and diffusion can be trained to predict different quantities, most commonly the data , the source noise , or the velocity . Although theoretically equivalent, these can lead to substantially different performances. We identify two main drivers for these differences: the source-data signal-to-noise ratio, and the information bottleneck induced by the neural architecture. We show that, beyond intrinsic data dimension, the factor affecting the optimal parametrization the most is a certain signal-to-noise ratio in each data covariance direction. From this analysis, we introduce new spectral hybrid parameterizations that adapt across time and data covariance directions; we show that these are optimal for Gaussian data. We also show that architecture-induced compression changes which parameterization is easier to learn, with -prediction being more sensitive to discarded directions than -prediction. Experiments across architectures and source scales show that our spectral parameterizations are robust across regimes, can substantially accelerate optimization, while incurring essentially no additional training cost compared with standard parameterizations.

Primary area:
generative models
Keywords:
flow matching, diffusion, spectral, prediction, parametrization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_W1QTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how source–data SNR and architectural compression affect prediction-target preferences in flow matching. It proposes a binary data/noise switch and a continuous Gaussian-based parameterization with a learned residual, both using covariance information to adapt across directions and time. The paper provides Gaussian analyses and evaluates the methods on synthetic data and image generation.

Critical strengths
  • Informative analysis of prediction targets. The controlled experiments help distinguish the effects of ambient dimension and source–data SNR. The Gaussian analysis gives testable predictions about how parameterization interacts with compression, and the bottleneck experiments provide useful support for these predictions.

  • Promising image-generation results. The hybrids show robust performance across source scales on CelebA and reach low FID in fewer optimization steps on Bedroom. The convergence gains are observed with both U-Net and ViT models.

Critical weaknesses
  • Limited evaluation at scale. Related work such as JiT (Li and He, 2025) and k-Diff (Jin and Wang, 2026) reports quantitative results on ImageNet. The experiments here remain smaller in scale, and the higher-resolution results are mainly qualitative. It is unclear whether the reported gains persist on larger and more diverse datasets. Actual training-time measurements would also help assess the efficiency claims.
  • Unclear added benefit of directional adaptation. The image-generation comparisons leave it unclear how much is gained from adapting individual directions beyond adapting to global SNR. The uniform-switch baseline uses a fixed switching time, so it does not test whether an SNR-dependent rule shared across all directions could provide similar gains.
Questions for authors
  1. Could you provide quantitative results on a larger image-generation benchmark, such as ImageNet?

  2. Could you compare the binary hybrid with a baseline that switches all directions together at a time determined by global SNR? For the continuous hybrid, could you compare with a version that uses one time-dependent coefficient for the analytical input term, shared across directions and based on average data variance and source-noise scale?

  3. How do the hybrids compare with fixed-target baselines in wall-clock time to reach a comparable FID, including preprocessing and spectral operations?

I would consider raising my rating if the response provides stronger evidence at scale and clarifies the benefit of directional adaptation.

2 comments

We thank the reviewer for their positive assessment and constructive suggestions.

W1 & Q1. Evaluation at scale

Limited evaluation at scale. / Could you provide quantitative results on a larger image-generation benchmark, such as ImageNet?

We extended our evaluation to ImageNet-100100 at 256×256256\times256. The results are very positive: at matched training stages, the continuous hybrid substantially outperforms x1x_1-prediction and converges markedly faster. See Point 1 of the general response for quantitative results and implementation details.

W2 & Q2. Benefit of direction-dependent adaptation

Unclear added benefit of directional adaptation. / Could you compare [...] with a baseline that switches all directions together at a time determined by global SNR?

We thank the reviewer for suggesting this ablation. We added global binary and continuous hybrids that retain SNR-dependent adaptation but apply the same rule to every covariance direction. See Point 3 of the general response for the definitions and quantitative results.

The result is particularly clear: both global hybrids perform worse than x1x_1-prediction, whereas both spectral hybrids substantially outperform it. This directly shows that adapting only to the global SNR is insufficient, and that direction-dependent adaptation is essential to the observed gains.

Q3. Wall-clock training efficiency

How do the hybrids compare with fixed-target baselines in wall-clock time to reach a comparable FID, including preprocessing and spectral operations?

The preprocessing cost is incurred only once: the spectral statistics are computed, cached, and reused across runs. For CelebA-6464, the full PCA requires about 0.60.6 GB of memory and takes 4 minutes to compute on a V100 GPU.

During training, the additional cost mainly consists of matrix-vector products, which are negligible compared with the network forward and backward passes. We now report FID as a function of wall-clock time on LSUN-Bedroom-6464 (Figure 6 b). The resulting curves preserve the gains observed versus optimization steps, showing that the spectral hybrids reach a given FID substantially earlier than the best fixed-target baseline.

At higher resolution, we avoid the O(D2)O(D^2) memory cost of full-image PCA by using scalable spectral representations: a fixed 2D DCT basis for AFHQ-256256 and patch-level PCA for ImageNet-100100 (see common answer, Point 1).

Thank you for the response and additional experiments. These additions largely address my main concerns, and I will raise my rating from 2 to 3. Please include the additional results and implementation details in the revised paper.

Reviewer_Ne3nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how prediction targets affect flow-matching training. It shows that noise scale and information lost by the network can change which target works best. It proposes two hybrid targets that adapt across time and data directions. Gaussian analysis and image experiments support improved robustness and faster training convergence.

Critical strengths
  1. The paper provides a clear explanation of prediction target choice. By changing the noise scale while keeping data geometry and loss weighting fixed, the paper shows that prediction target rankings can reverse. These controls support the conclusion that intrinsic dimension alone is insufficient to explain performance. This provides useful guidance for designing flow-matching models.

  2. The paper provides a useful connection between analysis and algorithm design. The hybrids adapt across time and covariance directions. Under the Gaussian model, the analysis explains how they preserve target variance in discarded directions. Image experiments also show benefits: on the compressed Bedroom-64 ViT, the continuous hybrid reaches FID around 60 with about one-third as many updates as data prediction. This supports its potential to improve optimization efficiency.

Critical weaknesses
  1. The continuous construction’s advance over existing preconditioning is unclear. Its analytical component is the standard Wiener estimator, followed by a learned residual. Its Gaussian optimality therefore does not provide a new estimation result. The binary switch and bottleneck analysis add useful ideas, but the experiments do not clearly isolate the continuous model’s benefit over simpler denoiser preconditioning. This limits assessment of its specific technical contribution.

  2. The continuous hybrid’s reliability remains insufficiently characterized. On the strongly non-Gaussian spiral, it often generates a Gaussian cloud instead of recovering the target structure, while the binary hybrid remains robust. This exposes sensitivity to the Gaussian approximation. At higher resolution, the AFHQ results provide image examples without quantitative FID comparisons. These gaps leave uncertainty about when the continuous model helps and how consistently its apparent higher-resolution benefits hold.

Questions for authors
  1. What are the main advances beyond Wiener estimation and existing denoiser preconditioning? Could a comparison using one shared variance across all directions, with the same architecture and training objective, isolate the benefit of directional adaptation?

  2. Given the spiral failure, what properties of the data or model indicate when the continuous hybrid should help? Can quantitative AFHQ results, such as FID under matched training and sampling settings, establish whether the visual improvements hold across generated samples?

2 comments

We thank the reviewer for their constructive assessment.

W1 & Q1. Contribution beyond existing preconditioning

"The continuous construction's advance over existing preconditioning is unclear. Its analytical component is the standard Wiener estimator, followed by a learned residual."

We agree that the Wiener estimator itself is standard, and do not claim otherwise. Our contribution is its direction-dependent integration into flow-matching parameterization, together with the analysis of its interaction with architectural bottlenecks.

In particular, EDM (Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022) already uses a Gaussian-motivated preconditioning with time-dependent scalar coefficients, but they use a single coefficient, common to every direction. Our approach extends this principle by exploiting the full covariance spectrum: the analytical coefficient depends not only on time and source noise scale, but also differ across directions through their dependency on the directional variance λj\lambda_j.

This distinction is central: different directions can have very different SNRs, and therefore benefit from different parameterizations at the same time.

To isolate this contribution, we have added the ablation suggested by reviewer W1QT on LSUN-Bedroom-64. We compare our spectral hybrids with two global counterparts: a binary hybrid using a single SNR-dependent switching time, and a continuous hybrid using a single time-dependent Wiener coefficient shared across all directions.

We find that both global hybrids perform worse than x1x_1-prediction, whereas our spectral hybrids outperform it. This demonstrates that the improvement is not simply due to Gaussian-based preconditioning, but specifically to adapting the parameterization across covariance directions. More details in the general answer (Point 3).

W2 & Q2. Reliability of the continuous hybrid

On the strongly non-Gaussian spiral, it often generates a Gaussian cloud instead of recovering the target structure, while the binary hybrid remains robust.

We do not yet have a complete explanation for this failure. Our hypothesis is that the continuous hybrid becomes less effective when its covariance-based analytical term captures too little of the structure needed for prediction. The new controlled experiment in Point 2 of the general response is consistent with this hypothesis: as the nonlinear gap increases, the continuous hybrid deteriorates, while the binary hybrid remains stable. On image data where the covariance information is more useful, the continuous hybrid performs very well.

W2 & Q2. Quantitative evaluation at higher resolution

At higher resolution, the AFHQ results provide image examples without quantitative FID comparisons.

We added quantitative AFHQ-256256 results and extended the evaluation to ImageNet-100100 at 256×256256\times256. The ImageNet results are particularly positive: at matched training stages, the continuous hybrid substantially outperforms x1x_1-prediction and converges markedly faster. See Point 1 of the general response for the quantitative results.

Thank you for your clarification and additional experimental results. For W1 & Q1, direction-dependent (anisotropic) spectral preconditioning is mathematically sound. For W2 & Q2, I thank your honest on the lack a complete theoretical explanation for the failure on the spiral dataset, but your hypothesis give a useful characterization of the failure mode. For W3 & Q2, the new experimental results resolve the lack of empirical rigor at higher resolutions. My concerns are mainly addressed, and I will raise my score.

Reviewer_aApZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that the relative performance of data, noise, and velocity prediction in flow matching depends on the source–data signal-to-noise ratio (SNR) and architectural information bottlenecks, alongside the intrinsic dimensionality of the data. It proposes spectral hybrid parameterizations that adapt across time and covariance directions. By disentangling dimensionality from SNR, the authors demonstrate that changing the source noise scale alone can reverse the performance ordering of prediction targets, and derive a direction-dependent switching rule between data and noise prediction through Gaussian analysis. The resulting binary hybrid and a continuous hybrid that combines a covariance-based Wiener estimate with a learned nonlinear correction are designed to reduce the residual the network must learn. The bottleneck analysis shows that prediction targets influence which directions the network preserves: in discarded directions, data prediction suppresses variability, velocity prediction retains the source noise, and both hybrids recover the target variance through an analytical path under the Gaussian model.

Critical strengths
  • The paper provides a principled account of performance differences across prediction parameterizations. Building on JiT’s observations about the benefits of data prediction, it uses controlled experiments to disentangle intrinsic dimensionality from source–data SNR. It further develops a time and covariance analysis that explains the relative advantages of data and noise prediction and derives their switching times, providing concrete guidance for parameterization design in diffusion and flow-matching models.

  • The paper effectively connects architectural information bottlenecks with prediction parameterization through theory and experiments. It analyzes both which directions each parameterization preserves and how discarded directions evolve during generation. The Gaussian analysis identifies a mechanism favoring data prediction under strong compression: data prediction suppresses variability in discarded directions, whereas velocity prediction retains the source noise variance. Experiments on image patches and learned ViT representations support this account, including a control using a bottleneck selected to favor velocity prediction.

  • The paper translates its findings into practical parameterizations with demonstrated benefits for image generation. The binary hybrid uses time switching, while the continuous hybrid combines a covariance-based analytical estimate with a learned correction. Experiments across image datasets and U-Net and ViT architectures demonstrate robustness to SNR variation and faster convergence. Comparisons using a common denoising objective and time weighting strengthen the evidence that these gains arise from the parameterization design.

Critical weaknesses
  • Quantitative validation at higher image resolutions is limited. The main SNR sweeps and convergence comparisons are conducted on 64×64 datasets. Although the paper includes AFHQ-256 experiments, these results are presented primarily through qualitative samples. Since the interaction between prediction parameterization and architectural compression is central to the paper, quantitative evaluations at 256×256 or higher—covering generation quality, convergence speed, and sensitivity to SNR and compression—would strengthen the evidence that the findings extend to higher-resolution image generation.
  • The explanation for the continuous hybrid’s failure is not sufficiently validated. In Figure 17, the continuous hybrid fails to recover the spiral structure in several settings, which the authors attribute to its reliance on an inaccurate Gaussian-based estimate. Experiments that vary non-Gaussianity while preserving covariance and SNR, or analyses of residual learning, would help clarify which data characteristics make the continuous hybrid effective.
  • The effects of the implementation’s additional stabilization choices are insufficiently analyzed. Appendix H states that the binary hybrid’s switching time is lower-bounded by 0.1. Thus, a direction whose theoretical switching time is 0.03 continues using data prediction until 0.1, changing the prediction target over that interval. The continuous hybrid similarly caps its analytical coefficient at 10 for CelebA-64 and 100 for AFHQ-Cat, changing the analytical estimate and the correction the network must learn whenever the cap is active. These are consequential design choices beyond the derived formulas. Ablations and sensitivity analyses would clarify how much they contribute to the reported performance and stability, and whether the results remain robust across different threshold values.
  • Minor presentation issue: Adding an explicit “Introduction” heading and renaming “Reminders” to “Preliminaries” or “Background” would make the paper’s organization easier to follow.
Questions for authors
  • Could you provide quantitative results at 256×256 or higher, including FID and convergence curves, and evaluate whether the benefits persist across different SNR levels and compression ratios?

  • What evidence supports attributing the failures in Figure 17 to the Gaussian-based estimate? Could you examine residual prediction errors or vary non-Gaussianity while keeping covariance and SNR fixed to identify when the continuous hybrid succeeds or fails?

  • How were the minimum switching time of 0.1 and the coefficient caps of 10 and 100 selected? How sensitive are performance and training stability to these values, and what happens when these restrictions are removed? How frequently do the restrictions become active in the reported experiments?

2 comments

We thank the reviewer for their positive assessment and constructive suggestions.

W1 & Q1. Higher-resolution evaluation

Quantitative validation at higher image resolutions is limited.

We now provide quantitative AFHQ-256256 (added in Figure 7 of the paper) and ImageNet-100100 results at 256×256256\times256. The continuous hybrid substantially improves over x1x_1-prediction at matched training stages. See the general response (Point 1) for the quantitative results and implementation details.

W2 & Q2. Continuous hybrid on the spiral

Could you [...] vary non-Gaussianity while keeping covariance and SNR fixed [...]?

We added exactly this controlled experiment, see general answer (Point 2). The continuous hybrid progressively deteriorates as the nonlinear gap increases, whereas the binary hybrid remains comparatively stable. See the general response for the full experiment and discussion.

W3 & Q3. Sensitivity to stabilization parameters

The stabilization choices are insufficiently analyzed. / How sensitive are the results to these choices?

We initially introduced the switching-time floor and coefficient cap for numerical stability: switching to x0x_0-prediction very close to t=0t=0 can be ill-conditioned, while very large analytical coefficients may amplify numerical errors.

We added a sensitivity analysis on CelebA-6464 and LSUN-Bedroom-6464, varying tmin⁡∈{0,0.05,0.1}t_{\min}\in \{0,0.05,0.1\} and amax⁡∈{5,10,∞}a_{\max}\in \{5,10,\infty\}. In the standard SNR regime, these parameters affect less than 0.2%0.2\% of covariance directions, and FID shows no systematic dependence on either choice.

In particular, removing the floor or cap does not cause instability or substantial degradation. Full results are reported in the revised paper, Appendix G.

Minor presentation issue

Adding an explicit 'Introduction' heading and renaming 'Reminders' to 'Preliminaries' [...]

We have made both changes in the revised manuscript.

Thank you for the detailed response. The high-resolution experiments, analysis of the continuous hybrid on the spiral dataset, and sensitivity analyses address most of my concerns. While some questions about practical applicability remain, I consider the paper’s extension of JiT’s intrinsic-dimension perspective to an SNR-based account a sufficient contribution to support acceptance. I will therefore raise my score.

Comments on the paper

4 comments
Area_Chair_WXXWOpenReview ↗

Engage in discussion

The authors and reviewers are requested to engage in the discussion phase.

Authors: please respond to the review comments.

Reviewers: please review the authors’ rebuttal and consider revising your assessment/score.

Dear reviewers, We are in the process on finalizing the additional experiments for large scale comparison on Imagenet, that required a bit of time. We will post our full answer to all comments soon. Thank you for your feedback

AuthorsOpenReview ↗

General answer to all reviewers

The reviewers raised three main concerns:

  1. the lack of quantitative large-scale evaluation beyond the qualitative AFHQ-256256 results;
  2. the failure of the continuous hybrid on the spiral distribution;
  3. the benefit of direction-dependent adaptation over simpler global SNR-dependent parameterizations.

We addressed each point with additional experiments.

1. Large-scale evaluation on ImageNet-100100

We extended our evaluation to ImageNet-100100 at 256×256256\times256 resolution, comparing the continuous hybrid with JiT using x1x_1-prediction. We use JiT-B/16 with bottleneck dimension 256256, corresponding to a patch-level compression factor of 768/256=3768/256=3.

A class-conditional full-image PCA is impractical at this resolution, since each image has dimension 3⋅2562=196,6083\cdot256^2=196{,}608. We therefore use a scalable patch-based approximation. For each class, we estimate a 768×768768\times768 covariance from pooled 16×1616\times16 RGB patches and apply the resulting PCA basis independently at the 256256 patch positions. This corresponds to the block-diagonal covariance approximation

Σ~c=I256⊗(Ucdiag(λc)Uc⊤)\tilde\Sigma_c = I_{256}\otimes \left(U_c\text{diag}(\lambda_c)U_c^\top\right)

and requires storing only patch-level spectral statistics.

ParameterizationFID at 49.6k49.6\mathrm{k}FID at 99.2k99.2\mathrm{k}
x1x_1-prediction76.7141.21
Continuous spectral38.7723.08

Thus, at matched training cost, the continuous hybrid approximately halves the FID of x1x_1-prediction. The improvement persists throughout training, and samples at matched epochs exhibit finer details. We added the quantitative FID curve and generated samples to the revised paper (Section 6).

2. Continuous hybrid on the spiral

At this stage we do not have a fully satisfying reason for why the continuous hybrid fails on the spiral dataset of Figure 17, apart that its second-order statistics carry almost no information about its structure.

Yet we also would like to emphasize that the spiral is a deliberately extreme toy distribution, a 2D curve embedded in a high-dimensional space. We use it for illustration, following the setting of Li & He, but we are not convinced that a fine-grained characterization of failure modes on this example would tell us much about the regimes our method targets in real situations.

To investigate this limitation, we added a controlled experiment where the target interpolates between a Gaussian distribution GG (α=0\alpha=0) and the original spiral XX (α=1\alpha=1), while keeping the mean, covariance, and source--data SNR fixed: αX+1−αG\sqrt{\alpha} X + \sqrt{1 - \alpha} G, XX and GG being centered and independent.

We define

Rlin(α)(t)=E[∥X1(α)−Lt(α)(Xt)∥2]R_{\mathrm{lin}}^{(\alpha)}(t)=\mathbb E[\|X_1^{(\alpha)}-L_t^{(\alpha)}(X_t)\|^2]

and

RMMSE(α)(t)=E[∥X1(α)−E[X1(α)∣Xt]∥2],R_{\mathrm{MMSE}}^{(\alpha)}(t) =\mathbb E[\|X_1^{(\alpha)}-\mathbb E[X_1^{(\alpha)}\mid X_t]\|^2],

where Lt(α)L_t^{(\alpha)} is the optimal linear estimator and RMMSE(α)(t)R_{\mathrm{MMSE}}^{(\alpha)}(t) is estimated by Monte Carlo. We then define

ΔNL(α)(t)=Rlin(α)(t)−RMMSE(α)(t)Rlin(α)(t),\Delta_{\mathrm{NL}}^{(\alpha)}(t) =\frac{R_{\mathrm{lin}}^{(\alpha)}(t)-R_{\mathrm{MMSE}}^{(\alpha)}(t)}{R_{\mathrm{lin}}^{(\alpha)}(t)},

which measures the relative decrease in prediction error obtained by replacing the optimal linear estimator with the optimal nonlinear conditional mean.

α\alphaW2W_2 continuousW2W_2 binaryΔNL\Delta_{\mathrm{NL}}
00.0500.0510%
0.750.1040.0420.39%
0.950.1990.0365.90%
10.2960.05346.69%

At the Gaussian endpoint, the binary and continuous hybrids perform similarly. As higher-order structure increases, the continuous hybrid progressively deteriorates together with the nonlinear gap, whereas the binary hybrid remains comparatively stable. This suggests that the continuous hybrid becomes less effective when the covariance-based analytical term does not capture enough of the structure needed for prediction.

Natural images are a much different case, for which the second-order structure (e.g., the decay of their PCA/DCT spectrum) is much more informative, so a Gaussian approximation is likely to be far more accurate. On such data (CelebA, LSUN, AFHQ, and ImageNet-100100) we have shown that the continuous hybrid consistently provides improved convergence and generation quality.

3. Directional adaptation versus global SNR adaptation

To isolate the benefit of adapting across covariance directions, we added two global counterparts of our hybrids. They use the same SNR-dependent principles but apply a single rule to all directions:

t_{\mathrm{switch}}^{\mathrm{glob}} &= \frac{1}{1+\sqrt{\rho}}, & a^{\mathrm{glob}}(t) &= \frac{t}{t^2+(1-t)^2/\rho}. \end{aligned}$$ Thus, these baselines retain adaptation to the global source--data SNR while removing dependence on individual covariance directions. On LSUN-Bedroom-$64$, the distinction is clear: | Parameterization | FID at epoch 165 | Time to FID $\leq 52$ (h) | |---|---:|---:| | $x_1$-prediction | 51.33 | 21.11 | | Binary global | 260.43 | -- | | Continuous global | 263.67 | -- | | Binary spectral | 19.46 | 12.88 | | Continuous spectral | **17.40** | **9.11** | Both global parameterizations perform substantially worse than $x_1$-prediction, whereas both spectral hybrids substantially outperform it. This directly shows that the gains cannot be explained by global SNR adaptation alone: adapting the parameterization to the heterogeneous variances of the covariance directions is essential. See Figure 6, b) in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.