acceptodds
Under review as a conference paper at ICLR 2027

Amortized Moment Matching for Visual Generation

Abstract

We propose amortized moment matching, utilizing neural networks to learn data moments as distributional training signals. By casting diffusion denoisers through polynomial projections, we establish a general framework for moment amortization, revealing that an -th degree projection explicitly identifies data moments up to order . Derived from the tractable affine case, we instantiate the Amortized Fr\'echet Distance (AMFD) loss. Unlike FD-loss which relies on explicit marginal moment calculations, AMFD is able to dynamically learn conditional moments via an alternating, matrix-free optimization pipeline that effortlessly scales to high-dimensional data. When operating on global representation features, AMFD serves as a powerful post-training objective; empirically, its neural formulation yields more robust training dynamics than exact statistical matching, substantially surpassing the FD baseline on the FDr metric and achieving superior one-step generation on ImageNet. Furthermore, it unlocks direct exploration within native generative spaces, suggesting that the first two moments can identify target distributions only in spaces with strong semantics. Finally, when scaled to text-to-image generation, the condition-aware nature of AMFD unlocks massive gains in instruction-following capabilities, enabling our one-step models to outperform their multi-step FLUX.2 [klein] 4B teachers on the GenEval benchmark while achieving on-par performance on PickScore.

Primary area:
generative models
Keywords:
generative models, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mcMROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a framework for amortized moment matching using neural diffusion denoisers. The authors establish a connection between polynomial denoisers and distribution moments, and develop the AMFD loss to match conditional moments without explicitly computing covariance matrices. The proposed method is evaluated on ImageNet and text-to-image generation, showing improvements over FD-loss and competitive performance against multi-step teacher models.

Critical strengths
  • The connection between diffusion denoisers and polynomial projections is interesting and provides a theoretical perspective on moment matching. The proposed method also avoids explicitly estimating covariance matrices through a matrix-free optimization approach.

  • The authors demonstrate an equivalence between the cross-entropy loss used in their framework and the noise prediction loss employed in standard diffusion models. This provides a useful connection between the proposed formulation and existing diffusion training objectives.

  • The experimental results are promising, particularly the improvements in one-step generation and the extension to text-to-image generation.

Critical weaknesses

One concern is the potential overfitting to the FID metric. Since the neural amortizer is trained in the Inception-v3 feature space, which is also used for FID evaluation, the model may learn to optimize specifically for this feature space. This could lead to lower FID scores without necessarily improving the overall generation quality. Additional evaluation using independent feature extractors or perceptual metrics would help address this concern.

Questions for authors
  • In Figure 2(b), is the order of the AMFD training pipeline reversed? My understanding is that the neural amortizer should first be trained using the Poly Diff loss, and then the trained, frozen amortizer is used to guide generator post-training. Could the authors clarify this?

  • In Equation (7), why is (v_n(x_t,t,c)) directly equated with (x_1-x_0)? Since (v_n) is a learned model optimized toward this target, the equality does not necessarily hold exactly unless the model achieves zero prediction error. Is this equality based on an assumption of an optimal denoiser?

  • Can the proposed method be understood as a more efficient way of performing diffusion model post-training with an FD-based loss, similar to Yang et al. (2026), but replacing explicit covariance computation with an affine approximation and neural amortization? It would be helpful to clarify the main conceptual and practical differences.

  • The higher loss observed for AMFD-C may suggest that the MLP-based amortizer has limited capacity to capture the underlying distribution. Have the authors considered using a more expressive architecture? It would be interesting to see whether this limitation comes from the amortization framework itself or simply from the model capacity.

5 comments

We thank the reviewer for the positive assessment and the constructive questions. We address each of your concerns in detail below.

W1. Evaluation beyond the Inception feature space.

We agree that FID cannot serve as an independent held-out metric when Inception-v3 is included in the SIM training ensemble (SigLIP2, Inception-v3, and MAE). To address this concern and rigorously evaluate out-of-domain generalizability, we additionally introduce FDr$^3$, defined as the average normalized Fréchet distance across three strictly held-out representations: ConvNeXt, DINOv2, and CLIP. Based on our per-encoder measurements, AMFD-U substantially improves FDr$^3$ over the FD-loss baseline, decreasing the score from 4.44 to 2.77 on JiT-H, and from 2.69 to 2.29 on pMF-H (detailed in Appendix Table 9 of the revised manuscript). This strong held-out performance confirms that the benefits of AMFD are not merely a result of overfitting to Inception features, but generalizing to held-out representation metrics.

Q1. Training order in Figure 2(b).

The intended procedure is an online alternating optimization loop, as specified in Algorithm 1 in the appendix. Within each iteration, we first update the amortizers using the current real and generated features, with generated features detached from the amortizer regression objective. We then freeze the amortizer parameters and update the generator using the resulting moment-matching signals. The amortizers are not trained once and permanently frozen; they continue adapting as the generated distribution evolves.

Q2. Regarding the optimality assumption in Eq. (7).

We thank the reviewer for this question. Yes, Eq. (7) is derived under the assumption that the polynomial denoiser reaches its population optimum, which is standard in theoretical analyses of diffusion models. In practice, the neural amortizer approximates this population optimum. Importantly, our amortizer is lightweight and continuously updated using samples from the current generated distribution through alternating optimization, allowing it to adapt as the generator evolves. Our empirical results demonstrate stable training and effective moment estimation, supporting the practical usefulness of this approximation, although exact population optimality is not assumed during training.

Q3 Relationship to FD-loss.

We thank the reviewer for this insightful interpretation. We agree that AMFD can be viewed as an approach to FD-based moment matching. The key distinction lies in amortization: how the distributional statistics are represented and optimized. While FD-loss explicitly computes the mean and covariance from data samples, AMFD implicitly encodes these statistics in the parameters of neural amortizers through affine denoising regression.

This amortization brings two practical advantages. First, the neural formulation provides more stable training dynamics, particularly when post-training generators from early pretraining checkpoints, where FD-loss can diverge while AMFD remains stable (Figures 6 and 7). Second, neural amortizers naturally support condition-dependent mean and covariance estimation. This capability is particularly beneficial for text-to-image generation, where AMFD-C consistently improves instruction-following performance over AMFD-U in our experiments.

We will clarify this connection to FD-loss and emphasize these two advantages in the revised manuscript.

Q4. Amortizer capacity and conditional matching.

We thank the reviewer for this insightful question. We would like to clarify our architectural choice and the potential reasons for the higher estimation error of AMFD-C.

First, MLPs are a natural choice for our representation-space setting. Our amortizers operate on globally pooled, one-dimensional feature vectors rather than spatial feature maps or token sequences. Since these representations lack explicit spatial structure, MLPs provide a straightforward and computationally efficient architecture for learning the corresponding moment operators. Nevertheless, AMFD is not restricted to MLPs; we also demonstrate its compatibility with Transformer-based amortizers in our native generative space experiments.

Second, we have investigated the effect of model capacity. As shown in Figure 5, increasing the MLP depth and width improves moment estimation accuracy, with diminishing gains near our default configuration. This suggests that simply scaling up the capacity may offer limited additional benefits.

Finally, we believe the higher estimation error of AMFD-C is largely related to the greater difficulty of conditional moment estimation. On ImageNet, AMFD-C must learn class-dependent statistics across 1,000 categories, with only approximately 1,300 samples per class. In contrast, AMFD-U estimates a single global marginal distribution using the entire dataset. Although the conditional amortizer shares parameters across classes, the substantially smaller per-condition sample size makes estimating fine-grained conditional statistics more challenging. Therefore, the observed gap may reflect both task complexity and statistical estimation difficulty, rather than solely insufficient model expressivity.

Thank you for your responses. I believe the current rating is fair and would like to keep it unchanged.

Reviewer_mF7yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for one-step image generation through moment matching. Compared to FD-loss, the method uses neural networks (amortizers) to estimate the statistics of real and generated feature distributions. The authors show that an n-th degree polynomial projection of a diffusion denoiser can identify data moments up to order n+1. Based on this, they focus on the affine case (n=1), where they train amortizers using flow matching and use the estimated statistics to build a surrogate loss for fine-tuning the generator.

The authors test the method on class-conditional image generation on ImageNet and text-to-image generation, using GenEval and PickScore for evaluation. They fine-tune several pretrained models, including JiT, pMF, PixelGen, and FLUX.2. The method generally performs better than FD-loss on FDr⁶ and improves prompt following with conditional moment matching.

Critical strengths
  1. The idea of using a neural network (here, an amortizer) to estimate feature distribution statistics under different conditions is interesting and novel. By leveraging the generalization ability of neural networks, the method allows conditional moment matching for more complex tasks like text-to-image generation, which is a nice extension over FD-loss.
  2. The experiments are quite thorough, covering both pixel-space and latent-space models, as well as class-conditional and text-to-image generation. The authors also provide detailed ablations on the accuracy of the estimated statistics and different design choices, which help explain when and why the method works.
Critical weaknesses
  1. Compared to FD-loss, the method introduces additional networks and alternating training, which may require careful tuning. Also, as the generator is updated, the fake image distribution keeps changing, so the amortizer needs to continuously adapt to it. It is unclear how well it can track these changes and whether this may cause training instability. In Table 2, AMFD-U generally performs better than AMFD-C, even though conditional moment matching is one of the main motivations of the method. This makes it unclear when conditional amortization is actually useful, especially given the additional training complexity.
  2. For the FLUX experiments, the authors use images generated by a pretrained teacher as reference data, including images generated from GenEval prompts and selected using a detector. Since these prompts are also used for evaluation, it is unclear how well the method generalizes to unseen prompts, or how much of the improvement comes from this data preparation. It would be useful to evaluate on held-out prompts. Also, the paper lacks side-by-side visual comparisons with the original models and other one-step baselines for text-to-image generation. While the generated images look good, it is hard to tell how much improvement comes from the proposed method rather than the strong pretrained generator.
Questions for authors

Please refer to the questions and concerns raised in the Critical Weaknesses section.

3 comments

We sincerely thank the reviewer for recognizing the novelty of neural moment amortization and the breadth of our experimental evaluation. We appreciate the constructive concerns regarding training stability, the effectiveness of conditional matching, and generalization beyond the curated FLUX training data.

W1. Alternating optimization, tracking stability, and the role of conditioning

i) Additional training complexity and hyperparameter tuning. Although AMFD introduces additional amortizers and alternating optimization, it requires minimal additional tuning. We retain the established FD-loss generator training settings and use fixed defaults for all amortizer configurations, including architecture, interpolation time, and noise sampling. The only additional optimization hyperparameter to determine is the amortizer learning rate. Empirically, we find that setting it sufficiently higher than the generator learning rate yields stable training and strong performance. Since the generator learning rates in our experiments are at most 10−510^{-5}, we consistently use 10−410^{-4} for all amortizers, without any per-model tuning. Detailed configurations are provided in the appendix.

ii) Stability and adaptation to the evolving generated distribution.

We thank the reviewer for raising this concern. In AMFD, the amortizers are continuously updated using freshly generated samples before each generator update, allowing them to adapt to the evolving generated distribution.

To directly examine tracking accuracy, we conducted additional experiments comparing AMFD-U with FD-loss using three EMA decay rates (0.9, 0.99, and 0.999). We monitor mean estimation MSE, symmetric covariance estimation MSE, and generator FID throughout 5,000 training steps.

At the final checkpoint, we obtain:

MethodMean MSE ↓Covariance MSE ↓FID ↓
FD-loss (EMA=0.9)0.0003320.0000822.44
FD-loss (EMA=0.99)0.0092190.0003957.89
FD-loss (EMA=0.999)0.0136590.00101511.03
AMFD-U (LR=1e-4)0.0005410.0001322.70

The evolution of the error metric across training steps is illustrated in Appendix C.4 in the revised manuscript. Despite starting from a randomly initialized amortizer, AMFD rapidly adapts to the evolving statistics and achieves lower tracking errors than the slower EMA baselines (0.99 and 0.999) by the end of training. Its final FID of 2.70 is also close to that of the faster EMA=0.9 baseline (2.44). These results demonstrate effective online moment tracking without requiring explicit EMA-based statistic updates.

Furthermore, our pretraining-checkpoint experiments (Figures 6 and 7) demonstrate that both AMFD variants can successfully post-train from substantially earlier JiT checkpoints, even in settings where FD-loss diverges.

Together, these experiments provide complementary evidence that AMFD can adapt to evolving generated statistics while maintaining robust generator optimization.

iii) Why AMFD-U outperforms AMFD-C on ImageNet, and when conditioning is beneficial.

We agree that AMFD-U generally outperforms AMFD-C on ImageNet. This is primarily due to the greater difficulty of estimating class-conditional moments: AMFD-U learns global statistics from the entire dataset, whereas AMFD-C must estimate separate statistics for 1,000 classes, each with approximately 1,300 training images. This estimation gap is also reflected in our amortizer ablation results.

However, conditional matching provides clear benefits in more challenging settings. First, our early-pretraining-checkpoint experiments (Figures 6 and 7) show that AMFD-C outperforms AMFD-U under the weakest generator initializations, indicating improved optimization robustness from class-aware supervision. Second, in text-to-image generation, where conditions encode much richer semantic information, AMFD-C substantially outperforms AMFD-U: 0.760 vs. 0.713 on PixelGen and 0.833 vs. 0.749 on FLUX Base in GenEval. Importantly, this advantage persists in our new LAION-prompt experiments (0.789 vs. 0.758), without the original GenEval-specific data curation.

These results indicate that the benefit of conditional amortization depends on both the difficulty of conditional moment estimation and the value of condition-specific supervision. While unconditional matching is often sufficient for well-pretrained ImageNet models, conditional matching offers additional robustness under weak initialization and stronger prompt alignment in complex text-to-image tasks.

W2. GenEval-specific curation, generalization, and qualitative comparisons

i) GenEval-specific curation and unseen-prompt generalization

We agree that our original FLUX results do not constitute held-out GenEval evaluation, as the original training corpus included a composition block derived from all 553 GenEval prompts, with 551 prompts contributing detector-verified teacher-generated reference images. To rigorously test whether the advantage of conditional amortization persists without this GenEval-specific composition corpus, we conducted new FLUX experiments using LAION-sourced prompts. For these experiments, we select the first 301,853 unique LAION prompts (matching the number of reference images in our original FLUX training corpus), and use the four-step FLUX.2 [klein] 4B teacher to generate the reference images.

The new results, now reported in the revised Table 3, are as follows:

MethodGenEval ↑PickScore ↑
FLUX.2 [klein] 4B Base0.77821.50
+FD-SIM0.73321.53
+AMFD-U-SIM0.75821.51
+AMFD-C-SIM0.78921.59
FLUX.2 [klein] 4B0.79421.85
+AMFD-C-SIM0.81321.78

Notably, AMFD-C-SIM still outperforms AMFD-U-SIM by 0.031 and FD-SIM by 0.056 on GenEval with FLUX.2 [klein] 4B Base, despite using an independently sourced LAION prompt corpus without the original GenEval-derived composition block.

These results provide strong evidence that the benefits of conditional amortization persist beyond the original GenEval-specific training protocol. Furthermore, using the same LAION-based reference data, AMFD-C-SIM post-training of the four-step FLUX.2 [klein] 4B achieves a GenEval score of 0.813, exceeding the four-step teacher's reported score of 0.794. This shows that teacher-surpassing GenEval performance can also be observed without the original benchmark-specific data curation.

ii) Visual comparisons among different models.

Following the reviewer's suggestion, we compare FLUX.2 [klein] 4B (the default 4-NFE), FLUX.2 [klein] 4B (1-NFE), FD-Loss, iRDM, AMFD-C in Appendix Figure 12 in the revised manuscript. As illustrated by the figure, despite compressing the four-step teacher into a single step, AMFD-C successfully preserves a generation quality highly comparable to the original multi-step model. Moreover, when compared to the one-step baselines (FD-Loss and iRDM), our approach yields noticeably clearer images with sharper edges and richer details.

We sincerely thank you again for your detailed and constructive review. As the discussion period approaches its end, we would greatly appreciate any further feedback you may have on our revisions. We would be happy to clarify any remaining questions.

Reviewer_ErM9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a generative post-training method that learns condition-dependent statistics through auxiliary neural networks, rather than directly computing the means and covariances of real and generated feature distributions. The method builds on the relationship between the population optimum of affine flow-matching regression and the mean/covariance, and applies covariance-related operators in a matrix-free manner using Jacobian-vector products (JVPs). The resulting Amortized Fréchet Distance (AMFD) framework alternates between training the generator and the amortizer, and is evaluated on ImageNet with JiT/pMF models and on pixel- and latent-space text-to-image models.

On ImageNet, AMFD-U achieves lower FDr⁶ than FD-loss across several models, while the FLUX experiments report high GenEval scores. However, the results for FID and individual encoders are mixed, and the FLUX training corpus includes detector-filtered images generated from GenEval prompts. While the practical effectiveness of the affine approach is demonstrated, the broader claims regarding higher-order moment identification and unseen-prompt generalization are not sufficiently validated.

Critical strengths
  • The paper develops affine moment amortization into a practical conditional training method. The affine regression result in Corollary 3.2 is valid: at the population optimum, the affine operator uniquely identifies the covariance for 0<t<10 < t < 1. Connecting this result to conditional amortizers and JVPs is a meaningful contribution, as it provides training signals without explicitly constructing large covariance matrices. Furthermore, the implementation details, including analytical JVPs and normalization, are also useful.

  • The paper demonstrates practical post-training improvements across multiple generators. In Table 2, AMFD-U improves FDr⁶ from 2.65 to 1.79 on JiT-H and from 1.89 to 1.75 on pMF-H, compared with FD-loss. In Table 8, the CLIP FDr scores also improve on all three backbones for which direct comparisons are available, even though CLIP is not included among the training encoders. Therefore, the improvements cannot simply be dismissed as being restricted to the training features. However, FID worsens from 0.75 to 0.83 on JiT-H and from 0.77 to 0.85 on pMF-H. On pMF-H, the DINOv2 score remains unchanged, while the ConvNeXt score worsens from 0.57 to 0.60. Moreover, since Inception is included among the SIM training encoders, FID is not an independent held-out evaluation metric.

Critical weaknesses
  • The central theoretical claims regarding moment identification and the interpretation of the covariance force extend beyond what is actually established. The normal equations in Theorem 3.1/B.1 provide EP[vn∗(Z)⊗Z⊗k]=EP[V⊗Z⊗k]\mathbb{E}_P[v_n^*(Z)\otimes Z^{\otimes k}]=\mathbb{E}_P[V\otimes Z^{\otimes k}]. While the triangular structure of the moment recurrence on the right-hand side is valid, the expectation on the left-hand side also depends on the unknown distribution PP. Therefore, knowing this expectation is not equivalent to knowing only the polynomial coefficients.

    For a concrete counterexample, consider D=1D=1, t=1/2t=1/2, and n=2n=2, with X0∼N(0,1)X_0\sim N(0,1) independent of X1X_1, Z=(X1+X0)/2Z=(X_1+X_0)/2, and V=X1−X0V=X_1-X_0. On the support (−2,−1,0,1,3)(-2,-1,0,1,3), let PP be the uniform distribution and let QQ have probabilities

    \frac{33279689}{153480506}, \frac{806699977}{6139220240}, \frac{3}{10}, \frac{222347523}{1534805060}, \frac{1270176539}{6139220240} \right).$$ Both probability vectors have strictly positive entries and sum to one. Computing the population normal equations using exact rational arithmetic shows that the two distributions have the same unique optimal degree-2 regressor: $$v_2^*(z)=-\frac{53}{666}+\frac{301}{333}z+\frac{7}{37}z^2.$$ However, their data moments differ: | Distribution | $E[X_1]$ | $E[X_1^2]$ | $E[X_1^3]$ | |---|---:|---:|---:| | P | 0.200000000 | 3.000000000 | 3.800000000 | | Q | 0.200488753 | 3.005663332 | 3.864978818 | Since Gaussian convolution ensures that the Gram matrix is positive definite, this counterexample does not rely on non-uniqueness of the minimizer. Therefore, the conclusion that a single degree-$n$ projection at a fixed time uniquely identifies the moments does not hold in general. A revised claim involving the full hierarchy of lower-degree projections, for example, would fall outside the scope of this counterexample, but would require a separate proof. Sufficient higher-order moment assumptions should also be stated for polynomial regression of general degree. There is a separate issue with the interpretation of the covariance force. Let $R=-I+tA$ and $J=I-tR$. When $J$ is invertible, the exact differential is $d\Sigma=(1-t)J^{-1}(dA)J^{-1}$. This differs from $J_g(A_g-A_r)J_g u$ in Eq. (17). Appendix C.2 calibrates a scalar weight at $\Sigma=I$, but this does not justify an exact Jacobian interpretation in a general anisotropic setting. Under the local analysis at the population affine optimum, $J$ is symmetric positive definite (SPD), and $J(dA)J=\frac{J^2(d\Sigma)J^2}{1-t}$, so there may be an interpretation as a preconditioned surrogate. However, the learned operator is not constrained to be symmetric (Appendix D.3), and replacing a finite real/fake difference with a differential introduces an additional approximation. Thus, this local population analysis does not by itself justify interpreting the implemented finite-difference force as the exact covariance-map Jacobian. The proposed force may still be practically effective as a preconditioned surrogate, but its interpretation and theoretical status need to be clarified.
  • The T2I experiments do not sufficiently disentangle benchmark-specific training from the generalization benefits of the proposed method. Appendix D.6 explicitly states that the composition block of the FLUX training corpus is constructed from the 553 GenEval prompts. For each prompt, multiple teacher samples are generated and detector-verified images are selected, resulting in 53,504 training images. No separate held-out evaluation prompt split is reported. Although I have not directly compared the exact evaluation prompt IDs, this setup makes it difficult to interpret the high GenEval scores as evidence of unseen compositional generalization.

    The AMFD-C versus AMFD-U comparisons provide evidence that conditioning is beneficial under the reported training protocol. However, they do not establish whether this advantage persists without GenEval-specific curation or transfers to unseen prompts and compositions.

    Furthermore, even in the FLUX experiments initialized from the Base model, the reference images are generated by the four-step teacher. The reported result that the student surpasses the teacher's average GenEval score is numerically valid, but it should be interpreted in the context of learning from a detector-filtered teacher-success distribution. In the PixelGen experiments, which use a separate training corpus, AMFD-C achieves a GenEval score of 0.760, below the teacher's 0.794. Thus, the teacher-surpassing result is specific to the reported FLUX settings and is not observed in PixelGen.

Questions for authors
  • What is the exact prompt overlap between FLUX training and GenEval evaluation? Could the authors provide the prompt split lists and clarify whether the conditional advantage persists when the composition block is excluded or when evaluation is performed on unseen prompts or compositions?

  • Does Theorem B.1 claim identification from the coefficients of a single projection or from the entire hierarchy of lower-degree projections? How can the theorem's conclusion and assumptions be revised to accommodate the counterexample above?

  • Is Eq. (17) intended as an exact Jacobian transformation or as a preconditioned surrogate? Could the authors clarify its derivation and validate its behavior beyond the isotropic equilibrium considered in Appendix C.2?

  • Appendix D.6 identifies several externally reported baseline results. Could the authors provide a row-wise provenance summary for the T2I comparisons and clarify the remaining differences in training data, text length, encoders, and evaluation settings?

7 comments

We sincerely thank the reviewer for the rigorous assessment, particularly for providing the concrete counterexample and raising valuable questions regarding our theoretical scope and experimental generalization. In response, we have revised our theoretical analysis and added new text-to-image experiments to address these concerns comprehensively.

[Evaluation metric] Independence of FID and Held-out Evaluation

We agree with the reviewer that FID cannot serve as an independent held-out metric when Inception-v3 is included in the SIM training ensemble (SigLIP2, Inception-v3, and MAE). We also acknowledge the FID trade-offs identified by the reviewer.

To rigorously evaluate out-of-domain generalizability, we report FDr$^3$, defined as the average normalized Fréchet distance across three strictly held-out representations: ConvNeXt, DINOv2, and CLIP. Based on our per-encoder measurements (detailed in Appendix Table 9), AMFD-U substantially improves FDr$^3$ over the FD-Loss baseline, decreasing the score from 4.44 to 2.77 on JiT-H, and from 2.69 to 2.29 on pMF-H.

Although AMFD exhibits slightly worse FID in the Inception feature space used for training, it consistently improves the FDr$^3$ metric. These results indicate that AMFD's benefits extend beyond the feature spaces directly optimized during training, and generalize to held-out representation metrics.

W1. Theoretical issues.

i) (W1,Q2) Higher-order moment identification (Theorem 3.1 / Appendix B.1)

We fully acknowledge that the reviewer's counterexample is valid and that our identification claim does not hold for general polynomial degrees. While the normal equations establish cross-moment identities, these expectations inherently depend on the unknown underlying data distribution. Consequently, they do not guarantee that the coefficients of a single degree-nn projection at a fixed time uniquely determine the first n+1n+1 data moments.

To resolve this, we have revised the theorem by strengthening its identification hypothesis and explicitly stating the moment assumption:

  • Strengthened Identification Condition (From a Single Time to an Open Interval): Our original theorem claimed that equality of the population-optimal degree-nn polynomial projection coefficients at a single fixed time sufficed to identify the first n+1n+1 data moments. We have now strengthened this condition by replacing the single fixed time with every tt in some nonempty open interval I⊂(0,1)I\subset(0,1).
  • Moment Boundedness Assumption: Under independent standard Gaussian corruption, we now explicitly require E[∥X1∥max⁡(2,2n)∣c]<∞\mathbb{E}[\|X_1\|^{\max(2,2n)}\mid c]<\infty.

Under these conditions, we establish that the time-dependent projection coefficients uniquely identify all conditional data moments up to order n+1n+1.

Crucially, the revised proof abandons the attempt to recover unknown cross-moments directly from the fixed-time normal equations. Instead, we take a different approach: we transform the velocity regression into a clean-data polynomial denoising problem and expand its coefficients in a multivariate Hermite basis.

Specifically, by writing a=t/(1−t)a = t/(1-t), we demonstrate that the appropriately rescaled coefficients are rational functions of q=a2q = a^2. We show that their limits as q→0q \to 0 satisfy a unit lower-triangular system that exactly coincides with the multivariate moment--cumulant recurrence. In particular, we have:

lim⁡a→0β! a−∣β∣ci,β(a)=κβ+ei.\lim_{a\to0} \beta! \, a^{-|\beta|} c_{i,\beta}(a) = \kappa_{\beta+e_i}.

Because these rescaled coefficient functions are rational and possess no poles on the non-negative real axis, their values on any nonempty open time interval uniquely determine their limits at a=0a=0. This guarantees the unique identification of all cumulants and moments through order n+1n+1.

This shift in the information requirement fundamentally resolves the counterexample. The reviewer's counterexample correctly demonstrates that the optimal polynomial coefficients can be identical at one fixed time for different distributions, but it does not establish equality of their coefficient functions over an open time interval.

The explicitly revised theorem, its assumptions, and the complete proof are now provided in Theorem 3.1 and Appendix B.

ii) (W1,Q3) The interpretation of the covariance force

We sincerely thank the reviewer for identifying the incorrect mapping direction in our explanation of Eq. (17). Your rigorous observation helped us uncover a misstatement in our manuscript's text. Our intended statistical objective is to measure the covariance discrepancy through the population affine-operator representation:

Ft(Σg;Σr)=12∥At(Σg)−At(Σr)∥F2.F_t(\Sigma_g;\Sigma_r) = \frac12\Vert A_t(\Sigma_g)-A_t(\Sigma_r)\Vert_F^2.

Unfortunately, the manuscript failed to state this objective explicitly and incorrectly attributed the sandwich construction to the Jacobian of the inverse map At↦ΣA_t\mapsto\Sigma. The reviewer’s expression for that inverse-map differential is absolutely correct.

However, for our intended objective, the chain rule relies on the adjoint derivative of the forward map Σ↦At(Σ)\Sigma\mapsto A_t(\Sigma). Specifically, writing

St(Σ)=(1+t)I−t2At(Σ)=(1−t)[t2Σ+(1−t)2I]−1,S_t(\Sigma) = (1+t)I-t^2A_t(\Sigma) = (1-t)[t^2\Sigma+(1-t)^2I]^{-1},

we can precisely compute the differential as:

DAt,Σ[H]=11−tSt(Σ)HSt(Σ).DA_{t,\Sigma}[H] = \frac{1}{1-t}S_t(\Sigma)H S_t(\Sigma).

Consequently, the exact gradient of our objective with respect to Σg\Sigma_g is:

∇ΣgFt=11−tSg(Ag−Ar)Sg.\nabla_{\Sigma_g}F_t = \frac{1}{1-t}S_g(A_g-A_r)S_g.

Thus, at the population affine optimum, the force δaJ\delta a_{\mathrm J} implemented in Eq. (17) is exactly proportional to this gradient projected along uu:

δaJ=(1−t)(∇ΣgFt)u.\delta a_{\mathrm J} = (1-t)(\nabla_{\Sigma_g}F_t)u.

Crucially, this identity holds exactly for general anisotropic covariances, and Ag−ArA_g-A_r is the exact gradient of the squared operator discrepancy (not a finite-difference approximation substituting for an infinitesimal differential). We have revised the objective definition and derivation in Section 4.2 to reflect this forward-map formulation.

We further clarify that the isotropic approximation Σ=I\Sigma=I in Appendix C.2 is used only to calibrate the relative weighting of the mean and covariance losses, rather than to justify the sandwich structure itself.

Regarding operator symmetry, we agree that the population-optimal affine operator is symmetric, whereas our JVP-based neural parameterization does not explicitly enforce this property. However, in Appendix C.3 in the revised manuscript, we show analytically that the first-order contributions of operator asymmetry cancel in the covariance-loss feature gradient, leaving only second-order corrections. And our new full-matrix diagnostics further confirm that, symmetrizing the learned operators changes the RMS covariance-loss gradient by only ~1.5%. These results indicate that operator asymmetry has a limited practical effect on the covariance-loss gradient.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.