- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a framework for amortized moment matching using neural diffusion denoisers. The authors establish a connection between polynomial denoisers and distribution moments, and develop the AMFD loss to match conditional moments without explicitly computing covariance matrices. The proposed method is evaluated on ImageNet and text-to-image generation, showing improvements over FD-loss and competitive performance against multi-step teacher models.
Critical strengths
-
The connection between diffusion denoisers and polynomial projections is interesting and provides a theoretical perspective on moment matching. The proposed method also avoids explicitly estimating covariance matrices through a matrix-free optimization approach.
-
The authors demonstrate an equivalence between the cross-entropy loss used in their framework and the noise prediction loss employed in standard diffusion models. This provides a useful connection between the proposed formulation and existing diffusion training objectives.
-
The experimental results are promising, particularly the improvements in one-step generation and the extension to text-to-image generation.
Critical weaknesses
One concern is the potential overfitting to the FID metric. Since the neural amortizer is trained in the Inception-v3 feature space, which is also used for FID evaluation, the model may learn to optimize specifically for this feature space. This could lead to lower FID scores without necessarily improving the overall generation quality. Additional evaluation using independent feature extractors or perceptual metrics would help address this concern.
Questions for authors
-
In Figure 2(b), is the order of the AMFD training pipeline reversed? My understanding is that the neural amortizer should first be trained using the Poly Diff loss, and then the trained, frozen amortizer is used to guide generator post-training. Could the authors clarify this?
-
In Equation (7), why is (v_n(x_t,t,c)) directly equated with (x_1-x_0)? Since (v_n) is a learned model optimized toward this target, the equality does not necessarily hold exactly unless the model achieves zero prediction error. Is this equality based on an assumption of an optimal denoiser?
-
Can the proposed method be understood as a more efficient way of performing diffusion model post-training with an FD-based loss, similar to Yang et al. (2026), but replacing explicit covariance computation with an affine approximation and neural amortization? It would be helpful to clarify the main conceptual and practical differences.
-
The higher loss observed for AMFD-C may suggest that the MLP-based amortizer has limited capacity to capture the underlying distribution. Have the authors considered using a more expressive architecture? It would be interesting to see whether this limitation comes from the amortization framework itself or simply from the model capacity.
5 comments
Q3 Relationship to FD-loss.
We thank the reviewer for this insightful interpretation. We agree that AMFD can be viewed as an approach to FD-based moment matching. The key distinction lies in amortization: how the distributional statistics are represented and optimized. While FD-loss explicitly computes the mean and covariance from data samples, AMFD implicitly encodes these statistics in the parameters of neural amortizers through affine denoising regression.
This amortization brings two practical advantages. First, the neural formulation provides more stable training dynamics, particularly when post-training generators from early pretraining checkpoints, where FD-loss can diverge while AMFD remains stable (Figures 6 and 7). Second, neural amortizers naturally support condition-dependent mean and covariance estimation. This capability is particularly beneficial for text-to-image generation, where AMFD-C consistently improves instruction-following performance over AMFD-U in our experiments.
We will clarify this connection to FD-loss and emphasize these two advantages in the revised manuscript.
Q4. Amortizer capacity and conditional matching.
We thank the reviewer for this insightful question. We would like to clarify our architectural choice and the potential reasons for the higher estimation error of AMFD-C.
First, MLPs are a natural choice for our representation-space setting. Our amortizers operate on globally pooled, one-dimensional feature vectors rather than spatial feature maps or token sequences. Since these representations lack explicit spatial structure, MLPs provide a straightforward and computationally efficient architecture for learning the corresponding moment operators. Nevertheless, AMFD is not restricted to MLPs; we also demonstrate its compatibility with Transformer-based amortizers in our native generative space experiments.
Second, we have investigated the effect of model capacity. As shown in Figure 5, increasing the MLP depth and width improves moment estimation accuracy, with diminishing gains near our default configuration. This suggests that simply scaling up the capacity may offer limited additional benefits.
Finally, we believe the higher estimation error of AMFD-C is largely related to the greater difficulty of conditional moment estimation. On ImageNet, AMFD-C must learn class-dependent statistics across 1,000 categories, with only approximately 1,300 samples per class. In contrast, AMFD-U estimates a single global marginal distribution using the entire dataset. Although the conditional amortizer shares parameters across classes, the substantially smaller per-condition sample size makes estimating fine-grained conditional statistics more challenging. Therefore, the observed gap may reflect both task complexity and statistical estimation difficulty, rather than solely insufficient model expressivity.
Thank you for your responses. I believe the current rating is fair and would like to keep it unchanged.
We thank the reviewer for the positive assessment and the constructive questions. We address each of your concerns in detail below.
W1. Evaluation beyond the Inception feature space.
We agree that FID cannot serve as an independent held-out metric when Inception-v3 is included in the SIM training ensemble (SigLIP2, Inception-v3, and MAE). To address this concern and rigorously evaluate out-of-domain generalizability, we additionally introduce FDr$^3$, defined as the average normalized Fréchet distance across three strictly held-out representations: ConvNeXt, DINOv2, and CLIP. Based on our per-encoder measurements, AMFD-U substantially improves FDr$^3$ over the FD-loss baseline, decreasing the score from 4.44 to 2.77 on JiT-H, and from 2.69 to 2.29 on pMF-H (detailed in Appendix Table 9 of the revised manuscript). This strong held-out performance confirms that the benefits of AMFD are not merely a result of overfitting to Inception features, but generalizing to held-out representation metrics.
Q1. Training order in Figure 2(b).
The intended procedure is an online alternating optimization loop, as specified in Algorithm 1 in the appendix. Within each iteration, we first update the amortizers using the current real and generated features, with generated features detached from the amortizer regression objective. We then freeze the amortizer parameters and update the generator using the resulting moment-matching signals. The amortizers are not trained once and permanently frozen; they continue adapting as the generated distribution evolves.
Q2. Regarding the optimality assumption in Eq. (7).
We thank the reviewer for this question. Yes, Eq. (7) is derived under the assumption that the polynomial denoiser reaches its population optimum, which is standard in theoretical analyses of diffusion models. In practice, the neural amortizer approximates this population optimum. Importantly, our amortizer is lightweight and continuously updated using samples from the current generated distribution through alternating optimization, allowing it to adapt as the generator evolves. Our empirical results demonstrate stable training and effective moment estimation, supporting the practical usefulness of this approximation, although exact population optimality is not assumed during training.