- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
Final justification after rebuttal
I appreciate the authors' efforts and detailed responses during the rebuttal.
I now believe that NFD offers a meaningful contribution as a diagnostic metric for measuring departures from pretrained reference geometry, particularly given its demonstrated value beyond existing metrics.
I also appreciate the authors' explicit acknowledgement that the causal relationship between normal drift and quality degradation remains to be established. I hope the authors will clearly communicate this limitation in the final manuscript to avoid potential misinterpretation by readers.
I trust that the authors will incorporate these new findings and their limitations into the revised manuscript (I hope to see it before the end of the discussion period).
Thank you again for your efforts.
This paper introduces NFD, which measures the normal drift of post-trained diffusion models (e.g., after DPO) to distinguish off-manifold drift (i.e., normal drift) from on-manifold drift (i.e., tangential drift), beyond what FID can distinguish.
Unlike FID, which gives the same value for equal-magnitude tangential and normal shifts in the paper's line-translation example, NFD measures normal departure while remaining zero for ideal tangential drift.
Since the exact reference manifold is unknown, the authors approximate it from finite pretrained samples using anchor-centered bounded local linear patches (conceptually similar to piecewise-linear approximations). Due to this approximation, they also report a nonzero NFD for the Base model.
What they find is that (i) the NFD-based model ranking is consistent across different prompt splits and etc, and (ii) NFD has some correlation with preference and aesthetic scores (e.g., Spearman ρ = 0.195 for ImageReward).
They also provide a set of qualitative results suggesting that overall visual quality decreases as NFD increases, with OCR and PickScore (PS) reported alongside the images.
Critical strengths
Critical Strength / Problem Formulation
- The problem formulation itself is interesting, since I also feel that FID cannot properly represent overall generation quality (e.g., for large models such as FLUX, whose training data are not fully disclosed; calculating FID against ImageNet or other publicly available datasets may not be very meaningful when the reference distributions are mismatched).
- Of course, many alternative measures have been proposed to address this issue (e.g., Jayasumana et al., 2024). Still, I think this kind of manifold-wise approach itself has great potential worth exploring.
Critical weaknesses
Critical Weakness / Soundness
TLDR: Overall, the experiments cannot fully support the paper's main claim.
Before discussing the experiments, let me reiterate how I understood the main problem formulation of this manuscript.
- FID cannot properly distinguish tangential drift from normal drift in certain cases.
- Tangential drift redistributes probability mass along the pretrained , which could be a desirable effect.
- However, normal drift moves samples away from the pretrained , which may lead to undesirable artifacts (e.g., reward hacking).
- Therefore, we need NFD.
At this point, I generally agree with this assumption. However, since the paper's main contribution relies on distinguishing these two types of drift, I think at least a controlled experiment showing the differences between tangential and normal drift under similar levels of FID discrepancy is necessary, especially for actual diffusion-generated images.
For example, I would expect a comparison along the following lines:
| Metric | Base | Tangential-dominated | Normal-dominated |
|---|---|---|---|
| FID | Reference | Matched | Matched |
| NFD | Reference | Report | Report |
| ImageReward | Reference | Report | Report |
| CLIPScore | Reference | Report | Report |
| Diversity / Recall | Reference | Report | Report |
| Human visual quality | Reference | Report | Report |
Here, the two types of drift should be controlled or identified independently of the proposed NFD.
-
Furthermore, looking at Table 10, I find it somewhat unclear what additional benefits NFD provides over existing metrics. The reported Spearman rank correlations between NFD and MMD$^2$, and etc are all 1.0, indicating that these metrics produce identical model rankings in this experiment. Given that the paper motivates NFD by highlighting the limitations of existing metrics, I believe it would be important to demonstrate what additional information or insights NFD can provide.
-
Additionally, I am not fully convinced that Tables 11 and 12 sufficiently support the manuscript's main claim. Table 11 reports correlations between NFD and preference/AES scores, while Table 12 presents human evaluations on selected FlowDPO-related pairs. Although these results suggest an association between NFD and image quality, I think further evidence is needed to establish why distinguishing normal drift from tangential drift is particularly important.
Critical Weakness / Insufficient Controlled Ablation
TLDR: At least the effects of feature normalization and norm-related artifacts should be examined.
Looking at Figure 6 and other qualitative results, some high-NFD images seem to exhibit oversaturation or distortion similar to the artifacts discussed in previous diffusion guidance studies, e.g., Sadat et al., 2025 and Jin et al., 2025.
In particular, Jin et al. (2025) discussed how latent norm amplification can lead to color distortions under strong guidance. Although the setting is different, I wonder whether similar norm-related effects might contribute to some of the observed qualitative results.
However, I could not find an ablation comparing NFD with and without explicit L2 normalization of the projected features, or an analysis of the relationship between the generated latent norm and NFD.
Since NFD directly measures Euclidean distances in feature space, I think it would be helpful to clarify the normalization procedure and, if possible, provide additional ablations to better understand the potential influence of magnitude-related effects on the reported results.
Questions for authors
Q1. I think this kind of measurement could also be applicable to inference-time guidance methods, since the evaluation only relies on generated samples and a fixed pretrained reference. Thus, it could be meaningful to evaluate other manifold-related guidance methods, such as Sun et al., 2023 and He et al., 2024, and could provide further insights into the role of manifolds in diffusion models.
Could the authors clarify whether NFD can be directly applied to these guidance methods, and whether there are any results demonstrating its advantages over existing metrics in such settings?
Q2. My main concern, as outlined in the Weaknesses section, is that the problem formulation itself is interesting, but the resulting experiments do not sufficiently demonstrate the practical importance of distinguishing normal drift from tangential drift. Consequently, I remain uncertain about the additional value of NFD beyond existing metrics.
Could the authors clarify whether I have misunderstood any aspect of the results or their implications? I would appreciate a clearer explanation of what NFD reveals that existing metrics cannot capture, particularly in actual diffusion-generated images.
I am open to the possibility that I have underestimated the value of this paper. A convincing clarification of its practical significance and the main experimental findings in the rebuttal could positively affect my assessment, and I would be willing to reconsider my score accordingly.
Author-Reviewer Discussion
Dear Reviewers,
Thank you for your time and thoughtful reviews. With the author–reviewer discussion period ending on November 18 (AoE), please review the authors' rebuttal and share your updated assessment. Specifically, let us know whether your main concerns have been resolved, and if not, which points remain open.
Your input before the deadline is greatly appreciated!
AC