- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces ENCORE to improve the quality of generated samples. The method is built on two systematic failures of few-step sampling: power loss in frequency bands and mean shift in feature space. It uses samples generated with more steps as reference and performs two stages. In the first stage (Stage S), the average band powers of the reference and generated sets are used to compute a per-band scaling factor. In the second stage (Stage M), given the last K differences between consecutive denoising outputs, transformed by Stage S, a least-squares regression is fitted so that the feature mean of the corrected samples matches that of the reference set. The method is tested on pixel-space, image-latent, and audio-latent diffusion models and demonstrates improvements across settings.
Critical strengths
- Experiments across different models show consistent improvement. The ablation study shows the importance of both components.
- The method is based on statistics from the sets and a closed-form least-squares formulation. The computational overhead is small.
Critical weaknesses
- For image generation, matching the mean Inception features to the mean features of the reference set might directly improve FID, since FID uses the same features. The result would be more convincing if other metrics rather than just FID also improve.
- The linear-displacement assumption is not clearly explained and formalized. It’s unclear how large the approximation error can be. I see the way is optimized is to propose good enough candidates, then select the best based on - the quantity we actually want to minimize. I wonder what the performance would be if a were optimized by directly minimizing this objective.
Questions for authors
- In Figure 2a, the band power after correction is above 100%. Does this mean that as long as you increase the band power, it’s alright? Is the band power of the same as ? If not, performing stage M also hurts stage S.
- Optional: Provide other metrics for image generation and what if is optimally trained without the need for the linear-displacement assumption.
7 comments
Response to Reviewer L8RW (2/2)
Weakness 1. Metrics other than FID
Please refer to our reply to Q2.
Weakness 2 (remaining point). The linear-displacement assumption and its approximation error
We thank the reviewer for pointing this out. Section 4.2 now states the linear-displacement assumption as Equation 5, with its approximation error :
where is the feature map, is the rescaled sample and are the rescaled directions. The assumption is that the feature-mean shift is linear in the coefficients, so the closed-form solve for sets to zero. To see how large the error is, Appendix C.10 measures on all 64 settings (Table 20). We compared the feature-mean shift predicted by Equation 5 with the shift measured after Stage M is applied on top of Stage S. The relative error, the norm of divided by the norm of the measured shift, lies between 0.05 and 0.81 with an average of 0.31. The cosine between the predicted and the measured shift is at least 0.90 in 59 of the 64 settings, and the lowest is 0.69. The effect of this error on FID is in our reply to Q2, where the closed form is compared with minimizing the discrepancy directly.
Thank you for the detailed responses and the additional experiments.
Q1, W1 Partially addressed, concern remains
Thanks for adding Appendix C.9 and Table 19. I agree that empirically, S2M is better than "M only" in most settings, except for FFHQ-64, so Stage S does help. However, S2M changes the band power again, which doesn’t match the stated contribution that the method “restores band power”. I see your goal is both to “restore band power” and “realign the feature centroid”, but to keep it simple, you optimize them in sequence, and you prefer M since it matters more ("M only" is better than "S only"). The S2M2S result also suggests that matching band power doesn’t matter in the end, so the role of S is unclear.
Why does S still matter even though, in the end, the objective S optimized has been changed?
Q2, W2 Addressed with additional questions
Thanks for the additional experiments and metrics. They addressed my concern that the gains might be limited to FID. On the linear-displacement assumption, the relative error lies between 0.05 and 0.81, with an average of 0.31, which is fairly large for an assumption of zero error. The high cosine shows the predicted shift has the right direction, but not necessarily the right magnitude. However, Table 21 shows the closed form performs close to direct minimization, so this error matters less in practice. Table 23 indeed shows that the direction is important, possibly helping the linear assumption work.
Is it the case that has a lower error than random vectors? If so, do you have any idea about the properties of that would make this the case?
Response to Reviewer L8RW (1/2)
We thank the reviewer for the careful and constructive review. We answer the two questions first and then the remaining point of the weaknesses. The revised PDF has the changes in blue, with the new tables in Appendix C and the human study in Section 5.3.
Q1. Band power above 100% in Figure 2(a)
The band power of the corrected output is not the same as that after Stage S. Stage S matches the band power of the sample to the target, and Stage M then adds the rescaled differences, which changes the band power again. Appendix C.9 reports the band power after each stage (Table 19). After Stage S alone, the 16 band powers lie between 0.90 and 1.08 of the target in all 64 settings. After both stages, they lie between 0.73 and 4.12, which is the overshoot in Figure 2(a).
Stage M therefore does move the band power away from the target that Stage S matched. Matching the band power is the goal of Stage S, but not of the correction as a whole: Stage M fits the feature mean, and the band power of its output can exceed the target. Increasing the band power is not the goal, and keeping it matched does not help either. S2M performs better than Stage S alone in 61 of the 64 settings (Table 10), and matching the band power again after Stage M makes FID worse. In a separate run on CIFAR-10 with Euler (50,000 samples), applying Stage S once more after S2M gives a higher FID at every NFE:
Q2. Other metrics, and optimizing ak without the linear-displacement assumption
Stage M fits the Inception mean, which is one of the two terms of FID, so we first split FID into its mean term and its covariance term (Appendix C.12, Table 23). Over the 64 pixel-space settings, the covariance term is lower than the baseline in 57 settings, although Stage M does not fit it, and in those settings its decrease accounts for 47% of the FID improvement on average. As the reviewer suggested, we also reported metrics other than FID on the same 64 settings. ENCORE achieves a lower KID than the baseline in 60 settings and a lower FD-DINOv2 in 46 settings (Table 24). It also achieves a higher precision in 51 settings and a higher recall in 56 settings. Recall is lower in 6 settings, five of them at NFE 10, by at most 0.02 (Table 25). On Stable Diffusion 1.4, KID, FD-DINOv2, precision, recall and the two human preference scores HPSv2.1 and ImageReward all improve at every NFE (Table 27). Section 5.3 also adds a blind user study on DiT-XL/2 with 32 participants, who chose ENCORE in 97.7%, 92.3% and 86.9% of the pairs at NFE 4, 6 and 8 (Table 4).
The closed form for ak relies on the linear-displacement assumption, so we also minimized the feature-mean discrepancy directly with Gauss–Newton, without this assumption and with the same K and band gains (Appendix C.10, Table 21). Compared with the closed form, directly minimizing the objective lowers FID in 30 of the 64 settings, leaves it unchanged in 14 and gives a higher FID in 20, with an average change of −0.13, so the closed form is close to the direct optimum in most settings.