- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes an adversarial Fréchet distance as a post-training objective for generators, aiming to address the Fréchet hacking problem. This objective can reveal the residual differences between real and generated data, and the learning is stabilized by introducing real feature whitening. Experimental results on models of multiple scales demonstrate the effectiveness of the proposed method.
Critical strengths
-
The research question and motivation are very clear, and the paper provides a deep and insightful analysis of the problems in the current FD loss.
-
The description of the method is fairly detailed, and no obvious errors appear in the formulas or the demonstration.
Critical weaknesses
-
The effectiveness of the method is mainly demonstrated through indirect experimental results, which makes some of the claims potentially too absolute. The authors claim that D_adv can reveal the residual differences between the real and generated distributions, but the premise is that maximizing the Fréchet distance leads to learning differences that are meaningful for visual quality. Actually, the learned differences may also correspond to features unrelated to human visual perceptual quality; the improvements in the experimental results may also stem from the additional trainable network parameters. Currently, no direct experimental evidence shows that D_adv learns meaningful visual differences.
-
The advantage of the proposed method over introducing more frozen encoders is unclear. The authors state that, apart from still being confined to static representations, introducing more frozen encoders also increases computational and memory costs. However, the proposed method introduces new network parameters and whitening computations. Evidence should be provided to show that the proposed method significantly outperforms introducing more frozen encoders, or to show that the overhead added by the proposed method is obviously lower than that of introducing more frozen encoders.
Questions for authors
-
Can the authors provide more direct, metric-based evidence showing that Fréchet hacking occurs in the training of the JiT and pMF generators in the experiments, and that it is alleviated after adopting the proposed method? The current results only indicate improvements in visual quality and quality metrics, and the source of the improvement cannot be clearly identified.
-
Can the authors show how many frozen encoders the currently added computational and memory overhead is equivalent to? It is also suggested to provide the generation results achieved after adding the corresponding number of frozen encoders.
2 comments
Rebuttal
Dear Reviewer KiVb,
Thank you again for your constructive review. Following your main concern that our evidence was indirect, we have added direct analyses, which we briefly summarize here:
- W1 & Q1 (direct evidence of Frechet hacking): (i) A universal perturbation optimized only to lower Inception FD visibly degrades the images and fools the static Inception (-7.0%), yet the adaptive representation learned by AdvFD, which never sees the perturbation, penalizes it by +39.6%. (ii) Tracking held-out FD-r3 (ConvNeXt, DINOv2, CLIP) during training, FD-loss reaches 7.59 at 37.5k steps and then rises to 9.47, a direct sign of Frechet hacking, whereas AdvFD keeps improving to 6.39 over the same interval.
- W2 & Q2 (vs. adding more frozen encoders): The overhead of AdvFD is comparable to one extra frozen encoder, yet AdvFD is clearly better on all three metrics. Even three frozen encoders, at more than twice the overhead, remain behind AdvFD (FID 0.97 vs. 0.92, FD-r6 5.13 vs. 4.26, FD-r3 7.34 vs. 6.43).
We would greatly appreciate it if you could let us know whether these responses address your concerns, or if anything remains unclear. We are happy to provide further clarification during the discussion period.
Best regards, Authors
Rebuttal to KiVb
Thanks for your reviews! Here is our rebuttal.
We thank the reviewer for finding the motivation clear, the analysis of FD-loss insightful, and the method description detailed and correct. The main concern is that our evidence is indirect. Below we add direct evidence for each claim, split over two comments.
W1 & Q1 — Does the adaptive branch learn visually meaningful differences, and is there direct evidence that Fréchet hacking occurs in our main experiments and is alleviated by AdvFD? We agree that our original evidence was indirect. We add two direct analyses on JiT-B.
Evidence 1: AdvFD closes a blind direction of the static encoder. Fréchet hacking arises when the generator exploits directions that a static encoder does not penalize, so we construct such a direction explicitly. As in Figure 2(a), we optimize a universal perturbation solely to lower Inception FID, and add it to JiT-B samples generated by FD-loss. The perturbation visibly degrades the images, yet the static Inception rewards it.
Each row is on its own scale, so the Change column is the comparison. The adaptive representation starts from the same Inception network and never sees the perturbation during training, yet it penalizes the perturbation by 39.6%. The differences it learns therefore include visible degradation that the static space misses, which is consistent with the human preference study in Appendix C.
Evidence 2: hacking occurs in our main setting, and AdvFD removes it. Figure 2(b) uses a single Inception objective and does not include AdvFD, so we repeat the analysis with the SIM objective. We evaluate JiT-B checkpoints every 12.5k steps and track FD-r3 over ConvNeXt, DINOv2, and CLIP, none of which is used in training. The full curves will be added to the appendix of the revised paper.
Under FD-loss, FD-r3 drops to 7.59 at 37.5k steps, then rises to 9.47 (+24.7%) as training continues. The generator keeps fitting the training encoders, while alignment in unseen feature spaces gets worse. This is Fréchet hacking. Under AdvFD, FD-r3 decreases over the same interval, from 7.24 to 6.39 (−11.8%). AdvFD therefore improves results by reducing hacking, not just by training better overall.
The gain does not come from extra parameters. The adaptive branch is used only in the loss and discarded at inference, so the generator is unchanged. With the same extra Inception branch, a scalar discriminator (Inception-D) does not improve FID or FD-r6 (1.73 / 6.72 versus 1.00 / 5.53 for FD-loss), and training the Inception inside SIM-FD jointly with the generator degrades all metrics (7.76 / 12.44 / 12.06). The gain appears only when the branch is updated adversarially under the Fréchet objective.
W2 & Q2 — AdvFD versus adding more frozen encoders. We add frozen encoders to the static SIM objective: one (ResNet-101) and three (ResNet-101, Swin-T, DeiT). None of them is part of FD-r6, so FD-r3 remains fully held-out. All results below are at 75k steps, under the same protocol, so absolute values differ from Table 1, which reports 125k steps.
Overhead (Q2). The overhead of AdvFD is equivalent to about one frozen encoder: +21.0% time and +13.9 GiB, versus +18.4% and +15.4 GiB for ResNet-101. Three frozen encoders cost more than twice as much (+45.8% time, +31.8 GiB) and bring peak memory to 76.0 GiB, close to the 80 GB capacity of our GPUs.
Quality (W2). At comparable overhead, one frozen encoder improves FID to 1.98 and FD-r3 to 8.20, but leaves FD-r6 unchanged (8.51 → 8.53), while AdvFD reaches 0.92, 4.26, and 6.43. Even three frozen encoders, at more than twice the overhead, remain behind AdvFD on all three metrics (0.97 / 5.13 / 7.34). Adding encoders therefore helps to a degree, but a fixed set cannot cover every aspect of the images, so blind spots remain. AdvFD instead adapts the feature space to the current generator, and achieves better results at lower cost.