- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper replaces multi-encoder Fréchet supervision with eight feature views from one frozen DINOv2 encoder, combining CLS and average-pooled patch representations at four depths. It evaluates five ImageNet generators using six representation spaces excluded from both methods' training objectives. The proposed supervision improves the aggregate held-out score while reducing measured training cost, with ablations examining feature depth and pooling.
Critical strengths
- The improvement extends beyond the supervised representation: all five generators obtain lower aggregate held-out FDr6, and each improves in five of the six individual evaluation spaces. ConvNeXt consistently favors FD-SIM, but the breadth of the other improvements makes the result more persuasive than a gain confined to the training features.
- The practical savings are substantial. In the matched H200 benchmark, JiT-B step time falls from 0.691 to 0.321 seconds and peak memory from 108.04 to 54.57 GiB (Table 3), with improvements also measured for pMF-B. The depth and pooling ablations further support using complementary features from the shared encoder.
Critical weaknesses
- It is unclear whether the held-out representations were also kept separate from configuration selection. The paper specifies the chosen layers and shows that different layer sets yield different scores, but does not explain how the layer set and final checkpoints were selected. This distinction matters: excluding an encoder from the training loss alone does not establish an independent evaluation. Clarifying the selection protocol would establish how independently these representations were evaluated.
- The comparison uses 50 post-training epochs for IEMV-FD and 100 for the released FD-SIM checkpoints. It demonstrates a practical advantage at those budgets, but does not separate the effects of supervision design from possible effects of training duration. The shorter run is useful, but the comparison alone does not establish the proposed explanation involving representation-specific overfitting.
Questions for authors
- What criteria selected the layer set and final checkpoints, and did any held-out evaluation encoders inform those choices? This would clarify how independently the generalization result was evaluated.
- If available, how does FD-SIM perform at 50 epochs, or how do its held-out scores evolve during post-training? Existing checkpoint results would help distinguish the supervision effect from training-duration effects.
2 comments
W2/Q2. Matched Training Duration and Optimization Settings
W2/Q2. Matched Training Duration and Optimization Settings
We retrained FD-SIM on pMF-B and JiT-B for 50 epochs, matching IEMV-FD's training duration and optimization settings. The learning rate/global batch size is 5 × 10⁻⁷/128 for pMF-B and 1 × 10⁻⁵/256 for JiT-B. Both methods use identical AdamW settings, cosine scheduling, five warm-up epochs, and model-weight EMA with a decay of 0.9996. Within each generator architecture, they start from the same pretrained generator checkpoint and are evaluated using EMA weights under the same sampling protocol.
Pretrained baselines are included for reference. Lower is better for both FDr⁶ metrics. Encoder parameter counts cover only the post-training supervision encoders and exclude generator parameters.
| Generator | Method | NFE | Supervision Encoders | Encoder Params | Post-training Epochs | FDr⁶ ↓ | Held-out FDr⁶ ↓ |
|---|---|---|---|---|---|---|---|
| pMF-B | Baseline | 1 | — | — | — | 13.83 | 15.13 |
| pMF-B | + FD-SIM | 1 | 3 | 755.0M | 50 | 4.31 | 9.47 |
| pMF-B | + Ours | 1 | 1 | 86.6M | 50 | 3.53 | 6.65 |
| JiT-B | Baseline | 50×2×2† | — | — | — | 15.81 | 18.57 |
| JiT-B | + FD-SIM | 1 | 3 | 755.0M | 50 | 5.85 | 11.00 |
| JiT-B | + Ours | 1 | 1 | 86.6M | 50 | 5.24 | 7.37 |
† We follow the Heun-sampler notation in the JiT paper. The individual factors are not interpreted as independent model calls.
Relative to FD-SIM, IEMV-FD reduces FDr⁶ by 18.1% on pMF-B and 10.4% on JiT-B, and held-out FDr⁶ by 29.8% and 33.0%, respectively.
To compare performance across the two evaluation sets, we define Δ = held-out FDr⁶ - FDr⁶. For a fixed checkpoint, Δ measures how much higher its held-out FDr⁶ is than its FDr⁶.This difference provides a descriptive check for preferential adaptation to particular representation spaces. For methods that directly optimize some FDr⁶ spaces, a larger positive Δ may reflect stronger adaptation to those spaces and less complete transfer to held-out representations. This pattern is consistent with representation-specific optimization bias. We interpret Δ jointly with the absolute scores because the metrics average different encoder sets.
| Generator | Baseline Δ | FD-SIM Δ (50 epochs) | IEMV-FD Δ (50 epochs) |
|---|---|---|---|
| pMF-B | 1.30 | 5.16 | 3.12 |
| JiT-B | 2.76 | 5.15 | 2.13 |
Both the baseline and IEMV-FD have smaller values of Δ than FD-SIM on pMF-B and JiT-B. FD-SIM directly optimizes SigLIP, Inception, and MAE, which account for three of the six FDr⁶ representation spaces. This supervision overlap can preferentially reduce FDr⁶ relative to held-out FDr⁶ and increase Δ. The baseline has no post-training with these FD objectives. IEMV-FD uses internal DINOv2-B views and does not directly optimize the six FDr⁶ evaluation representations.Under matched training duration and optimization settings, IEMV-FD achieves lower absolute scores for both FDr⁶ and held-out FDr⁶, together with a smaller Δ than FD-SIM. Its larger relative improvement on held-out FDr⁶ further supports transfer beyond the directly supervised representations. These observations are consistent with reduced representation-specific optimization bias.
W1/Q1. Configuration Selection and Held-out Evaluation
We thank the reviewer for recognizing the cross-representation improvements and practical efficiency of IEMV-FD, and for highlighting the need to clarify configuration selection and control training duration. Below, we explain the selection and evaluation protocol and report the matched-training comparison.
W1/Q1. Configuration Selection and Held-out Evaluation
The supervision encoder and layer configuration were selected empirically using FDr⁶. The final IEMV-FD checkpoint uses model-weight EMA at the fixed 50-epoch endpoint.
Held-out FDr⁶ was introduced after the initial training experiments were completed and the configurations were fixed. Its scores in Table 4 are retrospective evaluations of these configurations and checkpoints.
Layer exploration was conducted primarily on JiT-B, using both CLS-token and average-pooled patch-token representations throughout. All layer indices below are zero-based.
{11}{10,11}{8,9,10,11}{6,7,8,9,10,11}{2,5,8,11}{5,8,10,11}{4,9,10,11}These comparisons informed three empirical judgments:
{4,9,10,11}as the default based on its lowest observed FDr⁶, with{5,8,10,11}obtaining a close result.On iMF-B and pMF-B, we also compared
{4,9,10,11}with{5,8,10,11}. Both CLS-token and average-pooled patch-token representations are used for all rows below. JiT-B is included for reference.{4,9,10,11}{5,8,10,11}{4,9,10,11}{5,8,10,11}{4,9,10,11}{5,8,10,11}The two configurations obtain numerically close results on iMF-B and pMF-B, supporting their applicability beyond JiT-B. The default configuration
{4,9,10,11}achieves lower scores on both metrics across the three tested generators.We introduced held-out evaluation to examine whether improvements extend to representation spaces not directly used for supervision. FDr⁶ includes SigLIP, Inception, and MAE, which FD-SIM directly optimizes, while its DINOv2-L evaluator belongs to the same model family as our DINOv2-B supervision encoder. We therefore additionally evaluate both methods with six encoders excluded from their supervision objectives.The four additional encoders are DINOv3-L, PE-L, SwinV2-L, and EVA02-L. They provide complementary coverage of self-supervised and vision-language learning, hierarchical Transformer architectures, and masked image modeling (Table 7). Their selection was guided by this coverage, without using the two methods' comparative scores as a selection criterion.ConvNeXt and CLIP also contribute to FDr⁶ and therefore informed configuration selection through that metric. Results from the four additional encoders were computed after the initial experiments and did not inform supervision-encoder, layer, or checkpoint selection.