acceptodds
Under review as a conference paper at ICLR 2027

Intra-Encoder Multi-View Fréchet Supervision for Visual Generation

Abstract

Fr\'echet Distance (FD) has recently been adopted as a training objective for visual generative models, providing distribution-level supervision by matching the feature statistics of real and generated images in frozen visual representation spaces. Optimizing FD in a single representation space can overfit to the statistics captured by that representation, producing a low in-space FD while substantial discrepancies under other representations and poor visual quality remain. FD-loss addresses this issue with FD-SIM, which jointly aligns real and generated distributions across multiple pretrained visual encoders. This multi-encoder supervision broadens representation coverage but introduces substantial additional encoder computation during training. To obtain multi-view Fr\'echet supervision without relying on multiple independent encoders, we propose Intra-Encoder Multi-View Fr\'echet Supervision (IEMV-FD), which constructs multiple distributional supervision views from the internal representations of a single pretrained visual encoder. Specifically, we extract representations from multiple layers and form both CLS-token and average-pooled patch-token views at each selected layer. Existing Fr\'echet-based evaluation metrics may use encoders that also participate in training supervision as evaluation judges, creating supervision-evaluation overlap and making it difficult to isolate generalization to unseen representation spaces. We therefore introduce held-out , which averages normalized Fr\'echet distance ratios across 6 evaluation encoders that are not involved in the training of either IEMV-FD or FD-SIM. On ImageNet class-conditional generation, IEMV-FD achieves lower held-out than FD-SIM across multiple models, including reductions from 10.76 to 7.37 on JiT-B and from 8.01 to 6.65 on pMF-B, while using a single supervision encoder with only about 11.5% of the supervision-encoder parameters used by FD-SIM. Our code and model weights will be made publicly available.

Primary area:
generative models
Keywords:
Fréchet Distance, Multi-View Supervision, Visual Generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_4XHgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper replaces multi-encoder Fréchet supervision with eight feature views from one frozen DINOv2 encoder, combining CLS and average-pooled patch representations at four depths. It evaluates five ImageNet generators using six representation spaces excluded from both methods' training objectives. The proposed supervision improves the aggregate held-out score while reducing measured training cost, with ablations examining feature depth and pooling.

Critical strengths
  1. The improvement extends beyond the supervised representation: all five generators obtain lower aggregate held-out FDr6, and each improves in five of the six individual evaluation spaces. ConvNeXt consistently favors FD-SIM, but the breadth of the other improvements makes the result more persuasive than a gain confined to the training features.
  2. The practical savings are substantial. In the matched H200 benchmark, JiT-B step time falls from 0.691 to 0.321 seconds and peak memory from 108.04 to 54.57 GiB (Table 3), with improvements also measured for pMF-B. The depth and pooling ablations further support using complementary features from the shared encoder.
Critical weaknesses
  1. It is unclear whether the held-out representations were also kept separate from configuration selection. The paper specifies the chosen layers and shows that different layer sets yield different scores, but does not explain how the layer set and final checkpoints were selected. This distinction matters: excluding an encoder from the training loss alone does not establish an independent evaluation. Clarifying the selection protocol would establish how independently these representations were evaluated.
  2. The comparison uses 50 post-training epochs for IEMV-FD and 100 for the released FD-SIM checkpoints. It demonstrates a practical advantage at those budgets, but does not separate the effects of supervision design from possible effects of training duration. The shorter run is useful, but the comparison alone does not establish the proposed explanation involving representation-specific overfitting.
Questions for authors
  1. What criteria selected the layer set and final checkpoints, and did any held-out evaluation encoders inform those choices? This would clarify how independently the generalization result was evaluated.
  2. If available, how does FD-SIM perform at 50 epochs, or how do its held-out scores evolve during post-training? Existing checkpoint results would help distinguish the supervision effect from training-duration effects.
2 comments
AuthorsOpenReview ↗

W1/Q1. Configuration Selection and Held-out Evaluation

We thank the reviewer for recognizing the cross-representation improvements and practical efficiency of IEMV-FD, and for highlighting the need to clarify configuration selection and control training duration. Below, we explain the selection and evaluation protocol and report the matched-training comparison.

W1/Q1. Configuration Selection and Held-out Evaluation

The supervision encoder and layer configuration were selected empirically using FDr⁶. The final IEMV-FD checkpoint uses model-weight EMA at the fixed 50-epoch endpoint.

Held-out FDr⁶ was introduced after the initial training experiments were completed and the configurations were fixed. Its scores in Table 4 are retrospective evaluations of these configurations and checkpoints.

Layer exploration was conducted primarily on JiT-B, using both CLS-token and average-pooled patch-token representations throughout. All layer indices below are zero-based.

Layer configurationFDr⁶ ↓
Last layer {11}12.06
Last two layers {10,11}9.22
Last four layers {8,9,10,11}9.17
Last six layers {6,7,8,9,10,11}9.56
All 12 layers10.11
Evenly spaced {2,5,8,11}8.32
{5,8,10,11}5.43
{4,9,10,11}5.24

These comparisons informed three empirical judgments:

  1. Expanding the range of consecutive layers does not improve FDr⁶ monotonically: the last-two-layer and last-four-layer scores are close, while six or all twelve layers yield higher scores.
  2. Layer placement matters at a fixed four-layer count. Replacing layer 8 with layer 4 in the last-four-layer configuration reduces FDr⁶ from 9.17 to 5.24. Replacing layer 2 with layer 10 in the evenly spaced configuration reduces it from 8.32 to 5.43.
  3. The two strongest tested configurations combine an intermediate-depth layer with deeper layers. We chose {4,9,10,11} as the default based on its lowest observed FDr⁶, with {5,8,10,11} obtaining a close result.

On iMF-B and pMF-B, we also compared {4,9,10,11} with {5,8,10,11}. Both CLS-token and average-pooled patch-token representations are used for all rows below. JiT-B is included for reference.

GeneratorLayers (zero-indexed)FDr⁶ ↓Held-out FDr⁶ ↓
iMF-B{4,9,10,11}4.878.87
iMF-B{5,8,10,11}5.129.01
pMF-B{4,9,10,11}3.536.65
pMF-B{5,8,10,11}3.556.72
JiT-B{4,9,10,11}5.247.37
JiT-B{5,8,10,11}5.437.50

The two configurations obtain numerically close results on iMF-B and pMF-B, supporting their applicability beyond JiT-B. The default configuration {4,9,10,11} achieves lower scores on both metrics across the three tested generators.

We introduced held-out evaluation to examine whether improvements extend to representation spaces not directly used for supervision. FDr⁶ includes SigLIP, Inception, and MAE, which FD-SIM directly optimizes, while its DINOv2-L evaluator belongs to the same model family as our DINOv2-B supervision encoder. We therefore additionally evaluate both methods with six encoders excluded from their supervision objectives.The four additional encoders are DINOv3-L, PE-L, SwinV2-L, and EVA02-L. They provide complementary coverage of self-supervised and vision-language learning, hierarchical Transformer architectures, and masked image modeling (Table 7). Their selection was guided by this coverage, without using the two methods' comparative scores as a selection criterion.ConvNeXt and CLIP also contribute to FDr⁶ and therefore informed configuration selection through that metric. Results from the four additional encoders were computed after the initial experiments and did not inform supervision-encoder, layer, or checkpoint selection.

AuthorsOpenReview ↗

W2/Q2. Matched Training Duration and Optimization Settings

W2/Q2. Matched Training Duration and Optimization Settings

We retrained FD-SIM on pMF-B and JiT-B for 50 epochs, matching IEMV-FD's training duration and optimization settings. The learning rate/global batch size is 5 × 10⁻⁷/128 for pMF-B and 1 × 10⁻⁵/256 for JiT-B. Both methods use identical AdamW settings, cosine scheduling, five warm-up epochs, and model-weight EMA with a decay of 0.9996. Within each generator architecture, they start from the same pretrained generator checkpoint and are evaluated using EMA weights under the same sampling protocol.

Pretrained baselines are included for reference. Lower is better for both FDr⁶ metrics. Encoder parameter counts cover only the post-training supervision encoders and exclude generator parameters.

GeneratorMethodNFESupervision EncodersEncoder ParamsPost-training EpochsFDr⁶ ↓Held-out FDr⁶ ↓
pMF-BBaseline1———13.8315.13
pMF-B+ FD-SIM13755.0M504.319.47
pMF-B+ Ours1186.6M503.536.65
JiT-BBaseline50×2×2†———15.8118.57
JiT-B+ FD-SIM13755.0M505.8511.00
JiT-B+ Ours1186.6M505.247.37

† We follow the Heun-sampler notation in the JiT paper. The individual factors are not interpreted as independent model calls.

Relative to FD-SIM, IEMV-FD reduces FDr⁶ by 18.1% on pMF-B and 10.4% on JiT-B, and held-out FDr⁶ by 29.8% and 33.0%, respectively.

To compare performance across the two evaluation sets, we define Δ = held-out FDr⁶ - FDr⁶. For a fixed checkpoint, Δ measures how much higher its held-out FDr⁶ is than its FDr⁶.This difference provides a descriptive check for preferential adaptation to particular representation spaces. For methods that directly optimize some FDr⁶ spaces, a larger positive Δ may reflect stronger adaptation to those spaces and less complete transfer to held-out representations. This pattern is consistent with representation-specific optimization bias. We interpret Δ jointly with the absolute scores because the metrics average different encoder sets.

GeneratorBaseline ΔFD-SIM Δ (50 epochs)IEMV-FD Δ (50 epochs)
pMF-B1.305.163.12
JiT-B2.765.152.13

Both the baseline and IEMV-FD have smaller values of Δ than FD-SIM on pMF-B and JiT-B. FD-SIM directly optimizes SigLIP, Inception, and MAE, which account for three of the six FDr⁶ representation spaces. This supervision overlap can preferentially reduce FDr⁶ relative to held-out FDr⁶ and increase Δ. The baseline has no post-training with these FD objectives. IEMV-FD uses internal DINOv2-B views and does not directly optimize the six FDr⁶ evaluation representations.Under matched training duration and optimization settings, IEMV-FD achieves lower absolute scores for both FDr⁶ and held-out FDr⁶, together with a smaller Δ than FD-SIM. Its larger relative improvement on held-out FDr⁶ further supports transfer beyond the directly supervised representations. These observations are consistent with reduced representation-specific optimization bias.

Reviewer_UGxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary

This paper proposes Intra-Encoder Multi-View Fréchet Supervision (IEMV-FD), an efficient framework for Fréchet Distance (FD)-based generative model training. Building upon FD-loss, the authors aim to reduce the computational overhead of multi-encoder supervision while preserving representation diversity. Instead of employing multiple independent pretrained visual encoders, IEMV-FD extracts CLS-token and average-pooled patch-token representations from different layers of a single frozen DINOv2 encoder. Each representation serves as an independent supervision view with its own Fréchet loss, and these losses are jointly optimized. The authors further introduce held-out FDr6, an evaluation metric that measures distributional discrepancies across six representation spaces not directly involved in training supervision. Experiments on ImageNet 256×256 class-conditional generation demonstrate consistent improvements in held-out FDr6 over FD-SIM across five generator configurations, including JiT, pMF, and iMF. The method also substantially reduces supervision-encoder parameters, GPU memory consumption, and training time. Ablation studies further demonstrate the benefits of combining representations from multiple layers and different token aggregation strategies.

Critical strengths
  1. Strong empirical improvements with substantially reduced computational overhead.

The proposed method consistently achieves lower held-out FDr6 than FD-SIM across five generator configurations, covering both latent-space and pixel-space generation. Importantly, these improvements are achieved using only a single pretrained visual encoder, substantially reducing the computational cost of multi-view supervision. The efficiency experiments further demonstrate significant reductions in encoder GFLOPs (77.9%) and peak GPU memory consumption (49.5%–56.0%), together with a 1.67–2.15× increase in training throughput. These results demonstrate the practical value of the proposed framework and provide strong empirical support for its effectiveness.

  1. Meaningful validation of intra-encoder multi-view supervision and cross-representation transfer.

The ablation studies provide direct evidence that combining representations from different encoder layers and token aggregation strategies improves generative training. In particular, combining CLS-token and average-pooled patch-token representations across multiple layers consistently outperforms using either representation type alone. Furthermore, the proposed held-out FDr6 evaluates models across six representation spaces that are not directly involved in training supervision. The consistent improvements across these evaluation spaces suggest that the benefits of intra-encoder multi-view supervision extend beyond the directly optimized representation spaces. These findings support the central hypothesis that diverse internal representations of a single pretrained encoder can provide effective distribution-level supervision.

Critical weaknesses

Critical Weakness 1. Insufficiently controlled comparison with FD-SIM.

Although IEMV-FD demonstrates consistent improvements over FD-SIM, the comparison does not fully control for differences in the post-training configurations. Specifically, FD-SIM uses 100 training epochs, a global batch size of 1024, and no model-weight EMA, whereas IEMV-FD uses 50 epochs, batch sizes of 128 or 256, and model-weight EMA with a decay of 0.9996. The learning rates also differ for some generator configurations.

These differences may substantially affect the optimization outcome. First, despite using fewer epochs, IEMV-FD performs approximately 2–4 times more optimizer updates due to its smaller batch sizes, potentially leading to different convergence behavior. Second, batch size is particularly consequential for FD-based training, as it affects both gradient variance and the estimation of feature distribution statistics. Even with the same statistics-EMA decay, smaller batches change the variance and effective sample horizon of the estimated moments, which directly influence the FD objective and its gradients. Third, model-weight EMA introduces additional parameter smoothing that may affect the final generation quality independently of the proposed supervision strategy.

Consequently, it remains unclear whether the observed improvements arise primarily from intra-encoder multi-view supervision or from differences in the optimization protocol. A matched comparison with FD-SIM under identical training configurations, including batch size, optimization budget, and model-weight EMA, would be important for establishing the independent contribution of the proposed method.

  1. Limited independent evidence connecting held-out Fréchet improvements to actual generation quality.

The proposed held-out FDr6 provides a reasonable evaluation of distribution matching across representation spaces that are not directly involved in training supervision. However, previous studies have demonstrated that Fréchet-based metrics can be sensitive to the choice of representation space and may not always correlate with perceptual generation quality [1][2]. In particular, Appendix Table 8 shows that FD-SIM achieves substantially lower Inception FDr across all five generator configurations. For iMF-B, IEMV-FD even increases Inception FDr relative to the original baseline (2.81 vs. 2.07). Although this discrepancy may reflect the known limitations of Inception-based evaluation, it highlights that the relative performance depends on the evaluation representation. Furthermore, all six held-out metrics measure marginal feature-distribution discrepancies and do not directly assess whether generated images match their conditioning labels. Additional independent evidence, such as human preference evaluations or class-conditional fidelity measurements, would strengthen the claim that IEMV-FD improves generation quality beyond representation-level distribution matching.

References

[1] Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., and Lehtinen, J. The Role of ImageNet Classes in Fréchet Inception Distance. International Conference on Learning Representations (ICLR), 2023.

[2] Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., and Kumar, S. Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9307–9315.

Questions for authors

Q1. Clarification on the independence of held-out FDr6 evaluation.

The paper introduces held-out FDr6 to evaluate generalization across representation spaces that are not directly involved in training supervision. However, Table 4(b) compares different layer configurations using held-out FDr6, and the configuration {4, 9, 10, 11}, which achieves the best score, is adopted in the main experiments.

Could the authors clarify how this layer configuration was selected? Specifically, was held-out FDr6 used during hyperparameter selection, or was the configuration determined independently of the held-out evaluation encoders?

This distinction is important because, even if the evaluation encoders are excluded from the training objective, using their scores to select the supervision configuration would introduce indirect evaluation feedback. In that case, the held-out FDr6 results would not constitute a fully independent assessment of generalization to unseen representation spaces.

Q2. Clarification on the effective contribution of different Fréchet supervision views.

In Eq. (8), the authors combine the Fréchet losses from different encoder layers and token aggregation strategies using a uniform average. However, Fréchet Distance is sensitive to feature magnitude, and representations extracted from different layers or pooling operations may have substantially different feature distributions and scales. Consequently, assigning equal weights to all views does not necessarily imply that they contribute equally to the optimization.

Could the authors clarify whether the representations or individual Fréchet losses are normalized to ensure comparable contributions? Additionally, it would be informative to examine the relative loss magnitudes or gradient contributions of different views. This would help determine whether the improvements observed in Table 4 arise from genuinely complementary representation information or are primarily driven by a few dominant supervision views.

4 comments
AuthorsOpenReview ↗

W1. Matched post-training comparison with FD-SIM

W1. Matched post-training comparison with FD-SIM

We agree that differences in the post-training protocol could affect the comparison in the submitted manuscript. To address this concern, we retrained FD-SIM on pMF-B and JiT-B using the same post-training configuration as IEMV-FD within each generator architecture, while retaining FD-SIM's three-encoder supervision objective with SigLIP2, Inception-v3, and MAE.

Within each architecture, both methods start from the same pretrained generator checkpoint and undergo 50 post-training epochs with the same number of optimizer updates. The global batch size and learning rate are matched as follows:

GeneratorGlobal batch sizeLearning rate
pMF-B1285 × 10⁻⁷
JiT-B2561 × 10⁻⁵

We also match the AdamW optimizer, cosine learning-rate schedule with five warmup epochs, feature-statistics EMA decay of 0.999, and model-weight EMA decay of 0.9996. We evaluate the final model-weight EMA checkpoint at epoch 50 using the same single-step sampling protocol and 50,000 class-balanced generated images per checkpoint.

GeneratorMethodNFESupervision EncodersEncoder ParamsPost-training EpochsFDr⁶ ↓Held-out FDr⁶ ↓
pMF-BBaseline1———13.8315.13
pMF-B+ FD-SIM13755.0M504.319.47
pMF-B+ Ours1186.6M503.536.65
JiT-BBaseline50×2×2†———15.8118.57
JiT-B+ FD-SIM13755.0M505.8511.00
JiT-B+ Ours1186.6M505.247.37

† We follow the Heun-sampler notation in the original JiT paper. The factors are not interpreted as independent model calls.

Relative to the matched FD-SIM baselines, IEMV-FD reduces held-out FDr⁶ by 29.8% on pMF-B and 33.0% on JiT-B. Both aggregate metrics improve, and IEMV-FD achieves lower FDr in five of the six held-out representation spaces on each architecture. These results show that the advantage persists when training duration, optimizer-update count, batch size, learning rate, and both EMA settings are matched.

The retrained pMF-B FD-SIM baseline obtains an FDr⁶ of 4.31, close to the 4.20 reported by Yang et al. at 50 epochs under their original recipe in Table 1(c) of Representation Fréchet Loss for Visual Generation. IEMV-FD also achieves lower held-out FDr⁶ than the released 100-epoch FD-SIM checkpoints re-evaluated in Table 1 of our submitted manuscript: 6.65 vs. 8.01 on pMF-B, and 7.37 vs. 10.76 on JiT-B.

AuthorsOpenReview ↗

W2. Generation quality: human preference and CMMD

W2. Generation quality: human preference and CMMD

We agree that held-out Fréchet scores alone do not establish perceptual generation quality. We therefore supplemented them with a human preference study and CMMD, introduced by Jayasumana et al. in reference [2] cited by the reviewer.

Human preference. We compared IEMV-FD with FD-SIM on iMF-L, pMF-L, JiT-L, and JiT-H. For each generator and method, we sampled 1,000 class-balanced images, with corresponding images sharing the same class condition and initial noise, yielding 8,000 images in total.

Each question presented eight images from each method, giving 125 questions per generator and 500 questions overall. Twelve participants were divided into four groups of three, with each group evaluating one generator. Method identities were hidden, and the two methods were randomly assigned to the left and right panels. Participants made a forced binary choice based on overall visual quality. Each question received three ratings, and its winner was determined by majority vote.

GeneratorIEMV-FD majority-vote winsMajority-vote preference rate
iMF-L82/12565.6%
pMF-L69/12555.2%
JiT-L88/12570.4%
JiT-H84/12567.2%
Overall323/50064.6%

The study collected 1,500 individual ratings. IEMV-FD won 323 of 500 image-panel comparisons, corresponding to an overall majority-vote preference rate of 64.6%, with rates above 50% on all four generators. These percentages describe panel-level majority-vote outcomes, rather than the fraction of individual ratings or per-image wins.

CMMD. Following reference [2], we additionally evaluated kernel maximum mean discrepancy in CLIP feature space. We used the official implementation and default encoder, evaluating the same 50,000 generated images used for the other automatic metrics for each checkpoint.

GeneratorBaselineFD-SIMIEMV-FD
iMF-B0.770.380.33
iMF-L0.560.290.28
pMF-B0.700.280.27
pMF-L0.520.210.21
JiT-B0.710.430.35
JiT-L0.570.290.28
JiT-H0.470.270.26

CMMD-50K. Lower is better. Bold indicates the lowest reported value in each row, including ties.

IEMV-FD achieves lower CMMD on six generators and matches FD-SIM on pMF-L at the reported precision. For iMF-B, specifically highlighted by the reviewer, CMMD decreases from 0.77 for the baseline to 0.33 for IEMV-FD, compared with 0.38 for FD-SIM.

Together, these evaluations provide complementary perceptual and non-Fréchet evidence beyond held-out Fréchet scores. CMMD uses a different measure of distributional discrepancy, and the human study directly measures visual preference. We will include both evaluations and their complete protocols in the revised manuscript.

AuthorsOpenReview ↗

Q1. Configuration Selection and Held-out Evaluation

Q1. Configuration Selection and Held-out Evaluation

The supervision encoder and layer configuration were selected empirically using FDr⁶.Held-out FDr⁶ was introduced after the initial training experiments were completed and the configurations were fixed. Its scores in Table 4 are retrospective evaluations of these configurations.

Layer exploration was conducted primarily on JiT-B, using both CLS-token and average-pooled patch-token representations throughout. All layer indices below are zero-based.

Layer configurationFDr⁶ ↓
Last layer {11}12.06
Last two layers {10,11}9.22
Last four layers {8,9,10,11}9.17
Last six layers {6,7,8,9,10,11}9.56
All 12 layers10.11
Evenly spaced {2,5,8,11}8.32
{5,8,10,11}5.43
{4,9,10,11}5.24

These comparisons informed three empirical judgments:

  1. Expanding the range of consecutive layers does not improve FDr⁶ monotonically: the last-two-layer and last-four-layer scores are close, while six or all twelve layers yield higher scores.
  2. Layer placement matters at a fixed four-layer count. Replacing layer 8 with layer 4 in the last-four-layer configuration reduces FDr⁶ from 9.17 to 5.24. Replacing layer 2 with layer 10 in the evenly spaced configuration reduces it from 8.32 to 5.43.
  3. The two strongest tested configurations combine an intermediate-depth layer with deeper layers. We chose {4,9,10,11} as the default based on its lowest observed FDr⁶, with {5,8,10,11} obtaining a close result.

On iMF-B and pMF-B, we also compared {4,9,10,11} with {5,8,10,11}. Both CLS-token and average-pooled patch-token representations are used for all rows below. JiT-B is included for reference.

GeneratorLayers (zero-indexed)FDr⁶ ↓Held-out FDr⁶ ↓
iMF-B{4,9,10,11}4.878.87
iMF-B{5,8,10,11}5.129.01
pMF-B{4,9,10,11}3.536.65
pMF-B{5,8,10,11}3.556.72
JiT-B{4,9,10,11}5.247.37
JiT-B{5,8,10,11}5.437.50

The two configurations obtain numerically close results on iMF-B and pMF-B, supporting their applicability beyond JiT-B. The default configuration {4,9,10,11} achieves lower scores on both metrics across the three tested generators.

We introduced held-out evaluation to examine whether improvements extend to representation spaces not directly used for supervision. FDr⁶ includes SigLIP, Inception, and MAE, which FD-SIM directly optimizes, while its DINOv2-L evaluator belongs to the same model family as our DINOv2-B supervision encoder. We therefore additionally evaluate both methods with six encoders excluded from their supervision objectives.The four additional encoders are DINOv3-L, PE-L, SwinV2-L, and EVA02-L. They provide complementary coverage of self-supervised and vision-language learning, hierarchical Transformer architectures, and masked image modeling (Table 7). Their selection was guided by this coverage, without using the two methods' comparative scores as a selection criterion. ConvNeXt and CLIP also contribute to FDr⁶ and therefore informed configuration selection through that metric. Results from the four additional encoders were computed after the initial experiments and did not inform supervision-encoder or layer selection.

AuthorsOpenReview ↗

Q2. Per-view normalization and effective contributions

Q2. Per-view normalization and effective contributions

We thank the reviewer for raising this issue. Section 4.3 and Eq. (8) of the submitted manuscript omitted the per-view loss normalization used in the actual training implementation, leaving the meaning of “uniform averaging” insufficiently specified. We will include this key detail in the revised manuscript.

Let Ls\mathcal{L}_s denote the raw Fréchet loss for view ss. The normalized loss and training objective are

= \frac{\mathcal{L}_s}{\operatorname{sg}(\mathcal{L}_s)+c}, \qquad \mathcal{L}_{\mathrm{IEMV\text{-}FD}} = \frac{1}{|\mathcal{S}|} \sum_{s\in\mathcal{S}}\widetilde{\mathcal{L}}_s, \qquad c=0.01.$$ Here, $\operatorname{sg}$ denotes stop-gradient, and the same constant $c$ is used for all views. This normalization directly addresses differences in loss scale across views. In the training logs, the ratios of the largest to the smallest raw FD losses across views are approximately **13.24 on JiT-B**, **21.18 on pMF-B**, and **5.78 on iMF-B**. The logs record FD before normalization, whereas the quantities aggregated in the training objective are the normalized losses $\widetilde{\mathcal{L}}_s$. The fraction of the total normalized loss value assigned to view $s$ is $$p_s = \frac{\widetilde{\mathcal{L}}_s} {\sum_{j\in\mathcal{S}}\widetilde{\mathcal{L}}_j}.$$ When each view's raw loss is substantially larger than $c$, each $\widetilde{\mathcal{L}}_s$ approaches 1, so its fraction in the eight-view configuration approaches **12.5%**. This follows mathematically from the normalization mechanism. The corresponding per-view contribution to the generator gradient is $$g_s = \frac{1}{|\mathcal{S}|} \frac{\nabla_\theta\mathcal{L}_s} {\operatorname{sg}(\mathcal{L}_s)+0.01}.$$ Thus, “uniform weights” refers to identical aggregation coefficients applied to the normalized terms. The mechanism rescales each view's gradient according to its own loss scale. We will explicitly distinguish normalized loss-value fractions from gradient contributions in the revised manuscript. To examine the effects of different representations, we considered a layer ablation that keeps the number of views fixed. On JiT-B, both configurations use four layers, with CLS and AVG views at each layer. Changing the layer set from {8,9,10,11} to {4,9,10,11} replaces only the two views associated with one layer and reduces the original FDr⁶ from **9.17 to 5.24**, a **42.86% relative reduction**. This comparison keeps the eight supervision views and their aggregation coefficients fixed, showing that the choice of supervision representations materially affects the final result. The token-aggregation ablation further shows that, with the layer set fixed to {4,9,10,11}, combining CLS and AVG achieves a held-out FDr⁶ of **7.37**, compared with **8.30** for CLS alone and **9.57** for AVG alone. These correspond to relative reductions of **11.20%** and **22.99%**, respectively. These results support the complementary benefits of jointly using representations from different depths and the two token-aggregation strategies. We will include the complete normalization formula, gradient expression, and these ablation comparisons in the revised manuscript to clarify both the treatment of loss scale and the performance benefits of combining representations.
Reviewer_oWQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Intra-Encoder Multi-View Frechet Supervision (IEMV-FD) for multi-view Frechet supervision without relying on multiple independent encoders. It constructs multiple distributional supervision views from the internal representations of a single pretrained visual encoder, instead of multiple pretrained visual encoders introducing substantial additional encoder computation during training (used in FD-SIM). Specifically, it extracts representations from multiple layers and form both CLS-token and average-pooled patch-token views at each selected layer. In addition, this paper introduces held-out FDr6, which averages normalized Frechet distance ratios across 6 evaluation encoders that are not involved in the training of either IEMV-FD or FD-SIM. On ImageNet 256×256 class-conditional generation, the proposed IEMV-FD achieves lower held-out FDr6 than FD-SIM across multiple models, while using a single supervision encoder with only less parameters than that used by FD-SIM.

Critical strengths
  1. The paper reduces computational cost and improves training efficiency: it matches (and, on the held-out FDr6 metric, slightly surpasses) FD-SIM's performance while using far fewer supervision-encoder parameters, lowering per-image GFLOPs, and increasing throughput.
  2. The paper identifies the encoder-overlap problem in FDr6, where several evaluation encoders are also used for training supervision, and proposes a held-out FDr6 that excludes these encoders, offering the community a cleaner protocol for assessing generalization beyond directly supervised representation spaces.
Critical weaknesses
  1. The methodological novelty is limited. At its core, IEMV-FD combines multi-layer features with two token-aggregation strategies (CLS and average pooling) drawn from a single frozen encoder, and then averages the per-view Fréchet losses with uniform weights. Both of multi-layer feature extraction and CLS/average-pooled token aggregation are standard practices in representation learning, and the paper provides no theoretical analysis of why internal-layer views should yield supervision that generalizes better than independently trained encoders, nor any principled criterion for selecting layers or weighting views. The held-out FDr6 metric is conceptually a straightforward re-instantiation of FDr6 with a different encoder set. As a result, the contribution is better characterized as a well-executed and practically valuable engineering simplification than as a conceptually novel advance.
  2. The empirical evidence is weakened by potential confounds in the experimental setup, and several analyses are incomplete. First, FD-SIM is post-trained for 100 epochs while IEMV-FD uses only 50, and the two methods may also differ in learning rate (e.g., JiT uses 0.00001 in Table 5), leaving it unclear whether the reported gains reflect the method itself or differences in training budget and optimization. Second, the supervision encoder DINOv2-B and the evaluation encoder DINOv3-L belong to the same family, which may give IEMV-FD an unjustified advantage on the held-out FDr6 metric. Third, it is unclear whether the fixed random seeds were chosen specifically for this comparison. On the analysis side, the layer-selection ablation only reports a handful of combinations, without evidence on how sensitive the results are to other choices or whether {4,9,10,11} was selected independently of the JiT-B results. Moreover, the authors do not explain why FD-SIM achieves a slightly better performance under ConvNeXt, and this counterexample to the claimed "generalization to unseen representation spaces" is left unaddressed.
Questions for authors
  1. What is the information-theoretic basis for constructing multiple distributional supervision views from the internal representations of a single pretrained visual encoder, rather than from multiple independent pretrained encoders? Are these internal views statistically independent? If not, does their correlation weaken the intended effect of multi-view distributional supervision?
  2. Apart from empirical performance, what is the intrinsic, interpretability-grounded reason for selecting layers {4, 9, 10, 11}? What happens when more layers are included? And is the layer set {4, 9, 10, 11}, chosen via the ablation on JiT-B/16, also appropriate for the other generator architectures?
  3. What are the inclusion and exclusion criteria for the held-out FDr6 encoder set? Could you report the score when only its subset {ConvNeXtV2-B, CLIP ViT-L/14, SwinV2-L} is used?
4 comments
AuthorsOpenReview ↗

W2/Q3. Held-out encoder criteria, DINO-family effects, and ConvNeXt

W2/Q3. Held-out encoder criteria, DINO-family effects, and ConvNeXt

Held-out evaluation examines whether improvements over the original generator extend to representation spaces not directly used for supervision. The original FDr⁶ includes SigLIP2, Inception, and MAE, which FD-SIM directly optimizes, as well as DINOv2-L, which belongs to the same family as our DINOv2-B supervision encoder. We therefore added evaluation with representative pretrained encoders spanning different architectures and training objectives, excluding encoders used in either method's Fréchet supervision objective.

The resulting set is DINOv3-L, PE-L, SwinV2-L, EVA02-L, ConvNeXtV2-B, and CLIP ViT-L/14. Exact checkpoints and feature extraction settings are listed in Table 7. ConvNeXt and CLIP are retained from the original metric, while the other four provide additional evaluation spaces. The set was established after the original training experiments, without screening candidate encoders according to which method they favored. The aggregate follows the original FDr⁶ normalization and arithmetic averaging procedure.

We directly report the subset requested by the reviewer: ConvNeXtV2-B, CLIP ViT-L/14, and SwinV2-L. Averaging these three components of Table 9 gives:

GeneratorBaseline mean FDr ↓FD-SIM mean FDr ↓IEMV-FD mean FDr ↓Reduction relative to FD-SIM
iMF-B12.125.414.4417.9%
iMF-L7.713.192.7015.5%
pMF-B10.173.763.448.3%
pMF-L6.552.562.0719.1%
JiT-B15.227.814.7339.4%
JiT-L11.555.433.0643.6%
JiT-H8.924.492.6241.6%

On this subset, which contains no DINO evaluation encoder and retains ConvNeXt, IEMV-FD obtains lower mean FDr than FD-SIM on all seven generators, with reductions of 8.3% to 43.6%.

Generalization to unseen representations is assessed relative to the baseline generator before post-training. Table 9 shows that IEMV-FD obtains lower FDr than this baseline in all six held-out spaces across all seven generators, including ConvNeXt. The comparison with FD-SIM evaluates the relative performance of the two post-training supervision schemes.

AuthorsOpenReview ↗

W2. Matched post-training comparison with FD-SIM

W2. Matched post-training comparison with FD-SIM

We agree that differences in the post-training protocol could affect the comparison in the submitted manuscript. To address this concern, we retrained FD-SIM on pMF-B and JiT-B using the same post-training configuration as IEMV-FD within each generator architecture, while retaining FD-SIM's three-encoder supervision objective with SigLIP2, Inception-v3, and MAE.

Within each architecture, both methods start from the same pretrained generator checkpoint and undergo 50 post-training epochs with the same number of optimizer updates. The settings below apply to both FD-SIM and IEMV-FD:

GeneratorGlobal batch sizeLearning rate
pMF-B1285 × 10⁻⁷
JiT-B2561 × 10⁻⁵

We also match the AdamW optimizer, cosine learning-rate schedule with five warmup epochs, feature-statistics EMA decay of 0.999, and model-weight EMA decay of 0.9996. We evaluate the final model-weight EMA checkpoint at epoch 50 using the same single-step sampling protocol and 50,000 class-balanced generated images per checkpoint.

Matched post-training results. Each generator is compared using the same training and sampling settings. The FD-SIM rows below are the newly retrained 50-epoch models.

GeneratorPost-training methodSampling NFEFrozen supervision encoders (#)Frozen supervision params (M)Post-training epochsFDr⁶ ↓Held-out FDr⁶ ↓
pMF-BFD-SIM (matched retraining)13755.0504.319.47
pMF-BIEMV-FD1186.6503.536.65
JiT-BFD-SIM (matched retraining)13755.0505.8511.00
JiT-BIEMV-FD1186.6505.247.37

Parameter counts cover only the frozen visual supervision backbones, excluding the generator. Bold marks the lower value between the two post-training methods within each generator.

For reference, the original pMF-B generator obtains FDr⁶/held-out FDr⁶ of 13.83/15.13 at 1 NFE. The original JiT-B generator obtains 15.81/18.57 with its original Heun sampler, written as 50×2×2† in the source notation.

† The JiT sampling setting is reproduced in the original paper's notation and is not converted here to a standardized NFE count.

Relative to the matched FD-SIM runs, IEMV-FD reduces held-out FDr⁶ by 29.8% on pMF-B and 33.0% on JiT-B. Both aggregate metrics improve, and IEMV-FD achieves lower FDr in five of the six held-out representation spaces on each architecture. These results show that the advantage persists when training duration, optimizer-update count, batch size, learning rate, and both EMA settings are matched.

The retrained pMF-B FD-SIM model obtains an FDr⁶ of 4.31, close to the 4.20 reported by Yang et al. at 50 epochs under their original recipe in Table 1(c) of Representation Fréchet Loss for Visual Generation. IEMV-FD also achieves lower held-out FDr⁶ than the released 100-epoch FD-SIM checkpoints re-evaluated in Table 1 of our submitted manuscript: 6.65 vs. 8.01 on pMF-B, and 7.37 vs. 10.76 on JiT-B.

Random seeds and evaluation. The evaluation seed was fixed at 0 before evaluation for reproducibility, with no seed screening based on comparison scores. Within each architecture, both methods use the same seed and matched sampling settings. For each checkpoint, the same generated sample set is evaluated by all encoders.

AuthorsOpenReview ↗

W1/W2/Q2. Empirical layer selection, sensitivity, and use across generators

W1/W2/Q2. Empirical layer selection, sensitivity, and use across generators

The supervision encoder and layer configuration were selected empirically using FDr⁶. Held-out FDr⁶ was introduced after the initial training experiments were completed and the configurations were fixed. Its scores in Table 4 are retrospective evaluations of these configurations. Layer exploration was conducted primarily on JiT-B, using both CLS-token and average-pooled patch-token representations throughout. All layer indices below are zero-based.

Layer configurationFDr⁶ ↓
Last layer {11}12.06
Last two layers {10,11}9.22
Last four layers {8,9,10,11}9.17
Last six layers {6,7,8,9,10,11}9.56
All 12 layers10.11
Evenly spaced {2,5,8,11}8.32
{5,8,10,11}5.43
{4,9,10,11}5.24

These comparisons informed three empirical judgments:

  1. Expanding the range of consecutive layers does not improve FDr⁶ monotonically: the last-two-layer and last-four-layer scores are close, while six or all twelve layers yield higher scores.
  2. Layer placement matters at a fixed four-layer count. Replacing layer 8 with layer 4 in the last-four-layer configuration reduces FDr⁶ from 9.17 to 5.24. Replacing layer 2 with layer 10 in the evenly spaced configuration reduces it from 8.32 to 5.43.
  3. The two strongest tested configurations combine an intermediate-depth layer with deeper layers. We chose {4,9,10,11} as the default based on its lowest observed FDr⁶, with {5,8,10,11} obtaining a close result.

On iMF-B and pMF-B, we also compared {4,9,10,11} with {5,8,10,11}. Both CLS-token and average-pooled patch-token representations are used for all rows below. JiT-B is included for reference.

GeneratorLayers (zero-indexed)FDr⁶ ↓Held-out FDr⁶ ↓
iMF-B{4,9,10,11}4.878.87
iMF-B{5,8,10,11}5.129.01
pMF-B{4,9,10,11}3.536.65
pMF-B{5,8,10,11}3.556.72
JiT-B{4,9,10,11}5.247.37
JiT-B{5,8,10,11}5.437.50

The two configurations obtain numerically close results on iMF-B and pMF-B, supporting their applicability beyond JiT-B. The default configuration {4,9,10,11} achieves lower scores on both metrics across the three tested generators.

We introduced held-out evaluation to examine whether improvements extend to representation spaces not directly used for supervision. FDr⁶ includes SigLIP, Inception, and MAE, which FD-SIM directly optimizes, while its DINOv2-L evaluator belongs to the same model family as our DINOv2-B supervision encoder. We therefore additionally evaluate both methods with six encoders excluded from their supervision objectives. The four additional encoders are DINOv3-L, PE-L, SwinV2-L, and EVA02-L. They provide complementary coverage of self-supervised and vision-language learning, hierarchical Transformer architectures, and masked image modeling (Table 7). Their selection was guided by this coverage, without using the two methods' comparative scores as a selection criterion. ConvNeXt and CLIP also contribute to FDr⁶ and therefore informed configuration selection through that metric. Results from the four additional encoders were computed after the initial experiments and did not inform supervision-encoder or layer selection.

AuthorsOpenReview ↗

Information-theoretic basis for IEMV-FD

Q1. Information-theoretic basis for IEMV-FD

We sincerely thanks for your insightful suggestion.

We aim to establishes that different representations of the same input can share semantic information while retaining complementary characteristics as CRD [1].

In our method, each supervision view is obtained by combining different network depths and token aggregation strategies (CLS token or average-pooled patch tokens). Although these views are extracted from the same pretrained encoder, they capture different aspects of visual representations due to variations in feature transformation depth and aggregation mechanisms. Thus, using a single encoder does not imply that all views are redundant. Rather, it allows us to exploit the hierarchical and complementary representations already encoded in the pretrained model, without introducing additional independent backbones.

We acknowledge that these views are statistically dependent, as they originate from the same input and share the encoder parameters. However, statistical dependence does not necessarily imply representational redundancy. In information-theoretic terms, the conditional mutual information (I(X;Vi∣Vj)I(X;V_i\mid V_j)) can remain positive even when (I(Vi;Vj)>0I(V_i;V_j)>0), indicating that one view may retain information not fully captured by another.

Importantly, our method does not require statistical independence among views. Instead, each view defines a distinct feature space with its own Fréchet distribution-matching objective. Their joint optimization therefore imposes multiple distributional constraints from different representation perspectives, reducing the reliance on any single feature space.

The effectiveness of this design is further supported by our experiments: IEMV-FD consistently achieves better held-out FDr6 results across all seven evaluated generator configurations, suggesting that its benefits generalize beyond the directly supervised representation spaces.

In summary, our key insight is that multi-view supervision relies on representational complementarity rather than statistical independence. This enables effective distributional supervision using a single pretrained encoder while substantially reducing computational overhead.

We will refine and clarify this motivation and the statistical dependence among views in the revised version.

[1] Tian et al., Contrastive Representation Distillation. ICLR 2020.

Reviewer_tToFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds on FD-loss and proposes Intra-Encoder Multi-View Fréchet Supervision (IEMV-FD) as an efficient alternative to FD-SIM. It constructs eight supervision views from the CLS and average-pooled patch tokens of four layers in a single frozen DINOv2-B encoder. The paper also introduces held-out (FDr^6) to evaluate transfer beyond directly supervised representation spaces. On ImageNet (256\times256), IEMV-FD consistently outperforms FD-SIM across the tested generators.

Critical strengths
  • The proposed method is simple and computationally efficient. One ViT-B forward pass replaces three large encoders, and the savings are measured end to end (peak memory 108 → 55 GiB, 2.15× throughput on JiT-B/16, 1.67× throughput on pMF-B/16).
  • The held-out FDr$^6$ improvement is consistent across all five generators .
Critical weaknesses
  • Limited novelty, and the comparison does not isolate the idea.
    • Matching mean/covariance statistics over many layers of one frozen network is not new. GFMN [1] did exactly this over all layers of a pretrained ConvNet, including moving averages to avoid large batches, but the paper mentions it only in passing (L113). What is left is a new configuration of FD-loss [2].
    • IEMV-FD vs. FD-SIM changes the encoder (DINOv2 vs. SigLIP/Inception/MAE) and the views at the same time. Without a multi-layer SigLIP2/MAE variant or FD-SIM + DINOv2-B, I cannot tell whether the gain comes from the multi-view design or simply from DINOv2.
    • Single-view controls use only the final layer; intermediate-layer single-view baselines are absent. Thus, complementary-view benefits are not clearly distinguished from the contribution of a particularly effective intermediate representation.
    • The "11.5% of parameters" (repeated many times) comes from picking a ViT-B, not from the multi-view design.
  • The evaluation is built around the method.
    • FID and IS, which FD-loss reports, are dropped. Using the Inception column of Table 8 and FD(V,T) = 1.68 from [2], IEMV-FD's FID on iMF-B is ≈4.7, worse than the base model before post-training (≈3.5).
    • On the official FDr$^6$, the gains are marginal (3.53 vs. 3.55 on pMF-B, 2.11 vs. 2.12 on pMF-L). The main claim therefore rests on a metric the authors defined themselves.
    • This metric is not neutral either. Fig. 1 labels DINOv2 a "Non-Judge", but the official FDr$^6$ contains DINOv2 ViT-L/14 (Table B.1 in [2]), from the same distillation family as the training encoder. In Table 8, DINOv2 and CLIP are the only columns where IEMV-FD wins on all five models. The held-out set then adds DINOv3-L.
    • The layer set {4, 9, 10, 11} seems to be selected on JiT-B with the same held-out FDr$^6$ (Table 4). An evenly spaced set {2, 5, 8, 11} is even worse than the last layer alone (11.81 vs. 11.24).
    • There is no human study, although [2] states that FDr$^6$ is not a replacement for human evaluation and runs one.
  • Minor: Table 2 duplicates Table 9; "Strict FDr$^4$" is defined in Appendix B but never reported; "Muti-encoder" in Fig. 1. [1] dos Santos, Cicero Nogueira, et al. "Learning implicit generative models by matching perceptual features." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019. [2] Yang, Jiawei, et al. "Representation Fréchet loss for visual generation." arXiv preprint arXiv:2604.28190 (2026).
Questions for authors
  1. Please report FID/IS for all rows in Table 1. Why does IEMV-FD increase Inception FD over the un-post-trained iMF-B (2.81 vs. 2.07, Table 8)?
  2. To separate the encoder choice from the multi-view design, please add (a) IEMV with multi-layer SigLIP2 or MAE views and (b) FD-SIM with DINOv2-B as a fourth encoder.
  3. How was {4, 9, 10, 11} chosen? A layer rule fixed in advance would be more convincing than the best of five settings on the test metric.
  4. A pairwise human preference study against FD-SIM, as done in FD-loss [1], would matter more to me than another FD-based metric.
  5. Could FD-SIM also benefit from selecting a suitable intermediate layer separately for each encoder, rather than using fixed final-layer representations? If explored, how would this affect its held-out FDr$^6$ and quality–compute trade-off relative to IEMV-FD?  [1] Yang, Jiawei, et al. "Representation Fréchet loss for visual generation." arXiv preprint arXiv:2604.28190 (2026).
6 comments
AuthorsOpenReview ↗

Q1. FID/IS reporting and the iMF-B Inception discrepancy

Thank you for the detailed review and for recognizing the measured reductions in training cost. We have supplemented the representation-based evaluation with FID/IS, CMMD, and a blinded human preference study. Below, we address your questions and clarify the configuration-selection and evaluation protocols.

Q1. FID/IS reporting and the iMF-B Inception discrepancy

Thank you for pointing out this issue. Our careful investigation identified an incorrect VAE checkpoint in iMF-B post-training. We replaced sd-vae-ft-ema with sd-vae-ft-mse, as used in the official iMF implementation, and repeated post-training. The resulting FID is 3.18, improving on the pretrained baseline's 3.34 and resolving the previously reported increase. The Inception component of FDr⁶ also decreases from 2.81 to 1.93, below the baseline value of 2.07. The revised manuscript includes the updated iMF-B results, with FDr⁶ of 4.87 and held-out FDr⁶ of 8.87.

We now report FID and IS for the pretrained baseline, re-evaluated FD-SIM, and IEMV-FD checkpoints across all seven generator configurations covered by the revised main results. The complete results are provided below.

GeneratorBaseline FID ↓Baseline IS ↑FD-SIM FID ↓FD-SIM IS ↑IEMV-FD FID ↓IEMV-FD IS ↑
iMF-B3.34255.070.96306.383.18321.76
iMF-L1.85276.010.87301.701.66318.58
pMF-B3.15253.130.96311.712.93333.19
pMF-L2.53259.420.88307.781.65317.73
JiT-B3.61270.831.13324.412.58273.40
JiT-L2.40287.800.87316.981.71303.72
JiT-H1.90295.950.87312.221.40300.27

Bold indicates the better result between FD-SIM and IEMV-FD for each metric.

The corrected results show that IEMV-FD improves both FID and IS over the corresponding pretrained baseline on all seven generators. For iMF-B, IS increases from 255.07 to 321.76. IEMV-FD also achieves higher IS than FD-SIM on iMF-B/L and pMF-B/L.

Notably, FD-SIM achieves substantially lower FID across all seven generators. However, FD-SIM directly optimizes a Fréchet objective in Inception feature space, which overlaps with the feature space used for FID evaluation. Its exceptionally low FID, reaching 0.87, is even substantially below the FID of 1.68 obtained by ImageNet-1K validation images under the same evaluation protocol. This suggests potential overfitting to Inception feature statistics, meaning that FID may not fully reflect actual perceptual image quality. We therefore complement FID and IS with FDr⁶ and human preference evaluations to provide a more comprehensive assessment.

AuthorsOpenReview ↗

Q2. Separating encoder choice from multi-view supervision

Q2. Separating encoder choice from multi-view supervision

Thank you for suggesting this control. We have added multi-layer SigLIP2 experiments on JiT-B and pMF-B, addressing the SigLIP2 option in part (a). These experiments extend the within-encoder comparison to a supervision backbone outside the DINO family.

New within-SigLIP2 ablations. We vary the feature readouts while keeping the SigLIP2 backbone fixed. AVG denotes average pooling over patch tokens. MAP denotes the final-layer attention-pooled global feature returned as cls_token by the official FD-loss wrapper. Layer indices are zero-based.

MAP layersAVG layersJiT-B FDr⁶ ↓JiT-B held-out FDr⁶ ↓pMF-B FDr⁶ ↓pMF-B held-out FDr⁶ ↓
{26}None12.3317.796.1114.35
None{26}12.9816.757.3712.64
{26}{26}10.4211.666.0611.31
None{10,14,18,22,26}7.439.314.928.97
{26}{10,14,18,22,26}6.127.984.287.81

All values are from our SigLIP2 ablation. Bold marks the lowest value in each metric column. The final row uses one final-layer MAP view and five AVG views, for six views in total. This is an asymmetric readout variant. The default DINOv2-B configuration uses both CLS and AVG at each selected layer.

Two comparisons directly support the multi-view design. First, with AVG readout fixed, replacing final-layer supervision with five-layer supervision improves both metrics: on JiT-B, FDr⁶ decreases from 12.98 to 7.43 and held-out FDr⁶ from 16.75 to 9.31. On pMF-B, the corresponding scores decrease from 7.37 to 4.92 and 12.64 to 8.97. This comparison is independent of the additional MAP readout. Second, adding the final-layer MAP view to the same five AVG views further improves the results: 7.43/9.31 → 6.12/7.98 on JiT-B and 4.92/8.97 → 4.28/7.81 on pMF-B, where each pair denotes original FDr⁶ / held-out FDr⁶.

Complementary DINOv2-B controls. Our existing Table 4(a) holds DINOv2-B fixed and reports:

Layers, zero-indexedToken readoutOriginal FDr⁶ ↓Held-out FDr⁶ ↓
{11}CLS20.4215.01
{11}Average-pooled patch tokens14.0413.85
{11}CLS + average-pooled patch tokens12.0611.24
{4,9,10,11}CLS6.058.30
{4,9,10,11}Average-pooled patch tokens7.379.57
{4,9,10,11}CLS + average-pooled patch tokens5.247.37

Using the selected four layers instead of only the final layer, with both token readouts retained, reduces held-out FDr⁶ from 11.24 to 7.37. At the selected four layers, combining the two readouts also improves on CLS-only (8.30) and average-pooling-only (9.57) supervision. Together, the DINOv2-B and SigLIP2 results show improvements from intra-encoder multi-view supervision across both evaluated encoder families.

Comparison with FD-SIM. The strongest tested SigLIP2 configuration obtains lower held-out FDr⁶ than the re-evaluated FD-SIM checkpoints on JiT-B (7.98 vs. 10.76) and pMF-B (7.81 vs. 8.01). Its original FDr⁶ is higher, with corresponding values of 6.12 vs. 5.66 and 4.28 vs. 3.55, respectively.

AuthorsOpenReview ↗

Q3. Configuration Selection and Held-out Evaluation

Q3. Configuration Selection and Held-out Evaluation

The supervision encoder and layer configuration were selected empirically using FDr⁶. Held-out FDr⁶ was introduced after the initial training experiments were completed and the configurations were fixed. Its scores in Table 4 are retrospective evaluations of these configurations.

Layer exploration was conducted primarily on JiT-B, using both CLS-token and average-pooled patch-token representations throughout. All layer indices below are zero-based.

Layer configurationFDr⁶ ↓
Last layer {11}12.06
Last two layers {10,11}9.22
Last four layers {8,9,10,11}9.17
Last six layers {6,7,8,9,10,11}9.56
All 12 layers10.11
Evenly spaced {2,5,8,11}8.32
{5,8,10,11}5.43
{4,9,10,11}5.24

These comparisons informed three empirical judgments:

  1. Expanding the range of consecutive layers does not improve FDr⁶ monotonically: the last-two-layer and last-four-layer scores are close, while six or all twelve layers yield higher scores.
  2. Layer placement matters at a fixed four-layer count. Replacing layer 8 with layer 4 in the last-four-layer configuration reduces FDr⁶ from 9.17 to 5.24. Replacing layer 2 with layer 10 in the evenly spaced configuration reduces it from 8.32 to 5.43.
  3. The two strongest tested configurations combine an intermediate-depth layer with deeper layers. We chose {4,9,10,11} as the default based on its lowest observed FDr⁶, with {5,8,10,11} obtaining a close result.

On iMF-B and pMF-B, we also compared {4,9,10,11} with {5,8,10,11}. Both CLS-token and average-pooled patch-token representations are used for all rows below. JiT-B is included for reference.

GeneratorLayers (zero-indexed)FDr⁶ ↓Held-out FDr⁶ ↓
iMF-B{4,9,10,11}4.878.87
iMF-B{5,8,10,11}5.129.01
pMF-B{4,9,10,11}3.536.65
pMF-B{5,8,10,11}3.556.72
JiT-B{4,9,10,11}5.247.37
JiT-B{5,8,10,11}5.437.50

The two configurations obtain numerically close results on iMF-B and pMF-B, supporting their applicability beyond JiT-B. The default configuration {4,9,10,11} achieves lower scores on both metrics across the three tested generators.

We introduced held-out evaluation to examine whether improvements extend to representation spaces not directly used for supervision. FDr⁶ includes SigLIP, Inception, and MAE, which FD-SIM directly optimizes, while its DINOv2-L evaluator belongs to the same model family as our DINOv2-B supervision encoder. We therefore additionally evaluate both methods with six encoders excluded from their supervision objectives. The four additional encoders are DINOv3-L, PE-L, SwinV2-L, and EVA02-L. They provide complementary coverage of self-supervised and vision-language learning, hierarchical Transformer architectures, and masked image modeling (Table 7). Their selection was guided by this coverage, without using the two methods' comparative scores as a selection criterion.

ConvNeXt and CLIP also contribute to FDr⁶ and therefore informed configuration selection through that metric. Results from the four additional encoders were computed after the initial experiments and did not inform supervision-encoder or layer selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.