acceptodds
Under review as a conference paper at ICLR 2027

See More from Less: Sample-Efficient Generative Feedback for Fine-Grained Visual Understanding

Abstract

Generative feedback can improve the fine-grained visual understanding of Contrastive Language-Image Pre-training (CLIP), but existing methods require substantial data and computation. We identify two sources of sample inefficiency: first, projected representations can collapse toward a shared constant, weakening gradient feedback to CLIP; second, hierarchical information is underutilized, as supervision from only one or a few levels limits feedback per sample, while directly combining multiple levels can introduce conflicting gradients. We propose Sample-efficient Generative Feedback (SGF) to extract more useful supervision from each training sample. Margin regularization encourages matched conditions to yield lower reconstruction losses than mismatched ones by a positive margin, preserving image-specific information. Hierarchical boosting progressively integrates coarse-to-fine supervision while retaining earlier objectives, giving them time to stabilize before finer supervision is introduced, thereby reducing cross-level interference. Experiments across six CLIP backbones show that SGF achieves state-of-the-art fine-grained perception using only 84K training samples, at least 82% fewer than prior methods, with a - training speedup over the fastest competitor, demonstrating its improved sample efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generation model for understanding model, CLIP, Visual Autoregressive model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MCjhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SGF, which leverages feedback from a pre-trained generative model to improve the fine-grained recognition capability of visual encoders. The method consists of two main components: margin regularization to mitigate projected representation collapse, and progressive multi-scale training to gradually incorporate finer-grained generative supervision. The authors evaluate the method across six visual encoder configurations and report downstream results on large multi-modal models. SGF achieves performance comparable to or better than existing methods while using fewer training sample passes. However, the current experiments do not fully validate the sample efficiency advantage, nor do they isolate the contribution of the progressive schedule.

Critical strengths
  • The research problem has practical value. While generative feedback can improve visual encoders, high training costs typically limit its usage. This paper directly addresses how to reduce this cost.

  • Component designs are supported by experiments. The authors examine the differences in projected representations and backpropagation gradients. Ablation results also demonstrate that both components improve fine-grained recognition performance.

  • Experiments cover multiple model configurations. The paper tests three visual encoder families and provides multi-modal downstream evaluations along with diffusion model scaling. These results support the broad applicability of the method.

Critical weaknesses
  • It has not been evaluated whether the enhancement in fine-grained perception comes at the cost of degraded original capabilities. When updating the visual encoder, SGF uses only the generative reconstruction objective without directly reporting whether classification and image-text retrieval capabilities are maintained. This affects the comparison with DCR, because DCR focuses on both detail perception and discriminative capability. Existing MMVP-VLM and multi-modal QA results provide positive evidence, but are insufficient to determine this capability trade-off. Therefore, it remains unclear whether SGF can preserve an equally complete visual representation capability at a lower cost.

  • The explanation for the inefficiency of existing methods exceeds the current diagnostic evidence. The paper attributes projected representation collapse and insufficient hierarchical supervision as the efficiency bottlenecks of existing generative feedback methods, but the related diagnoses are primarily based on the training setup of this paper. There are significant differences among baselines: un²CLIP does not require an additional alignment projector, and DCR has already introduced conditional contrastive signals. The paper has not directly verified whether these baselines suffer from the same issues, nor whether these issues lead to their higher training costs. Consequently, the current results more directly support the effectiveness of the proposed training scheme, while the explanation for the efficiency bottlenecks of existing methods still requires further evidence.

  • The ablation study fails to disentangle the contributions of multi-scale supervision and progressive scheduling. Table 4 compares single-scale supervision with progressive multi-scale supervision while simultaneously altering the number of supervision scales and their inclusion order. Therefore, it cannot be determined whether the performance improvement stems from supervision at more scales or from the progressive schedule. Although Figure 4 compares progressive training with direct joint training, it only reports the coarse-scale reconstruction loss, which cannot be used to evaluate the final fine-grained recognition performance. To confirm the independent contribution of the progressive schedule, downstream results under a fixed set of scales and other identical training settings need to be compared between the two approaches.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to MCjh 1/2

Thank you for your thoughtful feedback and valuable recommendations. We have carefully revisited the points raised in your review and address them below. We hope the additional evidence and explanations help resolve your concerns.

Weakness 1: Evaluation of CLIP’s Original Capabilities

Following your suggestion, we evaluated zero-shot classification and retrieval performance across multiple CLIP backbones. The tables below report Recall@5 for retrieval and classification accuracy (ACC) for zero-shot classification. In the two summary columns, bold indicates the best result among enhanced methods, and a superscript † marks the second-best result. We evaluated all baselines using their publicly released checkpoints. Notably, our method achieves the best performance on fine-grained tasks while largely preserving CLIP’s original capabilities. We clarify that DCR targets the discriminability of CLIP’s visual features, which differs from the image–text discrimination required for zero-shot classification.

Table: Evaluation of Zero-Shot Classification and Retrieval on OpenAI ViT-L/14@224.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original19.276.787.161.197.379.495.675.875.686.731.9
DIVA25.976.787.061.097.279.795.576.375.586.831.7
GenHancer31.858.981.651.087.661.677.544.240.278.97.1
un2CLIP32.6$^{\dagger}$72.890.265.596.777.589.065.662.486.821.9
DCR33.359.982.252.188.963.578.745.341.479.57.4
Ours33.375.2$^{\dagger}$89.664.797.175.494.475.169.880.830.1

Table: Evaluation of Zero-Shot Classification and Retrieval on MetaCLIP ViT-L/14@224.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original23.782.793.671.498.785.097.684.179.289.945.2
DIVA27.482.793.771.498.685.197.583.979.089.845.2
GenHancer31.863.482.855.586.164.286.560.642.981.211.0
DCR32.6$^{\dagger}$63.482.755.486.264.186.560.842.981.311.0
Ours34.881.7$^{\dagger}$92.769.797.881.397.583.877.690.344.8

Table: Evaluation of Zero-Shot Classification and Retrieval on SigLIP ViT-SO-14@384.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original37.086.196.176.999.789.996.783.483.188.860.6
DIVA38.586.196.276.899.790.096.883.083.188.960.4
GenHancer40.084.996.878.899.790.593.675.880.889.258.5
un2CLIP41.585.5$^{\dagger}$96.578.799.990.495.679.081.189.359.3
DCR42.2$^{\dagger}$84.996.878.899.790.693.675.880.989.258.9
Ours43.086.196.377.399.790.196.582.982.988.960.6
AuthorsOpenReview ↗

Response to MCjh 2/2

Weakness 2: Further Explanation of the Efficiency Bottlenecks in Existing Methods

Thank you for your comments. We examined whether the two phenomena discussed in our paper occur in existing methods:

  • Representation collapse. We evaluated DIVA, GenHancer, and DCR, all of which train a projection layer. As shown below, each method exhibited signs of representation collapse as the number of processed training samples increased from 50K to 200K. We excluded un2CLIP from this analysis because it does not train a projection layer. Furthermore, reconstruction in all three methods was insensitive to mismatched conditions (see Appendix F.4 for the visualizations), indicating insufficient use of image-specific information in the conditions and a strong reliance on intermediate visual inputs.

    Table: Representation Collapse Across Different Methods

    DIVAGenHancerDCR
    50k0.720.560.50
    200k0.200.280.08
  • Underutilized hierarchical information. This issue is common in previous methods, which use only one or two denoising steps, leaving substantial supervisory information unused. We further verified that directly combining supervision from multiple timesteps leads to gradient conflicts.

As analyzed in our paper, these phenomena hinder effective gradient propagation and reduce sample efficiency, increasing training costs. We additionally incorporated SGF into these methods. As shown in the table below, SGF yields substantial improvements over their original formulations with a sample budget of only 84K. These results indicate that addressing the above issues with SGF helps unlock the potential of these methods.

Table: Performance Improvements from Incorporating SGF into Existing Methods Under an 84K Sample Budget

DIVAGenHancerun2CLIPDCR
w.o. SGF23.723.024.420.7
w. SGF25.228.228.925.2

Weakness 3: More Comprehensive Ablation Studies on Hierarchical Boosting

Thank you for this valuable suggestion. To isolate the contribution of progressive scheduling in Hierarchical Boosting (HB), we added a comparison with direct multi-scale supervision (MS) under the same experimental settings. As shown in the table below, directly applying multi-scale supervision yields limited performance gains while incurring additional computational overhead. Compared with direct multi-scale supervision, Hierarchical Boosting achieves better MMVP performance with less GPU time.

Table: More Comprehensive Ablation Results on Hierarchical Boosting. MS denotes multi-scale supervision, and HB denotes Hierarchical Boosting.

MRMSHBMMVP ScoreGPU Mem.GPU Hours
---22.218.5 GB3.3 h
√--25.220.0 GB4.7 h
-√-24.420.1 GB4.6 h
√√-26.720.1 GB6.7 h
--√28.120.1 GB3.4 h
√-√31.920.2 GB4.1 h

Thank you for the response and additional analyses, which address most of my concerns. The classification and retrieval results clarify the capability trade-offs, and the baseline diagnostics and direct multi-scale comparisons provide the evidence I was looking for. I still think the explanation of why existing methods are inefficient should be stated more cautiously. Overall, the results support the practical benefit of SGF, and I will raise my rating.

We sincerely appreciate your kind words and the improved score. We are pleased that our responses have addressed most of your concerns. Your constructive comments have greatly helped us enhance the soundness of our paper.

Following your suggestion, we have revised the manuscript to describe the causes of existing methods’ inefficiency more precisely and cautiously. We have also provided empirical analyses of representation collapse and gradient conflicts in Appendix F.4 to further support our analysis.

We will ensure that these revisions are properly incorporated into the final version.

Reviewer_BJtYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Sample-efficient Generative Feedback (SGF) to improve CLIP’s fine-grained visual understanding. It identifies two potential limitations of generative-feedback training: collapsed projector outputs and insufficient utilization of hierarchical supervision. SGF introduces a matched-versus-mismatched reconstruction margin during projector training and progressively activates coarse-to-fine generation levels during training. The main implementation uses a frozen Infinity-2B generator and a two-stage procedure comprising projector alignment and LoRA-based encoder refinement. Experiments cover six backbone configurations and a diffusion-based extension. The method processes 84K training samples and reports competitive or better performance with substantially lower adaptation costs.

Critical strengths
  1. Practical efficiency. SGF requires approximately 3.9–4.5 GPU hours for its main configurations. Achieving competitive fine-grained perception within this budget is useful, particularly for researchers with limited computational resources.
  2. Clear motivation. The proposed components address concrete issues in generative feedback. Comparing matched and shuffled conditions encourages the generator to use image-specific information, while progressive supervision provides a reasonable approach to integrating multiple generation levels.
  3. Meaningful ablations. Table 4 shows substantial improvements over the method’s own baseline: margin regularization increases MMVP-VLM performance from 22.2 to 25.2, hierarchical boosting reaches 28.1, and their combination reaches 31.9. These results provide useful evidence that both components contribute.
  4. Multiple implementations. Evaluation spans OpenAI CLIP, MetaCLIP, and SigLIP configurations. The diffusion extension and downstream MLLM experiments provide additional evidence that the approach is applicable beyond a single encoder–generator combination.
Critical weaknesses
  1. Incomplete controls. The main hierarchical-boosting ablation compares progressive multi-level supervision against single-level supervision, changing both the amount of supervision per image and its activation schedule. Figure 4 compares coarse-scale reconstruction losses against direct joint training, but does not report their corresponding MMVP-VLM performance. Consequently, the benefit of progressive activation remains insufficiently separated from the benefit of simply using multiple levels. Comparisons with all-level joint training, uniform activation intervals, and alternative scale orders would be informative. The effects of restricting supervision to seven scales and disabling margin regularization after scale five should also be separated from the proposed scheduling mechanism.
  2. Limited validation. The main fine-grained evaluation relies on MMVP-VLM, where SGF ties DCR on two backbone configurations and improves by only approximately 0.7–0.8 points on several others. Given the coarse category-level score increments, these differences are small and require uncertainty estimates. No training-seed variability or confidence intervals are reported. Downstream MLLM results are competitive, but several improvements over the strongest baseline are also marginal. Furthermore, the evaluation does not establish whether refinement preserves CLIP’s broader capabilities, such as zero-shot classification and image–text retrieval. Additional fine-grained benchmarks and general-capability evaluations would make the conclusions more convincing.
  3. Limited theoretical support. The propositions provide useful explanatory bounds, but their connection to the implemented algorithm is limited. Proposition 1 assumes a common locally affine projector, whereas the actual projector contains GELU and LayerNorm. It bounds a covariance-weighted feature-gradient quantity rather than the ordinary gradient norm, and excluding constant conditions does not guarantee informative representations or useful encoder updates. Proposition 2 establishes reduced marginal interference near stationarity, but the activation schedule is based on estimated information gain and manual allocation rather than measured stationarity. The appendix acknowledges these limitations; stronger empirical verification is needed to connect the bounds to the claimed training mechanism.
Questions for authors
  1. Efficiency: How do the strongest baselines perform when restricted to the same 84K processed samples or approximately four GPU hours? Can you provide performance-versus-budget curves, profiling fit errors, and clearly distinguish measured from extrapolated costs?
  2. Scheduling: What is the MMVP-VLM performance of direct joint supervision over scales 1–7 under matched sample and compute budgets? How sensitive are the results to uniform schedules, alternative activation orders, and the chosen scale cutoff?
  3. Robustness: Can you report results across multiple training seeds and paired confidence intervals? Do the enhanced encoders preserve zero-shot classification and retrieval performance, and do the improvements transfer to additional fine-grained benchmarks?
  4. Mechanism: Does hierarchical boosting measurably reduce gradient conflicts relative to joint training throughout optimization? Are earlier objectives close to stationarity when new scales are activated, and does margin regularization improve representation diversity beyond increasing the RMS radius?
2 comments
AuthorsOpenReview ↗

Response to BJtY 1/2

We appreciate the time and effort you devoted to reviewing our work and the helpful suggestions for strengthening the paper. Below, we respond to your comments in detail, organizing closely related concerns together for readability. We hope our responses clarify the issues you raised.

Weakness 1 & Question 2: More Comprehensive Ablation Studies on Hierarchical Boosting

To isolate the contribution of progressive scheduling in Hierarchical Boosting, we added a comparison with direct multi-scale supervision under the same experimental settings. As shown in the table below, directly applying multi-scale supervision yields limited performance gains while incurring additional computational overhead. Compared with direct multi-scale supervision, Hierarchical Boosting achieves better MMVP performance with less GPU time.

Table: More Comprehensive Ablation Results on Hierarchical Boosting. MR denotes Margin Regularization, MS denotes multi-scale supervision, and HB denotes Hierarchical Boosting.

MRMSHBMMVP ScoreGPU Mem.GPU Hours
---22.218.5 GB3.3 h
√--25.220.0 GB4.7 h
-√-24.420.1 GB4.6 h
√√-26.720.1 GB6.7 h
--√28.120.1 GB3.4 h
√-√31.920.2 GB4.1 h

We also evaluated several alternative schedules: uniformly spaced scale activation, reverse scheduling, and perturbing the scale activation points of the original schedule by ±10%. The results of uniformly spaced scale activation and reverse scheduling indicate that the coarse-to-fine order is necessary and that the allocation of training intervals affects performance. Meanwhile, the perturbed schedule shows that, as long as this allocation broadly follows the principles described in the paper, small perturbations have little effect on performance, suggesting robustness to variations in the schedule.

OriginalHB1: UniformHB2: ReverseHB3: PerturbedHB
MMVP Score20.025.921.529.631.9

Weakness 2 & Question 3: Additional Evaluation Benchmarks

We first evaluated zero-shot classification and retrieval performance across multiple CLIP backbones. The tables below report classification accuracy (ACC) for zero-shot classification and Recall@5 for retrieval. In the two summary columns, bold indicates the best result among enhanced methods, and a superscript † marks the second-best result. We evaluated all baselines using their publicly released checkpoints. Notably, our method achieves the best performance on fine-grained tasks while largely preserving CLIP’s original capabilities.

Table: Evaluation of Zero-Shot Classification and Retrieval on OpenAI ViT-L/14@224.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original19.276.787.161.197.379.495.675.875.686.731.9
DIVA25.976.787.061.097.279.795.576.375.586.831.7
GenHancer31.858.981.651.087.661.677.544.240.278.97.1
un2CLIP32.6$^{\dagger}$72.890.265.596.777.589.065.662.486.821.9
DCR33.359.982.252.188.963.578.745.341.479.57.4
Ours33.375.2$^{\dagger}$89.664.797.175.494.475.169.880.830.1

Table: Evaluation of Zero-Shot Classification and Retrieval on MetaCLIP ViT-L/14@224.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original23.782.793.671.498.785.097.684.179.289.945.2
DIVA27.482.793.771.498.685.197.583.979.089.845.2
GenHancer31.863.482.855.586.164.286.560.642.981.211.0
DCR32.6$^{\dagger}$63.482.755.486.264.186.560.842.981.311.0
Ours34.881.7$^{\dagger}$92.769.797.881.397.583.877.690.344.8

Table: Evaluation of Zero-Shot Classification and Retrieval on SigLIP ViT-SO/14@384.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original37.086.196.176.999.789.996.783.483.188.860.6
DIVA38.586.196.276.899.790.096.883.083.188.960.4
GenHancer40.084.996.878.899.790.593.675.880.889.258.5
un2CLIP41.585.5$^{\dagger}$96.578.799.990.495.679.081.189.359.3
DCR42.2$^{\dagger}$84.996.878.899.790.693.675.880.989.258.9
Ours43.086.196.377.399.790.196.582.982.988.960.6
AuthorsOpenReview ↗

Response to BJtY 2/2

Weakness 2 & Question 3: Additional Evaluation Benchmarks (Continued)

Second, we added evaluations on two fine-grained benchmarks, SugarCrepe and CountBench, using the OpenAI ViT-L/14@224. The results show that the improvements from generative feedback transfer to other fine-grained evaluation benchmarks. SugarCrepe evaluates compositional understanding; we report subtask, macro-average, and overall accuracy. CountBench evaluates object counting; we report accuracy and mean absolute error.

Table: Results on SugarCrepe

MethodReplace ObjectReplace AttributeReplace RelationSwap ObjectSwap AttributeAdd ObjectAdd AttributeMacro Avg.Overall Acc.
Original94.0179.0665.0861.2261.8678.3771.2472.9876.71
SGF94.0778.8168.9960.8264.7183.8579.3475.8079.92

Table: Results on CountBench

MethodACC (%) ↑MAE ↓
Original29.941.69
SGF31.161.64

Finally, to assess the stability of the performance gains, we conducted experiments with three different random seeds on each of the three backbones. We report the mean and 95% confidence interval (CI), calculated using the t-distribution across the three runs. The results, presented in the table below, show that SGF consistently achieves state-of-the-art performance under a limited sample budget.

Table: Performance Across Different Seeds

MMVP Avg.95% CI
OpenAI ViT-L@224Original19.2-
OpenAI ViT-L@224SGF32.8[30.8, 34.8]
OpenAI ViT-L@336Original20.0-
OpenAI ViT-L@336SGF31.4[30.2, 32.5]
MetaCLIP ViT-L@224Original23.7-
MetaCLIP ViT-L@224SGF35.3[31.5, 39.1]

Question 1: Comparative Experiments Under a Fixed Data Budget

We fixed the training sample budget at 84K and adapted and tuned each baseline for this limited budget, for example, by moderately increasing the learning rate and reducing the batch size. The table below reports average MMVP scores. Even after these adjustments, the baselines show substantially smaller gains than SGF under the same sample budget.

Table: Comparison of Average MMVP Scores Under a Fixed 84K Training Sample Budget

BackboneOriginalDIVAGenHancerun2CLIPDCRSGF
OpenAI ViT-L/14@22419.223.723.024.420.733.3
OpenAI ViT-L/14@33620.018.521.521.522.231.9
MetaCLIP ViT-L/14@22423.724.428.9-26.734.8
MetaCLIP ViT-H/14@22425.226.728.2-28.937.8
SigLIP ViT-SO/14@22437.838.539.3-38.543.7
SigLIP ViT-SO/14@38437.037.041.5-40.743.0

Additionally, taking GenHancer as an example, we provide a curve showing how performance varies with the sample budget. The underlying data are presented below, and the curve is included in Appendix F.2.

50K100K150K200K
MMVP Score22.223.023.024.4

Finally, to eliminate errors from extrapolated estimates, we directly measured efficiency on multiple RTX 4090 GPUs and updated Table 1 in the paper.

Weakness 3 & Question 4: Further Clarification of the Connection Between the Theory and the Algorithm

For Margin Regularization, the original Proposition 1 establishes, under a local affine assumption, that representation collapse limits effective gradient propagation. Here, GeffG_{\mathrm{eff}} measures the gradient magnitude along directions in which image features actually vary within the current batch.

  • To address your concern that the practical projector includes GELU and LayerNorm, we extended Proposition 1 to differentiable nonlinear projectors. We introduce a within-batch first-order linearization error term, ηP\eta_P, to capture the discrepancy between the local linear approximation and the actual changes in projected representations, yielding the new upper bound Geff≤(ε+ηP)GcG_{\mathrm{eff}}\leq(\varepsilon+\eta_P)G_c. This bound likewise shows how representation collapse constrains effective gradient propagation. The definitions and proof are given in Appendix C.1.

  • To assess whether Margin Regularization improves representation diversity, we report two metrics below: 5-NN cosine distance ↑, the average cosine distance from each image representation to its 5 nearest neighbors; and Vendi Score ↑, which measures global diversity using the spectral entropy of the cosine similarity matrix, retaining the shared mean component.

    Vendi Score ↑5-NN Distance ↑
    w.o. MR1.3020.014
    w. MR1.5540.023

For Hierarchical Boosting, we measured ∣Is∣|I_s| from Proposition 2 to assess whether Hierarchical Boosting reduces interference. Specifically, we selected multiple checkpoints from training with Hierarchical Boosting and with direct joint multi-scale optimization and computed ∣Is∣|I_s| at each checkpoint. The results below show that Hierarchical Boosting measurably reduces this interference.

w.o. HBw. HB
ckpt24002.23×10−52.23 \times 10^{-5}1.38×10−51.38 \times 10^{-5}
ckpt25002.45×10−52.45 \times 10^{-5}1.43×10−51.43 \times 10^{-5}
ckpt26002.52×10−52.52 \times 10^{-5}1.42×10−51.42 \times 10^{-5}
Reviewer_bRc5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that existing approaches that use generative feedback to enhance CLIP's fine-grained perception (DIVA, GenHancer, un2CLIP, DCR) are too expensive to train. It identifies two sources of sample inefficiency: (1) Representation collapse of the projector. A pretrained generator can reconstruct images from its own prior and the intermediate visual inputs. The projected conditions therefore drift toward a shared constant, and the gradients sent back to CLIP become weak. (2)Insufficient use of hierarchical information. Each update supervises only one or a few generation levels. Combining multiple levels directly leads to gradient conflicts.

The proposed method has two components to address these: (1)Margin Regularization (MR): conditions are shuffled within a batch, and the reconstruction loss under matched conditions must be lower than under mismatched conditions by at least a margin. (2) Hierarchical Boosting (HB): supervision levels are activated progressively in a coarse-to-fine manner, and previously activated levels are retained.

The main implementation builds on a frozen Infinity-2B (VAR architecture). It uses 84K CC3M samples in total across two stages and trains in 3.9–4.5 hours on a single RTX 4090. The authors report state-of-the-art or on-par results on MMVP-VLM across six CLIP backbones. They give two propositions (Proposition 1 and 2) as theoretical support and also extend the method to Hyper-SD.

Critical strengths
  1. Generative-feedback methods for enhancing CLIP are generally costly. Cutting training to about 4 hours on a single GPU with 84K samples is useful to the community and makes the work easy to reproduce.

  2. The diagnosis has empirical support. Reconstructions barely change when conditions are swapped (Fig. 3a), the condition radius is small, and the gradients are weak. Together these show fairly directly that projector collapse is real. MR is also simple to implement and cheap to run.

  3. The experiments cover six backbones across three model families. The paper provides implementations with both VAR and diffusion generators, releases code, and documents reproduction in detail.

Critical weaknesses
  1. The efficiency comparison is partly confounded: Training budgets for baselines are not aligned. Baselines are trained with their official configurations (467K–3.84M samples), which were never tuned for small budgets. A key control is missing: how do the baselines perform under the same 84K-sample budget? GPU hours are extrapolated. Baseline training times were measured on a 4090 with small batch sizes and then extrapolated linearly or quadratically. This makes the precision of the claimed "6.0–7.9× speedup" questionable.
  2. Under the local affine assumption, Proposition 1 follows directly from the Cauchy-Schwarz inequality. It states that gradients vanish under complete collapse and that the conditions cannot be constant once the margin is satisfied, which is close to tautological. The appendix itself admits that the proposition does not guarantee a positive gradient magnitude. Proposition 2 is a generic smoothness and first-order expansion bound that holds for any pair of losses. It neither shows that the HB schedule reaches a near-stationary point nor distinguishes HB from other multi-task methods. Presenting these two propositions as "theoretical support" overstates what they show.
  3. Novelty is limited. MR is essentially an in-batch negative contrast with a hinge constraint, which is very close in spirit to DCR's contrastive reconstruction. HB is a coarse-to-fine curriculum. The combination works well as engineering but offers limited methodological insight. In addition, the collapse may stem partly from the design choice of repeating the [CLS] token L times as the condition. Would the problem still arise if dense tokens were used as conditions, as in DIVA?
Questions for authors
  1. With the same generator (e.g., Infinity-2B for all methods) and the same 84K-sample budget, how do GenHancer- or DCR-style baselines perform? This is the only clean experiment that can separate efficiency gained from the method from efficiency gained from a stronger generator.
  2. Does the enhanced CLIP lose performance on ImageNet zero-shot classification and on retrieval tasks?
3 comments
AuthorsOpenReview ↗

Response to bRc5 1/3

We thank you for your careful review and constructive suggestions for improving our paper. We have considered each of your comments and provide point-by-point responses below, grouping related questions for clarity. We hope these explanations and additional results address your concerns.

Question 1 & Weakness 1: More Rigorous Controlled Experiments

Thank you for your suggestion. We conducted more rigorous controlled experiments. First, we fixed the training sample budget at 84K and tuned each baseline for this limited budget, for example, by moderately increasing the learning rate and reducing the batch size. The table below reports average MMVP scores. Even after these adjustments, the baselines show substantially smaller gains than SGF under the same sample budget.

Table: Comparison of Average MMVP Scores Under a Fixed 84K Training Sample Budget

BackboneOriginalDIVAGenHancerun2CLIPDCRSGF
OpenAI ViT-L/14@22419.223.723.024.420.733.3
OpenAI ViT-L/14@33620.018.521.521.522.231.9
MetaCLIP ViT-L/14@22423.724.428.9-26.734.8
MetaCLIP ViT-H/14@22425.226.728.2-28.937.8
SigLIP ViT-SO/14@22437.838.539.3-38.543.7
SigLIP ViT-SO/14@38437.037.041.5-40.743.0

Second, we adapted previous methods to use Infinity. For DIVA and DCR, we replaced the pretrained denoising model with Infinity. For GenHancer, we replaced the small denoiser originally trained in Stage 1 with a frozen Infinity model. We could not make this adaptation for un2CLIP because it relies on a specific generator. We likewise tuned these variants for the limited sample budget and trained them on 84K samples. As shown in the table below, simply replacing the generator with Infinity does not yield clear gains in fine-grained performance.

Table: Average MMVP Scores with a shared Infinity generator under an 84K sample budget.

BackboneOriginalDIVAGenHancerDCRSGF
OpenAI ViT-L/14@22419.222.222.221.533.3
OpenAI ViT-L/14@33620.020.023.720.731.9

Additionally, to address your concerns about the extrapolated efficiency estimates, we directly measured efficiency on multiple RTX 4090 GPUs and updated Table 1 in the paper; the measurement protocol is described in Appendix E.3. The measurements closely match the extrapolated estimates, and our conclusions remain unchanged. Specifically, we followed the configurations reported in the respective baseline papers. When a single GPU had insufficient memory, we used parallel execution across multiple GPUs. When reporting memory usage, we excluded duplicate copies of model parameters across GPUs.

Question 2: Performance of the Enhanced CLIP on Zero-Shot Classification and Retrieval Tasks

Following your suggestion, we evaluated zero-shot classification and retrieval performance across multiple CLIP backbones. The tables below report Recall@5 for retrieval and classification accuracy (ACC) for zero-shot classification. In the two summary columns, bold indicates the best result among enhanced methods, and a superscript † marks the second-best result. We evaluated all baselines using their publicly released checkpoints. Notably, our method achieves the best performance on fine-grained tasks while largely preserving CLIP’s original capabilities.

Table: Evaluation of Zero-Shot Classification and Retrieval on OpenAI ViT-L/14@224.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original19.276.787.161.197.379.495.675.875.686.731.9
DIVA25.976.787.061.097.279.795.576.375.586.831.7
GenHancer31.858.981.651.087.661.677.544.240.278.97.1
un2CLIP32.6$^{\dagger}$72.890.265.596.777.589.065.662.486.821.9
DCR33.359.982.252.188.963.578.745.341.479.57.4
Ours33.375.2$^{\dagger}$89.664.797.175.494.475.169.880.830.1

Table: Evaluation of Zero-Shot Classification and Retrieval on MetaCLIP ViT-L/14@224.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original23.782.793.671.498.785.097.684.179.289.945.2
DIVA27.482.793.771.498.685.197.583.979.089.845.2
GenHancer31.863.482.855.586.164.286.560.642.981.211.0
DCR32.6$^{\dagger}$63.482.755.486.264.186.560.842.981.311.0
Ours34.881.7$^{\dagger}$92.769.797.881.397.583.877.690.344.8
AuthorsOpenReview ↗

Response to bRc5 2/3

Question 2: Performance of the Enhanced CLIP on Zero-Shot Classification and Retrieval Tasks(Continued)

Table: Evaluation of Zero-Shot Classification and Retrieval on SigLIP ViT-SO/14@384.

MethodMMVP Avg.Original Capabilities Avg.Text2Image R@5Image2Text R@5Zero-shot Classification
Flickr30kCOCOFlickr30kCOCOCIFAR10CIFAR100ImageNet1kCaltech101Aircraft
Original37.086.196.176.999.789.996.783.483.188.860.6
DIVA38.586.196.276.899.790.096.883.083.188.960.4
GenHancer40.084.996.878.899.790.593.675.880.889.258.5
un2CLIP41.585.5$^{\dagger}$96.578.799.990.495.679.081.189.359.3
DCR42.2$^{\dagger}$84.996.878.899.790.693.675.880.989.258.9
Ours43.086.196.377.399.790.196.582.982.988.960.6

Weakness 2-1: Supplement to Proposition 1

The original Proposition 1 aims to explain why representation collapse limits generative feedback: collapse of the projected representations constrains the upper bound on the effective gradient, thereby affecting subsequent optimization. Margin Regularization can increase this upper bound, alleviating the restriction that collapse imposes on effective gradient propagation.

Moreover, we derived a lower bound on the effective gradient and provided experimental validation to address your concern that the original proposition does not guarantee a strictly positive effective gradient. The bound is strictly positive when δ>0\delta>0 and the path amplification condition below holds. Let BB denote the batch size, π\pi a batch permutation, and δ=Lshuf−Lrec\delta=\mathcal L_{\mathrm{shuf}}-\mathcal L_{\mathrm{rec}}. Along the feature path vb(t)=(1−t)vb+tvπ(b)v_b(t)=(1-t)v_b+t v_{\pi(b)}, define hb(t)=∇vℓb(Pω(v))∣v=vb(t),Geff(t)=1B∑bhb(t)⊤Σvhb(t),h_b(t)=\left.\nabla_v\ell_b(P_\omega(v))\right|_{v=v_b(t)}, G_{\mathrm{eff}}(t)= \sqrt{\frac1B\sum_b h_b(t)^\top\Sigma_vh_b(t)}, where the original feature covariance Σv\Sigma_v remains fixed. The fundamental theorem of calculus and Cauchy–Schwarz yield [δ]+2B≤∫01Geff(t),dt.\frac{[\delta]_+}{\sqrt{2B}} \le \int_0^1G_{\mathrm{eff}}(t)\\,dt. If the path amplification satisfies sup⁡t∈[0,1]Geff(t)≤CampGeff(0),\sup_{t\in[0,1]}G_{\mathrm{eff}}(t) \le C_{\mathrm{amp}}G_{\mathrm{eff}}(0), then [δ]+Camp2B≤Geff(0)\frac{[\delta]_+}{C_{\mathrm{amp}}\sqrt{2B}} \le G_{\mathrm{eff}}(0). The full derivation is provided in Appendix C.1.

In practice, we computed CampC_{\mathrm{amp}} over 256 samples, and all observed values were at most 4. We therefore set Camp=4C_{\mathrm{amp}}=4 to estimate the lower bound. With Margin Regularization, the estimated lower bound increased to approximately 2.2×2.2\times its original value, while the corresponding effective gradient increased to approximately 2.0×2.0\times its original value.

Estimated lower boundMeasured GeffG_{eff}
w.o. MR0.533×10−30.533 \times 10^{-3}3.105×10−33.105 \times 10^{-3}
w. MR1.178×10−31.178 \times 10^{-3}6.222×10−36.222 \times 10^{-3}

Weakness 3-1: Clarification of the Differences Between Margin Regularization and DCR's Loss

The contrastive losses of Margin Regularization and DCR may appear similar, but they actually differ in their objectives, implementations, and outcomes.

  • Objectives: Margin Regularization aims to mitigate collapse of the projected conditions and improve gradient propagation during subsequent optimization. DCR primarily targets CLIP’s visual features to enhance their discriminability.

  • Implementations: For a fixed reconstruction input, let aa, nn, and yy denote the matched prediction, mismatched prediction, and ground-truth target, respectively. DCR uses aa as the anchor, treats the augmented-view prediction and yy as positives, and scores the negative by cos⁡(a,n)\cos(a,n). Margin Regularization compares both predictions against the same target and penalizes an insufficient reconstruction loss gap via RMR=[m−Δ]+R_{\rm MR}=[m-\Delta]_+, where Δ=Lshuf−Lmatch\Delta=L_{\rm shuf}-L_{\rm match}. For batch-averaged losses, when the margin is unmet (Δ<m\Delta < m) and the other predictions are held fixed, the direct gradient norms with respect to nn are

    ∣∇nLDCR∣=p−Bτ∣n∣sin⁡θ,∣∇nRMR∣=1B∣∇nd(n,y)∣,\\|\nabla_n L_{\rm DCR}\\| =\frac{p_-}{B\tau\\|n\\|}\sin\theta, \qquad \\|\nabla_n R_{\rm MR}\\| =\frac{1}{B}\\|\nabla_n d(n,y)\\| ,

    where θ=∠(a,n)\theta=\angle(a,n), p−p_- is the negative’s softmax weight, τ\tau is the temperature, BB is the batch size, and dd is the reconstruction loss. When aa and nn point in the same direction, DCR’s direct gradient on nn vanishes. With Margin Regularization, the mismatched branch still receives a gradient whenever the margin is unmet and ∇nd(n,y)≠0\nabla_n d(n,y)\ne 0.

  • Outcomes: We observed substantial collapse during DCR training, along with insensitivity to mismatched conditions (see Appendix F.4). With Margin Regularization, condition shuffling produces more pronounced changes in reconstruction, and stronger gradients propagate back to CLIP’s visual features (Figure 3).

AuthorsOpenReview ↗

Response to bRc5 3/3

Weakness 3-2: Causes of Representation Collapse

To investigate whether representation collapse is caused by repeated CLS tokens, we reproduced the training procedure of DIVA, which uses dense representations for reconstruction, and observed collapse during training as well. For a fair comparison, we measured DIVA’s projected CLS tokens. As the number of training samples increased from 50K to 200K, the average pairwise cosine distance decreased from 0.72 to 0.20. Furthermore, like the other methods, DIVA was insensitive to mismatched conditions, indicating that it also failed to fully exploit image-specific information in the conditions. The corresponding measurements and visualizations are provided in Appendix F.4.

As a complementary control, we fixed the conditions and swapped the intermediate visual inputs, which produced substantial changes in reconstruction. This is consistent with our explanation in the paper: the generator can obtain substantial reconstruction information from intermediate visual inputs and pretrained priors, so minimizing reconstruction error alone may not sufficiently encourage the preservation of image-specific information in the conditions.

Weakness 2-2 & Weakness 3-3: Supplement to Hierarchical Boosting.

Methodological insight. The generator first establishes the overall structure of an image and then uses that structure to generate finer details. If CLIP has not captured the coarser structure, it cannot tell which part of the image a finer detail belongs to, even if it learns that detail from generative feedback. General multi-task methods typically do not use this dependency to decide when to introduce each objective. Hierarchical Boosting follows this order explicitly. It first gives CLIP direct supervision on the overall structure. It then introduces finer supervision while keeping the coarse objectives active, so that CLIP continues to learn the structure on which those details depend.

Theoretical clarification. Proposition 2 explains why progressively introducing finer scales in Hierarchical Boosting can limit interference with previously learned scales. As shown in the table below, Hierarchical Boosting yields smaller values of the interference measure ∣Is∣|I_s| defined in Proposition 2 than direct multi-scale supervision. To better distinguish Hierarchical Boosting from general multi-task methods, we examine the dependency between coarse structures and fine-grained details from an information-theoretic perspective. Let ZZ denote the CLIP representation, and CC and FF denote coarse structures and fine-grained details, respectively. Since CC is determined by FF, the entropy chain rule gives H(F∣Z)=H(C∣Z)+H(F∣C,Z)H(F\mid Z)=H(C\mid Z)+H(F\mid C,Z). This indicates that uncertainty in coarse structures also limits the representation of fine-grained semantics. For example, intermediate visual inputs may already provide the overall shape of a car, allowing the generator to produce wheels and headlights even when CLIP has not learned the car's overall structure. Such fine-scale feedback alone cannot ensure a complete semantic representation. Thus, unlike general multi-task methods that simply combine multiple objectives, Hierarchical Boosting uses their semantic dependency to determine the order of supervision.

w.o. Hierarchical Boostingw. Hierarchical Boosting
ckpt24002.23×10−52.23 \times 10^{-5}1.38×10−51.38 \times 10^{-5}
ckpt25002.45×10−52.45 \times 10^{-5}1.43×10−51.43 \times 10^{-5}
ckpt26002.52×10−52.52 \times 10^{-5}1.42×10−51.42 \times 10^{-5}

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Responses and Revisions

Dear Reviewers,

Thank you very much for your valuable feedback and constructive suggestions. To facilitate the discussion, we summarize our key responses and revisions below. We have also uploaded a revised manuscript incorporating these changes.

  1. Preservation of CLIP's original capabilities. We add zero-shot classification and retrieval evaluations across three backbones, showing that SGF largely preserves CLIP’s original capabilities while achieving state-of-the-art fine-grained performance. On MetaCLIP ViT-L/14@224, SGF outperforms DCR on MMVP (34.8 vs. 32.6) and the average classification/retrieval score (81.7 vs. 63.4). Results on SugarCrepe and CountBench further show that fine-grained gains extend beyond MMVP. (Reviewer bRc5 Q2; Reviewer BJtY W2, Q3; Reviewer MCjh W1)

  2. Comparisons with the same training budget and generator. SGF achieves higher MMVP scores across all six backbone configurations than baselines retuned for the same 84K processed-sample budget. This advantage persists when DIVA, GenHancer, and DCR are adapted to use a shared Infinity generator. Separate transfer experiments show that incorporating SGF into four existing methods improves their MMVP scores at the same sample budget. (Reviewer bRc5 W1, Q1; Reviewer BJtY Q1; Reviewer MCjh W2)

  3. More comprehensive ablations of Hierarchical Boosting. Compared with direct multi-scale supervision, Hierarchical Boosting improves MMVP from 26.7 to 31.9 while reducing training cost from 6.7 to 4.1 GPU hours. Additional schedule ablations support the importance of coarse-to-fine order and activation timing. (Reviewer BJtY W1, Q2; Reviewer MCjh W3)

  4. Further methodological and theoretical insights. We diagnose representation collapse and gradient conflicts in existing methods, showing that collapse also occurs with DIVA's dense conditioning and DCR's contrastive reconstruction loss. (Reviewer bRc5 W3; Reviewer MCjh W2) For Margin Regularization, we extend Proposition 1 to nonlinear projectors, derive a conditional lower bound on the effective gradient, and empirically verify that it increases the estimated lower bound. (Reviewer bRc5 W2; Reviewer BJtY W3) For Hierarchical Boosting, we clarify the coarse-to-fine rationale and report lower measured interference, consistent with Proposition 2's analysis. (Reviewer bRc5 W2–W3; Reviewer BJtY W3, Q4)

We would greatly appreciate it if you could review our responses and revisions and let us know whether any concerns remain. If you find that your concerns have been adequately addressed, we would be grateful if you could consider raising your rating accordingly.

Thank you again for your time and consideration!

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.