- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SGF, which leverages feedback from a pre-trained generative model to improve the fine-grained recognition capability of visual encoders. The method consists of two main components: margin regularization to mitigate projected representation collapse, and progressive multi-scale training to gradually incorporate finer-grained generative supervision. The authors evaluate the method across six visual encoder configurations and report downstream results on large multi-modal models. SGF achieves performance comparable to or better than existing methods while using fewer training sample passes. However, the current experiments do not fully validate the sample efficiency advantage, nor do they isolate the contribution of the progressive schedule.
Critical strengths
-
The research problem has practical value. While generative feedback can improve visual encoders, high training costs typically limit its usage. This paper directly addresses how to reduce this cost.
-
Component designs are supported by experiments. The authors examine the differences in projected representations and backpropagation gradients. Ablation results also demonstrate that both components improve fine-grained recognition performance.
-
Experiments cover multiple model configurations. The paper tests three visual encoder families and provides multi-modal downstream evaluations along with diffusion model scaling. These results support the broad applicability of the method.
Critical weaknesses
-
It has not been evaluated whether the enhancement in fine-grained perception comes at the cost of degraded original capabilities. When updating the visual encoder, SGF uses only the generative reconstruction objective without directly reporting whether classification and image-text retrieval capabilities are maintained. This affects the comparison with DCR, because DCR focuses on both detail perception and discriminative capability. Existing MMVP-VLM and multi-modal QA results provide positive evidence, but are insufficient to determine this capability trade-off. Therefore, it remains unclear whether SGF can preserve an equally complete visual representation capability at a lower cost.
-
The explanation for the inefficiency of existing methods exceeds the current diagnostic evidence. The paper attributes projected representation collapse and insufficient hierarchical supervision as the efficiency bottlenecks of existing generative feedback methods, but the related diagnoses are primarily based on the training setup of this paper. There are significant differences among baselines: un²CLIP does not require an additional alignment projector, and DCR has already introduced conditional contrastive signals. The paper has not directly verified whether these baselines suffer from the same issues, nor whether these issues lead to their higher training costs. Consequently, the current results more directly support the effectiveness of the proposed training scheme, while the explanation for the efficiency bottlenecks of existing methods still requires further evidence.
-
The ablation study fails to disentangle the contributions of multi-scale supervision and progressive scheduling. Table 4 compares single-scale supervision with progressive multi-scale supervision while simultaneously altering the number of supervision scales and their inclusion order. Therefore, it cannot be determined whether the performance improvement stems from supervision at more scales or from the progressive schedule. Although Figure 4 compares progressive training with direct joint training, it only reports the coarse-scale reconstruction loss, which cannot be used to evaluate the final fine-grained recognition performance. To confirm the independent contribution of the progressive schedule, downstream results under a fixed set of scales and other identical training settings need to be compared between the two approaches.
Questions for authors
See weakness.
4 comments
Response to MCjh 2/2
Weakness 2: Further Explanation of the Efficiency Bottlenecks in Existing Methods
Thank you for your comments. We examined whether the two phenomena discussed in our paper occur in existing methods:
-
Representation collapse. We evaluated DIVA, GenHancer, and DCR, all of which train a projection layer. As shown below, each method exhibited signs of representation collapse as the number of processed training samples increased from 50K to 200K. We excluded un2CLIP from this analysis because it does not train a projection layer. Furthermore, reconstruction in all three methods was insensitive to mismatched conditions (see Appendix F.4 for the visualizations), indicating insufficient use of image-specific information in the conditions and a strong reliance on intermediate visual inputs.
Table: Representation Collapse Across Different Methods
DIVA GenHancer DCR 50k 0.72 0.56 0.50 200k 0.20 0.28 0.08 -
Underutilized hierarchical information. This issue is common in previous methods, which use only one or two denoising steps, leaving substantial supervisory information unused. We further verified that directly combining supervision from multiple timesteps leads to gradient conflicts.
As analyzed in our paper, these phenomena hinder effective gradient propagation and reduce sample efficiency, increasing training costs. We additionally incorporated SGF into these methods. As shown in the table below, SGF yields substantial improvements over their original formulations with a sample budget of only 84K. These results indicate that addressing the above issues with SGF helps unlock the potential of these methods.
Table: Performance Improvements from Incorporating SGF into Existing Methods Under an 84K Sample Budget
| DIVA | GenHancer | un2CLIP | DCR | |
|---|---|---|---|---|
| w.o. SGF | 23.7 | 23.0 | 24.4 | 20.7 |
| w. SGF | 25.2 | 28.2 | 28.9 | 25.2 |
Weakness 3: More Comprehensive Ablation Studies on Hierarchical Boosting
Thank you for this valuable suggestion. To isolate the contribution of progressive scheduling in Hierarchical Boosting (HB), we added a comparison with direct multi-scale supervision (MS) under the same experimental settings. As shown in the table below, directly applying multi-scale supervision yields limited performance gains while incurring additional computational overhead. Compared with direct multi-scale supervision, Hierarchical Boosting achieves better MMVP performance with less GPU time.
Table: More Comprehensive Ablation Results on Hierarchical Boosting. MS denotes multi-scale supervision, and HB denotes Hierarchical Boosting.
| MR | MS | HB | MMVP Score | GPU Mem. | GPU Hours |
|---|---|---|---|---|---|
| - | - | - | 22.2 | 18.5 GB | 3.3 h |
| √ | - | - | 25.2 | 20.0 GB | 4.7 h |
| - | √ | - | 24.4 | 20.1 GB | 4.6 h |
| √ | √ | - | 26.7 | 20.1 GB | 6.7 h |
| - | - | √ | 28.1 | 20.1 GB | 3.4 h |
| √ | - | √ | 31.9 | 20.2 GB | 4.1 h |
Thank you for the response and additional analyses, which address most of my concerns. The classification and retrieval results clarify the capability trade-offs, and the baseline diagnostics and direct multi-scale comparisons provide the evidence I was looking for. I still think the explanation of why existing methods are inefficient should be stated more cautiously. Overall, the results support the practical benefit of SGF, and I will raise my rating.
We sincerely appreciate your kind words and the improved score. We are pleased that our responses have addressed most of your concerns. Your constructive comments have greatly helped us enhance the soundness of our paper.
Following your suggestion, we have revised the manuscript to describe the causes of existing methods’ inefficiency more precisely and cautiously. We have also provided empirical analyses of representation collapse and gradient conflicts in Appendix F.4 to further support our analysis.
We will ensure that these revisions are properly incorporated into the final version.
Response to MCjh 1/2
Thank you for your thoughtful feedback and valuable recommendations. We have carefully revisited the points raised in your review and address them below. We hope the additional evidence and explanations help resolve your concerns.
Following your suggestion, we evaluated zero-shot classification and retrieval performance across multiple CLIP backbones. The tables below report Recall@5 for retrieval and classification accuracy (ACC) for zero-shot classification. In the two summary columns, bold indicates the best result among enhanced methods, and a superscript † marks the second-best result. We evaluated all baselines using their publicly released checkpoints. Notably, our method achieves the best performance on fine-grained tasks while largely preserving CLIP’s original capabilities. We clarify that DCR targets the discriminability of CLIP’s visual features, which differs from the image–text discrimination required for zero-shot classification.
Table: Evaluation of Zero-Shot Classification and Retrieval on OpenAI ViT-L/14@224.
Table: Evaluation of Zero-Shot Classification and Retrieval on MetaCLIP ViT-L/14@224.
Table: Evaluation of Zero-Shot Classification and Retrieval on SigLIP ViT-SO-14@384.