See More from Less: Sample-Efficient Generative Feedback for Fine-Grained Visual Understanding

Generative feedback can improve the fine-grained visual understanding of Contrastive Language-Image Pre-training (CLIP), but existing methods require substantial data and computation. We identify two sources of sample inefficiency: first, projected representations can collapse toward a shared constant, weakening gradient feedback to CLIP; second, hierarchical information is underutilized, as supervision from only one or a few levels limits feedback per sample, while directly combining multiple levels can introduce conflicting gradients. We propose Sample-efficient Generative Feedback (SGF) to extract more useful supervision from each training sample. Margin regularization encourages matched conditions to yield lower reconstruction losses than mismatched ones by a positive margin, preserving image-specific information. Hierarchical boosting progressively integrates coarse-to-fine supervision while retaining earlier objectives, giving them time to stabilize before finer supervision is introduced, thereby reducing cross-level interference. Experiments across six CLIP backbones show that SGF achieves state-of-the-art fine-grained perception using only 84K training samples, at least 82% fewer than prior methods, with a $6.0$-$7.9\times$ training speedup over the fastest competitor, demonstrating its improved sample efficiency.