- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper observes that CLIP's native image-text function does not determine the Q/Q and K/K self-similarities used by training-free dense readers. Based on this, it proposes Learned G: a bounded low-rank per-head transform on the last visual block, trained with 256 VOC images and folded into the Q/K weights at no inference cost. It beats learned per-head scalars of ViT-B/16 and ViT-L/14 on VOC, and improves on top of a fixed trained output head. It also proposes a canonicalization that makes dense readers consistent across equivalent parameterizations, and a transport procedure that keeps a learned task metric under coordinate changes.
Critical strengths
- The observation is interesting.
- The method preserves CLIP's native function exactly, has very few parameters, and folds into existing weights with no inference overhead.
- The control experiments are well designed and the results are reported carefully.
Critical weaknesses
- Reader-only LoRA4 beats G by 1.72 mIoU on ViT-B/16 and by 6.5 on ViT-L/14 VOC. A rank-12 output head with a similar number of parameters beats G by 3.89. The only stated advantage of G is that it preserves CLIP's native function through the same Q/K weights. However, reader-only LoRA also preserves the native function by keeping the original path. So I think the motivation for G is weak.
- Once 256 labeled VOC images are used, the method is no longer training-free, and it should be compared with supervised or few-shot open-vocabulary segmentation methods.
- Evaluation uses VOC val, a custom 1000-image COCO-Object subset and partial ADE panels, instead of the standard full benchmarks. As a result, the numbers cannot be compared with the literature. Besides, the authors state that the VOC evaluation images were used in earlier experiments, and the main LoRA comparison (Table 3) was completed after earlier 512-image results were known, so it is post-hoc. With many studies, separately trained G baselines and different protocols per table, repeated iteration on the same evaluation set is a real risk of overfitting.
- The writing is very hard to follow. For example, they used many coined acronyms and many separately run studies with different protocols and baselines.
Questions for authors
Please refer to weaknesses.