- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces a method for semantic segmentation using a frozen SigLIP 2 encoder. To this end, the authors partially reuse SigLIP 2's MAP head: they first obtain the patch tokens after the last layer's LayerNorm, project them with the V matrix of the MAP head, aggregate them using a similarity matrix between all token pairs, and pass them through the output projection and MLP block.
The similarity matrix with entries is obtained by first computing the cosine similarities across every pair of patch-embedding outputs, then masking out tokens under a per-row similarity threshold , where is the highest off-diagonal similarity in the row. Finally, each row is converted into a probability distribution using a softmax with temperature .
The authors measure the performance of combining features and similarity matrices from every block pair, and find that combining patches from the last block and the similarity matrix obtained using patches after the patch embedding works best, with performance sharply degrading when patch features are selected from shallower blocks.
Using their aggregation strategy significantly improves performance across several semantic segmentation datasets, and outperforms other comparable aggregation strategies.
Furthermore, the authors show that their method extends to SigLIPv1 and CLIP-RN50.
Critical strengths
- The method is simple, requires no training, and no additional forwards.
- The method performs strongly across the reported segmentation datasets.
Critical weaknesses
- While the method is understandable, the paper is hard to read, often uses non-standard terminology, sections are dense, and the motivation of some experiments is unclear. For example, I find it particularly hard to understand what point is being made for the "Is the ordering a preference for smoother supports?" experiment.
- The evaluation is limited to ViT-B models. Do the findings extend to larger or smaller models within the SigLIP 2 family?
- The applicability of the method is relatively narrow, as to my understanding it can only be applied to models using a pooling head, and does not extend to e.g. vanilla CLIP ViTs, which, combined with other methods, is shown to perform similarly well as the proposed method in [1]. Furthermore, Table 5 shows that only performs marginally worse compared to the full proposed method on SigLIPv1. It would be interesting to see if a similar effect extends to CLIP-RN50.
- The final layer targets are hand-picked, and it's unclear how sensitive the method is to these values.
- The comparison with other methods is limited to Table 5. While other methods are cited in the related works, it's unclear why the authors did not report their performance.
- Similar observations in terms of visual discriminability and semantic content have been made in previous work [1]. Currently this is only mentioned in the related works, but I believe it should be discussed more thoroughly in the analysis section.
[1] Improving Visual Discriminability Of Clip For Training-free Open-vocabulary Semantic Segmentation, TMLR, 2026.
Questions for authors
- Does the performance improvement carry over to larger and smaller models?
- Could the authors explain how the target and have been picked, and the sensitivity of their method to these values?
- A more thorough comparison with other state of the art methods (or an explanation of why they are not comparable) would strengthen the paper.