acceptodds
Withdrawn from ICLR 2027

Decoupling Content and Spatial Support in Frozen Vision Language Readouts.pdf

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dense prediction with a frozen vision–language encoder depends on both the content supplied to its text-aligned head and the spatial support used to aggregate patches. We investigate whether the same representations are suitable for both roles by generalizing the pretrained pooling head into a support-set decoder with independently selectable content and support depths. An exhaustive 13x13 depth map on SigLIP 2 B/16 reveals distinct preferences: final-layer content and pre-contextual support. Holding content fixed, replacing pre-contextual support with final-layer support reduces VOC20 mIoU by 6.4 points. Statistical controls clarify the distinction: matching content from an earlier layer to the native head input's moments recovers most of its readout loss, while matching final-layer support features to pre-contextual moments substantially narrows the support gap. These interventions distinguish compatibility with the frozen head from the feature geometry used for spatial aggregation. Cross-encoder controls characterize where the shallow-support preference holds. Applying this separation yields SigMAP, a training-free decoder that combines final-layer content with pre-contextual support and improves on identity support across five datasets. Together, these findings identify content compatibility and spatial support quality as distinct criteria for selecting representations under a frozen readout.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, dense prediction, open-vocabulary semantic segmentation, representation analysis, SigLIP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1uUAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a method for semantic segmentation using a frozen SigLIP 2 encoder. To this end, the authors partially reuse SigLIP 2's MAP head: they first obtain the patch tokens after the last layer's LayerNorm, project them with the V matrix of the MAP head, aggregate them using a similarity matrix SS between all token pairs, and pass them through the output projection and MLP block.

The similarity matrix SS with entries (i,j)(i, j) is obtained by first computing the cosine similarities across every pair of patch-embedding outputs, then masking out tokens under a per-row similarity threshold miτm_i\tau, where mim_i is the highest off-diagonal similarity in the row. Finally, each row is converted into a probability distribution using a softmax with temperature TT.

The authors measure the performance of combining features and similarity matrices from every block pair, and find that combining patches from the last block and the similarity matrix obtained using patches after the patch embedding works best, with performance sharply degrading when patch features are selected from shallower blocks.

Using their aggregation strategy significantly improves performance across several semantic segmentation datasets, and outperforms other comparable aggregation strategies.

Furthermore, the authors show that their method extends to SigLIPv1 and CLIP-RN50.

Critical strengths
  1. The method is simple, requires no training, and no additional forwards.
  2. The method performs strongly across the reported segmentation datasets.
Critical weaknesses
  1. While the method is understandable, the paper is hard to read, often uses non-standard terminology, sections are dense, and the motivation of some experiments is unclear. For example, I find it particularly hard to understand what point is being made for the "Is the ordering a preference for smoother supports?" experiment.
  2. The evaluation is limited to ViT-B models. Do the findings extend to larger or smaller models within the SigLIP 2 family?
  3. The applicability of the method is relatively narrow, as to my understanding it can only be applied to models using a pooling head, and does not extend to e.g. vanilla CLIP ViTs, which, combined with other methods, is shown to perform similarly well as the proposed method in [1]. Furthermore, Table 5 shows that S=IS = I only performs marginally worse compared to the full proposed method on SigLIPv1. It would be interesting to see if a similar effect extends to CLIP-RN50.
  4. The final layer targets (τ=0.85,T=0.15)(\tau = 0.85, T = 0.15) are hand-picked, and it's unclear how sensitive the method is to these values.
  5. The comparison with other methods is limited to Table 5. While other methods are cited in the related works, it's unclear why the authors did not report their performance.
  6. Similar observations in terms of visual discriminability and semantic content have been made in previous work [1]. Currently this is only mentioned in the related works, but I believe it should be discussed more thoroughly in the analysis section.

[1] Improving Visual Discriminability Of Clip For Training-free Open-vocabulary Semantic Segmentation, TMLR, 2026.

Questions for authors
  1. Does the performance improvement carry over to larger and smaller models?
  2. Could the authors explain how the target τ\tau and TT have been picked, and the sensitivity of their method to these values?
  3. A more thorough comparison with other state of the art methods (or an explanation of why they are not comparable) would strengthen the paper.
Reviewer_Z8K8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies open-vocabulary dense prediction with a frozen vision–language encoder and generalizes the pretrained MAP pooling head into a support-set decoder whose single probe-attention row is replaced by an arbitrary row-stochastic matrix S. This decouples two previously confounded choices: the content depth c (which transformer layer feeds the frozen value/output path) and the support depth s (which layer supplies pairwise affinities for aggregation). An exhaustive 13×13 depth map on SigLIP 2 B/16 reveals opposite preferences: content prefers the final layer (c=12), support prefers the pre-contextual patch-embedding output (s=0), and holding content fixed while moving support from s=0 to s=12 loses 6.4 mIoU on VOC20. Statistical controls (token-norm, per-channel affine, and whitening–recoloring to depth-0 second moments) recover 87.1% of the content gap and 70.4% of the VOC20 support gap, indicating that second-moment feature statistics matter to both roles. A cross-encoder sweep associates the shallow-support preference with "bottleneck-aligned" encoders (SigLIP 2, SigLIP v1, CLIP-RN50) and shows it is absent in AIMv2-LiT and CoCa. Based on this separation the authors propose SigMAP, a training-free decoder that keeps final-layer content on the native head path and sources support from the encoder's own depth-0 patch embeddings; it improves the S=I value-path readout from 35.24 → 42.82 mean mIoU across five segmentation datasets (VOC20, Context59, COCO-Stuff, ADE20K, Cityscapes), with no additional parameters, no external model, and no extra encoder forward.

Critical strengths
  1. A principled measurement instrument. Generalizing the pooling probe row into an arbitrary row-stochastic S matrix, while keeping the entire value/output path and MLP block frozen, is a clean way to turn "which layer should feed the readout" into two orthogonal, controllable variables. The 13×13 depth sweep then falsifies the implicit assumption in prior CLIP-adaptation work that content and support must come from the same layer. This is the paper's real contribution and it reframes a diffuse body of related work (MaskCLIP/SCLIP/ClearCLIP/NACLIP/ProxyCLIP) in a unified way.
  2. The statistical-recovery experiments are the right kind of control for the claim. Showing that depth-10 content can be made 87.1% readable by matching its second moments to the native head input, and that depth-12 support can be made 70.4% better by whitening its moments to depth-0 geometry, is a non-trivial piece of evidence that the two gaps have different causes (interface calibration vs. aggregation geometry). This is more informative than another ablation and is what elevates the paper above a method note.
Critical weaknesses
  1. No head-to-head with recent OVSS adaptations on the main SigLIP 2 result. Table 4 compares SigMAP (42.82) only against the S=I value-path baseline (35.24) on the same SigLIP 2 B/16. The strong adaptations the paper situates itself against — ProxyCLIP, CLIP-DINOiser, SCLIP, NACLIP, CAT-Seg, CorrCLIP — are mentioned only in Related Work. The protocol-matched SigLIP v1 comparison (Table 5) does include ClearCLIP and the prior SigLIP adaptation of Zhou et al. (2026), where SigMAP is +1.65 over the S=I control (30.28 → 31.93) and +5.03 over the prior adaptation (26.9 → 31.93). On SigLIP 2 B/16 the reader cannot tell whether the +7.58 gain reflects a strong decoder or just a strong backbone the baselines haven't been tuned on. Given that the "practical outcome" of the analysis is SigMAP, this gap matters.
  2. The alignment-route story is under-determined. The dichotomy in Table 3 (bottleneck-aligned vs. post-hoc/captioning) is intriguing but is drawn from a single measurement per encoder (VOC20 shallow-minus-final support gap). Without varying the alignment route within a controlled encoder (e.g., SigLIP 2 with and without the auxiliary self-distillation/masked-prediction objectives, or a controlled post-hoc re-alignment of a bottleneck-aligned model), the causal claim rests on a 5-point correlation across models that also differ in architecture and data. The paper is honest in its Limitations about this, but the AIMv2-LiT/CoCa pair carries disproportionate weight in the narrative.
Questions for authors

In Table 2, DINOv2 patch-embedding support reaches 41.02 vs. SigMAP 42.82 (mean), while on Cityscapes DINOv2 is actually better (39.89 vs. 42.20 — please double-check this: Table 2 shows SigMAP 42.20 vs. DINOv2 39.70). What is the contribution of the encoder's patch-embedding projection (positional+linear) itself, versus the raw pixel grid? Table 2 does include a Gaussian-prior and bilateral baseline, but a controlled "position-only / color-only / position+color" decomposition of depth-0 would isolate how much of SigMAP's edge is positional information already present in the embedding vs. visual information.

Can the alignment-route claim in Table 3 be strengthened by an intra-model comparison? E.g., evaluating a SigLIP 2 variant trained without the auxiliary self-distillation/masked-prediction losses, or evaluating CoCa after disabling the captioning cross-attention branch? Without such a counterfactual, Table 3 is a 5-point correlation.

In 2.3 (why shallow content fails), whitening-recoloring to depth-12 second moments recovers 87.1% of the depth-10 content gap on VOC20 validation. What explains the remaining 12.9%? If it is first-moment or higher-order structure, a direct demonstration (e.g., matching 3rd/4th moments or using an invertible flow) would make the "interface constraint" framing much stronger.

How were τ=0.8384 and T=0.088345 at depth 0 selected relative to the development vs. validation split? The appendix (C.1) says a single calibration is shared across the five benchmarks, but shows these values only for depth 0. Is this calibration re-derived from scratch at each (c,s) cell in the 13×13 map, and how sensitive is the depth map to the ±0.1 nat tolerance on mean row entropy?

Table 5 lists "ClearCLIP transplant" at 2.3 mIoU on SigLIP v1 — this is strikingly low. Is this the ClearCLIP recipe applied verbatim to SigLIP's attention/MLP, or a faithful transplant that respects SigLIP's sigmoid loss and shared Head? If it is the former, the comparison risks being a straw man.

Reviewer_te12OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors turn SigLIP 2's attention-pooling head into a dense decoder by replacing its single probe-attention row with an N x N row-stochastic matrix, which yields per-patch predictions from a frozen encoder. They propose SigMAP: keep the content path at the final layer, and build the aggregation matrix from cosine affinity among the layer-0 patch embeddings. The choice is motivated by a 13x13 sweep over content and support depth. Mechanically this is a per-patch value readout smoothed by a substituted affinity, the recipe of MaskCLIP followed by ProxyCLIP-style affinity replacement, with the affinity taken from layer 0 instead of an external model.

Critical strengths

The paper is clearly written and the scope of each claim is stated carefully. The method is simple, costs nothing at inference, and could be reimplemented from the text alone.

Critical weaknesses
  • Decoupling the aggregation affinity from the content features is well explored, and is not a contribution of this paper. ProxyCLIP already substitutes a foundation-model correlation as proxy attention with its own masking and normalisation, which is what S does here, and CLIP-DINOiser already obtains DINO-like affinity from CLIP's own features in a single forward pass, which is SigMAP's claim minus the training. Both halves of SigMAP are prior art by the paper's own account, which credits SPAR for the SigLIP 2 value path and ProxyCLIP for an external S. The one new ingredient is taking the affinity from layer 0, and none of these methods is compared against in its own setting.

  • Neither finding of the depth map is new. That the frozen head needs its native input follows from its having been trained on depth-12 statistics, and the paper shows the drop is largely calibration, recoverable to 87.1% by whitening. That final-layer similarities group patches badly is the premise ProxyCLIP, CLIP-DINOiser and CorrCLIP were built on.

  • The preference holds only for one configuration of one encoder family, and the engineering claim behind it is untested. Table 3 reports it reversing on AIMv2-LiT and CoCa, and in Figure 2 it survives only at content depth 12, since at every shallower content depth the best support is mid-depth and layer 0 is the worst or near-worst. The alignment-route account offered for the cross-encoder pattern rests on five encoders differing in architecture, data and objective, which the paper concedes is descriptive. Layer 0 is the patch projection plus the positional embedding, so its cosine affinity is a learned appearance-and-position kernel; the five-set margin over the hand-crafted version is 1.76 points, a position-only prior beats SigMAP on VOC20, and the paper does not separate the two components.

Questions for authors

please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.