acceptodds
Withdrawn from ICLR 2027

Depth-wise Specialization in Vision-Language Models for Compositional Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive vision–language models such as CLIP and SigLIP achieve strong global image–text alignment, yet often struggle with fine-grained compositional distinctions such as modifier sensitivity and visually grounded relations. Prior work improves compositionality through stronger supervision, caption enrichment, or pooling modifications, but largely overlooks how compositional cues evolve across encoder depth. We reveal complementary depth specialization: noun-phrase selectivity strengthens toward later layers, whereas modifier- and relation-sensitive distinctions progressively attenuate. Depth-wise probes of broad noun-phrase selectivity, same-head modifier sensitivity, and relation-family discrimination consistently support this pattern. Motivated by this finding, we introduce RELIP, a depth-asymmetric fine-tuning method for contrastive dual encoders. RELIP retains each backbone's native global image–caption objective while using shallow patch representations as compositional evidence and a detached late relevance map to emphasize attribute–entity and grounded-relation cues that attenuate with depth. The auxiliary branch is training-only and leaves the original inference architecture unchanged. Under the same CC300K budget, RELIP improves both CLIP and SigLIP; with SigLIP, it reaches on SugarCrepe and on SugarCrepe++, outperforming CLIP by and points while preserving strong retrieval and zero-shot performance. These results show that effective compositional adaptation depends not only on what supervision is provided, but also on where and how it is applied.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Compositionality, Multimodal Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EhXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the lack of compositionality in CLIP and SigLIP models. The authors find that prior works did not consider how compositional knowledge exists across different depths of the model. They analyze the existing information in shallow, mid, and late layers and identify better “noun-phrase selectivity” in later layers and better “fine-grained compositional discrimination” in shallower layers. Based on this analysis, the authors suggest RELIP, a fine-tuning method, which improves compositionality as well as classification and retrieval, compared to standard fine-tuning on the same data.

The paper is hard to follow, with some terms not clearly defined (e.g., s_l(c,p)) or defined after being used for the first time (e.g., e_q), and with dense sentences (e.g., L048-051).

Critical strengths
  1. The analysis helps to understand the strengths and weaknesses of the pretrained CLIP ViT-L/14 model and gives a motivation for some of the suggested additions to the method.

  2. Evaluation on both compositionality metrics and general metrics (retrieval and classification), as well as on SigLIP and CLIP models, shows improvements compared to standard fine-tuning on the same data.

Critical weaknesses
  1. The analysis (and its conclusions) are based only on COCO. In addition, from the experiments, it is not clear whether there is good fine-grained discrimination in shallower layers.

  2. Method and results. RELIP leads to small compositionality improvements compared to fine-tuning the model on the same data, and from the tables it is not clear if it improves compared to previous baselines. In addition, it is not clear why certain design choices were made based on the analysis (please see questions).

Questions for authors
  1. Why is the analysis done on early, mid, and later layers, while the method operates only on early layers and the final layer (whose gradients are detached from the auxiliary loss)? Related to that, what do you mean by Shallow-Only and Late-Only in your ablations?

  2. Can it be that the margins are shrinking in later layers for modifier-bindings because both frisbee and red frisbee are getting closer to the full caption? Can you use a metric that does not compare to the full caption to check this?

  3. For the relation-family contrast: all averages are negative (and later layers are more negative on average). At least from this number, I don’t see why target relations are less separable in later layers, as it seems that all layers cannot separate the positives from the negatives. Does it mean that the model lacks relation contrast across all layers?

  4. How do you extract the phrase set, and how do you filter this set? Is this time included in your method (30 min)?

  5. In Table 1, it is unclear which models to compare with which. Are the fine-grained models and the composition-aware models fine-tuned on CC300k as well? Are they based on SigLIP or on CLIP ViT-B/16? In addition, some composition-aware methods that have better SugarCrepe results are missing, e.g., DAC, READ-CLIP.

  6. Do you understand why the compositionality ITT increases are relatively mild? Can these numbers improve if you train on more samples (from the same CC3M dataset)?

Reviewer_BiLAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how compositional information changes across the depth of contrastive vision-language encoders. The authors find that late layers exhibit stronger noun-phrase selectivity, while shallow layers preserve more fine-grained information about attributes and visual relations. Motivated by this observation, they propose RELIP, a depth-asymmetric fine-tuning method that uses shallow patch representations as compositional evidence and a detached late-layer relevance map to identify spatial regions where fine-grained cues are attenuated. The method adds an auxiliary phrase-level contrastive objective while retaining the backbone's original global image-text alignment objective. Experiments on CLIP and SigLIP show improvements on SugarCrepe and SugarCrepe++, while maintaining or improving image-text retrieval and zero-shot classification performance. The results suggest that explicitly exploiting differences in compositional information across encoder depth can improve compositional adaptation without changing the inference architecture.

Critical strengths
  • Interesting empirical observations about encoder depth. The paper identifies a potentially useful distinction between late-layer noun-phrase selectivity and shallow-layer attribute-binding and relation-sensitive information. The analysis considers multiple probes and both CLIP and SigLIP, providing evidence that compositional information is not uniformly preserved across encoder depth.
  • Consistent improvements over strong baselines. RELIP improves both CLIP and SigLIP on the evaluated compositional benchmarks. Under the same CC300K training budget, RELIP outperforms C2LIP by 1.4 points on SugarCrepe and 2.2 points on SugarCrepe++. It also improves retrieval and zero-shot classification performance, suggesting that the gains are not achieved simply by sacrificing general vision-language capabilities.
  • Practical implementation with no inference-time overhead. The auxiliary branch is used only during training and is removed at inference. This makes RELIP relatively easy to integrate into existing contrastive vision-language training pipelines without adding inference-time parameters or computation.
Critical weaknesses
  • The depth-wise analysis relies on proxy metrics whose interpretation could be better justified. The paper uses relevance-map alignment, phrase gaps, and contrast margins to characterize noun-phrase selectivity and compositional discrimination. However, a larger phrase gap does not necessarily imply better phrase grounding, and the relationship between these metrics and actual compositional understanding is not fully established. The analysis would be stronger with additional validation against independent grounding annotations or complementary measures of attribute binding and relation recognition.
  • The robustness of the reported improvements remains uncertain. Appendix B states that each controlled configuration is based on a single distributed training run. Since some improvements over strong baselines are relatively modest, it is difficult to determine whether the gains are consistently reproducible or sensitive to random initialization and training dynamics. Repeated runs with standard deviations or confidence intervals would strengthen the empirical claims.
  • The methodological novelty beyond the depth-wise observation could be better established. RELIP combines shallow patch-level evidence, a detached late-layer reference, differential spatial weighting, and an auxiliary phrase-level contrastive objective. Although the ablations support these design choices, the paper could more clearly explain why this particular formulation is preferable to simpler alternatives, such as directly applying phrase-level supervision to shallow features or using fixed spatial weighting. More direct comparisons with alternative depth-aware compositional adaptation strategies would clarify the method's specific contribution.
  • The scope of the depth-wise analysis remains limited. The analysis uses MSCOCO image-caption pairs and a set of phrase probes, while the relation study focuses on five relation families. Although the appendix includes additional SigLIP analyses, it remains unclear how broadly the observed specialization generalizes to other datasets, more diverse relational concepts, and different encoder architectures. Broader validation would help establish whether this is a general property of contrastive vision-language models or a pattern dependent on the evaluated settings.
  • The scope of the depth-wise analysis remains limited. The analysis uses MSCOCO image-caption pairs and a set of phrase probes, while the relation study focuses on five relation families. Although the appendix includes additional SigLIP analyses, it remains unclear how broadly the observed specialization generalizes to other datasets, more diverse relational concepts, and different encoder architectures. Broader validation would help establish whether this is a general property of contrastive vision-language models or a pattern dependent on the evaluated settings.
Questions for authors
  • Can the authors validate the proposed depth-wise metrics against independent spatial grounding or compositional reasoning annotations? In particular, does stronger noun-phrase selectivity reliably correspond to better phrase grounding, or can the two behaviors diverge?
  • Have the authors evaluated RELIP across multiple random seeds? Given that each controlled configuration uses a single training run, how much variance is observed in the reported improvements over the baseline and C2LIP?
  • How sensitive is RELIP to the choice of shallow and late layers? Would similar improvements be obtained with different layer combinations, or is the proposed shallow-late differential weighting dependent on the specific blocks selected in the experiments?
  • Can the authors provide further comparisons with simpler depth-aware alternatives, such as shallow-only phrase supervision with different spatial weighting strategies? This would help isolate the contribution of the detached late-layer reference and the differential weighting mechanism.
  • The paper does not cite or compare against UMGCLIP [1]. Could the authors discuss the relationship between RELIP and UMGCLIP and clarify their differences in motivation, methodology, and technical contributions? A direct comparison under matched backbones and training data budgets would help establish the novelty of RELIP and better demonstrate its advantages over existing approaches.

[1] Shi, Bowen, et al. "Umg-clip: A unified multi-granularity vision generalist for open-world understanding." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

Reviewer_ewkeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how compositional cues vary across the depth of contrastive vision-language encoders. It compares caption-conditioned patch relevance with phrase-conditioned relevance and reports increasing noun-phrase selectivity but decreasing modifier and relation margins in CLIP and SigLIP. RELIP uses shallow patch representations and a detached late relevance map to construct spatial weights from their normalized positive difference. This auxiliary image-phrase objective accompanies the backbone's native global objective and is removed at inference. Under CC300K fine-tuning, RELIP improves both backbones, with SigLIP reaching 86.1 on SugarCrepe and 66.9 on SugarCrepe++, compared with C$^2$LIP's 84.7 and 64.7.

Critical strengths
  1. RELIP makes a specific, testable change to phrase-conditioned training: it supervises shallow evidence using a detached late spatial reference. The positive difference between spatially z-normalized maps determines soft spatial weights, while the pooled score retains the original shallow cosine similarities. The auxiliary branch is discarded at inference, making this a practical adaptation of existing dual encoders rather than a new deployment requirement.

  2. The backbone-matched CC300K comparisons provide evidence of useful gains on compositional benchmarks. On SigLIP, improvements over C$^2$LIP extend across add, replace, and swap categories, while Flickr30K and MSCOCO retrieval also improve. The phrase-source control is particularly helpful: C$^2$LIP-Qwen changes little, whereas the depth variants improve, making a phrase-extractor-only explanation less plausible.

Critical weaknesses
  1. The three probes do not show what the paper concludes from them. Each score is the agreement between a phrase's patch map and the caption's patch map, computed with the same projection at every depth. The modifier score is the difference between "red frisbee" and "frisbee". Late layers match salient noun phrases better (the phrase gap rises from 0.1731 to 0.3256), and "frisbee" is such a phrase, so this difference can shrink with no loss of modifier information. Separating the two readings needs "red frisbee" against "blue frisbee". The relation probe is weaker: its mean margin is negative at every depth (-0.0101, -0.0210, -0.0300, Section 3.3.2), so on average shallow layers also rank the correct relation below its hardest alternative, and only 62% of episodes favor shallow. Contribution 1 rests on these probes.

  2. The ablation behind the method cannot be checked. Table 2 is the only place where the weighting is separated from the extra phrase loss, yet Shallow-Only and Late-Only are never defined, so I cannot tell what is held fixed. The gain over Shallow-Only is 0.7 on SugarCrepe and 1.6 on SugarCrepe++, with one run per setting, so 0.7 cannot be separated from run-to-run noise. Since the depth choice and the weighting are the only new pieces relative to C2LIP, FLAIR and FineCLIP, this comparison is what the new idea rests on. There is no all-depth uniform variant, and no measurement showing that the weights land on modifiers or relations; Figure 4 shows two examples.

Questions for authors
  1. What distinguishes the modifier-margin trend from increasing agreement with the bare head noun? Equation (2) measures caption-map agreement, not correct versus incorrect attribute binding; a correctly recognized object can occupy the same region for both queries. The shared projection in Section 3.1 also need not be equally suitable at every depth. Please separate the two alignment terms and test an entity-preserving incorrect-modifier or swapped-binding control. Persistence under an independently calibrated per-layer readout would further strengthen the specialization interpretation; otherwise, I would retain the empirical observation but narrow its meaning.

  2. Please specify the relation episode counts, candidate construction, scoring reference, and visual validation rules (Section 3.3.2, Appendix F). Do alternatives strictly preserve entities, and how are simultaneously valid descriptions, such as riding and being adjacent to an object, handled? Entity-controlled, visually verified episodes with image-level statistics would increase my confidence in the relation claim.

  3. Please give the exact pooling and loss definitions for Late-Only and Shallow-Only in Table 2, identifying all differences from RELIP. Apart from the selected depths and differential weighting, which temperature, pooling, calibration, collision-masking, and optimization choices are held fixed? Appendix B reports one run per configuration, while shallow versus late differs by 0.6/0.5 points and RELIP versus shallow by 0.7/1.6. Matched repeats with uncertainty estimates would strengthen the claim that depth and differential weighting explain the gains.

  4. Relative to which reference is general capability preserved? Table 1 shows RELIP improving ImageNet over matched fine-tuning, but remaining below pretrained accuracy: 70.5 versus 76.1 for SigLIP and 66.2 versus 68.4 for CLIP. Restricting the conclusion to no additional degradation relative to matched fine-tuning would resolve this concern. Evidence of retention at matched compositional accuracy would support a stronger claim.

  5. How do the approximately 30-minute training time and 1.46 seconds per optimizer step relate (Sections 5.1 and 5.3.4)? The stated 300K images, five epochs, and effective batch of 768 imply about 1,953 steps, or 47.5 minutes at that rate. Please report the actual update count and distinguish end-to-end training time from profiled step time. I would qualify the relative efficiency claim if profiling settings differ across methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.