- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the lack of compositionality in CLIP and SigLIP models. The authors find that prior works did not consider how compositional knowledge exists across different depths of the model. They analyze the existing information in shallow, mid, and late layers and identify better “noun-phrase selectivity” in later layers and better “fine-grained compositional discrimination” in shallower layers. Based on this analysis, the authors suggest RELIP, a fine-tuning method, which improves compositionality as well as classification and retrieval, compared to standard fine-tuning on the same data.
The paper is hard to follow, with some terms not clearly defined (e.g., s_l(c,p)) or defined after being used for the first time (e.g., e_q), and with dense sentences (e.g., L048-051).
Critical strengths
-
The analysis helps to understand the strengths and weaknesses of the pretrained CLIP ViT-L/14 model and gives a motivation for some of the suggested additions to the method.
-
Evaluation on both compositionality metrics and general metrics (retrieval and classification), as well as on SigLIP and CLIP models, shows improvements compared to standard fine-tuning on the same data.
Critical weaknesses
-
The analysis (and its conclusions) are based only on COCO. In addition, from the experiments, it is not clear whether there is good fine-grained discrimination in shallower layers.
-
Method and results. RELIP leads to small compositionality improvements compared to fine-tuning the model on the same data, and from the tables it is not clear if it improves compared to previous baselines. In addition, it is not clear why certain design choices were made based on the analysis (please see questions).
Questions for authors
-
Why is the analysis done on early, mid, and later layers, while the method operates only on early layers and the final layer (whose gradients are detached from the auxiliary loss)? Related to that, what do you mean by Shallow-Only and Late-Only in your ablations?
-
Can it be that the margins are shrinking in later layers for modifier-bindings because both frisbee and red frisbee are getting closer to the full caption? Can you use a metric that does not compare to the full caption to check this?
-
For the relation-family contrast: all averages are negative (and later layers are more negative on average). At least from this number, I don’t see why target relations are less separable in later layers, as it seems that all layers cannot separate the positives from the negatives. Does it mean that the model lacks relation contrast across all layers?
-
How do you extract the phrase set, and how do you filter this set? Is this time included in your method (30 min)?
-
In Table 1, it is unclear which models to compare with which. Are the fine-grained models and the composition-aware models fine-tuned on CC300k as well? Are they based on SigLIP or on CLIP ViT-B/16? In addition, some composition-aware methods that have better SugarCrepe results are missing, e.g., DAC, READ-CLIP.
-
Do you understand why the compositionality ITT increases are relatively mild? Can these numbers improve if you train on more samples (from the same CC3M dataset)?