- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies whether Structure-CLIP, a CLIP-based model that incorporates scene-graph structure, remains effective when captions contain multiple entities, attributes, and relations rather than simple controlled edits. The authors construct a controlled dataset of 666 examples across six structural categories, split into 350 training, 76 validation, and 240 test examples. Each example contains an image, two semantically equivalent positive captions, and one negative caption that changes a single structural binding. They then propose Complex-SCLIP, which extends Structure-CLIP by training both positive captions to score above the same negative and encouraging the two positives to produce consistent representations. The method improves performance on the held-out test split and shows some transfer to SUGARCREPE++, while slightly reducing COCO retrieval performance.
Critical strengths
-
The paper is clearly written and the setup is easy to follow.
-
The authors provide a controlled comparison with several compositional baselines and include ablations, which makes it easier to understand where the gains come from.
-
The paper is relatively transparent about some limitations, including the small dataset, reuse of the diagnosis set, and the drop in COCO retrieval performance.
Critical weaknesses
-
I do not find the novelty strong enough. The main problem, whether VLMs can distinguish small but important semantic or structural changes, has already been studied in many prior works such as ARO, SugarCrepe/SUGARCREPE++, and other related works.
-
The method itself is also quite limited. Most of the gain comes from adding the second positive caption during training. Table 3 shows that removing the additional consistency losses has almost no effect on the final Both accuracy. This makes it hard to see a substantial methodological contribution.
-
The model is trained on only 350 examples. This is very small, especially since the main test set follows the same P1/P2/N format used during training. Because of this, the strong in-distribution improvement is not very convincing evidence that the model learned a more general structural capability.
-
The out-of-distribution results are also not conclusive. SUGARCREPE++ uses a very similar positive/positive /negative setup, so it is not a very different evaluation setting. On COCO retrieval, the method performs worse than the Structure-CLIP initialization. Overall, I do not think the external evaluation is strong enough to show broad generalization.
Questions for authors
How was the semantic equivalence between P1 and P2 verified? Was there any independent human annotation or agreement measurement?
Since the model is trained on only 350 examples, how do the authors know the gains are not mainly due to fitting the specific P1/P2/N setup?
Thank you for the careful assessment and for recognizing the clear setup, ablations, and stated limitations. We clarify the contribution and provide additional evidence on the training and generalization concerns.
W1: Relation to existing compositional benchmarks. Our benchmark tests whether valid multi-entity descriptions outrank descriptions containing a single edit to an attribute, relation, host assignment, or spatial binding, and whether this ordering survives meaning-preserving reformulation. The design retains non-target facts, while the six-family breakdown helps identify which binding types are difficult. We build on existing fine-grained and dual-positive evaluation, complementing ARO and SUGARCREPE++ through this controlled diagnostic focus rather than claiming a new general-purpose reasoning benchmark.
W2: Role of dual-positive supervision. A single-positive ranking objective does not explicitly constrain an equivalent reformulation. Supervising both valid descriptions against the same negative is therefore central to our adaptation design. The submitted ablations already note the small effects of the consistency regularizers. To match access to P2 while separating this design from the full objective's added value, we trained the released Structure-CLIP objective on (P1,N) and (P2,N), averaging losses before one update. This is an added dual-positive variant, not the unchanged published model. Initialization, image order, positive exposure, frozen visual encoder, optimizer, 160 updates, and three seeds were matched. Both use fixed step-160 checkpoints with test KEE disabled; the control retains original weighting, fusion, and warmup conventions.
With test KEE disabled, Test240 Both is 61.25% for the control and 60.69% for Complex-SCLIP, versus 41.67% for the unchanged starting checkpoint. Both schemes improve over this starting point, showing that the dual-positive adaptation design also works with the original objective, with numerically close Both scores. The full method does not exceed the matched control on Both here; on SUGARCREPE++, however, its ITT and TOT are higher than the control in all three seeds (see W4). We distinguish these findings from the contribution of any individual auxiliary loss.
W3 and the format-fitting question. We use the 350 examples for targeted adaptation of pretrained Structure-CLIP, keeping its visual encoder frozen. This builds on existing representations rather than learning vision-language competence from scratch. As described in Section 4.1, P1 and P2 express the same visual facts in different wording, while N changes one target binding. Supervising both positives against the same negative makes the structural distinction, rather than one particular wording, the intended learning target.
Using this format in both training and evaluation aligns supervision with the property being tested: whether the ranking remains correct under meaning-preserving reformulation. This motivates the design but does not by itself rule out format-specific adaptation. The 240 test records were excluded from fine-tuning, though they informed the earlier diagnosis and are not development-independent. We therefore interpret the results as adaptation within this controlled task; W4 discusses external transfer and further cross-format evaluation.
W4: External transfer and capability preservation. We selected SUGARCREPE++ to examine whether adaptation on our collection transfers to a separate dataset measuring a related compositional task, without further fine-tuning. It shares the P1/P2/N format with our benchmark, as you note.
For a fair comparison, both methods were evaluated with test-time KEE disabled. Compared with the training-matched dual-positive control, Complex-SCLIP improves mean ITT and TOT by 1.02 and 3.87 percentage points across three seeds, respectively, with both metrics higher in every seed. These results support an additional benefit from the full method on related compositional tasks, but alone do not establish broad, format-independent generalization.
COCO retrieval performance is lower than the Structure-CLIP initialization. We therefore regard incomplete preservation of retrieval capability as a limitation of the current adaptation, rather than interpreting gains on a related task as improvements across all tasks.
In future work, we plan to evaluate benchmarks with different task formats to assess transfer beyond the current setting.
Question on P1/P2 equivalence. Two raters independently examined 100 test instances. Both confirmed the intended equivalent positive pair for 97/100 instances; agreement on equivalence was 99% (Cohen's kappa=0.795).