- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces minimal image–caption pairs covering physical-causal, perspectival, and social knowledge, with explicit and implicit captions for the same images. It evaluates shallow cross-modal mappings, CLIP, and generative VLMs against human baselines. Results show weak shallow alignment and stronger multimodal performance, with remaining gaps in social interaction and implicit perspective judgments.
Critical strengths
The explicit/implicit comparison holds visual evidence fixed and provides a useful diagnostic. Evaluation spans multiple model families and includes both human consensus and single-rater baselines.
Critical weaknesses
Benchmark size and coverage are unclear, making its breadth difficult to assess. (Section 2 omits total and per-category item counts; Appendix A.3 reports only annotation totals.)
The claimed minimality of the generated pairs is insufficiently validated; high human accuracy does not rule out unintended matching cues.
there are several pairwise benchmarks need discussion.: MMVP, MLLM-CompBench, and CompareBench. The paper needs to establish what its semantic domains and explicit/implicit captions add beyond existing pairwise evaluations.
Questions for authors
- How many unique image pairs and semantic contrasts does each subset contain?
- Which capabilities are newly tested relative to EqBen, VisMin, and SPEC?
- What fraction of generated pairs failed validation, and why?
Response to Reviewer J97r
We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue.
W1 / Q1: Benchmark size and coverage
The counts were missing from the paper and are now given in the new Appendix A.1.4 (Table 3). The suite has 369 items across six inferential distinctions:
W2 / Q3: Minimality validation (new Appendix A.7)
We added a quality-control appendix (new Appendix A.7) that measures every image pair and every caption pair against baselines of unrelated pairs, to verify that the two members of a pair are highly similar overall and differ only in the targeted way. Image pairs are compared at the pixel level, at the perceptual level (LPIPS, DreamSim) and in the DINOv2 and CLIP embedding spaces used for shallow alignment; caption pairs at the surface level (differing spans and word edits), at the level of sentence meaning (sentence embeddings, natural-language inference) and in the Llama and CLIP text embedding spaces. We found that:
Image pairs are minimal at every level (Figure 11). The matched partner is far closer than any unrelated pair, the edit is confined to a small, connected region of the frame, and in social interaction the partner is closer than other edits of the same scene.
Caption pairs are minimal for both explicit and implicit conditions (Figure 12). Explicit pairs differ at one point in the sentence, share the same frame, and are read as contradictions by an NLI model. Implicit pairs differ at one point fewer, since the explicit caption is the implicit caption plus exactly one span (the property word) in every contrast, and are closer still in embedding similarity.
Outlier flagging and handling is reported in Appendix A.6 Table 4.
W3 / Q2: Related work.
Thank you for the pointers; the revision adds a related-work section with a comparison table (Appendix B, to keep section numbers stable; it moves to Section 2 in the camera-ready). The benchmarks named sit on two sides of our design. MMVP, MLLM-CompBench and CompareBench sort fine-grained failures into visual categories, but compare images that differ in many respects and evaluate through a question, so a failure cannot be attributed to the category under test and a similarity space or alignment map cannot be scored at all. VisMin, EqBen and SPEC supply the control we build on: two images and two captions that differ in one respect, scored as a 2×2 group; we adopt that format directly and now credit EqBen and SPEC as its predecessors.
Our benchmark adds two things to this format. First, it extends the controlled minimal-pair design to new domains, physical-causal, perspectival and, above all, social: no existing benchmark tests social interaction or social inference as controlled image–caption pairs, and social content appears in prior work only as emotion labels in question-answering sets. Second, within each domain it adds a level of inference: the same image pair is tested with the property named and with it left implicit, so how far the alignment reaches beyond what is stated is measured with the visual evidence fixed.