- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SABER, a synthetic benchmark for testing attribute–object binding across text and image retrieval. It also proposes text-only fine-tuning with synthetic binding examples. The authors report improved results on SABER and existing binding benchmarks, with little change in overall MMEB-v3 performance.
Critical strengths
- SABER covers four modality pairings, varied attributes, and scenes with up to ten objects.
- It evaluates both multiple-choice and large-gallery retrieval.
- Text-only fine-tuning improves cross-modal binding without requiring image supervision.
- Gains also appear on existing benchmarks, negation, and instance retrieval.
Critical weaknesses
- SABER is relatively small. It contains 1,800 queries, fewer than several existing attribute-binding benchmarks. Its coverage of modality pairings and scene complexity is useful, but the limited sample size may make results sensitive to benchmark construction and sampling.
- The fine-tuning method has limited technical novelty. It applies a standard triplet-loss objective to synthetic text examples, with LoRA and LiNeS for adaptation. The main contribution appears to be the data and evaluation design rather than a new learning method.
- The gains on SABER raise questions about benchmark difficulty. After text-only fine-tuning, the 8B model reaches 91.1 mean mAP on independent retrieval, compared with 71.9 for the pretrained model. This large improvement, alongside 99.3% text–text accuracy, suggests that SABER may be relatively easy for a model trained on closely related binding examples.
Questions for authors
- Given that the fine-tuning objective uses a standard triplet loss, what is the paper’s technical contribution beyond the construction of the training data and benchmark?
- Why does text-only fine-tuning produce such large gains on SABER, particularly the 91.1 mean mAP on independent retrieval and near-saturated text–text accuracy? Can the authors report results on harder held-out binding patterns, attributes, or objects to establish that SABER measures robust generalization?
- Could the authors evaluate the fine-tuned model on broader compositional-understanding benchmarks, such as SugarCrepe[0] or VALSE[1]? This would help establish whether the gains generalize beyond attribute–object binding and are robust across compositional reasoning tasks.
[0] SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality [1] VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
3 comments
Response to Reviewer Y1ck (2/2)
Q3: Generalization to Broader Compositional-Understanding Benchmarks: Following the reviewer's suggestion, we additionally evaluated our finetuned model on Winoground, SugarCrepe, and VALSE.
On SugarCrepe, average accuracy improves from 86.8% to 88.5%, with gains in five of seven categories, particularly attribute swaps (+5.0), attribute additions (+3.7), and object swaps (+3.3). On Winoground, performance improves across all three metrics, including Group accuracy (16.5% to 18.5%). On VALSE, average accuracy increases from 80.0% to 80.6%, with notable improvements in coreference (standard: +8.1, clean: +6.8), although performance declines in some categories, including spatial relations and actions.
Overall, despite mixed results across individual categories, average performance improves on all three benchmarks, suggesting that the benefits of our finetuning extend beyond attribute–object binding and negation to broader compositional understanding. Full results are provided in Appendix G.
Tables 1–4 report mean MCQ accuracy (%) across nine scene sizes (2–10 objects), evaluated on progressively larger subsets of SABER. Subsets preserve the balance between independent and nested examples at each scene size.
Table 1: SABER Subsampling (T→T).
| Model | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|
| LABCLIP ViT-B/32 | 52.2 | 53.3 | 52.8 | 52.6 | 54.3 | 55.6 | 55.4 | 55.4 | 55.4 | 55.2 |
| PE-Core bigG-14-448 | 61.7 | 66.4 | 65.0 | 65.3 | 65.0 | 67.1 | 66.7 | 66.4 | 65.7 | 66.1 |
| LamRA-Ret | 50.6 | 47.5 | 47.2 | 47.5 | 48.4 | 49.6 | 50.6 | 49.5 | 49.3 | 49.1 |
| Qwen3-VL-Embedding-8B | 60.6 | 55.6 | 55.2 | 53.9 | 53.7 | 55.5 | 54.6 | 53.8 | 53.6 | 53.7 |
| Ours-8B | 98.9 | 99.4 | 99.4 | 99.4 | 99.3 | 99.4 | 99.4 | 99.3 | 99.3 | 99.3 |
Table 2: SABER Subsampling (T→I).
| Model | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|
| LABCLIP ViT-B/32 | 27.2 | 23.9 | 24.3 | 23.8 | 23.6 | 22.7 | 22.1 | 22.1 | 22.8 | 22.8 |
| PE-Core bigG-14-448 | 30.6 | 27.5 | 26.1 | 29.0 | 28.7 | 27.6 | 28.8 | 29.1 | 29.5 | 29.7 |
| LamRA-Ret | 34.4 | 32.2 | 31.9 | 33.8 | 33.0 | 31.9 | 31.7 | 30.9 | 30.9 | 30.6 |
| Qwen3-VL-Embedding-8B | 35.6 | 36.9 | 35.9 | 36.0 | 36.0 | 35.4 | 34.7 | 34.9 | 35.1 | 34.9 |
| Ours-8B | 68.9 | 68.3 | 70.4 | 70.3 | 68.3 | 68.2 | 68.0 | 68.1 | 68.6 | 69.0 |
Table 3: SABER Subsampling (I→T).
| Model | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|
| LABCLIP ViT-B/32 | 26.7 | 25.0 | 25.9 | 25.1 | 25.0 | 25.6 | 24.8 | 24.7 | 24.3 | 24.2 |
| PE-Core bigG-14-448 | 26.1 | 30.0 | 28.9 | 28.7 | 27.2 | 27.4 | 26.6 | 27.4 | 27.7 | 27.6 |
| LamRA-Ret | 33.3 | 29.7 | 30.0 | 28.6 | 27.4 | 28.0 | 27.8 | 26.6 | 26.6 | 27.0 |
| Qwen3-VL-Embedding-8B | 43.3 | 40.6 | 37.4 | 38.1 | 37.1 | 37.4 | 36.9 | 37.6 | 36.9 | 36.3 |
| Ours-8B | 72.8 | 75.3 | 75.7 | 76.2 | 74.8 | 74.8 | 74.3 | 74.3 | 73.6 | 74.1 |
Table 4: SABER Subsampling (I→I).
| Model | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|
| LABCLIP ViT-B/32 | 74.4 | 75.8 | 75.9 | 75.8 | 75.2 | 75.3 | 75.2 | 74.3 | 75.1 | 75.3 |
| PE-Core bigG-14-448 | 72.8 | 71.4 | 72.8 | 71.7 | 70.0 | 69.0 | 68.9 | 67.7 | 67.8 | 67.6 |
| LamRA-Ret | 61.7 | 65.3 | 64.8 | 64.9 | 64.9 | 64.8 | 65.5 | 65.3 | 65.3 | 64.9 |
| Qwen3-VL-Embedding-8B | 86.7 | 85.8 | 86.1 | 86.2 | 85.4 | 84.9 | 84.8 | 84.9 | 84.9 | 84.9 |
| Ours-8B | 97.2 | 94.7 | 95.7 | 95.3 | 94.8 | 94.2 | 94.4 | 94.4 | 94.4 | 94.3 |
Thanks for the author's response, which resolved most of my concerns and i have adjusted my score.
Response to Reviewer Y1ck (1/2)
We thank the reviewer for their feedback and for highlighting the scope of our evaluations and the benefits of text-only finetuning.
W1: SABER Benchmark Size and Sampling Sensitivity: SABER contains 1,800 queries, more than several existing attribute-binding benchmarks, such as COLA-Multi-Object (210 examples), BiVLC-Swap-Att. (278), and SugarCrepe-Swap-Att. (666). Beyond its 4-way MCQ evaluation, SABER also supports large-gallery retrieval, evaluating each query against 7,200 candidates.
Regarding coverage, SABER includes all 232 objects and 50 visual attributes in its vocabulary, covering 4,911 (42.3%) of the 11,600 possible attribute–object pairs in queries alone, increasing to 10,176 (87.7%) when including all candidates. This provides substantial diversity in the evaluated attribute–object combinations despite the benchmark's size.
To assess sensitivity to benchmark size, we additionally evaluated five models on progressively larger subsets of SABER, preserving the balance between independent and nested examples across all scene sizes. At 50% of the benchmark (900 queries), accuracies across all models and modality pairings differ by at most 2.4 percentage points from the full evaluation, decreasing to 1.3 points at 80%. These results suggest that the benchmark is sufficiently large to provide stable aggregate performance estimates, with limited expected benefit from substantially increasing the number of evaluation examples. Full subsampling results for all four modality pairings are provided in Tables 1–4 below.
W2 + Q1: Technical Novelty of Our Finetuning Approach: We acknowledge that our finetuning procedure relies on established components, including triplet loss, LoRA, and LiNeS. This simplicity is intentional, as our goal is to demonstrate that attribute–object binding in pretrained multimodal embedders can be substantially improved through text-only supervision, without requiring a new learning objective or any image data.
Nevertheless, to the best of our knowledge, we are the first to combine these components for text-only finetuning of pretrained MLLM-based multimodal embedders. Our LiNeS ablation (Appendix C.2) further demonstrates its importance for preserving general multimodal performance.
Beyond the finetuning approach, our work also contributes SABER, a novel benchmark for evaluating attribute–object binding, alongside extensive experiments examining how improvements in binding generalize across benchmarks and transfer to other tasks, such as negation understanding and instance retrieval, while largely preserving general multimodal performance.
W3 + Q2: SABER Difficulty: The relatively low performance of pretrained models on SABER (52.5% mean MCQ accuracy for the strongest baseline) suggests that the benchmark is not inherently easy. The large gains achieved by our finetuned model may therefore reflect the effectiveness of our targeted supervision, rather than simply low benchmark difficulty. The near-saturated text–text performance is also expected, given that text is the directly supervised modality. Nevertheless, SABER remains challenging in cross-modal settings, where our 8B model achieves only 69.0% and 74.1% MCQ accuracy on text–image and image–text, respectively. Moreover, despite achieving 91.1% mAP on independent retrieval, performance drops to 54.1% on the more challenging nested retrieval setting, compared with 38.3% for the pretrained model. These results indicate that substantial room for improvement remains, even after extensive binding-specific finetuning.
Q2: Generalization to Held-Out Objects, Attributes, and Binding Patterns: To investigate whether success on SABER reflects general attribute-binding ability rather than familiarity with its specific concepts and construction, we consider several complementary experiments.
First, we finetuned a model using none of SABER's 232 objects or 50 attributes (Appendix C.1, Table 11). Despite this complete vocabulary disjointness, the model achieves 82.5% mean MCQ accuracy on SABER, compared with 84.2% for our standard finetuned model. This suggests that high SABER performance does not depend on exposure to its specific vocabulary during finetuning.
Furthermore, our finetuned model outperforms all baselines not only on SABER but also on CLEVR-bind (Appendix H, Figure 7), which evaluates attribute binding using partial descriptions rather than attribute swaps. This provides additional evidence that the improvements measured by SABER extend beyond its specific binding formulation.