acceptodds
Under review as a conference paper at ICLR 2027

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Abstract

Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute–object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute–object bindings across different modality pairings and more complex settings. We therefore introduce SABER, a controlled benchmark for evaluating attribute–object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. Recent work also shows that capabilities learned through text-only supervision can transfer to visual modalities in MLLMs. We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image–image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
VLM, MLLM, Retrieval, Composition, Attribute-binding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Y1ckOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SABER, a synthetic benchmark for testing attribute–object binding across text and image retrieval. It also proposes text-only fine-tuning with synthetic binding examples. The authors report improved results on SABER and existing binding benchmarks, with little change in overall MMEB-v3 performance.

Critical strengths
  • SABER covers four modality pairings, varied attributes, and scenes with up to ten objects.
  • It evaluates both multiple-choice and large-gallery retrieval.
  • Text-only fine-tuning improves cross-modal binding without requiring image supervision.
  • Gains also appear on existing benchmarks, negation, and instance retrieval.
Critical weaknesses
  • SABER is relatively small. It contains 1,800 queries, fewer than several existing attribute-binding benchmarks. Its coverage of modality pairings and scene complexity is useful, but the limited sample size may make results sensitive to benchmark construction and sampling.
  • The fine-tuning method has limited technical novelty. It applies a standard triplet-loss objective to synthetic text examples, with LoRA and LiNeS for adaptation. The main contribution appears to be the data and evaluation design rather than a new learning method.
  • The gains on SABER raise questions about benchmark difficulty. After text-only fine-tuning, the 8B model reaches 91.1 mean mAP on independent retrieval, compared with 71.9 for the pretrained model. This large improvement, alongside 99.3% text–text accuracy, suggests that SABER may be relatively easy for a model trained on closely related binding examples.
Questions for authors
  • Given that the fine-tuning objective uses a standard triplet loss, what is the paper’s technical contribution beyond the construction of the training data and benchmark?
  • Why does text-only fine-tuning produce such large gains on SABER, particularly the 91.1 mean mAP on independent retrieval and near-saturated text–text accuracy? Can the authors report results on harder held-out binding patterns, attributes, or objects to establish that SABER measures robust generalization?
  • Could the authors evaluate the fine-tuned model on broader compositional-understanding benchmarks, such as SugarCrepe[0] or VALSE[1]? This would help establish whether the gains generalize beyond attribute–object binding and are robust across compositional reasoning tasks.

[0] SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality [1] VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

3 comments
AuthorsOpenReview ↗

Response to Reviewer Y1ck (1/2)

We thank the reviewer for their feedback and for highlighting the scope of our evaluations and the benefits of text-only finetuning.

W1: SABER Benchmark Size and Sampling Sensitivity: SABER contains 1,800 queries, more than several existing attribute-binding benchmarks, such as COLA-Multi-Object (210 examples), BiVLC-Swap-Att. (278), and SugarCrepe-Swap-Att. (666). Beyond its 4-way MCQ evaluation, SABER also supports large-gallery retrieval, evaluating each query against 7,200 candidates.

Regarding coverage, SABER includes all 232 objects and 50 visual attributes in its vocabulary, covering 4,911 (42.3%) of the 11,600 possible attribute–object pairs in queries alone, increasing to 10,176 (87.7%) when including all candidates. This provides substantial diversity in the evaluated attribute–object combinations despite the benchmark's size.

To assess sensitivity to benchmark size, we additionally evaluated five models on progressively larger subsets of SABER, preserving the balance between independent and nested examples across all scene sizes. At 50% of the benchmark (900 queries), accuracies across all models and modality pairings differ by at most 2.4 percentage points from the full evaluation, decreasing to 1.3 points at 80%. These results suggest that the benchmark is sufficiently large to provide stable aggregate performance estimates, with limited expected benefit from substantially increasing the number of evaluation examples. Full subsampling results for all four modality pairings are provided in Tables 1–4 below.

W2 + Q1: Technical Novelty of Our Finetuning Approach: We acknowledge that our finetuning procedure relies on established components, including triplet loss, LoRA, and LiNeS. This simplicity is intentional, as our goal is to demonstrate that attribute–object binding in pretrained multimodal embedders can be substantially improved through text-only supervision, without requiring a new learning objective or any image data.

Nevertheless, to the best of our knowledge, we are the first to combine these components for text-only finetuning of pretrained MLLM-based multimodal embedders. Our LiNeS ablation (Appendix C.2) further demonstrates its importance for preserving general multimodal performance.

Beyond the finetuning approach, our work also contributes SABER, a novel benchmark for evaluating attribute–object binding, alongside extensive experiments examining how improvements in binding generalize across benchmarks and transfer to other tasks, such as negation understanding and instance retrieval, while largely preserving general multimodal performance.

W3 + Q2: SABER Difficulty: The relatively low performance of pretrained models on SABER (52.5% mean MCQ accuracy for the strongest baseline) suggests that the benchmark is not inherently easy. The large gains achieved by our finetuned model may therefore reflect the effectiveness of our targeted supervision, rather than simply low benchmark difficulty. The near-saturated text–text performance is also expected, given that text is the directly supervised modality. Nevertheless, SABER remains challenging in cross-modal settings, where our 8B model achieves only 69.0% and 74.1% MCQ accuracy on text–image and image–text, respectively. Moreover, despite achieving 91.1% mAP on independent retrieval, performance drops to 54.1% on the more challenging nested retrieval setting, compared with 38.3% for the pretrained model. These results indicate that substantial room for improvement remains, even after extensive binding-specific finetuning.

Q2: Generalization to Held-Out Objects, Attributes, and Binding Patterns: To investigate whether success on SABER reflects general attribute-binding ability rather than familiarity with its specific concepts and construction, we consider several complementary experiments.

First, we finetuned a model using none of SABER's 232 objects or 50 attributes (Appendix C.1, Table 11). Despite this complete vocabulary disjointness, the model achieves 82.5% mean MCQ accuracy on SABER, compared with 84.2% for our standard finetuned model. This suggests that high SABER performance does not depend on exposure to its specific vocabulary during finetuning.

Furthermore, our finetuned model outperforms all baselines not only on SABER but also on CLEVR-bind (Appendix H, Figure 7), which evaluates attribute binding using partial descriptions rather than attribute swaps. This provides additional evidence that the improvements measured by SABER extend beyond its specific binding formulation.

AuthorsOpenReview ↗

Response to Reviewer Y1ck (2/2)

Q3: Generalization to Broader Compositional-Understanding Benchmarks: Following the reviewer's suggestion, we additionally evaluated our finetuned model on Winoground, SugarCrepe, and VALSE.

On SugarCrepe, average accuracy improves from 86.8% to 88.5%, with gains in five of seven categories, particularly attribute swaps (+5.0), attribute additions (+3.7), and object swaps (+3.3). On Winoground, performance improves across all three metrics, including Group accuracy (16.5% to 18.5%). On VALSE, average accuracy increases from 80.0% to 80.6%, with notable improvements in coreference (standard: +8.1, clean: +6.8), although performance declines in some categories, including spatial relations and actions.

Overall, despite mixed results across individual categories, average performance improves on all three benchmarks, suggesting that the benefits of our finetuning extend beyond attribute–object binding and negation to broader compositional understanding. Full results are provided in Appendix G.


Tables 1–4 report mean MCQ accuracy (%) across nine scene sizes (2–10 objects), evaluated on progressively larger subsets of SABER. Subsets preserve the balance between independent and nested examples at each scene size.

Table 1: SABER Subsampling (T→T).

Model10%20%30%40%50%60%70%80%90%100%
LABCLIP ViT-B/3252.253.352.852.654.355.655.455.455.455.2
PE-Core bigG-14-44861.766.465.065.365.067.166.766.465.766.1
LamRA-Ret50.647.547.247.548.449.650.649.549.349.1
Qwen3-VL-Embedding-8B60.655.655.253.953.755.554.653.853.653.7
Ours-8B98.999.499.499.499.399.499.499.399.399.3

Table 2: SABER Subsampling (T→I).

Model10%20%30%40%50%60%70%80%90%100%
LABCLIP ViT-B/3227.223.924.323.823.622.722.122.122.822.8
PE-Core bigG-14-44830.627.526.129.028.727.628.829.129.529.7
LamRA-Ret34.432.231.933.833.031.931.730.930.930.6
Qwen3-VL-Embedding-8B35.636.935.936.036.035.434.734.935.134.9
Ours-8B68.968.370.470.368.368.268.068.168.669.0

Table 3: SABER Subsampling (I→T).

Model10%20%30%40%50%60%70%80%90%100%
LABCLIP ViT-B/3226.725.025.925.125.025.624.824.724.324.2
PE-Core bigG-14-44826.130.028.928.727.227.426.627.427.727.6
LamRA-Ret33.329.730.028.627.428.027.826.626.627.0
Qwen3-VL-Embedding-8B43.340.637.438.137.137.436.937.636.936.3
Ours-8B72.875.375.776.274.874.874.374.373.674.1

Table 4: SABER Subsampling (I→I).

Model10%20%30%40%50%60%70%80%90%100%
LABCLIP ViT-B/3274.475.875.975.875.275.375.274.375.175.3
PE-Core bigG-14-44872.871.472.871.770.069.068.967.767.867.6
LamRA-Ret61.765.364.864.964.964.865.565.365.364.9
Qwen3-VL-Embedding-8B86.785.886.186.285.484.984.884.984.984.9
Ours-8B97.294.795.795.394.894.294.494.494.494.3

Thanks for the author's response, which resolved most of my concerns and i have adjusted my score.

Reviewer_QtJJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is concerned with the retrieval problem in VLMs. In particular, it focuses on attribute-object binding, the ability of a model to recognize which attribute belongs to which object. The paper's main contributions are 1) the SABER benchmark and 2) text-only fine-tuning of VLMs. Compared to previous work, SABER offers evaluations across four modality pairings (T2T, I2I, I2T, T2I), scenes with two to ten objects, diverse attribute types and objects (synthetically generated), and nested binding evaluation. The authors observe that VLMs struggle with binding on the SABER benchmark and propose text-only fine-tuning. The authors show that this offers substantial gains in various binding benchmarks while also handling negation and instance-level retrieval.

Critical strengths

The authors provide a realistic (but synthetically generated) dataset with multiple objects in a scene and corresponding negative samples. To my knowledge, only a limited number of datasets make the comparison between positive and negative bindings in multi-object scenes explicit. Thus, the community would appreciate a new, clean way to evaluate attribute-object binding in VLMs.

The provided model performs well on various benchmarks. The authors additionally provide performance on negation and instance retrieval tasks, and make sure the general performance does not degrade as a result of binding-specific training. Therefore, the evaluations seem complete.

Critical weaknesses

Text-only fine-tuning achieves good results, but it is not clear which component of the method is most responsible for the improvements. Would the method perform similarly if it used a different backbone or different data? Could the other methods in Table 2 achieve comparable results if they were fine-tuned on the same data? I think the authors need to elaborate on the main novel contribution of their method.

The authors should also discuss the limitations of their method more extensively. The limitations of the dataset are mentioned. However, the limitations of the proposed text-only fine-tuning were not discussed.

Questions for authors
  1. Table 1 seems to portray biased comparisons to previous datasets. Could the authors explain why all other datasets except theirs have crossed out retrieval? Retrieval can be performed on synthetic datasets such as CLEVR and PUG:SPAR and PUG:SPARE, unlike SugarCrepe, where the possible captions are fixed. Similarly, T2T and I2I could be constructed from these datasets as there are no fixed queries. Could the authors explain their decision-making in constructing Table 1?

  2. Could the dataset be extended to cover the spatial relations of the objects? If such metadata exists and is made available, even harder forms of retrieval could be evaluated, or models could be improved.

  3. Does the benchmark include full coverage of all attributes and all objects? From my understanding, it does not. Could the author elaborate on whether they considered this, and why they limited the number of independent examples to 100 (+100 for nested chains)? Full coverage could be very helpful for a deeper analysis of binding in VLMs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QtJJ (1/2)

We thank the reviewer for their helpful feedback. We appreciate their recognition of SABER's value as a benchmark for attribute–object binding, as well as the completeness of our evaluations.

W1: Role of Model Architecture versus Finetuning Supervision: Our approach is not specific to Qwen3-VL-Embedding. In Appendix D (Table 14), we apply the same finetuning recipe to UEmbed and WeMM-Embedding, each at 2B and 9B parameters. All four models improve across all modality pairings, with mean SABER accuracy gains of 18.2–26.7 percentage points, including image–image improvements of 4.9–14.9 points despite exclusively textual supervision.

Regarding training data, our ablations (Appendix C.1, Table 11) show that excluding all SABER objects and attributes from finetuning still yields 82.5% mean SABER accuracy, compared with 84.2% for our standard model. In contrast, finetuning on the general-purpose AllNLI dataset using the same recipe achieves only 48.9%, highlighting the importance of binding-specific supervision (see response to wvpf).

Finally, unlike MLLM-based embedders, CLIP-style dual encoders use separate text and image encoders. Consequently, text-only finetuning cannot directly improve their image representations or image–image retrieval performance. While such models may benefit in other modality pairings, they cannot exploit the same cross-modal transfer mechanism.

W1: Main Contributions and Novelty: We highlight several contributions of our work. First, we introduce SABER, a novel attribute-binding benchmark covering all four modality pairings, scenes with 2–10 objects, and both MCQ and large-gallery retrieval. We also introduce a synthetic text-only finetuning dataset designed to improve attribute–object binding.

Second, we demonstrate that finetuning pretrained MLLM-based multimodal embedders exclusively on textual data can substantially improve attribute–object binding across both text and image modalities, despite the absence of visual supervision. These improvements extend beyond SABER to existing attribute-binding benchmarks, while largely preserving general multimodal performance.

Finally, we show that binding-focused finetuning also improves performance on other tasks, including negation understanding and instance retrieval, suggesting that stronger attribute–object binding can benefit broader retrieval capabilities.

W2: Limitations of Text-Only Finetuning: We agree that the limitations of our approach deserve further discussion and have expanded this discussion in the revised paper.

Architectural Dependence: Our approach relies on shared trainable parameters across modalities, as in MLLM-based embedders. It is therefore not directly applicable to CLIP-style dual encoders, where text-only finetuning cannot modify image representations or improve image–image retrieval.

Limitations of Textual Supervision: Our approach is best suited to capabilities that can be expressed and supervised through language, such as attribute–object binding. However, inherently visual tasks, such as depth estimation, may require visual supervision that text alone cannot provide.

Q1: Clarification of Table 1: The modality columns in Table 1 reflect the evaluation protocols reported in the original benchmark papers, rather than all possible extensions of the underlying datasets. We agree that synthetic datasets such as CLEVR, PUG:SPAR, and PUG:SPARE could be extended to additional modality pairings, including T→T and I→I. However, their published matching evaluations focus on I→T. We have clarified this distinction in the table caption.

Regarding retrieval, our intention was to distinguish multi-positive retrieval from single-positive gallery retrieval, which can be constructed by pooling the candidates of almost any MCQ benchmark. We agree that CLEVR, PUG:SPAR, and PUG:SPARE can support multi-positive retrieval, as multiple images may depict the same attribute–object configuration. We have therefore updated their retrieval entries and clarified the intended definition in the table caption.

Q2: Extending SABER to Spatial Relations: We agree that incorporating spatial relations could enable additional retrieval evaluations. While SABER does not currently include spatial metadata, off-the-shelf object detection models could be used to localize objects and derive spatial relationships, such as above, below, left, and right. However, as our current work focuses on attribute–object binding rather than spatial relationships, we leave this extension for future work.

AuthorsOpenReview ↗

Response to Reviewer QtJJ (2/2)

Q3: SABER Vocabulary Coverage and Benchmark Size: We verified the coverage of the benchmark and found that SABER includes all 50 attributes and all 232 objects in both its queries and answers. Regarding attribute–object combinations, the benchmark covers 4,911 (42.34%) of the 11,600 possible pairs in queries alone, increasing to 10,176 pairs (87.72%) when including all candidates. Thus, while SABER does not exhaustively cover every possible pairing, it achieves substantial coverage of the attribute–object space.

Regarding benchmark size, we chose to balance the benchmark between 100 independent examples and 100 nested-chain examples for each scene size (2–10 objects). Independent examples provide greater diversity and coverage of attribute–object combinations, while nested chains preserve earlier bindings as scene complexity increases. Although less diverse, nested chains enable a more realistic and challenging retrieval setting, where relevant scenes may contain additional objects and attributes beyond those specified in the query. This allows us to evaluate containment and the retrieval of valid supersets, rather than only exact matches.

The primary limitation on the number of examples was the cost of generating high-quality images using GPT-Image-2. SABER includes 9,000 generated images, all manually inspected to ensure correct object and attribute rendering. This made substantially expanding the benchmark costly in both image generation and manual validation.

I thank the authors for the additional experiments and explanations, which have addressed most of my concerns. After considering the other reviews, I have decided to maintain my original score of weak accept.

Reviewer_wvpfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SABER, a synthetic benchmark for attribute-object binding. Each multiple-choice question has a query and four candidates, one preserving the bindings and three changing them. The objects and attributes stay fixed except for one negative in two-object scenes. Every item exists as a templated caption and a generated CLEVR-style image, so all four text and image pairings are evaluated, and the candidates are pooled for gallery retrieval where nested scenes give extra positives. Existing dual encoders and MLLM embedders struggle cross-modally. The authors then finetune Qwen3-VL-Embedding with a small LoRA on text-only triplets generated by Gemma 4, where the negative permutes the attributes or the objects of the positive, and apply LiNeS to limit forgetting. The finetuned model improves substantially on SABER and on existing binding benchmarks, keeps its general MMEB-v3 performance, and also improves on negation and instance retrieval. The appendix ablates the data, LiNeS and an attention loss, applies the recipe to other embedders, and compares text with image supervision.

Critical strengths

S1) The benchmark fills a gap. It combines all four modality pairings, up to ten bindings per scene and gallery retrieval with multiple positives. The construction is well described and the containment finding on the nested subset is interesting on its own.

S2) Text-only binding supervision is a neat idea. Bindings are explicit in text and the hard negatives are exact permutations of the positive, which is hard to get with images. It is interesting that a 3.8M-parameter LoRA trained this way for one epoch, with the vision tower frozen, transfers to image-text and image-image retrieval, to other embedders, and even to instance retrieval, while MMEB-v3 is preserved after LiNeS.

S3) The paper is quite complete. Many questions that came up while reading are answered by an experiment, mostly in the appendix: vocabulary overlap with SABER, the LiNeS trade-off, an attention loss, other embedders, text versus image supervision, instruction prompting, negation terms in the training data, and two held-out sentence constructions.

Critical weaknesses

W1) The broader gains might not come from binding supervision. The finetuning adapts the embedder with a triplet loss and uses binding-swap negatives, and there is no control with the same setup but non-binding negatives of similar difficulty. A general effect of text-only finetuning is also likely. Appendix F.2 shows better within-class retrieval but does not show that binding is the cause. The instance benchmarks are also run without distractor sets (Appendix F.1), including the 100M distractors of ILIAS. The protocol is the same for both models, but the gains at large gallery sizes are not tested.

Minor weaknesses:

MW1) Related work is missing. Cola [1] evaluates multi-object attribute binding in text-to-image retrieval and belongs in Table 1. TARA [2] already adapts MLLMs into embedders with text-only triplets and hard negatives and shows transfer to video retrieval. The paper should discuss how its recipe relates.

[1] Cola: A Benchmark for Compositional Text-to-image Retrieval, Ray et al., NeurIPS 2023 Datasets and Benchmarks

[2] Adapting MLLMs for Nuanced Video Retrieval (TARA), Bagad & Zisserman, ECCV 2026

MW2) The sub-chance text-text scores at two objects (Qwen3-VL-Embedding-2B 5.0%, chance 25%) are not explained. If the positive must reverse the query's pair order, each swap negative keeps either the query's object or attribute order, which is a possible order cue. Please state how the order is sampled and which negative is chosen.

MW3) The LiNeS values and how they were selected should be stated, "the default setting of Wang et al. (2025)" is ambiguous since that paper uses different alpha/beta per experiment. The 40 binding-specific instructions of Appendix H should also be listed. Comparing with a uniform scaling of the LoRA delta, as in WiSE-FT or CLoVe [3], would show whether the depth schedule matters.

[3] CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models, Castro et al., arXiv 2024

Questions for authors

Q1) What do NegBench, the instance benchmarks and MMEB-v3 give with the same setup but non-binding negatives of similar difficulty?

Q2) Is the binding-preserving candidate ever in the same pair order as the query, and which negatives do the sub-chance models select? Does the failure persist after controlling for word-order cues and exact query-positive text matches?

Q3) What LiNeS parameters were used, how were they selected, and can the 40 binding-specific instructions be provided?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wvpf (1/2)

We thank the reviewer for their helpful feedback. We appreciate their recognition of SABER's contributions and the completeness of our work.

W1 + Q1: Non-binding Finetuning Control: To test whether our gains arise from generic text-only finetuning, we finetuned Qwen3-VL-Embedding-8B on a comparably sized subset of AllNLI [1,2,3], which provides general semantic supervision with negatives of comparable difficulty. We used the same finetuning recipe as for our model.

We evaluated this model on SABER, MMEB-v3, NegBench, and instance retrieval.

The results show that AllNLI finetuning does not reproduce our improvements:

  • SABER: AllNLI reduces mean MCQ accuracy from 52.5% to 48.9%, compared with 84.2% for our model. Retrieval mAP similarly decreases on both independent (71.9% to 69.0%) and nested (38.3% to 37.1%) queries, whereas ours achieves 91.1% and 54.1%, respectively.
  • MMEB-v3: AllNLI reduces the overall score by 2.1 points (54.7 to 52.6), whereas our model improves it by 0.5 points (54.7 to 55.2).
  • NegBench: AllNLI improves by 1.4/4.2 points on COCO/VOC, compared with 9.3/10.5 for our model.
  • Instance Retrieval: AllNLI reduces performance across all nine settings by an average of 1.8 mAP points, whereas our model improves across all nine by an average of 5.1 points.

Together, these results suggest that the broader improvements on negation and instance retrieval are not simply a consequence of text-only finetuning, but are driven by our binding-specific supervision. The full tables can be seen at the end of the comment.

[1] Reimers, Nils, and Iryna Gurevych. "Sentence-bert: Sentence embeddings using siamese bert-networks." Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.

[2] Bowman, Samuel R., et al. "A large annotated corpus for learning natural language inference." Proceedings of the 2015 conference on empirical methods in natural language processing. 2015.

[3] Williams, Adina, Nikita Nangia, and Samuel R. Bowman. "A broad-coverage challenge corpus for sentence understanding through inference." Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long papers). 2018.

W1: Instance Retrieval Without Distractors: Following previous work [4], we exclude auxiliary distractor sets to reduce computational overhead and better expose differences in fine-grained instance discrimination. Both models are evaluated under identical conditions, ensuring a controlled comparison. We have clarified this motivation in Appendix F.1 of the revised paper.

[4] Green, Michael, et al. "Find your needle: Small object image retrieval via multi-object attention optimization." NeurIPS 2025.

MW1: COLA: We have added COLA's multi-object subset to Table 1 and evaluated several models on it in Table 3. Our finetuned 8B model achieves the best performance in text–image retrieval and the second-best performance in image–text retrieval.

MW1: Relationship to TARA: We agree that the recently published TARA method is related to our approach and have cited it on page 3 of the revised paper. Both methods use text-only supervision with hard negatives to improve multimodal representations, but differ in their training objectives. TARA uses a contrastive objective with in-batch negatives, whereas we use a margin-based triplet loss with two binding-specific hard negatives that preserve the underlying objects and attributes. We additionally apply LiNeS to preserve general multimodal embedding performance.

MW2 + Q2: Qwen3-VL-Embedding-2B Performance on SABER (Text–Text, 2 objects): Your understanding of the two-object MCQ construction is correct. The positive candidate preserves the query's bindings in reversed order, while the attribute-swapped and object-swapped candidates contain the same incorrect bindings in different orders. The fourth candidate preserves one binding while replacing an object or attribute in the other.

Qwen3-VL-Embedding (2B) strongly favors the swapped candidates, selecting attribute swaps 55% of the time and object swaps 40%, compared with 5% for the correct candidate and 0% for the replacement candidate. We hypothesize that this reflects a word-order bias: each swapped candidate preserves either the attribute or object order of the query, whereas the positive reverses both.

We deliberately avoid exact query–positive text matches, which would trivially yield a cosine similarity of 1. Unlike the two-object setting, scenes with 3–10 objects allow for more diverse permutations, reducing systematic word-order cues. In these settings, the extreme failure is substantially reduced, with accuracy ranging from 30% to 47.5%, consistently above the 25% chance level.

AuthorsOpenReview ↗

Response to Reviewer wvpf (2/2)

MW3 + Q3: LiNeS Configuration: We use alpha=0 and beta=1, corresponding to the default arguments in the LiNeS pseudocode provided in Appendix A of the LiNeS paper. These define a linear scaling schedule from 0 at the first decoder layer to 1 at the last. We apply this fixed configuration across all models and have added these values to Appendix B.5.

Regarding uniform scaling, we do not expect the precise depth-dependent schedule to be critical to our results. Our goal is not to optimize the scaling strategy, but to demonstrate that even a simple schedule can substantially help preserve general multimodal performance.

MW3 + Q3: 40 Binding-Specific Instructions: We added the list of instructions used to Tables 20 and 21 in the Appendix.


Table 1: AllNLI ablation on SABER. Mean MCQ accuracy (%) and retrieval mAP (%) over 2–10 objects, averaged across all four modality pairings. Best results are shown in bold.

ModelMCQIndependentNested
Qwen3-VL-Embedding-8B52.571.938.3
Ours-8B84.291.154.1
Qwen3-VL-Embedding-8B-ALLNLI48.969.037.1

Table 2: AllNLI ablation on MMEB-v3. Average Hit@1 (%) for image, video, and agent tasks, and NDCG@5 (%) for VisDoc and text tasks. Best results are shown in bold.

ModelImageVideoVisDocTextAgentAll
# Datasets3718275347182
Qwen3-VL-Embedding-8B73.759.674.343.739.154.7
Ours-8B72.558.775.443.142.355.2
Qwen3-VL-Embedding-8B-ALLNLI73.059.167.042.637.052.6

Table 3: AllNLI ablation on NegBench. MCQ accuracy (%) by correct-statement type on the COCO and VOC 2007 subsets. Best results are shown in bold.

NegBench (COCO)NegBench (VOC)
ModelAffirmationNegationHybridAllAffirmationNegationHybridAll
Qwen3-VL-Embedding-8B78.36.836.341.489.94.758.039.8
Ours-8B83.46.858.550.791.86.179.650.3
Qwen3-VL-Embedding-8B-ALLNLI89.94.930.542.898.04.465.144.0

Table 4: AllNLI ablation on instance retrieval. Mean average precision (mAP, %) across image–image (first eight columns) and text–image (last column) benchmarks. For INSTRE, S and M denote single- and multi-object subsets, respectively. Best results are shown in bold.

ModelILIASINSTRE S→SINSTRE M→SINSTRE M→S+MPODSParis6KOxford5KSculptures6KILIAS (T→I)
Qwen3-VL-Embedding-8B58.592.881.375.685.786.861.350.852.3
Ours-8B68.494.986.779.088.188.964.359.660.8
Qwen3-VL-Embedding-8B-ALLNLI58.291.179.675.085.285.558.547.748.1

I would like to thank the authors for the additional clarifications and experiments. The Cola results are meaningful addition. The clarified LiNeS configuration and the instructions disclosed in the appendix ensure easier reproduction of the work. As for the additional results with AllNLI finetuning, they indicate that generic finetuning would not yield similar improvements as the proposed binding method. Overall, my main concern was addressed and I would recommend acceptance, keeping my original score of weak accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their feedback. We are glad that they recognize SABER as addressing an important gap in the evaluation of attribute-object binding [wvpf, QtJJ]. Additionally, we appreciate their acknowledgment of the effectiveness of our text-only fine-tuning approach [wvpf, QtJJ, Y1ck], as well as the breadth and thoroughness of our experimental evaluations and analyses [wvpf, QtJJ].

In response to the reviewers' suggestions, we conducted three additional experiments:

  • Non-binding finetuning control: We finetuned on AllNLI using the same training recipe and comparably sized data, showing that general-purpose textual supervision does not reproduce our improvements on SABER, negation understanding, or instance retrieval.
  • Broader compositional understanding: We evaluated our model on Winoground, SugarCrepe, and VALSE, observing improvements in average performance across all three benchmarks, although results vary across individual categories.
  • COLA evaluation: We added COLA-Multi-Object to our evaluations, where our finetuned model achieves the best text–image performance and second-best image–text performance.

We also revised the manuscript to address concerns raised across the reviews, including clarifying the benchmark comparisons in Table 1, expanding the related-work and limitations discussions, and providing additional details on LiNeS, binding-specific instructions, and benchmark construction.

We thank the reviewers again for their constructive feedback, which has helped us strengthen and clarify the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.