acceptodds
Under review as a conference paper at ICLR 2027

CoAMA: A Hierarchical Benchmark for Diagnosing Cross-Object Attribute Misbinding in Vision-Language Models

Abstract

Large vision-language models (LVLMs) are increasingly used for visual understanding and decision-making, yet they may still produce plausible but incorrect visual interpretations, a phenomenon known as hallucination. In multi-object scenes, such failures can occur when an attribute belonging to one object is incorrectly assigned to another, undermining reliable scene understanding and downstream reasoning. However, existing hallucination benchmarks mainly evaluate object existence, class recognition, or attribute presence, while largely overlooking whether perceived attributes are correctly bound to their corresponding object instances. To address this gap, we introduce Cross-object Attribute Misbinding Assessment (CoAMA), a hierarchical diagnostic benchmark for evaluating instance-level attribute binding through three progressive levels: multi-object instance recognition, target-object attribute recognition, and cross-object attribute binding verification. Its core design employs paired in-image attribute-swap probes, where each negative probe transfers a ground-truth attribute from one marked object to another, directly testing whether the model preserves instance-level attribute ownership when multiple attributes are visually present in the same scene. CoAMA contains 787 multi-object images, 3,935 marked instances, and 19,460 scorable attributes, with standardized annotation and scoring protocols for consistent and reproducible evaluation. We evaluate representative LVLMs and find that even the best-performing model, Qwen3.7-Plus, achieves only 67.69% PairAcc under L3-Oracle-Direct, while most models exhibit higher false-acceptance rates for attributes belonging to other marked objects than for attributes absent from the marked-object set. These findings reveal cross-object attribute misbinding as a critical reliability challenge and provide a controlled foundation for improving LVLM reliability in complex visual environments.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Multimodal Large Language Models, Visual Hallucination, Attribute Binding, Cross-Object Attribute Misbinding, Visual Grounding, Benchmark, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6YqqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CoAMA, a benchmark for evaluating cross-object attribute misbinding in LVLMs. The benchmark contains 787 images, 3,935 marked instances, and 19,460 scorable attributes, with 2,535 pairs in the primary Level 3 evaluation. Experiments on various models reveals a substancial gaps, especially in attribute binding, with attributes belonging to other objects more likely to be falsely accepted than absent attributes.

Critical strengths
    1. Explicitly marked target–source pairs provide a clear way to evaluate attribute ownership, particularly when multiple objects share the same category.
    1. The benchmark includes human-verified attribute annotations and several diagnostic settings, making it a reliable and valuable resource for studying this failure mode.
Critical weaknesses
    1. Semantic confounds and language priors weaken the interpretation of binding errors: attribute labels are not always mutually exclusive, moreover, some cases in Level3 may reflect differences in object–attribute plausibility(which is affected by language priors). For instance, the model may correctly figure out a banana is not red, but maybe the choice is made based on knowledge, not on visual evidences. Though discussed in the appendix, the authors failed to give satisfing solutions.
    1. The evaluation does not fully disentangle Level 3 binding errors from recognition errors at Levels 1 and 2. Reporting Level 3 accuracy on the subset where the model correctly answers the corresponding Level 1 and Level 2 questions would provide more convincing evidence.
Questions for authors
  • Source masking yields only a marginal improvement. Does it significantly reduce false acceptance of the swapped attribute? How should this result be reconciled with the interpretation that the source object causes visual interference?
  • Beyond the existing attribute and scene breakdowns, can the authors provide deeper analysis of what drives binding errors, for example through attention or token-level analysis? It would also be useful to examine how these errors change with the number of objects, model size, and reasoning budget.
  • Does the binding failure persist in a multi-turn setting where the model first identifies the objects and their attributes, then answers the Level 3 question?
Reviewer_Wc8cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Coama is a benchmark to diagnose hallucinations connected to attributes attribution in images. The thesis of the authors is that MLLM tends to misattribute attributes in images containing multiple objects. To verify this they build an ad hoc benchmark and create positive and hallucinated pairs of questions like “Does <object X> has <attribute y>? yes/no” and use the generated samples to check whether models answer consistently. The GT labels for the task have been generated by human annotations. The authors evaluate multiple models (closed and open source) and few variations of the proposed task. The outcome is that current models still struggle in these settings. Few methodological flaws in my opinion exists in the paper and will be covered in the rest of the review.

Critical strengths
  • Intuitive hierarchical structure: Organizing the evaluation into object recognition, single-object attribute prediction, and paired cross-object attribute verification provides a logical progression on the same set of images.

  • High inter-annotator agreement: The manual double-annotation and adjudication process achieves strong agreement between the two annotators on the closed-set vocabulary.

Critical weaknesses

A. Target-crop experiment contradicts core motivation: In Table 2 and Table S15, cropping the image tightly around the target object (L3-TargetCrop) or masking out the distractor object (L3-SourceMasked) removes cross-object interference, yet metrics moves only marginally. This contradicts the motivation of the paper that attribute attribution is challenging due to distracting visual evidence in the image. The results to me seem to suggest that the driving factor of the low performance is rather intrinsic single-object attribute difficulty (already evident in Level 2) and model bias on leading yes/no questions.

B. Narrow novelty: Level 1 directly reuses the images and settings of ROPE, while Level 2 is already covered by other larger benchmarks: VAW, OVAD, PACO. Level 3 is the main contribution of the work, however adversarial in-scene attribute polling is very similar to the eval settings in H-POPE. The main novelty left is the reliance on multiple objects acting as distractors, but as pointed out in weaknesses A the experimental results do not support this as a major source of hallucinations. C. Overlapping attribute vocabulary: The closed-set taxonomy has non-mutually-exclusive labels within shape (e.g., rectangular vs. flat, thin, or long) and pose (e.g., standing vs. upright). Forcing a single ground-truth label per category unfairly penalizes valid alternative predictions in Level 2 and creates ambiguous swapped negatives in Level 3. D. Lack of experiment on input: The paper lacks exploration on the impact of basic model prompting and image resolution on the hallucinations. This kind of experimental analysis would have been cheap and useful to understand what is the dominant challenge in the task.

E. Missing human baseline: Table S3 only measures label agreement between the two dataset creators rather than evaluating an independent pool of human participants on the actual benchmark prompts. Without a human baseline it is hard to quantify whether the benchmark is hard or the models are weak.

Questions for authors

Can you comment on the raised weaknesses?

Reviewer_kCAPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoAMA, a hierarchical benchmark for evaluating cross-object attribute misbinding in large vision-language models. The benchmark consists of three evaluation levels: multi-object instance recognition, target-object attribute recognition, and cross-object attribute binding verification. Its core design uses paired attribute-swap probes to examine whether models incorrectly assign attributes from one object to another within the same image. CoAMA contains 787 images, 3,935 marked object instances, and 19,460 scorable attributes. Experiments across eight LVLMs reveal persistent difficulties in attribute binding, with the best model achieving 67.69% PairAcc.

Critical strengths
  1. The paper discusses an important challenge in multi-object visual understanding by evaluating whether visual attributes are associated with the correct object instances.
  2. The paper is generally well-written and logically organized. The benchmark construction and evaluation protocol are clearly described.
Critical weaknesses
  1. The core idea of evaluating attribute confusion using properties of other objects in the same image has been explored in prior work, particularly H-POPE, which introduces image-based adversarial attribute sampling. Related benchmarks such as SugarCrepe also investigates object–attribute associations through controlled negative examples. Although CoAMA introduces explicit instance annotations, paired verification, and additional diagnostic settings, the paper does not provide a direct comparison demonstrating the incremental diagnostic value of these design choices over existing approaches.
  2. Although the proposed hierarchy evaluates object recognition, attribute recognition, and attribute binding separately, it does not fully disentangle binding failures from recognition errors. In particular, an incorrect Level 3 prediction may result from inaccurate recognition of the target object or its attributes rather than incorrect attribute ownership. While the paper includes upstream error propagation experiments, these do not directly establish whether binding failures persist when the relevant objects and their attributes are correctly recognized.
Questions for authors
  1. Could the authors provide a more direct comparison with existing attribute-binding benchmarks, particularly H-POPE, under matched evaluation conditions?
  2. Could the authors provide a recognition-conditioned Level 3 analysis, restricting evaluation to cases where the model correctly identifies both the target and source objects and their relevant attributes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.