- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CoAMA, a benchmark for evaluating cross-object attribute misbinding in LVLMs. The benchmark contains 787 images, 3,935 marked instances, and 19,460 scorable attributes, with 2,535 pairs in the primary Level 3 evaluation. Experiments on various models reveals a substancial gaps, especially in attribute binding, with attributes belonging to other objects more likely to be falsely accepted than absent attributes.
Critical strengths
-
- Explicitly marked target–source pairs provide a clear way to evaluate attribute ownership, particularly when multiple objects share the same category.
-
- The benchmark includes human-verified attribute annotations and several diagnostic settings, making it a reliable and valuable resource for studying this failure mode.
Critical weaknesses
-
- Semantic confounds and language priors weaken the interpretation of binding errors: attribute labels are not always mutually exclusive, moreover, some cases in Level3 may reflect differences in object–attribute plausibility(which is affected by language priors). For instance, the model may correctly figure out a banana is not red, but maybe the choice is made based on knowledge, not on visual evidences. Though discussed in the appendix, the authors failed to give satisfing solutions.
-
- The evaluation does not fully disentangle Level 3 binding errors from recognition errors at Levels 1 and 2. Reporting Level 3 accuracy on the subset where the model correctly answers the corresponding Level 1 and Level 2 questions would provide more convincing evidence.
Questions for authors
- Source masking yields only a marginal improvement. Does it significantly reduce false acceptance of the swapped attribute? How should this result be reconciled with the interpretation that the source object causes visual interference?
- Beyond the existing attribute and scene breakdowns, can the authors provide deeper analysis of what drives binding errors, for example through attention or token-level analysis? It would also be useful to examine how these errors change with the number of objects, model size, and reasoning budget.
- Does the binding failure persist in a multi-turn setting where the model first identifies the objects and their attributes, then answers the Level 3 question?