acceptodds
Withdrawn from ICLR 2027

SimulVision: Diagnosing and Recovering Multi-Object Assignment in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-object errors in vision-language models (VLMs) are often interpreted as failures of visual binding or spatial reasoning, yet an incorrect global response does not reveal whether assignment-relevant information is genuinely unavailable. Inspired by the distinction between component access and multi-item integration, we study identity-location assignment under controlled evaluation. A component-preserving two-object diagnostic initially suggests a substantial mapping deficit: Qwen2.5-VL-7B fails on 31.4% of component-correct items under a fixed answer order. However, only 6.4% are incorrect under both answer orders, showing that many apparent assignment failures are response-unstable. We therefore introduce a closed-vocabulary multi-object assignment task that removes answer-position bias and free-form naming variability. On Qwen, local likelihood probing recovers substantial assignment-relevant signal even when monolithic response accuracy is low. Across both Qwen2.5-VL-7B and InternVL3.5-4B, enforcing one-to-one structure on the same local compatibility scores substantially improves exact assignment accuracy. We instantiate this intervention as Factorized Global Assignment (FGA), a training-free structured readout, and replicate the effect on an independently frozen 300-image confirmation set. These results show that multi-object evaluation should distinguish what assignment evidence can be elicited locally from how reliably several assignments are assembled into a globally consistent response.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multi-object reasoning, visual binding, structured prediction, assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VgStOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SimulVision, a framework for diagnosing multi-object assignment failures in vision-language models. It shows that many apparent assignment errors arise from response-order sensitivity rather than missing visual information. The proposed Factorized Global Assignment (GA) uses structured decoding to improve assignment accuracy without additional training.

Critical strengths
  1. The paper distinguishes component recognition errors from assignment failures and demonstrates the impact of answer-order bias on multi-object evaluation.
  2. FGA substantially improves assignment accuracy across two VLMs. Controlled comparisons and an independent confirmation set support the effectiveness of structured readout.
Critical weaknesses
  1. FGA relies on the standard Hungarian algorithm to enforce one-to-one assignments. Alternative structured decoders achieve comparable performance, limiting the algorithmic novelty.
  2. The evaluation assumes known bounding boxes and unique object labels, with at most four objects. It remains unclear whether the findings extend to more complex real-world scenarios.
Questions for authors
  1. What distinguishes FGA from existing structured assignment methods, given that Sinkhorn achieves comparable results in Table 1?
  2. Can FGA handle repeated object categories, incomplete candidate sets, or inaccurate bounding boxes?
  3. Can the authors report the additional inference cost and compare against independent per-box label decoding?
Reviewer_4bhoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper looks into the issue that common VLM benchmarks’ simple accuracy measures do not capture failure modes for queries involving multiple instances. It shows that models can give incorrect overall answers even when individual object–location matches can be recovered through separate queries. The authors combine these local scores using the Hungarian algorithm to produce more accurate and consistent assignments without retraining the models.

Critical strengths
  • The experimental design is thoughtful, containing many uncertainty analyses and confirmation experiments that strengthen the empirical results.
  • The paper is, in principle, very transparent and reproducible in its design.
Critical weaknesses
  • The paper builds on the interesting observation that multi-object responses can fail either through missing visual information or difficulties in assembling the assignments into an answer. But instead of analyzing the underlying mechanisms of that difference, the authors propose a simple and well-established method of constructing pairwise compatibility scores and solving a linear assignment problem. I don't see a strong contribution or novelty in that finding. The computational complexity of local search makes its usefulness questionable. In addition, the global prompt does not instruct a one-to-one mapping, which is algorithmically forced in the local search and evaluated.

  • The paper is very difficult to read in some parts. Especially Section 3 is hard to follow and could benefit from better structure and the introduction of mathematical notation. In addition, it is not clear which prompts have been used, references to the appendix and Figure 1 are missing, and the overall benefit of Figure 1C seems minor, since it adds no information beyond the text.

Questions for authors
  • Why is InterVL global prompt outperforming the local in contrast to Qwen?
  • How much of FGA’s improvement remains when global generation is explicitly instructed to use every label exactly once and decoding is constrained to valid one-to-one assignments?
  • What prompts are used in section 3?
  • Where is the GitHub Repository that is promised in the reproducibility statement?
Reviewer_bPybOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks if multi-object errors of vision-language models reflect that object–location information is missing or reflect failures to assemble evidence into a consistent answer. An object swap diagnostic on Qwen2.5-VL-7B shows that failures are considerably more frequent under a fixed answer order than when both orders are considered, suggesting that much of the required information is available but not consistently assembled. The authors then introduce a closed-vocabulary task involving numbered boxes on COCO images and compare monolithic generation with local scoring, where candidate labels are evaluated independently for each box using teacher-forced likelihood. They propose Factorized Global Assignment (FGA), which uses the Hungarian algorithm to find a globally consistent assignment from local scores. FGA improves exact assignment accuracy over independent local decoding on both Qwen2.5-VL-7B and InternVL3.5-4B.

Critical strengths
  1. Careful evaluation protocol. The confirmation set was frozen and hashed before any outputs were collected, contrasts are paired with image-cluster bootstrap intervals, and the within-image control fixes the image and object pool while varying kk. The order-reversal result is a useful caution for multi-object benchmarks that rely on a fixed answer order. The reported numbers are therefore trustworthy, even where their interpretation is in question.

  2. Factorized scoring with a one-to-one constraint outperforms exhaustive joint scoring. On Qwen, FGA reaches 54.8% at k=4k=4 against 34.2% for scoring all k!k! complete answers. Sinkhorn gives similar gains and Greedy-Unique smaller but still substantial ones. This is a falsifiable and practically relevant finding about how to read out multi-object answers, although it is shown for one model only.

Critical weaknesses
  1. The local–global dissociation is not separated from differences in the prompt and answer interface. The global prompt requires a formatted kk-slot generation, while the local prompt asks about a single box and scores each label by likelihood under kk box-specific prompts, one forward pass per box–label cell. The gap may therefore reflect the known difficulty of producing multi-object answers, not a failure to assemble available evidence. Two results weaken the claim that the dissociation is "consistent" (p. 2, l. 59; p. 5, l. 260). On Qwen, whole-sequence likelihood scoring roughly matches Local at k=3,4k=3,4. On InternVL, Local is below greedy generation at every kk. With only two models of at most 7B parameters (4-bit NF4, App. D) and no analysis of internal representations, the results also cannot distinguish the proposed account from limited model capacity or representational interference (e.g., Campbell et al., NeurIPS 2024). Matched-interface controls and narrowing the claims to what is shown would address this.

  2. The FGA "recovery" result cannot fail as tested. If the local argmax (Eq. 2) is a permutation, FGA returns it (App. G). Otherwise Local is already wrong on that item. FGA ≥ Local therefore holds item by item, and the gain equals the number of resolved duplicates (Sec. 5.3). The comparison is nevertheless presented as "a direct test of our interpretation" with confidence intervals excluding zero. The Local baseline also compares log-likelihoods computed under different box-specific prompts without calibration, and no per-box (row-wise) baseline or random-repair null is reported. Since FGA itself is a standard linear assignment, the contribution rests on the diagnosis, and these baselines are needed to support it.

Questions for authors
  1. Clarification. Could you report, from the stored score matrices, a per-box (row-wise) argmax, a row-normalized Local readout, and a random-repair null that resolves duplicates at random? If FGA keeps a clear margin over these, it would help answer my concern in Weakness 2.
  2. Clarification. Can you add a matched-interface control, for example single-box generation compared with kk-box generation under the same prompt template, and whole-sequence scoring for InternVL? This would show whether the gap in Fig. 2 is due to assembly or to the generation interface.
  3. Clarification. How do you reconcile Local being below greedy generation for InternVL at every kk (Fig. 4B) with the claim of a consistent local–global dissociation?
  4. Suggestion. Does the gap persist for a larger model of the same family (e.g., Qwen2.5-VL-32B on the confirmation set) and without 4-bit quantization on a subset? A gap that shrinks with scale would point to capacity, not assembly.
  5. Suggestion. Relating the findings to representational accounts of binding (Campbell et al., NeurIPS 2024; Feng & Steinhardt, ICLR 2024), for instance by probing hidden states for box–label information, would help test the proposed mechanism directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.