- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SimulVision, a framework for diagnosing multi-object assignment failures in vision-language models. It shows that many apparent assignment errors arise from response-order sensitivity rather than missing visual information. The proposed Factorized Global Assignment (GA) uses structured decoding to improve assignment accuracy without additional training.
Critical strengths
- The paper distinguishes component recognition errors from assignment failures and demonstrates the impact of answer-order bias on multi-object evaluation.
- FGA substantially improves assignment accuracy across two VLMs. Controlled comparisons and an independent confirmation set support the effectiveness of structured readout.
Critical weaknesses
- FGA relies on the standard Hungarian algorithm to enforce one-to-one assignments. Alternative structured decoders achieve comparable performance, limiting the algorithmic novelty.
- The evaluation assumes known bounding boxes and unique object labels, with at most four objects. It remains unclear whether the findings extend to more complex real-world scenarios.
Questions for authors
- What distinguishes FGA from existing structured assignment methods, given that Sinkhorn achieves comparable results in Table 1?
- Can FGA handle repeated object categories, incomplete candidate sets, or inaccurate bounding boxes?
- Can the authors report the additional inference cost and compare against independent per-box label decoding?