- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FindIt, which combines 13 existing datasets to evaluate generalist multimodal models on object detection, referring expression detection, instance detection, and video detection. The authors use a multi-stage search to select output configurations for each model and examine the effects of bounding-box representations and structured output formats.
Critical strengths
The experiments are fairly systematic, covering multiple localization tasks and revealing some multimodal models’ reliance on specific coordinate formats, as well as differences between open-source and frontier proprietary models in localization ability and adaptability to output formats.
Critical weaknesses
- I find the benchmark’s contribution somewhat limited. The data and tasks largely come from existing evaluations, while the new analysis mainly concerns the effects of coordinate representations and structured output formats. Without a direct comparison with similar benchmarks, its distinct value remains unclear.
- The baseline coverage is incomplete, missing newer open-source multimodal models and localization-specialized models such as LocateAnything.
Questions for authors
- What does FindIt add over existing localization benchmarks? Could the authors clarify this through a direct comparison?
- Why were newer open-source multimodal models and localization-specialized models such as LocateAnything not included? Are the latter unsuitable for the evaluation setup used in this paper?
3 comments
We have also updated our responses to reviewers L1Fz and sbVn. Please see for more details.
I understand the authors' emphasis on format-informed evaluation and appreciate the practical value of a unified evaluation protocol. However, I still find the contribution limited, particularly in terms of the new insights it offers beyond existing localization evaluations. I will therefore keep my current rating.
Thank you for the review.
[Weakness 1, Question 1] Contribution over existing benchmarks. FindIt’s contribution is in its format-informed evaluation setup rather than additional data or tasks. It compares many models across different localization axes, and it does so under a unified protocol. We are not aware of any benchmarks that provide a comparable setting or evaluation. Could the reviewer specify which benchmarks and what kind of comparison they would like to see? We are happy to provide it during the discussion period.
[W2, Q2] Additional open-source and localization-specialized models FindIt evaluates models whose output format can be controlled through prompting and who can address different localization tasks (detection, ref expression, instance detection, video). Methods like LocateAnything are not designed for this challenge as they are only suitable for one or two tasks (e.g. image detection and ref expression) and usually limited to a single format (e.g. for LocateAnything there is no documented support for alternative box representations or JSON output), so they would by design fail on most tasks.
If the question is about reporting an upper bound for each task separately, we are happy to refer to the respective current literature. If the reviewer has a specific model, they would like us to evaluate, could the reviewer let us know the reference? We will try to report results for them during the discussion period if possible or in the camera ready version if it needs larger resources.