acceptodds
Under review as a conference paper at ICLR 2027

FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) are predominantly evaluated on free-form vision-language tasks such as visual question answering, captioning, and summarization. However, their practical use is rapidly expanding to more structured computer vision settings, where users prompt models to perform localization-centric tasks such as object detection, often within larger agentic or decision-making systems. Despite this shift, there is currently no standardized benchmark that systematically evaluates these capabilities at scale. In this work, we introduce the first comprehensive benchmark specifically designed to assess the promptable localization abilities of generalist MLLMs. Our benchmark spans four core task categories: object detection, referring expression detection, instance-level detection, and video-based detection. To enable consistent and fair evaluation, we develop a unified framework that standardizes inputs, enforces parsable bounding box outputs, and defines transparent evaluation protocols across tasks. Using this suite, we evaluate a diverse set of open-source and proprietary MLLMs, providing an in-depth analysis of their performance and limitations. Beyond accuracy, we examine models' ability to adhere to output format specifications, showing that current systems are highly sensitive to formatting constraints and often fail to generalize even to minor variations. Our results highlight both the strengths and shortcomings of state-of-the-art MLLMs in localization settings, and point toward important directions for improving multimodal model design and evaluation.

Primary area:
datasets and benchmarks
Keywords:
visual grounding, MLLM evaluation, bounding box prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sbVnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FindIt, which combines 13 existing datasets to evaluate generalist multimodal models on object detection, referring expression detection, instance detection, and video detection. The authors use a multi-stage search to select output configurations for each model and examine the effects of bounding-box representations and structured output formats.

Critical strengths

The experiments are fairly systematic, covering multiple localization tasks and revealing some multimodal models’ reliance on specific coordinate formats, as well as differences between open-source and frontier proprietary models in localization ability and adaptability to output formats.

Critical weaknesses
  1. I find the benchmark’s contribution somewhat limited. The data and tasks largely come from existing evaluations, while the new analysis mainly concerns the effects of coordinate representations and structured output formats. Without a direct comparison with similar benchmarks, its distinct value remains unclear.
  2. The baseline coverage is incomplete, missing newer open-source multimodal models and localization-specialized models such as LocateAnything.
Questions for authors
  1. What does FindIt add over existing localization benchmarks? Could the authors clarify this through a direct comparison?
  2. Why were newer open-source multimodal models and localization-specialized models such as LocateAnything not included? Are the latter unsuitable for the evaluation setup used in this paper?
3 comments

Thank you for the review.

[Weakness 1, Question 1] Contribution over existing benchmarks. FindIt’s contribution is in its format-informed evaluation setup rather than additional data or tasks. It compares many models across different localization axes, and it does so under a unified protocol. We are not aware of any benchmarks that provide a comparable setting or evaluation. Could the reviewer specify which benchmarks and what kind of comparison they would like to see? We are happy to provide it during the discussion period.

[W2, Q2] Additional open-source and localization-specialized models FindIt evaluates models whose output format can be controlled through prompting and who can address different localization tasks (detection, ref expression, instance detection, video). Methods like LocateAnything are not designed for this challenge as they are only suitable for one or two tasks (e.g. image detection and ref expression) and usually limited to a single format (e.g. for LocateAnything there is no documented support for alternative box representations or JSON output), so they would by design fail on most tasks.

If the question is about reporting an upper bound for each task separately, we are happy to refer to the respective current literature. If the reviewer has a specific model, they would like us to evaluate, could the reviewer let us know the reference? We will try to report results for them during the discussion period if possible or in the camera ready version if it needs larger resources.

We have also updated our responses to reviewers L1Fz and sbVn. Please see for more details.

I understand the authors' emphasis on format-informed evaluation and appreciate the practical value of a unified evaluation protocol. However, I still find the contribution limited, particularly in terms of the new insights it offers beyond existing localization evaluations. I will therefore keep my current rating.

Reviewer_JAZNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FindIt evaluates bounding-box localization in generalist multimodal LLMs across output formats, four task families, and thirteen datasets. It selects each model's preferred format via a multi-stage search on Pascal VOC. Results show higher scores for closed-source models, comparable scores for open-source and previous-generation closed-source models, format sensitivity in most models, and weak correlation between parsable format and correct localization. Validation includes open and closed models.

Critical strengths

The paper treats output format as a controlled variable and measures format sensitivity across models and tasks. Prior MLLM grounding evaluations mostly fix one prompt format. The separation between format adherence and localization accuracy is documented at scale. The center-format failure analysis gives a concrete mechanism for parsable but wrong outputs.

Critical weaknesses

Format selection is done on the evaluation data. Stage 3 chooses the per-task configuration using the full 1,000-query results and those numbers appear in the main tables. This inflates scores by different amounts depending on how many format options work for each model. The main tables lack a fixed-format baseline. Without it, a reader cannot separate localization ability from format preference. This matters because the paper's premise is that format affects scores.

Questions for authors

Was the 1,000-query evaluation split used for Stage 3 selection. If yes, can you report a version where format is chosen on a validation split and the main tables are evaluated once on a test split.

Can you add a fixed unified format column to the main tables next to the best-format column.

How stable is the Stage 1 and Stage 2 selection with 50 and 20 queries. Do repeated runs or bootstrap estimates change the chosen box representation or JSON key.

Do you have a semantic format adherence measure for coordinate space, order, and range. The examples in A.4 suggest many adherent outputs are semantically wrong.

In video results Gemini 2.5 Flash often omits frame_idx and the current parser cannot assign boxes to frames. Does a more robust parser change the relative ranking of models on video.

Does FindIt risk dataset leakage or evaluation contamination, given that its public evaluation datasets may overlap with MLLM pretraining data.

5 comments

[Weakness 1, Question 1] Format selection on the evaluation queries

Yes, the 1,000-query evaluation split was used for Stage 3 selection. To address this concern, we repeated Stage 3 for all seven open-source models on a separate validation set, using 500 new queries per dataset (84 per RefCOCO split and 433 for iGround). RoboTools, HR-InsDet hard, and multi-label iGround were excluded because no unused queries were available. We then evaluated the validation-selected formats on the original evaluation split.

The validation set selects the same format in 29 of 35 cases. Across all 35 cases, using validation-selected formats reduces [email protected] on the evaluation split by only 0.2 points on average. The table below shows the six cases where the selected format changes.

[email protected] on the evaluation split. Validation gap: how much higher the validation-selected format scores than the paper's format on the validation set.

ModelTaskFormat (paper)F1Format (validation)F1Validation gap
Qwen3-VLSingle-label detectionunconstrained66.9JSON (xyxy, bbox_2d)66.00.1
Qwen3-VLInstance detectionJSON (xyxy, bbox_2d)62.3unconstrained62.11.9
Qwen3.5-ThinkingReferring expressionstext (xyxy)67.2unconstrained66.70.6
Qwen3.5Referring expressionsJSON (xyxy, bbox_2d)64.4text (xyxy)64.20.5
Qwen3.5Multi-label detectionunconstrained60.9JSON (xyxy, bbox_2d)60.80.9
Gemma-4Single-label detectiontext (yxyx)52.4JSON (yxyx, bounding_box)48.20.6

All six changes are between formats that differ by less than 2 F1 on the validation set. Five reduce evaluation F1 by at most 1 point. The exception is Gemma-4 on single-label detection, which drops from 52.4 to 48.2.

The overall model ranking remains unchanged. The only change in per-task rankings occurs in referring expressions where Qwen3.5-Thinking led Qwen2.5-VL by only 0.3 F1. Its 0.5 F1 drop moves it from third to fourth place. We will revise the main tables to use validation-selected formats and extend the analysis to closed-source models.

[W2, Q2] Fixed-format baseline

We add a fixed-format comparison for 9 of the 12 models, using text with xyxy ([x_min, y_min, x_max, y_max]) as the fixed format. The table reports [email protected] averaged over each task family.

Each cell shows the F1 with the best format compared to the fixed text xyxy (incl. delta). A single number is reported when the best format is text xyxy.

ModelSingle-label detectionMulti-label detectionReferring expressionsInstance detectionVideoOverall
GLM-4.6V65.562.568.660.050.960.9
Qwen3-VL66.9 / 56.9 (−10.0)63.8 / 55.2 (−8.6)67.7 / 66.0 (−1.7)62.3 / 61.8 (−0.5)41.1 / 39.1 (−2.0)59.1 / 55.7 (−3.4)
Qwen3.5-Thinking63.2 / 62.7 (−0.5)54.167.234.8 / 29.4 (−5.4)30.2 / 27.0 (−3.2)47.7 / 45.5 (−2.2)
Qwen3.562.2 / 54.7 (−7.5)60.9 / 12.0 (−48.9)64.4 / 64.2 (−0.2)37.69.443.2 / 36.1 (−7.1)
Qwen2.5-VL53.9 / 43.2 (−10.7)53.2 / 13.6 (−39.6)66.9 / 63.7 (−3.2)28.7 / 28.1 (−0.6)17.0 / 9.8 (−7.2)41.5 / 32.5 (−9.0)
GPT-5.442.544.544.825.3 / 24.1 (−1.2)31.536.3 / 36.0 (−0.3)
InternVL355.057.2 / 56.1 (−1.1)61.00.5 / 0.0 (−0.5)18.033.9 / 33.6 (−0.3)
Gemma-452.4 / 12.7 (−39.7)58.0 / 13.6 (−44.4)58.5 / 14.2 (−44.3)0.3 / 0.0 (−0.3)17.2 / 6.0 (−11.2)32.8 / 8.3 (−24.5)
Sonnet 4.524.5 / 23.7 (−0.8)27.0 / 26.5 (−0.5)23.3 / 21.9 (−1.4)11.8 / 11.1 (−0.7)15.919.2 / 18.5 (−0.7)

Under the fixed text xyxy format, five of the nine models lose more than 5 F1 points in at least one task family. Gemma-4's overall F1 falls from 32.8 to 8.3, while Qwen3.5 and Qwen2.5-VL lose 48.9 and 39.6 F1 points, respectively, on multi-label detection. These results demonstrate that a unified output format does not affect all models equally and can substantially alter their measured performance.

We will add the remaining three models (Gemini 2.5 Flash, Opus 5.5 and GPT-6 Astra) in the final version. In Stage 1 on Pascal, Gemini's text F1 falls from 37.7 with yxyx to 5.7 with xyxy (Table 8), so we expect it to score much lower with text xyxy.

Q3. Stability of Stages 1 and 2

In Appendix A.3.1, we assess whether the Stage 1 selection transfers across datasets by repeating the full sweep on RefL4 (referring expressions) and single-frame RoboTools (instance detection) for GLM-4.6V, Qwen3-VL, and GPT-5.4. For GLM-4.6V and Qwen3-VL, xyxy remains the best-performing representation on both datasets and in both output formats. GPT-5.4, meanwhile, shows similar performance across representations on all three datasets.

We further assess selection stability through 4,000 bootstrap resamples of the Stage 1 and Stage 2 queries (50 per model, or 20 for closed-source models). The tables report the original [email protected] and bootstrap standard deviation for the best and second-best options. Across both stages, the bootstrap standard deviation averages 6.0 F1.

Stage 1, box representation. [email protected] ± standard deviation over 4,000 bootstrap draws.

ModelText: bestText: second bestJSON: bestJSON: second best
GLM-4.6Vxyxy 78.8 ± 4.4xywh 16.7 ± 4.6xyxy 44.1 ± 8.6xywh 4.0 ± 1.8
Qwen3-VLxyxy 69.9 ± 7.5xywh 19.5 ± 5.0xyxy 74.6 ± 7.4xywh 17.2 ± 4.8
Qwen3.5-Thinkingxyxy 77.7 ± 5.5xywh 21.1 ± 5.2xyxy 76.7 ± 5.7xywh 20.8 ± 5.2
Qwen3.5xyxy 64.8 ± 10.4xywh 16.5 ± 5.5xyxy 77.7 ± 7.8xywh 17.4 ± 4.5
Qwen2.5-VLxyxy 40.6 ± 6.8xywh 18.0 ± 5.3xyxy 64.3 ± 8.2xywh 20.3 ± 5.5
InternVL3xyxy 79.8 ± 5.5xywh 17.9 ± 4.7xyxy 80.2 ± 3.4xywh 20.8 ± 5.8
Gemma-4yxyx 73.0 ± 4.9yxhw 14.4 ± 4.5yxyx 43.4 ± 6.9yxhw 7.5 ± 3.3
GPT-6 Astraxywh 79.7 ± 4.3cxcywh 78.7 ± 4.4cxcywh 78.7 ± 4.7xyxy 77.6 ± 4.6
Opus 5.5yxhw 84.6 ± 2.9all-labelled 84.6 ± 2.9yxhw 85.6 ± 2.4cxcywh 83.0 ± 2.5
Gemini 2.5 Flashyxyx 37.7 ± 12.5yxhw 24.6 ± 10.8yxyx 76.1 ± 7.8yxhw 11.9 ± 6.0
GPT-5.4xyxy 42.1 ± 9.1xywh 35.6 ± 7.9xywh 40.4 ± 6.1xyxy 37.6 ± 10.1
Sonnet 4.5xyxy 30.8 ± 7.0xywh 16.9 ± 8.4xywh 27.4 ± 8.6xyxy 26.7 ± 8.1

Similarly, Stage 2, JSON key.

ModelSingle-label: bestSingle-label: second bestMulti-label: bestMulti-label: second best
GLM-4.6Vcoordinates 58.9 ± 6.3box_2d 57.9 ± 6.6box_2d 66.4 ± 4.7bounding_box 64.6 ± 5.0
Qwen3-VLbbox_2d 74.6 ± 7.3bbox 74.0 ± 7.7coordinates 75.4 ± 5.2bbox_2d 74.1 ± 4.1
Qwen3.5-Thinkingbbox_2d 76.7 ± 5.5box_2d 16.2 ± 5.7bbox_2d 75.0 ± 5.0class_name 38.0 ± 8.4
Qwen3.5bbox_2d 77.7 ± 7.9coordinates 0.0 ± 0.0bbox_2d 80.0 ± 4.2coordinates 0.0 ± 0.0
Qwen2.5-VLbox_2d 68.4 ± 6.6bbox_2d 64.3 ± 8.1box_2d 83.2 ± 4.6bounding_box 73.2 ± 5.9
InternVL3coordinates 83.5 ± 3.9box_2d 83.2 ± 3.6coordinates 83.7 ± 3.2bbox 83.1 ± 3.6
Gemma-4coordinates 68.5 ± 5.1bounding_box 68.2 ± 5.2box_2d 75.8 ± 5.2coordinates 62.0 ± 6.7
GPT-6 Astrabounding_box 77.2 ± 7.8coordinates 77.2 ± 7.7class_name 82.7 ± 4.8bbox 82.7 ± 4.2
Opus 5.5bounding_box 83.9 ± 3.8bbox_2d 83.2 ± 3.5bbox_2d 88.0 ± 3.5class_name 84.3 ± 3.1
Gemini 2.5 Flashbox_2d 76.8 ± 7.2bbox_2d 76.1 ± 7.7class_name 86.6 ± 5.9bbox 85.4 ± 7.5
GPT-5.4coordinates 47.2 ± 8.5bbox 45.5 ± 9.4bbox 53.2 ± 11.8coordinates 50.5 ± 12.0
Sonnet 4.5bbox 29.8 ± 7.2bbox_2d 27.4 ± 8.6coordinates 32.6 ± 7.0box_2d 32.1 ± 9.8

To assess the robustness of the selections, we count the bootstrap samples in which an alternative scores higher than the selected option. In Stage 1, all seven open-source models show a clear preference. The selected representation outperforms the next best by 22.6–62.1 F1, and no alternative scores higher in any of the 4,000 bootstrap samples. In Stage 2, Qwen3.5 and Qwen3.5-Thinking in both settings, and Qwen2.5-VL and Gemma-4 in the multi-label setting, show similar strong preferences. Their selected JSON keys outperform every alternative by at least 10 F1, with alternatives scoring higher in at most 1.8% of bootstrap samples, by a median difference of just 1.0 F1. For the remaining selections, alternatives score higher in some bootstrap samples. In those samples, the median difference is small: 1.7 F1 points for Stage 1 representations of closed-source models and 2.6 points for Stage 2 JSON keys. These results show that the selected formats are either strongly preferred or have only small performance differences from alternatives.

[Q4] Semantic format adherence for coordinate representation, space, and range

We assess semantic format adherence along three aspects: coordinate representation, range, and space. We test whether predictions follow the requested representation, whether the resulting boxes have valid coordinates, and whether alternative coordinate spaces improve localization performance.

In Stage 1, each model is prompted to produce bounding boxes in one of five representations, R={xyxy,xywh,yxyx,yxhw,cxcywh}\mathcal{R}=\{\texttt{xyxy},\texttt{xywh},\texttt{yxyx},\texttt{yxhw},\texttt{cxcywh}\}. We denote by S(r,p) the [email protected] obtained when predictions requested in representation r are parsed as representation p∈Rp\in\mathcal{R}. The score reported in the paper corresponds to S(r,r).

We define the semantic adherence error as

Δ(r)=max⁡p∈RS(r,p)−S(r,r).\Delta(r)=\max_{p\in\mathcal{R}}S(r,p)-S(r,r).

An error of zero indicates that parsing the predictions as requested gives the highest F1. A positive error indicates that another representation yields a higher F1, suggesting that the model may not have followed the requested representation.

The tables below show the complete results for Qwen3-VL on Pascal, separately for text and JSON outputs. Rows indicate the requested representation, columns indicate the representation used for parsing, and bold values correspond to the requested representation.

Table 1: Text output

Requested \ parsed asxyxyxywhyxyxyxhwcxcywhError
xyxy69.919.613.58.60.00.0
xywh69.519.513.48.50.050.0
yxyx67.919.813.68.60.054.3
yxhw69.519.513.48.50.061.0
cxcywh69.519.513.48.50.069.5

Table 2: JSON output

Requested \ parsed asxyxyxywhyxyxyxhwcxcywhError
xyxy74.618.612.47.20.00.0
xywh75.617.211.56.70.058.4
yxyx77.219.312.27.10.065.0
yxhw76.018.311.56.70.069.2
cxcywh75.418.112.17.00.075.4

For both text and JSON outputs, Qwen3-VL achieves its highest F1 when predictions are interpreted as xyxy, irrespective of the requested representation. This suggests that the model frequently defaults to xyxy coordinates even when prompted otherwise. The large differences between the requested and alternative interpretations demonstrate that syntactically valid outputs need not follow the requested coordinate representation. We will include the corresponding results for all models, including multi-label detection, in the supplementary material and on the project website.

To assess coordinate range, we check whether predictions have valid coordinates. We parse each prediction in the requested representation and consider it valid if all boxes have positive width and height and lie within the image, allowing a 2% margin. All Qwen3-VL text predictions parse successfully, but only 18% are valid when xywh or yxhw is requested, and 8% when cxcywh is requested (JSON: 14%, 14%, and 8%, respectively). The invalid predictions contain boxes extending beyond the image, consistent with the model producing xyxy coordinates that are interpreted as widths, heights, or box centers.

To assess sensitivity to coordinate space, we interpret predictions as pixel coordinates, on a 0–1000 scale, and as normalized 0–1 coordinates, comparing each against the space selected for that model based on its documentation or observed behavior. For 9 of the 12 models, no alternative space improves F1 in any Stage 1 setting. The exceptions are text outputs from InternVL3, Opus 5.5, and Sonnet 4.5, with improvements of up to 8.9 F1 points (Appendix A.7).

Reviewer_L1FzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a comprehensive benchmark designed to evaluate the promptable localization capabilities of generalist Multimodal Large Language Models (MLLMs). Moving beyond conventional free-form vision-language tasks such as visual question answering and captioning, the proposed benchmark systematically evaluates promptable spatial localization across four distinct task families: object detection, referring expression detection, instance-level detection, and video-based detection. By repurposing existing datasets, the authors establish a unified evaluation pipeline that standardizes free-form MLLM outputs into bounding boxes across varying coordinate conventions and structured output formats. The empirical findings reveal that while most state-of-the-art MLLMs exhibit robust localization proficiency, their performance remains remarkably sensitive to output format constraints.

Critical strengths
  1. Identification of a Critical and Under-Explored Evaluation Bottleneck. The paper makes a valuable distinction between a model's intrinsic spatial reasoning capability ("can the model localize the object?") and its instruction-following reliability ("can the model express that localization under a strict output contract?"). Unlike specialized object detectors that output spatial coordinates through dedicated architecture heads, generalist MLLMs must predict bounding boxes autoregressively via natural language prompts. This setup introduces a key evaluation variable: even when underlying spatial representations remain constant, minor variations in coordinate conventions, JSON key specifications, or output format constraints can significantly degrade performance. Spotlighting this sensitivity provides a fresh and critical perspective for evaluating MLLMs in structural output settings.
  2. Comprehensive Benchmark Design and Empirical Coverage. The proposed benchmark provides a broad and systematic evaluation suite spanning 4 task families (object detection, referring expression detection, instance-level detection, and video-based detection) across 13 distinct datasets. The study empirically benchmarks 6 open-source and 5 proprietary models. Beyond reporting standard localization accuracy, the paper thoroughly investigates pivotal factors, including coordinate representation schemes, free-text versus JSON structured outputs, fine-grained instance handling, and temporal multi-frame inputs. This multi-dimensional analysis offers a rich, standardized diagnostic reference for the community.
Critical weaknesses
  1. Methodological Flaw in Configuration Selection (Risk of Oracle/Optimistic Evaluation). A major methodological limitation lies in the configuration selection protocol, which relies on test-set metrics to determine model-specific parameters, thereby introducing a substantial risk of optimistic (oracle) evaluation bias. As detailed in Appendix A.2, the paper employs a three-stage tuning pipeline to identify each model's optimal output configuration. While Stages 1 and 2 select coordinate conventions and JSON keys using Pascal VOC, Stage 3 executes candidate configurations across all 1,000 queries per evaluation split and selects whichever configuration maximizes the average [email protected] per task family. Consequently, the evaluation data serves a dual role: configuring the model and measuring its final performance. This design choice undermines the foundational premise of a standardized, fair benchmark. To ensure methodological rigor, the evaluation protocol must strictly decouple validation and test splits, for instance: Validation/Development Set -> Configuration Selection -> Freeze Configuration -> Held-Out Test Set -> Final Benchmark Score.
  2. Conflation of Format Robustness and Intrinsic Localization Capability. The paper argues that because different models rely on different default coordinate conventions and output formats, evaluating each model under its optimal configuration yields a fairer measurement of its true localization capability. While this rationale is partially justified, format sensitivity itself constitutes a vital dimension of an MLLM's practical utility. A model that achieves precise localization only under a narrow, handcrafted format contract differs fundamentally from one that robustly respects arbitrary formatting instructions in real-world agentic environments. Reporting exclusively peak-format performance risks obscuring format fragility, whereas enforcing a single rigid format may underestimate a model's upper-bound perceptual potential. Rather than conflating these two orthogonal attributes into a single primary score, the authors are strongly advised to decouple the evaluation into two complementary reporting dimensions: Upper-Bound Localization Capability: Evaluates peak spatial performance under each model's optimal configuration. Format Robustness & Generalization: Quantifies instruction-following reliability and cross-format stability across a standardized suite of structural format variations.
Questions for authors

A central finding of the paper is that output format constraints significantly impact empirical localization scores, with variations in coordinate representations (e.g., xyxyxyxy vs. xywhxywh) or JSON structural conventions yielding substantial performance disparities. The paper appears to treat this format sensitivity primarily as a confounding variable (or nuisance parameter) to be controlled or optimized away in order to isolate underlying visual-spatial capabilities. However, this stance raises a critical conceptual question regarding practical MLLM deployment. In real-world agentic workflows or downstream API integrations, models are explicitly required to adhere to rigid structural contracts (e.g., "return bounding boxes in normalized cxcywhcxcywh format under specific JSON key protocols"). In such operational settings, failing to conform to the requested format constitutes a genuine functional failure, as the system must jointly execute visual localization and strict instruction-following. Therefore, could the authors clarify why format sensitivity should be framed as an extraneous artifact to be eliminated via per-model optimization, rather than an integral component of a generalist MLLM's practical localization capability?

1 comment

[Weakness 1] Configuration selection on evaluation data

We acknowledge the question if the evaluation split for Stage 3 format selection could inflate the reported scores. To address this concern, we repeated Stage 3 for all seven open-source models on a separate validation set, using 500 new queries per dataset (84 per RefCOCO split and 433 for iGround). RoboTools, HR-InsDet hard, and multi-label iGround were excluded because no unused queries were available. We then evaluated the validation-selected formats on the original evaluation split.

The validation set selects the same format in 29 of 35 cases. Across all 35 cases, using validation-selected formats reduces [email protected] on the evaluation split by only 0.2 points on average. The table below shows the six cases where the selected format changes.

[email protected] on the evaluation split. Validation gap: how much higher the validation-selected format scores than the paper's format on the validation set.

ModelTaskFormat (paper)F1Format (validation)F1Validation gap
Qwen3-VLSingle-label detectionunconstrained66.9JSON (xyxy, bbox_2d)66.00.1
Qwen3-VLInstance detectionJSON (xyxy, bbox_2d)62.3unconstrained62.11.9
Qwen3.5-ThinkingReferring expressionstext (xyxy)67.2unconstrained66.70.6
Qwen3.5Referring expressionsJSON (xyxy, bbox_2d)64.4text (xyxy)64.20.5
Qwen3.5Multi-label detectionunconstrained60.9JSON (xyxy, bbox_2d)60.80.9
Gemma-4Single-label detectiontext (yxyx)52.4JSON (yxyx, bounding_box)48.20.6

All six changes are between formats that differ by less than 2 F1 on the validation set. Five reduce evaluation F1 by at most 1 point. The exception is Gemma-4 on single-label detection, which drops from 52.4 to 48.2.

The overall model ranking remains unchanged. The only change in per-task rankings occurs in referring expressions where Qwen3.5-Thinking led Qwen2.5-VL by only 0.3 F1. Its 0.5 F1 drop moves it from third to fourth place. We will revise the main tables to use validation-selected formats and extend the analysis to closed-source models.

[W2 and Question 1] Format robustness

We agree that adherence to a required output format is an important part of practical localization capability. Our goal is not to disregard format sensitivity, but to measure it separately from localization accuracy. FindIt already reports both dimensions. The main tables (Sec. 5.1) report localization performance under each model's selected configuration, while Secs. 5.2 and 5.3 evaluate all models on the same box representations, output formats, and JSON keys (Tables 7–9). Table 17 (A.3.2) directly compares these dimensions, reporting F1 at the selected representation alongside mean F1 and mean format adherence across all seven representations. GPT-5.4 scores below Qwen3-VL at the selected representation (42.1 vs. 69.9 F1 in text), but achieves more than twice its mean F1 across representations (36.4 vs. 15.9). Table 16 (A.3.1) extends this comparison to RefL4 and RoboTools. These results illustrate why both dimensions matter for model selection: Qwen3-VL is preferable when the output format can be chosen freely to maximize localization accuracy, while GPT-5.4 is better suited to settings where the required format may vary. We retain selected-format localization as the main score because a low score under a fixed format does not distinguish localization errors from failures to follow the requested coordinate convention. For example, when asked for cxcywh in text, Qwen3-VL scores 0.0 F1 when parsed as requested, but 69.5 when the same predictions are interpreted as xyxy (Table 21). This is a clear format-following failure, but the predictions still localize the objects. Conversely, enforcing text xyxy for all models reduces Gemma-4's average F1 across tasks from 32.8 to 8.3 (see our response to JAZN, W2/Q2). In agentic workflows, developers may be able to choose the model's output format and convert its predictions to the representation required by the pipeline. Other applications impose fixed format requirements that the model must follow directly. Reporting both localization accuracy and format robustness allows developers to choose a model based on these constraints, rather than relying on a single score that conflates the two. We will make this distinction clearer in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.