acceptodds
Under review as a conference paper at ICLR 2027

KNOWING HOW MANY IS NOT KNOWING WHICH ONES:NATIVE COMPLETE-SET GROUNDING IN MULTIMODAL LARGE LANGUAGE MODELS

45 views

Abstract

Generalized Referring Expression Comprehension (GREC) requires a model to recover all objects referred to by an expression, including no-target and multi-target cases. We identify a spurious-correlation failure mode in GREC: target cardinality remains partly predictable when visual evidence is removed or mismatched, revealing reliance on language and dataset priors. Yet accurate counting does not imply correct target selection. A strong query-based set decoder predicts the correct count on 98.38% of two-target expressions but achieves only 10.90% Pr for the complete target set. When the query changes to refer to a different pair of objects in the same image, the decoder usually preserves the count but rarely switches to the correct targets. We therefore propose Native Complete-Set Grounding, which trains the MLLM to select and localize all referred objects through its autoregressive localization pathway. In a matched comparison with a separate set-prediction decoder, Pr improves from 50.24% to 77.22%. Our 3B model achieves 81.24/76.18/68.11% Pr on gRefCOCO val/testA/testB, establishing state-of-the-art performance on all three splits. The gains persist at 7B scale while conventional REC performance remains strong.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Visual Grounding, Generalized Referring Expression Comprehension, Referring Expression Comprehension, Spurious Correlation, Autoregressive Localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Ph6VOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Native Complete-Set Grounding to address a key challenge in generalized referring expression comprehension (GREC): models may correctly predict the number of referred objects yet fail to identify the complete set of those objects. The proposed method achieves impressive experimental performance.

Critical strengths
  1. The proposed method demonstrates strong performance on the evaluated benchmarks.
  2. The paper provides thorough analyses that support the effectiveness of the proposed approach.
Critical weaknesses
  1. The motivation for converting unordered set prediction into ordered prediction requires further clarification. Existing GREC methods typically use ground-truth bounding boxes to supervise training, enabling models to learn both the number and locations of the referred objects. Introducing an ordering constraint appears to impose an additional requirement: the model must identify the correct objects while also predicting them in a prescribed order. Why is this reformulation necessary, and how does it improve complete-set grounding? Controlled comparisons with unordered set prediction would help substantiate this design choice.

  2. The framework in Figure 2 does not clearly convey the novelty of the proposed method or its core technical details. The figure should more explicitly illustrate the key components, their interactions, and the aspects that distinguish the proposed approach from existing methods.

  3. Table 1 show the results on regerring expression segmentation, but the topic of this paper is referring expression comprehension. Although these two task share some similarity, the inner logic is different

Questions for authors

see above mentioned.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ph6V

We thank the reviewer for the thoughtful comments and constructive feedback.

W1. Why serialize an unordered target set, and how does it compare with unordered set prediction?

The target is an unordered set of bounding boxes, while evaluation uses one-to-one IoU matching and is invariant to the order of predicted boxes. Serialization serves as a training representation for autoregressive generation, while prediction correctness remains independent of box order. It enables complete-set grounding through the MLLM's native localization pathway, allowing each generated box to be conditioned on the image, the referring expression, and previously generated boxes.

We compared this formulation with a permutation-invariant, Hungarian-matched DETR-style set decoder under the same 12k-train/3k-development protocol. Even with extended training (6 epochs), the unordered set decoder reaches 85.79% count accuracy but only 56.59% complete-set Pr (vs. 50.24% at 3 epochs), whereas Native achieves 77.22% Pr and improves directional switch accuracy on count-preserving query pairs from 4.02% to 65.68%.

We further tested how training order and the explicit count field affect performance using the same 12k/3k controlled protocol:

Training outputFixed target orderResampled target orderFixed − resampled (pp)
Explicit count + boxes78.8977.22+1.67
Free-form boxes81.8580.41+1.44
Free-form − explicit count (pp)+2.96+3.19

Fixed-order supervision improves complete-set Pr in both formats, while free-form boxes outperform explicit Count+Boxes under either ordering. These results support our autoregressive grounding formulation and show that training order affects complete-set recovery, while evaluation remains order-invariant.

W2. What is the technical distinction that Figure 2 should convey?

The key distinction is how target selection and localization are performed. The separate set decoder uses learned object queries, cross-attention to multimodal LM features, Hungarian matching for box supervision, and query ranking with a predicted target count. Native grounding instead supervises the complete variable-cardinality set through Qwen's autoregressive coordinate-generation pathway; the generated boxes are then parsed into an unordered set.

Figure 2 depicts the submitted Count+Boxes interface: its count is generated in the same response as the coordinates, rather than by a separate count head. We will clarify these two prediction pathways and the set-parsing step in the figure and caption.The figure's link is https://anonymous.4open.science/r/figure-materials-5B33/

Figure 2: Overview of Native Complete-Set Grounding and the separate set-prediction baseline. (A) Both approaches share the same Qwen2.5-VL backbone, with frozen visual components and LM-side LoRA adaptation. (B) Native performs target selection and localization within a single pretrained autoregressive pathway, generating target count and box coordinates in one response. Training uses valid orderings of unordered target sets with response-token causal LM supervision, while inference deterministically parses the generated response into an unordered set. (C) The separate baseline instead predicts cardinality with a count head and localizes targets through a newly trained query-based decoder over MLLM representations, using count classification and Hungarian-matched box/objectness supervision.

W3. Does Table 1 evaluate referring expression segmentation or comprehension?

Table 1 evaluates bounding-box Generalized Referring Expression Comprehension (GREC), not segmentation. Predictions and ground truth are bounding boxes; complete-set Pr requires the correct target count and one-to-one box matching at IoU ≥ 0.5, while N-acc measures correct empty-set prediction. gRefCOCO/GREC · GRES

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Reviewer_aPq8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the distinction between correct target cardinality and correct target membership in generalized referring expression comprehension. It introduces count-preserving query-pair diagnostics and adapts Qwen2.5-VL to generate complete box sets using its native autoregressive interface. It reports substantial gains on gRefCOCO and over a matched DETR-style head, with limitations in high-cardinality recovery and inference latency.

Critical strengths
  1. Count-preserving, same-image query pairs provide a useful diagnostic of expression-conditioned target selection beyond count accuracy.

  2. The method is simple, and the paper transparently retains malformed predictions in evaluation and reports model-selection and efficiency details.

Critical weaknesses
  1. Limited novelty. The method closely resembles ordinary multi-box LoRA/SFT through an existing Qwen2.5-VL coordinate interface. The advances beyond prior native autoregressive and multi-instance grounding, including GETok, need clearer delineation and comparison.

  2. Lack of comparison with stronger LLMs. Qwen2.5 is an "old" model, experiments on current SOTA models (e.g. Qwen3.5, even closed-source models) will be beneficial for the paper, making the claims more convincing.

  3. The architectural conclusion is supported by only one newly trained DETR-style head, whose complete-set Pr is 10.90% for K=2 and 0% for K≥3. Equal training budgets do not establish comparable convergence when Native reuses a pretrained localization pathway. Please test decoder capacity and training-budget sensitivity, multiple seeds, and a stronger proposal-based or pretrained modular baseline such as VGent (VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction) before generalizing the failure to separate target prediction.

  4. The practical scope is limited by high-cardinality recovery and decoding cost. For K≥5, complete-set Pr is only 5.36% on testA and 1.98% on testB, and remains low even with correct counts. Please provide cardinality-stratified comparisons with strong baselines, accuracy–latency trade-offs, and sensitivity to the 128-token generation limit.

Questions for authors

Please refer to the weaknesses section, especially W1-3.

13 comments
AuthorsOpenReview ↗

Response to Reviewer aPq8-1

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the first response.

W1. Novelty relative to existing native multi-object grounding, GETok, and VGent

Our work addresses a specific failure in variable-cardinality grounding: predicting the correct number of targets does not imply selecting the correct referred instances. Complete-set grounding requires jointly recovering which objects are referred to, how many there are, and where they are located, including empty sets.

We isolate this failure using count-preserving query pairs: the same image contains two expressions referring to different object sets of the same size. Even independently developed specialists exhibit the discrepancy. On 371 aligned pairs, InstanceVG and PropVG recover both complete sets in only 46.09% and 52.00% of cases, respectively, given that both predicted counts are correct.

GETok improves autoregressive spatial grounding using learned grid and offset tokens, including multi-instance grounding. VGent combines detector proposals with modular reasoning and target selection. In contrast, we investigate count-correct membership errors and address them by supervising the entire referred set through Qwen's existing autoregressive localization pathway, without an auxiliary set decoder or separate box-selection stage. GETok (CVPR 2026) · VGent (CVPR 2026)

W2. Does the formulation extend to a newer MLLM backbone?

Yes. We trained Qwen3.5-4B (non-thinking mode) with full-scale free-form complete-set supervision and used the same official gRefCOCO splits and evaluator as Qwen2.5-VL-3B:

Full-scale free-form backboneVal Pr (%)TestA Pr (%)TestB Pr (%)K ≥ 3 Pr (%) — testA/testB
Qwen2.5-VL-3B84.0677.2869.1828.50 / 20.93
Qwen3.5-4B (non-thinking)79.4377.6670.8037.72 / 29.43

On the newer backbone, testA/testB complete-set Pr reaches 77.66%/70.80%, with +9.22/+8.50 pp gains at K ≥ 3 relative to the 3B free-form model. This confirms that the supervision formulation transfers to a newer MLLM backbone.

AuthorsOpenReview ↗

Response to Reviewer aPq8-2

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the second response.

W3. Is the original detached decoder too weak to support the paper's claims?

We evaluated longer training, increased decoder capacity, and candidate-box quality under the same 12k-train/3k-development protocol and gRefCOCO val evaluator. The detached decoder's learned object queries already cross-attend to image and referring-expression features.

Controlled configuration (single run)Val Pr (%)Count accuracy (%)Pr given correct count (%)Raw-query oracle Pr (%)
Original detached decoder (12 queries, 3 layers, 3 epochs)50.2482.8060.6770.89
Extended training (6 epochs)56.5985.7965.9674.22
Increased capacity (24 queries, 6 layers)46.7883.1556.2762.98
Native grounding (controlled)77.2286.2789.52—

Extended training improves the decoder from 50.24% to 56.59% Pr, while Native reaches 77.22%, outperforming the original decoder by 26.98 pp (95% paired bootstrap CI: [+26.22, +27.75] pp). Under 6-epoch training, their count accuracies are nearly identical (85.79% vs. 86.27%), but Pr given correct count differs substantially (65.96% vs. 89.52%). With the extended decoder's predicted boxes held fixed, ground-truth-guided selection raises Pr to 74.22%, a 17.63 pp gain over its actual predictions. This identifies target selection and count decisions, beyond candidate localization alone, as major sources of error.

Multi-seed validation. We additionally evaluated detached-decoder training stability on official gRefCOCO val and the same 373 count-preserving query pairs:

Detached configurationSeedsVal Pr (%), mean ± SDDirectional switch (%), mean ± SD
Original decoder (3 epochs)350.94 ± 2.233.22 ± 1.88
Extended training (6 epochs)255.72 ± 1.228.04 ± 0.76
Increased capacity (24 queries, 6 layers)250.70 ± 5.5410.72 ± 7.20

Longer training consistently improves complete-set Pr, while the larger decoder shows greater seed sensitivity. Across these runs, both target counts are correct on 95.98%–97.86% of pairs, yet both complete sets are recovered on only 0%–0.54%. The count–membership discrepancy therefore persists across seeds, training budgets, and decoder capacities.

Matched-seed Native validation. Under the same controlled protocol, we evaluated Native on seeds 42/43/44 and compared it with the corresponding original detached decoder:

SeedNative Val Pr (%)Native directional switch (%)Detached Val Pr (%)Paired Pr gain (pp)
4277.2265.6850.24+26.99
4378.5960.8649.15+29.43
4474.7356.0353.43+21.30
Mean ± SD76.85 ± 1.9560.86 ± 4.8350.94 ± 2.23+25.91 ± 4.17

Across these seeds, Native recovers both complete sets in 27.88 ± 1.42% of the 373 query pairs. Paired Pr gains use the unrounded evaluation values.

We also evaluated released full-training-set GREC specialists: InstanceVG achieves 73.71%, PropVG 72.38%, and our full-scale Native model 81.24% official-val Pr. Their independent count-preserving diagnostics (W1) further demonstrate that correct cardinality does not ensure complete target membership. We could not identify a publicly released official VGent implementation or checkpoints for matched reproduction.

AuthorsOpenReview ↗

Response to Reviewer aPq8-3

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the third response.

W4. What happens at high target cardinality, under a longer output cap, and at inference time?

Complete-set Pr by cardinality on testA/testB (%) is:

ModelK ≥ 3 Pr (%)K ≥ 5 Pr (%)
Submitted Qwen2.5-VL-3B (Count+Boxes)31.16 / 22.615.36 / 1.98
Qwen2.5-VL-3B (free-form)28.50 / 20.936.25 / 4.40
Qwen2.5-VL-7B (free-form)29.36 / 21.666.79 / 1.54
Qwen3.5-4B (free-form, non-thinking)37.72 / 29.43—
InstanceVG14.75 / 10.300.00 / 0.22
PropVG11.99 / 5.821.43 / 0.22

Native retains a substantial relative advantage at K ≥ 3. High-cardinality errors involve both counting and membership: 80.82%/81.81% of failed K ≥ 3 cases for free-form 3B under-count targets, while the submitted model achieves only 24.59%/15.79% Pr at K ≥ 5 even when counts are correct.

Token-cap sensitivity. Across all 281 evaluation instances reaching the 128-token limit, raising the cap to 256 tokens increased complete-set recoveries only from 1 to 2, while mean latency increased by approximately 25%–40% across the measured models. A longer output budget therefore incurs additional decoding cost with little recovery benefit.

End-to-end accuracy–latency comparison. We evaluated four models on the same 500 testA examples, including image preprocessing, full model computation, and output decoding, on an RTX 2080 Ti (batch size 1). The Qwen-based models use FP16, while InstanceVG and PropVG use FP32:

ModelSubset complete-set Pr (%)Mean end-to-end latency (ms/example)
Controlled Detached38.60291.55
Controlled Native75.801,615.71
InstanceVG73.4050.69
PropVG72.8050.57

On identical samples, Controlled Native gains +37.20 pp over Controlled Detached at 5.54× mean end-to-end latency. Relative to InstanceVG and PropVG, Controlled Native achieves +2.40/+3.00 pp higher Pr at approximately 31.9× the mean latency. A separate RTX 4090, BF16 end-to-end test of Controlled Native versus Detached on 500 aligned testA examples similarly yields +37.80 pp at 8.57× mean latency.

Reviewer_hM2eOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Generalized Referring Expression Comprehension (GREC) and points out that even when a model predicts the correct number of targets, it does not necessarily select the correct targets. As a reference, the authors attach a DETR-style set-prediction decoder to Qwen2.5-VL-3B. This decoder predicts the correct count for 98.38% of two-target expressions but recovers the complete target set for only 10.90%. On query pairs that share the same image and the same number of targets but refer to different objects, both counts are correct for 97.59% of the pairs, yet the prediction switches to the correct targets for only 4.02%. Based on this, the authors propose Native Complete-Set Grounding, which directly fine-tunes the MLLM with LoRA to autoregressively output all target boxes in the format "Count: K; Boxes: [...]", without introducing an additional detection head, matching loss, or counting loss. In the controlled comparison on a 12k training subset, Pr on the official val split improves from 50.24 to 77.22. The 3B model trained on the full training set reaches 81.24/76.18/68.11 Pr on gRefCOCO val/testA/testB, surpassing existing specialist models. For the 7B model, Pr increases from 72.33 without fine-tuning to 78.35, and the conventional REC performance on the RefCOCO family also remains at a high level. The appendix provides a visual-evidence ablation, serialization controls, a failure analysis by target count, and an inference-efficiency comparison (14.27× slower than the separate decoder).

Critical strengths

The results on gRefCOCO are strong, and the evaluation protocol is fairly rigorous. On val/testA/testB, the 3B model exceeds the previous best specialist models in Pr by 7.07, 4.99, and 7.31 points, respectively. A single interface handles no-target, single-target, and multi-target expressions, and N-acc also reaches 80.39/75.13/70.38.

Critical weaknesses

The novelty of the method is limited. Native Complete-Set Grounding essentially uses the standard causal language-modeling loss to make the MLLM output all target boxes as text, which is a common practice for multi-target localization with MLLMs.

In the Qwen2.5-VL technical report, the 3B model achieves 85.2/85.7 on RefCOCOg val/test, which is higher than the 82.95/83.63 of the fine-tuned model in Table 5. This shows that the base model itself is already very strong.

Performance degrades very severely in scenarios with many targets. Without a comparison with specialist methods on results broken down by K, there is no evidence to determine whether the SOTA performance mainly comes from no-target, single-target, and two-target samples.

Questions for authors

Could the authors run the same count-preserving query-pair diagnostic on published specialist GREC models (e.g., using the released weights of InstanceVG, SSDE, or PropVG) to show whether this failure mode is common among existing methods?

Which MLLM hidden states does the separate decoder read (which layer, and image tokens or text tokens)? Are the image tokens placed before the referring expression in the sequence, so that they cannot see the expression under causal attention? What do the count head and the box queries read, respectively? Could the authors provide a version that ensures the box queries can see the expression, or a version that is fully trained on the full dataset?

In the Qwen2.5-VL technical report, the 3B model achieves 85.2/85.7 on RefCOCOg val/test. What is the benefit of the proposed method compared with the base model? What are the results of the unadapted Qwen2.5-VL-3B on gRefCOCO?

Since the free-form output (81.85) outperforms "explicit K + resampling" (77.22) in the 12k setting, why does the main model still use the latter? What are the results of training the free-form variant on the full dataset?

6 comments
AuthorsOpenReview ↗

Response to Reviewer hM2e-1

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the first response.

W1. Novelty beyond standard multi-box autoregressive fine-tuning

Our central finding is that correct target cardinality does not guarantee correct target membership in generalized referring expression comprehension. Recovering a complete set requires selecting all and only the referred instances, not merely generating the right number of boxes.

We demonstrate this with count-preserving query pairs: two expressions for the same image, each with K = 2 but no shared target instance IDs. Among 373 pairs, the detached decoder predicts both counts correctly in 97.59% of cases but achieves only 4.02% directional switch accuracy; controlled Native achieves 95.44% and 65.68%, respectively. A directional switch means each predicted set matches its own expression's ground truth better (in set F1) than the other expression's ground truth. Q1 shows that independent GREC specialists also exhibit count-correct membership failures.

We address this problem by supervising target selection and localization together through the MLLM's autoregressive coordinate-generation pathway, handling empty, single-target, and multi-target expressions without an auxiliary set decoder or separate target-selection module.

W2. Do RefCOCOg scores imply that complete-set adaptation harms conventional REC?

The Qwen2.5-VL technical report reports 85.2/85.7 on RefCOCOg val/test, while our adapted model obtains 82.95/83.63. We therefore evaluated the unadapted and adapted Qwen2.5-VL-3B checkpoints under the same RefCOCOg Top-1 protocol:

RefCOCOg modelVal Top-1 (%) · 4,896 examplesTest Top-1 (%) · 9,602 examples
Unadapted Qwen2.5-VL-3B80.9480.25
Adapted Qwen2.5-VL-3B82.9583.63
Adapted − unadapted (pp; full precision)+2.00+3.37

Although the absolute RefCOCOg scores differ from those reported in the technical report, our matched evaluation uses identical prompts, preprocessing, and Top-1 evaluation settings for both checkpoints, showing improvements of +2.00/+3.37 pp on val/test. Qwen2.5-VL Technical Report, Table 6

W3. Does the reported SOTA mainly come from low-cardinality cases?

We compared cardinality-stratified complete-set Pr on testA/testB (%):

ModelK = 0K = 1K = 2K ≥ 3K ≥ 5
Submitted Qwen2.5-VL-3B (Count+Boxes)75.13 / 70.3889.57 / 76.4685.57 / 74.0731.16 / 22.615.36 / 1.98
Qwen2.5-VL-3B (free-form)81.65 / 74.7988.93 / 76.6986.18 / 73.6428.50 / 20.936.25 / 4.40
InstanceVG71.09 / 65.1887.66 / 70.2280.35 / 68.2814.75 / 10.300.00 / 0.22
PropVG69.87 / 64.9788.52 / 71.7077.19 / 61.8711.99 / 5.821.43 / 0.22

The gain is not limited to empty or single-target cases. At K ≥ 3, the submitted model achieves 31.16%/22.61%, compared with 14.75%/10.30% for InstanceVG and 11.99%/5.82% for PropVG. At K ≥ 5, the full-scale free-form model reaches 6.25%/4.40%, also exceeding both specialists, although exact complete-set recovery remains challenging at high cardinality.

Q1. Does the count-preserving diagnostic hold for published GREC specialists?

Yes. We evaluated the released InstanceVG and PropVG checkpoints on 371 aligned count-preserving pairs:

Pair-level metric (%)InstanceVGPropVG
Both counts correct61.9947.17
Both complete sets recovered28.5724.53
Both sets recovered given both counts correct46.0952.00
Directional switch accuracy64.1560.38

Even when both counts are correct, both complete sets are recovered in only 46.09% and 52.00% of pairs. This independently validates the count–membership discrepancy. The public SSDE repository does not currently provide executable code or checkpoints for the same diagnostic.

AuthorsOpenReview ↗

Response to Reviewer hM2e-2

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the second response.

Q2. Can the separate decoder's box queries actually access the referring expression, and does further training close the gap?

Yes, the box queries directly access the referring expression. Although image tokens precede the referring expression in the causal LM prompt, the subsequent text tokens attend to all image tokens across the LM layers, and the decoder's learned object queries cross-attend jointly to the final-layer LM hidden states of both image and text tokens (including the referring expression). Its count head reads the final text hidden state, which is likewise conditioned on both the image and the referring expression. Thus, the observed membership errors are not caused by a missing text-conditioning pathway.

We also tested whether extended training or larger decoder capacity resolves this bottleneck under the same 12k/3k protocol:

Controlled configuration (single run)Val complete-set Pr (%)Count accuracy (%)Pr given correct count (%)Raw-query oracle Pr (%)
Original detached decoder (12 queries, 3 layers, 3 epochs)50.2482.8060.6770.89
Extended training (6 epochs)56.5985.7965.9674.22
Increased capacity (24 queries, 6 layers)46.7883.1556.2762.98
Native grounding (controlled)77.2286.2789.52—

Doubling the training budget raises the detached decoder's count accuracy to 85.79% (nearly matching Native's 86.27%), yet its Pr given correct count reaches only 65.96% (vs. 89.52% for Native). Moreover, ground-truth-guided selection from the 6-epoch decoder's unchanged candidate pool improves complete-set Pr from 56.59% to 74.22%, demonstrating the substantial contribution of target selection and count decisions to the remaining errors.

Across multiple seeds, official-val Pr is 50.94 ± 2.23% (original, 3 seeds), 55.72 ± 1.22% (6 epochs, 2 seeds), and 50.70 ± 5.54% (larger decoder, 2 seeds). On the same 373 query pairs, both target counts remain correct for 95.98%–97.86% of pairs, but both complete sets are recovered in only 0%–0.54%.

Across seeds 42/43/44, controlled Native achieves 76.85 ± 1.95% official-val Pr and 60.86 ± 4.83% directional switch accuracy, versus 50.94 ± 2.23% and 3.22 ± 1.88% for the corresponding original detached decoder. The same-seed Pr advantage is +25.91 ± 4.17 pp.

Q3. What does adaptation add over the unadapted 3B backbone on gRefCOCO?

We evaluated the unadapted and adapted Qwen2.5-VL-3B using the same Count+Boxes interface, official gRefCOCO splits, and complete-set evaluator:

Qwen2.5-VL-3B configurationVal Pr (%)TestA Pr (%)TestB Pr (%)
Unadapted, Count+Boxes65.6765.7758.69
Adapted, Count+Boxes (submitted)81.2476.1868.11
Matched-interface gain (pp)+15.57+10.41+9.42

The unadapted model already produces parseable outputs on essentially all examples (99.99%/99.99%/100.00%), confirming that the gains reflect improved complete-set grounding rather than output-format compliance.

AuthorsOpenReview ↗

Response to Reviewer hM2e-3

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the third response.

Q4. Why use Count+Boxes in the submission if free-form performs better?

The submitted model used our original Count+Boxes, resampled-order formulation to explicitly represent target cardinality within the autoregressive grounding response. The controlled 12k comparison of 81.85% (free-form, fixed order) versus 77.22% (Count+Boxes, resampled order) varied both output format and target ordering, and was not a matched full-training-set comparison. We therefore evaluated the free-form, fixed-order variant at full scale on the same gRefCOCO splits:

SplitSubmitted Count+Boxes Pr (%)Full-scale free-form Pr (%)Gain (pp)95% paired image-cluster bootstrap CI (pp)
Val81.2484.06+2.83[+2.37, +3.28]
TestA76.1877.28+1.10[+0.60, +1.59]
TestB68.1169.18+1.07[+0.42, +1.70]

Our controlled 2 × 2 study isolates both choices: removing the explicit count field improves Pr by +2.96 pp (fixed order) and +3.19 pp (resampled order); fixed-order supervision adds +1.67 pp (Count+Boxes) or +1.44 pp (free-form). These factor-specific improvements are established in the controlled 12k setting, while the full-scale comparison demonstrates the gain of the combined free-form, fixed-order configuration. Full-scale gains are calculated from unrounded scores, with 10,000 paired image-cluster bootstrap resamples for the confidence intervals.

Reviewer_5AsbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Generalized Referring Expression Comprehension, an expression may refer to zero, one, or multiple objects and a prediction is correct only if the full target set is recovered. The authors argue that a model can predict the right number of targets without selecting the right objects. That target count can be partly inferred from language and dataset priors. They propose Native Complete-Set Grounding fine-tunes qwen to output the complete box set through its native autoregressive coordinate generation instead of a separate set-prediction decoder. They report sota Pr on all three gRefCOCO splits.

Critical strengths
  1. Simply fine-tuning an MLLM to emit the full box set in its native output format beats specialized grey architectures by +7. +4. +7. Pr on val testA testB over the best prior results in Table 1. This is a useful empirical data point for GREC.
  2. The cardinality-wise analysis show both counting and set membership break down as the number of targets grows. for example, Pr given correct count falls to 24.59% for K>=5 on testA, so high-cardinality grounding is the main remaining difficulty.
Critical weaknesses
  1. the central claim rests on a weak baseline the authors built themselves: a 12-query DETR-style decoder trained on a 12k-sample subset. It gets 0.00% Pr on K>=3. When the expression is changed to a different object set in the same image, its predictions still overlap for 86.86% of the pairs. Its val Pr 50.24% is more than 20 point below the published query-based methods 72.20 to 74.17%. so the evidence only shows that this baseline is weak, and the 50.24 to 77.22 comparison says little about where target selection should happen. I am confused.
  2. The visual-evidence ablation does not support the prior-reliance claim. With a mean or mismatched image, count acc is still scored against the original ground truth. Most of the remaining 50% comes from K=0 samples. On positive-target expressions, count acc drops from 88.38% to 25.12% and 27.51%. so I think the model does rely on the image.
Questions for authors
  1. Was ablation run on the Native model rather than the separate decoder? Its normal-condition val count acc. match the main 3b model. It is also pooled over val and testA B, while the decoder is only evaluated on val. If so, the ablation shows neither that the decoder relies on priors nor that Native reduces that reliance.
  2. Could you add a competitive set-prediction baseline, for example one trained on the full training sets or with explicit query text interaction? could you also run the count-preserving query-pair diagnostic on published grec method?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5Asb-1

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the first response.

W1. Competitiveness of the separate-decoder baseline and scope of the architectural comparison

We compared Native grounding with a separate DETR-style set decoder using the same 12k-train/3k-development protocol and gRefCOCO val evaluator. Notably, the detached decoder already incorporates explicit query–text interaction: its learned object queries cross-attend directly to the final LM hidden states of both image and referring-expression tokens, and its count head reads the final text hidden state conditioned on the image and expression. To examine whether additional training or decoder capacity closes the gap, we evaluated the following configurations:

Configuration (single run)Val complete-set Pr (%)Count accuracy (%)Pr given correct count (%)Raw-query oracle Pr (%)
Original detached decoder (12 queries, 3 layers, 3 epochs)50.2482.8060.6770.89
Extended training (6 epochs)56.5985.7965.9674.22
Increased capacity (24 queries, 6 layers)46.7883.1556.2762.98
Native grounding (controlled)77.2286.2789.52—

With extended training, the detached decoder reaches 85.79% count accuracy, close to Native's 86.27%, but only 65.96% Pr given correct count, versus 89.52% for Native. Selecting ground-truth-matched boxes from the decoder's unchanged predicted candidate pool raises its complete-set Pr from 56.59% to 74.22%. This 17.63 pp gap identifies target selection and count decisions, in addition to candidate localization, as major sources of error.

On the original 373 count-preserving query pairs, the detached decoder produces overlapping predictions for 86.86% of pairs; Native reduces this to 24.66% and improves directional switch accuracy from 4.02% to 65.68%. Q2 additionally tests the count–membership discrepancy on independently developed, full-dataset GREC models.

Across independent seeds, the original, 6-epoch, and larger detached decoders achieve 50.94 ± 2.23% (3 seeds), 55.72 ± 1.22% (2 seeds), and 50.70 ± 5.54% (2 seeds) official-val Pr, respectively. On the same 373 query pairs, both counts are correct in 95.98%–97.86% of cases across these runs, whereas both complete sets are recovered in only 0%–0.54%. Thus, the count–membership discrepancy persists with additional training and decoder capacity.

Under the same controlled protocol, Native achieves 76.85 ± 1.95% official-val Pr and 60.86 ± 4.83% directional switch accuracy across seeds 42/43/44. Its same-seed Pr advantage over the original detached decoder is +25.91 ± 4.17 pp, and both complete target sets are recovered in 27.88 ± 1.42% of the query pairs.

W2 and Q1. Which model was visually ablated, and what does the residual count accuracy establish?

The visual ablation was performed on the trained Native model rather than the separate decoder, pooling val/testA/testB. We replaced the input image while keeping the referring expression and original target count for evaluation. We report count accuracy under different visual inputs:

ExpressionsOriginal image (%)Mean image (%)Mismatched image (%)Majority baseline (%)
All expressions84.0551.8950.5036.42
Positive-target expressions88.3825.1227.51—

The substantial drop on positive-target expressions confirms that Native relies on visual evidence. Under mean-image ablation, count accuracy remains 97.89% for K = 0 but falls to 46.62% for K = 2, showing how empty-set cases can sustain aggregate count accuracy without valid visual evidence. This ablation is intended to characterize Native’s aggregate counting behavior; prior reliance in the detached decoder is not inferred from it. Expression-conditioned target selection is instead evaluated directly by the count-preserving query-pair diagnostics in W1 and Q2.

AuthorsOpenReview ↗

Response to Reviewer 5Asb-2

We thank the reviewer for the thoughtful comments and constructive feedback.Here is the second response.

Q2. Competitive full-dataset set-prediction baselines and count-preserving diagnostics on published GREC methods

Competitive full-scale baselines. We evaluated the released, full-training-set InstanceVG and PropVG checkpoints under their official configurations. They achieve 73.71% and 72.38% complete-set Pr on gRefCOCO val, respectively, compared with 81.24% for our full-scale Native model. These results establish competitiveness against independently developed GREC systems; W1 provides the controlled decoder comparison.

Independent count-preserving diagnostic. We then evaluated these specialists on 371 aligned query pairs, each containing the same image and target count (K = 2) but different referred instances:

Pair-level metric (%)InstanceVGPropVG
Both counts correct61.9947.17
Both complete sets recovered28.5724.53
Both sets recovered given both counts correct46.0952.00
Directional switch accuracy64.1560.38

Even when both counts are correct, at least one referred set is missed in 53.91% of pairs for InstanceVG and 48.00% for PropVG. Thus, correct cardinality does not ensure correct target membership even in competitive, fully trained GREC systems.

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summarized Response to All Reviewers

We thank the Reviewers, Area Chairs, and Program Chairs for their time and thoughtful feedback on our submission.We have posted detailed responses to the comments from Reviewer Ph6V, Reviewer aPq8, Reviewer hM2e and Reviewer 5Asb in their respective discussion threads and uploaded a revised manuscript, with revisions highlighted in blue.

We appreciate the reviewers' recognition of several aspects of our work:

  • Strong GREC performance. Reviewer Ph6V recognized the strong benchmark results. Our submitted 3B model achieves 81.24/76.18/68.11% complete-set Pr on gRefCOCO val/testA/testB.
  • Value of the count-preserving diagnostic. Reviewer aPq8 highlighted the usefulness of same-image query pairs for examining expression-conditioned target selection beyond count accuracy.
  • Experimental analysis. Reviewer Ph6V acknowledged the thorough empirical analyses supporting the effectiveness of our approach.

In response to the reviewers' questions, we conducted additional controlled experiments, extended the evaluation to independent GREC systems and newer MLLMs, examined frontier closed-source models, and audited the conventional REC evaluation protocol. We also revised the methodological presentation and expanded the appendix with detailed experimental evidence.

Our major revisions include the following:

  • Method clarification and controlled comparisons. Redesigned Figure 2, clarified how Native differs from separate set prediction, and added multi-seed, training-budget, decoder-capacity, and serialization controls.
  • Extended grounding diagnostics. Evaluated independently developed GREC systems on count-preserving query pairs and expanded the high-cardinality and visual-evidence analyses.
  • Generalization and evaluation. Added full-scale free-form training, Qwen3.5-4B transfer, zero-shot evaluations of four frontier closed-source MLLMs, and a detailed RefCOCOg protocol audit.
  • Efficiency and limitations. Extended end-to-end latency comparisons and examined sensitivity to the generation-length limit.

The following table maps the major revisions to their locations in the revised manuscript and the reviewer concerns they address.

PageLocationRevisionReviewer
4–5Sec. 3, Fig. 2Clarified Native's complete-set formulation, unordered serialization, and distinction from separate set prediction; redesigned Figure 2.Ph6V, aPq8, hM2e
7, 16–17Sec. 4.3, App. B.1, Tables 9–10Added three-seed and decoder training-budget/capacity controls (+25.91 ± 4.17 pp paired Pr).5Asb, aPq8, hM2e
7, 17Sec. 4.3, App. B.3, Table 12Tested InstanceVG and PropVG on 371 count-preserving query pairs.5Asb, aPq8, hM2e
8, 18Sec. 4.4, App. CClarified positive-target visual ablation and no-target effects on aggregate counting.5Asb
8, 15Sec. 4.5, App. A.1, Tables 4, 7Added 2×2 serialization controls and full-scale free-form results (84.06/77.28/69.18% Pr).Ph6V, hM2e
9, 18–19Sec. 4.6, App. DExtended high-cardinality analysis and comparison with published GREC specialists.aPq8, hM2e
9, 21Sec. 4.7, App. F.1, Table 22Evaluated complete-set adaptation on the newer Qwen3.5-4B backbone.aPq8
10, 22–23Sec. 4.7, App. G.1–G.3, Tables 24–26Added four frontier MLLMs on 400 stratified expressions and 371 query pairs.aPq8
23–24App. G.4–G.6, Tables 27–29Added identical-subset Base/Native results, matched Qwen2.5 adaptation (+10.50 pp overall; +26.00 pp at K=2), prediction-validity audit, and Qwen3.5 protocol calibration.aPq8
10, 22Sec. 4.7, App. F.2, Table 23Audited RefCOCOg prompt and coordinate extraction under strict and diagnostic parsing.hM2e
10, 20–21Sec. 4.8, App. E, Tables 18–20Reported end-to-end latency and output-length sensitivity.aPq8, 5Asb

Together, these revisions strengthen the evidence for our central finding: predicting the correct number of targets does not guarantee recovery of the complete referred set. They also clarify how Native performs target selection and localization through the MLLM's existing autoregressive pathway, while providing broader evidence across architectures, model scales, output formats, and zero-shot frontier MLLMs.

We thank the reviewers again for the questions and suggestions that helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.