acceptodds
Withdrawn from ICLR 2027

UniSeek: Unified Referring Expression Localization with Count-Guided Instance Slots and Lightweight Add-ons

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Referring expression localization identifies a target set from natural language and represents its spatial extent using bounding boxes or masks. Unifying these tasks within a multimodal large language model (MLLM) requires handling variable target counts and heterogeneous geometric outputs while preserving fine spatial information. We propose UniSeek, a single model for referring expression comprehension (REC), referring expression segmentation (RES), and their generalized counterparts (GREC/GRES). UniSeek connects the MLLM backbone to downstream lightweight decoders through instance slots trained with explicit counting supervision. A Sparse Localization Decoder and a Dense Localization Decoder predict bounding boxes and masks, respectively. A Lightweight Image Local Detail Injector provides local detail features without an external pretrained image encoder. Experiments on both RefCOCO/+/g and gRefCOCO show that UniSeek achieves leading box-level localization results and competitive segmentation performance. For example, UniSeek-8B achieves a mean REC Acc@IoU=0.5 of 93.96% across eight splits, outperforming all compared baselines on every split. These evaluations clearly demonstrate that the proposed UniSeek re-calibrates the modern state-of-the-art performance under most settings of the interested tasks here.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Grounding, Multimodal Large Language Models, Computer Vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_azUuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

UniSeek provides a simple and straightforward unified framework for four referring expression localization tasks. By reusing the native visual features of Qwen3-VL and incorporating lightweight modules, it achieves strong performance across multiple benchmarks.

Critical strengths

Unified Framework: UniSeek provides a simple and coherent framework that unifies REC, RES, GREC, and GRES, supporting both bounding-box and mask predictions with varying target counts.

Strong Experimental Performance: The method achieves strong results across multiple benchmarks, particularly in REC and GREC, demonstrating its effectiveness in referring expression localization.

Critical weaknesses

Limited Methodological Novelty and Missing Key Ablation Studies for the Core Architectural Designs.

Questions for authors

Rating and Rebuttal Considerations

Regarding my rating, I initially intended to give this paper a Borderline score and then either maintain or increase my rating depending on the authors' rebuttal. If the authors satisfactorily address my concerns, I would be willing to increase my score from 2 to 3.

Weaknesses

1. Limited Methodological Novelty.

The core designs of this paper overlap substantially with existing research. For example, CoHD and HieA2G have already explored explicit target counting, while GSVA and PixelLM utilize the hidden states of special MLLM tokens for downstream segmentation. Moreover, DETR and SAM provide well-established foundations for task-specific geometric decoding. Similarly, PixelLM and Qwen3-VL-Seg have investigated native visual feature reuse and lightweight spatial feature enhancement.

Therefore, the main novelty of this work appears to lie in the integration of these existing techniques, particularly the unified interface that constrains instance-slot generation based on predicted target counts. While such an integration has practical value, it remains unclear what distinct and indispensable technical advantages the proposed design offers over existing approaches.

2. Missing Key Ablation Studies for the Core Architectural Designs.

Considering that Count-Guided Instance Slots constitute one of the main methodological contributions, I believe the current ablation studies are insufficient to fully demonstrate the necessity of this mechanism. In particular, I have the following questions:

(a) Why is it necessary to predict the target count before generating instance slots? I suggest comparing Count-Guided Slots with conventional fixed-size Instance Queries under otherwise comparable settings.

(b) Does explicit counting actually improve the localization of complete target sets? Table 6 primarily reports N-acc and T-acc. I suggest additionally reporting Exact Counting Accuracy and GREC Strict Set-Matching Accuracy for the same ablation configurations.

(c) Do the performance improvements mainly come from the auxiliary Counting Loss or from Constrained Decoding? I suggest disentangling these two components by always retaining the Count Token while independently enabling or disabling the auxiliary Counting Loss and the count-based constraints during decoding.

I understand that conducting all these experiments may be challenging within the rebuttal period. If the authors have already performed such experiments, reporting the results would be helpful. Otherwise, I would particularly appreciate the comparison in (a). The evaluation proposed in (b) should be relatively straightforward if the corresponding trained checkpoints are available, as it mainly requires additional inference and evaluation.

3. Missing Comparisons with Important Recent Baselines.

The REC experiments do not include WeDetect-Ref-4B, published at CVPR 2026, which achieves higher performance than UniSeek-8B on certain RefCOCO/RefCOCOg splits.

Similarly, the GRES experiments omit several relevant methods, including DeRIS-L, CoHD, and SSDE.

I recommend including these methods in the experimental comparisons to provide a more comprehensive assessment of UniSeek's competitiveness against recent approaches.

4. Insufficient Analysis of Computational Efficiency.

The paper emphasizes its Lightweight Add-ons and reports that the newly introduced modules contain approximately 28M parameters. However, the overall framework still relies on a multi-billion-parameter MLLM backbone and generates instance slots autoregressively.

Therefore, a small number of additional parameters does not necessarily imply efficient end-to-end inference.

I suggest reporting actual inference latency, FLOPs, throughput (e.g., FPS), and peak GPU memory consumption, particularly the inference cost under different numbers of target instances.

Additional Suggestions

The following comments are relatively minor and would not substantially affect my overall rating.

1. Figure 1 could be improved in terms of presentation and clarity.

The current layout appears somewhat simplistic, with considerable unused whitespace. I suggest making better use of the available rectangular space to improve the visual organization. In addition, Figure 1 lacks some important annotations and explanations, which makes the architecture less intuitive to follow.

2. Consider streamlining the main performance tables.

I would suggest reducing the number of non-MLLM-based methods in the main comparison tables. Including a few representative state-of-the-art approaches, or prioritizing more recent methods (e.g., those published in or after 2025), may make the comparisons more focused and easier to interpret. Other baselines could be moved to the appendix.

3. Key ablation results should be presented in the main paper.

Ablation studies that directly establish the necessity and effectiveness of the core architectural contributions should be included in the main text rather than being placed only in the appendix.

4. The abstract could better motivate the proposed method.

The abstract would benefit from a clearer explanation of the key limitations of existing approaches. Ideally, it should first identify the main challenges that previous methods have not adequately addressed, then explain how UniSeek specifically tackles these challenges and what improvements or new capabilities it provides. This would make the motivation and contributions more compelling.

Reviewer_BDQkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

UniSeek uses a shared multimodal large language model (MLLM) for referring expression comprehension and segmentation in both standard and generalized settings: REC, RES, GREC, and GRES. Built on Qwen3-VL-2B or 8B, it first predicts the number of targets for each expression, then generates instance slots whose hidden states condition separate box and mask decoders. For predicted counts from 0 to 9, constrained decoding enforces that many slots in each requested task section. Counts of 10 or more use an overflow token without an exact slot quota. The decoders reuse native multilevel visual features, supplemented by sub-patch detail injection, shared semantic feature aggregation, and query-conditioned mask refinement. Results on RefCOCO/+/g and gRefCOCO show competitive box localization and segmentation. Component ablations, evaluations at different IoU thresholds, and counting analyses examine the major design; exact counting deteriorates markedly for large target sets.

Critical strengths
  1. This method can be adapted to four downstream tasks involving bounding box localization and segmentation, and demonstrates strong performance on the two bounding box localization tasks, REC and GREC. UniSeek-8B has the highest REC accuracy among the methods in Table 1 on all eight splits. As shown in Table 3, the strict set matching accuracy of the GREC task shows an improvement compared to other comparison methods.
  2. The architectural implementation and experimental design of the entire method such as count prediction, autoregressive instance-slot generation and so on are relatively comprehensive. The MLLM approach, which employs two geometric decoders and provides a usable interface without requiring four independent backbone networks, can also be viewed as a significant contribution.
Critical weaknesses
  1. The segmentation results show uneven gains, and the counting analysis raises concerns about reliability for larger target sets. UniSeek-8B does not achieve the SOTA performance on RES or GRES tasks. Exact-count accuracy falls to 23.06–28.65% for 5–10 targets and 3.33–11.43% for more than 10. These cases are rare in the evaluated benchmarks, as the authors acknowledge, but aggregate results consequently provide limited evidence of high-cardinality reliability. Since exact-count predictions constrain the number of generated slots, the paper should quantify how undercounting, overcounting, and false-zero predictions affect downstream localization, and distinguish these failures from those under overflow decoding.
  2. Compared to existing MLLM models that combine four downstream tasks or previous work on counting supervision and prediction, UniSeek’s specific contribution lies in how count prediction guides instance-slot generation and connects to the geometric decoders. However, Table 6 evaluates the counting variants only through N-acc and T-acc, while Figure 5 reports exact-count accuracy only for the full models. These experiments do not isolate the mechanism’s contribution to exact cardinality or complete target-set localization. The absence of quantitative comparisons with GREx/enhanced ReLA and CoHD further limits the assessment of this contribution. Beyond that, Figures 6–7 illustrate successful cases but lack side-by-side comparisons with baselines and counting ablations on the same inputs. Together, these limitations make it difficult to determine whether the proposed slots improve instance coverage and correspondence.
Questions for authors
  1. What does count-guided slot generation improve beyond existing unified outputs and counting-based localization? Please try to provide comparable results for GREx/enhanced ReLA and CoHD, or explain the protocol differences preventing comparison.
  2. How much downstream error results from undercounting, overcounting, and false-zero predictions? Please try to report these errors and geometric performance by target cardinality, separating exact-count and overflow decoding. This allows for a more intuitive assessment of how counting prediction errors and overflow logic affect this method.
  3. Could the authors include side-by-side qualitative comparisons with relevant methods for box localization and segmentation across the four tasks? Figures 6–7 mainly illustrate UniSeek’s successful predictions, making its relative strengths and failure modes difficult to assess. Showing the same image–expression pairs with ground truth, predicted counts, and individual outputs from UniSeek, a relevant baseline, and a counting ablation would help distinguish improvements in counting, target coverage, and geometric accuracy.
  4. How should the large cIoU gap in Table 9 be interpreted when SAM2 receives the same UniSeek-predicted boxes? The magnitude of the difference raises questions about whether it reflects segmentation capability, differences in conditioning and task-specific training, or evaluation and preprocessing choices. Since these factors are not isolated, what conclusion does this comparison support about the proposed decoder?
Reviewer_CNSzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniSeek, a unified MLLM framework for REC, RES, GREC, and GRES. Its central design uses explicit count prediction to guide the generation of instance slots, whose hidden representations condition task-specific localization decoders. These decoders reuse multi-level features from the MLLM’s visual encoder, enhanced by a lightweight local detail injector. The paper evaluates REC/RES on RefCOCO/+/g and GREC/GRES on gRefCOCO, and presents several component ablations.

Critical strengths

UniSeek achieves strong REC and GREC results across the evaluated datasets and splits. Using explicit count prediction to organize instance slots and condition localization decoders is well motivated. Integrating this mechanism into a shared MLLM framework provides a coherent approach to handling variable target cardinalities and different geometric outputs.

Critical weaknesses
  1. The technical contributions are not sufficiently distinguished from prior work. REC/RES are single-target special cases of GREC/GRES, so supporting all four tasks does not, by itself, establish substantial novelty. Explicit counting supervision and count-guided localization have been seen in prior arts, such as GREx [1] and CoHD [2]. The visual detail module also combines established techniques, including multi-layer feature aggregation, shallow detail extraction, and gated fusion. The specific integration of count prediction with MLLM instance-slot generation may offer an incremental contribution, but its advantages over alternative counting head prediction and output-generation designs are not sufficiently established.

  2. The counting ablation in Table 6 reports only N-acc and T-acc, which measure target-presence recognition rather than exact counting or complete target-set localization. So, these results do not establish whether the proposed counting mechanism improves the final GREC/GRES performance, particularly the strong "strict set-matching accuracy" emphasized in the paper.

[1] GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation.

[2] CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentation.

Questions for authors
  1. Training and reproducibility: Could the authors provide a complete description of the training procedure, including the initialization, training, and freezing schedule of each module, and clarify whether all stages are included in the reported training budget?

  2. Counting ablation: Could Table 6 be supplemented with exact counting accuracy and the final GREC/GRES metrics, particularly strict set-matching accuracy for GREC and gIoU/cIoU for GRES?

  3. SAM2 comparison: In Table 9, UniSeek-8B box + SAM2-base achieves 46.84 cIoU on RefCOCO validation, whereas UniSeek achieves 82.64. To help explain this substantial gap, could the authors detail the SAM2 evaluation pipeline, including image preprocessing, box-coordinate transformations, candidate-mask selection, and postprocessing? Could they also report SAM2-base results using ground-truth boxes to help verify the pipeline?

  4. Evaluation coverage: Could the authors evaluate generalized referring performance on an additional benchmark, such as Ref-ZOM for GRES, to assess whether the improvements extend beyond gRefCOCO?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.