acceptodds
Under review as a conference paper at ICLR 2027

OneSEG: Improving Generalized Referring Segmentation with Semantic-Instance Decomposition

Abstract

Generalized referring segmentation extends conventional referring segmentation to scenarios where a referring expression may correspond to zero, one, or multiple instances, requiring models to jointly reason about referential semantics, instance cardinality, and target existence. Existing MLLM-based methods either autoregressively generate multiple [SEG] tokens to represent individual targets, making subsequent segmentation vulnerable to missing or duplicated tokens, or employ mask queries for parallel prediction, underutilizing the MLLM's native autoregressive reasoning capability. In this work, we propose OneSEG, a decoupled framework that preserves autoregressive referential reasoning while deferring instance prediction to decoding. Specifically, OneSEG employs a single [SEG] token to capture the shared referential semantics of all potential targets. Our Semantic-to-Instance Query Decomposition (SIQD) then transforms this holistic representation into instance-aware queries conditioned on visual evidence, enabling flexible multi-instance prediction without explicit instance enumeration during multimodal reasoning. Furthermore, we introduce an Instance Evidence Verifier (IEV) that verifies decoded candidates using explicit spatial evidence. Extensive experiments on generalized referring segmentation benchmarks demonstrate that OneSEG achieves state-of-the-art performance, validating the effectiveness of decoupling semantic reasoning from instance prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Generalized Referring Segmenation, Reasoning Segmentation, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YxVwOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OneSEG, a framework for GRES. Instead of generating multiple [SEG] tokens for individual instances, OneSEG uses a single [SEG] token to capture the shared referential semantics and performs instance prediction in the downstream decoder. Specifically, the SIQD module transforms the shared semantic representation into instance-aware queries using visual features. These queries are then decoded into segmentation masks. An IEV further evaluates each candidate mask using region-level visual evidence to filter invalid predictions.

Critical strengths

1.Clear motivation and effective framework design.
2.Strong experimental results. 3.Well-designed figures.

Critical weaknesses

None

Questions for authors

1.In Table 8, the authors compare OneSEG with single-[SEG] and multi-[SEG] baselines. However, OneSEG also introduces additional components, including SIQD, IEV, and a refined mask decoder. Therefore, it is difficult to determine how much of the improvement comes from the proposed single-[SEG] decomposition strategy rather than the additional modules. Could the authors provide a more controlled comparison using the same backbone, decoder, verifier, and training settings? This would help clarify the effectiveness of the core design.

2.OneSEG achieves substantial improvements in N-acc on gRefCOCO, which may also contribute significantly to its gIoU gains, given the large proportion of no-target samples. Could the authors report segmentation performance separately on target-present samples? Additionally, since the proposed method uses a fixed number of six instance queries, it would be helpful to provide instance-level evaluation across different target cardinalities, especially when the number of targets exceeds six. These results would better demonstrate the claimed multi-instance prediction capability.

2 comments
AuthorsOpenReview ↗

Response to Question 1

Response to Q1:

Thank you for the suggestion. We have updated Table 8 to isolate the contribution of the single-[SEG] decomposition strategy from the additional components. The controlled baselines use the same Qwen3-VL-4B backbone, basic mask decoder, and training settings. IEV, VSF, and the refinement decoder are excluded from all baseline variants, so the comparison focuses on how instance representations are formed.

MethodInstance representationgIoUcIoUN-acc
Single-[SEG] BaselineOne [SEG] for a global mask73.270.071.1
Multi-[SEG] BaselineAutoregressive multiple [SEG]76.772.478.6
Query-based BaselineIn-MLLM queries75.371.076.5
OneSEG BaselineOne [SEG] + downstream decomposition78.673.880.4
OneSEG (full model)One [SEG] + downstream decomposition82.576.288.5

Under these matched conditions, the OneSEG baseline achieves 78.6 gIoU, improving over the single-[SEG], multi-[SEG], and query-based baselines by 5.4, 1.9, and 3.3 points, respectively. It also achieves the highest cIoU and N-acc among the baseline variants. These gains are obtained without the additional fusion, verification, or refinement components.

The comparison with the single-[SEG] baseline examines the benefit of converting shared referential semantics into separate instance hypotheses instead of directly predicting one global mask. The multi-[SEG] and query-based baselines provide complementary comparisons: both maintain multiple instance representations, but form them through autoregressive token generation or queries within the MLLM. OneSEG instead forms these representations downstream, conditioned on the shared [SEG] representation and visual evidence. Its stronger performance under matched settings supports this decomposition strategy.

AuthorsOpenReview ↗

Response to Question 2

Response to Q2:

We agree that the improvement in N-acc contributes to the overall gIoU gain. Figures 7 and 8 already provide the sample counts and cardinality-specific segmentation results on gRefCOCO val and MUSE test, allowing target-present performance to be examined separately from no-target recognition. Figure 7 reports the number of samples in each target-cardinality group, and Figure 8 reports the corresponding gIoU and cIoU. Aggregating the nonzero-cardinality groups by their sample counts, OneSEG achieves 78.5 gIoU on target-present samples in gRefCOCO val.

The improvement in no-target recognition is also an intended outcome of our design. GRES requires both identifying the correct regions when valid referents exist and returning an empty prediction when they do not. OneSEG separates shared semantic understanding from downstream instance prediction, allowing target-existence decisions to incorporate spatial evidence from decoded candidates. In particular, IEV evaluates each candidate using its mask-conditioned visual features, global context, and instance representation. This enables the model to suppress candidates whose predicted regions do not support the referring expression.

Our objective is therefore to improve target identification and existence prediction through semantic-instance decoupling, rather than primarily to improve mask boundary quality. The gains in N-acc reflect this objective, while the target-present results separately characterize segmentation performance when valid targets exist. Reporting both provides a clearer view of the model’s behavior across the two types of GRES samples.

Regarding the fixed query count, Figure 8 includes results for samples with more than six targets. Despite using six instance queries, OneSEG maintains competitive gIoU and cIoU in this group. Notably, on MUSE test, both metrics for the more-than-six-target group exceed the corresponding averages over the full test set. Thus, the reported segmentation performance remains competitive beyond the default query count, as measured by the benchmark’s gIoU and cIoU metrics. The sample counts in Figure 7 are useful for interpreting these results. High-cardinality samples are relatively scarce in gRefCOCO val, whereas MUSE test contains substantially more multi-target examples. The MUSE results therefore complement the gRefCOCO analysis by evaluating performance on a dataset with more extensive coverage of multi-target cases.

For easier reference, we provide the numerical results from Figure 8 in tabular form below.

On gRefCOCO val:

Number of targetsSample countgIoU (%)
06,96688.5
13,99973.4
25,29780.8
346771.5
49660.5
52848.9
6561.8
>61253.5
Target-present9,90477.0
On MUSE test:
Number of targetsSample countgIoU (%)
------:---:
00N/A
130762.4
272769.9
316363.3
42,27666.9
567466.7
615767.9
>64872.4
Target-present4,35267.0
Reviewer_3fBNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes OneSEG, which aims to utilize one [seg] token for VLM to express all the candidate referred regions whether target regions are in input images. Constructed on the base VLM, OneSEG proposes Semantic-to-Instance Query Decomposition and Instance Evidence Verifier to distinguish the latent targets and refine the final masks. The former is a DETR like structure which uses initial query tokens to generate masks, which would be refine in the latter module by a score threshold. Extensive experiments prove the SOTA performance.

Critical strengths
  1. The structure of the paper is easy to understand and follow.

  2. Experiments shows great improvement compared with other VLM-based methods.

Critical weaknesses
  1. Novelty. The kernel design of OneSEG concentrates on the second half of the model. The MLLM is more like a bigger encoder to generate the visual features and the part of language appears to have minor impact. By this way, I'm concerned about the necessity to introduce VLM. Besides that, the design Semantic-to-Instance Query Decomposition is common from MDETR[1]-series methods like CGFormer[2] and so on.

[1] MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding [2] Contrastive Grouping with Transformer for Referring Image Segmentation

Questions for authors
  1. Instance Evidence Verifier (IEV) is more likely a common module for filter low quality masks. How it works if plug it on other existing methods with multiple [seg] tokens to filter and fuse candidate masks?

  2. In Tab.1, the comparisons involve with the specialized methods, which are missed in Tab.3. I wonder the entire comparisons to support your claims in general situations.

  3. I wonder the necessity to introduce MLLM. In fact, we hope VLM to bring explanations or better alignment. But in the paper, MLLM performs like a bigger backbone. In this way, the combination between convince multiple [seg] VLM and IEV can also achieve the motivation in abs., segmentation vulnerable. So I wonder if there are any quantity results to support your claims solider.

6 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses

Response to Critical Weakness 1:

Our use of an MLLM is motivated by the strong multimodal understanding required in complex referring scenarios, particularly those involving implicit instructions and multiple targets. Referring Segmentation methods can broadly be viewed as perception-oriented or cognition-oriented. Previous work SimVG[1], DeRIS[2] has highlighted insufficient multimodal understanding as a key bottleneck in grounding, motivating improvements in cognition beyond perceptual capacity alone. Smaller, task-specific models, such as MDETR [3], CGFormer [4], and OneRef [5], achieve strong performance on conventional grounding and referring segmentation tasks, while efficient architectures offer practical advantages for resource-constrained deployment. However, scenarios such as MUSE [6] require reasoning over implicit textual instructions to identify multiple targets, demanding semantic understanding beyond matching explicit object descriptions. MLLM-based methods [6–9] exploit the multimodal understanding capabilities of large models to address these complex referring and reasoning segmentation tasks. Our work follows this direction, using the MLLM to interpret the referring intent and provide semantic guidance for downstream instance prediction.

Regarding the SIQD module, our contribution is a decoupled way of bringing the established query-based paradigm into MLLM-based referring segmentation, combining the strengths of two existing directions while addressing their limitations. We acknowledge that SIQD uses query mechanisms familiar from MDETR [3] and CGFormer [4]. However, as discussed in the Introduction, the LISA–GSVA line [7][8] departs from this query-based paradigm, instead representing targets through autoregressively generated [SEG] tokens. This preserves autoregressive generation but abandons parallel query-based set prediction, making instance enumeration dependent on token generation. The PSALM–HyperSeg line [10][11], conversely, transfers mask queries from task-specific models into MLLMs but forgoes the MLLM’s native autoregressive generation capability. Our starting point is precisely this trade-off: how can the widely used query-based paradigm be adapted to MLLMs while retaining their autoregressive capability? OneSEG generates a single holistic referential representation and decomposes it into instance queries outside the MLLM, thereby decoupling semantic understanding from instance enumeration. Its novelty therefore lies in this integration of autoregressive semantic modeling and parallel query-based prediction, rather than in introducing entirely new building blocks. To clarify this positioning, we have cited MDETR and CGFormer in Related Work and added a comparison with CGFormer to Table 3.

[1] Dai et al. SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion. NeurIPS, 2024.

[2] Dai et al. DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergy. ICCV, 2025.

[3] Kamath et al. MDETR—Modulated Detection for End-to-End Multi-Modal Understanding. ICCV, 2021.

[4] Tang et al. Contrastive Grouping with Transformer for Referring Image Segmentation. CVPR, 2023.

[5] Xiao et al. OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling. NeurIPS, 2024.

[6] Ren et al. PixelLM: Pixel Reasoning with Large Multimodal Model. CVPR, 2024.

[7] Lai et al. LISA: Reasoning Segmentation via Large Language Model. CVPR, 2024.

[8] Xia et al. GSVA: Generalized Segmentation via Multimodal Large Language Models. CVPR, 2024.

[9] Zhang et al. SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction. arXiv:2605.20110, 2026.

[10] Zhang et al. PSALM: Pixelwise Segmentation with Large Multi-modal Model. ECCV, 2024.

[11] Wei et al. HyperSeg: Hybrid Segmentation Assistant with Fine-grained Visual Perceiver. CVPR, 2025.

[12] Wei et al. InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models. ICCV, 2025.

AuthorsOpenReview ↗

Response to Questions

Response to Q1:

IEV can also be applied to the multi-[SEG] setting, where each [SEG] token represents an individual instance. For each candidate mask, we perform mask pooling and global pooling on the MLLM’s visual features and combine these features with the hidden state of the corresponding [SEG] token, following Figure 3. The verifier then predicts a confidence score for each candidate, allowing masks associated with multiple [SEG] tokens to be filtered individually.

Response to Q2:

Thank you for your reasonable suggestions. We have expanded Table 3 to include specialized methods: PropVG, LAVT, CGFormer, and DETRIS, providing a more comprehensive comparison on conventional referring segmentation benchmarks.

AuthorsOpenReview ↗

Response to Question 3

Response to Question 3

Our use of an MLLM is motivated by the semantic understanding required in complex referring scenarios, while our core contribution is to decouple shared referential semantics from the formation of instance-specific representations. Previous work [1][2] identifies insufficient multimodal understanding as a key bottleneck in grounding. Although specialized models such as MDETR [3], CGFormer [4], and OneRef [5] perform well on conventional referring tasks, scenarios such as MUSE [6] require interpreting implicit instructions to identify multiple targets. Following MLLM-based segmentation methods [6–9], we use the MLLM to address these semantic demands. Specifically, OneSEG jointly processes the image and referring expression, producing a single [SEG] hidden state that captures shared visual-language semantics conditioned on the input context. Its role therefore extends beyond visual feature extraction.

Regarding to other variants like multi [SEG] or query-based, as discussed in the Introduction, query-based methods use the MLLM primarily as a vision-language encoder without exploiting autoregressive generation for segmentation, whereas multi-[SEG] methods couple visual-language understanding and instance representation within the same autoregressive generation process. OneSEG preserves autoregressive semantic modeling while deferring instance decomposition to downstream decoding.

To quantitatively evaluate this distinction, we have added controlled comparisons in Table 8 using the same MLLM, basic decoder, and training settings, with VSF, IEV, and the refinement decoder excluded. Under these matched conditions, OneSEG outperforms both the multi-[SEG] and query-based variants. These results support the contribution of the decoupled paradigm independently of IEV and the other auxiliary components.

Reviewer_BoZYOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

the paper studies mllm-based gres, an expression can refer to zero or one or multiple instances. Existing methods either generate multiple [SEG] tokens or insert multiple mask queries into the mllm. OneSEG instead keeps a single [SEG] token. It leaves instance decomposition and existence verification (IEV) to a lightweight decoder outside the MLLM. It is built on Qwen3-VL-4B and evaluated on gRefCOCO, MUSE and RefCOCO.

Critical strengths
  1. The main idea is simple and useful: instance enumeration don't have to happen inside the mllm and can be left to a decoder conditioned on a single [SEG] representation. The critique of multi [SEG] generation is reasonable.
  2. The framework is modular and trains on four RTX 4090 GPUs in about 48 hours. It reports strong results on GRES and RES benchmarks, so it could serve as a convenient baseline.
Critical weaknesses
  1. OneSEG uses Qwen3-VL-4B, which is much newer than the LLaVA backbone behind LISA and GSVA. 4.2 still credit the decoupled design for beating larger models. The paper's own numbers point to the backbone instead. On gRefCOCO val, a LISA-style baseline on Qwen3-VL-4B gets 73.2 gIoU against 65.2 for LISA-13B. The “Direct [SEG]” variant with no instance decomposition have already reached 78.3 close to the best prior methods. It is hard to tell how much of the gain comes from the method.
  2. No matched comparison with query-based methods. paper argues against this paradigm but never tests it under the same setup. table 8 has no mask-query variant, PSALM and HyperSeg are only zero-shot on gRefCOCO. Nothing query-based is reported on MUSE. The motivation is also weak. I am confused that OneSEG emits [SEG] after a fixed template with no reasoning before it. so is it basically a query-based model with one query inside the MLLM? The paper does not show that this works better than using several queries.
Questions for authors
  1. Can you re implement the strongest baselines on Qwen3-VL-4B or train OneSEG on a backbone used in prior work, so that the effect of the method itself can be isolated?
  2. How does OneSEG compare with a variant that puts Nq mask queries inside Qwen3-VL-4B with the same fusion module, decoder, verifier and training data including on MUSE?
3 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses

Response to Critical Weakness 1:

We acknowledge that a stronger backbone contributes to improvements on RES/GRES. The key distinction is that the 73.2 baseline in Table 4 and the 78.3 Direct [SEG] variant in Table 5 use different component configurations, despite sharing the same Qwen3-VL-4B backbone. Table 4 progressively introduces the proposed components, whereas Table 5 retains VSF, the refined mask decoder, and IEV to isolate the contribution of SIQD. Consequently, the strong Direct [SEG] result reflects both the backbone and the components already incorporated into that variant.

Table 4 quantifies the incremental contributions of our components with the backbone held fixed:

SettinggIoUcIoUN-acc
Baseline73.270.071.1
+ VSF74.271.974.6
+ SIQD79.474.781.9
+ Mask Decoder80.375.683.5
+ IEV82.576.288.5

Starting from the Qwen3-VL-4B baseline, VSF improves gIoU from 73.2 to 74.2. Adding SIQD then increases gIoU to 79.4, a gain of 5.2 points, while also improving cIoU by 2.8 points and N-acc by 7.3 points. Subsequently incorporating the refined mask decoder and IEV yields further gIoU gains of 0.9 and 2.2 points, respectively. The complete model improves over the same-backbone baseline by 9.3 gIoU, 6.2 cIoU, and 17.4 N-acc points. These results show that substantial gains remain after accounting for the stronger backbone.

Table 5 addresses a complementary question: how much does instance decomposition contribute when the other components are already present? To answer this, we keep VSF, the refined mask decoder, and IEV unchanged and progressively introduce the operations within SIQD. Direct [SEG] directly decodes the shared representation into a global mask without instance decomposition, but it is not the initial LISA-style baseline from Table 4.

SettinggIoUcIoUN-acc
Direct [SEG]78.373.780.4
Query + [SEG]80.174.283.6
+ Self-Attention81.174.885.5
+ Semantic Gating82.576.288.5

Even from this stronger Direct [SEG] starting point, conditioning learnable instance queries on [SEG] increases gIoU from 78.3 to 80.1. Self-attention further improves it to 81.1, and semantic-guided visual gating brings it to 82.5. The corresponding N-acc increases progressively from 80.4 to 83.6, 85.5, and 88.5. Thus, the full SIQD module provides an additional 4.2 gIoU, 2.5 cIoU, and 8.1 N-acc points while the backbone and surrounding components remain fixed. This supports the benefit of forming distinct instance hypotheses and grounding them in referent-relevant visual evidence.

The two tables therefore provide complementary evidence: Table 4 measures the incremental gains from the overall architecture, while Table 5 examines the contribution of SIQD within the otherwise complete framework. The SIQD gains differ between the tables because they are measured under different surrounding component configurations. We have clarified these settings in the discussion of Table 5 and will revise Section 4.2 to distinguish the overall comparisons across backbones from the controlled evidence supporting our architectural contributions.

AuthorsOpenReview ↗

Response to Weakness 2

We have added controlled comparisons with the query-based paradigm and updated Table 8 in the manuscript. All variants use Qwen3-VL-4B with matched training settings. The baseline variants share the same basic decoder and exclude IEV, VSF, and the refinement decoder, allowing us to compare the core segmentation paradigms.

MethodInstance representationgIoUcIoUN-acc
Single-[SEG] BaselineOne [SEG] for a global mask73.270.071.1
Multi-[SEG] BaselineAutoregressive multiple [SEG]76.772.478.6
Query-based BaselineIn-MLLM queries75.371.076.5
OneSEG BaselineOne [SEG] + downstream decomposition78.673.880.4
OneSEG (full model)One [SEG] + downstream decomposition82.576.288.5

Under these controlled conditions, the OneSEG baseline outperforms both the multi-[SEG] and query-based baselines, improving gIoU by 1.9 and 3.3 points, respectively. These results support the effectiveness of downstream instance decomposition independently of the additional components. We have also fine-tuned PSALM and HyperSeg on gRefCOCO and MUSE and added their results to the revised manuscript. To the best of our knowledge, prior work has not reported evaluations of these query-based methods on MUSE, so we supplemented this comparison with our own fine-tuning experiments.

Regarding reasoning, OneSEG follows the LISA/GSVA response template without generating an explicit reasoning trace before [SEG], maintaining a consistent setup for comparison. Explicit reasoning could be introduced through chain-of-thought supervision, allowing the model to generate a reasoning trace before [SEG].

As discussed in the Introduction, [SEG] serves as a shared visual-semantic representation in OneSEG, while instance-specific queries are formed downstream. Its hidden state is conditioned on both the image and referring expression. SIQD then combines this representation with learnable queries and visual evidence to obtain separate instance hypotheses. This differs from methods such as PSALM and HyperSeg, which place multiple instance-prediction queries inside the MLLM and use it primarily as a vision-language encoder, without exploiting autoregressive generation for segmentation. The distinction therefore concerns where and how instance representations are formed, rather than simply using one query instead of several.

To further examine this distinction on MUSE, we implemented an in-MLLM query variant using Qwen3-VL-4B with the same number of queries, fusion module, decoder, verifier, and training data as OneSEG. This variant uses backbone forward passes without calling the model’s generate method.

Methodval gIoUval cIoUtest gIoUtest cIoU
In-MLLM Queries51.652.959.562.0
OneSEG59.761.567.068.3

OneSEG consistently outperforms this variant, with gIoU gains of 8.1 points on validation and 7.5 points on test. These matched comparisons provide quantitative support for our design on multi-target reasoning segmentation.

AuthorsOpenReview ↗

Response to Question 2

Response to Question 2:

We implemented an additional query-based variant with (N_q=6) mask queries inside Qwen3-VL-4B, matching the number of instance queries used by OneSEG. This variant processes the queries through the MLLM backbone without calling its generate method. The fusion module, mask decoder, verifier, and training data are kept identical to OneSEG. This comparison therefore evaluates the two approaches with the surrounding segmentation components held fixed.

The principal difference is where instance-specific representations are formed. In the query-based variant, multiple mask queries are processed within the MLLM and subsequently used for instance prediction. In OneSEG, the MLLM generates a single [SEG] representation conditioned on the image and referring expression, and SIQD transforms this shared representation into instance-specific queries using visual evidence outside the MLLM. Both variants predict multiple candidate masks, but they organize the interaction between semantic understanding and instance formation differently.

The results on MUSE are as follows:

Methodval gIoUval cIoUtest gIoUtest cIoU
In-MLLM Queries51.652.959.562.0
OneSEG59.761.567.068.3

OneSEG improves gIoU by 8.1 points and cIoU by 8.6 points on the validation set. On the test set, the corresponding gains are 7.5 and 6.3 points. The improvements are consistent across both splits and both metrics, demonstrating the benefit of our design when the backbone, fusion module, decoder, verifier, and training data are matched.

MUSE is particularly relevant to this comparison because its queries require interpreting implicit instructions and identifying multiple targets, rather than simply matching explicit object descriptions. The results support our motivation to retain the MLLM’s autoregressive generation process for shared semantic modeling while moving instance formation to downstream decoding. In this setting, maintaining multiple instance queries inside the MLLM is less effective than deriving them from a shared referential representation through SIQD.

This experiment complements the baseline comparison in Table 8: Table 8 evaluates the core paradigms without the auxiliary components, whereas the MUSE comparison retains those components in both variants. OneSEG performs better in both settings, providing consistent evidence for the proposed decoupling strategy.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Question 1

Response to Question 1:

We have conducted controlled experiments using the same Qwen3-VL-4B backbone to separate the contributions of the segmentation paradigm and the proposed components from those of the backbone. These experiments address the question at two levels: comparing how instance representations are formed under matched settings, and measuring the incremental gains from the additional components.

Specifically, the updated Table 8 compares single-[SEG], multi-[SEG], query-based, and OneSEG baselines using the same backbone, basic mask decoder, and training settings. All baseline variants exclude VSF, IEV, and the refinement decoder, so the comparison focuses on the core segmentation paradigms.

MethodInstance representationgIoUcIoUN-acc
Single-[SEG] BaselineOne [SEG] for a global mask73.270.071.1
Multi-[SEG] BaselineAutoregressive multiple [SEG]76.772.478.6
Query-based BaselineIn-MLLM queries75.371.076.5
OneSEG BaselineOne [SEG] + downstream decomposition78.673.880.4
OneSEG (full model)One [SEG] + downstream decomposition82.576.288.5

Under these controlled conditions, the OneSEG baseline achieves 78.6 gIoU, exceeding the single-[SEG], multi-[SEG], and query-based baselines by 5.4, 1.9, and 3.3 points, respectively. It also improves cIoU and N-acc over all three baselines. The comparison with the single-[SEG] baseline evaluates the benefit of decomposing shared referential semantics into separate instance hypotheses. The comparisons with the multi-[SEG] and query-based baselines examine whether these hypotheses are better formed through token generation, within the MLLM, or through downstream decomposition. The results support our choice of downstream instance formation under the same backbone and training setup.

Complementing this comparison, Table 4 evaluates the incremental contributions of the components within the same Qwen3-VL-4B framework:

SettinggIoUcIoUN-acc
Baseline73.270.071.1
+ VSF74.271.974.6
+ SIQD79.474.781.9
+ Mask Decoder80.375.683.5
+ IEV82.576.288.5

Adding SIQD to the baseline with VSF improves gIoU by 5.2 points, while the refined mask decoder and IEV provide further gains of 0.9 and 2.2 points. Together, Tables 8 and 4 distinguish the benefit of the core decomposition strategy from the additional improvements provided by feature fusion, mask refinement, and instance verification. As discussed in our response to Critical Weakness 2, we have incorporated these controlled comparisons into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.