acceptodds
Withdrawn from ICLR 2027

SEmO: Seg+Edge minus Others Token with Progressive Mask Decoding for Reasoning Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning segmentation aims to generate precise segmentation masks from complex and implicit language queries. Prior approaches mainly rely on a single <SEG> token to represent all segmentation-relevant information and predict the final mask in a single decoding stage. This design makes it difficult to capture fine-grained target edges while suppressing false positives in the background. We propose SEmO, a novel framework that augments the conventional <SEG> token with two role-specific tokens, <EDGE> and <OTHERS>, together with a Progressive Mask Decoder(PMD). The three tokens explicitly represent the target region, its edge, and the background, respectively, providing complementary information for mask prediction. At each resolution stage, PMD aggregates their role-specific predictions and propagates the refined result to the next stage for coarse-to-fine mask refinement. These designs enable SEmO to predict more accurate target edges while effectively suppressing background false positives. Experiments demonstrate that SEmO achieves state-of-the-art performance on ReasonSeg and improvements on referring segmentation benchmarks. Moreover, PMD requires only 0.69M parameters and achieves faster inference than existing mask decoders, showing that the performance gains are obtained without sacrificing efficiency. All code and model parameters of SEmO will be made available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reasoning segmentation, multimodal large language models, progressive mask decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SwKZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SEmO, a reasoning segmentation framework using three tokens: SEG, EDGE, and OTHERS. The proposed Progressive Mask Decoder (PMD) combines these tokens to improve segmentation quality. Experiments on ReasonSeg and RefCOCO datasets show improved performance with a lightweight decoder.

Critical strengths
  1. The paper addresses important problems in reasoning segmentation, particularly inaccurate object boundaries and false-positive predictions.

  2. The proposed role-specific tokens provide a simple and understandable approach to separating target regions, boundaries, and non-target regions.

  3. The method achieves competitive performance on ReasonSeg, with improvements over the reported baseline methods.

Critical weaknesses
  1. Comparisons with related methods are limited. Important methods such as PixelLM and M²SA are missing from the segmentation performance comparisons.

  2. Reasoning segmentation is evaluated only on ReasonSeg. Additional benchmarks, such as MUSE and MMR, are needed to better evaluate the generalization of the method.

  3. The framework does not support multiple targets in a single query. This limits its use in more complex reasoning segmentation tasks.

  4. The ablation study does not clearly separate the effects of the new tokens from those of additional supervision. More controlled experiments are needed to support the main contribution.

  5. The proposed PMD is not compared with other progressive or multi-scale decoders. This makes it difficult to evaluate the contribution of the decoder design.

Questions for authors
  1. Can the authors provide performance comparisons with PixelLM(CVPR2024) and M²SA(ICLR2025), or explain why these methods were excluded?

  2. Can the authors evaluate the method on other reasoning segmentation datasets, such as MUSE or MMR?

  3. How can the OTHERS token be applied when a query refers to multiple target objects?

  4. Can the authors compare the proposed tokens with a single SEG token using the same supervision and separate prediction heads?

  5. Can the authors compare PMD with existing multi-scale or progressive decoders under similar settings?

Reviewer_spZuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends LISA-style reasoning segmentation from one SEG token to three role-specific tokens: SEG, EDGE, and OTHERS, representing the target, an inner-boundary band, and the background. The new labels are derived from existing masks without extra annotation. PMD predicts the three role maps at multiple resolutions, combines target and edge logits while subtracting background logits, and adds logits across stages. The evaluation covers ReasonSeg and RefCOCO/+/g, with token and stage ablations and a 13B model. The results are generally strong and the decoder is small, but the current evidence does not isolate the token mechanism, and some reported aggregation and comparison claims need clarification.

Critical strengths
  1. The role decomposition is clear and practically motivated. The edge and background labels are derived from the existing mask, and the appendix specifies adaptive erosion and empty-mask handling, so no extra manual annotation is required.

  2. PMD is lightweight and easy to follow. Table 3 reports 0.69M decoder parameters versus 4.06M for the listed SAM/LISA decoder, or about 83% fewer, together with lower decoder-only latency. The evaluation also spans reasoning segmentation, referring segmentation, and two model sizes.

Critical weaknesses
  1. The evaluation aggregation needs clarification. In Table 4(b), N=1 has short, long, and overall cIoU values of 45.5, 53.9, and 55.2. If short and long are disjoint, complete parts of overall and use the same predictions under Section 5.1, overall cIoU must lie between the two subset values. The paper does not provide enough split, count, and checkpoint details to determine whether this is a table or protocol issue. The headline gains are also not always measured against the strongest listed baseline: the table-level gain is 1.5 points for validation cIoU over READ and 1.5 points for test-overall gIoU over FlowSeg, rather than the larger quoted values.

  2. The ablations do not isolate the claimed role-token and progressive-decoding effects. Removing EDGE or OTHERS also removes its branch, aggregation term, and loss. Changing the number of PMD stages can also change resolution, capacity, and the number of supervised loss terms. The current results therefore support the complete package, but not the independent benefit of role-specific token states or logit propagation. Direct boundary and background false-positive metrics are also missing.

Questions for authors
  1. Are short and long disjoint and complete parts of overall? Please provide sample counts, split membership, checkpoints, and aggregation statistics, explain the 45.5/53.9/55.2 values in Table 4(b), and state whether corrected values change any ranking or conclusion.

  2. With the same PMD, data, three derived labels, losses, and parameter budget, how does one SEG hidden state with three projection heads compare with the full three-token model? For the stage ablation, can the final resolution and capacity be fixed and the total supervision weight normalized? Can the existing predictions also be evaluated with a boundary metric and false-positive area outside the target?

Reviewer_jRDhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates reasoning segmentation from complex and implicit language queries, focusing on imprecise target boundaries and background false positives in existing embedding-as-mask approaches. To address these issues, the paper proposes SEmO, which augments the conventional ⟨SEG⟩ token with ⟨EDGE⟩ and ⟨OTHERS⟩ tokens to separately represent the target region, its boundary, and the background. Supervision for the additional tokens is derived from existing mask annotations without requiring further labeling. To integrate these representations, the paper introduces a Progressive Mask Decoder (PMD) that aggregates role-specific predictions at each resolution and propagates the resulting logits across stages for coarse-to-fine mask refinement. Experiments on ReasonSeg and referring segmentation benchmarks indicate improved segmentation performance, while the 0.69M-parameter PMD achieves lower decoding latency than the compared mask decoders. Evaluation confined to LLaVA-1.5 leaves the applicability of the improvements to more recent VLM backbones, such as Qwen2.5-VL or Qwen3-VL, unclear.

Critical strengths
  • Role-specific modeling without additional annotation. SEmO assigns separate tokens and supervision targets to the target region, its boundary, and the background. The auxiliary labels are derived directly from existing segmentation masks, allowing these roles to be supervised without additional manual labeling.

  • Lightweight mask decoding. The proposed PMD combines predictions across resolutions using feature projection, similarity matching, and logit aggregation. It contains 0.69M parameters and reports lower decoder latency than the compared SAM and PixelLM decoders (Table 3).

  • Evaluation across tasks and model scales. Experiments cover ReasonSeg and RefCOCO/+/g, with additional results using a 13B backbone. Ablations on role-specific tokens and the number of decoding stages provide evidence about the contributions of the proposed components (Table 4, Appendix D).

Critical weaknesses
  • The methodological novelty appears incremental. The main distinction is the explicit assignment of region, boundary, and background roles to separate tokens, while multiple-token representations and multiscale decoding have already been explored in prior work such as PixelLM. PMD mainly combines feature projection, similarity matching, and logit accumulation within a lightweight architecture.

  • The motivation is generally clear: the paper aims to improve boundary precision and suppress background false positives through role-specific tokens and progressive decoding. However, the connection between these errors and the proposed architectural changes is not sufficiently established. The qualitative examples illustrate segmentation failures but do not demonstrate that these failures specifically arise from overloading a single ⟨SEG⟩ token or using single-stage decoding. Decoder capacity, feature resolution, and supervision could also contribute. Controlled comparisons that isolate these factors, together with boundary-specific metrics and false-positive measurements, would better substantiate the rationale for the proposed design.

  • The comparisons in Tables 1 and 2 omit several recent and relevant baselines. For reasoning segmentation, InstructSeg, RESAnything, X-SAM, LENS, and SAM 3 Agent also report ReasonSeg results but are absent from Table 1. For referring segmentation, Table 2 lacks comparisons with RESAnything (neurips25), FlowSeg/STAMP(cvpr26), SAM3Agent (iclr26)... (For Table 2, comparisons with a few representative recent baselines would sufficiently contextualize the generalization results; state-of-the-art performance on the RefCOCO benchmarks is not necessary.)

  • The experiments are limited to LLaVA-1.5, leaving it unclear whether the improvements generalize to more recent VLM backbones, such as Qwen2.5-VL or Qwen3-VL.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.