acceptodds
Under review as a conference paper at ICLR 2027

Look Closer, Segment Better: Query-Guided Visual Token Unfolding

Abstract

Multimodal large language models (MLLMs) demonstrate strong vision–language understanding capabilities, yet precise pixel-level segmentation remains challenging. Existing methods primarily focus on mask representation and generation, while the visual tokens fed into the large language model are typically spatially compressed, weakening local details and limiting segmentation accuracy for complex objects. To address this issue, we propose query-guided ***V***isual ***T***oken ***U***nfolding (***VTU***), which enables fine-grained visual tokens to directly perform in vision–language interaction before mask prediction. ***VTU*** first processes coarse-grained visual tokens to achieve a global understanding of the image and the text query, and employs a lightweight router to select tokens for unfolding. The corresponding tokens are inserted before the segmentation token, allowing the segmentation representation to integrate global context and local details. We further introduce router supervision targeting mixed-region and a simple-sample skipping strategy to incorporate informative tokens while reducing unnecessary unfolding. Extensive experiments demonstrate that ***VTU*** achieves state-of-the-art performance across multiple segmentation tasks and different model scales with only a small number of additional visual tokens, while preserving general visual understanding capabilities. Anonymous link: [https://anonymous.4open.science/r/VTU](https://anonymous.4open.science/r/VTU).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Model, Image Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i27DOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a common issue of MLLM-based segmentation approaches where the visual tokens passed into the model are spatially compressed which limits the models to capture finer local details especially near the object boundaries leading to poorer segmentation accuracy. The authors mitigate the issue by proposing Visual Token Unfolding (VTU), which allows visio-linguistic interaction with fine-grained visual tokens before mask prediction. VTU first uses the coarse-grained visual tokens to perform global reasoning and then uses a lightweight router to selectively choose which fine-grained tokens to unfold. The authors implement their extension on InternVL3-2B and InternVL3-8B, achieving improvements across several segmentation benchmarks with limited computational overhead.

Critical strengths

The observation that spatial compression of visual tokens leads to a loss of local detail around object boundaries and in complex regions is intuitive and clearly motivated. The proposed approach, which uses a lightweight router to first identify complex coarse regions and then unfolds the corresponding finer-grained visual features and feeds them to the LLM before mask prediction, is interesting and offers a promising direction for improving segmentation accuracy without increasing the visual token budget.

Critical weaknesses

However the paper has a few critical issues which are listed below:

  • Although the paper is framed as a general approach for MLLM-based segmentation, all experiments are conducted with the InternVL3 family of models (2B and 8B). InternVL’s spatial compression mechanism yields both coarse-grained and fine-grained visual tokens, which the proposed approach relies on. It is unclear to me whether the method would extend to MLLMs with different visual encoders and token compression strategies, such as Qwen2.5-VL or Gemma 3. Evaluating the approach on additional MLLM families would substantially strengthen the claims of generality.

  • The authors mention in Appendix A.2 that they use a straight-through estimator (STE) to propagate gradients through discrete routing decisions, but they do not explain its implementation in detail. It is unclear how gradients are approximated through the threshold filtering and top-K selection, both of which are discrete operations, and how the downstream segmentation losses help the router learn its routing decisions.

  • The paper lacks a direct ablation that isolates the importance of query-guided routing. Although the authors compared against random and uniform selection, they do not establish whether conditioning of the router on the language query is necessary at all. Comparison with a visual-only routing mechanism (e.g., scoring tokens from the visual hidden states without the <UNFOLD> query) would better isolate the contributions of query-guided token selection. Similarly, an oracle routing experiment would help quantify the upper-bound on how effectively the learned router identifies informative regions.

Questions for authors

Please see the weakness section especially the issue with framing and generalizability.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

W1: Generalize to other multimodal large language models

Thank you for your constructive suggestions! To validate whether our method can be generalized to multimodal backbones other than InternVL3, we adapted and trained the SELF1E-style segmentation baseline and VTU on Qwen2.5-VL-3B [1] and Gemma-3-4B [2] respectively. These two MLLMs adopt different visual encoding and spatial compression mechanisms:

  • Qwen2.5-VL-3B employs a vision Transformer supporting dynamic resolution, which splits images into 14×1414\times14 pixel patches. Taking a 448×448448\times448 input as an example, the vision encoder generates 32×3232\times32 spatial tokens. Afterwards, the vision-language connector concatenates adjacent 2×22\times2 features and projects them via a two-layer MLP, compressing the 1024 fine-grained tokens into 256 coarse-grained tokens, where each coarse-grained position corresponds to 4 fine-grained positions.
  • Gemma-3-4B uses the SigLIP [3] vision encoder to process each image view at a resolution of 896×896896\times896, producing 64×6464\times64 spatial tokens. Afterwards, 4096 fine-grained tokens are compressed into 256 coarse-grained tokens via 4×44\times4 average pooling, which are then projected into the embedding space of the language model, where each coarse-grained position corresponds to 16 fine-grained positions. | MLLM | Mehtods | RefCOCO | RefCOCO+ | RefCOCOg | | :--- | :--- | :--- | :--- | :--- | | InternVL3-2B | SELF1E | 84.3 | 78.9 | 80.4 | | | VTU | 85.0 (+0.7) | 81.3 (+2.4) | 82.8 (+2.4) | | Qwen2.5-VL-3B | SELF1E | 84.5 | 78.6 | 81.1 | | | VTU | 86.1 (+1.6) | 81.9 (+3.3) | 83.6 (+2.5) | | Gemma-3-4B | SELF1E | 83.9 | 77.4 | 79.7 | | | VTU | 86.4 (+2.5) | 82.3 (+4.9) | 83.9 (+4.2) |

The experimental results are shown above. We can observe that:

  • VTU achieves consistent improvements across different multimodal backbones. Compared with the SELF1E segmentation baseline, VTU yields gains of 1.6%, 3.3%, and 2.5% on Qwen2.5-VL-3B, and gains of 2.5%, 4.9%, and 4.2% on Gemma-3-4B. Notably, the baseline performance on Gemma 3 is relatively low. One possible reason is that it adopts a higher spatial compression ratio. Each coarse-grained token is obtained by average pooling over 4×4 fine-grained positions, which weakens some local details before they are fed into the language model. After introducing VTU, the model can revisit query-related fine-grained visual features and achieve substantial improvements. This observation aligns with our design motivation of supplementing fine-grained visual tokens.
  • From a methodological perspective, the core of VTU is to supplement query-related fine-grained visual evidence rather than relying on one specific compression operation. When the visual representations input to the LLM undergo spatial compression or only provide coarse-grained information, the model’s access to local details may be restricted. By selectively introducing fine-grained features before compression, VTU enables the language model to further acquire such information before generating segmentation representations. Therefore, as long as the model retains accessible high-resolution visual features and can establish correspondences between coarse- and fine-grained positions, it is in principle eligible for VTU adaptation. The above results support that this design can function across different visual compression mechanisms.

We have added relevant experiments and detailed discussions (please see Appendix B.3) on generalization in the revised version.

[1] Qwen2.5-VL Technical Report. arXiv 2025.

[2] Gemma 3 Technical Report. arXiv 2025.

[3] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2: Implementation of straight-through estimator

We apologize for not elaborating on this implementation in the manuscript, and we have added the relevant implementation details in the Appendix A.2.

During the model backpropagation process, the mask loss is backpropagated through segmentation representations to the language model and our Router module. However, the discrete threshold filtering and Top-KK selection within our routing module are inherently non-differentiable. For discrete routing decisions, we adopt the Straight-Through Estimator (STE), enabling the routing module to be jointly optimized via explicit routing supervision and downstream task losses.

Straight-Through Estimator (STE) [1] is a commonly used technique in neural network training. It aims to preserve discrete operations in the forward pass while providing usable surrogate gradients for the backward pass. Drawing on the implementation ideas from NeuralSort [2] and differentiable top-K [3], we adopt hard selection in the forward pass and soft sorting in the backward pass.

Specifically, let the selection probabilities of the Router for NN coarse-grained visual location predictions be p=(p1,...,pN)p=(p_1,...,p_N), where a larger pip_i indicates that location ii has a higher priority for expansion. In the forward pass, threshold filtering and top-K selection are performed according to these probabilities to retain m≤Km \le K locations. The selection result is represented by a matrix H∈{0,1}m×NH\in\{0,1\}^{m\times N}. If the jj-th selected location is ii, then Hji=1H_{ji}=1, otherwise, Hji=0H_{ji}=0.

In backpropagation, to implement a differentiable surrogate path, we draw on the idea of NeuralSort [2] and construct a soft sorting matrix PP. Each row of PP corresponds to a rank, and each column corresponds to a candidate location. PjiP_{ji} denotes the continuous weight of assigning location ii to the jj-th rank. For example, the first row represents the soft assignment of each candidate location to the highest rank, and the second row represents its soft assignment to the second-highest rank. Unlike hard sorting, where each rank corresponds to exactly one location, the weights in each row of soft sorting sum to 1 and vary continuously with the predicted probabilities from the Router, which enables the provision of surrogate gradients for sorting decisions.

For threshold filtering, we employ smooth gating and construct a soft selection matrix with the same shape as HH: gi=σ(pi−τTg),Aji=Pjigi,j=1,...,m,g_i=\sigma(\frac{p_i-\tau}{T_{g}}),\quad A_{ji}=P_{ji}g_i,\quad j=1,...,m, where σ\sigma denotes the Sigmoid function, τ\tau is the selection threshold, and Tg>0T_g>0 controls the smoothness of gating. We then construct it using the straight-through estimator: AST=stopgrad(H−A)+A,Zj=∑iN(AST)jiFi.A_{ST}=stopgrad(H-A)+A,\quad Z_j=\sum_{i}^{N}(A_{ST})_{ji}F_i. Among them, stopgradstopgrad stands for stopping gradient propagation, FiF_i is a set of fine-grained visual tokens corresponding to position ii, and ZjZ_j is the jj-th group of tokens fed into the subsequent segmentation process.

During the forward pass, AST=HA_{ST} = H, so the model still obtains fine-grained features via hard selection. During the backward pass, gradients are estimated through the derivative of the Router probability with respect to the soft selection matrix AA, allowing the downstream task loss to participate in the optimization of the Router.

[1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv 2013.

[2] Stochastic Optimization of Sorting Networks via Continuous Relaxations. ICLR 2019.

[3] Differentiable Top-k Operator with Optimal Transport. NeurIPS 2020.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W3: Ablation of visual-only routing and query-guided routing

Thank you for your insightful comments! To address your concerns, we have conducted additional ablation experiments to validate the superiority of our query-guided routing.

We adopt VisionZip [1] to evaluate the effectiveness of the visual-only routing. This work scores each spatial location using the attention within the visual encoder without introducing representations related to text queries. We aggregate the scores into a coarse-grained grid over corresponding 2×2 regions, and then select up to 32 fine-grained tokens (K=8K=8) for expansion.

In addition, to measure the upper bound of improvement achievable by routing, we conduct GT-guided (Oracle) experiments. We bypass the Router's prediction and directly select unfolding positions based on Router labels constructed from the ground-truth mask. To maintain the same budget of at most 32 fine-grained tokens (K=8K=8), when there are more than 8 positive GT label positions, we uniformly sample 8 positions without replacement, if there are fewer than 8, all positions are retained.

Token selectionRefCOCORefCOCO+RefCOCOgAvg.
Visual-only (VisionZip-based)80.678.279.479.4
Query-guided (ours)85.081.382.883.0
GT-guided (Oracle)85.882.083.583.8

The results are shown in the table above, and we can draw the following conclusions:

  • Query-guided routing outperforms vision-only routing across all three datasets, with improvements of 4.4%, 3.1%, and 3.4% respectively. We attribute this to the fact that vision-only attention mainly reflects the overall information distribution of an image, yet visually salient regions are not necessarily the areas requiring fine-grained discrimination for the current query. For instance, for queries targeting different objects within the same image, the vision-only strategy does not adjust selected positions according to the referring expressions, and the limited expansion budget may be allocated to regions irrelevant to the target. In contrast, the query-guided routing can allocate fine-grained visual information conditioned on the current query. These results validate the effectiveness of query-guided routing.
  • Compared with the GT-guided (Oracle) reference, our method has gaps of 0.8%, 0.7%, and 0.7% on the three datasets, respectively. This indicates that the learned Router can achieve segmentation performance close to that of the GT-guided reference, while there remains room for improvement in position selection.

We have included detailed discussions on vision-only routing and Oracle evaluation (please see Appendix B.2) in the revised manuscript.

[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models. CVPR 2025.


Please feel free to contact us if you have any other questions.

Reviewer_RyxUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Multimodal large language models support language-guided segmentation but commonly process spatially compressed visual tokens. This compression motivates providing additional local evidence before constructing the segmentation representation. The paper proposes Visual Token Unfolding (VTU), which uses query-conditioned routing and mixed-region supervision to insert selected fine-grained tokens before the segmentation token. Experiments across multiple segmentation tasks and two model sizes report consistent improvements, including average referring-segmentation cIoU of 83.2% and 85.6% with at most 32 additional tokens.

Critical strengths
  1. VTU provides a technically coherent way to inject query-relevant spatial detail into segmentation-token generation while reusing existing visual features and avoiding an external mask decoder.
  2. Reported improvements span referring, generalized referring, reasoning, and semantic segmentation across both 2B and 8B models.
  3. Ablations meaningfully examine token selection, mixed-region supervision, sample skipping, router architecture, token budgets, and alternative model-based supervision.
Critical weaknesses
  1. The open-vocabulary training protocol is ambiguous because Section 4.2 specifies COCO-Stuff-only training, whereas Table 8 includes ADE20K, PASCAL Context, and PASCAL VOC without identifying the corresponding checkpoints.
  2. Comparisons do not clearly establish a no-unfolding baseline trained with the identical data mixture and update budget, or matched-budget coarse-token reinsertion, leaving the specific contribution of fine-grained unfolding incompletely isolated.
  3. The boundary-detail motivation lacks quantitative boundary and small-object evaluation, particularly since the router labels cells with foreground occupancy below 10% as negative, potentially overlooking thin or tiny targets.
  4. The “negligible latency” claim lacks a matched local baseline without unfolding, while Table 12 compares local VTU measurements against published baseline timings.
  5. Preserving general visual capabilities is overstated because both model sizes decline on six of eight benchmarks relative to InternVL3, although Appendix B.2 acknowledges these trade-offs.
Questions for authors
  1. Which training datasets were used for each starred and unstarred checkpoint in Table 4?
  2. Were the SELF1E-SEG baseline and VTU ablations trained with identical datasets, sampling schedules, resolutions, and trainable components?
  3. Does fine-grained unfolding outperform reinserting the same number of selected coarse tokens under otherwise identical conditions?
  4. What are the boundary-quality and small/thin-object gains, and how often does the router actually skip unfolding?
  5. What is the measured overhead against a local no-unfolding baseline, including end-to-end semantic segmentation requiring multiple category queries?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

W1&Q1: Clarification of open-vocabulary training protocols

We apologize for not clarifying this distinction in our manuscript. The VTU-2B/8B checkpoints use COCO-Stuff as the only semantic segmentation training dataset, without training on ADE20K, PASCAL Context (PC-59), or PASCAL VOC (PAS-20). The VTU-2B*/8B* checkpoints additionally incorporate ADE20K, PC-59, and PAS-20 to facilitate comparison with SELF1E* and HiMTok*, which include these datasets in their training splits.

In fact, under the strict open-vocabulary setting, the categories used in the evaluation set should not be included in the training set. We report this result only for comparison against other methods.

Thank you for your valuable feedback! We have refined our Table 10 in the revised version to clarify this point.


W2&Q2&Q3: Isolating the contribution of fine-grained unfolding

Thank you for your insightful comments! First, we would like to clarify that the baseline in Table 6 is SELF1E-SEG-2B, whose training data composition and sampling scheme differ slightly from VTU. Both adopt an input resolution of 448×448448\times448, and the trainable components of the visual encoder and LLM backbone are kept consistent.

To further distinguish the contributions of the training configuration and fine-grained token unfolding, we conducted additional experiments and designed the comparisons in the table below:

  • Baseline*:We use the same mixture training data and sampling schedules as our VTU, without visual token unfolding, while the Baseline refers to the results reported in SELF1E's original paper.
  • Coarse-token reinsertion:Retain the input format and routing mechanism of VTU, and insert the coarse-grained visual tokens at selected positions into the sequence as additional visual information.
  • VTU:Use the default setting K=8K=8 (i.e., a maximum of 32 visual tokens can be insertedd), and insert the fine-grained visual tokens corresponding to the selected positions. | Variants | Max. Additional Tokens | RefCOCO | RefCOCO+ | RefCOCOg | | :--- | :--- | :--- | :--- | :--- | | Baseline | - | 84.3 | 78.9 | 80.4 | | Baseline* | - | 84.4 (+0.1) | 79.2 (+0.3) | 80.1 (-0.3) | | Coarse-token reinsertion | 2 | 84.3 | 79.3 | 80.7 | | | 4 | 84.4 | 79.2 | 80.9 | | | 8 | 84.2 | 79.0 | 80.6 | | | 16 | 84.0 | 79.1 | 80.6 | | | 32 | 83.8 | 78.8 | 80.2 | | | 64 | 82.6 | 78.0 | 79.1 | | VTU (ours) | 32 (K=8K=8) | 85.0 | 81.3 | 82.8 |

We draw two key findings:

  • The performance gain of our method does not stem from the training configuration. As can be seen from the results in the first and second rows, when the same mixture training data and sampling schedules are adopted, there is no significant improvement compared with the metrics reported in SELF1E's original paper, and even a drop is observed (from 80.4 to 80.1 on RefCOCOg). This demonstrates that the performance improvement of VTU mainly comes from our methodological design.
  • The gain from coarse-grained reinsertion is limited and unstable. Under a small token budget, coarse-grained reinsertion achieves certain performance improvements over the baseline, yet the performance declines as the budget increases. With the same maximum budget of adding 32 tokens, VTU outperforms the baseline by 1.2%, 2.5% and 2.6% in cIoU on the three datasets respectively. Further increasement of coarse-grained tokens does not yield additional performance gains. Coarse-grained reinsertion allows the model to reprocess existing visual information but does not supplement independently accessible fine-grained features. In contrast, VTU unfolds the fine-grained regions corresponding to coarse-grained visual tokens and feeds them into the LLM, enabling it to distinguish local details more meticulously. The experimental results validate that the performance gain of our method arises from the introduction of fine-grained visual information.

We have updated the manuscript and included these results and discussions (please see Appendix B.4) in the revised version.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W3&Q4-1: Analysis of boundary and small/thin-object

Thank you for your valuable comment! We would like to clarify that filtering out cells with a router label threshold below 0.1 is not equivalent to excluding small objects. Objects that occupy a small area of the image may still have a high coverage ratio within the local cell they reside in. For example, even if a small object occupies only a tiny fraction of the whole image, a positive routing label will be assigned to that position as long as it covers more than 10% of the region corresponding to a certain token. In addition, negative routing labels only apply to the supervision of region unfolding. The model retains the original coarse-grained visual information and receives full mask supervision.

To further address your concerns, we conducted additional experiments. Specifically, we compared the boundary segmentation, small-object segmentation, and thin-object segmentation performance of VTU-2B and SELF1E-2B-SEG on RefCOCO/+/g validation sets.

We use Boundary IoU [1] to measure boundary quality. Objects whose ground-truth mask area accounts for no more than 10% of the image area are defined as small objects, and objects whose minimum rotated bounding rectangle of the ground-truth mask has an aspect ratio of at least 3 are defined as thin objects. We report the cIoU for these two subsets separately.

DatasetsMethodsBoundary IoUSmall-object cIoUThin-object cIoU
RefCOCOSELF1E-2B-SEG60.172.171.2
VTU-2B66.4 (+6.3)78.3 (+6.2)77.1 (+5.9)
RefCOCO+SELF1E-2B-SEG56.664.863.9
VTU-2B64.3 (+7.7)72.5 (+7.7)72.3 (+8.4)
RefCOCOgSELF1E-2B-SEG58.466.968.5
VTU-2B64.9 (+6.5)72.5 (+5.6)74.2 (+5.7)

As shown in the table above, we have the following findings:

  • VTU-2B achieves superior boundary segmentation quality on all three datasets. The Boundary IoU increases by 6.3%, 7.7%, and 6.5% respectively, which provides direct evidence that our method can improve the segmentation of fine spatial details.
  • Performance gains are also observed for small objects and slender objects. The cIoU for small objects increases by 6.2%, 7.7%, and 5.6% respectively, and the cIoU for slender objects increases by 5.9%, 8.4%, and 5.7% respectively. This indicates that the benefits of our method are not limited to large-area objects, but also extend to objects that are more sensitive to local visual details.

It is worth noting that we select a local coverage threshold greater than 0.1 to balance the preservation of detailed regions and the reliability of routing supervision. The goal of the routing module is to select regions to be unfolded under a limited token budget. If all positions with any intersection with the foreground are labeled as positive samples, regions containing only a small number of edge pixels will receive the same positive supervision, weakening the supervision's discriminative ability for regions with actual target content. Therefore, we adopt a relatively low coverage threshold to reduce weakly relevant labels generated by slight boundary overlaps while retaining local details related to the target.

We include the experiments and discussions related to boundaries, small objects and slender objects in Sec. 4.4 of the revised version, and add comparison figures showing the segmentation improvements achieved by our method in the Appendix C.2.

[1] Boundary IoU: Improving Object-Centric Image Segmentation Evaluation. CVPR 2021.


Q4-2: Analysis of unfolding skip rates

We apologize for not clarifying this in the manuscript. For simple sample skipping, we calculate the proportion of skipped queries to the total number of queries across three datasets, as shown in the table below.

DatasetsQuery countSkip countSkip rate (%)
RefCOCO10834390336.0
RefCOCO+10758378635.2
RefCOCOg4896167434.2

We can observe that the query skip rates of the three datasets are 36.0%, 35.2%, and 34.2%, respectively, with relatively consistent performance across datasets. This indicates that the skipping mechanism is not triggered only for a small number of queries, instead, it avoids the expansion of additional visual tokens for a considerable proportion of queries. This strategy represents a trade-off between reducing unnecessary expansions and preserving fine segmentation capabilities.

We have included the above experiments and discussions (please see Sec. 4.4) in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W4&Q5: Latency comparison with the no-unfolding baseline

Thank you for your valuable feedback! To address your questions, we conducted additional experiments. Specifically, we forced our model to directly output the segmentation token after generating the <UNFOLD> without appending any fine-grained visual tokens. We measured the average latency per query and the per-image latency for semantic segmentation tasks with different numbers of categories.

Task / Datasetsettings2B8B
Single-query segmentation (s/query)No unfolding0.06820.106
VTU (K=8K=8)0.0689 (+0.0007)0.114 (+0.008)
ADE20K (s/image)No unfolding10.14016.272
VTU (K=8K=8)10.265 (+0.125)17.372 (+1.10)
PC-59 (s/image)No unfolding4.0426.415
VTU (K=8K=8)4.081 (+0.039)6.898 (+0.483)
PAS-20 (s/image)No unfolding1.3302.175
VTU (K=8K=8)1.342 (+0.012)2.305 (+0.13)

As shown in the table above, VTU improves segmentation performance while introducing only limited additional latency. For single-query segmentation, the latency increases by just 0.7 ms and 8.0 ms for the 2B and 8B models, respectively. For semantic segmentation requiring class-specific queries, the additional cost accumulates with the number of categories, yet the relative latency increase remains below 10% across all evaluated settings. Together with the segmentation results, these findings demonstrate that VTU achieves a favorable accuracy–efficiency trade-off by selectively introducing fine-grained visual information at modest computational cost.

We have incorporated the above experiments and explanations (please see Appendix B.6) into the revised manuscript.


W5: Trade-off between general visual capability and segmentation capability

Thank you for your constructive suggestions! We would like to emphasize two points:

  • Compared with the SELF1E baseline, VTU achieves stronger segmentation performance while remaining competitive on general visual understanding tasks. Both model scales show a consistent trend of change. Compared with the original InternVL3 model, our method achieves performance gains on the GQA and POPE benchmarks. VTU-2B/8B obtains improvements of 1.2%/3.9% on GQA and 1.6%/1.2% on POPE, respectively. This demonstrates that the model trained with segmentation supervision still retains certain image understanding capabilities.
  • On the other hand, our method suffers performance degradation on the other six datasets compared with the original model. Among them, OKVQA requires out-of-image knowledge, TextVQA and DocVQA rely on accurate text reading and contextual understanding. AI2D involves symbols and relationships in schematic diagrams. Our segmentation supervision mainly constrains target regions and their spatial structures and does not directly provide the supervision required for the aforementioned capabilities. During joint training, the adaptation of shared parameters to the segmentation task may disrupt the balance of original capabilities.

We will rephrase the relevant conclusions in the revised manuscript and further clarify this trade-off in the Limitation (please see Lines 26-27, Line 107 and Lines 1339-1343).


If you have additional concerns, we would be happy to discuss them with you further.

Reviewer_NUZEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper has 3 contribution: 1. showing that MLLM-based segmentation methods might compress the image's fine-grained information due to the Patch-Merger module 2. propose VTU, using text interspersed with images input to inject key fine-grained patches towards MLLM. 3. SoTA performance.

Critical strengths
  1. SoTA performance and comprehensive evaluation.
  2. Sounded mechanism designs to ensure the router catch the key fine-grained patches.
Critical weaknesses
  1. The router selects edge regions of the input image for enhancement. While this design is intuitive, the paper would benefit from a more thorough analysis of why edge-focused enhancement is effective. In particular, the improvement appears more pronounced when using a larger MLLM (e.g., scaling from 2B to 8B with InternVL3) than with other methods. Since many recent MLLMs are trained with interleaved image–text inputs (e.g., <image><text><image>), could the authors clarify whether the gains primarily come from edge enhancement, or whether they are partly attributable to the input format or the model’s ability to process interleaved image–text inputs?

  2. To better isolate the contribution of edge enhancement, it would be helpful to include a controlled baseline for SELFIE-SEG that uses the same <image><text><image> input format but does not apply edge enhancement. Comparing this baseline with the proposed method would clarify how much of the performance gain is specifically attributable to edge enhancement.

  3. The baseline results reported in Table 6 (84.3, 78.9, and 80.4 for VTU-2B) appear quite strong and seem comparable to the state-of-the-art results reported in the paper. Could the authors clarify the baseline’s exact configuration, including its training and evaluation setup, and explain what may account for its strong performance? This information would help readers better assess the contribution of the proposed method over the baseline.

Questions for authors

Authors should solve my concerned weakness so that I can improve my rates (show me why the edged enhancement can improve performance, why reasoning Seg can also be improved, if <I><T><I> affect performance but don't perform isolated ablation.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

W1-1: In-depth analysis of the effectiveness of edge enhancement

Thank you for your insightful comments! Our motivation for applying edge enhancement stems from observations of the predictions produced by existing MLLM segmentation methods. Even when the model can localize targets well, its predictions may still contain local errors at locations with complex contours and intertwined foreground and background.

To further quantify this phenomenon, we conducted a region-level error analysis on the predictions of the baseline model SELF1E-SEG-2B over the RefCOCO/+/g validation sets. Specifically, we divide the image into three mutually exclusive regions: the GT boundary neighborhood, the foreground interior, and the background. The boundary neighborhood covers both the inside and outside of the contour. The error rate of each region is defined as the ratio of misclassified pixels to the valid pixels within that region.

DatasetsBoundary area proportion (%)Boundary errors in total (%)Boundary error rate (%)Foreground interior error rate (%)Background error rate (%)
RefCOCO10.8267.0812.044.060.48
RefCOCO+10.9055.2113.327.090.91
RefCOCOg10.5856.2412.656.420.78

The results show that areas near boundaries remain challenging regions that strong baselines have not yet adequately addressed. Across the three datasets, boundary neighborhoods account for only approximately 11% of pixels but contribute 55%–67% of all errors, with a markedly higher error rate than the foreground interior and the background.

These phenomena align with our design motivation. A coarse-grained visual unit near the boundary may cover both the foreground and the background at the same time. Spatial compression aggregates information from different locations, limiting the ability of the language model to access local visual evidence separately. To this end, we use mixed-region supervision to guide the Router to focus on positions requiring fine-grained discrimination and supplement the corresponding fine-grained features before generating segmentation representations.

We have added boundary comparison visualizations in the Appendix C.2 in the revised manuscript to demonstrate the improvement of our method on boundary errors.


W1-2&W2: Isolating the performance gains of interleaved image–text inputs

Thank you for your valuable feedback! To clarify whether the performance gains stem from boundary enhancement or interleaved image-text input, we investigate several interleaved image–text input variants to determine whether the gains arise from the <I><T><I> format or also depend on the visual information introduced after the query.

Firstly, we append foreground visual tokens after the text query while retaining the original segmentation mechanism of SELF1E-SEG-2B.

Additional Visual FeaturesRefCOCORefCOCO+RefCOCOgAvg.
-84.378.980.481.2
Foreground tokens84.5 (+0.2)79.3 (+0.4)80.7 (+0.3)81.5 (+0.3)

We can find that appending foreground visual information merely through the interleaved input format yields relatively limited gains. With the original segmentation mechanism of SELF1E-SEG-2B retained, foreground tokens are appended after the query, resulting in cIoU improvements of 0.2%, 0.4%, and 0.3% across the three datasets respectively. This shows that the interleaved inputs can yield certain improvements, yet it cannot explain all the gains obtained by our mixed region selection.

Secondly, we compare fine-grained tokens selected from the full image, foreground, intra-foreground, and mixed-regions (ours) based on our VTU.

Additional Visual FeaturesRefCOCORefCOCO+RefCOCOgAvg.
Full image81.178.479.779.7
Foreground82.779.880.881.1
Intra-foreground81.979.180.080.3
Mixed-regions (ours)85.081.382.883.0

We can observe that variants using the same interleaved input format also exhibit clear performance differences. Full-image selection distributes the limited token budget across the entire scene, reducing coverage of target-related details. Foreground selection focuses on the target but still includes interior regions already adequately represented by coarse-grained tokens. Intra-foreground selection excludes boundaries and foreground–background interfaces that require fine-grained representations, limiting the contribution of additional information to local detail refinement. In contrast, our method concentrates the budget on locations where pixel assignments are more difficult, achieving an average cIoU of 83.0%. These experiments show that interleaved inputs do not necessarily yield consistent performance improvements, and the selection of fine-grained visual features is critical to obtaining these performance gains.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

(continue)

Intuitively, interleaved image–text inputs allow models to associate visual content with its textual context. For example, MM1.5 [1] leverages interleaved pre-training to support in-context learning and multi-image reasoning, while MANTIS [2] uses interleaved multi-image instruction tuning to improve cross-image co-reference, comparison, and temporal understanding. In these settings, different images often provide complementary evidence, and interleaving helps establish their relationships with the text. In contrast, our task requires pixel-level prediction for a target within a single image, where appending visual inputs does not necessarily introduce the spatial details needed for precise segmentation. Repeating coarse-grained features or supplementing already well-represented intra-foreground allows the model to revisit visual content after the query, but may offer limited information for resolving local ambiguities. VTU instead introduces fine-grained features from mixed foreground–background regions, exposing local details that are difficult to distinguish in compressed representations and thereby supporting more accurate boundary delineation and pixel assignment.

[1] MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning. ICLR 2025.

[2] MANTIS: Interleaved Multi-Image Instruction Tuning. arXiv 2024.


W1-3: Analysis of model scale

Thank you for raising this important point! To analyze the effect of model scale on performance gains, we compare foreground and mixed-regions selection on InternVL3-2B and 8B. Both settings use the same interleaved input format, keeping the input format consistent across the comparison.

MethodsAdditional Visual FeaturesRefCOCORefCOCO+RefCOCOgAvg.
VTU-2BForeground82.779.880.881.1
Mixed-regions85.0 (+2.3)81.3 (+1.5)82.8 (+2.0)83.0 (+1.9)
VTU-8BForeground84.181.982.782.9
Mixed-regions87.3 (+3.2)83.8 (+1.9)84.9 (+2.2)85.3 (+2.4)

The results show that when scaling from 2B to 8B, average performance increases from 81.1 to 82.9 with foreground selection, a gain of approximately 1.8%, and from 83.0% to 85.3% with mixed-regions selection, a gain of approximately 2.3%. Despite sharing the same interleaved input format, mixed-regions selection benefits more from scaling, with its advantage over foreground selection increasing from approximately 1.9% to 2.4%. This suggests that the scaling gains are not determined solely by the shared input format, but also depend on the content of the additional visual information and how effectively the model uses it.

We believe that larger backbones may have stronger capabilities for distinguishing local features and integrating visual and linguistic information, enabling them to better exploit fine-grained evidence at mixed foreground–background regions. Model capacity and fine-grained visual information thus appear complementary, allowing VTU to make more effective use of detail enhancement and achieve greater performance gains as model scale increases.

We have included the relevant experiments and analyses in the Sec. 4.4 of revised manuscript.


W3: Clarification of baseline configuration

Thank you for your valuable feedback! We would like to address this question from two perspectives:

  • The baseline in Table 6 is SELF1E-SEG-2B trained with enhanced segmentation. Its cIoU on the RefCOCO, RefCOCO+, and RefCOCOg validation sets are 84.3%, 78.9%, and 80.4% respectively, consistent with the results of the corresponding model in the original SELF1E paper and our Table 1. Built upon high-resolution visual feature modeling, this version further strengthens segmentation capability by adjusting the sampling strategy for multi-task data. Specifically, a 20× sampling is applied to the RefCOCO/+/g training data to increase the training frequency of the referring segmentation task. We have specified its name and supplemented the corresponding training and evaluation settings in the revised manuscript.
  • VTU achieves further improvements over the baseline on multiple tasks. For referring segmentation, compared with SELF1E-SEG-2B, the average cIoU of VTU-2B across eight evaluation splits increases from 81.4 to 83.2 (+1.8). On other tasks, against the baseline model, the average cIoU/gIoU of ReasonSeg rises from 60.8 to 66.2 (+5.4), gRefCOCO goes from 45.6 to 48.6 (+3.0), and the average mIoU over three open-vocabulary semantic segmentation benchmarks improves from 35.2 to 53.0 (+17.8). These results demonstrate that VTU delivers additional improvements on top of a strong baseline and shows its applicability across diverse segmentation tasks, rather than only bringing benefits on a single benchmark.
AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Q1: Gains for reasoning segmentation

Thank you for your insightful suggestions! Reasoning segmentation requires a model to possess both visual reasoning capability and segmentation capability. The former is used to understand the implicit intent in the query and locate the target, while the latter is used to accurately predict the pixel-level mask of the target.

Accordingly, we designed two diagnostic experiments to validate these two components respectively. specifically, we first prompt the model to directly output the category name or a brief description of the referred target based on the original image and reasoning query. We then check whether it matches the ground-truth target and calculate the target recognition accuracy. After that, we rewrite the target name or description into a direct segmentation instruction, such as "Please segment the wheelchair in the image", to replace the original reasoning query. The same instruction is fed to both models to validate their segmentation performance under the condition of a clearly specified target.

MethodsRecognition accuracyClear target segmentationOriginal reasoning segmentation
SELF1E-SEG-2B54.760.457.6
VTU-2B61.6 (+6.9)70.1 (+9.7)65.9 (+8.3)

The results are shown in the table above, and we can observe that:

  • Fine-grained visual information helps identify the target referred to by the reasoning query. The target recognition accuracy of VTU is improved from 54.7% to 61.6%. Reasoning queries often do not directly provide category names. Instead, they describe targets through object attributes, usages, or relationships with surrounding objects. Therefore, merely understanding the overall semantics of an image is insufficient, and it is necessary to distinguish local visual cues relevant to the query. Fine-grained visual information can supplement the unclear attributes and details in coarse-grained representations, helping the model distinguish candidate objects with similar semantics and map the implicit conditions in the query to specific targets more accurately.
  • With a clearly defined target, VTU can further improve the fineness of masks. After the segmentation category name is given, the model still needs to accurately determine the boundaries and local details of the target. Our method supplements fine-grained visual information at these positions to enable more sufficient characterization of target contours and details. Under the same explicit target description, the segmentation metric increases from 60.4% to 70.1%, indicating that the benefits lie not only in target recognition but also in precise segmentation after recognition.

In summary, our method improves both target recognition and fine segmentation capabilities in reasoning segmentation. It helps the model better understand the target referred to by the query, and characterizes the spatial extent and local details of the target more accurately by supplementing fine-grained visual information. Together, these two aspects boost the performance of reasoning segmentation.


We hope our response addresses your concerns. Please feel free to contact us anytime if you have any further questions.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer NUZE,

We hope this message finds you well. As the discussion period will conclude in about two days, we would like to ensure that our responses have addressed all of your concerns. If there are any additional points or feedback you would like us to consider, please feel free to let us know. Your insights have been extremely valuable in improving our work, and we are committed to resolving any remaining issues. Thank you again for the time and effort you have devoted to reviewing our work.

Sincerely,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time, effort, and careful evaluation of our manuscript. To help the Area Chair and reviewers evaluate our work more conveniently and comprehensively, we summarize our core contributions, the reviewers’ main concerns, the corresponding major revisions, and the feedback received during the rebuttal phase.


Core Contributions

This paper explores query-guided image segmentation with multimodal large language models. Spatially compressed visual tokens limit access to local details during segmentation-representation generation. We propose Visual Token Unfolding (VTU) to select query-relevant fine-grained visual tokens before mask prediction while retaining global context.

  • Fine-grained tokens for segmentation. VTU uses query-guided routing to introduce selected fine-grained features into the language model before mask prediction.
  • Targeted unfolding under a limited budget. Mixed-region supervision prioritizes detail-sensitive regions, while simple-sample skipping avoids unnecessary computation.
  • Effectiveness across tasks and model scales. Experiments demonstrate improvements across multiple segmentation tasks and model scales while maintaining competitive general visual capabilities.

Responses to Main Concerns and Major Revisions

We appreciate the reviewers' recognition of our work. NUZE highlighted the performance and routing design, RyxU valued the technical soundness and comprehensive evaluation, and i27D recognized the clear motivation and promise of selective unfolding.

  • Sources of performance gains (NUZE, RyxU): We added a no-unfolding baseline matched to VTU's training configuration, coarse-token reinsertion, and multiple interleaved-input controls. Training adjustments or interleaved inputs alone yield limited gains. Targeted fine-grained information from mixed regions contributes to the improvements, with greater benefits at larger model scales. These analyses are included in Sec. 4.4 and Appendix B.4.
  • Boundary enhancement and reasoning segmentation (NUZE, RyxU): Region-level analysis shows that boundary neighborhoods occupy approximately 11% of pixels but account for 55%–67% of baseline errors, motivating targeted detail enhancement. Additional evaluations confirm improvements in boundary quality and small/thin-object segmentation. Separate reasoning-segmentation diagnostics show gains in both target recognition and segmentation with explicit target descriptions. Quantitative analyses and visualizations are included in Sec. 4.4 and Appendix C.2, respectively.
  • Cross-model generalization and query-guided routing (i27D): VTU outperforms the corresponding segmentation baselines on Qwen2.5-VL-3B and Gemma-3-4B. Query-guided routing also surpasses VisionZip-based visual-only selection and approaches the oracle GT-guided reference, supporting cross-model applicability and the value of query conditioning. These experiments are included in Appendices B.2–B.3.
  • Implementation details and training settings (NUZE, RyxU, i27D): We clarify both the implementation and training setup. Appendix A.2 details straight-through gradient estimation and router optimization, while the revised training-data table distinguishes the datasets used by different model variants. We also clarify that SELF1E-SEG is a segmentation-enhanced baseline and provide matched-configuration experiments to better isolate VTU's contribution.
  • Efficiency and general visual capability trade-offs (RyxU): Local comparisons show latency overhead below 10%, with roughly one-third of queries skipping unfolding (Sec. 4.4, Appendix B.6). We also clarify general visual capability trade-offs and revise the corresponding claims.

Rebuttal Feedback and Summary

We thank Reviewer i27D for raising the score from 2 to 3 and explicitly stating that our detailed responses and additional experiments adequately addressed most of their concerns. Reviewer RyxU also confirmed that our rebuttal addressed their concerns and maintained their positive score of 3. For Reviewer NUZE, although we have not yet received further feedback, we have provided point-by-point experimental evidence and analyses addressing the concerns raised. The corresponding experiments, additions, and clarifications have been incorporated into the revised manuscript and will be retained in the final version.


We sincerely thank the AC and reviewers again for their time and constructive suggestions. We would be grateful if the AC could take into account the positive follow-up feedback, the additional evidence addressing the remaining concerns, and the corresponding manuscript revisions during the overall assessment.

Best Regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.