- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses a common issue of MLLM-based segmentation approaches where the visual tokens passed into the model are spatially compressed which limits the models to capture finer local details especially near the object boundaries leading to poorer segmentation accuracy. The authors mitigate the issue by proposing Visual Token Unfolding (VTU), which allows visio-linguistic interaction with fine-grained visual tokens before mask prediction. VTU first uses the coarse-grained visual tokens to perform global reasoning and then uses a lightweight router to selectively choose which fine-grained tokens to unfold. The authors implement their extension on InternVL3-2B and InternVL3-8B, achieving improvements across several segmentation benchmarks with limited computational overhead.
Critical strengths
The observation that spatial compression of visual tokens leads to a loss of local detail around object boundaries and in complex regions is intuitive and clearly motivated. The proposed approach, which uses a lightweight router to first identify complex coarse regions and then unfolds the corresponding finer-grained visual features and feeds them to the LLM before mask prediction, is interesting and offers a promising direction for improving segmentation accuracy without increasing the visual token budget.
Critical weaknesses
However the paper has a few critical issues which are listed below:
-
Although the paper is framed as a general approach for MLLM-based segmentation, all experiments are conducted with the InternVL3 family of models (2B and 8B). InternVL’s spatial compression mechanism yields both coarse-grained and fine-grained visual tokens, which the proposed approach relies on. It is unclear to me whether the method would extend to MLLMs with different visual encoders and token compression strategies, such as Qwen2.5-VL or Gemma 3. Evaluating the approach on additional MLLM families would substantially strengthen the claims of generality.
-
The authors mention in Appendix A.2 that they use a straight-through estimator (STE) to propagate gradients through discrete routing decisions, but they do not explain its implementation in detail. It is unclear how gradients are approximated through the threshold filtering and top-K selection, both of which are discrete operations, and how the downstream segmentation losses help the router learn its routing decisions.
-
The paper lacks a direct ablation that isolates the importance of query-guided routing. Although the authors compared against random and uniform selection, they do not establish whether conditioning of the router on the language query is necessary at all. Comparison with a visual-only routing mechanism (e.g., scoring tokens from the visual hidden states without the <UNFOLD> query) would better isolate the contributions of query-guided token selection. Similarly, an oracle routing experiment would help quantify the upper-bound on how effectively the learned router identifies informative regions.
Questions for authors
Please see the weakness section especially the issue with framing and generalizability.
5 comments
Rebuttal by Authors (2/3)
W2: Implementation of straight-through estimator
We apologize for not elaborating on this implementation in the manuscript, and we have added the relevant implementation details in the Appendix A.2.
During the model backpropagation process, the mask loss is backpropagated through segmentation representations to the language model and our Router module. However, the discrete threshold filtering and Top- selection within our routing module are inherently non-differentiable. For discrete routing decisions, we adopt the Straight-Through Estimator (STE), enabling the routing module to be jointly optimized via explicit routing supervision and downstream task losses.
Straight-Through Estimator (STE) [1] is a commonly used technique in neural network training. It aims to preserve discrete operations in the forward pass while providing usable surrogate gradients for the backward pass. Drawing on the implementation ideas from NeuralSort [2] and differentiable top-K [3], we adopt hard selection in the forward pass and soft sorting in the backward pass.
Specifically, let the selection probabilities of the Router for coarse-grained visual location predictions be , where a larger indicates that location has a higher priority for expansion. In the forward pass, threshold filtering and top-K selection are performed according to these probabilities to retain locations. The selection result is represented by a matrix . If the -th selected location is , then , otherwise, .
In backpropagation, to implement a differentiable surrogate path, we draw on the idea of NeuralSort [2] and construct a soft sorting matrix . Each row of corresponds to a rank, and each column corresponds to a candidate location. denotes the continuous weight of assigning location to the -th rank. For example, the first row represents the soft assignment of each candidate location to the highest rank, and the second row represents its soft assignment to the second-highest rank. Unlike hard sorting, where each rank corresponds to exactly one location, the weights in each row of soft sorting sum to 1 and vary continuously with the predicted probabilities from the Router, which enables the provision of surrogate gradients for sorting decisions.
For threshold filtering, we employ smooth gating and construct a soft selection matrix with the same shape as : where denotes the Sigmoid function, is the selection threshold, and controls the smoothness of gating. We then construct it using the straight-through estimator: Among them, stands for stopping gradient propagation, is a set of fine-grained visual tokens corresponding to position , and is the -th group of tokens fed into the subsequent segmentation process.
During the forward pass, , so the model still obtains fine-grained features via hard selection. During the backward pass, gradients are estimated through the derivative of the Router probability with respect to the soft selection matrix , allowing the downstream task loss to participate in the optimization of the Router.
[1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv 2013.
[2] Stochastic Optimization of Sorting Networks via Continuous Relaxations. ICLR 2019.
[3] Differentiable Top-k Operator with Optimal Transport. NeurIPS 2020.
Rebuttal by Authors (3/3)
W3: Ablation of visual-only routing and query-guided routing
Thank you for your insightful comments! To address your concerns, we have conducted additional ablation experiments to validate the superiority of our query-guided routing.
We adopt VisionZip [1] to evaluate the effectiveness of the visual-only routing. This work scores each spatial location using the attention within the visual encoder without introducing representations related to text queries. We aggregate the scores into a coarse-grained grid over corresponding 2×2 regions, and then select up to 32 fine-grained tokens () for expansion.
In addition, to measure the upper bound of improvement achievable by routing, we conduct GT-guided (Oracle) experiments. We bypass the Router's prediction and directly select unfolding positions based on Router labels constructed from the ground-truth mask. To maintain the same budget of at most 32 fine-grained tokens (), when there are more than 8 positive GT label positions, we uniformly sample 8 positions without replacement, if there are fewer than 8, all positions are retained.
| Token selection | RefCOCO | RefCOCO+ | RefCOCOg | Avg. |
|---|---|---|---|---|
| Visual-only (VisionZip-based) | 80.6 | 78.2 | 79.4 | 79.4 |
| Query-guided (ours) | 85.0 | 81.3 | 82.8 | 83.0 |
| GT-guided (Oracle) | 85.8 | 82.0 | 83.5 | 83.8 |
The results are shown in the table above, and we can draw the following conclusions:
- Query-guided routing outperforms vision-only routing across all three datasets, with improvements of 4.4%, 3.1%, and 3.4% respectively. We attribute this to the fact that vision-only attention mainly reflects the overall information distribution of an image, yet visually salient regions are not necessarily the areas requiring fine-grained discrimination for the current query. For instance, for queries targeting different objects within the same image, the vision-only strategy does not adjust selected positions according to the referring expressions, and the limited expansion budget may be allocated to regions irrelevant to the target. In contrast, the query-guided routing can allocate fine-grained visual information conditioned on the current query. These results validate the effectiveness of query-guided routing.
- Compared with the GT-guided (Oracle) reference, our method has gaps of 0.8%, 0.7%, and 0.7% on the three datasets, respectively. This indicates that the learned Router can achieve segmentation performance close to that of the GT-guided reference, while there remains room for improvement in position selection.
We have included detailed discussions on vision-only routing and Oracle evaluation (please see Appendix B.2) in the revised manuscript.
[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models. CVPR 2025.
Please feel free to contact us if you have any other questions.
Rebuttal by Authors (1/3)
Thank you for your constructive suggestions! To validate whether our method can be generalized to multimodal backbones other than InternVL3, we adapted and trained the SELF1E-style segmentation baseline and VTU on Qwen2.5-VL-3B [1] and Gemma-3-4B [2] respectively. These two MLLMs adopt different visual encoding and spatial compression mechanisms:
The experimental results are shown above. We can observe that:
We have added relevant experiments and detailed discussions (please see
Appendix B.3) on generalization in the revised version.[1] Qwen2.5-VL Technical Report. arXiv 2025.
[2] Gemma 3 Technical Report. arXiv 2025.
[3] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.