acceptodds
Under review as a conference paper at ICLR 2027

GrabMe: Accelerating ViTs for Segmentation through Graph-based Token Merging

Abstract

Segmentation is a widely studied computer vision task in which Vision Transformers (ViTs) have demonstrated notable performance. Nevertheless, the self-attention mechanism in ViTs incurs quadratic computational complexity with respect to the number of tokens, which has motivated the development of a range of token-merging strategies. Still, existing merging methods based on global matching or rigid local grouping may perturb critical spatial relationships and degrade the preservation of fine-grained details. Furthermore, reliance solely on pairwise similarity metrics may be inadequate for preserving object boundaries and maintaining the integrity of small objects during the merging process. In this work, we introduce Graph-based Token Merging (GrabMe), a training-free token-merging method for efficient segmentation. To more faithfully preserve relative spatial structure, GrabMe constrains merging to one-hop neighbors on a spatial adjacency graph, where resulting token groups inherit adjacencies of their constituent tokens. For parallel merging, GrabMe employs dynamic programming to construct a conflict-free pool of candidates with maximum cardinality. Subsequently, complementary structural context derived from one-hop similarity distributions is incorporated into the merge decision, thereby mitigating the loss of fine-grained details. Extensive experiments with ViT backbones across semantic, instance, and panoptic segmentation benchmarks demonstrate that GrabMe effectively improves the accuracy–efficiency trade-off of pretrained backbones for segmentation without requiring any additional training. Our code is available at this https://anonymous.4open.science/r/GrabMe-7889/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
token merging, vision transformers, image segmentation, efficient inference, training-free acceleration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_sYs9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

To address the inference bottleneck in Vision Transformers (ViTs) caused by quadratic self-attention complexity, as well as the spatial distortion and loss of small objects in existing merging methods, the paper proposes a training-free Graph-based Token Merging framework (GrabMe). It preserves spatial structure via one-hop constraints on a spatial adjacency graph and constructs a conflict-free, maximum-cardinality candidate pool using dynamic programming for O(N) parallel merging. Additionally, a similarity-based context score regulates merge decisions to protect fine-grained details. It substantially reduces tokens and accelerates inference while maintaining mIoU performance across semantic, instance, and panoptic segmentation benchmarks.

Critical strengths
  1. The paper is well-motivated and clearly presented. It introduces one-hop spatial graph constraints, a dynamic-programming-based conflict-free candidate pool, and complementary structural context to improve boundary distortion and performance degradation for small objects.
  2. The paper conducts thorough experiments across diverse semantic, instance, and panoptic segmentation benchmarks, demonstrating its effectiveness.
Critical weaknesses
  1. Although the motivation is reasonable, GrabMe does not elaborate on its essential differences from existing local/graph-structure-based merging methods (such as G2TM, DHTM, etc.). The authors need to better illustrate its differences at the level of topological modeling or merging decisions, as well as provide solid corresponding quantitative comparisons.
Questions for authors
  1. Although the motivation is reasonable, GrabMe does not elaborate on its essential differences from existing local/graph-structure-based merging methods (such as G2TM, DHTM, etc.). The authors need to better illustrate its differences at the level of topological modeling or merging decisions, as well as provide solid corresponding quantitative comparisons.
  2. While Table 4a demonstrates that combining pairwise similarity and context scores outperforms either component alone, it also indicates that the fusion ratio is performance-critical;nevertheless, no sensitivity analysis over the fusion coefficients (½, ½) is reported
  3. The concluding claim "Taken together, these results demonstrate that GrabMe substantially accelerates inference while maintaining superior performance across a diverse set of segmentation tasks" is not supported by the reported evidence. For semantic segmentation, the performance is on par with baseline (as shown in Table 1), while on instance and panoramic segmentation tasks, the performance exhibits a drop (as shown in Table 3).
Reviewer_PAGeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents GrabMe, a training-free token merging method for Vision Transformers (ViTs) for image segmentation. GrabMe merges tokens locally in a progressive manner, meaning that each layer further merges tokens to reduce the number of tokens processed by the ViT. To determine which tokens are merged, it considers direct neighbors of (merged) tokens in each layer. Then, it merges tokens if (i) their embeddings have a high similarity to ensure that merged tokens have similar content, and (ii) the mean similarity between the tokens and all their neighbors is high and the standard deviation of these similarities is low, to only merge tokens if they are in a semantically homogeneous region and do not belong to boundaries or small objects. With this design, GrabMe can locally merge tokens in a way that adapts to the shape of objects, while largely avoiding the merging of tokens that should not be merged (e.g., belonging to different classes or lying on object boundaries), which is something that existing models could not do. With experiments across multiple datasets and segmentation tasks, GrabMe is shown to yield a more favorable accuracy-speed trade-off than existing token methods, showing its effectiveness.

Critical strengths
  1. The method presented in the paper, GrabMe, is well-designed and well-motivated. It addresses multiple limitations of existing works with a single, elegant method.

  2. By conducting local merging progressively and allowing (previously merged or non-merged) tokens to be merged with any neighboring token in each layer, GrabMe can merge tokens in arbitrary patterns, adapting to irregular shapes of objects. This is something many existing local token merging methods (e.g., ALGM, LoTM, Expedite) cannot do, because they restrict token merging to fixed local windows or specific patterns, or only apply it in a single layer. This is also clearly visible in Fig. 5.

  3. By selecting disjoint sets of tokens as candidates for merging using dynamic programming, GrabMe ensures that all tokens can be merged in parallel for each layer, which is more efficient than the sequential merging of token pairs used by some existing methods (e.g., DHTM).

  4. GrabMe does not only use the pairwise similarity between tokens to determine if they can be merged, but also a ‘context score’. This score takes into account the mean and standard deviation of the similarities between a token and all of its direct neighbors. A token gets a high score if the mean token similarity is high and the standard deviation of this similarity is low, as this indicates that the token likely lies in a semantically homogenous region. By also taking this score into account to determine merging, GrabMe discourages the merging of tokens that lie on object boundaries or belong to small objects, which would result in the loss of information and detail that's necessary for segmentation.

  5. The method’s effectiveness is extensively evaluated and demonstrated across multiple datasets (ADE20K, Cityscapes, COCO), segmentation tasks (semantic, panoptic, instance), and ViT-based segmentation models (Segmenter, EoMT, EVA + ViT-Adapter). Across all of these experiments, GrabMe achieves a more favorable accuracy-speed trade-off than existing, state-of-the-art token merging methods. This demonstrates the general effectiveness of the proposed method. With its strong performance, this method can be useful for both practitioners and researchers, providing a strong foundation for follow-up research.

Critical weaknesses

The paper does not have any critical weaknesses that are cause to reject this paper. There are, however, several smaller points for improvement. I’ve listed these in the ‘Questions for Authors’ section. Considering both the strengths and these smaller points for improvement, I initially give a ‘Weak Accept’ rating. If the authors address the points for improvement, I’m willing to upgrade my rating to ‘Clear Accept’.

Questions for authors

There are several, smaller points for improvement, which could further strengthen the paper. If the authors address these points and strengthen the paper, I am willing to upgrade my rating.

  1. It is not fully clear whether GrabMe only merges pairs of two tokens in a single layer, or whether it merges sets of any arbitrary number of tokens. Sec. 3 repeatedly talks about candidate ‘pairs’ (e.g., L149, L150, L219), which suggests that only pairs of two tokens are merged (where a token can of course consist of multiple constituent tokens that were merged into it in earlier layers). However, L153 mentions that Av\mathcal{A}_v consists of the token vv itself and the ‘tokens’ (plural) that select vv as their target, and L197 says that the candidate set of mergeable tokens is constructed ‘while allowing multiple tokens to share the same target’, suggesting that more than two tokens can be merged into one, in a single layer. This is inconsistent. The paper should clarify which of these settings is correct.
  2. While the paper demonstrates that GrabMe obtains a more favorable accuracy-speed trade-off than existing methods at fixed mIoU values or fixed merging rates, the paper does not provide a full picture of this accuracy-speed trade-off. The paper could be strengthened by including a plot with the speed and accuracy of GrabMe and other token merging methods for different reduction rates, as this would give insights into the relative performance of GrabMe across a wide range of settings, and it could highlight the settings in which it is most effective. Fig. 6 in the appendix contains these results partly, but it does not report the model speed/throughput.
  3. Currently, while the paper does report the overall model throughput, the paper does not provide insight into the efficiency of the merging algorithm itself. The latency of the merging algorithm would be particularly interesting as the dynamic programming algorithm used to obtain the candidate pairs is a new component that potentially introduces a considerable latency overhead. The paper could be strengthened by reporting this latency.
  4. The paper does not mention whether the implementation uses FlashAttention [a] or not. FlashAttention significantly reduces the burden of quadratic complexity and speeds up transformers. It would be relevant to know whether the current efficiency improvements are made in combination with FlashAttention, or whether standard attention implementations are used. If the implementation uses FlashAttention, GrabMe’s efficiency improvements would be much more remarkable and valuable.
  5. It is not clear how the ‘spatial displacement’ is computed for Figure 3. Is it the Euclidean distance between each token's original position and the center of the token group that it is merged into, or is it measured differently?
  6. It is not clear whether the reported reduction rate in the tables (e.g., Tab. 1 & 2) represents the reduction in tokens in the final layer, or the average reduction across the model. Using the reduction rate at the final layer could be misleading, as some methods (like ALGM) already produce and use the final reduced set of tokens early in the model, allowing more ViT layers to be sped up by feeding them fewer tokens, while in GrabMe the reduction rate (and efficiency improvement) in different layers varies, and is lower in earlier layers. The paper should report which reduction rate it uses, and motivate this choice.

[a] Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022.


Note after rebuttal: my concerns have been adequately addressed. Therefore, I upgrade my rating to 'clear accept'.

Reviewer_11LEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

GrabMe introduces a training-free, graph-based token-merging method for accelerating ViT-based segmentation. It combines merging between spatially adjacent token groups with a context score derived from one-hop feature-similarity distributions. Dynamic programming constructs a maximum-cardinality, conflict-free candidate pool to support parallel merging. The reported experiments demonstrate high token reduction rates while preserving backbone performance in semantic segmentation, as well as smaller accuracy drops than the evaluated merging baselines under aggressive token reduction in instance and panoptic segmentation.

Critical strengths
  • The motivation for incorporating neighborhood similarity distributions beyond pairwise similarity is clear and well supported.
  • The proposed candidate-selection strategy enables efficient parallel merging and achieves higher throughput than greedy selection in Table 4b.
  • The quantitative and qualitative results support the claimed improvements in the accuracy–efficiency trade-off across multiple segmentation tasks and backbones.
Critical weaknesses
  • The conceptual novelty of the context score could be better clarified in relation to prior methods that infer boundaries from spatial variation in learned representations, such as ABDiv in TRACE [1].
  • The explanation for the accuracy advantage of DP-based candidate selection over greedy selection remains incomplete. Appendix B.4 reports larger candidate pools and higher semantic consistency, but does not clearly disentangle the contributions of candidate-pool size, scoring criteria, and greedy selection order.

[1] Jo et al., "TRACE: Your Diffusion Model Is Secretly an Instance Edge Detector," ICLR 2026.

Questions for authors
  • TRACE [1] uses divergence between self-attention distributions at opposite spatial neighbors to identify instance boundaries at a selected diffusion timestep. Although GrabMe uses a different representation and formulation, both approaches exploit local variation to identify boundary-sensitive regions. Could the authors clarify the conceptual contribution of the proposed context score relative to this prior work?
  • In Table 4b, greedy selection yields lower mIoU at the same token reduction rate and permits less token reduction at matched mIoU. Appendix B.4 attributes the advantage of DP-based selection to larger candidate pools and higher semantic consistency. Could the authors explain more concretely how candidate-pool size and greedy selection order contribute to these differences? Please also clarify whether all selection strategies use the same final pairwise-and-context scoring rule.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.