- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work presents SparseSAM, a structured sparsification framework for accelerating Segment Anything Models (SAMs). It introduces two strategies, Stripe-Sort Attention and Residual-Consistency MLP, to sparsify the attention and MLP modules, respectively. The experimental results demonstrate that the proposed method achieves strong performance.
Critical strengths
- The motivation is very clear and supported by extensive quantitative analysis. In particular, the three observations effectively motivate the separate acceleration strategies for the attention and MLP modules and explain the intuition behind their designs.
- The paper is well written, with a clear and natural flow that makes the core ideas easy to grasp.
- The comprehensive experimental results provide convincing evidence for the effectiveness of the proposed method.
Critical weaknesses
Overall, I have a positive impression of this work and did not identify any major weaknesses. I only have a few minor concerns:
- In Table 2, each method is associated with two rows of results at different density levels. Adding horizontal separators between methods would improve readability.
- During training, token saliency can be estimated using gradient information. How is token saliency determined or handled during inference, when such gradient information is unavailable?
Questions for authors
Could sparsification methods originally designed for general Vision Transformers, rather than specifically for SAM, also be effective in this setting, e.g., ExpediteViT [r1] and DToP [r2]? I am not requesting additional experiments, but a brief analysis or discussion would help clarify the relationship between SparseSAM and these approaches.
[r1] Liang, Weicong, et al. "Expediting large-scale vision transformer for dense prediction without fine-tuning." Advances in Neural Information Processing Systems 35 (2022): 35462-35477.
[r2] Tang, Quan, et al. "Dynamic token pruning in plain vision transformers for semantic segmentation." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.
2 comments
Relation to ExpediteViT and DToP
General ViT sparsification methods. Thank you for the suggestion. We already include several general-purpose ViT sparsification methods as baselines in the paper, and their results highlight why SAM requires a tailored design. For example, on MS-COCO with SAM-L (Table 2, lines 324-345):
| Method | Designed for | Density | mAP | Speedup |
|---|---|---|---|---|
| Dense | – | 100% | 49.5 | 1.00× |
| ToMe (token merging) | General ViTs | 50% | 42.5 | 0.86× |
| SpargeAttn (sparse attention) | General transformers | 25% | 46.1 | 1.23× |
| PISA (sparse attention) | General transformers | 25% | 48.3 | 0.68× |
| SparseSAM, attention-only | SAM | 25% | 48.7 | 2.15× |
| SparseSAM, joint | SAM | 25% | 46.8 | 2.95× |
ToMe loses 7 mAP and is slower than the dense baseline because merging and unmerging operations across SAM's 4,096-token grid introduce more overhead than the speedup gain. SpargeAttn and PISA preserve more accuracy but deliver limited speedups due to selection overhead relative to SAM's small attention windows.
We expanded the related work discussion in the revised paper to cover ExpediteViT and DToP (lines 518-526), specifically:
-
ExpediteViT (Liang et al., NeurIPS 2022) is a training-free method for dense prediction that clusters high-resolution tokens into fewer centers in intermediate layers and subsequently reconstructs high-resolution features. Our Observation 1, showing that replacing tokens with their nearest -means centroids largely preserves SAM's accuracy, is consistent with its premise. However, unlike ExpediteViT, SparseSAM retains all token positions and sparsifies only attention connections and MLP execution, avoiding clustering and reconstruction errors and latency overheads, particularly around object boundaries where mask quality is critical.
-
DToP (Tang et al., ICCV 2023) targets semantic segmentation by using auxiliary heads trained at intermediate stages to finalize predictions for easy tokens early. This requires additional training and relies on per-class confidence. In contrast, SAM's image encoder operates before receiving any prompt and has no predefined class vocabulary, making such confidence-based early exits unsuitable. SparseSAM instead uses prompt-agnostic, training-free token ranking, which better fits promptable segmentation.
How is saliency computed at inference without gradients, and Table 2 clarity
We thank the reviewer for the positive feedback and for finding the motivation clear and the experiments convincing. We address the remaining points below.
Saliency at inference. Thank you for this question. "Spatial gradient" in SparseSAM means the image-space gradient of the feature map, computed with a fixed 3×3 Sobel filter. It is not a back-propagated gradient of any loss. SparseSAM is entirely training-free: nothing is learned or estimated during training, and the same computation runs at inference.
Concretely, for each image we take the patch-embedding output X(0)∈R64×64×D, apply the Sobel operator over the token grid, and use the gradient magnitude (averaged over channels) as each token's score. Tokens are grouped into 2×2 Morton groups, and each group is scored by the mean of its four tokens. The resulting ranking is then cached and reused by every encoder block to avoid any further latency overhead.
We will rename the term to "Sobel feature saliency" throughout the revision and state explicitly that no backpropagation is involved.
Table 2 readability. We already added horizontal rules between methods in Table 2 (and the other density tables) in the revision.