acceptodds
Under review as a conference paper at ICLR 2027

SparseSAM: Structured Sparsification of Activations in Segment Anything Models

Abstract

The Segment Anything Model (SAM) achieves strong promptable segmentation, but its ViT-based image encoders dominate inference latency and memory footprint. Existing activation-compression methods, such as token merging, shorten the processed sequence but introduce non-trivial runtime overhead and can suffer severe quality degradation under high compression. Sparse-attention methods typically leave the MLP dense, limiting end-to-end speedup. We propose SparseSAM, a training-free structured-sparsification framework that jointly accelerates attention and MLP layers while preserving token identity. SparseSAM introduces Stripe-Sort Attention, which forms fixed spatial groups using Z-order, ranks those groups with an image-dependent saliency permutation computed once per image, and applies a fixed A-shaped sparse pattern compiled into the attention kernel. SparseSAM further introduces a Residual-Consistency MLP that routes only informative tokens through the MLP while propagating the remaining tokens through the residual pathway. On SAM-L , attention-only SparseSAM reaches latency speedup at 25% density with an absolute mAP decrease of 0.8 mAP points, while joint attention–MLP compression reaches latency speedup with an absolute decrease of 2.7 mAP points.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Efficient AI, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_P8vTOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work presents SparseSAM, a structured sparsification framework for accelerating Segment Anything Models (SAMs). It introduces two strategies, Stripe-Sort Attention and Residual-Consistency MLP, to sparsify the attention and MLP modules, respectively. The experimental results demonstrate that the proposed method achieves strong performance.

Critical strengths
  • The motivation is very clear and supported by extensive quantitative analysis. In particular, the three observations effectively motivate the separate acceleration strategies for the attention and MLP modules and explain the intuition behind their designs.
  • The paper is well written, with a clear and natural flow that makes the core ideas easy to grasp.
  • The comprehensive experimental results provide convincing evidence for the effectiveness of the proposed method.
Critical weaknesses

Overall, I have a positive impression of this work and did not identify any major weaknesses. I only have a few minor concerns:

  • In Table 2, each method is associated with two rows of results at different density levels. Adding horizontal separators between methods would improve readability.
  • During training, token saliency can be estimated using gradient information. How is token saliency determined or handled during inference, when such gradient information is unavailable?
Questions for authors

Could sparsification methods originally designed for general Vision Transformers, rather than specifically for SAM, also be effective in this setting, e.g., ExpediteViT [r1] and DToP [r2]? I am not requesting additional experiments, but a brief analysis or discussion would help clarify the relationship between SparseSAM and these approaches.

[r1] Liang, Weicong, et al. "Expediting large-scale vision transformer for dense prediction without fine-tuning." Advances in Neural Information Processing Systems 35 (2022): 35462-35477.

[r2] Tang, Quan, et al. "Dynamic token pruning in plain vision transformers for semantic segmentation." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

2 comments
AuthorsOpenReview ↗

How is saliency computed at inference without gradients, and Table 2 clarity

We thank the reviewer for the positive feedback and for finding the motivation clear and the experiments convincing. We address the remaining points below.

Saliency at inference. Thank you for this question. "Spatial gradient" in SparseSAM means the image-space gradient of the feature map, computed with a fixed 3×3 Sobel filter. It is not a back-propagated gradient of any loss. SparseSAM is entirely training-free: nothing is learned or estimated during training, and the same computation runs at inference.

Concretely, for each image we take the patch-embedding output X(0)∈R64×64×DX^{(0)} \in \mathbb{R}^{64\times64\times D}, apply the Sobel operator over the token grid, and use the gradient magnitude (averaged over channels) as each token's score. Tokens are grouped into 2×2 Morton groups, and each group is scored by the mean of its four tokens. The resulting ranking is then cached and reused by every encoder block to avoid any further latency overhead.

We will rename the term to "Sobel feature saliency" throughout the revision and state explicitly that no backpropagation is involved.

Table 2 readability. We already added horizontal rules between methods in Table 2 (and the other density tables) in the revision.

AuthorsOpenReview ↗

Relation to ExpediteViT and DToP

General ViT sparsification methods. Thank you for the suggestion. We already include several general-purpose ViT sparsification methods as baselines in the paper, and their results highlight why SAM requires a tailored design. For example, on MS-COCO with SAM-L (Table 2, lines 324-345):

MethodDesigned forDensitymAPSpeedup
Dense–100%49.51.00×
ToMe (token merging)General ViTs50%42.50.86×
SpargeAttn (sparse attention)General transformers25%46.11.23×
PISA (sparse attention)General transformers25%48.30.68×
SparseSAM, attention-onlySAM25%48.72.15×
SparseSAM, jointSAM25%46.82.95×

ToMe loses 7 mAP and is slower than the dense baseline because merging and unmerging operations across SAM's 4,096-token grid introduce more overhead than the speedup gain. SpargeAttn and PISA preserve more accuracy but deliver limited speedups due to selection overhead relative to SAM's small attention windows.

We expanded the related work discussion in the revised paper to cover ExpediteViT and DToP (lines 518-526), specifically:

  • ExpediteViT (Liang et al., NeurIPS 2022) is a training-free method for dense prediction that clusters high-resolution tokens into fewer centers in intermediate layers and subsequently reconstructs high-resolution features. Our Observation 1, showing that replacing tokens with their nearest kk-means centroids largely preserves SAM's accuracy, is consistent with its premise. However, unlike ExpediteViT, SparseSAM retains all token positions and sparsifies only attention connections and MLP execution, avoiding clustering and reconstruction errors and latency overheads, particularly around object boundaries where mask quality is critical.

  • DToP (Tang et al., ICCV 2023) targets semantic segmentation by using auxiliary heads trained at intermediate stages to finalize predictions for easy tokens early. This requires additional training and relies on per-class confidence. In contrast, SAM's image encoder operates before receiving any prompt and has no predefined class vocabulary, making such confidence-based early exits unsuitable. SparseSAM instead uses prompt-agnostic, training-free token ranking, which better fits promptable segmentation.

Reviewer_wwocOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a token pruning method for the Segment Anything Model (SAM). The pruning method manipulates two types of modules in the SAM: The Stripe-Sort Attention prunes the attention module and the Residual-Consistency MLP is for pruning the MLP module. In the Stripe-Sort Attention, tokens are grouped in Z-order, and the groups are ranked according to saliency. During pruning, retaining a prefix of rr complete stripes preserves rr representatives from every group, yielding uniform spatial coverage. For MLP pruning, the same ordering is used to prune MLP tokens. The authors conducted experiments on the MS-COCO and the HQ-44K datasets.

Critical strengths

The authors conducted experiments and observed that replacing the encoder’s tokens with their nearest clustering centroids retains a high percentage of the baselines mIoU. Motivated by this observation, the authors proposed the Stripe-Sort Attention method that is able to sort the groups and prune the tokens in each group. The remaining tokens yield uniform spatial coverage.

The authors observed that MLP's magnitudes are highly non-uniform. Therefore, the authors proposed the Residual-Consistency MLP to update only the background tokens and keep foreground tokens with no MLP update.

Critical weaknesses

The authors applied this method to SAM, which is a bit old. Does the whole method also work well for SAM3 or the architectures with Mixture-of-Experts (MoEs)? Also, the method is applied to DINO, which is old. Now we have DINOv3, why not test the method on the DINOv3 architecture?

What group size G is used in the experiments? I think G is an important parameter to apply Sobel kernels.

Questions for authors

It's unclear to me which attention heads does Eq. 1 apply to, and which layers does Eq. 1 apply to? Does Eq. 1 apply to some heads? If yes, how to combine the results from different attention heads? How to combine the ranking from different layers?

6 comments
AuthorsOpenReview ↗

DINO in the paper is the detector to produce prompt inputs, not the targeted backbone

We thank the reviewer for the feedback. We answer them directly below and have revised the paper accordingly.

DINO in our experiments. We apologize for the confusion. In our COCO experiments, "DINO" refers to the DINO object detector (Zhang et al., ICLR 2023), a DETR variant used solely to generate bounding-box prompts for SAM, alongside H-DETR and YOLOX. SparseSAM does not depend on which detector produces the prompts. Here, the detector runs as a separate preprocessing stage shared across all methods to provide box prompts. We have clarified this in our revised paper.

A DINOv3-based detector. To follow the reviewer's suggestion, we added DEIMv2-X, a recent real-time detector built on a DINOv3 (ViT-S+) backbone, as a fourth box-prompt source. It reaches 57.8 box AP on COCO val2017 in our pipeline, and we segment its boxes with SAM-HQ (COCO mask mAP):

SAM backboneDenseSparseSAM attention-only 25%SparseSAM joint 25%
SAM-B44.443.8 (−0.6)43.3 (−1.1)
SAM-L47.146.5 (−0.6)45.3 (−1.8)
SAM-H47.547.1 (−0.3)46.0 (−1.5)

With DINOv3-based boxes, SparseSAM loses at most 0.6 mAP under attention-only compression and at most 1.8 mAP under joint compression, no more than with the DINO, H-DETR, and YOLOX boxes in Table 8 (lines 1242-1293) in the appendix. Again, SparseSAM does not depend on which detector produces the prompts.

AuthorsOpenReview ↗

Does it work on SAM 3 or MoE architectures?

** Application to SAM 3.** We already tested SparseSAM on the SAM 3 encoder in the paper, alongside Segmenter ViT-L, on ADE20K semantic segmentation (Table 6, lines 462-471):

ModelDensityADE20K mIoUSpeedup
SAM 3100% (dense)70.361.00×
SAM 370%71.171.28×
SAM 350%67.581.43×
SAM 330%62.511.60×
Segmenter ViT-L100% (dense)50.811.00×
Segmenter ViT-L70%51.611.52×
Segmenter ViT-L50%52.141.72×
Segmenter ViT-L30%45.381.99×

The same method, with no retraining, gives 1.28× on SAM 3 with no accuracy loss and up to 1.99× on Segmenter. SAM 3 is more sensitive than SAM at aggressive densities.

Application to Mixture-of-Experts. Our Residual-Consistency routing performs token-level conditional computation, allowing each token to either pass through the MLP or retain its residual based on its saliency. In principle, this mechanism could be extended to models with MoE layers, provided they are trained using similar inductive biases as SAM-based models. Moreover, we are able to sparsify the attention layer, which is useful for MoE architectures too. Yet, while MoE architectures have been mostly explored in LLMs, they remain uncommon in promptable segmentation models like SAM, SAM2, SAM3, and Segmenter. Evaluating MoE integration is therefore beyond the scope of this work, and we leave this direction for future investigation rather than making claims about its effectiveness without empirical validation.

AuthorsOpenReview ↗

Which group size G, and how does it relate to the Sobel kernel?

Group size GG. We use G=4G=4 in all experiments, grouping tokens into non-overlapping 2×22\times2 blocks along the Morton (Z-order) curve. This is the smallest square grouping on the Morton curve. Importantly, GG and the Sobel kernel size are independent design choices. We first compute the token saliency using Sobel independently for each token on the token grid (Eq. 1), then assign each group the mean saliency of its GG constituent tokens (Section 3.1). Thus, GG controls only the granularity of token ranking and interleaving; it does not affect the Sobel operator's receptive field. We clarified this choice in Section 3.1 of the revised paper (lines 160-161).

Group-size ablation. We vary GG on SAM-L with joint compression and measure average mIoU/Boundary IoU over the four HQ-44K datasets all 1,537 images). SAM's windowed blocks use 14×1414\times14 windows of 196 tokens, which can only use G∈{1,2,4}G \in \{1, 2, 4\}, so G≥16G \geq 16 is applied to the global-attention blocks only, while the windowed blocks keep their max G=4G = 4. Each cell in the table below is mIoU / Boundary IoU:

GGGroup shape25%30%50%75%
4 (ours)2×22\times286.59 / 78.2787.02 / 78.8988.39 / 80.8388.78 / 81.38
16 (global blocks)4×44\times484.14 / 75.6186.75 / 76.4787.19 / 80.4788.73 / 81.31
64 (global blocks)8×88\times881.68 / 72.5183.40 / 73.6586.51 / 75.5087.55 / 77.99

As expected, G=4G = 4 is the most accurate at every density, in both mIoU and Boundary IoU. The gap grows as the density drops, while non-square pairs (G=2G = 2) are also consistently worse.

Why We Need a Small GG. Stripe-Sort interleaves tokens across groups in a round-robin fashion (i.e the first token from each group is selected, followed by the second token from each group, and so on). With larger groups, however, the selected tokens cluster in the earliest positions in Morton order, which correspond to a small localized spatial region of the image. Similarly, the MLP updates only the top-ranked groups, potentially leaving large spatial regions without any MLP updates. In contrast, G=4G=4 provides finer-grained spatial coverage and gives the MLP greater flexibility to capture complex object structures. This combination is crucial for preserving accuracy at low token densities. We added this ablation study to Appendix A10 in the revised paper.

Reviewer_kwjHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SparseSAM, a training-free structured computation sparsification method for prompt-based image segmentation with the Segment Anything Model (SAM), which predicts pixel-level object masks from user-provided prompts such as points and bounding boxes. The authors identify substantial feature redundancy in SAM’s image encoder and considerable variation in MLP update magnitudes across spatial tokens. Sparsifying attention alone limits acceleration, while the overhead of dynamic selection or token merging and restoration can offset computational savings. SparseSAM therefore ranks spatial token groups using spatial gradients and reuses the ranking across layers. It reduces attention connections through a fixed sparse pattern and applies MLP updates only to high-saliency tokens, allowing the remaining tokens to pass through the residual pathway. Fused GPU kernels further reduce execution overhead. Experiments on COCO and HQ-44K demonstrate favorable speed-accuracy trade-offs. On COCO with SAM-L at 25% density, attention-only compression achieves a 2.15x speedup with a 0.8-point mAP drop, while joint compression achieves a 2.95x speedup with a 2.7-point mAP drop.

Critical strengths
  1. This paper addresses computational bottlenecks in both attention and MLP layers while accounting for the overhead of dynamic selection and reordering, making the design directly relevant to practical inference efficiency.

2.SparseSAM reduces computation through sparse attention connections and selective MLP execution while preserving all tokens and their spatial correspondence, helping retain the regional and boundary information needed for dense prediction.

3.Stripe interleaving balances spatial coverage with prioritization of salient regions, while the fixed sparse pattern supports efficient execution. Fused kernels further reduce reordering, restoration, and routing overhead, translating computational savings into practical speedups.

Critical weaknesses

1.SparseSAM relies on the initial Sobel ranking to continue identifying important tokens in subsequent layers, although spatial gradients do not directly correspond to semantic importance or MLP update magnitude. The appendix compares cached and refreshed rankings, but also shows that refreshing substantially reduces error in some deeper layers. Additional statistics across images and layers, together with an evaluation of how refresh frequency affects final accuracy and latency, would strengthen this justification.

2.Appendix Table 8 shows that optimizing the dense implementation alone yields a 1.51x speedup, while joint sparsification provides an additional 1.83x speedup over that implementation. Consequently, the speedups relative to the original implementation in the main tables do not isolate the benefits of the sparsification strategy. The optimized dense implementation should be included in the main comparisons, alongside a comparison between simple uniform selection and the proposed method using the same kernel.

3.The paper reports attention-only and joint compression results but lacks a systematic evaluation of MLP-only compression and experiments that independently vary attention and MLP densities. Additional component ablations and accuracy–latency curves would clarify the benefits of joint compression and support a more informed allocation of computational budgets.

4.The evaluation primarily uses mAP and mIoU, with limited analysis of boundaries, small objects, and thin structures. It also lacks failure cases involving low-contrast objects or highly textured backgrounds, where the saliency proxy may be unreliable. Furthermore, speedups for some detector configurations are estimated, and the timing excludes detection, preprocessing, and postprocessing. Measured and estimated results should be distinguished more clearly, and the scope of end-to-end acceleration should be explicitly defined.

Questions for authors
  1. Please compare the optimized dense implementation, uniform spatial selection, and the proposed method in terms of accuracy and latency using the same fused kernel framework, to separately assess the contributions of kernel optimization and sparsification.

  2. It is suggested to investigate how well the initial saliency ranking remains effective across network layers, and report the effects of periodically refreshing the ranking on final segmentation accuracy and inference latency, focusing on the deeper layers where the appendix shows substantial error reductions after refreshing.

  3. Please provide experiments with MLP-only compression and independently varied attention and MLP densities to clarify the individual contributions of the two modules and assess the appropriateness of computational budget allocation under joint compression.

5 comments
AuthorsOpenReview ↗

Separate kernel gains from sparsification gains and compare with uniform selection.

We thank the reviewer for the detailed and constructive review and for noting that SparseSAM accounts for selection and reordering overhead to give practical speedups. We have run all the requested experiments, the results are below and are also included in the revised paper.

Kernel optimization vs. sparsification (W2, Q1). We agree that the benefits of kernel optimization and sparsification should be evaluated separately. To this end, we added the fused-dense implementation as a baseline in the revised Table 2 (Lines 324–345) and moved speedup attribution to the main paper (Table 5, lines 426-431). Specifically, here is the speedup attribution of SAM-L on the NVIDIA L4 GPU at 30% density :

ConfigurationLatency (ms)Peak memory (MB)vs. originalvs. fused dense
Original dense191.512221.51.00×0.66×
Fused dense (our kernels, 100% density)127.05765.31.51×1.00×
sparse attention + dense MLP96.79765.31.98×1.31×
dense attention + sparse MLP91.252109.62.09x1.40x
sparse attention + sparse MLP (SparseSAM)69.45765.32.76×1.83×

Notably, sparse attention methods such as SpargeAttention and PISA also use the same fused-dense kernel as the implementation baseline, yet their runtime overhead from sparsity selection and indexing can offset the computational savings, resulting in limited speedups or even slower execution than the unfused baseline. In contrast, SparseSAM effectively reduces computation while keeping runtime overhead low, delivering actual speedups beyond those from kernel fusion alone. Here are sparse-attention baselines at 25% density using SAM-L on MS-COCO with DINO boxes (Table 2, Lines 324–345):

MethodmAPSpeedup vs. original
Dense49.51.00×
Fused Dense49.51.51×
SpargeAttn46.11.23×
PISA48.30.68×
SparseSAM, attention-only48.72.15×
SparseSAM, joint46.82.95×

Uniform vs. Saliency Ranking. We added an ablation study in the revised paper (Figure 9) comparing saliency-based ranking against uniform selection of Morton groups, while keeping the fused kernel and sparsity density fixed for a fair comparison. Both strategies use the same kernel, isolating the impact of the ranking strategy on accuracy.

Joint compression, SAM-L, same kernel and density:

DensityRankingCOCO mAP (DINO boxes)HQ-44K mIoU
100%Dense49.588.91
30%Uniform45.885.63
30%Saliency (SparseSAM)47.8 (+2.0)87.02 (+1.39)
25%Uniform45.485.05
25%Saliency (SparseSAM)46.8 (+1.4)86.59 (+1.54)

The results demonstrate that saliency-based ranking is key to preserving accuracy at the same sparsity levels, outperforming uniform selection by 2.0 mAP on COCO and 1.39 mIoU on HQ-44K (Figure 9, lines 462–491).

AuthorsOpenReview ↗

Does the first-layer Sobel ranking stay valid in deeper layers?

Reusing the initial ranking across layers (W1, Q2). We use the initial ranking as a lightweight proxy for where MLP updates concentrate, rather than as a measure of semantic importance. As shown in Figure 6(b) (Lines 216–236), dense SAM concentrates MLP updates on a small set of representative, highly dissimilar tokens, motivating the use of Sobel responses as an inexpensive proxy for token ranking. After routing, the inter-token similarity structure remains close to that of the dense baseline. Moreover, final segmentation quality also depends on how the decoder leverages inter-token regional dissimilarities to generate segmentation masks.

To further evaluate the effectiveness of reusing the initial ranking, we compare different ranking refresh schedules on SAM-L with HQ-44K at 25% density. Speedup is measured end-to-end relative to the same fused-kernel implementation at 100% density.

Refresh scheduleBlocks refreshedHQ-44K mIoUΔ vs. cachedEnd-to-end speedup
Never (default)086.59–1.83×
Each global block386.72+0.131.72×
Every 4 blocks586.87+0.281.41×
Last stage only786.61+0.021.35×
Every 2 blocks1186.93+0.341.21×
Every block2386.94+0.360.99×

Refreshing the ranking at every block yields only a marginal improvement of 0.36 mIoU, while eliminating the end-to-end speedup. These results demonstrate diminishing returns from more frequent ranking updates at a substantial computational cost. Reusing a single cached ranking per image provides the best accuracy–latency trade-off. Further details are provided in Appendix A.4 (Table 7, Lines 1079–1111).

AuthorsOpenReview ↗

MLP-only compression and independent densities

MLP-only compression and independent densities (W3, Q3). We added an ablation study in the appendix that independently varies attention density (dad_a) and MLP density (dmd_m), using a shared token ranking on SAM-L with HQ-44K. We report mean mIoU and end-to-end speedup relative to the original implementation. Here is the result table, with each column representing (da,dmd_a,d_m):

MetricOriginalFused dense(25%, 25%)(25%, 50%)(25%, 75%)(25%, 100%)(50%, 25%)(50%, 50%)(50%, 75%)(50%, 100%)(75%, 25%)(75%, 50%)(75%, 75%)(75%, 100%)(100%, 25%)(100%, 50%)(100%, 75%)
Speedup (×)1.001.512.842.782.421.992.592.532.391.812.312.171.731.622.111.811.57
mIoU (%)88.9188.9186.5987.5087.8988.0287.5788.3988.6588.7687.7488.5588.7888.8887.8088.5888.81

The results reveal several key findings:

  • Both modules contribute comparable speedups. At 25% density, attention-only and MLP-only compression achieve 2.35× and 2.40× speedups, respectively, while combining both reaches 2.94×, demonstrating that their computational savings are complementary.
  • MLP compression is slightly more sensitive to sparsification. When compressed independently down to 25% density, attention and MLP compression incur mIoU drops of 0.89 and 1.11 points, respectively.
  • Balanced density allocation better preserves accuracy. At the same total density, the balanced setting (50%, 50%) achieves 88.39 mIoU, outperforming (25%, 75%) and (75%, 25%), which achieve 87.89 and 87.74, respectively. This supports our use of equal attention and MLP densities in the main experiments.

The complete density sweep is included in the revised Appendix A9 (lines 1343-1359).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their constructive feedback. Reviewers found the motivation clear and well supported (Reviewer P8vT), the design relevant to practical inference efficiency (Reviewer kwjH), and the paper well written (Reviewers P8vT, kwjH). We summarise the main clarifications and additions here, and more details are in the individual responses. All changes in the revised paper are marked in blue, and line numbers below refer to the revised PDF.

Clarifications

  1. Saliency is a spatial (Sobel) gradient of the features, not a back-propagated gradient (Reviewers P8vT, kwjH, wwoc). It is computed once per image during the forward pass, scored over 2×2 Morton groups, and reused by all later blocks and heads at negligible cost (Section 3.1, Lines 158–185).
  2. "DINO" is the box-prompt detector, not the DINO backbone; SparseSAM is applied to SAM only, and the detector is a separate stage shared by all methods (Reviewer wwoc).
  3. End-to-end latency scope. End-to-end latency is measured on the SAM model after the box prompts are obtained from the detector, and all reported speedups are measured (Section 4, Line 305; Appendix A.11, Lines 1386–1418) (Reviewer kwjH).

New results in the revision

  1. Kernel optimisation vs. sparsification (Table 2, Lines 324–345; Section 4.2 and Table 5, Lines 426–443): the fused kernels and sparsification contribute separate, compounding speedups (Reviewer kwjH).
  2. Saliency vs. uniform selection in the same kernel (Section 4.2 and Fig. 9, Lines 460–485): at the same kernel and density, saliency ranking is what preserves accuracy (Reviewer kwjH).
  3. Ranking refresh and ranking quality across layers (Appendix A.4 and Table 7, Lines 1079–1110): a single cached ranking per image gives the best accuracy–latency trade-off (Reviewer kwjH).
  4. Independent attention and MLP densities, including MLP-only (Appendix A.9 and Table 10, Lines 1342–1362; summary in Section 4.2, Line 471): both modules contribute to the speedup, and the balanced allocation used in the paper is the most accurate (Reviewer kwjH).
  5. Group-size ablation (Appendix A.10 and Table 11, Lines 1363–1385): G=4G = 4 is the best choice at every density (Reviewer wwoc).
  6. Boundary IoU on HQ-44K (Appendix A.8 and Table 9, Lines 1296–1340): SparseSAM keeps its advantage over the baselines on boundary quality (Reviewer kwjH).
  7. Hard-case analysis and failure cases (Appendix A.16, Table 14 and Fig. 15, Lines 1622–1670): low object–background contrast is the main failure mode (Reviewer kwjH).
  8. Stage-wise latency breakdown (Appendix A.11 and Table 12, Lines 1402–1418): the image encoder dominates the pipeline, so its speedup carries over end-to-end (Reviewer kwjH).

Writing

A clearer method description of heads, layers and GG (Section 3.1, Lines 158–185), a related-work discussion of general ViT acceleration including ExpediteViT and DToP (Section 5, Lines 516–517) (Reviewer P8vT), and table formatting.

All of the new experiments and updates for clarification and writing are included in the revised paper as blue text. If you have any other concerns, please let us know, and we are happy to discuss and include more results in the remaining time of the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.