acceptodds
Withdrawn from ICLR 2027

Dense Checkpoints, Sparse Execution: Retraining-Free Acceleration of Query-Based Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Query-based segmentation repeatedly materializes dense query-pixel tensors during decoding and output processing, generating substantial memory traffic at high image resolutions. We use a frozen checkpoint's predicted masks and logits as an input-dependent execution plan to reduce this traffic. Mask-guided sparse cross-attention with a fixed budget of 128 feature cells per query preserves dense-level panoptic, instance, and semantic accuracy across compatible checkpoints on Cityscapes, COCO, and ADE20K, without retraining or checkpoint-specific tuning. Coarse intermediate mask prediction at the next attention stride avoids temporary stride-4 masks. Task-sufficient output processing evaluates masks only in regions that can affect the final output. For panoptic segmentation, we find that masks can influence pixel assignments even where they predict background, and derive conditions for safely skipping computation. Given identical FP32 logits from a released Mask2Former R50 checkpoint, output processing exactly reproduces the original panoptic maps, instance masks, and labels on all 500 Cityscapes validation images. On Mask2Former at native Cityscapes resolution, these transformations reduce FP32 DRAM traffic by 67% in the transformer decoder and over 80% in output processing. On an AGX Orin, output processing achieves a panoptic speedup and a instance speedup over optimized dense processing. With a PPM-FPN pixel decoder under mixed precision, complete-model inference achieves speedups of up to for panoptic and for instance segmentation over an optimized dense reference. Dense segmentation checkpoints can therefore be deployed more efficiently without retraining by targeting the inference paths that dominate memory traffic.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Query-based segmentation, efficient inference, sparse attention, retraining-free acceleration, high-resolution segmentation, edge deployment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZxE5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper speeds up inference of frozen, dense-trained query-based segmentation models (mainly Mask2Former and relatives) without retraining, by using the model's own intermediate masks and logits as an execution plan. There are three transformations:

  1. Mask-guided sparse cross-attention. At strides 16 and 8, each query attends to at most M=128M=128 feature cells inside its predicted mask, chosen in a fixed hash order. Stride 32 stays dense.
  2. Coarse intermediate mask prediction. This uses Rs(EF4)=E Rs(F4)R_s(EF_4)=E\,R_s(F_4) to predict intermediate masks directly at the next attention stride, rather than at stride 4 followed by downsampling.
  3. Task-sufficient output processing. Instance masks are rendered only over tiles with positive support. For panoptic, conservative per-region score bounds skip queries that provably cannot win any pixel, and mask areas are counted separately.

Results on Cityscapes, COCO and ADE20K show accuracy within about ±0.4 points of dense. Output processing exactly reproduces the original panoptic and instance outputs on Cityscapes val. On an AGX Orin, the paper reports 1.8× / 4.2× faster output processing and 1.24–1.59× complete-model speedups over an optimized dense reference.

Critical strengths

1. The output-processing part is careful and convincing. The observation that a query can still win panoptic pixels where its own mask predicts background is a nice catch, and the authors handle it properly: they use conservative per-region score bounds, keep possible ties, and count mask areas independently of the winners. Exact agreement with the original panoptic maps and instance masks on all 500 Cityscapes val images is a strong sanity check, and the 1.8× / 4.2× output-stage speedups over an already optimized dense path are real gains.

2. The paper is honest about scope and generalizes reasonably well. The same M=128M=128 setting is used unchanged across several checkpoints and three datasets, with accuracy within about ±0.4 points. DRAM traffic is measured with hardware counters rather than estimated. The limitations are stated openly (small gains at low resolution, the FastInst conversion losing 1.2 AP, no TensorRT), which I appreciated.

Critical weaknesses

(1) The methodological contribution feels thin for ICLR. Taking the three parts one by one:

  • When a mask permits at most MM cells, the sparse attention computes the same thing as Mask2Former's existing masked attention, just implemented with a gather. The new part is the hash-uniform subsampling for larger masks, which is a reasonable heuristic but not much more.
  • Coarse mask prediction is the linearity of bilinear resizing, i.e., an identity in real arithmetic.
  • The output processing is a well-engineered rewrite of post-processing.

All of this is useful engineering, but I don't see much that teaches us something new about learning or representations. It reads more like a strong systems or deployment paper for the Mask2Former family.

The sparse-attention gains are also narrower than the framing suggests. Under fused SDPA (Table 10), sparse attention at stride 16 is actually slower than dense (9.54 vs 8.22 ms), and making stride 16 sparse changes total decoder latency by only about 1% (54.57 vs 55.10 ms). So the decoder speedup essentially comes from stride 8 alone. It also shrinks to about 1.03× at 512×1024512\times1024. End to end, the complete-model gain over the optimized dense reference is 1.24–1.59×.

(2) The practical impact is hard to judge from the current evaluation. The setting where this matters most is edge deployment, but all latency numbers come from PyTorch/CUDA on one Orin. TensorRT is left for future work, even though compiled pipelines are exactly where dense post-processing might already be fused or much cheaper. The exactness guarantee for panoptic output is also tied to one specific GPU, software build and kernel configuration (including a lookup table over the actual sigmoid kernel), and needs revalidation whenever any of these change. That makes "exact" much less portable than the abstract suggests.

Some headline numbers also depend on reference choices:

  • The instance DRAM reduction is 80.5% relative to the "+ additional optimizations" variant, which uses more DRAM (8147.3 MB) than the previous step (7046.3 MB). Measured against the latter, it would be about 77.5%.
  • Tables 1–2 use single-process medians, and MM and the schedule were selected on the same Cityscapes val split used to report them.

None of these alone is serious, but together they make it hard to tell how much of the speedup would survive in a production deployment.

Questions for authors

1. Why sparsify stride 16 at all? Given Table 10, where stride-16 sparse attention is slower than fused dense SDPA and contributes about 1% to decoder latency, what is the accuracy and latency of a stride-8-only sparse schedule? If it matches, that would simplify the method and remove one source of approximation.

2. How do the gains hold up against a compiled dense baseline? Could you report complete-model latency with TensorRT or torch.compile for both the dense reference and your pipeline, even for just one model? Related: how much effort does the panoptic numerical certification take when moving to a different GPU or software version? Is the sigmoid lookup table regenerated per device?

3. Which dense reference should the DRAM numbers use? In Table 7, the instance "+ additional optimizations" variant has higher DRAM traffic than "+ early class filtering" (8147.3 vs 7046.3 MB) while being faster. What causes the increase? Would you consider reporting the reduction against both references, so the "over 80%" claim is clearly scoped?

Reviewer_zXLrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes three training-free inference transformations for query-based segmentation models, using a frozen checkpoint’s own predicted masks and logits to decide which computation can be skipped, in order to reduce the memory traffic that dominates decoder and post-processing cost at high resolution. They replace dense masked cross-attention with a fixed budget of 128 feature cells per query selected within the predicted mask, predict intermediate masks directly at the next attention stride, and restrict output processing to regions that can affect the final result, with conservative score bounds that make the panoptic and instance outputs exactly match the original. Evaluation on Cityscapes, COCO, and ADE20K across several model families shows accuracy preserved within 0.4 points without retraining, with complete-model speedups of up to 1.35× (panoptic) and 1.59× (instance) over an optimized dense reference on an AGX Orin, and over 80% less DRAM traffic in output processing.

Critical strengths
  1. The transformations require no training and no per-checkpoint tuning: a single configuration selected on Cityscapes is applied unchanged to released checkpoints from five model families and transfers to COCO and ADE20K with accuracy changes within a few tenths of a point. Since most prior work on efficient query-based segmentation relies on architectural changes and dedicated training, a method that applies directly to existing dense-trained checkpoints addresses a different and practically relevant deployment scenario.
Critical weaknesses
  1. Two of the three components offer limited methodological novelty. Mask-guided sparse attention is essentially the mask-guided sampling of SparseOcc applied without training, and coarse intermediate mask prediction follows from the linearity of bilinear resizing; for these components the contribution is an empirical demonstration that frozen checkpoints tolerate the change rather than a new mechanism, which leaves the task-sufficient output processing as the only technically substantive part of the paper.
  2. The complete-model gains are modest and confined to a narrow operating regime. Speedups of 1.2–1.6× are obtained at native Cityscapes resolution with batch size one on a single edge GPU, decoder gains nearly vanish at lower resolutions, and all measurements use PyTorch rather than a deployment runtime such as TensorRT, where an optimized dense baseline would likely recover part of the memory-traffic advantage the method relies on. For a paper motivated by edge deployment, this leaves the practical headroom of the approach uncertain.
Questions for authors
  1. Section 2 distinguishes mask-guided sparse attention from SparseOcc by the training-free setting and the retained attention projections, and coarse intermediate mask prediction is equivalent in real arithmetic by the linearity of bilinear resizing. Beyond these points, could the authors clarify what the technical contribution of these two components is? If the substantive novelty lies in the output-processing component, the framing of the contributions should reflect this.
  2. All latency measurements use PyTorch on an AGX Orin. Could the authors provide results, even for one configuration, under a deployment runtime such as TensorRT, or otherwise show that the memory-traffic advantage persists against a compiled and fused dense baseline?
  3. The attention budget and schedule are selected on the Cityscapes validation set on which the main results are reported, with the held-out check in Table 6 limited to one model and task; the FastInst conversion loses accuracy without an explanation. Could the authors state a criterion for which checkpoints are compatible with the sparse attention, and report latency in addition to accuracy for the COCO and ADE20K transfer experiments?
Reviewer_GqixOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes three optimizations to speed up query-based segmentation models without sacrificing the performance:

  1. For the cross attention between queries and image feature maps, instead of full attention, only up to 128 feature cells are allowed.
  2. Alternate the calculating order of downsampling and mask decoding to achieve same result with less compute.
  3. Process the outputs on-demand to reduce DRAM traffic.
Critical strengths
  1. The proposed optimization achieves a better latency while keeping the performance almost lossless across different query-based segmentation methods.
Critical weaknesses
  1. Writing needs improvement and the current framing is hard to follow.
  2. All contributions are more engineering efforts than academic findings.
Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.