acceptodds
Under review as a conference paper at ICLR 2027

Where Precision Matters: Selection-Regret Learning for Quantized SAM 2

Abstract

Segment Anything Model 2 (SAM 2) generates multiple mask candidates for each prompt and selects one using its predicted-IoU head, but existing quantization objectives evaluate candidates independently and do not protect this selection process. We propose a mixed-precision quantization method that jointly learns the weights and a bit width for every layer, together with a teacher-relative selection-regret objective that penalizes only the additional mask quality forfeited by the quantized selector compared with its full-precision teacher. A controlled precision sweep over video propagation units shows that hard selection accuracy degrades significantly already at eight bits, even when candidate quality remains within half a percent of full precision, and that the resulting damage increases with propagation depth. Independently, precision learned using only the segmentation loss concentrates around the same predicted-IoU interface, identifying it as a critical quantization bottleneck. At an average precision of 4.79 bits, our method achieves 71.1 SA-V J&F and 0.743 COCO mIoU on SAM 2.1 Base+, whereas two mixed-precision baselines at the same budget preserve COCO accuracy but fall below 2.4 on video. At a matched bit budget, the regret objective raises candidate-set and selected-mask quality, each with a paired interval excluding zero.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
mixed-precision quantization, quantization-aware training, vision foundation models, video object segmentation, mask candidate selection, teacher-relative selection regret

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vq93OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how quantization affects mask generation and candidate selection in SAM 2. It jointly learns layerwise precision and weights and adds a teacher-relative selection-regret loss. Precision sweeps suggest that selection deteriorates before candidate quality, and experiments report strong segmentation results at approximately 4.79 average bits.

Critical strengths
  1. The paper examines a consequential decision within the segmentation pipeline. It distinguishes oracle candidate quality, selected-mask quality, and selection accuracy instead of reporting only final segmentation scores. The precision sweep uses matched propagation units and trajectory-clustered bootstrapping, showing that selection can deteriorate even when candidate quality changes little.
  2. The precision profiles and segmentation experiments provide useful observations. The profile learned with segmentation loss alone retains relatively high precision in the predicted-IoU head, consistent with the selection diagnostics. Experiments cover three SAM 2.1 scales, and the Base+ result substantially improves over the reported uniform low-bit configurations. Comparing precision-only training with the regret-loss variants also helps assess what the added objective contributes.
Critical weaknesses
  1. The reported average precision is not clearly tied to an implementable quantization budget. Section 4.2 reports a parameter-weighted average of continuous precision variables used in fake quantization. The paper does not explain how these values are converted to supported integer bit widths or report accuracy and storage after conversion. An average of 4.79 therefore does not, by itself, establish the cost of an exported quantized model.

    The stated objective also contains no explicit bit-cost penalty or budget constraint. Clipping and smoothing stabilize the updates, but do not explain how the budget is enforced. The missing controller appendix is important here. Table 1 further labels HAWQv2 and MixQSAM with weight-only budgets, while the proposed method has a weight-and-activation budget. More detail is needed to establish whether these comparisons use equivalent costs.

  2. The results show improved candidate quality more clearly than improved selection. The candidate diagnostics use training videos rather than held-out data. In Table 3, the proposed loss improves selected-mask quality, but also substantially improves mean candidate quality. Selection accuracy changes only from 0.558 to 0.563. The conclusion likewise states that candidate quality improves without meaningful changes in the selection statistics. This is a useful result, but it does not establish that the loss resolves the selector failure identified by the precision sweep.

    On held-out SA-V, J&F increases from 70.47 to 71.09, while the top-tail variant reaches 71.34. The supplied PDF provides no seed variability or uncertainty for these differences. Held-out selection diagnostics and an ablation restricting updates to the predicted-IoU head would help clarify the loss’s specific contribution.

Questions for authors
  1. How is the precision budget enforced during optimization? Please provide the controller equations and settings, along with the exported integer allocation, actual storage, and segmentation performance after conversion.
  2. What activation precisions and quantization scopes are used for HAWQv2 and MixQSAM? How do these baselines perform when calibrated on video propagation data? Can a SAM 2-specific baseline be evaluated under the same budget and protocol?
  3. Does the regret loss improve hard regret, selection accuracy, or normalized selection skill on held-out videos? Please separate improvements in candidate generation from improvements in choosing among candidates.
  4. Does retaining higher precision only in the predicted-IoU head recover the selection losses? Please also report multi-seed uncertainty for the precision-only, teacher-relative, absolute-regret, and top-tail variants.
Reviewer_ZdAmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, authors introduce a mixed-precision quantization method for SAM 2. It jointly learns layer-wise weights and bit widths parallel with a teacher-relative selection-regret objective. They also proposed a novel loss explicitly penalizes performance loss in candidate mask selection caused by quantization, safeguarding the predicted-mIoU selection mechanism across video propagation units. As a result, the method preserves both candidate-set and selected-mask quality, significantly outperforming conventional mixed-precision baselines on video segmentation tasks.

Critical strengths

1: The paper identifies a crucial and previously overlooked vulnerable issue in quantizing SAM 2: the failure mode of the candidate selection mechanism. The controlled precision sweep cleanly shows that selection accuracy degrades significantly before candidate mask quality dropping, with selection errors compounding over propagation depth in videos. 2: Rather than evaluating candidate masks independently or relying on traditional pointwise distillation, the authors introduce a novel Teacher-Relative Selection-Regret Loss (LSR\mathcal{L}_{SR}). By penalizing only the additional regret incurred by the student selector relative to the full-precision teacher, the objective specifically protects the candidate selection process without directly over-optimizing mask outputs. 3:The proposed method outperforms existing mixed-precision baselines (e.g., HAWQv2, MixQSAM) on video segmentation tasks by a massive margin, this demonstrates the effectiveness of the proposed method.

Critical weaknesses

1: While the selection-regret objective raises candidate-set and selected-mask quality, its impact on the final end-task benchmark scores is modest. As shown in the ablation study (Table 3), adding LSR\mathcal{L}_{SR} moves the SA-V J&F score from 70.47 to 71.09 (+0.62) and COCO mIoU from 0.7344 to 0.7433 (+0.0089) compared to the precision-only baseline. 2:The paper proposed method which was compared against recent SAM 2 quantization frameworks (e.g., Q-SAM2, MixQSAM2) using quoted numbers instead of running them under a protocol-matched environment. Since these external methods are using different training subsets, prompt protocols, or backbones, it seems very difficult to isolate the exact algorithmic advantage over existing SotA SAM 2 quantization techniques.

Questions for authors

1: Table 3 shows that by adding the teacher-relative selection-regret objective (LSR\mathcal{L}_{SR}) significantly improves candidate-set (OO) and selected-mask (HH) quality, but the resulting gains on end-task benchmarks (SA-V J&F and COCO mIoU) remain relatively modest compared to the precision-only baseline (+0.62 J&F and +0.0089 mIoU). Can authors explain what factors prevent these candidate-level improvements from translating into larger downstream segmentation gains, and could tuning the temperature (τ\tau) or margin (γ\gamma) bridge this gap?

2: The analysis clearly shows how quantization hurts the performance of SAM 2's predicted-IoU selection head before affecting candidate mask generation. Do authors observe similar early selection failure modes in other video segmentation models that score or rank their own candidate outputs (such as original SAM or diffusion-based candidate generation networks)?

Reviewer_qXCXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how low-bit quantization affects SAM 2, showing that candidate-mask selection deteriorates before substantial degradation in mask quality, with errors accumulating during video propagation. The main contributions include (1) identifying selection failure as a key quantization bottleneck, (2) learning layerwise mixed precision to identify precision-sensitive components, and (3) introducing a teacher-relative selection-regret loss. Experiments on SAM 2.1 Tiny, Small, and Base+ using SA-V and COCO validate the approach through precision sweeps, baseline comparisons, ablations, and targeted interventions. At 4.79 average bits, Base+ achieves 71.09 SA-V J&F and 0.7433 COCO mIoU. Although the proposed loss improves candidate quality, gains in selection accuracy and downstream segmentation remain modest.

Critical strengths
  1. A novel and well-supported insight into quantization-induced selection failures. The paper demonstrates that candidate-selection accuracy deteriorates significantly at 8-bit precision, even when candidate-mask quality remains nearly unchanged. The analysis of temporal error propagation further highlights a failure mechanism overlooked by conventional quantization approaches. This provides a meaningful conceptual contribution beyond standard accuracy-preserving quantization.

  2. Strong empirical validation of precision-sensitive components. The learned precision profiles, controlled precision sweeps, and targeted interventions consistently highlight the importance of decoder-related components. At 4.79 average bits, the method achieves 71.09 SA-V J&F and 0.7433 COCO mIoU, substantially outperforming the adapted mixed-precision baselines on video segmentation. These complementary experiments strengthen confidence in the practical relevance of the proposed precision-allocation strategy.

Critical weaknesses
  1. Limited evidence that the proposed selection-regret objective substantially improves selection performance. Although the loss improves candidate-mask quality, hard selection accuracy increases only from 55.8% to 56.3%, while SA-V J&F improves marginally from 70.47 to 71.09 (Table 3). Given the observed variability across random seeds, it remains unclear whether the regret objective provides a meaningful downstream advantage beyond learned precision allocation. This weakens the empirical support for the paper's central methodological contribution.

  2. Potential limitations in baseline fairness and comparative validation. The adapted HAWQv2 and MixQSAM baselines rely on single-frame synthetic calibration without temporal propagation (Appendix Table 16), potentially explaining their exceptionally poor video performance. Without stronger temporally informed baselines under comparable conditions, it is difficult to determine how much of the reported improvement comes from the proposed method rather than differences in calibration and training protocols. This limits the strength of the comparative evidence supporting acceptance.

Questions for authors
  1. Effectiveness of the selection-regret objective (Clarification). In Table 3, the proposed loss improves selected-mask quality from 0.6260 to 0.6723, but selection accuracy increases only from 55.8% to 56.3%, and SA-V J&F from 70.47 to 71.09. Could the authors clarify whether these improvements primarily result from better candidate generation rather than more accurate selection? Are the downstream improvements statistically significant across random seeds? This distinction is important for assessing the contribution of the proposed loss.

  2. Fairness of baseline comparisons (Clarification). Appendix Table 16 indicates that HAWQv2 and MixQSAM use single-frame synthetic calibration without temporal propagation, potentially contributing to their extremely low video segmentation scores. Could the authors clarify whether the performance gap would persist under temporally informed calibration or comparable video-based adaptation? This would help determine whether the improvements arise from the proposed precision-learning strategy or differences in experimental protocols.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.