- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how quantization affects mask generation and candidate selection in SAM 2. It jointly learns layerwise precision and weights and adds a teacher-relative selection-regret loss. Precision sweeps suggest that selection deteriorates before candidate quality, and experiments report strong segmentation results at approximately 4.79 average bits.
Critical strengths
- The paper examines a consequential decision within the segmentation pipeline. It distinguishes oracle candidate quality, selected-mask quality, and selection accuracy instead of reporting only final segmentation scores. The precision sweep uses matched propagation units and trajectory-clustered bootstrapping, showing that selection can deteriorate even when candidate quality changes little.
- The precision profiles and segmentation experiments provide useful observations. The profile learned with segmentation loss alone retains relatively high precision in the predicted-IoU head, consistent with the selection diagnostics. Experiments cover three SAM 2.1 scales, and the Base+ result substantially improves over the reported uniform low-bit configurations. Comparing precision-only training with the regret-loss variants also helps assess what the added objective contributes.
Critical weaknesses
-
The reported average precision is not clearly tied to an implementable quantization budget. Section 4.2 reports a parameter-weighted average of continuous precision variables used in fake quantization. The paper does not explain how these values are converted to supported integer bit widths or report accuracy and storage after conversion. An average of 4.79 therefore does not, by itself, establish the cost of an exported quantized model.
The stated objective also contains no explicit bit-cost penalty or budget constraint. Clipping and smoothing stabilize the updates, but do not explain how the budget is enforced. The missing controller appendix is important here. Table 1 further labels HAWQv2 and MixQSAM with weight-only budgets, while the proposed method has a weight-and-activation budget. More detail is needed to establish whether these comparisons use equivalent costs.
-
The results show improved candidate quality more clearly than improved selection. The candidate diagnostics use training videos rather than held-out data. In Table 3, the proposed loss improves selected-mask quality, but also substantially improves mean candidate quality. Selection accuracy changes only from 0.558 to 0.563. The conclusion likewise states that candidate quality improves without meaningful changes in the selection statistics. This is a useful result, but it does not establish that the loss resolves the selector failure identified by the precision sweep.
On held-out SA-V, J&F increases from 70.47 to 71.09, while the top-tail variant reaches 71.34. The supplied PDF provides no seed variability or uncertainty for these differences. Held-out selection diagnostics and an ablation restricting updates to the predicted-IoU head would help clarify the loss’s specific contribution.
Questions for authors
- How is the precision budget enforced during optimization? Please provide the controller equations and settings, along with the exported integer allocation, actual storage, and segmentation performance after conversion.
- What activation precisions and quantization scopes are used for HAWQv2 and MixQSAM? How do these baselines perform when calibrated on video propagation data? Can a SAM 2-specific baseline be evaluated under the same budget and protocol?
- Does the regret loss improve hard regret, selection accuracy, or normalized selection skill on held-out videos? Please separate improvements in candidate generation from improvements in choosing among candidates.
- Does retaining higher precision only in the predicted-IoU head recover the selection losses? Please also report multi-seed uncertainty for the precision-only, teacher-relative, absolute-regret, and top-tail variants.