Boundary Adaptation for Sparse Segmentation Refinement

To be computationally efficient, segmentation networks typically predict masks at a low resolution. These coarse masks have errors concentrated along the boundary, which are particularly significant for small objects. Refinement methods have been proposed to correct these coarse masks, often updating a sparse subset of pixels for computational efficiency. Existing sparse methods use entropy for sparse pixel selection, but in high-quality segmentation networks (like SAM) high-entropy pixels do not strongly correlate with pixels that require correction. We present a novel sparse pixel selection strategy for segmentation refinement called Boundary Adaptation, and a corresponding pointwise refinement architecture called Boundary-SAM. We show the proposed sparse refinement method outperforms high-quality mask-based methods on natural image datasets, a remote sensing dataset, and a CT scan dataset.