acceptodds
Under review as a conference paper at ICLR 2027

Boundary Adaptation for Sparse Segmentation Refinement

Abstract

To be computationally efficient, segmentation networks typically predict masks at a low resolution. These coarse masks have errors concentrated along the boundary, which are particularly significant for small objects. Refinement methods have been proposed to correct these coarse masks, often updating a sparse subset of pixels for computational efficiency. Existing sparse methods use entropy for sparse pixel selection, but in high-quality segmentation networks (like SAM) high-entropy pixels do not strongly correlate with pixels that require correction. We present a novel sparse pixel selection strategy for segmentation refinement called Boundary Adaptation, and a corresponding pointwise refinement architecture called Boundary-SAM. We show the proposed sparse refinement method outperforms high-quality mask-based methods on natural image datasets, a remote sensing dataset, and a CT scan dataset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
segmentation, refinement, boundary

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JzANOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Boundary Adaptation (BA), a sparse mask-refinement policy that iteratively reclassifies boundary pixels. Boundary-SAM implements it through native-resolution pointwise decoding using frozen HQ-SAM features. Topology-aware updates, heuristic artifact removal, and multiscale processing are used to stabilize refinement. Experiments report competitive results on natural-image benchmarks, with benefits for fine structures and small objects, plus remote-sensing and 2D CT-slice results after domain-specific retraining.

Critical strengths
  1. BA provides a simple, well-motivated refinement policy that targets boundary errors in high-quality coarse masks. The oracle analysis and iterative-update/multiscale ablations provide useful empirical support for the refinement procedure, making its practical design a substantive strength.
  2. The method demonstrates useful improvements for fine structures and small objects. On ThinObjects-5K, it improves over the input SAM masks by 3.78 mIoU and 6.72 bIoU points. The size-stratified experiment in Figure 6 and native-resolution ablation in supplementary Table 7 further support its value for small objects in high-resolution images.
Critical weaknesses
  1. The theoretical reduction to independent pixel classification is not justified as stated. In Proposition 1, Eq. (6) omits interactions between boundary pixels, although the candidate set permits both labels to vary; these terms generally prevent the factorization in Eq. (7). Moreover, minimizing energy requires maximizing the defined ratio, whereas Appendix A minimizes it. The four-pass update schedule in Section 4.1 may support a restricted formulation under suitable interaction assumptions, but these assumptions are not stated in the proposition. Consequently, the claimed MAP interpretation requires correction.
  2. The experiments do not fully isolate the pixel-selection policy, which is the paper’s stated core contribution. The BA-versus-entropy comparison uses different training-mask distributions. Supplementary Table 8 provides useful boundary-band evidence, but does not report matched query budgets or clearly specify training adaptations for band selection. The CRM-based patch comparison in Table 9 also changes the decoding formulation, and its failure is attributed to missing SAM context-token conditioning. These limitations leave uncertainty about BA’s advantage independently of training and decoder effects.
Questions for authors
  1. Clarification of the theory: Is Proposition 1 intended to apply to single-pixel or noninteracting-subset updates, such as the four-pass schedule in Section 4.1, rather than simultaneous optimization over the full boundary? Please state the required interaction assumptions, account for boundary–boundary terms in Eqs. (5)–(7), and clarify whether the defined energy ratio should be maximized.
  2. Assessment-critical empirical comparison: Does BA retain its advantage over entropy and boundary-band selection using the same B-SAM decoder, matched training-mask distributions, and comparable total pixel-query budgets, including repeated queries? Please clarify which controls already hold in Section 5.3 and supplementary Table 8, and provide a controlled comparison if these conditions are not currently matched.
1 comment

W1 & Q1. We thank the reviewer for raising this important point. Indeed, Proposition 1 applies only to noninteracting-subset updates, such as the four-pass schedule in Section 4.1. We have revised Section A so boundary updates are executed on a subset of pixels according to their color class. This will remove the coupling between boundary terms in Eq 6, which makes the canceling of terms valid.

W2 & Q2. To more directly compare the selection policy, we include the Entropy-based Oracle method and the Entropy-based B-SAM method in Tables 1, 2, 5, and 6. The Entropy Oracle method queries the same number of points as the BA Oracle method, and the Entropy B-SAM method queries the same number of points as the BA B-SAM method. Note too that Entropy B-SAM uses the same B-SAM architecture as BA B-SAM.

However, the training procedure for BA B-SAM and Entropy B-SAM is necessarily different. BA B-SAM is trained on synthetically deformed ground-truth masks, whereas Entropy B-SAM is trained directly on coarse SAM masks and logits because Entropy-based selection requires access to the model's predicted probabilities. Importantly, both methods are evaluated on coarse SAM masks in Tables 1 and 2. Thus, the training distributions are not matched, and this difference potentially favors Entropy B-SAM, as its training data more closely resembles the coarse-mask inputs encountered during evaluation. Our training distribution of BA B-SAM is selected to match methods designed for refinement like CPSP and CRM.

In Table 10, the banded update experiment uses the same number of queries as our standard network (including repeated queries), the same B-SAM decoder, and the same training-mask distribution. These controls are already present in the originally submitted experiments.

Reviewer_zzAtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Boundary Adaptation (BA), an iterative sparse segmentation refinement strategy that reclassifies pixels along the evolving mask boundary rather than relying on entropy-based selection or full-mask prediction. The proposed Boundary-SAM combines frozen HQ-SAM features, low-level image features, mask information, and pixel coordinates for native-resolution point-wise refinement. Topology-preserving updates and a coarse-to-fine schedule are introduced to suppress artifacts and mitigate error accumulation. The method is evaluated through oracle experiments, five high-resolution natural image benchmarks, and separately retrained models for remote sensing and medical image segmentation.

Critical strengths
  1. The method addresses an important limitation of existing refinement approaches: full-mask processing can be computationally unnecessary when initial masks are already accurate, while low-resolution refinement tends to lose fine boundaries and small objects. Concentrating predictions on the evolving boundary enables native-resolution refinement while leveraging pretrained semantic features and local image details.

  2. The topology-preserving updates and multiscale schedule address two important problems: disconnected artifacts and accumulated prediction errors. The ablations in Section 5.3 and Appendix Figure 10 support the effectiveness of these components. The experiments on thin and small objects further demonstrate the practical potential of the proposed approach.

Critical weaknesses
  1. The claimed spatial independence of the Ising energy ratios does not hold in general. In deriving Eq. (6) from Eq. (5), the proof omits pairwise interactions between boundary pixels that are simultaneously allowed to change. When both pixels are variable, the interaction term JijsisjJ_{ij}s_i s_j is not constant and cannot be absorbed into independent unary terms. For example, consider two variable boundary pixels coupled by a single nonzero interaction JijJ_{ij}. Their four energy ratios satisfy R00R11/(R01R10)=eJij≠1R_{00}R_{11}/(R_{01}R_{10})=e^{J_{ij}}\neq 1, contradicting the claimed factorization into independent per-pixel functions. Such a factorization may hold for single-pixel conditional updates or mutually noninteracting update subsets, but not for the entire variable boundary as stated. Furthermore, pointwise binary cross-entropy supervision does not establish equivalence to joint MAP inference. These issues substantially weaken the theoretical justification of the proposed method.

  2. The experiments rely heavily on ground-truth bounding boxes and, in specialized datasets, ground-truth connected-component decomposition. These settings favor the assumption that initial masks already identify the relevant objects, while the proposed topology-preserving refinement cannot readily correct major structural errors. Moreover, Table 1 shows that B-SAM does not consistently achieve the highest boundary accuracy: its bIoU is lower than CascadePSP on UHRSD (77.94 vs. 79.49) and HQSAMRefiner on DIS5K (63.38 vs. 66.38). On BIG, even after adjusting the regularization parameter, its bIoU remains below CascadePSP. These results raise concerns about the generality of the performance claims, particularly regarding boundary localization. The cross-domain experiments also demonstrate adaptation through retraining rather than direct transfer.

Questions for authors
  1. Why are pairwise interactions between simultaneously variable boundary pixels omitted in Eq. (6)? Does the proposed factorization hold only for single-pixel conditional updates or mutually noninteracting subsets? Please state the precise mathematical conditions and explain how they correspond to the four-group update procedure.

  2. Are BA and entropy-based selection evaluated under matched coarse-mask training distributions, total point-query budgets, iteration counts, and topology postprocessing? Please clarify which factors are isolated by this comparison and which remain confounded with the selection strategy.

  3. How does BA perform when bounding boxes and connected components are obtained from predictions rather than ground-truth annotations? Do the experiments cover missing components or incorrectly connected objects? Please clarify the structural conditions required of the initial masks.

  4. How do the authors explain the inconsistent rankings between mIoU and bIoU, particularly on UHRSD, DIS5K, and BIG? Do these results suggest that B-SAM primarily improves region overlap rather than boundary localization in certain settings? Please clarify the scope of the claimed superiority.

  5. Table 4 reports a runtime of 776 ms for B-SAM, compared with 567 ms for HQSAMRefiner and 577 ms for SAMRefiner when refining ten segmentations. Although Appendix B states that features are extracted once per image and reused across BA iterations, does the reported runtime include feature extraction, and are features shared across object instances? Please clarify the timing protocol and provide evidence of how total runtime scales with image resolution, object count, and cumulative boundary queries.

2 comments

W1 & Q1. We thank the reviewer for identifying an omission in the original derivation. We agree that the factorization in Eq. (6) does not hold for an arbitrary set of simultaneously variable boundary pixels because pairwise interactions between such pixels cannot be absorbed into independent unary terms. In the revised manuscript, we have updated Proposition 1 to apply only to independent subsets of the Ising interaction graph. Specifically, under our 8-point-connected interaction model, pixels assigned to the same color group are nonadjacent and therefore have no pairwise interactions with one another. Consequently, when conditioning on all other pixels, the energy decomposes into independent per-pixel contributions within each group. We perform the boundary updates sequentially across the four color groups, ensuring that the factorization is used only for valid conditional updates rather than for the entire boundary simultaneously. We have revised the mathematical derivation and clarified this distinction in Section A.

W2. Your comment correctly points out that Tables 1 and 2 use favorable initial coarse masks by prompting a SAM network with the ground-truth bounding box. However, we refer the reviewer to Tables 5 and 6 in the revised manuscript (Table 3 in the original submission) for a comparison of refinement methods acting on lower-quality masks from MaskRCNN networks. We address the concerns about B-SAM’s consistency in our response to Q4, but quickly note that masks refined with our BA B-SAM method have the best mIoU and bIoU metric on 5 of the 7 datasets on Tables 1 and 2, which is more consistent than any competing method. Finally, the reviewer states that cross-domain experiments are demonstrated through re-training; we emphasize that B-SAM, HQSAM-Refiner, CPSP, and CRM are all fine-tuned for each appropriate datasets (KITS and SegFVG) as these methods were not originally trained to refine medical images nor remote sensing data, and comparing networks on out-of-distribution data is less reliable.

Q2. We clarify that both methods use the same total point-query budget and topology postprocessing described in Section 4 (see paragraph labeled “Heuristic Exceptions”). However, they are trained using different training-distribution masks but this mismatch favors the competing Entropy-based selection method.

BA and Entropy B-SAM refine exactly the same total number of query points across all masks in all datasets (Tables 1, 2, 5, and 6). Specifically, for each image and mask instance, Entropy B-SAM selects the top-N highest-entropy points, where N equals the total number of points queried by BA across all iterations for that instance. Importantly, the query budget counts individual query operations rather than unique spatial locations; repeated queries to the same point across iterations are therefore counted separately. For example, if BA queries 1,000 points across its iterations for mask ID A of image ID 0, Entropy B-SAM selects the 1,000 highest-entropy points for that same instance. However, unlike BA, Entropy B-SAM does not use an iterative query procedure; therefore, the two methods are matched in their total point-query budgets, but not in their number of iterations

BA B-SAM is trained on synthetically deformed ground-truth masks, whereas Entropy B-SAM is trained directly on coarse SAM masks and logits because Entropy-based selection requires access to the model's predicted probabilities. Notably, both methods are evaluated on coarse SAM masks in Tables 1 and 2. Thus, the training distributions are not matched, and this difference potentially favors Entropy B-SAM, as its training data more closely resembles the coarse-mask inputs encountered during evaluation. Our training distribution of BA B-SAM is selected to match methods designed for refinement like CPSP and CRM.

Q3. We clarify that the main experiments in Tables 1 and 2 use ground-truth bounding boxes to prompt SAM, representing a setting in which an approximate object location is provided, as in interactive image editing. Tables 5 and 6 further evaluate refinement from Mask R-CNN predictions, which do not use any ground-truth information for prediction and provide a more challenging initialization.

We acknowledge that B-SAM is intended primarily to refine a reasonably good mask, rather than recovery from arbitrary structural errors. In particular, topology-preserving refinement cannot generally recover components absent from the initial mask or separate objects that have already been incorrectly merged.

To assess the upper-limit of our method due to topological errors in the coarse mask, Table 7 (or Table 5 in the original manuscript) reports the results after refining with our BA Oracle method until improvement is no longer possible. While an impractical algorithm, the experiment suggests that topological errors are not the main problem in existing segmentation methods.

Q4. In Tables 1 and 2, BA B-SAM achieves the best mIoU and bIoU on 5 of the 7 datasets. However, the source of the inconsistency in rankings is unclear. We point out that in Table 3 (or Table 1 of the original manuscript), this inconsistency persists beyond our proposed method. For example; HQSAM-Refiner is typically better than SAM-Refiner, but reports an inferior on bIoU the ObjMix dataset; SAM-Refiner reports a better mIoU compared to CRM in only 4 of the 7 cases. The only mask property in which BA B-SAM reliably improves the mask quality over other methods is mask size; BA B-SAM is significantly better than other refinement methods on small masks as reported in Figure 6 (or Figure 7 in the original submission).

Q5. In Table 8 of our revised manuscript (and Table 4 in the original submission), we include the average wall-clock runtime of each method and peak memory consumption across the entire BIG dataset (100 masks). Note in the revised manuscript, we also included the query density of each method averaged across the BIG dataset. Before measuring the wall-clock runtime, the refinement network, image, and coarse mask are loaded into memory; the wall-clock runtime itself measures the time each refinement method takes to return the refined mask. This means the total time of all cumulative boundary queries and feature extraction are included in the reported runtime.

In all natural image datasets, each image is associated with only one mask. In the alternative datasets when there are many masks associated with each image, we do not share features across object instances.

In Table R2, we include a breakdown of the wall-clock runtime for three of the highest-quality methods. We note that BA B-SAM is about 30% - 60% slower than Entropy B-SAM.

Table R2. Wall-Clock Runtime by Resolution (ms)

Ave (H, W)BA B-SAMEntropy B-SAMHQSAM-Refiner
(1692, 2405)741.83524.73582.94
(2342, 3220)853.58598.78700.76
(2952, 3560)976.84663.90796.94
(3845, 5074)1358.51857.701096.80
Reviewer_4pKHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Boundary Adaptation (BA) for sparse mask refinement, with a multiscale variant (MS-BA) and a SAM-based decoder (B-SAM). Experiments on natural, remote-sensing, and medical images show improvements over SAM and several refinement baselines, particularly for some small objects. However, the evaluation mainly uses ground-truth-box-initialized SAM masks, and the contribution of BA is not fully separated from other B-SAM components, limiting the broader claims.

Critical strengths
  1. The paper addresses a practical problem in high-resolution mask refinement, where small objects and fine boundary details remain challenging. Restricting refinement to sparse boundary pixels is a sensible direction for avoiding unnecessary dense computation.

  2. The oracle experiments provide a clear motivation for BA and show that many residual errors in strong SAM masks can be reached through boundary updates. The evaluation also covers natural images, remote sensing, and medical imaging, which gives the study useful breadth.

Critical weaknesses
  1. BA assumes that the coarse mask already captures the relevant connected components, while the main experiments use ground-truth box prompts to initialize SAM. This favorable setting reduces missing-object and major topology errors, so the results mainly validate refinement of already well-localized masks. Evaluation without ground-truth box prompts and with more structurally incorrect masks would provide stronger evidence.
  2. B-SAM combines BA with strong SAM features and several additional design choices, while the direct BA-versus-Entropy comparison is only reported on ThinObjects-5K. More controlled experiments with the same backbone, decoder, training setup, and query/computation budget are needed to isolate the contribution of BA.
Questions for authors
  1. The manuscript repeatedly refers to Sections A, B, and C, but these sections are not included in the provided file. Are they part of the official submission? If not, where are the corresponding theoretical, architectural, and ablation details?
  2. Can the authors compare BA with Entropy and other sparse/boundary selection strategies using the same B-SAM backbone and decoder, with matched query and computation budgets across more datasets?
  3. How does BA handle a coarse mask that misses a connected component or has an incorrect topology? Results without ground-truth bounding-box prompts would help clarify this limitation.
  4. Please report an end-to-end computational comparison, including latency, memory, total queried-pixel ratio, and the cost of SAM/HQ-SAM feature extraction. This is important because inference uses 12+9+3 iterative updates.
1 comment

Q1. The supplemental section is available as a pdf under the Supplementary Material.

Q2/W2. In our revised manuscript, we have included Entropy B-SAM in Tables 1, 2, 5, and 6. The two methods use the same backbone and decoder, with matched query and computation budgets. The only difference in their training setups is the choice of training masks. Entropy-based selection requires access to the model's predicted probabilities (logits) and is therefore trained directly on coarse SAM masks and their corresponding logits, rather than the standard synthetically deformed masks used to train BA B-SAM. Importantly, both methods are evaluated on coarse SAM masks in Tables 1 and 2. Thus, their training distributions are not matched, and this difference potentially favors Entropy B-SAM, whose training data more closely resembles the coarse-mask inputs encountered during evaluation. We choose the training distribution for BA B-SAM to remain consistent with methods specifically designed for mask refinement, such as CPSP and CRM.

The number of query points each method uses is exactly equal on every mask and the computation budgets are matched. We are unaware of well-known alternative selection strategies, so we have created a few of our own. In Section 5.3, we demonstrate that updating all the pixels with a B-SAM network is not as effective as the iterative BA inference. In Tables 10 and 11 in Section D, we executed B-SAM using banded iterative updates and mask-based sparse update strategy. No alternative method we tried is experimentally better than our BA inference procedure.

Q3. Our proposed BA method cannot recover a missing connected component or correct certain topological errors in the initial coarse mask. For example, if a coarse mask of a donut incorrectly covers its center hole, BA will be unable to recover the hole. Nevertheless, coarse masks can often be meaningfully refined, as demonstrated by our Oracle case. Although SAM masks initialized with ground-truth bounding-box prompts provide a favorable starting point, we demonstrate the effectiveness of the BA Oracle and BA B-SAM methods using coarse masks from MaskR-CNN networks predicted without ground-truth box prompts (Tables 5 and 6). These results provide additional evidence that BA can meaningfully refine coarse masks beyond the ground-truth-box-prompted SAM setting.

Q4. In Table 8 of our revised manuscript (and Table 4 in the original submission), we report the wall-clock runtime and peak memory consumption of each method. Following your suggestion, we have also included the average query density of each method across the BIG dataset. The SAM/HQ-SAM image feature extraction step represents a shared computational cost across all methods using the same image encoder. Importantly, this cost is already included in the reported wall-clock runtime and memory consumption. The reported runtime also accounts for all 12+9+3 iterative updates.

Comments on the paper

1 comment

We thank the reviewers for their comments and we are encouraged by the positive feedback. In particular, we are glad reviewers believe our work addresses an important problem [4pKH,zzAT], and they find our oracle experiments provide a clear motivation for our proposed method [4pKH,JzAN]. Reviewer zzAT states our experiments “demonstrate the practical potential of the proposed approach”, and reviewer JzAN states our method “demonstrates useful improvements for fine structures and small objects”.

In response to the reviewers’ feedback, we have made several revisions to the manuscript. First, we corrected an error in Proposition 1 in Appendix A by reformulating the BA steps as updating only one color class of boundary pixels in parallel, rather than the entire boundary simultaneously. This revised theoretical formulation is consistent with our practical implementation, as described in Section 4.1.

Second, we have added the relevant theoretical derivation of the Potts term in Eq. 3 (Eqs. 3 and 4 in the original manuscript).

Finally, we have updated Tables 1, 2, 5, and 6 to include the Oracle and B-SAM entropy-based selection methods. These additions provide a more direct comparison between our BA selection strategy and entropy-based selection. Importantly, the number of queried points for each entropy-based method exactly matches the number of queried points used by the corresponding BA method, ensuring a fair comparison.

In addition to the revisions described above, we identified an issue in our original evaluation procedure and two methodological details that were not explicitly documented in the initial submission. We have addressed these points in the revised manuscript and provide further clarification below, including a discussion of their implications for the reported results.

1. Correction to the evaluation procedure and clarification of coarse-mask filtering

We identified an inconsistency in our evaluation code that affected our quantitative results. Specifically, when reporting IoU-based metrics, the original implementation incorrectly applied the filtering criterion to the refined masks (excluding masks with an mIoU below 10) rather than to the corresponding coarse masks (excluding masks with an mIoU below 50). This criterion reflects our objective of evaluating mask refinement given an initial mask of reasonable quality. We have corrected this inconsistency in the revised manuscript.

To provide additional context for the filtering procedure, 85.56% of coarse masks across all datasets satisfy the mIoU threshold of 50, as reported in Table R1.

Table R1. Percentage of Coarse Masks with mIoU > 50

DatasetMasks (%)
ThinObjects78.47
HRSOD95.00
UHRSD90.59
BIG89.00
DIS5K65.96
ObjMix94.75
KITS95.86
SegFVG97.61
Overall85.56

We also examined the performance of the refinement methods on coarse masks with an mIoU below 50. For these masks, HQSAM-Refiner and BA B-SAM improve only 15.64% and 7.4% of masks, respectively, to an mIoU of at least 75.

The corrected evaluation does not change any major conclusions, although it does affect one finding from our ablation study in Table 12, which investigates the capacity of the B-SAM network. Whereas the original results suggested that the B-SAM network might benefit from additional training data, the corrected results instead indicate that the network does not yet fully utilize the available training data.

2. Clarification of the B-SAM refinement procedure on natural image datasets

We also clarify a methodological detail that was not explicitly stated in the original manuscript. For experiments on natural image datasets (BIG, DIS5K, ThinObjects, HRSOD, and UHRSD), both B-SAM variants, using Boundary Adaptation and Entropy Selection, execute a single HQSAM-Refiner step before applying their respective sparse update strategies.

Consequently, the mask decoder is executed twice in total for both B-SAM variants (one initial HQSAM-Refiner step followed by one sparse update), compared with five decoder executions for HQSAM-Refiner. This detail is relevant to interpreting the computational comparison in Table 8. We further investigate the effect of varying the number of HQSAM-Refiner steps preceding the BA and Entropy refinement strategies in the ablation studies presented in Tables 14 and 15.

Our experiments indicate that a single HQSAM-Refiner step provides the best performance for both B-SAM refinement strategies, outperforming both B-SAM without an initial HQSAM-Refiner step and HQSAM-Refiner alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.