acceptodds
Under review as a conference paper at ICLR 2027

RCP: Residual-Complementary Prompting for Specialist-Guided Binary Segmentation

Abstract

Foundation segmentation models can complement task-specific specialists, but their predictions may be unreliable or redundant. We study this collaboration in binary polyp and camouflaged-object segmentation using a DINOv3-B specialist and frozen SAM3. On intervention-selected polyp cases, direct SAM3 replacement reduces Dice by 2.0 points, although a ground-truth-guided image-level choice between the specialist and direct SAM3 shows a potential 5.2-point gain. We propose Residual-Complementary Prompting (RCP), which generates SAM3 hypotheses that correct distinct specialist errors. A spatial arbiter combines the hypotheses with the specialist prediction, and a failure-aware module selects inputs for intervention. RCP improves Dice by 5.0 points over the specialist on intervention-selected cases and consistently improves performance across five polyp and three camouflaged-object benchmarks. A fixed training-calibrated threshold routes 31.2% of test inputs and captures 91.6% of specialist failures; a separate retrospective 30% budget analysis captures 92.4%. The direct-SAM3 oracle uses different candidates and does not bound RCP performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Prompt Learning, Multiple-Hypothesis Learning, Segmentation, Complementary Learning, Prompt Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LZckOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a binary image segmentation framework based on Residual-Complementary Prompting (RCP). The authors observe that, even when a general-purpose foundation model for segmentation underperforms compared to a specialized model (a "Specialist"), its predictions may still contain complementary information useful for correcting the Specialist's errors. Building on this insight, the paper shifts the objective: rather than requiring the foundation model to produce independent, accurate segmentation results, it optimizes multiple prompts to elicit segmentation hypotheses specifically designed to rectify the residual errors made by the Specialist. Overall, this work explores a valuable research direction by reframing the role of the foundation model—moving it from direct prediction toward targeted error correction for a specialized model.

Critical strengths
  1. The research motivation is well-grounded, introducing a meaningful perspective on the complementary collaboration between a specialist model and a foundation model. While existing adaptation and prompt-learning methods for foundation-based segmentation models typically focus on enhancing the segmentation accuracy of the foundation model itself, this work—RCP—goes a step further: it explores whether a foundation model can provide targeted assistance when a high-performing specialist model is already available.
  2. The method design demonstrates strong logical consistency. The three proposed modules address when to intervene, what hypotheses to generate, and how to fuse predictions, respectively; the overall framework follows a clear and coherent design logic.
Critical weaknesses
  1. Inconsistency in reported performance improvements.Tables 1 and 9 report a 4.4-point overall Dice gain with approximately 30% intervention, while Tables 3 and 5 show only a 5.0-point gain on intervened samples. Since non-intervened predictions remain unchanged, this would imply an overall gain of approximately 1.5 points under consistent image-wise averaging. The authors should clarify the aggregation methods and reconcile these results.
  2. Insufficient distinction from existing multiple-hypothesis learning methods.The set-level objective resembles Best-of-K learning, while the redundancy loss and spatial arbiter are similar to existing diversity regularization and prediction fusion methods. Stronger comparisons are needed to establish the unique contribution of specialist-relative correction.
Questions for authors

1.Could the authors clarify the relationship between the routed-subset improvements and the overall improvements reported in Tables 1, 3, 5, and 9? 2.Can the authors provide a stronger baseline that isolates the contribution of specialist-relative residual optimization? For example, a conventional best-of-K or multiple-choice learning objective could be used to train the same prompt generator without explicitly incorporating the specialist prediction into the set-level objective.

1 comment

We thank the reviewer for recognizing the motivation and coherent design of RCP. Following the review, we repeated the affected evaluations, corrected the descriptions underlying the performance discrepancy, and added the requested conventional best-of-KK baseline. The manuscript revisions are highlighted in blue.

Response: Parts (a) and (b) address your first and second questions, respectively.

(a) Relationship between routed-subset improvements and overall improvements: The table numbers in your question refer to the original submission. In the revised manuscript, original Tables 3, 5, and 9 correspond to Tables 4, 7, and 12, respectively; Table 1 retains its number. New Table 3 reconciles routed-subset and overall gains. Under pooled image-wise averaging, the overall gain equals the routed-subset gain weighted by the intervention fraction, because non-routed predictions remain unchanged. After repeating the affected evaluations, we identified that the original Tables 3 and 5 described a separate diagnostic subset but incorrectly labeled it as the operational routed subset underlying Tables 1 and 9. Consequently, their reported 5.0-point gain was not the routed-subset gain corresponding to the overall results. We corrected these descriptions and added the following matched decomposition in revised Table 3:

PopulationImagesSpecialist DiceFinal Dice
Routed2390.7604430.916000
Non-routed5590.8955350.895535
Complete evaluation7980.8550750.901664

Intervention covers 239 of 798 polyp evaluation images, or 29.95%. On routed images, Dice increases from 0.760443 to 0.916000, an improvement of approximately 15.56 percentage points. Weighting this gain by the intervention fraction produces approximately 4.66 percentage points of pooled overall improvement, matching the increase from 0.855075 to 0.901664. Table 1 reports a different aggregation: mean image-wise Dice is calculated within each complete evaluation dataset, followed by an unweighted mean across datasets. For polyp segmentation, this dataset-macro score increases from 0.8818 to 0.9258, reported as an increase from 0.882 to 0.926. Original Table 9, now Table 12, uses the same dataset-macro Dice convention; its description as pooled image-wise Dice was incorrect and has been corrected. The diagnostic comparison in original Tables 3 and 5, now Tables 4 and 7, concerns the separate 200-image subset explicitly defined in Appendix A.1.4. Its improvement from 0.866 to 0.916 therefore should not be multiplied by the operational intervention rate to reconstruct the overall gain. Thus, the original discrepancy involved both an incorrect subset description and insufficiently distinguished aggregation conventions. We clarified the intervention protocol and aggregation in Sections 3.2 and 4 and the Table 1 caption, added the matched decomposition in Section 5.1 and Table 3, defined the diagnostic subset in Appendix A.1.4, and corrected the affected descriptions and captions, including Table 12.

(b) Stronger baseline isolating the contribution of specialist-relative residual optimization: We added the conventional best-of-KK baseline in Section 5.2 and Table 5. Its prompt generator minimizes the segmentation loss of the best individual hypothesis, without explicitly rewarding specialist-relative residual correction. Both methods use the same specialist-guided prompt-generator architecture, K=3K=3, frozen SAM3, training schedule, spatial-arbiter architecture, and arbiter-training procedure. They are evaluated on identical selected image IDs for each seed. The controlled difference is therefore the prompt-training objective, rather than hypothesis count or fusion architecture. The results across three random seeds are:

DatasetBest-of-KK + ArbiterRCP + ArbiterDice improvement
Kvasir0.952±0.0030.952 \pm 0.0030.970±0.0020.970 \pm 0.0021.8 points
ClinicDB0.959±0.0030.959 \pm 0.0030.978±0.0020.978 \pm 0.0021.9 points
ColonDB0.865±0.0040.865 \pm 0.0040.889±0.0030.889 \pm 0.0032.4 points
CVC-3000.914±0.0050.914 \pm 0.0050.935±0.0040.935 \pm 0.0042.1 points
ETIS0.829±0.0060.829 \pm 0.0060.857±0.0050.857 \pm 0.0052.8 points

RCP achieves higher mean Dice on all five datasets. Table 5 additionally reports increases in residual-error coverage of 8.1–10.1 percentage points and reductions in correction redundancy of 0.15–0.18 relative to best-of-KK. Final Dice is evaluated over each complete dataset, while coverage and redundancy are evaluated on routed images. These results support the contribution of specialist-relative optimization: under the same architecture and routing conditions, RCP generates hypotheses that collectively correct more specialist errors with less overlap between corrective contributions. The baseline definition, matched experimental protocol, three-seed results, and discussion have been added to Section 5.2 and Table 5.

Reviewer_ats3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Residual-Complementary Prompting (RCP), a framework for collaboration between a task-specific segmentation specialist and a frozen foundation segmentation model, SAM3. The central idea is that a foundation model may provide useful corrective information even when its standalone segmentation performance is inferior to that of the specialist. The proposed framework consists of three components: a failure-aware estimator that determines whether foundation-model intervention is needed, an RCP module that generates multiple prompts to produce complementary segmentation hypotheses, and a spatial arbiter that integrates the specialist prediction with the generated hypotheses. The RCP objective combines residual-error coverage, correction redundancy reduction, and individual hypothesis validity. The method is evaluated on five polyp segmentation benchmarks and three camouflaged-object segmentation benchmarks. The authors report improvements over the specialist and several controlled intervention baselines, including an average Dice increase from 0.882 to 0.926 for polyp segmentation and from 0.842 to 0.888 for camouflaged-object segmentation. Ablation studies, failure-routing analyses, and computational evaluations are also provided.

Critical strengths

The paper addresses a meaningful limitation of conventional foundation-model assistance: a foundation model with inferior standalone performance may nevertheless correct errors made by a specialist. The proposed residual-based objective explicitly encourages generated hypotheses to address different specialist errors, rather than simply maximizing individual mask quality or generic prediction diversity. This formulation provides a clear motivation for the proposed method and is supported by comparisons against multi-prompt and mask-diversity baselines.

Critical weaknesses

The reported average inference latency decreases from 286 ms per image for RCP applied to every image to 115 ms for failure-aware RCP, compared with 41 ms for the specialist alone. Although this demonstrates a reduction in computational cost, the selective method remains substantially slower than the specialist. Moreover, the reported failure-capture rates are based on a predefined segmentation-quality threshold and should be interpreted alongside the fixed-threshold routing results. Additional evidence on the stability of the routing mechanism and the quality-latency trade-off would help establish its practical value.

Questions for authors
  1. Could the authors provide additional evidence that the residual-complementary objective is responsible for the performance gains, beyond encouraging multiple predictions to differ? In particular, how sensitive are the results to the formulation of Lset, Lred, and Lvalid, and do the generated hypotheses consistently correct distinct specialist errors across different datasets and random seeds? The existing ablations are useful, but further evidence would clarify the extent to which the proposed objective contributes independently of the multi-hypothesis architecture and spatial arbiter.

  2. How robust are the reported gains to the choice of the specialist-quality threshold δ and the intervention threshold τ? Since the failure labels depend on whether specialist Dice falls below δ=0.85, and the intervention threshold is calibrated to target approximately 30% intervention, could the authors clarify how changes in these two thresholds affect failure-estimation performance, intervention frequency, and final Dice?

  3. How does RCP compare with simpler alternatives that use the specialist prediction to guide foundation-model assistance? The current experiments compare against direct SAM3 prompting, multi-prompting, mask-diversity prompting, and selective replacement. Could the authors clarify whether simpler strategies, such as combining a single SAM3 hypothesis with a learned spatial arbiter or using a specialist-guided correction mask, achieve comparable performance under the same intervention budget? This comparison would help establish the incremental value of generating multiple residual-complementary hypotheses.

Reviewer_7BPvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RCP, a framework that uses frozen SAM3 to help correct a task-specific segmenter. It learns prompts that produce complementary corrections, combines the resulting masks with the specialist prediction through a spatial arbiter, and uses predicted failure risk to select images for intervention. Experiments cover polyp and camouflaged-object segmentation.

Critical strengths
  1. The paper gives a concrete definition of complementarity based on which specialist errors each hypothesis corrects. The training objectives follow this motivation closely.

  2. Evaluation across eight datasets in two domains also provides a useful breadth of evidence.

Critical weaknesses
  1. The overall gains are difficult to reconcile with the gains on intervention-selected cases. Different averaging weights alone cannot explain this discrepancy. The intervention rates, evaluation subsets, and aggregation rules need clarification.
  2. The failure-recall results in Section 5.3 also appear inconsistent. With the same risk scores and evaluation set, the former selection should contain the latter and therefore cannot have lower failure recall. This affects the evidence supporting the routing module.
Questions for authors

Does Table 1 use 30% selective intervention? Please specify the evaluation subsets, intervention rates, and Dice aggregation rules for the overall and selected-case results, and explain the discrepancy in their gains.

Do the two routing results in Section 5.3 use different runs, risk scores, or evaluation sets? If the conditions are the same, why does routing 31.2% of images yield lower failure recall than selecting the highest-risk 30%?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.