acceptodds
Withdrawn from ICLR 2027

Boundary-Informed Support-Guided Evidence Fusion for Training-Free Multimodal One-Shot Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How should category text and visual support be combined when they disagree about the identity or extent of a query object? We introduce Support-Guided Evidence Fusion (SGEF), which treats the annotated support as both a prompt and a reliability reference. A frozen SAM3 evaluates one query-text view and two layouts of the same one-shot support-query pair. Their aligned fields form a semantic-rescue branch and a layout-consistent signed-correction branch, while support-region overlap and inner-outer boundary fidelity determine an episode-specific mixing weight. This boundary-informed rule adds no learned parameters and requires no task-specific training or test-time optimization. Across eight benchmarks, SGEF reaches 89.28, 77.42, and 61.82 foreground mIoU on PASCAL, COCO, and LVIS, and obtains the highest score among the evaluated methods on four of five additional benchmarks. Component replacements and episode-level weight reassignment isolate the contributions of directional correction and support-boundary fidelity. Across six frozen encoder-decoder configurations evaluated on complete PASCAL and COCO fold 0, fusion improves the corresponding reference field in all twelve comparisons by 0.57-1.85 mIoU points. These results establish annotated support boundaries as an effective control signal for training-free multimodal one-shot segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
one-shot segmentation, few-shot semantic segmentation, multimodal segmentation, training-free inference, text-visual fusion, support-guided segmentation, boundary-informed evidence fusion, promptable segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x1AEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SGEF, a training-free method for one-shot segmentation that combines frozen SAM3 with category text and visual support. SGEF constructs three prompt views: a query-text view, a vertical support-query layout, and a horizontal support-query layout. The query outputs form a rescue field and a signed correction field. A scalar weight, computed from support-region fidelity and support-boundary fidelity, controls how much the signed correction contributes to the final output. Experiments on eight datasets, including PASCAL, COCO, LVIS, FSS-1000, PerSeg, DAVIS, ISIC, and Kvasir-SEG, compare SGEF against FSS-SAM3, CG-ICS, and several training-free baselines. SGEF reports gains on PASCAL, COCO, and LVIS, and ablations examine boundary fidelity, weight assignment, and alternative frozen evidence providers.

Critical strengths
  1. The boundary-informed reliability signal is a simple but effective idea. Using inner and outer bands around the support contour to measure foreground retention and background rejection is a concrete way to detect support errors that region overlap alone may miss. The ablation showing that region-only reliability loses 1.40 points on FSS-1000, 1.68 on PASCAL, and 0.62 on COCO supports the claim that boundary information matters. This is the main reason the paper is worth considering.
  2. The experimental coverage is broad. Eight datasets, multiple folds, class-partitioned benchmarks, and a provider-transfer study give a reasonably complete picture of where the method works and where it does not. The weight-assignment controls and the alternative frozen evidence provider study add useful evidence that the fusion rule is not tied to a single encoder or decoder.
Critical weaknesses
  1. The comparison against FSS-SAM3 is not compute-matched or evidence-matched, so the source of the gains is not isolated. SGEF uses three views: one query-text view and two paired support-query layouts. FSS-SAM3 uses one paired view. The paper says all views reuse the same annotated support image, preserving the one-shot protocol, but inference cost and evidence access are not the same. Table 7 shows SGEF at 1.327 seconds per query, slower than single-view FSS-SAM3, though faster than some other methods. The accuracy gains may partly come from additional views and additional frozen model evaluations. Without a compute-matched ablation, for example SGEF with only one paired view plus the query-text view, or FSS-SAM3 with an equivalent three-view setup, it is difficult to isolate how much of the gain comes from the boundary-informed fusion rule versus simply using more evidence. This is central because the paper’s main claim is that the fusion rule, not the extra views, drives the improvement. A similar issue applies to text access: SGEF uses category text, while several baselines use visual support only, and CG-ICS generates text from support. The comparison therefore mixes differences in input information with differences in fusion design.
  2. The method is essentially a hand-designed combination of existing frozen components, and the paper does not provide a theoretical or mechanistic justification strong enough for ICLR. The signed correction uses fixed exponential scaling, fixed attenuation, a fixed offset, and clipping. The reliability weight uses fixed constants for the sigmoid center and width. The final threshold is fixed at 0.35, and the support split ratio is fixed at 0.6. These values are claimed to be fixed across datasets, but there is no derivation, no learning, and no sensitivity study. The paper reports that the fusion rule helps empirically, but it does not explain why boundary fidelity should control the mixture in this particular functional form, why the coefficients take these values, or under what conditions the scalar reliability estimate should be trusted. For a training-free method, this is not automatically a fatal flaw, but ICLR expects either a strong conceptual advance or a thorough empirical characterization. The current paper provides neither at the level needed to support its broad claims.
  3. The reliability signal is a single global scalar per episode, and the method has not been tested when the support mask is noisy, incomplete, or ambiguous. The paper acknowledges that one scalar cannot express spatially varying trust and suggests a spatial reliability field as future work. But the current design means that a single bad region or boundary in the support can misweight the entire query. This is particularly concerning for objects with thin structures, multiple instances, occlusion, or large appearance variation between support and query. The boundary bands are constructed with a radius that depends on support area and is clipped to a small pixel range, which may behave differently across resolutions and object scales. No experiment tests support masks with missing parts, coarse boundaries, or annotation noise. Since the method’s central claim is that support fidelity controls fusion, robustness to support-mask quality is not a side issue; it is a prerequisite for trusting the method in realistic one-shot settings.
  4. The evaluation has unresolved failure cases and lacks statistical grounding. SGEF trails PR-MaGIC and CG-ICS on DAVIS, and the paper only briefly notes that the relative contributions of frame appearance change, support mismatch, and episode-level reliability remain unresolved. This is a clear limitation that deserves a diagnostic, not a future-work sentence. In addition, many gains are small, for example 0.91 points over CG-ICS on PASCAL, and the paper does not report standard deviations, confidence intervals, or significance tests across episodes or folds. Without variance, it is unclear whether the reported differences are meaningful. The paper also does not report per-episode or per-class failure rates, so the reader cannot tell whether SGEF improves broadly or wins on average through a subset of easy episodes. For an ICLR submission making general claims about training-free one-shot segmentation, this is a substantive evidentiary gap.
Questions for authors
  1. Compute-matched comparison. Can you report SGEF using only one paired view plus the query-text view, or FSS-SAM3 using an equivalent three-view setup with the same fusion rule? This would isolate how much of the gain comes from the boundary-informed fusion rule versus the additional views and frozen model evaluations.
  2. Hyperparameter sensitivity. How sensitive is SGEF to the fixed coefficients in the signed correction, the weight mapping, the final threshold, and the support split ratio? Please report a sweep or at least a robustness analysis around the chosen values.
  3. Failure analysis on DAVIS. SGEF trails PR-MaGIC and CG-ICS on DAVIS. Can you provide a per-episode analysis showing whether the failures come from frame appearance change, support mismatch, or incorrect reliability estimates? A spatial reliability field is suggested as future work, but a diagnostic now would help.
  4. Variance and significance. Many gains are small. Can you report standard deviations or confidence intervals across episodes or folds, and clarify whether the reported differences are statistically significant?
Reviewer_88UcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a training-free one-shot segmentation method based on boundary information. It extracts semantic-rescue information from query-text and support-conditioned predictions and uses support and query images to generate horizontal and vertical canvases. These canvases are used to generate signed corrections, which are further combined with the rescue information. To determine the weight for the linear combination of rescue and correction, the authors use erosion- and dilation-based processing of the support mask, along with region overlap. They report improved performance on PASCAL-5i and COCO-20i in the one-shot setting.

Critical strengths

The paper shows improved performance on PASCAL-5i and COCO-20i in the one-shot setting compared to existing methods. These improvements demonstrate the effectiveness of the proposed approach, although the method has limited novelty and relies heavily on heuristics.

Critical weaknesses
  1. The paper has a substantial lack of novelty compared to [1], where [1] already uses support and query images placed on a shared canvas to perform few-shot segmentation. [1] uses support and query images in a canvas to generate predictions, where a bounding box is provided for the support image to guide query segmentation in the vision-only setting, and category text is additionally provided along with the bounding box in the vision+text setting. Instead of the single forward pass in [1], the proposed method uses three different forward passes for each of the following: query image + text, query image + support image canvas (horizontal), and query image + support image canvas (vertical). The authors then perform heuristic-based refinement on the resulting query predictions.

  2. The paper relies heavily on heuristics without providing sufficient motivation or justification for the introduced components. For example, the need for Equation (1) and the use of its output as a cubic factor in Equation (2) are not clearly justified. Equations (4), (5), (6), (10), and (11) introduce several heuristic operations with fixed constants, making the method largely heuristic-based.

  3. The paper is also poorly written and motivated. The introduction does not adequately explain the need for each introduced component, and the methodology similarly lacks sufficient justification for several design choices. The paper also does not report five-shot performance, which is evaluated in related few-shot segmentation work such as [1].

[1] Few-Shot Semantic Segmentation Meets SAM3, arXiv, April 2026.

Questions for authors

Please see weakness

Reviewer_VoXTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of reconciling inconsistent predictions—arising from class-based text prompts versus visual support samples—within the context of training-free, multimodal, one-shot segmentation. Building upon a frozen SAM3 architecture, the proposed method constructs three distinct views: a query image paired with text, a vertical support-query concatenation, and a horizontal support-query concatenation. A "semantic remedy" branch is employed to preserve and augment foreground regions, while residuals derived from the two layout-based views are used to reinforce or suppress specific predictions. Finally, fusion weights are determined based on regional overlap with support samples, as well as the fidelity of inner and outer boundaries. The approach is evaluated across eight datasets, with additional experiments covering component substitution, weight permutation, and the integration of alternative frozen encoders and segmentation models. The results demonstrate that this fusion strategy consistently outperforms single-view FSS-SAM3, with particularly notable gains observed on the LVIS dataset.

Critical strengths
  1. The primary results cover a broad scope, demonstrating a consistent level of improvement. Compared to FSS-SAM3—which utilizes the same categories of textual and visual support—SGEF achieves gains of 2.87, 3.60, and 8.69 mIoU points on the PASCAL, COCO, and LVIS datasets, respectively. Improvements are observed across all four splits of COCO and all ten splits of LVIS, indicating that the results are not driven by any single, particularly favorable split. The evaluation further extends to diverse data types, including personalized objects, video frames, and medical imagery. The authors also transparently report that their method underperforms relative to PR-MaGIC and CG-ICS on the DAVIS dataset, reflecting a measured and cautious stance regarding the scope of the method's applicability.
  2. The ablation studies account for potential confounding factors introduced by multi-view inference and adaptive weighting. Table 4 compares different fusion strategies while keeping the three input views constant; this approach is more compelling than a simple comparison against a single-view model. Table 5 further isolates the branch predictions—varying only the weight allocation—to demonstrate that adaptive weighting cannot be fully replaced by a simple uniform (average) weighting scheme. Additionally, the method yields positive gains across six different "frozen-model" configurations, providing preliminary evidence for its generalizability.
Critical weaknesses
  1. The additional contributions relative to existing work still require more explicit validation. Support-based approaches—such as query concatenation, freezing the SAM3 model, and combining text with visual prompts—are already core design elements of FSS-SAM3. Furthermore, the use of prediction quality from support samples to modulate the contribution of information on the query side has a precedent in the TACC mechanism within MM-FSS. The authors have cited these works; therefore, the issue is not an oversight of existing literature, but rather whether the newly introduced boundary metrics and residual fusion rules provide sufficiently clear and generalizable improvements.
  2. The method incorporates several empirical choices, such as the cubic power used in semantic remediation, the asymmetry coefficient for residuals, a fixed offset, and reliability mapping. While the local stability of three of these settings has been examined—a point that deserves recognition—the process for selecting these parameters remains unexplained. Furthermore, the fixed offset of 0.05, as well as the center and temperature parameters of the sigmoid function, have not been analyzed individually. The ablation study on symmetric correction simultaneously removes both the quality-dependent component of the positive coefficient and the square-root factor of the negative term; consequently, while this demonstrates the overall effectiveness of this set of design choices, it fails to isolate the specific contribution of each individual component.
  3. Table 4 presents a setup in which the joint reliability parameters are altered, while the center and temperature of the sigmoid function remain unchanged. This not only removes boundary information but also modifies the scale and distribution of the reliability scores. Consequently, the observed performance degradation cannot be attributed solely to the absence of boundary information. The permutation experiments in Table 5 support the notion that "task-aligned weights are beneficial," but they do not, on their own, prove that "boundary indicators provide valid information beyond what is captured by regional indicators."
  4. Whether the segmentation quality of the model—when provided with box prompts derived from ground-truth annotations—can predict the relative performance of the two branches on the query side is a question that requires direct verification. A single example, as shown in Figure 2, is insufficient to answer this.
Questions for authors
  1. The rationale for the selection of fixed coefficients and thresholds is not specified. While Table 8 in Appendix C provides sensitivity analysis results for some parameters, please further clarify the data sources used to determine these parameters. Additionally, for those fixed offsets and reliability mapping parameters not yet analyzed, do the primary performance advantages persist if their values ​​are moderately altered?
  2. Does boundary information still provide performance gains after accounting for the effects of score calibration? Since the value distributions for regional reliability and joint reliability may differ, applying the same mapping function to both could bias the comparison; therefore, please calibrate each separately before making comparisons. Furthermore, it is recommended to include results for the semantic remedy branch used in isolation, the correction branch used in isolation, and a configuration employing an optimal fixed mixing weight. This will help clarify the specific contribution of the adaptive fusion mechanism.
  3. Do the reliability scores assigned to support samples accurately reflect the actual gains achieved on query images? Please analyze the relationship between these scores and the difference in IoU (Intersection over Union) between the two branches for query images, and report the fusion gains observed across different reliability intervals. Furthermore, oes this relationship—as well as the primary performance improvement—remain stable when the query image is held constant while the support image is varied?
  4. Please provide the ablation results for the final weighted fusion step in the cross-model experiments. Table 6 currently compares the baseline predictions with the final results obtained using Equation (17). It is recommended to also report the performance after fusing the two branches described in Equation (16). Comparing these three sets of results would help clarify the effectiveness of the core fusion step itself, as well as the impact of the final weighted combination with the baseline predictions, thereby providing a clearer explanation for the performance gains observed in the cross-model experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.