acceptodds
Under review as a conference paper at ICLR 2027

EvidenceSAM: Hierarchical Source Evidence Fusion with SAM for Multimodal Semantic Segmentation

Abstract

Multimodal semantic segmentation improves dense scene understanding by combining complementary observations from heterogeneous sensors. However, the usefulness of different modalities is inherently spatially and semantically dependent: a source that is informative for one region or category may be redundant or unreliable for another. Most existing approaches address this challenge primarily through feature-level interaction, where source-specific information is progressively compressed into a shared representation. This becomes particularly important when adapting large segmentation foundation models, where multimodal reasoning should exploit complementary sensing cues without sacrificing the transferable representations learned during pretraining. We introduce EviSAM2, a parameter-efficient SAM2-based framework that preserves source-specific evidence beyond the initial feature-fusion stage. Rather than treating multimodal fusion as a single operation, EviSAM2 organizes prediction into three complementary levels: Cross-Modal Feature Fusion (CMF) constructs the shared multimodal representation, Risk-Decomposed Expert Aggregation (RDEA) integrates source-specific semantic evidence, and Context-Guided Semantic Refinement (CGSR) further refines the aggregated prediction using class-level context. This formulation explicitly separates feature participation, semantic contribution, and contextual refinement, allowing different sources to play different roles throughout the prediction process. EviSAM2 retains most of the pretrained SAM2 backbone and requires only 4.0M trainable parameters for multimodal adaptation. On MCubeS, the four-modality model achieves 55.46% mIoU, these results demonstrate the effectiveness of maintaining explicit source-wise evidence throughout multimodal foundation-model adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ULn4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Through a diagnostic experiment on MCubeS, the authors find that source participation at the feature level is nearly uncorrelated with the improvement that per-source experts bring to the prediction, suggesting that "participation does not equal contribution." This indicates that after multimodal feature fusion, the prediction stage still needs to account for how much each modality's source-specific information actually affects the prediction. EvidenceSAM therefore performs multimodal feature fusion in the CMF stage while retaining source information; in the RDEA stage, it uses source features to separately model corrective and degradation evidence with respect to the base prediction, so as to assess prediction-level source contribution; CGSR further decomposes the semantic states provided by RDEA on a per-class basis. On MCubeS / DeLiVER, the method improves over the strongest baselines by +2.58 / +2.33 mIoU.

Critical strengths
  1. The main text is clearly structured.

  2. The motivation is clear: the paper challenges the implicit assumption—adopted by many existing multimodal fusion methods for multimodal semantic segmentation—that source participation is roughly equivalent to its contribution, and proposes the EvidenceSAM framework to validate the necessity of source identity at the decision-decoding stage after multimodal fusion.

  3. The method achieves state-of-the-art performance on multimodal semantic segmentation, with clear improvements.

Critical weaknesses
  1. Insufficient comparative experiments. There is no comparison isolating TASR, so its actual effect remains unverified. Could TASR be replaced by a standard per-pixel spatial gating? What is the necessity of this design? TASR is also not shown in the architecture figure (Fig. 2).

  2. The paper defines and uses a quantity called "semantic contribution," but prediction-level semantic contribution is not explored for the first time here; closely related concepts were studied earlier in Perceptual Score (NeurIPS 2021) and PID (Liang et al., NeurIPS 2023). The paper should discuss its relation to prior work on measuring semantic contribution / modality dependence, and avoid describing itself as the first to raise this question.

  3. The figures and the notation system have clear defects. In the RDEA section, symbols are reused across different meanings and under-explained, which makes the method hard to follow; several variables are not clearly defined.

Questions for authors
  1. TASR receives no explicit task supervision and is trained only end-to-end. How is "task alignment" actually achieved?

  2. Please provide the construction of EBE_B, CmC_m, RmdR_m^d, and HmdH_m^d (formulas or code), and report the values of λ\lambda, δ\delta, TsT_s, and κ\kappa, together with a sensitivity analysis.

  3. RDEA, as the core module of the method, is not described in sufficient detail to ensure reproducibility. Do the authors plan to release the source code to ensure the transparency of the method?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ULn4: TASR, RDEA Construction, and Sensitivity Analysis

We thank the reviewer for the constructive suggestions concerning TASR, the relationship to prior work, and the mathematical transparency of RDEA. We have revised the method description and added controlled mechanism comparisons and parameter sensitivity analyses.

  1. TASR and task alignment

TASR is now explicitly illustrated in Figure 2, including pairwise scoring, bounded source exchange, and the updated fused representation.

Sections 3.2 and Appendix 6.1 describe the antisymmetric transfer mechanism. Each pairwise transfer increases one source's allocation while decreasing another's, preserving total support and keeping unsupported sources inactive.

Task alignment arises from the downstream segmentation objective. Appendix 6.1, Eq. (26), derives the gradient of the task loss with respect to the pairwise exchange. This relation connects the learned redistribution to task-dependent gradients through the difference between source features, without requiring an additional alignment target.

We have also added separately trained mechanism comparisons on both MCubeS and DeLiVER (Table 3). Replacing TASR with pixel-wise spatial gating yields 52.44%/62.56% mIoU, compared with 55.46%/67.71% for EvidenceSAM. The improvements are 3.02/5.15 points. Removing TASR while retaining support-normalized fusion yields 52.46%/62.63%.

  1. Relationship to prior work

We have expanded the relevant discussion in Section 2 by citing Perceptual Score (Gat et al., 2021) and the multimodal information decomposition framework of Liang et al. (2023).

These studies examine modality dependence and task-relevant multimodal information. EvidenceSAM addresses a complementary problem: evaluating source-associated semantic alternatives relative to a fused base prediction and incorporating their evidence into spatially varying semantic decisions. Our contribution is the use of this distinction in the prediction process, rather than introducing modality contribution as a new general concept.

  1. RDEA evidence construction and notation

We have clarified the construction of E_B, C_m, R_m^d, and H_m^d in Section 3.3 and Appendix 6.2.

Appendix 6.2, Eqs. (27)–(30), specifies the base and expert prediction statistics, coarse verifier features, class-belief weighting, and detail readouts. The four sigmoid evidence factors estimate base-error evidence, expert-correctness evidence, direct corrective evidence, and direct degradation evidence.

Eq. (9) combines these factors into corrective and degradation scores. Eqs. (10)–(11) define expert admission, ranking, and posterior aggregation. Appendix 7.2, Eqs. (48)–(50), further specifies the auxiliary training targets and losses.

  1. Parameter values and sensitivity analysis

The revised manuscript reports the RDEA parameters as lambda = 1.25, delta_adm = 0.01, T_s = 0.35, and kappa = 6.0.

Appendix 9.3 and Table 13 report a one-parameter-at-a-time sensitivity analysis on MCubeS using the trained checkpoint. Across 17 distinct inference configurations, the maximum decrease from the default performance is only 0.095 mIoU points.

The routing behavior remains responsive to the parameters. Increasing lambda from 0.625 to 1.875 changes expert admission from 70.79% to 17.89%, while final segmentation accuracy varies only slightly. Further analysis shows that 96.14%–99.92% of pixels with changed routing decisions retain their aggregated semantic class under the evaluated extreme settings.

The separately trained direct-utility-head ablation provides complementary evidence for the risk decomposition itself, with improvements of 3.16/5.49 points on MCubeS/DeLiVER.

  1. Reproducibility and code release

The revised manuscript provides detailed expert inputs, evidence construction, selection rules, auxiliary objectives, gradient boundaries, and training configurations in Sections 3.3 and Appendices 6–7. We also plan to release the source code and corresponding experiment configurations publicly to support reproducibility.

We hope these revisions address the questions regarding TASR's mechanism and learning signal, the relationship to earlier research, and the construction and stability of RDEA.

Thank you for the extensive revisions. The revised version improves the readability and transparency of the paper, and substantially completes the description of the relevant modules and the evidence for their effectiveness. This is a substantive improvement, and I am willing to raise my score accordingly.

That said, readability and transparency should have been ensured before submission so that review effort is not spent on reconstructing the method. The lack of clarity in the submitted version made the paper substantially harder to understand, and this was avoidable.

Thank you for taking the time to carefully review our revised manuscript and for acknowledging the improvements. We appreciate your candid feedback and agree that the method should have been presented more clearly in the original submission. Your comments have helped us improve the clarity, technical presentation, and overall quality of the paper. We sincerely appreciate your thoughtful consideration.

Reviewer_f9joOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EvidenceSAM for four-modality semantic segmentation on MCubeS and DeLiVER. Its key insight is that a modality’s role in feature fusion does not necessarily reflect its contribution to the final prediction. Using a mostly frozen SAM2/Hiera-B+ backbone with 12.9M trainable parameters, EvidenceSAM combines three stages: CMF adjusts fusion weights based on modality availability and quality, RDEA evaluates each modality’s corrective and degradation evidence to decide whether to use its prediction, and CGSR combines class-level semantic evidence using prototypes and precision-weighted pooling. The method achieves 55.46% mIoU on MCubeS and 67.71% on DeLiVER, outperforming MemorySAM by 2.58 and 2.33 percentage points, respectively. Experiments on MCubeS further examine component contributions, the relationship between fusion weights and expert gains, and alternative decision rules.

Critical strengths
  1. Clear and useful idea. The paper distinguishes a modality’s role in feature fusion from its actual contribution to the final prediction. This is a sensible approach that could also work for other multimodal tasks.

  2. Efficient and well documented. EvidenceSAM trains only 12.9M parameters and outperforms MemorySAM and MLE-SAM on both benchmarks. The appendix provides useful implementation details, and the authors openly discuss limitations in their diagnostic experiments.

Critical weaknesses
  1. Unfair baseline comparisons: EvidenceSAM uses a 119M-parameter pretrained SAM2 backbone, while most non-SAM baselines use the much smaller MiT-B0. Larger-backbone results from methods such as CMNeXt are missing.

  2. Missing baselines and efficiency comparisons: Several relevant methods, including MM-SAM, DFormerv2, and OmniSegmentor, are not compared. The paper also lacks comparisons of total parameters, FLOPs, and inference speed.

  3. Unclear evaluation protocols: It is unclear whether baseline results were reproduced or taken from prior work, and whether all methods use the same evaluation settings.

  4. No appropriate SAM2 baseline or proper ablations: Without a simple SAM2 baseline using the same training recipe, it is difficult to attribute improvements to the proposed modules. Removing modules only at inference also fails to establish their individual contributions.

  5. Limited evidence for the core novelty: Alternative RDEA decision rules reduce mIoU by only about 0.19 points, while the correlation analysis does not convincingly establish the claimed distinction between fusion weights and semantic contribution.

  6. Limited statistical validation: Experiments use only one random seed and 102 MCubeS test images, with no corresponding ablation study on DeLiVER.

Questions for authors
  1. SAM2 baseline: What are the results of a simple-fusion SAM2.1/Hiera-B+ baseline using the same training setup?

  2. Ablation studies: Can you provide retrained incremental ablations and RDEA variants, with mean ± std over multiple seeds on both datasets?

  3. Baseline comparisons: How does EvidenceSAM compare with stronger versions of CMNeXt, MAGIC, and StitchFusion, as well as MM-SAM, DFormerv2, and OmniSegmentor? Please include parameters, FLOPs, and runtime.

  4. Evaluation protocol: Were baseline results reproduced or taken from prior papers? Were all methods evaluated under the same settings?

  5. Robustness: How does the method perform with missing, noisy, or degraded modalities, and when fewer than four modalities are available?

1 comment
AuthorsOpenReview ↗

Response to Reviewer f9jo: Controlled Comparisons, Efficiency, and Robustness

We thank the reviewer for the detailed comments on experimental fairness, component attribution, and robustness. We have expanded the experiments and clarified the evaluation protocols to better distinguish the contributions of EvidenceSAM from those of the pretrained backbone.

  1. Same-backbone baseline and retrained ablations

We have added a simple-fusion SAM2.1/Hiera-B+ baseline using the corresponding dataset-specific training and evaluation protocols. It achieves 49.24% mIoU on MCubeS and 61.81% on DeLiVER.

Table 2 reports independently trained incremental configurations on both datasets. Adding CMF yields 51.81%/63.84%, adding the source experts and RDEA yields 54.57%/66.35%, and the complete model achieves 55.46%/67.71%. EvidenceSAM therefore improves over simple-fusion SAM2 by 6.22/5.90 points.

Table 3 further examines separately trained mechanism replacements. TASR outperforms pixel-wise spatial gating by 3.02/5.15 points, while risk-decomposed expert aggregation outperforms a direct-utility head by 3.16/5.49 points. These comparisons provide direct evidence for the proposed designs under the same SAM2 backbone family.

  1. Stronger baselines and evaluation protocols

We have clarified the distinction between published external comparisons and our controlled experiments. Table 1 uses reported MiT-B0 configurations for non-SAM methods and the published backbones for SAM-family approaches. Appendix 8.4 additionally reports stronger MiT-B2 configurations of CMNeXt, MAGIC, and StitchFusion, together with OmniSegmentor.

The external results are taken from the corresponding publications rather than reproduced under one unified training protocol. Appendix 7.4 now specifies this distinction. Accordingly, Tables 2 and 3 provide the primary controlled evidence for attributing performance gains to our components.

We have also examined the other suggested methods. MM-SAM does not report directly corresponding four-modality results on MCubeS or DeLiVER, while DFormerv2 reports RGB-D rather than four-modality DeLiVER segmentation. OmniSegmentor's published DeLiVER result is included in Appendix 8.4.

  1. Model complexity and efficiency

Appendix 8.5 and Table 9 now report total parameters, trainable parameters, GFLOPs, and inference latency. EvidenceSAM has 89.84M total parameters, including 12.90M trainable parameters, with 4,190.66 GFLOPs and 269.03 ms per-image inference latency under the reported configuration.

Compared with MemorySAM, EvidenceSAM improves mIoU by 2.58/2.33 points on MCubeS/DeLiVER while using fewer trainable parameters (12.90M versus 13.80M). We report the inference costs separately from the parameter-efficient adaptation results.

  1. Evidence for source contribution

Beyond the original fixed-weight interventions, the retrained direct-utility-head comparison in Table 3 evaluates the risk decomposition itself, with improvements of 3.16/5.49 points.

The MCubeS diagnostic further finds Pearson correlations of -0.009, 0.038, and -0.022 between initial CMF weights and conditional source-expert gains. The bootstrap intervals include zero. We interpret these results as weak linear associations in the evaluated setting, supporting separate assessment of prediction-level source evidence.

  1. Noise and missing-modality evaluation

Appendix 8.3 now includes auxiliary-only, RGB-only, and joint corruption settings, together with missing-modality experiments on MCubeS.

Under auxiliary-modality corruption, EvidenceSAM achieves 53.32% average mIoU versus 41.51% for simple-fusion SAM2. The corresponding drops from clean performance are 2.14 and 7.73 points. At medium severity, EvidenceSAM reduces the performance loss by 4.75 points relative to the baseline, with a paired 95% bootstrap interval of [3.20, 6.57].

With RGB retained, EvidenceSAM also outperforms the baseline across all seven auxiliary-modality omission settings, with an average advantage of 4.39 points. RGB-only inference achieves 49.92% versus 45.10% mIoU.

We believe these new controlled ablations, stronger published comparisons, efficiency measurements, and degradation experiments substantially strengthen the empirical evaluation of EvidenceSAM.

Reviewer_xe73OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper present EvidenceSAM, a SAM2 based methode for multimodal semantic segmentation.

It has three main patrts: CMF combines features from different modalities. RDEA decides whether predictions from source experts should contribute to the base prediction. CGSR uses class level context to refine the result.

Experiments on MCubeS and DeLiVER show improvements over the reported baselines.

Critical strengths
  1. Improved results on two datasets. The paper reports improvements on MCubeS and DeLiVER. Compared with MemorySAM using the same backbone, improvment are 2.58 and 2.33 mIoU points. These results show that the full framework is useful for multimodal segmentation.

  2. A meaningful problem and a useful research direction. The paper studies an important problem in multimodal fusion. Different modalities may help in different regions and for different classes. The method keeps information from each modality after feature fusion. It then checks whether a source expert can improve the prediction. If an expert may make the result worse, the model can keep its base prediction.

Critical weaknesses
  1. The design choices are not clearly explained. The method section contains many equations, but some quantities and operators are not clearly explained when they first appear. For example, the meaning of (D_m) in Eq. (8) is only explicitly stated in the appendix. The paper does not clearly connect the problem being addressed. It remains unclear why this combination is suitable for the proposed decision process. Even I can follow the equations, I find it difficult to understand why the full sequence of steps is needed. This makes the method hard to assess.

  2. Lack of ablation studies with retraining. The tests in Tables 2 and 6 only disable components after training. The models are not retrained. Ablations with retraining are needed to show how much each component helps.

Questions for authors

Could you provide ablation results with retraining to show the benefit of each main component?

Could you explain what problem each main step solves and why the chosen operations are needed? Please also define the symbols clearly and explain how the steps connect.

1 comment
AuthorsOpenReview ↗

Response to Reviewer xe73: Method Clarification and Retrained Ablations

We thank the reviewer for recognizing the motivation and empirical improvements of EvidenceSAM. We have revised the method presentation and added independently trained ablations on both MCubeS and DeLiVER to address the concerns about design rationale and component contributions.

  1. Method rationale and clarity

We have revised Sections 3.1–3.4 to clarify the role of each component and the connections between them. CMF constructs shared features while retaining source-specific representations. These representations allow RDEA to generate source-associated semantic alternatives and evaluate their corrective and degradation evidence relative to the base prediction. RDEA selectively aggregates eligible experts, with the base prediction available as a fallback. CGSR then uses class-conditioned evidence and stored context to refine features and predictions, addressing semantic differences that a single source-level decision cannot fully capture.

We have also clarified the notation where the quantities first appear. In particular, D_m is now defined before Eq. (8) as the class-complete residual tensor produced by each source expert. The construction connecting the RDEA aggregated posterior to the CGSR reference prediction is explained in Section 3.1 and Appendix 6.3. Figure 2 has also been revised to improve the presentation of the architecture.

  1. Retrained ablation studies

We have added separately trained component configurations on both datasets, as reported in Table 2. On MCubeS, simple-fusion SAM2 achieves 49.24% mIoU, increasing to 51.81% with CMF, 54.57% with RDEA, and 55.46% with the complete model. On DeLiVER, the corresponding results are 61.81%, 63.84%, 66.35%, and 67.71%.

These results show that CMF improves mIoU by 2.57/2.03 points, adding the source experts and RDEA provides a further 2.76/2.51 points, and CGSR contributes another 0.89/1.36 points on MCubeS/DeLiVER.

Table 3 further reports separately trained mechanism replacements. Compared with pixel-wise spatial gating, TASR improves mIoU by 3.02/5.15 points. Compared with a direct-utility head, risk-decomposed expert aggregation improves mIoU by 3.16/5.49 points.

These new experiments distinguish the contributions of the individual components from the benefits of their specific designs. The retrained ablations are also clearly separated from the fixed-checkpoint diagnostic interventions in Appendix 9.

We hope these revisions address the concerns about the clarity, necessity, and individual contributions of the proposed stages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.