acceptodds
Withdrawn from ICLR 2027

DSS-Net: Downstream Task-Driven Multi-modal Medical Image Fusion via Decoupled Spatial-Spectral Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient multi-modal medical image fusion is vital for precision diagnostics. However, existing methods face a dilemma: models focusing solely on image fusion often prioritize visual aesthetics while neglecting semantic details required for downstream task. Meanwhile, unified fusion-downstream networks often suffer from feature entanglement and progressive semantic attenuation due to insufficient cross-task interaction. To address this, we propose DSS-Net, a novel framework that harmonizes high-fidelity fusion with downstream task performance. Central to our architecture are two specialized modules designed to reconcile representational conflicts: the Spatial Complementary Interaction (SCI) and the Global Spectral Integration (GSI). Specifically, SCI enables inter-modal complementary modulation to extract spatial visual features, while GSI executes spectral-domain aggregation to derive modality-invariant semantic representations. These decoupled features are then processed by dual-path decoders to achieve high-fidelity fusion and superior downstream utility. By decoupling representations into vision-driven spatial enhancement and semantic-driven spectral alignment, DSS-Net preserves intricate anatomical details while boosting downstream effectiveness. Extensive experiments demonstrate state-of-the-art performance in fusion quality and downstream task accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Image Fusion, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bXXYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DSS-Net is a joint framework for multimodal medical image fusion and downstream segmentation. It uses a Spatial Complementary Interaction module to model cross-modal spatial information and a Global Spectral Integration module to integrate features in the frequency domain. The resulting spatial and semantic features are processed by separate fusion and segmentation decoders. Experiments on BraTS 2020, 2021, and 2022 evaluate both image fusion quality and tumor segmentation performance, with additional ablation and efficiency analyses.

Critical strengths

DSS-Net gives a coherent joint design for image fusion and segmentation. The SCI and GSI modules separate spatial detail modeling from spectral semantic integration, while the two decoders use these features for different downstream goals.

Critical weaknesses
  1. The paper uses segmentation performance to support the downstream value of the fused representation, but the segmentation branch directly uses the shared semantic features. The final fused image itself is not tested with an independent downstream model.
  2. The evaluation is limited to T1/T2 brain MRI from the BraTS datasets and uses 2D axial slices. The generalization to other modality pairs or full 3D settings is unclear.
Questions for authors
  1. In the downstream segmentation comparison, is a common segmentation model applied to the fused images from all methods, or does each method use its own segmentation branch?
  2. Since the segmentation decoder operates on GSI features instead of the fused image, what evidence supports attributing the segmentation gains to the fusion output itself?
Reviewer_54y3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DSS-Net, a joint framework for multimodal medical image fusion and tumor segmentation. The architecture combines a Spatial Complementary Interaction (SCI) module based on state-space modeling with a Global Spectral Integration (GSI) module that exchanges Fourier amplitude information across modalities while retaining their respective phases. Separate decoders reconstruct the fused image and predict segmentation masks from shared intermediate representations. Experiments on T1/T2 axial slices from BraTS 2020–2022 report competitive fusion and segmentation performance.

Critical strengths
  • Relevant problem formulation. The paper addresses the gap between visually appealing fusion and task-relevant representations.
  • Encouraging component-level results. Table 2 shows that combining SCI and GSI improves mIoU from 76.85% to 84.82% and SSIM from 0.72 to 0.82 relative to the baseline.
Critical weaknesses
  • Converting 3D volumes to 2D slices discards volumetric context, which is detrimental for realistic 3D tumor boundaries; only brain MRI T1/T2 is covered, so cross-modality generalization is unverified.
  • The segmentation decoder consumes only GSI features, never the fused image; gains reflect joint multi-task learning, not fusion-driven causation. Missing same-backbone segmentation-only control (no fusion branch/loss) leaves the central attribution unsupported.
  • SCI overlaps ICFusion and ClinicalFMamba, which are uncited; GSI (Fourier amplitude exchange) is established (HFF-Net, Y-Net, FreMIM). Differentiating axes unarticulated.
  • Table 2 ablates only whole modules; the claimed key designs(Discrepancy Elimination (Eq. 3), Spectral Self-Enhancement (Eq. 5), entanglement-avoidance) are never individually ablated or visualized.
  • All experiments use only brain MRI T1/T2, one organ, one dataset family (BraTS versions share cases). Generalization/clinical-applicability claims exceed evidence.
Questions for authors
  • Please specify the exact rule for selecting "representative axial slices" (tumor-only? how many per volume?), the train/validation/test split, and the seeds. Please rerun on the official 3D split or on all slices to demonstrate there is no selection bias.
  • What are the patient counts and patient-level splits for each dataset? How were representative slices selected, were tumor-free slices included, and was cross-year patient overlap checked?
  • Can you provide a same-backbone, segmentation-loss-only control (no fusion branch, no fusion loss) to prove the downstream gain comes from decoupling/fusion rather than the architecture? Also, please provide quantitative or visual evidence that decoupling actually prevents feature entanglement.
  • Do you have results on other modality pairs/datasets (PET-MRI, CT-MRI) or with FLAIR/T1CE to support "clinical applicability"?
  • How are “decoupling” and “modality invariance” operationally defined? What existing evidence supports these properties, and why should Equation (3) identify conflicting information?
Reviewer_wxKmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose DSS-Net, which is a medical image segmentation task-driven medical image fusion model. The main goal is to improve both image quality and downstream segmentation performance. The method includes two main modules: Spatial Complementary Interaction (SCI), which uses Mamba to combine spatial features from different MRI images, and Global Spectral Integration (GSI), which uses Fourier transforms to combine information in the frequency domain. The model then uses two separate decoders for image fusion and segmentation. Experiments on BraTS 2020, 2021, and 2022 show improvements over several existing methods.

Critical strengths
  1. Clear motivation. The paper addresses a useful problem: improving image fusion while also supporting downstream medical tasks. The authors explain why image quality alone may not be enough for medical applications.
  2. Reasonable model design. The SCI and GSI modules are designed to capture different types of information. The use of separate decoders for fusion and segmentation is also reasonable.
  3. Good experimental results. The method reports strong results on three BraTS benchmarks. The ablation study also shows that both SCI and GSI help improve performance, and combining them gives further gains.
Critical weaknesses
  1. Limited novelty. The main method is largely based on existing techniques, including Mamba, Fourier transforms, attention, and multi-task learning. The combination is reasonable, but the paper does not clearly explain what is new beyond combining these modules. It seems like a combination of existing architecture and methods.
  2. Weak support for the main claims. The paper claims that SCI and GSI can separate spatial and semantic information and reduce conflicts between fusion and segmentation. However, there is no direct experiment showing that these conflicts exist or that the proposed modules solve them. In particular, it is unclear why working in the frequency domain should lead to better semantic understanding.
  3. Missing important baselines. The paper does not compare against a strong segmentation model that directly uses the original MRI images without image fusion. This makes it difficult to tell whether the fusion process is truly needed for better segmentation. More comparisons with simpler multi-task models would also help show the value of the proposed design.
  4. Limited experimental analysis. The experiments mainly focus on the BraTS datasets, which are closely related. The paper does not clearly explain the patient-level data splits or how the 2D slices are selected. It also lacks results across multiple random seeds and tests on more diverse medical imaging data. These details are important for judging how reliable and general the results are.
  5. Unclear evidence for task decoupling. Although the paper presents DSS-Net as a task-decoupled framework, both tasks still share most of the feature extraction network. Using separate decoders alone does not prove that the two tasks learn independent features. More analysis is needed to support this key claim. Based on the paper, SCI and GSI still form the shared backbone network, and the gradients for both tasks are still propagated through shared features.
Questions for authors

Please see Weaknesses section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.