acceptodds
Withdrawn from ICLR 2027

Can Hybrid-Space Representation Learning Advance Multimodal Fusion?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal fusion—joint learning of representations from heterogeneous data has shown significant potential in healthcare by integrating imaging, text, and omics for comprehensive analysis. However, prior methods face two major limitations: (1) inadequate modeling of complex hierarchical structural dependencies at high computational cost, which constrains the performance–efficiency trade-off ; and (2) limited robustness to missing modalities. These limitations restrict their applicability in resource-constrained medical AI. To address these challenges, we introduce Hybrid Space Attention Refiner (HySAR), a plug-and-play cross-modal refiner that jointly exploits Euclidean and non-Euclidean manifolds to capture complex hierarchical structural dependencies for learning robust shared representations. This, in turn, improves multimodal fusion performance and missing-modality robustness, while HySAR’s channel-wise geometric refinement and complementary multi-scale information learning further reduce computational cost. Across 10 multimodal benchmarks, HySAR-integrated learners achieve ≈ 4.8-point on average improvements over baselines, up to ≈ 84.1% parameter and ≈ 88.0% FLOP reductions, and up to ≈ 8.7-point gains in missing-modality robustness, enabling efficient and robust clinical prediction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, multimodal fusion, Medical data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7VC3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Space Attention Refiner (HySAR), a plug-and-play cross-modal refinement framework designed to improve multimodal fusion performance, computational efficiency, and robustness to missing modalities. The framework consists of three components: Multi-scale Channel Information Learning Attention (MILA), which extracts multi-scale Euclidean spatial-channel information; Dual Hyperbolic Fusion Attention (DUHA), which combines Poincaré-aware and Proper-Velocity manifold representations to capture hierarchical dependencies; and Hybrid-Space Multi-Cross Fusion (HFuse), which integrates complementary geometric information across modalities. The authors evaluate HySAR on ten multimodal benchmarks covering medical survival prediction, medical image classification, semantic segmentation, and image-text classification. Experiments across multiple fusion architectures and backbone families report consistent performance improvements over existing refiners, together with favorable computational trade-offs and improved performance under missing-modality settings.

Critical strengths

Strength 1: The proposed MILA module uses depth-wise, group-wise, and dilated convolutions to capture spatial information at different scales. It combines these features through channel attention, providing an efficient way to preserve fine-grained information with relatively low computational cost.

Strength 2: The proposed DUHA module combines Poincaré and Proper-Velocity representations to model hierarchical relationships. HFuse further integrates these hyperbolic features with Euclidean information across modalities. This design allows HySAR to benefit from both geometric spaces while remaining compatible with different multimodal fusion architectures and supporting missing-modality scenarios.

Strength 3: The paper evaluates HySAR on ten multimodal benchmarks using different fusion architectures and backbones. The results show consistent improvements over existing refiners in prediction performance, with favorable computational efficiency and missing-modality robustness.

Critical weaknesses

Weakness 1: The IMF formulation appears to cancel out the MGHC and MDDHC outputs, yet Table 3(a) reports gains from these branches. This makes it unclear whether the formulation matches the implementation used in the experiments.

Weakness 2: All evaluated benchmarks use only two modalities, so the benefit of HFuse’s softmax-based weighting across complementary modalities remains untested. The reported improvements therefore do not establish the contribution of this particular mechanism.

Weakness 3: When only one modality is available, HFuse uses the DUHA response but leaves out the corresponding MILA response. The paper provides limited justification for this choice and does not compare it with retaining both responses during single-modality inference.

Questions for authors

Question 1: In Section 2.1, the IMF operation \mathcal I(a,b,c)=(a+b+c)+(a-b-c) simplifies to 2a2a, so the MGHC and MDDHC outputs would cancel out as written. However, Table 3(a) reports gains from these branches. Could the authors provide the exact IMF formulation used in the implementation and explain how these branches contribute to the reported improvements?

Question 2: According to Table 6, all ten benchmarks contain only two modalities. In this setting, the softmax in Equation (13) operates over only one complementary modality, making its weight always equal to one. Could the authors clarify how the adaptive weighting mechanism contributes to the reported improvements? Would experiments involving three or more modalities help demonstrate its effectiveness compared with simpler fusion strategies?

Question 3: Algorithm 1 computes both MILA and DUHA for each available modality, but Algorithm 4 only uses DUHA when a single modality remains. Could the authors provide an ablation study comparing the current DUHA-only design with a variant that combines both responses under single-modality inference?

Reviewer_Fk9oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a plug-and-play cross-modal feature refinement module called HySAR, designed to replace the existing refiner in multi-modal fusion frameworks. HySAR consists of three core components: MILA, DUHA, and HFuse. In the experiments, the proposed module replaces the original refiner within various host frameworks. In the medical domain, the approach was evaluated using six host frameworks—MOTCAT, DRIFA-Net, HyPCA-Net, HEALNet, MAIL, and CURE—paired with four backbone networks (ResNet18, ViT-Tiny, Mamba-Tiny, and Inception-V3). Evaluations were conducted on four TCGA survival analysis cohorts, as well as the HAM10000 and SIPaKMeD datasets. Furthermore, experiments addressing missing modalities were performed on the BLCA and KIRP cohorts, and an ablation study of the individual components was carried out using MAIL as the host framework to assess the model's robustness to missing modalities and the specific contribution of each component.

Critical strengths

In these experiments, only the refiner is replaced while the rest of the architecture remains unchanged. The evaluation spans 6 host frameworks and 4 types of backbones across three distinct tasks—survival analysis, classification, and segmentation. We report results based on 5 random seeds, alongside metrics for parameter count, FLOPs, and inference latency. This controlled experimental design allows us to attribute performance variations directly to the refiner itself. Within the refiner, missing modalities are handled via masking and re-normalization. This approach requires no modifications to the host framework, thereby reducing the cost of integration.

Critical weaknesses

There is a fundamental contradiction in the description of the methodology, which undermines the claims regarding the contributions of the proposed "hybrid geometry." In Equation 5, the expression I(a,b,c)=(a+b+c)+(a−b−c)=2aI(a,b,c)=(a+b+c)+(a-b-c)=2a implies that the components associated with MGHC, MDDHC, as well as the GAP and GMN terms in Equation 6, have no impact on the forward pass. This contradicts the assertion in Table 3(a) that these branches contribute to an improvement of 1–3 C-index points. Furthermore, since all attention mechanisms are derived from channel vectors obtained via global pooling—essentially amounting to channel reweighting—the method lacks explicit spatial selectivity. Consequently, the claim that it "preserves fine-grained, multi-scale spatial-channel context" is unsubstantiated. Moreover, the geometric necessity of the hyperbolic component remains unproven: the element-wise multiplication in Equation 10 is not an intrinsic manifold operation, and the exp–FC–log structure of PVRL has not been shown to offer any advantage over standard non-linear transformations. Additionally, the Poincaré and PV models are merely isometric representations of the same hyperbolic space. The paper fails to verify whether the features actually possess a hyperbolic structure, nor does it provide a comparison against a Euclidean baseline with a matched number of parameters. There are several questionable aspects regarding data partitioning, evaluation protocols, and the reported numerical results, rendering the experimental reliability insufficient to support the paper's conclusions. For the HAM10000 dataset, it is unclear whether the data was partitioned by lesion (grouping samples from the same lesion together) or whether the dx_type field—which could lead to label leakage—was utilized. Similarly, for the SIPaKMeD dataset, it is not specified whether the data was partitioned based on the original image clusters; furthermore, given that this dataset does not inherently contain clinical data, the source of the "second modality" remains unexplained. The approaches of compressing Whole Slide Images (WSIs) into pseudo-images and rearranging omics vectors into image formats also deviate from the original configurations established in works like MOTCAT, thereby limiting the comparability of these results with existing research.

Questions for authors

1.Are the HAM10000 data partitioned by lesion ID, and is the SIPaKMeD dataset partitioned by source cluster images? Please list all table fields used and specify whether dx_type is included. 2.How does the performance change when PARL and PVRL are replaced with Euclidean MLPs or SE-gating mechanisms that have an equivalent number of parameters? Please provide a comparison with SE, ECA, and MMTM using the same replacement protocol to disentangle the contributions of "hybrid geometry" from those of the channel-gating mechanism itself. 3.Do the main conclusions still hold when using 5-fold cross-validation on the TCGA cohort and performing paired significance tests? 4.Was "modality dropout" employed during training for the missing-modality experiments? In Algorithm 4, the single-modality fallback path bypasses MILA—which contradicts the description in the main text—and the comment "must match the released code" remains present. 5.Beyond the inclusion of PVRL, what are the substantive differences between HySAR and CURE/HySAM in terms of their core mechanisms and design philosophy?

Reviewer_24SCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an approach for modulating the cross-modal attention over learnt representations for better multimodal fusion. More specifically, the authors introduce HySAR, which updates the learnt per-modality representations by combining Euclidean and non-Euclidean attention mechanisms. The proposed HySAR combines three components: (i) MILA which uses multi-scale convolutions and channel attention to learn Euclidean representations; (ii) DUHA that combines two hyperbolic formulations (Poincare and Proper-Velocity) to model the hierarchical dependencies; (iii) and finally HFuse, which integrates the resulting representations across modalities. The authors highlight that HySAR is modular and applicable to different multimodal architectures. The presented experiments on 10 datasets (6 medical and 4 non-medical) comparing HySAR to several baseline methods show that it can lead to performance improvements while being robust to missing data. Finally, a small ablation study points to the fact that all three different components (and in particular the MILA) contribute to these performance improvements.

Critical strengths
  • The motivation of the paper is clear and relevant. Multimodal learning, especially when it considers non-VL modalities, is an important topic. The focus on practical trade-offs in terms of computational performance and robustness to missing modalities vs. predictive performance is also important.

  • The modular aspect of HySAR is good, as it can allow integration with different multimodal architectures.

Critical weaknesses
  • W1. Overall, the experimental setup is unclear. The authors report results of baselines, but for comparability, these baselines have to be re-implemented. There are many details missing on how the experiments were set up and the baselines benchmarked. For instance, HEALNet has a fusion layer that gets constantly updated; MOTCAT has a fusion operator based on optimal transport; MANGO uses a normalizing flow and therefore has an invertible cross-attention, etc. Therefore, it's unclear how such mechanisms can be adapted to HySAR, which components are replaced, as they may likely regress parts of their originally intended capabilities. Therefore is difficult to assess the validity of the reported improvements.

  • W2. The novelty of the work is also rather narrow. Specifically, the work is closely related to [1] as it both focus on combining Euclidean/non-Euclidean representations for multimodal fusion. While [1] frames the work around image modalities, HySAR is claimed to work on heterogeneous modalities (see W3). The other closely related work, CURE, which also implements hyperbolic representations and is benchmarked, it is a bit unclear to what extent the non-Euclidian representations in HySAR differ from HySAM (see Q2).

  • W3. HySAR (and esp. MILA) assumes representations from spatial input (WxHxCv). While these are fine for image modalities, it is unclear why and how they are appropriate for tabular modalities (such as -omics). Moreover, the multi-granularity arguments are less convincing in this setting, as tabular data do not have inherent spatial/arrangement structure or feature ordering. Therefore, it's really unclear how these representations will affect the downstream performance.

  • W4. There are many inconsistencies between the manuscript and the provided code implementation in how some of the proposed mechanisms are reported and implemented. Moreover, there are many details missing for reproducing the results (like baselines, data pre-processing steps, etc.)

[1] Xiao et al "A Hybrid Space Model for Misaligned Multi-modality Image Fusion" AAAI 2026.

Questions for authors
  • See weaknesses

  • I would like a clarification on the discrepancies between the manuscript and the code. For instance, one of the claimed contributions of this work is the PVRL (proper-velocity) representation learning. In the provided code, however, there is no mention of this component but reference to Lorenz representations. This can be relevant for the comparison against CURE, which also implements similar hyperbolic representations.

  • How are non-image modalities mapped into the particular tensors (Appendix D)? In particular, what determines the spatial arrangement of omics features?

  • One of the claimed benefits of HEALNet is its capability to work with missing modalities (at train and inference time). However, the authors do not compare to it in the missing-modality benchmark. This needs clarification. Moreover, the authors do not evaluate the contrubutions of HFuse (and its capabilities to be aware of available modalities) against simpler alternatives, such as single modal alternatives. As such, it is unclear whether the reported improvements can be attributed to the proposed missing-modality handling or to the underlying representation learning.

  • How are ISA, DRIFA, DVCA, EMCAM, OTCA, HSA, HySAM, and CA implemented and combined with HySAR?

  • The main experimental setup (for Table 1) is a bit confusing. The manuscript suggests that the authors test "host" multimodal pipelines by replacing the different fusion components. Given that each of these has a ResNet-18 backbone, how much do the architectures actually differ at the end? Ie. What's the difference between MOTCAT w/ HySAR and HEALNet w/ HySAR (which components of the "original architectures" are preserved) ?

  • Would the authors clarify Eq 13 and the resulting A_i?

Minor comment:

  • Fig. 1 is unclear, especially the legend.
  • The "IMF" equation (page 5) reads really odd, suggesting the different responses cancel each other out.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.