- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces LATFUSE, a multimodal fusion framework based on a Boolean lattice. The authors formalize the fusion process as a vector-valued set function defined over subsets of modalities, employing Möbius inversion to decompose subset states into interaction components of varying orders. Building on this, LATFUSE utilizes shared, context-conditioned operations to expand unimodal representations, constructing representations for each subset of modalities layer by layer; final predictions are then generated through order-based aggregation and a residual readout mechanism. The training objective incorporates task-specific supervision, subset-level supervision, edge-gating calibration, and the decorrelation of interaction components. The paper demonstrates strong performance across several benchmarks. Overall, it proposes a fusion architecture with a well-defined mathematical structure and provides an extensive set of experimental results.
Critical strengths
-
The authors propose a structured fusion architecture that accounts for the context-dependent contributions of individual modalities. The paper unifies subset states, unimodal extensions over covering edges, and Möbius components within a Boolean lattice, providing a consistent representation for both local context conditioning and global interaction decomposition. The recursive construction ensures that each subset state depends solely on the modalities within that subset; consequently, the same architecture can perform subset-based inference even when certain modalities are missing.
-
The method achieves strong experimental results across multiple tasks, supported by targeted ablation analyses. LATFUSE demonstrates superior performance on four datasets, which encompass a diverse range of modality combinations and task types. Removing either the context conditioning or the intermediate subset states leads to a significant drop in performance, confirming the practical efficacy of the complete subset-construction mechanism.
Critical weaknesses
-
The unique contribution of higher-order interaction modeling has not yet been sufficiently substantiated. The primary novelty of the proposed method lies in the specific architecture of LATFUSE; however, the current ablation studies do not fully rule out the possibility that the observed performance gains stem simply from an increased number of intermediate states, computational paths, or supervision applied to subsets of components. Furthermore, the synthetic experiments rely on fANOVA to analyze incremental gains at the readout stage, rather than directly demonstrating a correspondence between the Möbius components and the interaction terms inherent in the actual tasks. The authors should provide an explanation or conduct analytical experiments to address this issue.
-
The scope of the experiments concerning robustness to missing modalities is limited. All evaluations involving missing modalities are restricted to the removal of a single modality; the study does not test scenarios where multiple modalities are missing simultaneously, nor does it address cases where arbitrary proportions of modal content are absent[1][2]. Consequently, while the current results support the claim that LATFUSE exhibits good robustness to missing modalities under the tested protocols, they are insufficient to establish a general advantage across a broader range of arbitrary missing-modality conditions.
References: [1] Towards Robust Multimodal Sentiment Analysis with Incomplete Data, NeurIPS 2024 [2] Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data, ACL 2025
Questions for authors
-
The unique contribution of higher-order interaction modeling has not yet been sufficiently substantiated. The primary novelty of the proposed method lies in the specific architecture of LATFUSE; however, the current ablation studies do not fully rule out the possibility that the observed performance gains stem simply from an increased number of intermediate states, computational paths, or supervision applied to subsets of components. Furthermore, the synthetic experiments rely on fANOVA to analyze incremental gains at the readout stage, rather than directly demonstrating a correspondence between the Möbius components and the interaction terms inherent in the actual tasks. The authors should provide an explanation or conduct analytical experiments to address this issue.
-
The scope of the experiments concerning robustness to missing modalities is limited. All evaluations involving missing modalities are restricted to the removal of a single modality; the study does not test scenarios where multiple modalities are missing simultaneously, nor does it address cases where arbitrary proportions of modal content are absent[1][2]. Consequently, while the current results support the claim that LATFUSE exhibits good robustness to missing modalities under the tested protocols, they are insufficient to establish a general advantage across a broader range of arbitrary missing-modality conditions.
-
How can one distinguish the gains from LATFUSE’s interactive modeling from those derived from increased computation and supervision? Table 6 shows that removing the intermediate subset states leads to a significant drop in performance; however, this ablation also removes the associated computation, supervision, and reuse pathways.
-
The baseline lacks the latest papers from 2026. The authors should incorporate relevant research published in top-conferences such as ICML 2026, CVPR 2026, and ICLR 2026.
8 comments
Rebuttal by Authors (Part 2/5)
Question 1 (sequel)
Synthetic Experiments and Interactions.
Furthermore, ... to address this issue.
The synthetic experiments primarily aim to verify LatFuse's ability to organise higher-order interactions in a task-adaptive manner. We do not claim that the Möbius components correspond exactly to interactions of the same order in the true target. Analysing this correspondence generally requires partial information decomposition (PID) to quantify unique, synergistic and redundant information in multimodal interactions [1][2]. Establishing an exact correspondence between individual Möbius components and the true target interactions remains a separate question.
To further explore the relationship between LatFuse's subset representations and multimodal interaction information, we used PID to analyse readouts from real CMU-MOSI data. Below, we present the text transcripts of three representative samples, their subset predictions on the sentiment scale , and four-component PID decompositions for T--A and T--V in nats:
I DIDNT REALLY I MEAN TRASURE PLANET WAS OKIF YOU LIKE EXPLOSIONS REALLY RIDICULOUS ACTION SEQUENCES MALODRAMATIC PLOT AND ACTING THIS IS THE RIGHT MOVIE TO SEEUM I DONT THINK THAT THIS OVE GOT AS MUCH BUZZ AND AS MUCH UM MEDIA ATTENTION AND UM WORD OF MOUTH YOU KNOW SPREADING AROUND AND STUFF
| Sample | Ground Truth Label | T | A | V | T+A | T+V | T+A+V |
|---|---|---|---|---|---|---|---|
| 1 | -1.0 | +0.083 | +0.560 | −2.179 | +0.089 | −1.031 | −0.976 |
| 2 | -0.8 | +0.268 | −0.959 | +0.278 | −0.181 | +0.408 | −0.084 |
| 3 | -0.8 | +0.015 | −0.130 | −0.546 | −0.026 | −0.152 | −0.036 |
| Sample | PID Pair | Redundant | T Unique | A/V Unique | Synergistic |
|---|---|---|---|---|---|
| 1 | T--A | +0.0499 | ≈0 | −0.2159 | −0.4317 |
| T--V | +0.0499 | ≈0 | +0.2111 | +0.2109 | |
| 2 | T--A | −0.1324 | ≈0 | +0.7367 | −0.3333 |
| T--V | −0.1324 | ≈0 | +0.2287 | −0.3421 | |
| 3 | T--A | −0.0550 | ≈0 | +0.3561 | −0.5168 |
| T--V | +0.1566 | −0.2116 | ≈0 | −0.0681 |
The results show that subset predictions reveal the context dependence of modality contributions. In sample 1, text alone is nearly neutral (+0.083), audio is positive (+0.560), and video is strongly negative (−2.179). Adding text to audio lowers the prediction, whereas adding text to video raises it. Text therefore moderates both overly positive and overly negative judgements, yielding a final prediction of −0.976, which closely matches the ground-truth label of −1.0. PID also reveals positive synergistic information for T--V (+0.2109 nat), whereas the T--A synergy term is −0.4317 nat. This shows that text can adjust the sentiment intensity conveyed by other modalities according to the non-verbal context, with a sentiment contribution that varies in direction.
Interaction analysis can also reveal modality conflicts and their effects on decisions. In sample 2, T and V predict +0.268 and +0.278; while audio predicts −0.959, which is closer to the ground-truth label of −0.8. After adding T, T+A remains negative (−0.181), while T+V becomes more positive (+0.408). This shows that negative audio cues can change the sentiment judgement from T, while visual information reinforces its positive tendency. Although the local synergy terms for T--A and T--V are −0.3333 and −0.3421 nat, respectively, the two combinations produce predictions with opposite polarities. PID and subset predictions thus provide complementary explanations: the former characterises interactions in task information, while the latter reveals how these interactions change model decisions.
These analyses can also identify specific failure mechanisms in multimodal prediction. In sample 3, the text prediction is nearly neutral (+0.015), while the video prediction is −0.546, which agrees much more closely with the negative ground-truth sentiment (−0.8). Adding text makes the T+V prediction less negative (−0.152), and the full model moves further towards neutrality (−0.036). PID shows that T--A contains +0.3561 nat of information unique to audio and a synergy term of −0.5168 nat, whereas T--V combines positive redundant information (+0.1566 nat), a negative local text-unique term (−0.2116 nat) and a synergy term of −0.0681 nat. This reveals clear differences in the information relationships involving text across modality combinations and pinpoints the model's excessive suppression of negative non-verbal evidence.
These samples show that LatFuse's lattice structure also offers a way to analyse how decisions form, alongside its role in modelling interactions. Combining the Boolean lattice with PID allows us to trace predictions to specific modality relationships and distinguish mechanisms such as effective cross-modal correction, evidence suppression and single-modality dominance. These observations complement the synthetic experiments by demonstrating context-dependent contributions of modalities in real samples.
Rebuttal by Authors (Part 3/5)
Question 2 (Weakness 2)
The scope of the experiments concerning robustness to missing modalities is limited. All evaluations involving missing modalities are restricted to the removal of a single modality; the study does not test scenarios where multiple modalities are missing simultaneously, nor does it address cases where arbitrary proportions of modal content are absent[1][2]. Consequently, while the current results support the claim that LATFUSE exhibits good robustness to missing modalities under the tested protocols, they are insufficient to establish a general advantage across a broader range of arbitrary missing-modality conditions.
LatFuse's core contribution is to model the interactions required by multimodal tasks through explicit local semantic conditioning. We did not design it specifically for missing modalities. Because LatFuse computes on a lattice, it can perform inference directly on a sublattice when modalities are missing at inference time. The missing-modality experiments assess the robustness of this sublattice inference. Our existing single-modality removal evaluations show that LatFuse's sublattices adapt well to missing modalities in this setting, without additional interpolation or data generation.
In a three-modality setting, removing two modalities leaves only one, reducing the task to unimodal prediction and precluding the evaluation of cross-modal fusion and higher-order interactions. With more modalities, LatFuse retains the same inference mechanism and operator-reuse scheme across observed sublattices of any size. Removing a single modality therefore tests sublattice inference while preserving interactions among the remaining modalities, including pairwise interactions in the three-modality setting.
To further investigate LatFuse under more general missing-modality conditions, we also tested:
- Modality-level missing content at arbitrary rates on TCGA-BRCA;
- Single-modality removal on TCGA-BRCA after training baseline methods with modality dropout;
- Sample-level missing modality content at arbitrary rates on CMU-MOSI.
We have added the full results to Section 5.4 of the revised manuscript and to Appendix F.
Modality-Level Missingness on TCGA-BRCA.
We have added missing-modality tests at arbitrary rates on TCGA-BRCA. After standard training across all modalities, we set the missing rate for each modality to 0.1 -- 1.0 at test time. The table below reports ACC averaged across the missing rates.
| Method | Full | −mRNA | −miRNA | −DNA |
|---|---|---|---|---|
| MM-Dynamics | 86.94 | 60.13 (−26.81) | 83.48 (−3.46) | 84.43 (−2.51) |
| MOHGCN | 87.91 | 79.84 (−8.07) | 86.19 (−1.72) | 85.73 (−2.18) |
| LatFuse (ours) | 89.39 | 83.09 (−6.30) | 88.79 (−0.60) | 87.72 (−1.67) |
The results show that LatFuse also offers a robustness advantage over the baseline methods on this task across the tested missing rates.
Single-Modality Removal with Modality Dropout for TCGA-BRCA Baselines.
We trained several baselines on TCGA-BRCA with different modality dropout rates. The table below reports ACC under the Full and heterogeneous missing-modality conditions.
| Method | Dropout | Full | Hetero. |
|---|---|---|---|
| MM-Dynamics | 0.0 | 86.94 | 72.96 (−13.98) |
| 0.2 | 86.97 | 82.58 (−4.39) | |
| 0.5 | 86.93 | 82.88 (−4.05) | |
| MOHGCN | 0.0 | 87.91 | 82.50 (−5.41) |
| 0.2 | 83.73 | 80.69 (−3.04) | |
| 0.5 | 84.35 | 81.15 (−3.20) | |
| DMIB | 0.0 | 86.22 | 81.82 (−4.40) |
| 0.2 | 85.48 | 81.65 (−3.83) | |
| 0.5 | 85.23 | 81.96 (−3.27) | |
| CLCLSA | 0.0 | 86.09 | 82.00 (−4.09) |
| 0.2 | 85.17 | 82.86 (−2.31) | |
| 0.5 | 83.88 | 82.31 (−1.57) | |
| MOSGAT | 0.0 | 87.17 | 82.43 (−4.74) |
| 0.2 | 86.54 | 83.92 (−2.62) | |
| 0.5 | 86.39 | 83.63 (−2.76) | |
| LatFuse (ours) | 0.0 | 89.39 | 85.39 (−4.00) |
The results show that:
- Modality dropout markedly improves the missing-modality performance of MM-Dynamics. However, MM-Dynamics still suffers a larger performance drop than LatFuse in the condition with the largest change in the metric.
- Modality dropout has a detrimental effect on MOHGCN and CLCLSA: it substantially reduces their performance with all modalities, even though it reduces the absolute drop caused by missing modalities.
- Modality dropout can improve the missing-modality performance of DMIB and MOSGAT, but selecting an appropriate dropout rate introduces an additional tuning difficulty. LatFuse avoids this difficulty by supervising all subsets.
Rebuttal by Authors (Part 1/5)
We thank the reviewer for the thorough evaluation and insightful questions.
Question 1 (Weakness 1)
Sources of the Performance Gains.
In our view, the performance gains come from context-conditioned modelling. The lattice structure is the most natural technical choice for implementing this modelling, but the computations on the lattice do not, by themselves, explain the gains. Our existing A1 ablation shows that removing context conditioning from the computational flow while retaining the other design choices substantially reduces performance.
To distinguish the contribution of context conditioning from that of additional supervision, we conducted an ablation that removes Lsubset (L3). Together with the existing A1 ablation, this gives a 2x2 ablation grid:
The table below shows the experimental results. We have added the full results to Section 5.3 of the revised manuscript.
Even with Lsubset removed, context conditioning improves the reported metrics by 2.81 points on TCGA-BRCA, 0.62 points on MLOmics-Pan-cancer and 1.93 points on CMU-MOSI (A1 + L3 → L3). On MLOmics-Pan-cancer and CMU-MOSI, context construction and subset supervision have a stronger joint effect. Comparing A1 + L3 → A1 with L3 → Full shows that adding Lsubset alone yields smaller gains without context conditioning, for example +0.76 on Pan-cancer. With context conditioning, the gain increases substantially to +2.81 on Pan-cancer. The gain from subset supervision is larger when the model uses predecessor states as contexts, consistent with its role in training task-relevant intermediate representations.
Specifically, during edge refinement, the predecessor state hV directly serves as the initial query vector in cross-attention to retrieve complementary marginal information from the newly added modality xm. Without Lsubset, the lower-level intermediate state hV has a long gradient path to the task representation. Direct training may therefore fail to ensure that this state encodes the complete task semantics of subset V. The role of Lsubset is thus to ensure that every predecessor state hV provides a semantically adequate context.
LatFuse's contribution is to represent modality subset states as reusable contexts for fusion. We consider context-conditioned modelling a local operation with clear semantics, enabling models to learn more readily across different tasks. Learning higher-order interactions directly at the global level requires a model capable of fitting joint representations across many combinations of modalities. LatFuse decomposes this complex problem into local objectives with consistent semantics and organises them on the Boolean lattice. Both the lattice structure and subset supervision serve the core design of context-conditioned modelling. Subset supervision also adds only limited overhead relative to the core computational flow.