acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Fusion on the Boolean Lattice: From Local Context Dependence to Global Higher-Order Interactions

Abstract

In multimodal fusion, a modality's marginal contribution can depend on the combination of modalities already present, which defines its context. Viewing fusion as a vector-valued set function over modality subsets shows that this context dependence is equivalent to higher-order interactions of order two or above. We therefore develop a formulation of multimodal fusion on the Boolean lattice of modality subsets, capturing global higher-order interactions by structuring local context dependence. Building on this formulation, we introduce LatFuse, which constructs fused representations for each subset of modalities in a bottom-up manner on the Boolean lattice, using single-modality extensions as reusable units for fusion. Across four benchmarks comprising three or four modalities and spanning multiomics classification, sentiment analysis, and semantic segmentation, LatFuse outperforms the strongest baselines on all task metrics. By constructing every subset state, LatFuse can infer directly on sublattices and is more robust than the baselines under most missing-modality conditions. On synthetic tasks with known interaction structures, LatFuse learns interactions that largely match the true interaction orders, supporting analysis of higher-order interactions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Fusion, Multimodal Learning, Boolean Lattice

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EcKBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces LATFUSE, a multimodal fusion framework based on a Boolean lattice. The authors formalize the fusion process as a vector-valued set function defined over subsets of modalities, employing Möbius inversion to decompose subset states into interaction components of varying orders. Building on this, LATFUSE utilizes shared, context-conditioned operations to expand unimodal representations, constructing representations for each subset of modalities layer by layer; final predictions are then generated through order-based aggregation and a residual readout mechanism. The training objective incorporates task-specific supervision, subset-level supervision, edge-gating calibration, and the decorrelation of interaction components. The paper demonstrates strong performance across several benchmarks. Overall, it proposes a fusion architecture with a well-defined mathematical structure and provides an extensive set of experimental results.

Critical strengths
  1. The authors propose a structured fusion architecture that accounts for the context-dependent contributions of individual modalities. The paper unifies subset states, unimodal extensions over covering edges, and Möbius components within a Boolean lattice, providing a consistent representation for both local context conditioning and global interaction decomposition. The recursive construction ensures that each subset state depends solely on the modalities within that subset; consequently, the same architecture can perform subset-based inference even when certain modalities are missing.

  2. The method achieves strong experimental results across multiple tasks, supported by targeted ablation analyses. LATFUSE demonstrates superior performance on four datasets, which encompass a diverse range of modality combinations and task types. Removing either the context conditioning or the intermediate subset states leads to a significant drop in performance, confirming the practical efficacy of the complete subset-construction mechanism.

Critical weaknesses
  1. The unique contribution of higher-order interaction modeling has not yet been sufficiently substantiated. The primary novelty of the proposed method lies in the specific architecture of LATFUSE; however, the current ablation studies do not fully rule out the possibility that the observed performance gains stem simply from an increased number of intermediate states, computational paths, or supervision applied to subsets of components. Furthermore, the synthetic experiments rely on fANOVA to analyze incremental gains at the readout stage, rather than directly demonstrating a correspondence between the Möbius components and the interaction terms inherent in the actual tasks. The authors should provide an explanation or conduct analytical experiments to address this issue.

  2. The scope of the experiments concerning robustness to missing modalities is limited. All evaluations involving missing modalities are restricted to the removal of a single modality; the study does not test scenarios where multiple modalities are missing simultaneously, nor does it address cases where arbitrary proportions of modal content are absent[1][2]. Consequently, while the current results support the claim that LATFUSE exhibits good robustness to missing modalities under the tested protocols, they are insufficient to establish a general advantage across a broader range of arbitrary missing-modality conditions.

References: [1] Towards Robust Multimodal Sentiment Analysis with Incomplete Data, NeurIPS 2024 [2] Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data, ACL 2025

Questions for authors
  1. The unique contribution of higher-order interaction modeling has not yet been sufficiently substantiated. The primary novelty of the proposed method lies in the specific architecture of LATFUSE; however, the current ablation studies do not fully rule out the possibility that the observed performance gains stem simply from an increased number of intermediate states, computational paths, or supervision applied to subsets of components. Furthermore, the synthetic experiments rely on fANOVA to analyze incremental gains at the readout stage, rather than directly demonstrating a correspondence between the Möbius components and the interaction terms inherent in the actual tasks. The authors should provide an explanation or conduct analytical experiments to address this issue.

  2. The scope of the experiments concerning robustness to missing modalities is limited. All evaluations involving missing modalities are restricted to the removal of a single modality; the study does not test scenarios where multiple modalities are missing simultaneously, nor does it address cases where arbitrary proportions of modal content are absent[1][2]. Consequently, while the current results support the claim that LATFUSE exhibits good robustness to missing modalities under the tested protocols, they are insufficient to establish a general advantage across a broader range of arbitrary missing-modality conditions.

  3. How can one distinguish the gains from LATFUSE’s interactive modeling from those derived from increased computation and supervision? Table 6 shows that removing the intermediate subset states leads to a significant drop in performance; however, this ablation also removes the associated computation, supervision, and reuse pathways.

  4. The baseline lacks the latest papers from 2026. The authors should incorporate relevant research published in top-conferences such as ICML 2026, CVPR 2026, and ICLR 2026.

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/5)

We thank the reviewer for the thorough evaluation and insightful questions.

Question 1 (Weakness 1)

The unique contribution of higher-order interaction modeling has not yet been sufficiently substantiated. The primary novelty of the proposed method lies in the specific architecture of LATFUSE; however, the current ablation studies do not fully rule out the possibility that the observed performance gains stem simply from an increased number of intermediate states, computational paths, or supervision applied to subsets of components. Furthermore, the synthetic experiments rely on fANOVA to analyze incremental gains at the readout stage, rather than directly demonstrating a correspondence between the Möbius components and the interaction terms inherent in the actual tasks. The authors should provide an explanation or conduct analytical experiments to address this issue.

Sources of the Performance Gains.

The unique contribution ... or supervision applied to subsets of components.

In our view, the performance gains come from context-conditioned modelling. The lattice structure is the most natural technical choice for implementing this modelling, but the computations on the lattice do not, by themselves, explain the gains. Our existing A1 ablation shows that removing context conditioning from the computational flow while retaining the other design choices substantially reduces performance.

To distinguish the contribution of context conditioning from that of additional supervision, we conducted an ablation that removes Lsubset\mathcal{L}_\mathrm{subset} (L3). Together with the existing A1 ablation, this gives a 2x2 ablation grid:

with context conditioningw/o context conditioning
with Lsubset\mathcal{L}_{\mathrm{subset}}FullA1
w/o Lsubset\mathcal{L}_{\mathrm{subset}}L3A1 + L3

The table below shows the experimental results. We have added the full results to Section 5.3 of the revised manuscript.

VariantTCGA-BRCA (m.F1F_1)Pan-cancer (m.F1F_1)CMU-MOSI (ACC$_7$)
Full87.0895.2650.95
A183.60 (−3.48)92.59 (−2.67)45.42 (−5.53)
L385.14 (−1.94)92.45 (−2.81)44.61 (−6.34)
A1 + L382.33 (−4.75)91.83 (−3.43)42.68 (−8.27)

Even with Lsubset\mathcal{L}{\mathrm{subset}} removed, context conditioning improves the reported metrics by 2.81 points on TCGA-BRCA, 0.62 points on MLOmics-Pan-cancer and 1.93 points on CMU-MOSI (A1 + L3 →\to L3). On MLOmics-Pan-cancer and CMU-MOSI, context construction and subset supervision have a stronger joint effect. Comparing A1 + L3 →\to A1 with L3 →\to Full shows that adding Lsubset\mathcal{L}_{\mathrm{subset}} alone yields smaller gains without context conditioning, for example +0.76 on Pan-cancer. With context conditioning, the gain increases substantially to +2.81 on Pan-cancer. The gain from subset supervision is larger when the model uses predecessor states as contexts, consistent with its role in training task-relevant intermediate representations.

Specifically, during edge refinement, the predecessor state hVh_{\mathcal{V}} directly serves as the initial query vector in cross-attention to retrieve complementary marginal information from the newly added modality xmx_m. Without Lsubset\mathcal{L}{\mathrm{subset}}, the lower-level intermediate state hVh_{\mathcal{V}} has a long gradient path to the task representation. Direct training may therefore fail to ensure that this state encodes the complete task semantics of subset V\mathcal{V}. The role of Lsubset\mathcal{L}{\mathrm{subset}} is thus to ensure that every predecessor state hVh_{\mathcal{V}} provides a semantically adequate context.

LatFuse's contribution is to represent modality subset states as reusable contexts for fusion. We consider context-conditioned modelling a local operation with clear semantics, enabling models to learn more readily across different tasks. Learning higher-order interactions directly at the global level requires a model capable of fitting joint representations across many combinations of modalities. LatFuse decomposes this complex problem into local objectives with consistent semantics and organises them on the Boolean lattice. Both the lattice structure and subset supervision serve the core design of context-conditioned modelling. Subset supervision also adds only limited overhead relative to the core computational flow.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/5)

Question 1 (sequel)

Synthetic Experiments and Interactions.

Furthermore, ... to address this issue.

The synthetic experiments primarily aim to verify LatFuse's ability to organise higher-order interactions in a task-adaptive manner. We do not claim that the Möbius components correspond exactly to interactions of the same order in the true target. Analysing this correspondence generally requires partial information decomposition (PID) to quantify unique, synergistic and redundant information in multimodal interactions [1][2]. Establishing an exact correspondence between individual Möbius components and the true target interactions remains a separate question.

To further explore the relationship between LatFuse's subset representations and multimodal interaction information, we used PID to analyse readouts from real CMU-MOSI data. Below, we present the text transcripts of three representative samples, their subset predictions on the sentiment scale [−3,3][-3, 3], and four-component PID decompositions for T--A and T--V in nats:

  1. I DIDNT REALLY I MEAN TRASURE PLANET WAS OK
  2. IF YOU LIKE EXPLOSIONS REALLY RIDICULOUS ACTION SEQUENCES MALODRAMATIC PLOT AND ACTING THIS IS THE RIGHT MOVIE TO SEE
  3. UM I DONT THINK THAT THIS OVE GOT AS MUCH BUZZ AND AS MUCH UM MEDIA ATTENTION AND UM WORD OF MOUTH YOU KNOW SPREADING AROUND AND STUFF
SampleGround Truth LabelTAVT+AT+VT+A+V
1-1.0+0.083+0.560−2.179+0.089−1.031−0.976
2-0.8+0.268−0.959+0.278−0.181+0.408−0.084
3-0.8+0.015−0.130−0.546−0.026−0.152−0.036
SamplePID PairRedundantT UniqueA/V UniqueSynergistic
1T--A+0.0499≈0−0.2159−0.4317
T--V+0.0499≈0+0.2111+0.2109
2T--A−0.1324≈0+0.7367−0.3333
T--V−0.1324≈0+0.2287−0.3421
3T--A−0.0550≈0+0.3561−0.5168
T--V+0.1566−0.2116≈0−0.0681

The results show that subset predictions reveal the context dependence of modality contributions. In sample 1, text alone is nearly neutral (+0.083), audio is positive (+0.560), and video is strongly negative (−2.179). Adding text to audio lowers the prediction, whereas adding text to video raises it. Text therefore moderates both overly positive and overly negative judgements, yielding a final prediction of −0.976, which closely matches the ground-truth label of −1.0. PID also reveals positive synergistic information for T--V (+0.2109 nat), whereas the T--A synergy term is −0.4317 nat. This shows that text can adjust the sentiment intensity conveyed by other modalities according to the non-verbal context, with a sentiment contribution that varies in direction.

Interaction analysis can also reveal modality conflicts and their effects on decisions. In sample 2, T and V predict +0.268 and +0.278; while audio predicts −0.959, which is closer to the ground-truth label of −0.8. After adding T, T+A remains negative (−0.181), while T+V becomes more positive (+0.408). This shows that negative audio cues can change the sentiment judgement from T, while visual information reinforces its positive tendency. Although the local synergy terms for T--A and T--V are −0.3333 and −0.3421 nat, respectively, the two combinations produce predictions with opposite polarities. PID and subset predictions thus provide complementary explanations: the former characterises interactions in task information, while the latter reveals how these interactions change model decisions.

These analyses can also identify specific failure mechanisms in multimodal prediction. In sample 3, the text prediction is nearly neutral (+0.015), while the video prediction is −0.546, which agrees much more closely with the negative ground-truth sentiment (−0.8). Adding text makes the T+V prediction less negative (−0.152), and the full model moves further towards neutrality (−0.036). PID shows that T--A contains +0.3561 nat of information unique to audio and a synergy term of −0.5168 nat, whereas T--V combines positive redundant information (+0.1566 nat), a negative local text-unique term (−0.2116 nat) and a synergy term of −0.0681 nat. This reveals clear differences in the information relationships involving text across modality combinations and pinpoints the model's excessive suppression of negative non-verbal evidence.

These samples show that LatFuse's lattice structure also offers a way to analyse how decisions form, alongside its role in modelling interactions. Combining the Boolean lattice with PID allows us to trace predictions to specific modality relationships and distinguish mechanisms such as effective cross-modal correction, evidence suppression and single-modality dominance. These observations complement the synthetic experiments by demonstrating context-dependent contributions of modalities in real samples.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/5)

Question 2 (Weakness 2)

The scope of the experiments concerning robustness to missing modalities is limited. All evaluations involving missing modalities are restricted to the removal of a single modality; the study does not test scenarios where multiple modalities are missing simultaneously, nor does it address cases where arbitrary proportions of modal content are absent[1][2]. Consequently, while the current results support the claim that LATFUSE exhibits good robustness to missing modalities under the tested protocols, they are insufficient to establish a general advantage across a broader range of arbitrary missing-modality conditions.

LatFuse's core contribution is to model the interactions required by multimodal tasks through explicit local semantic conditioning. We did not design it specifically for missing modalities. Because LatFuse computes on a lattice, it can perform inference directly on a sublattice when modalities are missing at inference time. The missing-modality experiments assess the robustness of this sublattice inference. Our existing single-modality removal evaluations show that LatFuse's sublattices adapt well to missing modalities in this setting, without additional interpolation or data generation.

In a three-modality setting, removing two modalities leaves only one, reducing the task to unimodal prediction and precluding the evaluation of cross-modal fusion and higher-order interactions. With more modalities, LatFuse retains the same inference mechanism and operator-reuse scheme across observed sublattices of any size. Removing a single modality therefore tests sublattice inference while preserving interactions among the remaining modalities, including pairwise interactions in the three-modality setting.

To further investigate LatFuse under more general missing-modality conditions, we also tested:

  1. Modality-level missing content at arbitrary rates on TCGA-BRCA;
  2. Single-modality removal on TCGA-BRCA after training baseline methods with modality dropout;
  3. Sample-level missing modality content at arbitrary rates on CMU-MOSI.

We have added the full results to Section 5.4 of the revised manuscript and to Appendix F.

Modality-Level Missingness on TCGA-BRCA.

We have added missing-modality tests at arbitrary rates on TCGA-BRCA. After standard training across all modalities, we set the missing rate for each modality to 0.1 -- 1.0 at test time. The table below reports ACC averaged across the missing rates.

MethodFull−mRNA−miRNA−DNA
MM-Dynamics86.9460.13 (−26.81)83.48 (−3.46)84.43 (−2.51)
MOHGCN87.9179.84 (−8.07)86.19 (−1.72)85.73 (−2.18)
LatFuse (ours)89.3983.09 (−6.30)88.79 (−0.60)87.72 (−1.67)

The results show that LatFuse also offers a robustness advantage over the baseline methods on this task across the tested missing rates.

Single-Modality Removal with Modality Dropout for TCGA-BRCA Baselines.

We trained several baselines on TCGA-BRCA with different modality dropout rates. The table below reports ACC under the Full and heterogeneous missing-modality conditions.

MethodDropoutFullHetero.
MM-Dynamics0.086.9472.96 (−13.98)
0.286.9782.58 (−4.39)
0.586.9382.88 (−4.05)
MOHGCN0.087.9182.50 (−5.41)
0.283.7380.69 (−3.04)
0.584.3581.15 (−3.20)
DMIB0.086.2281.82 (−4.40)
0.285.4881.65 (−3.83)
0.585.2381.96 (−3.27)
CLCLSA0.086.0982.00 (−4.09)
0.285.1782.86 (−2.31)
0.583.8882.31 (−1.57)
MOSGAT0.087.1782.43 (−4.74)
0.286.5483.92 (−2.62)
0.586.3983.63 (−2.76)
LatFuse (ours)0.089.3985.39 (−4.00)

The results show that:

  1. Modality dropout markedly improves the missing-modality performance of MM-Dynamics. However, MM-Dynamics still suffers a larger performance drop than LatFuse in the condition with the largest change in the metric.
  2. Modality dropout has a detrimental effect on MOHGCN and CLCLSA: it substantially reduces their performance with all modalities, even though it reduces the absolute drop caused by missing modalities.
  3. Modality dropout can improve the missing-modality performance of DMIB and MOSGAT, but selecting an appropriate dropout rate introduces an additional tuning difficulty. LatFuse avoids this difficulty by supervising all subsets.
Reviewer_J46xOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LatFuse, a framework that learns multimodal interactions by considering the marginal contribution of each component modality in a bottom-up manner while accounting for the other modalities that are present. It does this by building a Boolean lattice, constructing multimodal representations that are refined by the addition of each modality, and then decomposing the contributions of individual modality subset-specific interaction components via Mobius inversion. These representations are input to a task-specific prediction head for downstream inference. The authors contribute both this decomposition formulation as well as the methodological implementation. LatFuse outperforms the baselines across multiple datasets (each containing 3-4 modalities) and tasks, demonstrates lower performance degradation in the presence of missing modalities, and recovers known interaction structures on synthetic data. Ablation experiments also demonstrate the utility of the components of LatFuse.

Critical strengths

• Strong performance over baselines, as well as the verification on synthetic tasks helps to confirm that the underlying basis of the method is working as expected.

• Novel approach that goes beyond prior work by considering the context-dependence of adding modalities while modeling cross-modal interactions. Method and experiments are clearly described, and the paper is easy to follow.

Critical weaknesses

• More clarity needed on the practical utility of the functional decompositions – beyond performance gains, is there utility in having these readouts (e.g. for interpretability, etc.). This would strengthen the significance of the approach, as well as help readers understand the unique benefits that this approach contributes over the several other existing multimodal approaches. Current results on the synthetic data are promising and insightful; discussing some potential real-world use cases would further enhance the importance of the results.

• Model complexity with increasing modalities – how prohibitive is this? (see Question 1). This is an important limitation of the method, which can constrain its usage in practice. However, the authors do address this and provide suggestions for future work to mitigate this, which is helpful.

Questions for authors

Clarification Requests

  1. Clarification on the latency / inference time – how does runtime compare to the baselines, for both training and inference?

  2. What was T set to in these experiments? How was this tuned? How does performance vary with the number of iterations / how sensitive is performance to the number of iterations?

  3. In Related Works, it would be helpful to more explicitly state why LatFuse’s approach is an improvement / advancement upon the prior approaches. Not just what LatFuse does differently, but why that’s important/beneficial.

  4. Why do the results on DeLiVER only report standard deviation for StitchFusion + LatFuse? What does the variability look like for the competing methods?

  5. Although there are no ground-truth decompositions in the real data, did the authors ever look at the readouts from the real data to analyze the model-identified modality contributions across different contexts? How can this knowledge be used in real world settings – could it be used for interpretability / data-driven insights?

  6. How were baseline methods chosen for different tasks? I’m especially wondering why the baselines are not the same for both multi-omics classification tasks. Also, how were the two baselines chosen for the missingness experiments in Table 7?

Suggestions for Future Work

  1. Expanded missingness experiments:

    a. Assessing different types of missingness – MAR, MNAR, MCAR.

    b. Assessing how the model performs when trained on data with missing modalities. For example, this could be useful for the case when data are very sparse and being able to include incomplete examples during training would provide a substantial increase in sample size.

  2. Expanded synthetic data experiments:

    a. As a follow-up / future experiment, it might be interesting to include more pairwise interactions – for example, if s1s3 and s2s3 were present, would we see the band across the full set of 2-modality subsets?

    b. Additionally, it could be interesting, if possible, to alter the strength of the different types of interactions and see if the readout can recover which types of interactions are most dominant. I’d imagine that in real data, perhaps all possible interactions could be present, so I’m wondering if the readouts provided by the model could potentially provide useful information / insights into the dataset.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/5)

We thank the reviewer for the positive assessment and constructive questions.

Question 1

Clarification on the latency / inference time -- how does runtime compare to the baselines, for both training and inference?

We benchmarked training runtimes, parameter counts, and single-pass forward FLOPs across the evaluation benchmarks on a single RTX 3090 GPU.

During inference, LatFuse executes only the forward lattice construction and the Möbius readout, omitting the shared subset predictor and auxiliary loss terms. The single-pass FLOPs and epoch runtimes below therefore reflect both training and inference efficiency relative to the baselines.

TCGA-BRCA.

Method#Params (M)FLOPs (G)Time (ms/epoch)
MM-Dynamics4.7942.5205.93
MOGLAM5.3724.5798.67
DPNET3.6206.87118.49
DeepMKL1.6460.8643.17
DMIB6.5352.2484.70
MOSGAT2.5227.82229.44
CLCLSA2.9451.4535.16
MOHGCN8.03718.51077.60
LatFuse (ours)1.2524.0396.24

LatFuse uses the fewest parameters among all compared methods while maintaining competitive forward FLOPs and per-epoch runtime.

MLOmics-Pan-cancer.

We have added parameter counts, FLOPs for a single forward pass and training time measurements on MLOmics-Pan-cancer, with comparisons against the baselines.

Method#Params (M)FLOPs (G)Time (s/epoch)
MM-Dynamics55.34944.5294.89
CustOmics25.66944.8706.77
DMIB33.14618.2283.08
MOSGAT17.13086.10412.23
LatFuse (ours)35.36867.6685.07

In this four-modality setting, LatFuse operates within the same runtime and compute range as the baselines.

Other Benchmarks.

On CMU-MOSI, we found that the BERT-base text encoder accounts for most of the end-to-end parameter count and FLOPs, while fusion accounts for a limited share of the cost. All baselines use a BERT-base encoder, so we did not compare LatFuse's costs with those of the other baselines.

ModuleBERT-base emb.BERT-base encoder.Fuser
#Params (M)23.83785.0544.869 (4.28%)
FLOPs (G/sample)08.5864.020 (31.89%)

On DeLiVER, we replaced the StitchStep module in StitchFusion. This module accounts for 1.03% of the parameters in the full model, so the additional overhead is almost negligible relative to the end-to-end pipeline.

ModuleStitchFusionStitchFusion + LatFuse
#Params (M)29.87430.132 (+0.86%)
FLOPs (G)606.91614.23 (+1.21%)

Question 2

What was T set to in these experiments? How was this tuned? How does performance vary with the number of iterations / how sensitive is performance to the number of iterations?

We set T=2T=2 for CMU-MOSI and T=1T=1 for TCGA-BRCA, MLOmics-Pan-cancer and DeLiVER. During model design, we hypothesised that the optimal number of iterations TT depends on the token structure of the input modalities. With compact latent tokens or pointwise features, a single cross-attention step is sufficient to extract the marginal contribution of the newly added modality. For longer sequences with cross-modal temporal dependencies, such as N=50N=50 in CMU-MOSI, one additional refinement iteration can further improve global sequence alignment.

We therefore made the number of edge-refinement steps TT configurable, allowing single- or multi-step context alignment based on each task's token sequence length and attention span. We select the value on the validation set from the small range 1,2,3,4{1, 2, 3, 4}. The additional results cover different values of TT on TCGA-BRCA and CMU-MOSI.

TCGA-BRCA.

TACCWeighted F1F_1Macro F1F_1
189.3989.7587.08
289.1389.4286.69
388.5888.9085.92
489.0389.3786.38

CMU-MOSI.

TMAECorrACC$_7$ACC$_2$ (has0/non0)F1F_1 (has0/non0)
10.6780.80548.8384.59 / 87.0184.53 / 86.98
20.6620.82350.9586.17 / 88.6686.15 / 88.61
30.6700.81449.3585.88 / 88.2485.81 / 88.19
40.6730.81149.0785.42 / 87.8185.30 / 87.75

Across the tested values of TT, TCGA-BRCA ACC ranges from 88.58 to 89.39, while CMU-MOSI ACC$_7$ ranges from 48.83 to 50.95. The best results occur at T=1T=1 for TCGA-BRCA and at T=2T=2 for CMU-MOSI. Increasing TT beyond these values brings no consistent improvement. This shows that LatFuse's performance advantage does not depend on extensive tuning of the iteration count or costly multi-step computation.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/5)

Question 3

In Related Works, it would be helpful to more explicitly state why LatFuse's approach is an improvement / advancement upon the prior approaches. Not just what LatFuse does differently, but why that's important/beneficial.

We have revised the Related Work section accordingly. Reusable subset states provide explicit contexts for adding modalities, allowing their contributions to adapt to the information already present. The Boolean lattice organises these local updates into higher-order interactions.

Question 4

Why do the results on DeLiVER only report standard deviation for StitchFusion + LatFuse? What does the variability look like for the competing methods?

The baseline results on DeLiVER come from our inference runs using the single weights file released by each method's authors. We did not retrain these baseline models, so we do not report standard deviations. Our reproduced metrics closely match those in the original papers; none of these three original papers reports standard deviations:

MethodmIoU reprod.mIoU paper
CMNeXt66.3366.30
StitchFusion68.2168.18
Mul-VMamba68.9968.98
AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/5)

Question 5

Although there are no ground-truth decompositions in the real data, did the authors ever look at the readouts from the real data to analyze the model-identified modality contributions across different contexts? How can this knowledge be used in real world settings -- could it be used for interpretability / data-driven insights?

To examine how modality contributions vary across contexts in real data, we used partial information decomposition (PID) to analyse LatFuse's subset readouts on the CMU-MOSI dataset [1][2]. Below, we present the text transcripts of three representative samples, their subset predictions on the sentiment scale [−3,3][-3, 3], and four-component PID decompositions for T--A and T--V in nats:

  1. I DIDNT REALLY I MEAN TRASURE PLANET WAS OK
  2. IF YOU LIKE EXPLOSIONS REALLY RIDICULOUS ACTION SEQUENCES MALODRAMATIC PLOT AND ACTING THIS IS THE RIGHT MOVIE TO SEE
  3. UM I DONT THINK THAT THIS OVE GOT AS MUCH BUZZ AND AS MUCH UM MEDIA ATTENTION AND UM WORD OF MOUTH YOU KNOW SPREADING AROUND AND STUFF
SampleGround Truth LabelTAVT+AT+VT+A+V
1-1.0+0.083+0.560−2.179+0.089−1.031−0.976
2-0.8+0.268−0.959+0.278−0.181+0.408−0.084
3-0.8+0.015−0.130−0.546−0.026−0.152−0.036
SamplePID PairRedundantT UniqueA/V UniqueSynergistic
1T--A+0.0499≈0−0.2159−0.4317
T--V+0.0499≈0+0.2111+0.2109
2T--A−0.1324≈0+0.7367−0.3333
T--V−0.1324≈0+0.2287−0.3421
3T--A−0.0550≈0+0.3561−0.5168
T--V+0.1566−0.2116≈0−0.0681

The results show that subset predictions reveal the context dependence of modality contributions. In sample 1, text alone is nearly neutral (+0.083), audio is positive (+0.560), and video is strongly negative (−2.179). Adding text to audio lowers the prediction, whereas adding text to video raises it. Text therefore moderates both overly positive and overly negative judgements, yielding a final prediction of −0.976, which closely matches the ground-truth label of −1.0. PID also reveals positive synergistic information for T--V (+0.2109 nat), whereas the T--A synergy term is −0.4317 nat. This shows that text can adjust the sentiment intensity conveyed by other modalities according to the non-verbal context, with a sentiment contribution that varies in direction.

Interaction analysis can also reveal modality conflicts and their effects on decisions. In sample 2, for example, text and video predict +0.268 and +0.278, respectively, while audio predicts −0.959, which is closer to the ground-truth label of −0.8. After adding text, T+A remains negative (−0.181), while T+V becomes more positive (+0.408). This shows that negative audio cues can change the sentiment judgement from text, while visual information reinforces its positive tendency. Although the local synergy terms for T--A and T--V are −0.3333 and −0.3421 nat, respectively, the two combinations produce predictions with opposite polarities. PID and subset predictions thus provide complementary explanations: the former characterises interactions in task information, while the latter reveals how these interactions change model decisions.

These analyses can also identify specific failure mechanisms in multimodal prediction. In sample 3, the text prediction is nearly neutral (+0.015), while the video prediction is −0.546, which agrees much more closely with the negative ground-truth sentiment (−0.8). Adding text makes the T+V prediction less negative (−0.152), and the full model moves further towards neutrality (−0.036). PID shows that T--A contains +0.3561 nat of information unique to audio and a synergy term of −0.5168 nat, whereas T--V combines positive redundant information (+0.1566 nat), a negative local text-unique term (−0.2116 nat) and a synergy term of −0.0681 nat. This reveals clear differences in the information relationships involving text across modality combinations and pinpoints the model's excessive suppression of negative non-verbal evidence.

These samples show that LatFuse's Boolean lattice structure also offers a way to analyse how decisions form, alongside its role in modelling interactions. Combining the Boolean lattice with PID allows us to trace predictions to specific modality relationships and distinguish mechanisms such as effective cross-modal correction, evidence suppression and single-modality dominance. We believe this analysis could be a valuable direction for future research.

Reviewer_F1tgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper treats multimodal fusion as a set function over modality subsets, and uses Möbius inversion to show that a modality’s context-dependent contribution is equivalent to higher-order interactions. Building on this view, the authors propose LatFuse, which builds a representation for every modality subset bottom-up on the Boolean lattice, using shared single-modality extensions. Because each subset representation depends only on the modalities it contains, the model can naturally handle missing modalities at inference. Experiments on four benchmarks covering multiomics, sentiment analysis, and segmentation show consistent gains over the compared baselines.

Critical strengths
  1. the motivation is reasonable, treating fusion as a set function over modality subsets, and using Möbius inversion to link context-dependent contributions to higher-order interactions, gives the method a clear theoretical basis.
  2. Each subset representation depends only on the modalities it contains (Proposition 2), so the same trained model can run on any observed subset without retraining or imputation. That’s a practical advantage, and it follows naturally from the design.
  3. The experiments cover multiomics classification, sentiment analysis, and semantic segmentation, and the gains are consistent across them.
Critical weaknesses

Please refer to the questions.

Questions for authors
  1. Can you compare LatFuse against the most closely related structured fusion methods, such as TFN, LMF, and especially Multimodal Residual Optimization, using the same encoders and training setup? Right now it’s hard to tell whether the gains come from the lattice construction itself or from other design choices.
  2. Were the baseline results reproduced by you or taken from the original papers? Some, like those on DeLiVER, have no standard deviation.
  3. In the missing-modality experiments, LatFuse is trained with supervision on every subset, while baselines only get zero-filled inputs at test time. How do the results change if the baselines are also trained with modality dropout?
  4. Can you add an ablation without L_subset? It could be a major source of the gains, and it’s the only loss term that wasn’t ablated.
  5. What are the parameter counts, FLOPs, and runtimes on the other benchmarks, and how does the cost grow with the number of modalities?
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/4)

We thank the reviewer for the constructive feedback and insightful questions.

Question 1

Can you compare LatFuse against the most closely related structured fusion methods, such as TFN, LMF, and especially Multimodal Residual Optimization, using the same encoders and training setup? Right now it's hard to tell whether the gains come from the lattice construction itself or from other design choices.

Structured Fusion Methods.

We reproduced TFN, LMF and MRO on CMU-MOSI using the official code released by their authors with the same encoder and training configuration as LatFuse. The table below shows the results. We have added the full results, including standard deviations, to Section 5.2 of the revised manuscript.

MethodMAECorrACC$_7$ACC$_2$ (has0/non0)F1F_1 (has0/non0)
TFN (2017)1.0070.62932.0773.71 / 74.9573.76 / 75.07
LMF (2018)1.0110.64030.6674.29 / 75.5674.35 / 75.69
MRO (2022)0.9640.66033.5376.38 / 77.1376.43 / 77.25
LatFuse (ours)0.6620.82350.9586.17 / 88.6686.15 / 88.61

These three structured baselines perform below the other baselines in our comparison. To examine how their interaction structures behave when input content is missing, we also evaluated them under sample-level missing-modality conditions alongside more recent methods.

We used standard training with all modalities. At test time, we set the missing rate to 0.0 -- 0.9 and independently masked tokens or frames at random in each of the three modalities within each sample. In the table below, Missing denotes the results averaged across missing rates. We have added the full results to Section 5.4 of the revised manuscript and to Appendix F.

MethodCond.MAECorrACC$_7$ACC$_2$ (has0/non0)F1F_1 (has0/non0)
TFNFull1.0070.62932.0773.71 / 74.9573.76 / 75.07
Missing1.237 (+0.230)0.340 (−0.289)25.37 (−6.70)63.36 (−10.35) / 65.47 (−9.48)56.47 (−17.29) / 58.90 (−16.17)
LMFFull1.0110.64030.6674.29 / 75.5674.35 / 75.69
Missing1.238 (+0.227)0.333 (−0.307)23.74 (−6.92)62.92 (−11.37) / 65.00 (−10.56)56.92 (−17.43) / 59.28 (−16.41)
MROFull0.9640.66033.5376.38 / 77.1376.43 / 77.25
Missing1.234 (+0.270)0.364 (−0.296)24.75 (−8.78)63.82 (−12.56) / 65.25 (−11.88)59.88 (−16.55) / 61.52 (−15.73)
LNLNFull0.7070.79748.1583.43 / 85.5783.28 / 85.43
Missing1.052 (+0.345)0.534 (−0.263)30.09 (−18.06)69.86 (−13.57) / 71.40 (−14.17)66.92 (−16.36) / 68.27 (−17.16)
ALMTFull0.6840.80648.0784.07 / 85.6684.03 / 85.67
Missing1.051 (+0.367)0.527 (−0.279)30.61 (−17.46)69.09 (−14.98) / 70.66 (−15.00)64.84 (−19.19) / 65.99 (−19.68)
DLFFull0.7470.77444.2283.02 / 84.6883.01 / 84.73
Missing1.060 (+0.313)0.523 (−0.251)30.36 (−13.86)70.09 (−12.93) / 71.09 (−13.59)67.95 (−15.06) / 66.93 (−17.80)
LatFuse (ours)Full0.6620.82350.9586.17 / 88.6686.15 / 88.61
Missing1.053 (+0.391)0.537 (−0.286)30.55 (−20.40)71.23 (−14.94) / 72.30 (−16.36)70.51 (−15.64) / 71.66 (−16.95)

TFN, LMF and MRO explicitly model modality interactions of different orders, but they underperform state-of-the-art baselines with both complete inputs and random masking. This shows that explicitly constructing interaction terms does not in itself imply more effective fusion. LatFuse still achieves the highest Corr, ACC$_2$ and F1F_1 under random masking. The ablations below further examine the contribution of context-conditioned modelling to LatFuse's performance.

Sources of the Performance Gains.

The ablations show that context-conditioned refinement contributes substantially to the performance gains, with the Boolean lattice providing the structural scaffold. Traversing the lattice without conditioning each step on the predecessor context fails to yield these improvements. Our existing A1 ablation shows that removing context conditioning from the computational flow while retaining the other design choices substantially reduces performance. To further isolate the gains from additional supervision, we conducted an ablation that removes Lsubset\mathcal{L}_\mathrm{subset} (L3). Together with the existing A1 ablation, this gives a 2x2 ablation grid:

with context conditioningw/o context conditioning
with Lsubset\mathcal{L}_{\mathrm{subset}}FullA1
w/o Lsubset\mathcal{L}_{\mathrm{subset}}L3A1 + L3
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/4)

Question 1 (sequel)

The table below shows the experimental results. We have added the full results to Section 5.3 of the revised manuscript.

VariantTCGA-BRCA (m.F1F_1)Pan-cancer (m.F1F_1)CMU-MOSI (ACC$_7$)
Full87.0895.2650.95
A183.60 (−3.48)92.59 (−2.67)45.42 (−5.53)
L385.14 (−1.94)92.45 (−2.81)44.61 (−6.34)
A1 + L382.33 (−4.75)91.83 (−3.43)42.68 (−8.27)

These results show that context construction has substantial independent value on TCGA-BRCA, where removing additional supervision causes only a limited performance drop. On MLOmics-Pan-cancer and CMU-MOSI, context construction and subset supervision have a stronger joint effect. Comparing A1 + L3 →\to A1 with L3 →\to Full shows that adding Lsubset\mathcal{L}_{\mathrm{subset}} alone yields smaller gains without context conditioning. On Pan-cancer, for example, the gain rises substantially from +0.76 without context conditioning to +2.81 with it. These results show that subset supervision strengthens the benefit of context conditioning, consistent with its role in making predecessor states task-relevant.

Specifically, during edge refinement, the predecessor state hVh_{\mathcal{V}} directly serves as the initial query vector in cross-attention to retrieve complementary marginal information from the newly added modality xmx_m. Without Lsubset\mathcal{L}{\mathrm{subset}}, the lower-level intermediate state hVh_{\mathcal{V}} has a long gradient path to the task representation. Direct training may therefore fail to ensure that this state encodes the complete task semantics of subset V\mathcal{V}. Subset supervision trains each predecessor state hVh_{\mathcal{V}} to provide a task-relevant context.

LatFuse's contribution is to represent modality subset states as reusable contexts for fusion. We consider context-conditioned modelling a local operation with clear semantics, enabling models to learn more readily across different tasks. Learning higher-order interactions directly at the global level requires a model capable of fitting joint representations across many combinations of modalities. LatFuse decomposes this complex problem into local objectives with consistent semantics and organises them on the Boolean lattice. Both the lattice structure and subset supervision serve the core design of context-conditioned modelling. Subset supervision also adds only limited overhead relative to the core computational flow.

Question 2

Were the baseline results reproduced by you or taken from the original papers? Some, like those on DeLiVER, have no standard deviation.

For the DeLiVER baselines, we reproduced the results by running inference using the single weights file released by each method's authors. We did not retrain these models, so we do not report standard deviations. For the other datasets, we trained and evaluated each method across multiple seeds using its official source code. We compared the resulting metrics with those in the original papers to ensure faithful reproduction.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/4)

Question 3

In the missing-modality experiments, LatFuse is trained with supervision on every subset, while baselines only get zero-filled inputs at test time. How do the results change if the baselines are also trained with modality dropout?

We trained several baselines on TCGA-BRCA with different modality dropout rates. The table below reports ACC under the Full and heterogeneous missing-modality conditions. We have added the full results to Appendix F of the revised manuscript.

MethodDropoutFullHetero.
MM-Dynamics0.086.9472.96 (−13.98)
0.286.9782.58 (−4.39)
0.586.9382.88 (−4.05)
MOHGCN0.087.9182.50 (−5.41)
0.283.7380.69 (−3.04)
0.584.3581.15 (−3.20)
DMIB0.086.2281.82 (−4.40)
0.285.4881.65 (−3.83)
0.585.2381.96 (−3.27)
CLCLSA0.086.0982.00 (−4.09)
0.285.1782.86 (−2.31)
0.583.8882.31 (−1.57)
MOSGAT0.087.1782.43 (−4.74)
0.286.5483.92 (−2.62)
0.586.3983.63 (−2.76)
LatFuse (ours)0.089.3985.39 (−4.00)

The results show that:

  1. Modality dropout markedly improves the missing-modality performance of MM-Dynamics. However, MM-Dynamics still suffers a larger performance drop than LatFuse in the condition with the largest change in the metric.
  2. Modality dropout has a detrimental effect on MOHGCN and CLCLSA: it substantially reduces their performance with all modalities, even though it reduces the absolute drop caused by missing modalities.
  3. Modality dropout can improve the missing-modality performance of DMIB and MOSGAT, but selecting an appropriate dropout rate introduces an additional tuning difficulty. LatFuse avoids this difficulty by supervising all subsets.

Without modality dropout, most baselines suffer larger performance drops than LatFuse. LatFuse also achieves better final results than every baseline included in the comparison across the different dropout configurations.

Question 4

Can you add an ablation without L_subset? It could be a major source of the gains, and it's the only loss term that wasn't ablated.

We added the requested ablation, L3, which removes Lsubset\mathcal{L}_\mathrm{subset}. The results appear in our response to Question 1, where we combine L3 with A1 to examine subset supervision and context conditioning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Authors' Responses

We thank all reviewers for their constructive feedback. Here, we summarise the additional experiments and clarifications addressing questions raised by multiple reviewers. We have revised the manuscript in purple text accordingly.

1. Sources of LatFuse's Performance Gains (Reviewer F1tg, Q1 & Q4; Reviewer EcKB, Q1 & Q3)

To further distinguish the sources of LatFuse's performance gains. To address this concern, we conducted a new ablation L3 that removes Lsubset\mathcal{L}_{\mathrm{subset}} and combined it with the existing ablation A1, which removes predecessor-context conditioning, to form a 2×2 ablation grid. The results show that even without Lsubset\mathcal{L}{\mathrm{subset}}, context conditioning improves macro F1F_1 by +2.81 percentage points on TCGA-BRCA. On MLOmics-Pan-cancer, adding Lsubset\mathcal{L}{\mathrm{subset}} improves macro F1F_1 by only +0.76 points without context conditioning, compared with +2.81 points when context conditioning is enabled.

These findings indicate that context-conditioned modelling drives LatFuse's performance gains and constitutes the central contribution of our approach. Both the computational flow on the Boolean lattice and the subset supervision provided by Lsubset\mathcal{L}_{\mathrm{subset}} directly support this design objective. We have added the full results to Section 5.3 of the revised manuscript.

2. Missing-Modality Robustness and Modality Dropout (Reviewer F1tg, Q3; Reviewer J46x, Q6; Reviewer EcKB, Q2)

We evaluated LatFuse under a broader range of missingness protocols, including:

  1. Comparisons with baseline models trained using modality dropout on TCGA-BRCA;
  2. Tests with arbitrary rates of missing modality content on TCGA-BRCA, including additional baselines;
  3. Tests with arbitrary rates of sample-level missingness on CMU-MOSI.

The results show that:

  1. LatFuse outperforms all compared baselines across different baseline dropout configurations;
  2. LatFuse exhibits greater robustness than several baselines under arbitrary rates of missing modality content;
  3. LatFuse achieves fine-grained discrimination comparable to several baselines under arbitrary rates of sample-level missingness, while outperforming them on binary classification metrics.

These findings support missing-modality robustness as a desirable property of LatFuse. Its sublattice inference adapts well across different missingness settings without additional interpolation or data generation. We have added the full results to Section 5.4 of the revised manuscript and to Appendix F.

3. Interaction Analysis on Real Data (Reviewer J46x, Q5; Reviewer EcKB, Q1)

To examine how the subset representations learned by LatFuse relate to task-relevant interactions and provide interpretability on real data, we combined partial information decomposition (PID) with subset predictions on CMU-MOSI to quantify unique, redundant, and synergistic information across different modality pairs. By tracing subset predictions for representative utterances, we illustrate how the text modality modulates conflicting acoustic and visual sentiment cues, and how redundant or unique non-verbal signals help explain reversals in sentiment polarity and prediction errors.

4. Parameter Counts, Computational Cost, and Runtime (Reviewer F1tg, Q5; Reviewer J46x, Q1)

We compared LatFuse's parameter counts, FLOPs and runtimes with the baseline models on the two multiomics datasets using a single RTX 3090 GPU. The results show that LatFuse has computational costs comparable to those of baselines with different architectures on both benchmarks, while using fewer parameters than all compared baselines on TCGA-BRCA. For the other two benchmarks, we explained why LatFuse's computational structure is not the primary performance bottleneck in either task.

5. Baseline Reproduction and Standard Deviations on DeLiVER (Reviewer F1tg, Q2; Reviewer J46x, Q4)

We evaluated the DeLiVER baselines through deterministic inference using the single pretrained weights file released by each method's authors. The reproduced mIoU scores differ from those reported in the original papers by no more than 0.03. For TCGA-BRCA, MLOmics-Pan-cancer and CMU-MOSI, we retrained and evaluated all baseline models across multiple random seeds using their official source code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.