- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Lluna, combining private prediction paths from unimodal features with residual common collaboration and progressive cross-state consistency. Evaluation on three domains show an improvement in Mean-Missing performance.
Critical strengths
Overall, the paper was clear to read and easy to follow. The experiments were quite extensive spanning a range of domains and comparing a range of past methods. The code was also shared, which will aid reproducibility for future work. The proposed method is an interesting direction exploring collaborative unimodal models and will be of interest for the community.
Critical weaknesses
Originality and significance: The motivation behind the desired collaboration strategy is unclear. The paper focuses on compatibility, where it encourages agreement in task predictions. But it needs to discuss on why this is the optimal choice. For instance, it is likely that in the cases of conflict one modality predictor might be correct and confident and that should be the one that should be considered? The paper should also discuss on why the graph should remain fixed during training and inference as it is possible that the relationship between modalities and their contribution to a task can differ across tasks.
The paper further lacks a discussion with prior work [1,2,3] that have considered a decomposition of contributions of individual modalities and their interaction on a task, which makes the positioning challenging.
The paper should also discuss why modality-specific signals become less accessible after collaboration, what is a private base prediction and why is it required. For missing modality, the dependence and constraint of removal of exactly one modality should be discussed further as well. I'd also suggest to further elaborate the problem in section 3.5. Without confidence intervals, its also challenging to see the contributions of individual components in Table 3.
I'd also suggest to discuss the scenarios and application of the method when missingness is not at random and when multiple modalities are missing.
Clarity: The overview section is challenging to comprehend without the introduction of terms. For instance, collaboration path or private prediction path, compatibility graph are not self-evident terms to provide an overview for the readers. The result and ablation tables can be enhanced for readability. Currently it contains a lot of numbers, which can be hard for readers to interpret. I'd suggest to have the detailed results in appendix while highlighting the takeaways in the main paper. The meaning of underlined results is unclear as well. Further, how were the hyper-parameters in eq 3 were chosen? Did the authors conduct an ablation analysis on the same? Minor, but why is the method called Lluna with a double "l"?
[1] Fang et al. Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition.
[2] Madaan et al. Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning.
[3] Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.
Questions for authors
Please refer to my comments in the weakness section above.
8 comments
Question 2. How does Lluna relate to the work of Fang, Madaan and Liang on individual modality contributions and cross-modal interactions?
Fang et al. [1] and Liang et al. [3] study how modalities contribute to a prediction. Madaan et al. [2] explicitly combine multiple predictors. Our question is how independently pretrained unimodal experts can continue to collaborate when only some of their inputs are available.
Information decomposition. Fang et al. [1] separate the contributions of audio and video to a multimodal language model’s decision given the language prompt. Their analysis finds a stronger reliance on vision even on tasks designed to use both. Liang et al. [3] link the drop in accuracy after a modality is removed to the unique information it carried and, when that contribution is small, to information available only jointly. Both works measure how a model uses its modalities. Lluna builds this distinction into the predictor: each surviving expert’s own cues reach the prediction through the private path, and the common path uses the interactions that remain possible. Appendix D.2 applies a related analysis to Lluna itself, with Shapley values for modality contributions and pairwise interactions.
Combining unimodal and joint predictors. Madaan et al. [2] combine one classifier per modality with a classifier over both inputs as a product of experts, which adds their log-probabilities. Lluna differs in three respects. (1) Missing inputs. Their model is trained and evaluated with every modality present, and its joint classifier takes both inputs. Lluna trains a single predictor that handles every nonempty input subset. The private head reads whichever experts are available, and the common branch passes messages only among them. (2) Weighting. A product of experts weights its terms equally. Lluna treats the common prediction as a correction with a learned scale. Fixing that scale to one, which weights the two terms equally, lowers Mean-Missing on six of eight tasks, by 4.70% on average (Table 3a). (3) Consistency across input states. PCC aligns the common representation of each state with that of its neighbor with one more input. Removing PCC lowers Mean-Missing by 2.27% on average (Table 3c).
An interaction-aware baseline. I²MoE (Xin et al., 2025), which assigns separate experts to different types of modality interaction, is already among our baselines. With missing-input training on the shared expert pool, Lluna outperforms it on all three tasks in Table 2: MAE 0.3421 vs. 0.3731 on CH-SIMS v2, accuracy 33.82% vs. 30.27% on WorldSense, and balanced accuracy 84.17% vs. 77.71% on ADNI. It is also one of the seven baselines in Fig. 4.
The revised paper adds these comparisons to §2 and links them to the analysis in Appendix D.2.
[1] Wanlong Fang et al. Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition. ICML, 2026.
[2] Divyam Madaan et al. Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning. NeurIPS, 2024.
[3] Paul Pu Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework. NeurIPS, 2023.
Question 3a. Why can modality-specific cues become harder to recover after collaboration, and how does the private base prediction help?
Pooling compresses several experts’ features into one representation. A detail carried by one modality, such as tone of voice, can become harder to read back from it. In Fig. 1(b), we first predict a modality’s features from the other modalities and take the part left unexplained as our target. We then measure how well a linear probe recovers that same target from the attention-pooling model. Across eight tasks, the probe’s mean falls from 58.52% with the modality alone to 33.47% with all modalities pooled. The lower score means that the probe can explain less of the variation in the modality-specific features after pooling.
Lluna’s private head makes a base prediction directly from the surviving experts’ features, before pooling. A cue such as tone can therefore affect the answer even if it is hard to recover from the shared representation. The revised paper defines the base prediction in §3.3 and explains what the probe measures in the Fig. 1(b) caption.
Question 3b. Can Lluna handle several missing modalities at inference? Why does each PCC training step remove exactly one modality?
Yes. Lluna can predict when several modalities are missing at the same time. It processes the observed inputs in one forward pass. Table 6 reports all three ways that two modalities can be missing on MELD. The same trained model handles each input combination, and scores are averaged over five runs.
| Available input | Inputs missing | Lluna W-F1 (%) | Strongest baseline | Baseline W-F1 (%) |
|---|---|---|---|---|
| Text | Audio and video | 63.60 | LIMSSR | 61.72 |
| Audio | Text and video | 53.63 | LIMSSR | 51.33 |
| Video | Text and audio | 37.86 | SimMLM | 36.81 |
During training, a view with little evidence receives guidance from a teacher with just one additional modality. For example, the text–audio state receives guidance from the full text–audio–video state and in turn guides the text-only state. This breaks one large change in evidence into two smaller ones. Full-to-Minimal also uses a teacher that sees all the student’s inputs, but it asks the text-only student to match the full state directly and gives the intermediate state no consistency target. Fig. 1(c) supports the intuition that adjacent targets are closer. Across the six three-input tasks, the mean cosine distance is 0.279 for adjacent states and 0.456 from full to minimal input. In Table 3c, adjacent PCC outperforms Full-to-Minimal on all seven tasks where the pair sets differ. The revised §3.4 uses this example to explain how progressive guidance works during training.
Question 3c. What does the stability result in §3.5 establish, and how uncertain are the component effects reported in Table 3?
Consider the same clip with all three inputs, then with video removed, and finally with only text left. Section 3.5 shows that the distance between the first and last normalized common representations cannot exceed the distance moved in the first step plus the distance moved in the second. If PCC keeps both changes small, the text-only representation stays close to the full-input representation even though training only aligns neighboring states.
We computed paired 95% bootstrap intervals for fixed, archived full-model and ablation checkpoints. Each of 10,000 resamples keeps a sampling unit and all its views together. Positive differences favor the full model. Units are MAE × 100 for SIMS/MOSI and pp elsewhere.
| Dataset | w/o Private | w/o Common | w/o PCC |
|---|---|---|---|
| CH-SIMS v2 | 1.14 [0.50, 1.77] | 1.31 [0.87, 1.77] | 0.66 [0.25, 1.06] |
| CMU-MOSI | 3.07 [−0.09, 6.09] | 3.07 [1.56, 4.61] | 1.53 [0.75, 2.47] |
| MELD | 0.59 [0.06, 1.12] | 0.92 [0.37, 1.47] | 0.11 [−0.33, 0.55] |
| UR-FUNNY v2 | 2.60 [1.32, 3.87] | 0.99 [0.54, 1.46] | 0.60 [0.35, 0.87] |
| Daily-Omni | 0.85 [−4.47, 6.01] | 2.26 [−3.32, 7.65] | 0.00 [0.00, 0.00] |
| WorldSense | 1.32 [−0.47, 3.12] | 1.57 [0.47, 2.68] | 1.67 [0.20, 3.14] |
| CirCor | 10.69 [1.37, 21.27] | 4.44 [0.08, 9.62] | 3.85 [0.62, 7.76] |
| ADNI MCI36 | 18.54 [9.37, 28.50] | 4.79 [1.24, 8.36] | 3.33 [0.00, 9.17] |
Removing either prediction path worsens the point estimate on all eight tasks. The paired interval lies above zero on five tasks for the private path and seven for the common path. The revised paper works through the two-removal example in §3.5 and reports the paired intervals and resampling protocol in Appendix C (Table 25), referenced from the Table 3 analysis.
Thank you for the thorough review and for pointing us to related work on modality interactions. Your suggestions helped us improve the paper. We respond to each point below and have uploaded a revised PDF with the changes marked in blue. Table and section numbers refer to the revised PDF.
Question 1a. Why use agreement to guide collaboration? When experts disagree, would it be better to follow the most confident expert?
Lluna uses agreement across training examples to weight interactions in the common path. When one expert disagrees with the others on an example, the private path still gives the prediction direct access to that expert’s features. Table 3b tests this choice. The agreement-based graph improves Mean-Missing over a uniformly weighted graph on seven of eight tasks and ties on ADNI.
To test whether following the most confident expert works better when experts disagree, we first identify views where the reference experts for at least two available modalities predict different answers. For each such view, we compare Lluna’s answer with that of the most confident available expert from the full expert pool, after calibrating each expert on validation data. Both answers are scored against the same label. The table reports accuracy (%) and the paired difference (Lluna minus expert, in pp), with incomplete views pooled.
With missing inputs, Lluna gains 12.14 pp on MELD and 21.84 pp on CirCor over choosing the most confident expert. Daily-Omni has only one modality left after a removal, so there are no incomplete views with a disagreement between modalities.
The second table focuses on conflicts where the selected expert was highly confident. We split those records by whether the expert was correct, then count how often Lluna kept a correct answer or corrected a wrong one. On MELD with missing inputs, Lluna kept 43 of the expert’s 46 correct answers and corrected 31 of its 52 errors.
The revised paper clarifies the role of agreement in §3.2 and adds both conflict analyses and their subset definitions to Appendix E.1 (Table 46).
Question 1b. Why keep the compatibility graph fixed during training and inference when relationships between experts can vary across tasks and examples?
We fit a separate graph for each task using its training examples. For example, the MELD graph has only two links across modalities, while the ADNI graph retains all 36 expert pairs. Within a task, the fitted edges stay fixed so that two experts can still exchange useful cues on a sample where their predictions disagree. Each sample’s available experts determine which edges are active, and the message gates and pooling weights determine how much each expert contributes. The revised §3.2 makes this distinction explicit and notes learning sample-specific edges as future work.