acceptodds
Under review as a conference paper at ICLR 2027

Lluna: Collaborative Unimodal Models for Missing-Modality Inference

Abstract

Pretrained unimodal models offer a flexible route to multimodal prediction through collaboration, but this collaboration can become less accurate than a surviving unimodal model when modalities are missing. We study this setting using fixed surviving-model anchors and find that collaborative gains can weaken or reverse as inputs disappear, even when the anchor's own input is unchanged. Our diagnostics further reveal reduced recovery of modality-specific information after aggregation and state-dependent collaborative representations. We propose Lluna, which combines a direct private prediction path from adapted unimodal features with a common graph-based collaboration branch that provides a residual correction. Progressive Collaboration Consistency organizes richer-to-poorer guidance along single-modality removals, aligning normalized common representations while task supervision adapts predictions to the available evidence. Across eight datasets spanning affective computing, scene understanding, and medical prediction, \luna achieves the best evaluated average performance over non-full input states, improving Mean-Missing by 4.26% on average relative to the strongest evaluated baseline. The fixed-anchor diagnostic also shows positive gains in mean collaboration under minimal inputs on all eight tasks. Code is available at https://anonymous.4open.science/r/Lluna.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Unimodal Model, Missing-Modality, Model Collaboration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WGymOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Lluna, combining private prediction paths from unimodal features with residual common collaboration and progressive cross-state consistency. Evaluation on three domains show an improvement in Mean-Missing performance.

Critical strengths

Overall, the paper was clear to read and easy to follow. The experiments were quite extensive spanning a range of domains and comparing a range of past methods. The code was also shared, which will aid reproducibility for future work. The proposed method is an interesting direction exploring collaborative unimodal models and will be of interest for the community.

Critical weaknesses

Originality and significance: The motivation behind the desired collaboration strategy is unclear. The paper focuses on compatibility, where it encourages agreement in task predictions. But it needs to discuss on why this is the optimal choice. For instance, it is likely that in the cases of conflict one modality predictor might be correct and confident and that should be the one that should be considered? The paper should also discuss on why the graph should remain fixed during training and inference as it is possible that the relationship between modalities and their contribution to a task can differ across tasks.

The paper further lacks a discussion with prior work [1,2,3] that have considered a decomposition of contributions of individual modalities and their interaction on a task, which makes the positioning challenging.

The paper should also discuss why modality-specific signals become less accessible after collaboration, what is a private base prediction and why is it required. For missing modality, the dependence and constraint of removal of exactly one modality should be discussed further as well. I'd also suggest to further elaborate the problem in section 3.5. Without confidence intervals, its also challenging to see the contributions of individual components in Table 3.

I'd also suggest to discuss the scenarios and application of the method when missingness is not at random and when multiple modalities are missing.

Clarity: The overview section is challenging to comprehend without the introduction of terms. For instance, collaboration path or private prediction path, compatibility graph are not self-evident terms to provide an overview for the readers. The result and ablation tables can be enhanced for readability. Currently it contains a lot of numbers, which can be hard for readers to interpret. I'd suggest to have the detailed results in appendix while highlighting the takeaways in the main paper. The meaning of underlined results is unclear as well. Further, how were the hyper-parameters in eq 3 were chosen? Did the authors conduct an ablation analysis on the same? Minor, but why is the method called Lluna with a double "l"?

[1] Fang et al. Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition.
[2] Madaan et al. Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning.
[3] Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.

Questions for authors

Please refer to my comments in the weakness section above.

8 comments

Thank you for the thorough review and for pointing us to related work on modality interactions. Your suggestions helped us improve the paper. We respond to each point below and have uploaded a revised PDF with the changes marked in blue. Table and section numbers refer to the revised PDF.


Question 1a. Why use agreement to guide collaboration? When experts disagree, would it be better to follow the most confident expert?

Lluna uses agreement across training examples to weight interactions in the common path. When one expert disagrees with the others on an example, the private path still gives the prediction direct access to that expert’s features. Table 3b tests this choice. The agreement-based graph improves Mean-Missing over a uniformly weighted graph on seven of eight tasks and ties on ADNI.

To test whether following the most confident expert works better when experts disagree, we first identify views where the reference experts for at least two available modalities predict different answers. For each such view, we compare Lluna’s answer with that of the most confident available expert from the full expert pool, after calibrating each expert on validation data. Both answers are scored against the same label. The table reports accuracy (%) and the paired difference (Lluna minus expert, in pp), with incomplete views pooled.

DatasetFull: expert / LlunaFull differenceMissing: expert / LlunaMissing difference
MELD48.60 / 62.84+14.2445.28 / 57.42+12.14
UR-FUNNY v271.52 / 73.46+1.9467.31 / 68.69+1.38
Daily-Omni20.97 / 24.19+3.23——
WorldSense30.16 / 30.160.0029.55 / 30.17+0.62
CirCor65.63 / 87.50+21.8864.08 / 85.92+21.84
ADNI MCI3684.62 / 84.620.0076.92 / 82.69+5.77

With missing inputs, Lluna gains 12.14 pp on MELD and 21.84 pp on CirCor over choosing the most confident expert. Daily-Omni has only one modality left after a removal, so there are no incomplete views with a disagreement between modalities.

The second table focuses on conflicts where the selected expert was highly confident. We split those records by whether the expert was correct, then count how often Lluna kept a correct answer or corrected a wrong one. On MELD with missing inputs, Lluna kept 43 of the expert’s 46 correct answers and corrected 31 of its 52 errors.

High-confidence conflict subsetInputRecords / unique examplesCorrect expert answers keptExpert errors corrected
MELDFull67 / 6735/3717/30
MELDMissing98 / 6743/4631/52
UR-FUNNY v2Full61 / 6146/505/11
UR-FUNNY v2Missing77 / 6156/607/17

The revised paper clarifies the role of agreement in §3.2 and adds both conflict analyses and their subset definitions to Appendix E.1 (Table 46).


Question 1b. Why keep the compatibility graph fixed during training and inference when relationships between experts can vary across tasks and examples?

We fit a separate graph for each task using its training examples. For example, the MELD graph has only two links across modalities, while the ADNI graph retains all 36 expert pairs. Within a task, the fitted edges stay fixed so that two experts can still exchange useful cues on a sample where their predictions disagree. Each sample’s available experts determine which edges are active, and the message gates and pooling weights determine how much each expert contributes. The revised §3.2 makes this distinction explicit and notes learning sample-specific edges as future work.


Question 2. How does Lluna relate to the work of Fang, Madaan and Liang on individual modality contributions and cross-modal interactions?

Fang et al. [1] and Liang et al. [3] study how modalities contribute to a prediction. Madaan et al. [2] explicitly combine multiple predictors. Our question is how independently pretrained unimodal experts can continue to collaborate when only some of their inputs are available.

Information decomposition. Fang et al. [1] separate the contributions of audio and video to a multimodal language model’s decision given the language prompt. Their analysis finds a stronger reliance on vision even on tasks designed to use both. Liang et al. [3] link the drop in accuracy after a modality is removed to the unique information it carried and, when that contribution is small, to information available only jointly. Both works measure how a model uses its modalities. Lluna builds this distinction into the predictor: each surviving expert’s own cues reach the prediction through the private path, and the common path uses the interactions that remain possible. Appendix D.2 applies a related analysis to Lluna itself, with Shapley values for modality contributions and pairwise interactions.

Combining unimodal and joint predictors. Madaan et al. [2] combine one classifier per modality with a classifier over both inputs as a product of experts, which adds their log-probabilities. Lluna differs in three respects. (1) Missing inputs. Their model is trained and evaluated with every modality present, and its joint classifier takes both inputs. Lluna trains a single predictor that handles every nonempty input subset. The private head reads whichever experts are available, and the common branch passes messages only among them. (2) Weighting. A product of experts weights its terms equally. Lluna treats the common prediction as a correction with a learned scale. Fixing that scale to one, which weights the two terms equally, lowers Mean-Missing on six of eight tasks, by 4.70% on average (Table 3a). (3) Consistency across input states. PCC aligns the common representation of each state with that of its neighbor with one more input. Removing PCC lowers Mean-Missing by 2.27% on average (Table 3c).

An interaction-aware baseline. I²MoE (Xin et al., 2025), which assigns separate experts to different types of modality interaction, is already among our baselines. With missing-input training on the shared expert pool, Lluna outperforms it on all three tasks in Table 2: MAE 0.3421 vs. 0.3731 on CH-SIMS v2, accuracy 33.82% vs. 30.27% on WorldSense, and balanced accuracy 84.17% vs. 77.71% on ADNI. It is also one of the seven baselines in Fig. 4.

The revised paper adds these comparisons to §2 and links them to the analysis in Appendix D.2.

[1] Wanlong Fang et al. Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition. ICML, 2026.
[2] Divyam Madaan et al. Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning. NeurIPS, 2024.
[3] Paul Pu Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework. NeurIPS, 2023.


Question 3a. Why can modality-specific cues become harder to recover after collaboration, and how does the private base prediction help?

Pooling compresses several experts’ features into one representation. A detail carried by one modality, such as tone of voice, can become harder to read back from it. In Fig. 1(b), we first predict a modality’s features from the other modalities and take the part left unexplained as our target. We then measure how well a linear probe recovers that same target from the attention-pooling model. Across eight tasks, the probe’s mean R2R^2 falls from 58.52% with the modality alone to 33.47% with all modalities pooled. The lower score means that the probe can explain less of the variation in the modality-specific features after pooling.

Lluna’s private head makes a base prediction directly from the surviving experts’ features, before pooling. A cue such as tone can therefore affect the answer even if it is hard to recover from the shared representation. The revised paper defines the base prediction in §3.3 and explains what the probe measures in the Fig. 1(b) caption.


Question 3b. Can Lluna handle several missing modalities at inference? Why does each PCC training step remove exactly one modality?

Yes. Lluna can predict when several modalities are missing at the same time. It processes the observed inputs in one forward pass. Table 6 reports all three ways that two modalities can be missing on MELD. The same trained model handles each input combination, and scores are averaged over five runs.

Available inputInputs missingLluna W-F1 (%)Strongest baselineBaseline W-F1 (%)
TextAudio and video63.60LIMSSR61.72
AudioText and video53.63LIMSSR51.33
VideoText and audio37.86SimMLM36.81

During training, a view with little evidence receives guidance from a teacher with just one additional modality. For example, the text–audio state receives guidance from the full text–audio–video state and in turn guides the text-only state. This breaks one large change in evidence into two smaller ones. Full-to-Minimal also uses a teacher that sees all the student’s inputs, but it asks the text-only student to match the full state directly and gives the intermediate state no consistency target. Fig. 1(c) supports the intuition that adjacent targets are closer. Across the six three-input tasks, the mean cosine distance is 0.279 for adjacent states and 0.456 from full to minimal input. In Table 3c, adjacent PCC outperforms Full-to-Minimal on all seven tasks where the pair sets differ. The revised §3.4 uses this example to explain how progressive guidance works during training.


Question 3c. What does the stability result in §3.5 establish, and how uncertain are the component effects reported in Table 3?

Consider the same clip with all three inputs, then with video removed, and finally with only text left. Section 3.5 shows that the distance between the first and last normalized common representations cannot exceed the distance moved in the first step plus the distance moved in the second. If PCC keeps both changes small, the text-only representation stays close to the full-input representation even though training only aligns neighboring states.

We computed paired 95% bootstrap intervals for fixed, archived full-model and ablation checkpoints. Each of 10,000 resamples keeps a sampling unit and all its views together. Positive differences favor the full model. Units are MAE × 100 for SIMS/MOSI and pp elsewhere.

Datasetw/o Privatew/o Commonw/o PCC
CH-SIMS v21.14 [0.50, 1.77]1.31 [0.87, 1.77]0.66 [0.25, 1.06]
CMU-MOSI3.07 [−0.09, 6.09]3.07 [1.56, 4.61]1.53 [0.75, 2.47]
MELD0.59 [0.06, 1.12]0.92 [0.37, 1.47]0.11 [−0.33, 0.55]
UR-FUNNY v22.60 [1.32, 3.87]0.99 [0.54, 1.46]0.60 [0.35, 0.87]
Daily-Omni0.85 [−4.47, 6.01]2.26 [−3.32, 7.65]0.00 [0.00, 0.00]
WorldSense1.32 [−0.47, 3.12]1.57 [0.47, 2.68]1.67 [0.20, 3.14]
CirCor10.69 [1.37, 21.27]4.44 [0.08, 9.62]3.85 [0.62, 7.76]
ADNI MCI3618.54 [9.37, 28.50]4.79 [1.24, 8.36]3.33 [0.00, 9.17]

Removing either prediction path worsens the point estimate on all eight tasks. The paired interval lies above zero on five tasks for the private path and seven for the common path. The revised paper works through the two-removal example in §3.5 and reports the paired intervals and resampling protocol in Appendix C (Table 25), referenced from the Table 3 analysis.

Reviewer_C1puOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors present Lluna, a collaboration approach for multimodal inference that improves performance in the presence of missing modalities via: (1) preserving modality-specific predictive information, and (2) updating these predictions via common representations that capture cross-modal interaction information via a fitted compatibility graph. The work introduces Progressive Collaboration Consistency (PCC) to encourage consistency among common representations by aligning common representations of poorer (leave-one-modality-out) modality subsets to common representations of richer modality subsets. Results demonstrate strong performance across eight datasets spanning three domains compared to several baselines, and ablation experiments show the benefit of several architectural choices. Further analyses demonstrate robustness under increased sparsity, improved consistency among common representations, the effect of varying some hyperparameters, and comparison of performance and model size across baselines.

Critical strengths

• Strong performance gains across a comprehensive set of baseline comparison and ablation experiments – shows that this method does consistently outperform other methods across several datasets, and analyzes how different model variations perform, confirming the model design choices as superior overall. Thorough set of experiments and results demonstrate performance of the model from several angles, strengthening the claims of this model’s advantages over other methods (including in terms of number of parameters and robustness to increasing missingness levels) and the contribution and effectiveness of each of its components.

• Several analyses demonstrate the components of Lluna are working as expected, for example, the reduction in common-state drift and visualization of the compatibility graphs of various datasets. These help to support other results, for example, the dense connections within ADNI reflect similar performance between Lluna and the uniform graph variant in the ablation study.

Critical weaknesses

• Minor clarifications needed to support the understanding of the methodology and the experiments underlying its results – see Questions.

• Question of what the major claims are regarding utility of collaboration vs. non-collaboration, and whether the baseline experiments were a fair comparison – do the authors claim that collaboration methods outperform other method types (such as missing-modality methods) in general, or that Lluna is, more specifically, an improvement upon collaboration methods? Also, more clarification needed on why certain decisions were made regarding the baseline methods and the potential impact of these choices on the results (such as using different encoding strategy for the omni models) – see Questions 3+4. Clarity on these points would help to support the overall contribution of this paper in the context of the wider multimodal integration field, and to provide more confidence in the reported baseline comparison results.

Questions for authors
  1. Please specify the input and output dimensions of gprivg_{priv} and gcomg_{com}. Does gprivg_{priv} output a single prediction per sample? If so, how does it aggregate predictions across each modality? Are features pip_i from each modality concatenated across the embedding dimension before input to gprivg_{priv}? This would help clarify how the different model components come together in equation (1) as well as the final dimensionality of y^\mathbf{\hat{y}}.

  2. Is there another normalization or transformation step within equation (1) on the entire sum between the gprivg_{priv} and gcomg_{com} terms, especially if the output is a probability distribution over classes? I’m wondering if this sum as presented in equation (1) could lead to invalid outputs, and if so, how the model makes sure the resulting sum remains in the valid domain.

  3. Please provide clarification on the discrepancy between using gcomg_{com} as a mapping from z\mathbf{z} to a residual correction, versus how it is used in the objective function which supervises the output of gcomg_{com} to match the true label. Why is it useful to have this auxiliary loss supervision of the common prediction, when it is used as a residual correction in the final model output (rather than a standalone prediction that’s pooled with the modality-specific prediction)?

  4. Did the authors investigate how Lluna’s approach of collaboration among frozen models would compare with missing-modality methods if they were not constrained to use the frozen models as encoders, as these non-collaboration methods do not necessarily rely on pre-existing, frozen foundation models as encoders? I may be misunderstanding, but it seems to me that in treating the missing-modality methods in this way, they almost become another collaboration method, just differing in how they combine the representations from the frozen encoders. Is the collaboration approach really better than non-collaboration approaches, or does Lluna just have exceptional performance over other collaborative frameworks? What is the benefit of using model collaboration frameworks over non-collaboration in general?

  5. Concern over fairness of the comparison against the omni models – the omni models’ backbones were frozen and were given the raw inputs rather than the features generated by the shared model pool used by the other methods – could this approach potentially confound the effectiveness of multimodal inference with merely the effectiveness of the initial frozen feature encoders?

  6. Please clarify what is shown in Table 1 and how it differs from the ‘Training with missing-input views’ section – does this mean that Table 1 results were evaluating models that were trained on a dataset restricted to complete samples only? A brief clarification in the text would be helpful.

  7. I’m curious why the all-pairs version of version of PCC tends to do worse than the adjacent-state PCC, as I had thought that the adjacent-state PCC was a more efficient proxy for the all-pairs version. Any intuition on why this might be the case?

  8. Please clarify in Figure 3a what ‘Same-level’ means.

4 comments

Thank you for reading the paper so closely. Your questions helped us clarify the method and the fairness of the baseline comparisons. We answer each question below and have uploaded a revised PDF with the changes marked in blue. Table and section numbers refer to the revised PDF.


Question 1. Does gprivg_{\mathrm{priv}} concatenate the expert features to make one prediction per sample? What are the input and output dimensions of both heads in Eq. (1)?

Yes. gprivg_{\mathrm{priv}} concatenates the NN experts’ adapted features pip_i along the feature dimension and makes one prediction per sample. With private feature width dpd_p and pooled common feature width dcd_c, gprivg_{\mathrm{priv}} maps Ndp+NNd_p+N inputs to DD outputs, and gcomg_{\mathrm{com}} maps dc+Nd_c+N inputs to DD outputs. The extra NN entries indicate which experts are available. For classification, the combined prediction y^\hat{\mathbf y} has one entry per class. Regression produces one scalar, and QA scores each candidate separately. The revised paper states these dimensions beside Eq. (1).


Question 2. How does Eq. (1) turn the sum of the two heads’ outputs into a valid class probability distribution?

For each class, Eq. (1) multiplies the common head’s score by tanh⁡(s)\tanh(s), where ss is a learned scalar, and adds the result to the private head’s score for that class. We then apply softmax once across the combined class scores to obtain nonnegative probabilities that sum to one. For multiple-choice QA, we apply softmax over the valid candidates. Regression uses the combined score directly. The revised paper spells out these steps beside Eq. (1).


Question 3. Why supervise the common head with the true label if it supplies a residual correction to the private prediction?

The true label gives the common head a direct learning signal for which cross-model interactions help predict the outcome. With only the final loss, the gradient reaching the common head through Eq. (1) is multiplied by tanh⁡(s)\tanh(s) and can be weak when this scale is small. The auxiliary loss supplies task supervision independent of that scale. The final loss still trains the combined prediction and determines how strongly the common scores should correct the private scores.

The revised paper explains the role of label supervision beside Eq. (3).


Question 4. What is the benefit of using a shared pool of frozen experts, and how does the comparison extend to missing-modality methods with their own trainable encoders?

Lluna reuses pretrained unimodal encoders and trains only the adapters, collaboration layers, and prediction heads. Independently trained experts can pick up different cues from the same example. Collaboration lets the prediction use those complementary cues whenever their inputs are available. In Table 1, GIML and MARS receive features from our frozen expert pool. For this comparison, we trained each method with its own native encoders, using the same data splits and missing-modality evaluation protocol as Table 1 on CMU-MOSI, MELD, and UR-FUNNY v2. Lluna outperforms both on all three tasks. Compared with MARS, the stronger of the two on each task, Lluna reduces MAE by 0.1192 on CMU-MOSI, improves weighted F1 by 9.40 percentage points (pp) on MELD, and improves accuracy by 20.27 pp on UR-FUNNY v2.

MethodCMU-MOSI MAE ↓MELD weighted F1 ↑UR-FUNNY v2 accuracy ↑
GIML with its original encoders1.249934.61%49.98%
MARS with its original encoders1.086346.70%50.35%
Lluna with the shared expert pool (Table 1)0.967156.10%70.62%

The revised paper reports this comparison in Appendix B.5 (Table 21(a)).


Question 5. Why do the frozen Omni models receive raw inputs, and how might their encoders affect the comparison?

An Omni model is pretrained to encode raw inputs through its own processors. We therefore give it the available raw modalities and keep its pretrained backbone intact. Feeding it the expert pool’s embeddings would require a new interface, and the result would also depend on how well that interface was trained. This design compares a unified pretrained model with a pool of unimodal experts on the same available sources.

Because encoder quality can affect the original comparison, we evaluated two ways of combining features from the same frozen Omni checkpoint. We used the Table 1 splits and missing-modality protocol for CMU-MOSI, MELD, and UR-FUNNY v2. In the native configuration, the Omni backbone fuses its front-end features and a task head reads out the joint representation. In the Lluna configuration, we take features before cross-modal fusion and train Lluna to combine them and predict the label.

Frozen Omni checkpointFusion methodCMU-MOSI MAE ↓MELD weighted F1 ↑UR-FUNNY v2 accuracy ↑
Qwen2.5-Omni-7BNative0.996739.58%53.91%
Qwen2.5-Omni-7BLluna0.856951.92%62.06%
InteractiveOmni-8BNative1.355626.52%51.36%
InteractiveOmni-8BLluna0.773052.21%59.46%

Lluna improves on the corresponding Omni task head on all three tasks with both checkpoints. On MELD, for example, using Lluna raises weighted F1 by 12.34 pp with Qwen2.5-Omni-7B and by 25.69 pp with InteractiveOmni-8B. The revised paper reports these comparisons in Appendix B.5 (Table 21(b)) and clarifies the encoder setup in §4.1.


Question 6. Which methods see incomplete inputs during training in Table 1, and how does that training differ from Table 2?

Table 1 starts from complete training examples. Whether a method sees incomplete views of those examples during training depends on its protocol. CLARGA, LIMSSR, and ShaSpec uniformly sample one nonempty input configuration per batch. SimMLM and Flex-MoE start with five epochs on complete inputs, then train on nested pairs of views and sampled incomplete views, respectively. MARS adaptively samples partial views, while GIML uses feature degradation. Logit stacking, Concat MLP, attention pooling, Soft MoE, FoE, and Set Transformer train on complete inputs, as do the frozen Omni models’ task heads (Appendix B.2, Table 16 and accompanying text).

Table 2 gives all ten comparison methods missing-input supervision on SIMS, WorldSense and ADNI. Lluna ranks first on all three tasks. The revised paper states each method’s training views where Table 1 is introduced (§4.1), and §4.2 describes the missing-input training behind Table 2.


Question 7. Why does adjacent-state PCC perform better than All-Pairs if it uses fewer pairs?

Adjacent PCC aligns states that differ by a single removal. When text–audio–video guides text–audio, the teacher sees every cue the student still has, including the speaker’s tone. All-Pairs can also align text–audio with text–video. If text–video guides text–audio, the teacher cannot hear the speaker’s tone, while its target may depend on a facial expression the student cannot see. Matching that target could discourage the student from using an informative audio cue. Adjacent PCC transfers guidance across larger gaps through successive removals.

Table 48 shows why the extra pairs in All-Pairs can change what PCC teaches. In the attention-pooling diagnostic, two views of the same example with different modalities but the same input count were about 2.8 times farther apart on average than views separated by one removal (mean cosine distance 0.752 vs. 0.270). All-Pairs therefore asks the model to close larger gaps, including ones that can reflect useful cues specific to either view. In the direct training comparison, All-Pairs performed worse than adjacent PCC on all eight tasks, with an average relative decline of 1.42% in Mean-Missing (Table 3c). The revised §4.3 connects this diagnostic and the ablation.


Question 8. What does “Same-level” mean in Fig. 3(a)?

“Same-level” means two views of the same example with the same number of available modalities but a different composition, such as text–audio and text–video. Each point shows how much their mean common-representation cosine distance decreases when PCC is enabled. These pairs are evaluated to measure how training on adjacent removal pairs affects other state pairs. The revised Fig. 3 caption defines “Same-level” with this example.

Thank you for the clarifying responses and for the additional results. I still have some reservations regarding the motivation behind the collaboration strategy in general, as opposed to non-collaboration such as missing-modality approaches, but the authors have performed a comprehensive set of experiments whose results appear strong, including comparisons against several alternative non-collaboration methods. My concerns have been addressed, and I will maintain my score.

Thank you very much for your continued support and positive rating. We greatly appreciate your recognition of our comprehensive experiments, strong baseline comparisons, and analyses. Your feedback helped us significantly improve the paper, and we are glad our responses addressed your concerns.

Reviewer_1k66OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies how pretrained unimodal models can collaborate effectively for multimodal prediction when some modalities are missing. The authors propose LLuna, which engineers several components together in a new way to preserve modality-specific information, allow collaboration between different models, and keep consistent representations between different missing-modality settings. Their reported results are strong and outperform the evaluated baselines across eight datasets, with an average relative improvement of 4.26% over the strongest baseline on each task.

Critical strengths
  1. The authors report strong improvements over the strongest evaluated baseline on eight datasets in different application domains.

  2. Separating direct predictions from cross-model collaboration is well motivated. The general idea that collaboration should help refine instead of replacing the information from individual models is coherent and reasonable when modalities can disappear.

Critical weaknesses
  1. A large part of the contribution is primarily in the combination of pre-existing methods rather than the novelty of individual components, and the main rather novel aspect of the method is PCC.

  2. The authors claim that conventional model collaboration can become detrimental as modalities disappear. But a decrease in collaborative gains seems rather obvious when complementary information is removed, and the negative gains in Figure 4 are mainly shown for when only one modality remains (instead of when multiple inputs remain available),while the analysis averages seven baseline methods and could hide the behavior of stronger individual methods.

Questions for authors
  1. Could the authors clarify the methodological novelty of LLuna beyond the combination of existing techniques?

  2. Could the authors provide the individual collaboration gains for the seven baselines averaged in Figure 4? It would be useful to understand whether the reported negative gains under missing modalities are consistent across methods, or are primarily driven by particular methods.

  3. Could the authors explain the motivation behind choosing different evaluation metrics (BAcc, Acc, and W-F1) for the classification tasks in Table 1? The choice of evaluation metrics in table 1 is not currently justified properly for the classification tasks, and it is not clear whether the reported superiority holds under alternative relevant metrics.

7 comments

Thank you for your careful review. Your questions on Fig. 4 and on the choice of metrics prompted additional analyses that improved the paper. We answer each question below and have uploaded a revised PDF with the changes marked in blue. Table and section numbers refer to the revised PDF.


Question 1. What is methodologically new in Lluna beyond combining existing techniques?

What is new is a finding about collaboration under missing inputs and three design choices that follow from it. Each choice is tested against its conventional alternative in Table 3 or Table 22.

We study missing-modality inference as a collaboration problem among independently pretrained experts. Collaboration can become less accurate than a surviving expert alone, even though that expert’s own input is unchanged (Fig. 4; see Question 2). Lluna is built around this finding: the surviving experts’ own evidence makes the prediction, and their interactions are only allowed to correct it.

1. A private base prediction with a scaled correction. Fusion methods usually predict from the pooled representation. In Lluna, a private head predicts directly from the available experts’ features, and the common branch adds a correction through a learned scale. Predicting from the common branch alone lowers Mean-Missing on all eight tasks (−6.53% on average, Table 3a). So does concatenating the two branches (Table 22). Fixing the scale to one lowers it on six of eight (−4.70%).

2. A common branch for experts that can drop out. A shared aligner maps the heterogeneous experts into one space. Each expert receives messages only from available neighbors, incoming weights are renormalized over them, and an expert left on its own keeps a self-loop. Edge weights come from the experts’ agreement on the task’s training examples. This graph outperforms uniform weights on seven tasks and ties on ADNI (Table 3b).

3. PCC. You identified PCC as the most novel part. It makes three choices, each with a matched ablation (Table 3c–d): (a) Adjacent states only. PCC aligns states that differ by one removal, so an intermediate state learns from its richer neighbor and then guides its poorer one. Aligning the full state directly with each single-input state is worse on all seven tasks where the pair sets differ (−2.20% on average). Aligning every pair of states is worse on all eight (−1.42%). (b) Common representation only. PCC aligns the direction of the common representation and leaves the private path free to use cues specific to the observed inputs. Aligning the private representation instead, or both, is worse on all eight tasks (−1.40% and −1.88%). (c) One-way guidance. The richer state is a gradient-stopped target. Letting gradients reach both states is worse on seven tasks and ties on ADNI (−5.58%).

Related methods. ShaSpec [1] learns features shared across modalities, uses them to estimate a missing modality’s embedding, and predicts from the fused features. Lluna does not reconstruct the missing input. It predicts from the experts that remain. CLARGA [2] builds a graph among modalities from each sample’s features. In Lluna, the nodes are independently pretrained experts, and their agreement on the task’s training examples determines how strongly they exchange information.

The revised paper adds these comparisons to §2 and states the design choices and their ablations in §3.

[1] Hu Wang et al. Multi-Modal Learning With Missing Modality via Shared-Specific Feature Modelling. CVPR, 2023.
[2] Santosh Patapati and Trisanth Srinivasan. Multimodal Graph Representation Learning over Arbitrary Sets of Modalities. WACV, 2026.


Question 2. How do the seven baselines in Fig. 4 behave individually, and can collaboration underperform a surviving expert while two inputs are still available?

The decline in collaborative gain holds for each baseline individually. With two inputs remaining, each of the seven baselines gains less than with full input on at least six of the seven eligible tasks (45 of 49 comparisons).

Collaboration can also become less accurate than the fixed expert even though that expert’s own input is unchanged. With two inputs remaining, this already happens in some cases. Relative to the fixed expert, logit stacking on WorldSense goes from +1.60 to −2.44 percentage points (pp) and Concat MLP on CirCor from +1.42 to −9.94 pp, while Lluna’s gain stays positive on both tasks (+4.04 and +5.02 pp). With one input remaining, each baseline has negative gains on at least two tasks. Lluna retains a positive gain on all eight (+1.70 to +11.13 pp). Its gain is the highest among all methods on all seven eligible tasks with two inputs remaining and on seven of eight tasks with one.

No single baseline drives the average in Fig. 4. Omitting each of the seven baselines in turn preserves the decline in the mean gain and Lluna’s advantage at both endpoints on every task (Appendix E.1).

The table reports accuracy differences in pp, averaged over eligible expert–view pairs.

DatasetInputs leftLogit stackConcat MLPAttn. poolSoft MoEI²MoECLARGASet Trans.Lluna (ours)
CH-SIMS v23 (full)+11.03+8.41+8.70+11.70+11.41+10.16+9.09+10.93
2+6.71+5.25+4.93+2.64+8.00+8.22+6.45+9.09
1−2.71+0.77−2.26−4.55+2.61+2.64+1.55+4.48
CMU-MOSI3 (full)+20.75+16.23+21.91+22.21+13.31+13.61+8.50+23.23
2+9.43+9.57+4.42+12.20+11.66+11.03+5.59+21.48
1−2.14+0.78−10.30+0.44+4.28+3.26+3.16+11.13
MELD3 (full)+15.11+14.46+17.45+17.45+17.68+16.10+15.80+19.79
2+7.55+8.21+8.58+7.82+14.34+13.42+12.82+16.31
1−5.42−1.90−2.99−6.42+7.72+8.09+8.26+8.93
UR-FUNNY v23 (full)+12.54+9.73+10.73+11.54+7.51+6.41+6.71+14.05
2+6.20+4.02+5.50+2.01+5.40+3.99+3.96+9.76
1−0.50−5.03−0.60−7.44−0.87−1.38−0.67+3.29
Daily-Omni2 (full)−1.41+1.98−1.41+1.98−0.85+1.98+3.11+4.24
1−7.34−2.54−4.52+1.69−2.26−0.28+0.57+5.93
WorldSense3 (full)+1.60+2.23+2.23+2.23+1.81+3.28+1.60+3.70
2−2.44+0.56+1.32+0.84+0.70+1.74+1.19+4.04
1−6.42−0.56−0.14−0.42−0.98−0.21+0.35+2.86
CirCor4 (full)−64.49+1.42+3.69−0.85+3.69+1.42−64.49+5.97
2−64.49−9.94+2.37−1.61+1.61+2.75−64.49+5.02
1−60.23−14.77−9.09−5.11+1.14+1.42−64.49+1.70
ADNI MCI363 (full)+11.33+5.33+11.33+9.33+5.33+5.33+15.33+15.33
2+8.67+3.33+8.67+4.67+6.00+2.00+9.33+13.33
1−0.67−0.670.000.00+1.33+4.67+4.00+3.33

The revised paper adds this per-method breakdown to Appendix E.1 (Table 38(b)) and states alongside Fig. 4 how often gains decline and how often collaboration underperforms a surviving expert.


Question 3. Why does Table 1 use different classification metrics, and does Lluna still lead under alternative metrics?

We follow the MELD benchmark in reporting support-weighted F1 for seven-class emotion recognition [3] and UR-FUNNY in reporting accuracy for binary humor detection [4]. On Daily-Omni [5] and WorldSense [6], accuracy measures how often the model selects the correct answer. For CirCor and ADNI, we use balanced accuracy so that sensitivity and specificity count equally despite class imbalance. Prior work on ADNI conversion prediction uses the same metric [7].

We rescored the same predictions under alternative metrics. Lluna retains the highest Mean-Missing score in all five comparisons, so its better performance persists when the evaluation metric changes.

DatasetAlternative metricLluna (mean ± SD)Strongest baselineBaseline scoreGain (pp)
MELDAccuracy58.746 ± 0.243LIMSSR57.885+0.861
MELDMacro-F137.543 ± 0.415LIMSSR34.479+3.064
UR-FUNNY v2Macro-F170.572 ± 0.240LIMSSR68.065+2.507
CirCorAccuracy89.334 ± 1.602GIML88.945+0.390
ADNI MCI36Accuracy86.733 ± 1.588GIML82.200+4.533

The revised paper explains these choices in §4.1 and reports the alternative-metric results in Appendix A.3 (Table 12).

[3] Soujanya Poria et al. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ACL, 2019, Table 11.
[4] Md Kamrul Hasan et al. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. EMNLP-IJCNLP, 2019, Table 4.
[5] Ziwei Zhou et al. Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities. arXiv:2505.17862, 2025.
[6] Jack Hong et al. WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs. ICLR, 2026.
[7] Jonathan Young et al. Accurate multimodal probabilistic prediction of conversion to Alzheimer's disease in patients with mild cognitive impairment. NeuroImage: Clinical, 2013.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.