acceptodds
Withdrawn from ICLR 2027

CoMoR: A Conflict-Aware Modality Routing Framework for Multimodal Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal learning combines complementary evidence from heterogeneous sources, but high confidence in each modality's prediction does not guarantee agreement across modalities. Therefore, a general challenge is to determine how modality-specific judgments should influence joint prediction. We propose CoMoR, a Conflict-Aware Modality Routing framework. The framework uses these judgments, their disagreement, and uncertainty to guide multimodal integration. CoMoR does not combine diagnostic predictions into a final answer. It uses them to adjust the audio-visual inputs, which a frozen large language model then processes together with the original text to predict sentiment. We evaluate CoMoR on four widely used English and Chinese sentiment benchmarks. Ablations on CMU-MOSI and CH-SIMS support the use of diagnostic information and sample-specific routing. Among the methods compared, CoMoR obtains the lowest mean absolute error on all four datasets and leads on most reported metrics, with competitive results on the remaining ones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Frozen Large Language Models, Conflict-Aware Modality Routing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jUHpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CoMoR, a conflict-aware modality routing framework for adapting frozen large language models to multimodal sentiment analysis. Shared Sentiment Diagnosis produces modality-specific distributions over ordered sentiment anchors using a shared frozen LLM and a shared trainable head. A router receives these distributions, their pairwise Wasserstein-1 distances, and predictive entropies, and scales audio-visual pseudo-token blocks before joint prediction with the original text. Training combines a generative task loss with auxiliary diagnostic supervision. Experiments cover four English and Chinese sentiment benchmarks, with component ablations, alternative routing designs, multiple backbones, and computational analyses.

Critical strengths
  1. A coherent approach with promising empirical improvements. CoMoR uses modality-specific predictions to modulate continuous inputs while preserving the original evidence for joint prediction. The reported MAE improves over the reproduced MSE-Adapter baseline on all four datasets, including 0.672 to 0.581 on CMU-MOSI and 0.395 to 0.349 on CH-SIMS. Improvements across English and Chinese benchmarks make the approach practically interesting.
  2. Useful controls and detailed implementation disclosure. The paper evaluates shared versus independent diagnostic heads, alternative disagreement measures, uniform and learned global gates, and feature-driven routing. The appendix reports variability across training seeds and specifies prompts, masking, gradient flow, and numerical decoding. These details make the method easier to assess and reproduce.
Critical weaknesses
  1. The evidence connecting the diagnostics to effective conflict-aware routing remains limited. All diagnostic branches are supervised using the same multimodal label. Moreover, the text diagnostic branch pools non-contextual token embeddings before the LLM, making it invariant to token order (Appendix A.1). Consequently, disagreement can reflect limitations of the diagnostic branches rather than useful differences in sentiment evidence. The paper appropriately acknowledges that these distributions cannot identify the correct modality, but stronger empirical validation is needed to establish when they provide useful routing signals. Table 13 offers supporting evidence, although it uses one seed and groups defined by the full model itself. The case study also does not isolate the effect of routing, since the final LLM receives the original ordered text and may correct the misleading diagnosis through that path.
  2. The comparisons do not fully establish the contribution relative to recent LLM adaptation methods or its computational cost. The reproduced MSE-Adapter comparison is valuable, but Table 1 omits the cited Prototype-as-Prompt and MGSI methods, which are directly relevant to the paper’s positioning. Existing routing controls support the design, but do not provide a parameter- or compute-matched comparison with a stronger adapter baseline. Relative to MSE-Adapter, trainable parameters increase from 2.61M to 6.72M, training time increases by approximately 1.95 times, and inference latency increases by 21.8%. These costs make stronger comparisons important. Furthermore, Table 2 compares CoMoR across backbones without the corresponding baseline on each backbone, establishing compatibility more directly than consistent benefits from the proposed mechanism.
Questions for authors
  1. Diagnostic validity: Can the modality-specific annotations available in CH-SIMS or CH-SIMS v2 be used for evaluation only to examine whether diagnostic disagreement corresponds to independently annotated sentiment differences? This would help distinguish useful modality disagreement from diagnostic errors without changing the training protocol.
  2. Routing mechanism: Does the high-conflict advantage in Table 13 persist across the other training seeds? A fixed-checkpoint comparison with unit audio-visual scaling or shuffled diagnostic descriptors would also help isolate the contribution of sample-specific routing. 3.Comparisons: Are comparable results available for Prototype-as-Prompt and MGSI under the same splits, features, and metric conventions? If their settings are incompatible, please clarify the specific differences. Is there evidence that the improvement persists against an adapter baseline with a comparable parameter or training-compute budget?
  3. Backbone dependence: For the alternative LLMs in Table 2, what is the gain over the corresponding MSE-Adapter or uniform-gate baseline? These comparisons would materially affect the assessment of effectiveness across backbones.
Reviewer_5n5uOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper a conflict-aware modality routing framework for MSA. Shared Sentiment Diagnosis produces modality-specific sentiment distributions, while a router uses these distributions, pairwise disagreement, and predictive entropy to scale audio-visual pseudo-tokens before joint prediction. Experiments on four datasets shows improvements over the compared methods.

Critical strengths
  1. Evaluation covers four benchmarks, with consistent MAE improvements and ablations examining diagnostic-head sharing, routing strategies, and conflict measures.
  2. The distinction between diagnostic prediction and final joint prediction is conceptually clear. Diagnostic distributions guide input adaptation while preserving richer features for subsequent multimodal reasoning.
Critical weaknesses

The use of LLMs in the framework requires further justification. for example, the audio-visual input design, the extent to which the frozen backbone’s pretrained capabilities are utilized, and the advantages over pretrained LLMs remain unclear. Please see the questions below.

Questions for authors
  1. What advantages does CoMoR offer over directly using a pretrained multimodal LLM, such as Qwen2.5-Omni-7B?

  2. Why does the audio-visual pipeline rely on provided LSTM features rather than stronger pretrained encoders (like ViT-based visual models and Whisper) ? While fixed features facilitate comparison with conventional MSA methods, this choice seems unnecessarily restrictive for an LLM-based framework.

  3. What extent does CoMoR exploit the backbone’s general pretrained knowledge?

  4. Why are more recent LLM backbones, such as Qwen3-1.7B and Qwen3-0.6B not considered?

Reviewer_Nx32OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CoMoR, a method for deciding how much each modality should contribute when a frozen large language model is used for multimodal sentiment analysis. It builds on the MSE-Adapter setup with ChatGLM3-6B and adds two components. The first asks the frozen LLM for a separate sentiment estimate from each of text, audio and video, using a small shared head trained with an auxiliary loss. The second looks at how much these estimates disagree and how uncertain each one is, and turns this into sample-specific weights that scale the audio and visual inputs before the LLM produces the final prediction. Experiments on sentiment datasets show gains over a rerun of MSE-Adapter, with the clearest improvements in MAE.

Critical strengths
  1. Detailed and mostly honest reporting. The appendix gives exact prompts, the decoding and parsing rules, the masking rules, and the cost numbers. The appendix is upfront about several limitations: the text branch is order-invariant, W1 and entropy add no information beyond the distributions, and conflict does not identify the correct modality. That makes the paper much easier to assess.
Critical weaknesses

Reproducibility! Source code is not provided, which makes it hard to verify the claims independently. Besides that the 2 critical weaknesses in my view are:

  1. The evidence does not show that conflict-aware routing is what drives the gains

This is the main issue for me. The paper's story is that diagnostic distributions plus disagreement plus entropy drive sample-specific routing, and that this is why CoMoR works. But the evidence and numbers point somewhere else.

  • Most of the gain over MSE-Adapter is already there with no routing at all. "Uniform gates" bypasses the router entirely but keeps everything else, including the auxiliary diagnostic loss. It gets MOSI MAE 0.600 vs 0.672 for MSE-Adapter, and CH-SIMS 0.363 vs 0.395. That is about 75% of CoMoR's total MAE improvement. So the main driver is something other than routing. My guess is the auxiliary loss, which pushes the AV pseudo-tokens through the frozen LLM under a sentiment prompt and acts like a Self-MM style (AAAI 2021) unimodal multi-task signal. But this is not isolated, and the paper credits the improvement to the routing idea.
  • The routing-specific gaps are within seed noise. I ran Welch tests based on the means and stds in Tables 9 and 11 and p-values are about what chance would give. Some patterns are also non-monotone. On CH-SIMS, removing both C and U gives a lower MAE than removing only U. Statements like "dropping either C or U degrades both metrics" and "the results suggest that using explicit summaries ... works better ..." are not fully supported. Three seeds is also simply too few for gaps analysis and no significance tests are reported anywhere.
  • The routing mechanism is weak, and nothing shows it does much at test time. The paper reports no distribution of the gains over the test set, no relation between the gains and C or U, and no test-time intervention, such as forcing the gains to 1 on the trained full model. Without these, "routing" and "contribution" are labels rather than demonstrated behavior and what the router actually learned is never shown.
  • The case study does not support the story. In Fig. 3 the text diagnosis is the wrong one (positive for a clearly negative sentence). The visual diagnosis leans negative. Yet the router shrinks both AV blocks, so the implied text weight goes up. The correct final answer most likely comes from the LLM reading the raw sentence, which is unambiguous. No counterfactual is given, for example what uniform gates would predict for this sample.
  • What is called "conflict" is mostly unimodal predictor error. All three diagnostic branches are trained on the same multimodal label, and the text branch is an order-invariant bag of embeddings, so the text diagnosis is crippled. That is probably why it says +1 for "It reached the new heights of horrendous". The T-A and T-V distances then largely measure how bad the text branch is, not real cross-modal conflict. CH-SIMS ships unimodal labels, which would allow a direct check of whether CmnC^{mn} tracks ∣ym−yn∣|y^m - y^n|, but they are not used. In Table 13, the high-conflict group has the lowest MAE for both models. That contradicts the motivation that conflicting samples are the hard ones. It is also a single seed with no CI.
  1. Incremental novelty, and SOTA claims need narrower scoping

The building blocks are all known. The projector comes from MSE-Adapter. Auxiliary unimodal heads trained alongside the main prediction appear in Self-MM and in the multi-task baselines of the CH-SIMS paper. Two-hot ordinal targets are used in MuZero and DreamerV3, and the closed-form CDF expression for 1D W1 is standard. Sample-level dynamic fusion from unimodal confidence or uncertainty has been studied in QMF, Predictive Dynamic Fusion (Cao et al., ICML 2024), DynMM (Xue and Marculescu, CVPR 2023), and more. Learned gates on prefix or adapter tokens appear in LLaMA-Adapter (zero-init gating). What is new is the combination: unimodal readouts through the same frozen LLM feeding a scalar input gain. Several of the closest works (PDF, DynMM, RCML, LLaMA-Adapter gating) are not cited or discussed. The "multimodal learning" framing is also not tested. The title and abstract present a general framework, but only sentiment regression is evaluated, and the W1 conflict measure needs ordered labels, so it does not carry over directly to nominal classification tasks, which are where most of the conflict-aware fusion literature is evaluated.

The empirical claims also need narrower scoping. The paper cites MGSI and follows its protocol, but leaves it out of Table 1. With the same ChatGLM3-6B, MGSI reports MOSI Acc-2/F1 of 89.60/89.57 (CoMoR: 89.28/89.21), CH-SIMS Acc-2/F1 of 82.67/81.81 (CoMoR: 81.98/80.82) and SIMS-v2 MAE of 0.267 (CoMoR: 0.271). So claims like "lowest MAE on all four datasets", "first on all four MOSI metrics" etc does not hold. The published MSE-Adapter numbers should also be shown. For ChatGLM3-6B-base, the original MSE-Adapter paper reports numbers very close to CoMoR's on Acc-2/F1. I believe the rerun is fair, since MGSI got similar rerun numbers, but both should be reported.

Questions for authors
  1. On the trained full model, what happens to MAE/F1 at test time if you (a) force both AV gains to 1, or (b) shuffle router outputs across test samples?
  2. How much of the gain over MSE-Adapter comes from L_aux? Please run uniform gates without L_aux, and MSE-Adapter with your encoders and training recipe, so the routing contribution is separated from everything else.
  3. What are the parse-failure rates for CoMoR, MSE-Adapter and the ablations?
  4. Can you report Acc-7 (MOSI/MOSEI) and Acc-5 (SIMS), and rerun the main comparison and key ablations with 5 or more matched seeds and significance tests?
  5. On CH-SIMS, does CmnC^{mn} correlate with the disagreement between the provided unimodal labels? This is the most direct test of whether "conflict" means what the paper says it means.
  6. Why is the text diagnosis an order-invariant pooled embedding rather than the frozen LLM reading the real text with the diagnostic prompt?
  7. For the backbone comparison in Table 2, can you add MSE-Adapter (or uniform gates) per model, with stds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.