acceptodds
Under review as a conference paper at ICLR 2027

ContriFuse: Contribution-Aware Fusion of Auxiliary Descriptions for Multimodal Sentiment Analysis

Abstract

Multimodal Sentiment Analysis (MSA) integrates textual, acoustic, and visual signals to infer sentiment polarity and intensity. Existing methods typically encode inputs from different modalities into feature representations and perform sentiment prediction through cross-modal fusion. Recent studies further introduce auxiliary information, such as descriptions of audio-visual content and contextual knowledge, to enrich the semantic information contained in modality representations. Existing approaches mainly use such auxiliary textual information to augment or align modality representations, followed by joint, gated, or routed fusion for sentiment prediction. However, auxiliary descriptions from different modalities do not contribute equally to sentiment prediction; their usefulness depends not only on the corresponding original modality information, but also on auxiliary descriptions from other modalities. To address this issue, we propose ContriFuse, a multimodal sentiment analysis framework that adaptively regulates the contribution of each modality’s auxiliary descriptions before integrating them with the original modality features. First, ContriFuse constructs an auxiliary description representation for each modality from descriptions generated by multiple large language models. The resulting auxiliary representations are then mapped into shared and modality-specific spaces, allowing subsequent regulation to consider both information shared across modalities and information specific to each modality. ContriFuse further combines the original modality features, auxiliary description representations, and semantic similarities among descriptions from different modalities to determine the extent to which each modality’s auxiliary description should contribute to the current prediction. Finally, ContriFuse fuses the modulated auxiliary description representations with the corresponding modality features to predict sentiment polarity and intensity. Experiments on four widely used English and Chinese sentiment analysis benchmarks evaluate ContriFuse using both regression and classification metrics and demonstrate its effectiveness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Auxiliary Descriptions, Contribution-Aware Routing, Multimodal Fusion, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_vC5kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ContriFuse, a framework for multimodal sentiment analysis that regulates the contribution of LLM-generated auxiliary descriptions before integrating them with original modality features. Descriptions from multiple LLMs are aggregated using semantic agreement, organized into shared and modality-specific representations, and modulated by a contribution-aware router conditioned on source features, auxiliary representations, cross-modal relations, and modality availability. The modulated descriptions then condition temporal feature fusion. The framework is evaluated on four English and Chinese benchmarks, with component ablations, missing-modality experiments, and additional analyses of contribution regulation.

Critical strengths

The paper addresses a relevant problem through a coherent design. Generated auxiliary descriptions can provide useful sentiment information while also introducing inconsistent or weakly supported interpretations. Explicitly regulating these descriptions before source-feature integration is a reasonable approach. The distinction between original modality features and auxiliary descriptions is maintained throughout the architecture, making the proposed information flow understandable. The experiments provide useful evidence for adaptive pre-fusion regulation. The evaluation covers four benchmarks and reports five-seed statistics for the full model. Table 3 shows improvements over equal contribution, while Appendix C.13 compares pre-fusion regulation with a controlled post-fusion variant. The description-stratified analysis in Appendix C.14 is particularly relevant: larger improvements on opposite-polarity descriptions provide supporting evidence that adaptive regulation is useful when auxiliary descriptions conflict.

Critical weaknesses
  1. The contribution beyond conventional learned fusion is not sufficiently isolated under matched inputs and resources. CAR is implemented as a sigmoid MLP gate conditioned on source features, auxiliary representations, pairwise similarities, and availability. Table 3 and Appendix C.13 support adaptive pre-fusion regulation within the proposed pipeline, but do not establish its advantage over strong conventional concatenation, attention, or learned-gating alternatives receiving the same descriptions and source information. Furthermore, the auxiliary pathway introduces additional acoustic/visual descriptors and descriptions from three external LLMs. More controlled comparisons are needed to distinguish the benefits of these additional inputs from those of the proposed representation organization and routing design. This distinction is central to assessing the methodological contribution.
  2. The independent roles of agreement aggregation and shared–private organization remain insufficiently established. The generator-combination experiments change the available descriptions and therefore do not isolate the effect of agreement-based weighting. In particular, Equations (7)–(8) reduce exactly to equal averaging when two generators are used; for the default three-generator setting, a comparison against uniform aggregation is missing. The interpretation of DERL also requires stronger evidence: the shared loss admits constant shared representations, while the private loss can be minimized by fixed, mutually orthogonal modality representations. Task supervision may prevent these solutions, but the current module-level ablation does not establish that the learned components capture the claimed shared and complementary information. These gaps limit the evidence supporting two important mechanisms in the framework.
Questions for authors
  1. Comparison protocol and baseline provenance: Which feature extractors, feature dimensions, and normalization procedures produce the benchmark feature arrays used by ContriFuse? Which Table 2 baselines were reproduced with these same inputs, and which results were taken from prior publications? In particular, please identify the source or reproduction protocol for MSE-LLaMA2-7B’s MOSEI Non-0 Acc-2/F1 values of 69.90/69.85 and its MOSI and CH-SIMS results.
  2. Agreement aggregation: Have you compared agreement-based aggregation against uniform averaging using exactly the same three generators, cached descriptions, and downstream training protocol? For two generators, cosine symmetry makes the two agreement scores identical and hence both weights equal to 0.5. Please distinguish the benefit of combining multiple generators from the benefit of the weighting rule itself.
  3. Shared–private representations: What evidence shows that the shared representations retain nontrivial sample-dependent information and that the private representations capture complementary content? Statistics of representation variance and branch-specific predictive performance, or controlled shared-only/private-only analyses, would help determine whether the regularizers produce the intended organization rather than a degenerate or redundant decomposition.
  4. Stability of the incremental gains: Are the generator, CAR, and pre-/post-fusion comparisons evaluated over matched random seeds? For example, on MOSI, the best single-generator configuration has MAE 0.609 versus 0.602 for the full model, while the full model’s reported standard deviation is 0.0075. This does not establish that the difference is insignificant, but paired multi-seed results would clarify whether the additional complexity yields a consistent improvement.
5 comments
AuthorsOpenReview ↗

Reply 1|2.1 Methodological Contribution

Response to Reviewer 2 [1/5]: Methodological Contribution Under Matched Inputs and Resources (Weakness 1)

We sincerely thank the reviewer for the thorough evaluation of our manuscript and the constructive feedback. We particularly appreciate the concerns regarding methodological novelty, baseline provenance, agreement aggregation, representation complementarity, and experimental stability. We have conducted additional controlled comparisons and analyses to clarify these issues. Our point-by-point responses follow.

Weakness 1.

The contribution beyond conventional learned fusion is not sufficiently isolated under matched inputs and resources. CAR is implemented as a sigmoid MLP gate conditioned on source features, auxiliary representations, pairwise similarities, and availability. Table 3 and Appendix C.13 support adaptive pre-fusion regulation within the proposed pipeline, but do not establish its advantage over strong conventional concatenation, attention, or learned-gating alternatives receiving the same descriptions and source information. Furthermore, the auxiliary pathway introduces additional acoustic/visual descriptors and descriptions from three external LLMs. More controlled comparisons are needed to distinguish the benefits of these additional inputs from those of the proposed representation organization and routing design. This distinction is central to assessing the methodological contribution.

Response.

We thank the reviewer for emphasizing the importance of isolating methodological contributions under matched inputs and computational resources. We address this concern through two controlled comparisons.

(a) Fusion with matched inputs and similar computational cost. We hold the source features, cached LLM descriptions, DERL representations, temporal backbone, data splits, optimization protocol, and auxiliary dimensionality fixed. Only the integration mechanism varies.

MethodParams (M)Rel. FLOPsMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Direct concatenation2.411.00×0.6190.8600.4880.861
Simple learned gate2.441.01×0.6120.8640.4800.866
Source-to-auxiliary attention2.501.04×0.6090.8660.4780.868
Full CAR2.471.02×0.6020.8710.4720.873

Learned gating and attention improve over direct concatenation. However, Full CAR achieves the lowest MAE and highest correlation on both datasets at similar parameter counts and FLOPs.

The simple-gate comparison isolates the benefit of incorporating cross-modal relations and modality availability, while the attention comparison evaluates CAR against conventional feature interaction using identical auxiliary inputs.

(b) Auxiliary inputs with the downstream architecture fixed. We keep DERL, CAR, and MTAF unchanged and vary only the construction of auxiliary information.

Auxiliary inputMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Feature only0.6420.8420.5050.842
Direct descriptors, no LLM0.6250.8540.4930.855
Deterministic textualization, no LLM0.6180.8600.4860.862
LLM-generated descriptions0.6020.8710.4720.873

Direct eGeMAPS/OpenFace descriptors improve over feature-only input. Deterministic textualization provides further improvements, while LLM-generated interpretation yields additional MAE reductions of 0.016 on MOSI and 0.014 on MOSEI relative to deterministic textualization.

Overall, these matched comparisons separate the effects of auxiliary information from those of the fusion mechanism. They show that the improvements cannot be explained solely by additional descriptors or increased model capacity, providing stronger evidence for the contribution of CAR under controlled conditions.

AuthorsOpenReview ↗

Reply 2|2.2 Baseline Provenance

Response to Reviewer 2 [2/5]: Comparison Protocol, Source Features, and Baseline Provenance (Question 1)

Question 1.

Comparison protocol and baseline provenance: Which feature extractors, feature dimensions, and normalization procedures produce the benchmark feature arrays used by ContriFuse? Which Table 2 baselines were reproduced with these same inputs, and which results were taken from prior publications? In particular, please identify the source or reproduction protocol for MSE-LLaMA2-7B's MOSEI Non-0 Acc-2/F1 values of 69.90/69.85 and its MOSI and CH-SIMS results.

Response.

We thank the reviewer for requesting clarification of the feature protocol and baseline provenance. We have clarified the source features, distinguished published and reproduced results, and revised Table 2 to ensure consistent provenance at the dataset level.

(a) Source and auxiliary feature protocols.

ContriFuse uses pre-extracted benchmark feature sequences with the following configurations. Each modality-specific encoder projects its input into a common 128-dimensional hidden space.

DatasetTextAudioVision
CMU-MOSIBERT, 768COVAREP, 74FACET, 47
CMU-MOSEIBERT, 768COVAREP, 74FACET, 35
CH-SIMSBERT, 768LibROSA, 33OpenFace, 709
CH-SIMSv2BERT, 768LibROSA, 33OpenFace, 709

All datasets use training-set z-score normalization.

The auxiliary-description pathway uses the original transcript for text, 88-dimensional eGeMAPSv02 functionals for audio, and OpenFace facial-action, validity, pose, and gaze statistics for vision. Generated descriptions are encoded by the frozen all-MiniLM-L6-v2 encoder into 384-dimensional embeddings, followed by a unified PCA fitted exclusively on training-set embeddings to obtain 128-dimensional representations.

(b) Baseline provenance.

We prioritize results explicitly reported in original publications. For established baselines, we also use comparison results compiled in CICA. When complete numerical results are unavailable, we use reproduced dataset-level results where available. Unreported metrics are marked with "—".

The revised provenance is summarized below.

MethodMOSIMOSEICH-SIMSCH-SIMSv2
TFN, LMF, MulTCICACICACICACICA
MISACICACICA——
Self-MM, ALMTCICACICACICACICA
3WD-DRT, CICAOriginalOriginalOriginalOriginal
EBMCOriginalOriginal——
KuDA, TETFN, TMBL——OriginalOriginal
DEVAOriginalOriginalOriginalReproduced
TEXTOriginalOriginalOriginal (partial)Reproduced
MSE-LLaMA2-7BReproducedOriginal (partial)ReproducedOriginal (partial)

For DEVA, CH-SIMSv2 results are reproduced because the original publication evaluates MOSI, MOSEI, and CH-SIMS. For TEXT, the Chinese benchmark comparisons are primarily graphical; we retain explicitly reported CH-SIMS values and reproduce CH-SIMSv2.

For MSE-LLaMA2-7B, the original publication reports MOSEI and CH-SIMSv2 results. The MOSI and CH-SIMS results in our comparison are reproduced.

Published baseline results were not all re-evaluated using ContriFuse's feature arrays. The matched-input experiments in Section 2.1 directly evaluate the proposed mechanism under identical source and auxiliary inputs.

(c) MSE-LLaMA2-7B reproduction and evaluation.

For reproduced MOSI and CH-SIMS results, we use the LLaMA2-7B backbone with MSE-Adapter and the corresponding benchmark feature arrays.

ItemConfiguration
BackboneLLaMA2-7B / MSE-Adapter
SplitsOfficial train/validation/test
Batch size16
Learning rate2 × 10^-4
Maximum epochs20
Random seeds3
Checkpoint selectionValidation MAE
EvaluationContriFuse metric protocol

For MOSEI, the original publication reports Has-0 Acc-2/F1 of 86.74/86.51, alongside MAE, correlation, and Acc-7. The previously included Non-0 values of 69.90/69.85 came from our separate reproduction.

We have removed these Non-0 values from revised Table 2 and marked them as "—" to maintain consistent dataset-level provenance.

Similarly, the original CH-SIMSv2 results report Acc-2, F1, MAE, and correlation. Since Acc2-weak is distinct from Acc-3, we mark Acc-3 as unavailable.

These revisions make published and reproduced results explicitly traceable while preserving consistent comparisons and the controlled evaluation of ContriFuse.

AuthorsOpenReview ↗

Reply 3|2.3 Agreement Weighting

Response to Reviewer 2 [3/5]: Agreement Weighting and Shared–Private Organization (Question 2, Weakness 2)

Weakness 2.

The independent roles of agreement aggregation and shared–private organization remain insufficiently established. The generator-combination experiments change the available descriptions and therefore do not isolate the effect of agreement-based weighting. In particular, Equations (7)–(8) reduce exactly to equal averaging when two generators are used; for the default three-generator setting, a comparison against uniform aggregation is missing. The interpretation of DERL also requires stronger evidence: the shared loss admits constant shared representations, while the private loss can be minimized by fixed, mutually orthogonal modality representations. Task supervision may prevent these solutions, but the current module-level ablation does not establish that the learned components capture the claimed shared and complementary information. These gaps limit the evidence supporting two important mechanisms in the framework.

Question 2.

Agreement aggregation: Have you compared agreement-based aggregation against uniform averaging using exactly the same three generators, cached descriptions, and downstream training protocol? For two generators, cosine symmetry makes the two agreement scores identical and hence both weights equal to 0.5. Please distinguish the benefit of combining multiple generators from the benefit of the weighting rule itself.

Response.

We thank the reviewer for highlighting the need to independently validate agreement-based aggregation and shared–private representation organization. We address both mechanisms through controlled comparisons and representation diagnostics.

(a) Agreement weighting versus uniform averaging.

For two generators, cosine symmetry yields identical agreement scores and assigns each generator a weight of 0.5. Thus, the two-generator experiments measure the benefit of combining generators, not the weighting rule itself.

To isolate agreement weighting, we compare uniform weights (1/3, 1/3, 1/3) with agreement-derived softmax weights using the same three generators, cached descriptions, frozen sentence encoder, PCA projection, downstream architecture, data splits, and optimization protocol.

Aggregation strategyMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Best single generator0.6090.8640.4790.866
Three-generator uniform averaging0.6080.8660.4780.867
Agreement weighting0.6020.8710.4720.873

Uniform averaging provides a small improvement over the best single generator. Agreement weighting further reduces MAE by 0.006 on both MOSI and MOSEI.

We also evaluated both aggregation strategies over five matched random seeds.

Paired statisticMOSIMOSEI
Mean MAE reduction0.0060.006
Std. of paired reduction0.0020.002
Agreement weighting wins5/55/5

Agreement weighting achieves lower MAE in every matched run on both datasets. This directly isolates its contribution beyond generator diversity.

(b) Evidence for shared–private organization.

We additionally examined whether DERL learns nondegenerate and complementary shared (SS) and private (PP) representations.

First, shared-only and private-only prediction experiments show that both branches retain predictive information. On MOSI, SS achieves MAE 0.617 and PP achieves 0.622, while their combination reduces MAE to 0.602. On MOSEI, the corresponding MAEs are 0.485, 0.489, and 0.472.

Second, the cross-modal cosine similarity of SS decreases from 0.46 for matched samples to 0.21 for shuffled pairs, indicating sample-dependent shared information. The corresponding values for PP are 0.14 and 0.11, consistent with weaker cross-modal alignment.

Third, the mean feature-wise standard deviation is 0.38 for SS and 0.42/0.39/0.40 for the text/audio/visual private branches. These nonzero values rule out constant representations.

Furthermore, linear CKA between SS and PP is 0.21/0.18/0.19 for text/audio/vision, indicating limited linear redundancy.

Finally, removing both regularizers while retaining the DERL architecture increases MAE from 0.602 to 0.613 on MOSI and from 0.472 to 0.481 on MOSEI.

These diagnostics support sample-dependent, noncollapsed, and complementary representations, although they do not establish formal semantic identifiability.

Overall, the controlled aggregation comparison demonstrates that agreement weighting provides consistent gains beyond combining generators. The representation diagnostics independently support the intended shared–private organization of DERL. Together, these experiments provide direct evidence for both mechanisms questioned in Weakness 2.

Reviewer_dtExOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ContriFuse for MSA. It aggregates modality-specific auxiliary descriptions generated by multiple LLMs, organizes their representations into shared and modality-specific components, and learns contribution coefficients before integrating them with temporal source features. Experiments on four datasets show competitive performance.

Critical strengths
  1. generated auxiliary descriptions can be inconsistent or unevenly useful, making adaptive contribution regulation well motivated.

  2. well performance

Critical weaknesses
  1. The auxiliary pathway introduces eGeMAPS/OpenFace-derived information alongside LLM-generated interpretations. Existing comparisons do not clearly establish whether the gains require LLM interpretation or could be achieved by directly incorporating the same descriptors.

  2. Removing DERL/CAR shoes their utility, but does not establish that DERL separates shared and modality-specific semantics, or that CAR responds appropriately to misleading descriptions.

  3. The pipeline contains several processing stages and requires up to nine LLM generations per unseen complete-modality sample. Do the performance gains justify the substantial computational cost?

Questions for authors
  1. Could you compare DESC with directly encoding the same eGeMAPS/OpenFace descriptors or their deterministic textualizations, without LLM-generated interpretations?

  2. Does cross-LLM agreement weighting outperform uniform averaging of the same description embeddings?

  3. Does CAR outperform a simpler gate using only the source feature and its corresponding auxiliary representation? Under controlled description corruption or replacement, do its coefficients and prediction errors change appropriately?

13 comments
AuthorsOpenReview ↗

Reply 1|1.1 Value of LLM Interpretation

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive and insightful feedback. We particularly appreciate the questions concerning LLM-generated interpretations, agreement-based aggregation, representation organization, contribution-aware routing, and computational efficiency. We have conducted additional controlled experiments and analyses to address these concerns. Our point-by-point responses are provided below.

1.1 Value of LLM Interpretation Beyond Deterministic Descriptors

Weakness 1.

The auxiliary pathway introduces eGeMAPS/OpenFace-derived information alongside LLM-generated interpretations. Existing comparisons do not clearly establish whether the gains require LLM interpretation or could be achieved by directly incorporating the same descriptors.

Question 1.

Could you compare DESC with directly encoding the same eGeMAPS/OpenFace descriptors or their deterministic textualizations, without LLM-generated interpretations?

Response.

We compared two controlled baselines with ContriFuse while holding DERL, CAR, MTAF, the data splits, the optimization protocol, and the auxiliary representation dimension fixed.

(1) Direct descriptors (no LLM). We map the eGeMAPSv02 and OpenFace measurements used by DESC directly to the same 128-dimensional auxiliary space. The text branch encodes the original transcript with the same frozen sentence encoder. This baseline uses no LLM generation.

(2) Deterministic textualization (no LLM). We encode the exact modality-specific LLM inputs Qn,mQ_{n,m} using the same frozen sentence encoder and train-fitted PCA. These inputs are the original transcript, the deterministic eGeMAPS textualization, and the deterministic OpenFace textualization. This baseline also uses no LLM generation.

Auxiliary inputMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Feature only0.6420.8420.5050.842
Direct descriptors, no LLM0.6250.8540.4930.855
Deterministic textualization, no LLM0.6180.8600.4860.862
LLM-generated descriptions0.6020.8710.4720.873

Direct descriptors improve over feature-only input, and deterministic textualization improves performance further. With the underlying observations held fixed, LLM interpretation reduces MAE by an additional 0.016 on MOSI and 0.014 on MOSEI. These reductions account for approximately 40.0% and 42.4%, respectively, of the full improvement over feature-only input.

These controlled comparisons distinguish the benefits of incorporating additional acoustic/visual descriptors from those of LLM-generated interpretation. The results demonstrate that the performance gains cannot be attributed solely to the additional descriptor information.

AuthorsOpenReview ↗

Reply 2|1.2 Agreement Weighting

1.2 Agreement Weighting Versus Uniform Aggregation

Question 2.

Does cross-LLM agreement weighting outperform uniform averaging of the same description embeddings?

Response.

We thank the reviewer for highlighting the importance of distinguishing the benefit of combining multiple generators from the benefit of agreement-based weighting.

With two generators, cosine symmetry gives both generators the same agreement score, so each receives a weight of 0.5. The two-generator rows in the submitted manuscript therefore measure the benefit of combining two generators rather than the benefit of the weighting rule itself.

To isolate the weighting rule, we compared agreement weighting with uniform weights of (1/3, 1/3, 1/3) using the same cached descriptions from gpt-5.4-mini, deepseek-chat, and qwen-plus. The sentence embeddings, PCA projection, downstream architecture, data splits, and training protocol were identical.

AggregationMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Best single generator0.6090.8640.4790.866
Three-generator uniform averaging0.6080.8660.4780.867
Agreement weighting0.6020.8710.4720.873

Uniform averaging yields a small improvement over the best single generator on each dataset. Agreement weighting further reduces MAE from 0.608 to 0.602 on MOSI and from 0.478 to 0.472 on MOSEI.

We additionally compared the two three-generator strategies over the same five random seeds.

Paired statisticMOSIMOSEI
Mean MAE reduction0.0060.006
Std. of paired reduction0.0020.002
Agreement weighting wins5/55/5

Agreement weighting achieves lower MAE in every matched run on both datasets.

These results isolate the benefit of agreement-derived weighting under identical generator outputs and downstream training conditions. They demonstrate that the proposed aggregation mechanism provides consistent improvements beyond simply combining multiple LLM-generated descriptions.

AuthorsOpenReview ↗

Reply 3|1.3 DERL Representation

1.3 Representation Behavior of DERL

Weakness 2.

Removing DERL/CAR shows their utility, but does not establish that DERL separates shared and modality-specific semantics, or that CAR responds appropriately to misleading descriptions.

Response.

We thank the reviewer for requesting stronger evidence of the internal behavior of DERL and CAR. We address DERL here and examine CAR's response to misleading descriptions in Section 1.4.

DERL decomposes the aggregated auxiliary representation En,mE_{n,m} into a shared component Sn,mS_{n,m} and a modality-specific component Pn,mP_{n,m}, which are combined into the organized representation Gn,mG_{n,m}.

The submitted ablation compares DERL with direct use of En,mE_{n,m}. We additionally conducted four diagnostics.

(1) Shared-only and private-only prediction

We evaluated each branch separately with the same downstream CAR and MTAF pipeline.

RepresentationMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
E without DERL0.6250.8560.4910.858
S only0.6170.8620.4850.864
P only0.6220.8590.4890.861
S+P0.6020.8710.4720.873

Each branch retains predictive information, and their combination achieves the lowest MAE and highest correlation on both datasets.

(2) Matched versus shuffled cross-modal similarity

We compared cross-modal cosine similarity for matching samples against randomly shuffled cross-sample pairs.

RepresentationMatched similarityShuffled similarity
S0.460.21
P0.140.11

The larger matched-versus-shuffled gap for S supports its role in capturing sample-dependent information shared across modalities. The smaller gap for P is consistent with weaker cross-modal alignment.

(3) Representation variation and shared–private redundancy

We measured the mean feature-wise standard deviation across samples to examine potential representation collapse.

RepresentationMean feature-wise std.
E0.44
S0.38
P_T0.42
P_A0.39
P_V0.40

The nonzero variation indicates that the learned representations do not collapse into constant outputs.

We further measured linear centered kernel alignment (CKA) between shared and modality-specific representations.

ModalityLinear CKA between S and P
Text0.21
Audio0.18
Vision0.19

The low CKA values indicate limited linear redundancy between the shared and private representations.

(4) Effect of shared and private regularization

We removed both regularization terms while retaining the complete DERL architecture.

SettingMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
λ_s = λ_p = 00.6130.8640.4810.866
Full DERL0.6020.8710.4720.873

The performance degradation at fixed architecture isolates the predictive contribution of the regularizers.

Overall, these diagnostics support task-supervised organization into noncollapsed, sample-dependent, and complementary shared–private representations. While formal semantic identifiability remains open, the results provide stronger evidence for the intended behavior of DERL beyond module-level ablation.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Reply 1|3.1 Agreement Weighting

Response to Reviewer 3 [1/8]: Agreement Weighting Versus Uniform Aggregation (Weakness 1)

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive feedback. We particularly appreciate the questions concerning agreement-based aggregation, CAR's routing inputs, missing-modality robustness, coefficient design, and pre-fusion regulation. We have conducted additional controlled experiments and analyses to address these concerns.

Weakness 1.

The cross-LLM consistency aggregation lacks a clear justification. The paper presents cross-LLM agreement as a principled aggregation mechanism. However, it provides no evidence for it against uniform averaging. Its contribution beyond using more generators is therefore unknown.

Response.

We thank the reviewer for highlighting the need to isolate the contribution of agreement-based weighting from the benefit of combining multiple generators.

With two generators, cosine symmetry gives both generators the same agreement score and therefore a weight of 0.5 each. Thus, the two-generator rows in Table 3 measure the benefit of combining generators rather than agreement weighting itself.

To isolate the weighting rule, we compare uniform averaging with agreement-based aggregation using exactly the same three generators, cached descriptions, frozen sentence encoder, PCA projection, downstream architecture, data splits, and optimization protocol.

Uniform averaging assigns fixed weights of (1/3, 1/3, 1/3), whereas agreement-based aggregation uses agreement-derived softmax weights.

Aggregation strategyMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Best single generator0.6090.8640.4790.866
Three-generator uniform averaging0.6080.8660.4780.867
Agreement weighting0.6020.8710.4720.873

Uniform averaging slightly improves over the best single generator. Agreement weighting further reduces MAE from 0.608 to 0.602 on MOSI and from 0.478 to 0.472 on MOSEI.

We additionally evaluated both three-generator aggregation strategies over the same five random seeds.

Paired statisticMOSIMOSEI
Mean MAE reduction0.0060.006
Std. of paired reduction0.0020.002
Agreement weighting wins5/55/5

Agreement weighting consistently achieves lower MAE across all five matched runs on both datasets.

Overall, these controlled comparisons distinguish the effects of generator combination and agreement-based weighting. The results show that agreement weighting provides consistent additional gains beyond uniform averaging of the same three descriptions, supporting its independent contribution to ContriFuse.

AuthorsOpenReview ↗

Reply 2|3.2 CAR Routing Inputs

Response to Reviewer 3 [2/8]: CAR Scope and the Four Routing Inputs (Weakness 2, Question 1)

Weakness 2.

The paper motivates CAR by saying that an LLM-generated description may not be reliable. But CAR's inputs are the source feature, the organized auxiliary representation, the cross-modal relation, and the availability mask. None of these inputs compares a description with the modality it describes. So CAR cannot detect an unreliable description.

Question 1.

Are the four routing inputs sufficient to estimate the contribution coefficient? Can different inputs lead to the same coefficient? What does the paper assume about this relation?

Response.

We thank the reviewer for highlighting this important distinction between contribution estimation and factual verification.

(a) Scope and assumptions of CAR.

The contribution coefficient controls how strongly an auxiliary representation influences the current sentiment prediction. CAR jointly conditions its routing decision on four inputs:

ComponentDefinition
Contribution coefficientrn,m=Mn,mgn,mr_{n,m}=M_{n,m}g_{n,m}
Routing coefficientgn,m=σ(zn,m)g_{n,m}=\sigma(z_{n,m})
Router inputsSource feature, organized auxiliary representation, cross-modal relations, and modality availability

Here, the router logit is produced by a modality-specific MLP conditioned on the four routing inputs.

The router maps a high-dimensional input to a scalar coefficient, so different inputs can produce the same coefficient. CAR assumes that auxiliary usefulness varies across samples and modalities and that these four signals provide contextual information for estimating it.

Importantly, CAR does not explicitly verify whether a generated description is factually consistent with its source modality. Such verification would require a separate source–description comparison.

(b) Controlled description mismatch.

To examine whether CAR responds to misleading auxiliary information, we replace one modality's description at a time while holding all source features fixed.

Random replacement draws another sample's same-modality description. Opposite-polarity replacement uses a same-modality description with the opposite generated majority polarity. Selection uses generated-description polarity alone.

ConditionMean contributionMOSI MAE ↓
Clean0.610.602
Random replacement0.540.630
Opposite-polarity replacement0.460.651

As mismatch becomes stronger, the assigned contribution decreases (0.61 to 0.54 to 0.46), while MAE increases. This indicates sensitivity to description mismatch rather than explicit factual verification.

(c) Contribution of cross-modal relations.

Since replacement also changes the cross-modal relation input, we repeated the intervention using the submitted router variant without cross-modal relations.

ConditionFull CAR weightFull CAR MAE ↓No-relation weightNo-relation MAE ↓
Clean0.610.6020.590.613
Random replacement0.540.6300.560.641
Opposite-polarity replacement0.460.6510.530.662

The no-relation router also reduces the contribution of replaced descriptions. However, full CAR exhibits a larger reduction under opposite-polarity replacement (0.15 versus 0.06) and consistently achieves lower MAE.

This indicates that cross-modal relations provide additional contextual information for contribution regulation under mismatch.

(d) Additional compatibility and agreement inputs.

We further examined whether CAR benefits from explicitly incorporating a source–auxiliary compatibility score (kappa) or an average generator agreement score (a).

These signals were appended individually and jointly to the original four routing inputs while retaining the downstream architecture and evaluation protocol.

Router inputsMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
F, G, c, M0.6020.8710.4720.873
F, G, c, M + a0.6020.8710.4720.873
F, G, c, M + kappa0.6010.8720.4710.873
F, G, c, M + a + kappa0.6010.8720.4710.874

Adding generator agreement leaves the reported metrics unchanged, while explicit compatibility provides only marginal improvements.

These results suggest that the original four routing inputs already capture useful contextual signals for contribution estimation on the evaluated benchmarks.

Overall, the controlled experiments support the empirical adequacy of CAR's four-input design. CAR adaptively estimates task-conditioned auxiliary contributions and responds to contextual mismatch, but it should not be interpreted as an explicit factual verification mechanism. The additional compatibility and agreement experiments further show that introducing explicit reliability-related signals yields only marginal improvements over the original design.

AuthorsOpenReview ↗

Reply 3|3.3 Missing-Modality Robustness

Response to Reviewer 3 [3/8]: Missing-Modality Robustness and Matched Routing (Weakness 3)

Weakness 3.

The paper uses the missing-modality results to support pre-fusion contribution regulation. But in these settings text is missing. The drop in performance may simply come from losing text, not from how the coefficient is applied. So these results do not support the claim.

Response.

We thank the reviewer for identifying the need to distinguish the effect of missing text from the benefit of adaptive contribution regulation. We agree that missing-modality performance alone does not establish the superiority of pre-fusion coefficient placement.

(a) Effect of modality availability. The original missing-modality experiment evaluates robustness to input availability.

On MOSI, retaining text gives MAE values of 0.608 (T), 0.605 (T+V), and 0.604 (T+A), close to 0.602 with all three modalities. Removing text substantially increases MAE to 1.377–1.383. MOSEI exhibits a similar pattern.

This confirms that the large degradation primarily reflects the importance of textual information, rather than directly proving the benefit of pre-fusion regulation.

(b) Equal contribution versus CAR under matched availability.

To isolate the benefit of adaptive routing, we compared equal contribution with full CAR within each modality-availability pattern.

Both variants use identical source features, cached auxiliary representations, missing-modality sampling, data splits, and evaluation procedures.

AvailabilityMOSI Equal ↓MOSI CAR ↓MOSEI Equal ↓MOSEI CAR ↓
T+A+V0.6170.6020.4860.472
T0.6190.6080.5000.488
T+V0.6180.6050.4860.473
T+A0.6160.6040.4890.476
A+V1.4101.3770.7740.749
A1.4041.3770.7920.770
V1.4161.3830.7900.767

CAR achieves lower MAE in all seven matched availability conditions.

The mean absolute MAE reductions are 0.0206 on MOSI and 0.0174 on MOSEI, corresponding to relative reductions of approximately 2.15% and 2.83%.

With text available, the mean reductions are 0.0128/0.0130 on MOSI/MOSEI. With text absent, they increase to 0.0310/0.0233.

These within-pattern comparisons isolate adaptive routing benefits while holding available inputs fixed.

(c) Independent evidence for pre-fusion regulation.

To distinguish routing benefits from coefficient placement, we also compare pre-fusion and post-fusion regulation with the same available modalities and auxiliary inputs.

Regulation strategyMOSI MAE ↓MOSI Corr ↑MOSEI MAE ↓MOSEI Corr ↑
Post-fusion0.6150.8660.4830.867
Pre-fusion CAR0.6020.8710.4720.873

Pre-fusion regulation improves MAE by 0.013 on MOSI and 0.011 on MOSEI relative to post-fusion regulation.

Overall, the missing-modality results establish sensitivity to input availability, while matched within-pattern comparisons establish the benefit of adaptive routing. The separate pre-/post-fusion comparison provides direct evidence concerning coefficient placement. These are distinct findings and should not be conflated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.