- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ContriFuse, a framework for multimodal sentiment analysis that regulates the contribution of LLM-generated auxiliary descriptions before integrating them with original modality features. Descriptions from multiple LLMs are aggregated using semantic agreement, organized into shared and modality-specific representations, and modulated by a contribution-aware router conditioned on source features, auxiliary representations, cross-modal relations, and modality availability. The modulated descriptions then condition temporal feature fusion. The framework is evaluated on four English and Chinese benchmarks, with component ablations, missing-modality experiments, and additional analyses of contribution regulation.
Critical strengths
The paper addresses a relevant problem through a coherent design. Generated auxiliary descriptions can provide useful sentiment information while also introducing inconsistent or weakly supported interpretations. Explicitly regulating these descriptions before source-feature integration is a reasonable approach. The distinction between original modality features and auxiliary descriptions is maintained throughout the architecture, making the proposed information flow understandable. The experiments provide useful evidence for adaptive pre-fusion regulation. The evaluation covers four benchmarks and reports five-seed statistics for the full model. Table 3 shows improvements over equal contribution, while Appendix C.13 compares pre-fusion regulation with a controlled post-fusion variant. The description-stratified analysis in Appendix C.14 is particularly relevant: larger improvements on opposite-polarity descriptions provide supporting evidence that adaptive regulation is useful when auxiliary descriptions conflict.
Critical weaknesses
- The contribution beyond conventional learned fusion is not sufficiently isolated under matched inputs and resources. CAR is implemented as a sigmoid MLP gate conditioned on source features, auxiliary representations, pairwise similarities, and availability. Table 3 and Appendix C.13 support adaptive pre-fusion regulation within the proposed pipeline, but do not establish its advantage over strong conventional concatenation, attention, or learned-gating alternatives receiving the same descriptions and source information. Furthermore, the auxiliary pathway introduces additional acoustic/visual descriptors and descriptions from three external LLMs. More controlled comparisons are needed to distinguish the benefits of these additional inputs from those of the proposed representation organization and routing design. This distinction is central to assessing the methodological contribution.
- The independent roles of agreement aggregation and shared–private organization remain insufficiently established. The generator-combination experiments change the available descriptions and therefore do not isolate the effect of agreement-based weighting. In particular, Equations (7)–(8) reduce exactly to equal averaging when two generators are used; for the default three-generator setting, a comparison against uniform aggregation is missing. The interpretation of DERL also requires stronger evidence: the shared loss admits constant shared representations, while the private loss can be minimized by fixed, mutually orthogonal modality representations. Task supervision may prevent these solutions, but the current module-level ablation does not establish that the learned components capture the claimed shared and complementary information. These gaps limit the evidence supporting two important mechanisms in the framework.
Questions for authors
- Comparison protocol and baseline provenance: Which feature extractors, feature dimensions, and normalization procedures produce the benchmark feature arrays used by ContriFuse? Which Table 2 baselines were reproduced with these same inputs, and which results were taken from prior publications? In particular, please identify the source or reproduction protocol for MSE-LLaMA2-7B’s MOSEI Non-0 Acc-2/F1 values of 69.90/69.85 and its MOSI and CH-SIMS results.
- Agreement aggregation: Have you compared agreement-based aggregation against uniform averaging using exactly the same three generators, cached descriptions, and downstream training protocol? For two generators, cosine symmetry makes the two agreement scores identical and hence both weights equal to 0.5. Please distinguish the benefit of combining multiple generators from the benefit of the weighting rule itself.
- Shared–private representations: What evidence shows that the shared representations retain nontrivial sample-dependent information and that the private representations capture complementary content? Statistics of representation variance and branch-specific predictive performance, or controlled shared-only/private-only analyses, would help determine whether the regularizers produce the intended organization rather than a degenerate or redundant decomposition.
- Stability of the incremental gains: Are the generator, CAR, and pre-/post-fusion comparisons evaluated over matched random seeds? For example, on MOSI, the best single-generator configuration has MAE 0.609 versus 0.602 for the full model, while the full model’s reported standard deviation is 0.0075. This does not establish that the difference is insignificant, but paired multi-seed results would clarify whether the additional complexity yields a consistent improvement.
5 comments
Reply 2|2.2 Baseline Provenance
Response to Reviewer 2 [2/5]: Comparison Protocol, Source Features, and Baseline Provenance (Question 1)
Question 1.
Comparison protocol and baseline provenance: Which feature extractors, feature dimensions, and normalization procedures produce the benchmark feature arrays used by ContriFuse? Which Table 2 baselines were reproduced with these same inputs, and which results were taken from prior publications? In particular, please identify the source or reproduction protocol for MSE-LLaMA2-7B's MOSEI Non-0 Acc-2/F1 values of 69.90/69.85 and its MOSI and CH-SIMS results.
Response.
We thank the reviewer for requesting clarification of the feature protocol and baseline provenance. We have clarified the source features, distinguished published and reproduced results, and revised Table 2 to ensure consistent provenance at the dataset level.
(a) Source and auxiliary feature protocols.
ContriFuse uses pre-extracted benchmark feature sequences with the following configurations. Each modality-specific encoder projects its input into a common 128-dimensional hidden space.
| Dataset | Text | Audio | Vision |
|---|---|---|---|
| CMU-MOSI | BERT, 768 | COVAREP, 74 | FACET, 47 |
| CMU-MOSEI | BERT, 768 | COVAREP, 74 | FACET, 35 |
| CH-SIMS | BERT, 768 | LibROSA, 33 | OpenFace, 709 |
| CH-SIMSv2 | BERT, 768 | LibROSA, 33 | OpenFace, 709 |
All datasets use training-set z-score normalization.
The auxiliary-description pathway uses the original transcript for text, 88-dimensional eGeMAPSv02 functionals for audio, and OpenFace facial-action, validity, pose, and gaze statistics for vision. Generated descriptions are encoded by the frozen all-MiniLM-L6-v2 encoder into 384-dimensional embeddings, followed by a unified PCA fitted exclusively on training-set embeddings to obtain 128-dimensional representations.
(b) Baseline provenance.
We prioritize results explicitly reported in original publications. For established baselines, we also use comparison results compiled in CICA. When complete numerical results are unavailable, we use reproduced dataset-level results where available. Unreported metrics are marked with "—".
The revised provenance is summarized below.
| Method | MOSI | MOSEI | CH-SIMS | CH-SIMSv2 |
|---|---|---|---|---|
| TFN, LMF, MulT | CICA | CICA | CICA | CICA |
| MISA | CICA | CICA | — | — |
| Self-MM, ALMT | CICA | CICA | CICA | CICA |
| 3WD-DRT, CICA | Original | Original | Original | Original |
| EBMC | Original | Original | — | — |
| KuDA, TETFN, TMBL | — | — | Original | Original |
| DEVA | Original | Original | Original | Reproduced |
| TEXT | Original | Original | Original (partial) | Reproduced |
| MSE-LLaMA2-7B | Reproduced | Original (partial) | Reproduced | Original (partial) |
For DEVA, CH-SIMSv2 results are reproduced because the original publication evaluates MOSI, MOSEI, and CH-SIMS. For TEXT, the Chinese benchmark comparisons are primarily graphical; we retain explicitly reported CH-SIMS values and reproduce CH-SIMSv2.
For MSE-LLaMA2-7B, the original publication reports MOSEI and CH-SIMSv2 results. The MOSI and CH-SIMS results in our comparison are reproduced.
Published baseline results were not all re-evaluated using ContriFuse's feature arrays. The matched-input experiments in Section 2.1 directly evaluate the proposed mechanism under identical source and auxiliary inputs.
(c) MSE-LLaMA2-7B reproduction and evaluation.
For reproduced MOSI and CH-SIMS results, we use the LLaMA2-7B backbone with MSE-Adapter and the corresponding benchmark feature arrays.
| Item | Configuration |
|---|---|
| Backbone | LLaMA2-7B / MSE-Adapter |
| Splits | Official train/validation/test |
| Batch size | 16 |
| Learning rate | 2 × 10^-4 |
| Maximum epochs | 20 |
| Random seeds | 3 |
| Checkpoint selection | Validation MAE |
| Evaluation | ContriFuse metric protocol |
For MOSEI, the original publication reports Has-0 Acc-2/F1 of 86.74/86.51, alongside MAE, correlation, and Acc-7. The previously included Non-0 values of 69.90/69.85 came from our separate reproduction.
We have removed these Non-0 values from revised Table 2 and marked them as "—" to maintain consistent dataset-level provenance.
Similarly, the original CH-SIMSv2 results report Acc-2, F1, MAE, and correlation. Since Acc2-weak is distinct from Acc-3, we mark Acc-3 as unavailable.
These revisions make published and reproduced results explicitly traceable while preserving consistent comparisons and the controlled evaluation of ContriFuse.
Reply 3|2.3 Agreement Weighting
Response to Reviewer 2 [3/5]: Agreement Weighting and Shared–Private Organization (Question 2, Weakness 2)
Weakness 2.
The independent roles of agreement aggregation and shared–private organization remain insufficiently established. The generator-combination experiments change the available descriptions and therefore do not isolate the effect of agreement-based weighting. In particular, Equations (7)–(8) reduce exactly to equal averaging when two generators are used; for the default three-generator setting, a comparison against uniform aggregation is missing. The interpretation of DERL also requires stronger evidence: the shared loss admits constant shared representations, while the private loss can be minimized by fixed, mutually orthogonal modality representations. Task supervision may prevent these solutions, but the current module-level ablation does not establish that the learned components capture the claimed shared and complementary information. These gaps limit the evidence supporting two important mechanisms in the framework.
Question 2.
Agreement aggregation: Have you compared agreement-based aggregation against uniform averaging using exactly the same three generators, cached descriptions, and downstream training protocol? For two generators, cosine symmetry makes the two agreement scores identical and hence both weights equal to 0.5. Please distinguish the benefit of combining multiple generators from the benefit of the weighting rule itself.
Response.
We thank the reviewer for highlighting the need to independently validate agreement-based aggregation and shared–private representation organization. We address both mechanisms through controlled comparisons and representation diagnostics.
(a) Agreement weighting versus uniform averaging.
For two generators, cosine symmetry yields identical agreement scores and assigns each generator a weight of 0.5. Thus, the two-generator experiments measure the benefit of combining generators, not the weighting rule itself.
To isolate agreement weighting, we compare uniform weights (1/3, 1/3, 1/3) with agreement-derived softmax weights using the same three generators, cached descriptions, frozen sentence encoder, PCA projection, downstream architecture, data splits, and optimization protocol.
| Aggregation strategy | MOSI MAE ↓ | MOSI Corr ↑ | MOSEI MAE ↓ | MOSEI Corr ↑ |
|---|---|---|---|---|
| Best single generator | 0.609 | 0.864 | 0.479 | 0.866 |
| Three-generator uniform averaging | 0.608 | 0.866 | 0.478 | 0.867 |
| Agreement weighting | 0.602 | 0.871 | 0.472 | 0.873 |
Uniform averaging provides a small improvement over the best single generator. Agreement weighting further reduces MAE by 0.006 on both MOSI and MOSEI.
We also evaluated both aggregation strategies over five matched random seeds.
| Paired statistic | MOSI | MOSEI |
|---|---|---|
| Mean MAE reduction | 0.006 | 0.006 |
| Std. of paired reduction | 0.002 | 0.002 |
| Agreement weighting wins | 5/5 | 5/5 |
Agreement weighting achieves lower MAE in every matched run on both datasets. This directly isolates its contribution beyond generator diversity.
(b) Evidence for shared–private organization.
We additionally examined whether DERL learns nondegenerate and complementary shared () and private () representations.
First, shared-only and private-only prediction experiments show that both branches retain predictive information. On MOSI, achieves MAE 0.617 and achieves 0.622, while their combination reduces MAE to 0.602. On MOSEI, the corresponding MAEs are 0.485, 0.489, and 0.472.
Second, the cross-modal cosine similarity of decreases from 0.46 for matched samples to 0.21 for shuffled pairs, indicating sample-dependent shared information. The corresponding values for are 0.14 and 0.11, consistent with weaker cross-modal alignment.
Third, the mean feature-wise standard deviation is 0.38 for and 0.42/0.39/0.40 for the text/audio/visual private branches. These nonzero values rule out constant representations.
Furthermore, linear CKA between and is 0.21/0.18/0.19 for text/audio/vision, indicating limited linear redundancy.
Finally, removing both regularizers while retaining the DERL architecture increases MAE from 0.602 to 0.613 on MOSI and from 0.472 to 0.481 on MOSEI.
These diagnostics support sample-dependent, noncollapsed, and complementary representations, although they do not establish formal semantic identifiability.
Overall, the controlled aggregation comparison demonstrates that agreement weighting provides consistent gains beyond combining generators. The representation diagnostics independently support the intended shared–private organization of DERL. Together, these experiments provide direct evidence for both mechanisms questioned in Weakness 2.
Reply 1|2.1 Methodological Contribution
Response to Reviewer 2 [1/5]: Methodological Contribution Under Matched Inputs and Resources (Weakness 1)
We sincerely thank the reviewer for the thorough evaluation of our manuscript and the constructive feedback. We particularly appreciate the concerns regarding methodological novelty, baseline provenance, agreement aggregation, representation complementarity, and experimental stability. We have conducted additional controlled comparisons and analyses to clarify these issues. Our point-by-point responses follow.
Weakness 1.
Response.
We thank the reviewer for emphasizing the importance of isolating methodological contributions under matched inputs and computational resources. We address this concern through two controlled comparisons.
(a) Fusion with matched inputs and similar computational cost. We hold the source features, cached LLM descriptions, DERL representations, temporal backbone, data splits, optimization protocol, and auxiliary dimensionality fixed. Only the integration mechanism varies.
Learned gating and attention improve over direct concatenation. However, Full CAR achieves the lowest MAE and highest correlation on both datasets at similar parameter counts and FLOPs.
The simple-gate comparison isolates the benefit of incorporating cross-modal relations and modality availability, while the attention comparison evaluates CAR against conventional feature interaction using identical auxiliary inputs.
(b) Auxiliary inputs with the downstream architecture fixed. We keep DERL, CAR, and MTAF unchanged and vary only the construction of auxiliary information.
Direct eGeMAPS/OpenFace descriptors improve over feature-only input. Deterministic textualization provides further improvements, while LLM-generated interpretation yields additional MAE reductions of 0.016 on MOSI and 0.014 on MOSEI relative to deterministic textualization.
Overall, these matched comparisons separate the effects of auxiliary information from those of the fusion mechanism. They show that the improvements cannot be explained solely by additional descriptors or increased model capacity, providing stronger evidence for the contribution of CAR under controlled conditions.