acceptodds
Under review as a conference paper at ICLR 2027

PIDFusion: Selective Non-Text Supervision via Partial Information Decomposition for Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis (MSA) integrates language, acoustic behavior, and visual expression, yet the dominance of language can cause models to underuse informative non-text cues. Applying additional non-text supervision uniformly is also suboptimal because audio and vision do not contribute unique sentiment evidence in every utterance. To address this issue, we propose PIDFusion, a partial-information-decomposition-guided framework that treats text and the joint audio–visual stream as two information sources for the sentiment target. A neural label-information probe combines source and joint label predictors with label-conditioned Sinkhorn transport to characterize redundant, unique, and synergistic information in continuous representations. The probe is optimized on detached features, separating information measurement from predictive representation learning. We then calibrate a sample-wise audio–visual uniqueness score and use it to selectively allocate an auxiliary text-blind regression objective, thereby strengthening non-text representations where they provide distinct sentiment evidence. In parallel, modality-specific and joint audio–visual regression heads provide direct auxiliary supervision to the corresponding representations. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate the effectiveness of PIDFusion across English and Chinese MSA benchmarks. Ablation studies further show that PID-guided sample selection is more effective than uniform or matched-random allocation and validate the contributions of the proposed training objectives.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Multimodal Sentiment Analysis, Partial Information Decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eJ1YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PIDFusion for multimodal sentiment analysis, which treats text and the joint audio–visual (AV) representation as two sources of a partial information decomposition (PID) with respect to the sentiment label. A BATCH-style neural probe with label-conditioned Sinkhorn coupling, trained on detached features with its own optimizer, estimates redundant, unique and synergistic quantities; a batch-centered local AV-unique proxy is calibrated by a one-time injection sweep and thresholded to decide which training samples receive a text-blind AV-only regression loss. Together with unimodal auxiliary heads and a detached gated fusion, the model is evaluated on CMU-MOSI, CMU-MOSEI and CH-SIMS within MMSA, with matched-random and uniform selection controls on MOSI.

Critical strengths
  1. The question of which samples need extra non-text supervision is well posed, and the matched-random and uniform controls are the right experiments to test whether sample identity matters.
  2. Gradient boundaries are clearly specified (detached probe, separate optimizer, detached selector), and the paper is candid that the atoms are operational estimates rather than exact PID.
  3. Hyperparameters, feature shapes and the efficiency profile (including the admission that FLOPs are the highest among compared models) are fully documented in Appendix C and E.
Critical weaknesses
  1. All results are single runs with no seeds, standard deviations or significance tests. The headline margins are within typical seed noise on these benchmarks: MOSI Acc-2 85.37 vs. 85.06 (Self-MM), MAE 0.705 vs. 0.710; MOSEI F1 85.32 vs. 85.27 (DLF) and Acc-2 below DLF; SIMS Acc-3 below TETFN/ALMT. The central ablation (full vs. matched-random: +0.92 Acc-2, +0.44 Acc-7; full vs. uniform: +0.77 Acc-2) is likewise a single-run difference of under one point.
  2. The PID machinery is largely decorative with respect to what the method uses. Only the local AV-unique proxy enters training, and by Eq. (16) it equals the AV probe's per-sample log-likelihood ratio log p_AV(c_i|z_AV,i)/π(c_i) shifted by a batch-level constant and clipped at zero; the Sinkhorn coupling only sets that constant. The natural control, selecting samples by AV-head confidence or loss without any PID estimator, is missing, as is any analysis of the estimated R, U_T, S atoms or validation of the local score (CH-SIMS provides modality-specific labels that would directly test whether selected samples are those where AV annotations disagree with text).
  3. The fusion gate in Section 3.5 is a sigmoid MLP whose parameters "remain at their initialization" because its output is detached and no objective trains it, so the main prediction mixes z_T and z_J through a random input-dependent gate. This is neither motivated nor ablated (learned gate, fixed 0.5, no gate), and it raises doubts about which part of the pipeline the reported numbers reflect.
  4. The baseline set is dated and mixed. Table 1 combines reproductions with numbers copied from DLF, and recent methods reporting substantially stronger MOSI results in the same MMSA setting (e.g., CICA, CaReFlow, EBMC with Acc-7 ≈ 49–51 and MAE ≈ 0.62–0.69) are absent, so "ranking first on all four metrics" does not hold against the current state of the art. On CH-SIMS only three baselines are compared, one of which (CENET, Corr 0.123) is a collapsed reproduction; standard SIMS baselines (Self-MM, MISA, MMIM, EMOE) are omitted.
  5. Ablations are restricted to MOSI; the threshold δ differs per dataset (0.05/0.10/0.02) with no stated selection procedure; there is no ablation of injection calibration, the unimodal-head weights, or the probe's coupling. The method roughly doubles FLOPs over Self-MM (16.05 vs. 8.60 G) for gains that cannot be distinguished from noise. Code is only promised upon acceptance.

The selective-supervision question is worth asking and the control design is sound, but the evidence is single-run and within noise, the effective selection signal reduces to AV-probe confidence, and the untrained fusion gate is unexplained. I would reconsider if the authors report mean±std over seeds for the main tables and ablations, add a confidence-based selection control, ablate or justify the frozen gate, and validate the local score against the CH-SIMS modality-specific annotations.

Questions for authors

see above

Reviewer_bAZuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies language dominance in multimodal sentiment analysis. Strong text encoders make it possible to achieve a low training loss while only weakly exploiting audio and vision, and applying non-text supervision uniformly is suboptimal because audio and vision do not contribute unique sentiment evidence in every utterance. The paper proposes PIDFusion, which treats text and the joint audio-visual stream as two information sources and uses a BATCH-inspired neural probe with label-conditioned Sinkhorn transport to characterize redundant, unique, and synergistic information on detached features. A calibrated sample-wise audio-visual uniqueness score then determines which samples receive an auxiliary text-blind regression objective, while inference keeps the standard full-modality prediction. Experiments cover CMU-MOSI, CMU-MOSEI, and CH-SIMS, with matched-random and uniform-selection controls.

Critical strengths
  1. This paper raises a strong point. Prior methods add a separate loss branch for each modality, but audio-visual cues are not useful for every sample. Instead of asking how to make better use of non-text information, this paper asks where it is actually worth using.
Critical weaknesses
  1. All results appear to come from a single run, with no number of seeds or variance reported. The gains over baselines and the matched-random control are small, so it is unclear whether they reflect genuine improvement or training noise.
  2. The selection score in Eq. 16 does not use text at the sample level. Within a batch, \hat{U}{AV} and the batch mean of \iota{AV} are constants, so the ranking depends only on \iota_{AV,i}, i.e., how well the AV predictor alone predicts the label. The calibration in Eq. 18 has a positive slope and keeps this ranking. A redundant sample (text and AV both correct) and an AV-unique sample (only AV correct) with the same pAVp_{AV} thus get the same score. The rule acts as an AV-confidence filter, and the gain over matched-random in Table 3 does not show that the benefit comes from AV-unique information. A baseline that selects directly by ιAV,i\iota_{AV,i}, and one that uses a text-aware score such as ιAV,i−ιT,i\iota_{AV,i} - \iota_{T,i}, would clarify this.
Questions for authors
  1. Would it be possible to provide results over multiple random seeds on MOSI, MOSEI, and CH-SIMS, with means and standard deviations reported?
  2. It would also be helpful to see examples or statistics showing where audio-visual cues change the prediction compared with a text-only baseline, reported separately for samples selected by the threshold rule and for the remaining samples. This would directly illustrate whether the selected samples are indeed the ones where non-text information matters.
  3. Since the within-batch ranking in Eq. 16 depends only on ιAV,i\iota_{AV,i}, could the authors report the distribution of pT(ci∣zT,i)p_T(c_i \mid z_{T,i}) for selected versus non-selected samples? If most selected samples are already predicted correctly by text, what evidence shows that AV provides information beyond text on these samples?
Reviewer_2Ns1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes PIDFusion for MSA. It address language dominance through selective non-text supervision. It uses a partial information decomposition probe to estimate the information contributed by text and joint audio–visual representations. A calibrated local AV-unique score selects training samples for an auxiliary text-blind regression objective, encouraging the audio–visual branch to learn complementary sentiment cues. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show competitive performance.

Critical strengths
  1. The motivation is clear and soundness.
  2. The use of PID to guide auxiliary supervision is interesting.
Critical weaknesses
  1. The method introduces substantial complexity for relatively modest performance gains.
  2. The selection score’s interpretation as AV-unique information is insufficiently validated. I think that the comparisons with simpler confidence-based selectors are missing.
  3. The rationale for freezing the fusion gate and the necessity of injection calibration lack support.
Questions for authors
  1. The PID pipeline is complex. Could simpler selectors based on AV confidence or text–AV prediction differences achieve similar gains?
  2. What evidence shows that the selection score captures AV-unique information rather than AV predictability? Is injection calibration necessary?
  3. Why is the fusion gate frozen at initialization? Please compare it with a trainable gate and fixed mixing weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.