- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes STREAM, a multimodal sentiment analysis framework combining hierarchical representation learning, input-conditional operator routing, PID-inspired mixture-of-experts, cross-level alignment, and Collaborative Bi-Mamba fusion. The model constructs low- and high-level modality representations and adaptively routes them through different operators and experts to capture heterogeneous cross-modal interactions. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMSv2 report competitive performance, supported by component ablations and routing behavior analyses.
Critical strengths
-
The hierarchical routing framework is reasonably motivated. Separating low- and high-level representations and allowing sample-dependent operator and expert selection provides a flexible approach to modeling heterogeneous cross-modal interactions.
-
The paper provides relatively extensive experimental analyses.
Critical weaknesses
-
Eqs. (6)–(8) impose different responses to modality perturbations, but these behaviors do not necessarily correspond to unique, redundant, and synergistic task-relevant information. In particular, Eq. (7) encourages the Synergy Expert to produce dissimilar representations under modality masking, which may primarily promote sensitivity to perturbations rather than information that is predictive only through joint modality interactions. Similarly, the stability encouraged by Eq. (8) does not necessarily establish that the Redundancy Expert captures shared sentiment-relevant information. Although Appendix E.5 provides representation divergence and perturbation-sensitivity analyses, these results mainly confirm the behaviors encouraged by the training objectives rather than independently validating the intended information decomposition. More direct evidence of the experts' predictive roles would strengthen the central methodological claim.
-
I²MoE [1] already introduces interaction-specific experts with weakly supervised objectives and sample-level reweighting to model unique, redundant, and synergistic multimodal information, making it particularly relevant to the proposed APMoE. However, the paper does not adequately discuss this connection or isolate the contributions beyond existing interaction-aware expert designs. Moreover, recent methods such as TEXT [2], which combines sparse expert routing with Mamba-based temporal alignment for multimodal sentiment analysis, are absent from the comparisons. The authors also acknowledge that the main baseline results are quoted rather than reproduced under a common feature and evaluation protocol. Consequently, the current comparisons do not sufficiently establish whether the reported gains arise from the proposed routing and alignment mechanisms or from differences in experimental settings and architectural capacity.
Refs:
[1] I²MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts, Jiayi Xin et al., ICML 2025.
[2] A Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis, Dongning Rao et al., AAAI 2026.
Questions for authors
-
What evidence supports interpreting the perturbation-based objectives in Eqs. (6)–(8) as learning unique, redundant, and synergistic task-relevant information, rather than different forms of perturbation sensitivity? In particular, how does Eq. (7) distinguish genuine cross-modal synergy from sensitivity to modality corruption?
-
Do the existing analyses show that the Synergy Expert captures predictive information unavailable from individual modalities, and that the Redundancy Expert preserves sentiment-relevant information shared across modalities? How do the authors distinguish these properties from the representation stability and divergence reported in Appendix E.5?
-
How does APMoE differ methodologically from I²MoE [1], particularly regarding interaction-specific experts, perturbation-based supervision, and sample-level expert weighting? Which contributions are specifically attributable to the proposed two-level operator routing?
-
Please broaden the related-work discussion and experimental comparison to include closely related approaches such as I²MoE [1] and TEXT [2], where the tasks and evaluation settings are comparable. Clarifying the differences in feature extraction and evaluation protocols would also strengthen the empirical assessment.
-
Please consider more targeted controlled comparisons that isolate the benefits of the two-level routing mechanism, PID-inspired objectives, and Collaborative Bi-Mamba against simpler alternatives with comparable model capacity. These would help determine whether the improvements arise from the proposed mechanisms rather than increased architectural complexity.
More direct validation of the expert roles and clearer comparisons with recent approaches could substantially improve my assessment.
2 comments
We have conducted complementary experiments in response to the comments. The revised PDF has now been uploaded. We summarize the changes and their locations below.
1. Validation of PID-inspired expert roles (W1, Q1, Q2).
We agree that the perturbation-based objectives in Eqs. (6)–(8) are surrogate training signals and do not, by themselves, establish an exact information-theoretic decomposition. We now clarify this limitation in Section 3.4 and add three complementary tests. Frozen-factor probes recover matched unique factors (93.7–95.6%), shared information through Redundancy (96.1%), and joint-only XOR factors through Synergy (92.2–95.1%), while unrelated unique factors remain near chance (Appendix E.6, Table 13). To test sentiment relevance rather than perturbation sensitivity alone, frozen Synergy and Redundancy features reduce CMU-MOSEI MAE from 0.574 to 0.546 and 0.555, respectively; combining all PID experts reaches 0.533 (Table 14). Expert-output knockouts further show selective prediction losses, including larger effects of Synergy removal on XOR and conflict cases and of Redundancy removal on shared and consensus cases (Table 15). These findings support distinct predictive roles.
2. Methodological distinction from I²MoE and two-level routing (W2, Q3).
We have expanded the discussion of I²MoE in Related Work (p. 2). Both approaches use interaction-aware experts and sample-dependent expert weighting, but STREAM additionally makes input-conditional Top-K operator choices at the edge and expert-node stages, independently at low and high semantic levels (Sections 2.3 and 3.2). Under comparable parameter budgets, static, single-level dynamic, and full two-level routing achieve Acc-2 of 83.15%, 84.82%, and 85.74%, respectively (MAE 0.539, 0.535, and 0.531). This isolates the benefit of adapting the transformation operators, not just the expert weights (Appendix E.7, Table 16).
3. Comparable I²MoE/TEXT controls and evaluation protocols (W2, Q4).
We added I²MoE-style and TEXT-style controls using the same hierarchical backbone, extracted features, data splits, evaluation protocol, and comparable parameter budgets. On CMU-MOSEI, they achieve Acc-2/MAE of 83.99/0.538 and 84.44/0.537, versus 85.74/0.531 for STREAM (Appendix E.7, Table 16). We also specify feature inputs, label discretization, zero-label handling, and evaluation metrics in the Reproducibility Guide.
4. Isolating routing, PID supervision, and Bi-Mamba (W2, Q5).
We extended the ablations under matched experimental conditions and comparable capacity (Appendix E.7, Table 16). Removing both PID experts and loss yields 82.93 Acc-2 / 0.541 MAE. Keeping the PID experts without their auxiliary loss gives 83.92 / 0.539; using standard experts with that loss gives 83.53 / 0.540. Removing Collaborative Bi-Mamba gives 82.38 / 0.536, versus 85.74 / 0.531 for full STREAM. Together with the static and single-level routing controls, these results distinguish architectural contributions from auxiliary supervision and overall capacity.
We appreciate the reviewer's guidance. The revised evidence more directly tests expert predictions, identifies the contribution of two-level operator routing, and clarifies the fairness and limits of our comparisons.
We have conducted complementary experiments in response to the comments. The revised PDF will be uploaded soon.
Q1/Q2/W1: Can perturbation objectives establish task-relevant unique, redundant, and synergistic information?
A: Perturbation losses alone show training responses; new tests establish learned information and predictive value. In Table newa, frozen probes on mixed-factor samples recover modality-specific factors from the matched Uniqueness Experts (95.6/94.3/93.7%, versus ~52% for unmatched factors), the shared factor from Redundancy (96.1%), and joint-only XOR-2/3 from Synergy (95.1/92.2%). Synergy therefore encodes joint-dependent information beyond corruption sensitivity. In Table newb, adding frozen Synergy or Redundancy representations to the same unimodal additive predictor reduces CMU-MOSEI MAE 0.574→0.546/0.555; all experts reach 0.533, establishing sentiment relevance. Table newc provides direct interventions: removing Synergy raises XOR-2/3 loss 24.6/27.3%, while removing Redundancy raises shared-factor loss 21.8%. On CH-SIMSv2, Redundancy removal affects consensus more (12.4% vs 3.9%), and Synergy removal affects conflict more (13.7% vs 4.2%). Together, probing, prediction, and knockout results establish distinct, task-relevant roles.
Q3/W2: How does STREAM differ from I²MoE, and what does two-level routing contribute?
A: I²MoE uses interaction-specific experts, perturbation-guided supervision, and sample-dependent expert weights. STREAM additionally selects input-dependent Top-K operators independently at low and high semantic levels, adapting transformations beyond expert reweighting. Table newd isolates this contribution: static routing achieves 83.15 Acc-2 / 0.539 MAE, single-level dynamic routing 84.82 / 0.535, and two-level STREAM 85.74 / 0.531. This isolates the gain from dynamic, level-specific operator selection.
Q4/W2: Comparison with I²MoE/TEXT and feature/evaluation protocols?
A: We expanded related work and added I²MoE-style and TEXT-style controls under the same backbone, features, data splits, evaluation protocol, and comparable parameter budgets. In Table newd, I²MoE-style achieves 83.99/0.538, TEXT-style 84.44/0.537, and STREAM 85.74/0.531 (Acc-2/MAE). The TEXT-style control uses text-routed experts without extra inputs. The Reproducibility Guide specifies pre-extracted text/audio/visual features, fixed splits, continuous-score MAE/Corr, discretized Acc-5/7, and binary Acc-2 excluding neutral ground-truth labels. Thus all mechanisms are compared on the same empirical basis.
Q5/W2: Are routing, PID objectives, and Bi-Mamba isolated under matched capacity?
A: Yes. Both the original ablations and new comparisons control parameter budgets under identical settings. In Table newd, removing PID experts and loss gives 82.93/0.541; retaining experts without the loss gives 83.92/0.539, and standard experts with the loss give 83.53/0.540. Removing Collaborative Bi-Mamba gives 82.38/0.536, versus full STREAM's 85.74/0.531 (Acc-2/MAE). Combined with the routing controls, these isolated controls show that the improvements come from the proposed mechanisms, rather than added capacity.
Table newa — Frozen synthetic-factor probes: accuracy (%) and nuisance shift.
Table newb — MOSEI frozen-feature prediction (selected).
Table newc — Expert knockout: relative task-loss increase (%).
Synthetic: cross-entropy; CH-SIMSv2 consensus (n=459) and conflict (n=149): MAE.
Table newd — CMU-MOSEI capacity-controlled ablations.