acceptodds
Under review as a conference paper at ICLR 2027

STREAM: Sample-Adaptive Two-level Routing Experts with Aligned Mamba for Multimodal Representation Learning

Abstract

Multimodal Sentiment Analysis (MSA) aims to understand human emotions by integrating acoustic, visual, and textual signals. However, existing approaches often rely on static fusion architectures with limited interpretability, overlooking the dynamic and hierarchical nature of cross-modal interactions. To address these limitations, we propose STREAM, an interpretable dynamic fusion framework for MSA. STREAM introduces an input-conditional operator routing mechanism combined with a Partial Information Decomposition (PID)-inspired Mixture-of-Experts design, enabling sample-specific modeling of unique, redundant, and synergistic cross-modal patterns. Furthermore, STREAM incorporates hierarchical representation alignment and a Collaborative Bi-Mamba module to effectively capture interactions across different semantic levels and temporal dependencies. Beyond prediction, STREAM provides interpretable insights by analyzing routing behaviors, expert activation patterns, and modality-specific contributions. We evaluate STREAM on three benchmark datasets, including CMU-MOSI, CMUMOSEI, and CH-SIMSv2. Experimental results demonstrate competitive performance across these benchmarks, achieving state-of-the-art results on CH-SIMSv2 and strong results on CMU-MOSEI, with an Acc-7 of 54.58%, Acc-2 of 85.74%, MAE of 0.531, and Corr of 0.772. By offering both adaptive fusion and interpretable decision analysis, STREAM provides a promising foundation for future affective computing applications, including intelligent human-computer interaction and emotion-aware systems.

Primary area:
interpretability and explainable AI
Keywords:
Routing Experts, Interpretable Dynamic Fusion, Multimodal Sentiment Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9AHcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STREAM, a multimodal sentiment analysis framework combining hierarchical representation learning, input-conditional operator routing, PID-inspired mixture-of-experts, cross-level alignment, and Collaborative Bi-Mamba fusion. The model constructs low- and high-level modality representations and adaptively routes them through different operators and experts to capture heterogeneous cross-modal interactions. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMSv2 report competitive performance, supported by component ablations and routing behavior analyses.

Critical strengths
  1. The hierarchical routing framework is reasonably motivated. Separating low- and high-level representations and allowing sample-dependent operator and expert selection provides a flexible approach to modeling heterogeneous cross-modal interactions.

  2. The paper provides relatively extensive experimental analyses.

Critical weaknesses
  1. Eqs. (6)–(8) impose different responses to modality perturbations, but these behaviors do not necessarily correspond to unique, redundant, and synergistic task-relevant information. In particular, Eq. (7) encourages the Synergy Expert to produce dissimilar representations under modality masking, which may primarily promote sensitivity to perturbations rather than information that is predictive only through joint modality interactions. Similarly, the stability encouraged by Eq. (8) does not necessarily establish that the Redundancy Expert captures shared sentiment-relevant information. Although Appendix E.5 provides representation divergence and perturbation-sensitivity analyses, these results mainly confirm the behaviors encouraged by the training objectives rather than independently validating the intended information decomposition. More direct evidence of the experts' predictive roles would strengthen the central methodological claim.

  2. I²MoE [1] already introduces interaction-specific experts with weakly supervised objectives and sample-level reweighting to model unique, redundant, and synergistic multimodal information, making it particularly relevant to the proposed APMoE. However, the paper does not adequately discuss this connection or isolate the contributions beyond existing interaction-aware expert designs. Moreover, recent methods such as TEXT [2], which combines sparse expert routing with Mamba-based temporal alignment for multimodal sentiment analysis, are absent from the comparisons. The authors also acknowledge that the main baseline results are quoted rather than reproduced under a common feature and evaluation protocol. Consequently, the current comparisons do not sufficiently establish whether the reported gains arise from the proposed routing and alignment mechanisms or from differences in experimental settings and architectural capacity.

Refs:

[1] I²MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts, Jiayi Xin et al., ICML 2025.

[2] A Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis, Dongning Rao et al., AAAI 2026.

Questions for authors
  1. What evidence supports interpreting the perturbation-based objectives in Eqs. (6)–(8) as learning unique, redundant, and synergistic task-relevant information, rather than different forms of perturbation sensitivity? In particular, how does Eq. (7) distinguish genuine cross-modal synergy from sensitivity to modality corruption?

  2. Do the existing analyses show that the Synergy Expert captures predictive information unavailable from individual modalities, and that the Redundancy Expert preserves sentiment-relevant information shared across modalities? How do the authors distinguish these properties from the representation stability and divergence reported in Appendix E.5?

  3. How does APMoE differ methodologically from I²MoE [1], particularly regarding interaction-specific experts, perturbation-based supervision, and sample-level expert weighting? Which contributions are specifically attributable to the proposed two-level operator routing?

  4. Please broaden the related-work discussion and experimental comparison to include closely related approaches such as I²MoE [1] and TEXT [2], where the tasks and evaluation settings are comparable. Clarifying the differences in feature extraction and evaluation protocols would also strengthen the empirical assessment.

  5. Please consider more targeted controlled comparisons that isolate the benefits of the two-level routing mechanism, PID-inspired objectives, and Collaborative Bi-Mamba against simpler alternatives with comparable model capacity. These would help determine whether the improvements arise from the proposed mechanisms rather than increased architectural complexity.

More direct validation of the expert roles and clearer comparisons with recent approaches could substantially improve my assessment.

2 comments

We have conducted complementary experiments in response to the comments. The revised PDF will be uploaded soon.

Q1/Q2/W1: Can perturbation objectives establish task-relevant unique, redundant, and synergistic information?

A: Perturbation losses alone show training responses; new tests establish learned information and predictive value. In Table newa, frozen probes on mixed-factor samples recover modality-specific factors from the matched Uniqueness Experts (95.6/94.3/93.7%, versus ~52% for unmatched factors), the shared factor from Redundancy (96.1%), and joint-only XOR-2/3 from Synergy (95.1/92.2%). Synergy therefore encodes joint-dependent information beyond corruption sensitivity. In Table newb, adding frozen Synergy or Redundancy representations to the same unimodal additive predictor reduces CMU-MOSEI MAE 0.574→0.546/0.555; all experts reach 0.533, establishing sentiment relevance. Table newc provides direct interventions: removing Synergy raises XOR-2/3 loss 24.6/27.3%, while removing Redundancy raises shared-factor loss 21.8%. On CH-SIMSv2, Redundancy removal affects consensus more (12.4% vs 3.9%), and Synergy removal affects conflict more (13.7% vs 4.2%). Together, probing, prediction, and knockout results establish distinct, task-relevant roles.

Q3/W2: How does STREAM differ from I²MoE, and what does two-level routing contribute?

A: I²MoE uses interaction-specific experts, perturbation-guided supervision, and sample-dependent expert weights. STREAM additionally selects input-dependent Top-K operators independently at low and high semantic levels, adapting transformations beyond expert reweighting. Table newd isolates this contribution: static routing achieves 83.15 Acc-2 / 0.539 MAE, single-level dynamic routing 84.82 / 0.535, and two-level STREAM 85.74 / 0.531. This isolates the gain from dynamic, level-specific operator selection.

Q4/W2: Comparison with I²MoE/TEXT and feature/evaluation protocols?

A: We expanded related work and added I²MoE-style and TEXT-style controls under the same backbone, features, data splits, evaluation protocol, and comparable parameter budgets. In Table newd, I²MoE-style achieves 83.99/0.538, TEXT-style 84.44/0.537, and STREAM 85.74/0.531 (Acc-2/MAE). The TEXT-style control uses text-routed experts without extra inputs. The Reproducibility Guide specifies pre-extracted text/audio/visual features, fixed splits, continuous-score MAE/Corr, discretized Acc-5/7, and binary Acc-2 excluding neutral ground-truth labels. Thus all mechanisms are compared on the same empirical basis.

Q5/W2: Are routing, PID objectives, and Bi-Mamba isolated under matched capacity?

A: Yes. Both the original ablations and new comparisons control parameter budgets under identical settings. In Table newd, removing PID experts and loss gives 82.93/0.541; retaining experts without the loss gives 83.92/0.539, and standard experts with the loss give 83.53/0.540. Removing Collaborative Bi-Mamba gives 82.38/0.536, versus full STREAM's 85.74/0.531 (Acc-2/MAE). Combined with the routing controls, these isolated controls show that the improvements come from the proposed mechanisms, rather than added capacity.


Table newa — Frozen synthetic-factor probes: accuracy (%) and nuisance shift.

ExpertU-TU-AU-VSharedXOR-2XOR-3Shift↓
Text-Uni95.651.652.574.151.850.90.014
Audio-Uni51.794.352.073.551.552.00.015
Vision-Uni52.452.293.773.752.852.10.016
Redundancy53.854.253.696.153.453.10.009
Synergy53.753.052.956.395.192.20.019
Standard MoE71.871.069.982.883.577.60.046

Table newb — MOSEI frozen-feature prediction (selected).

FeaturesMAE↓Corr↑
Unimodal additive0.5740.743
+ Generic expert0.5600.753
+ Redundancy0.5550.756
+ Synergy0.5460.764
+ All PID experts0.5330.771

Table newc — Expert knockout: relative task-loss increase (%).

RemovedUniqueSharedXOR-2XOR-3ConsensusConflict
Matched Uni18.72.52.22.0——
Unmatched Uni2.81.71.91.5——
Redundancy3.021.83.33.012.44.2
Synergy2.52.824.627.33.913.7

Synthetic: cross-entropy; CH-SIMSv2 consensus (n=459) and conflict (n=149): MAE.

Table newd — CMU-MOSEI capacity-controlled ablations.

ConfigurationAcc-2↑MAE↓
w/o PID Experts & Loss82.930.541
w/o DCCA80.240.533
w/o Bi-Mamba82.380.536
PID Experts w/o PID Loss83.920.539
Standard Experts + PID Loss83.530.540
Static Global Routing83.150.539
Single-Level Dynamic84.820.535
I²MoE-style (same backbone)83.990.538
TEXT-style (no explanations)84.440.537
Full STREAM85.740.531

We have conducted complementary experiments in response to the comments. The revised PDF has now been uploaded. We summarize the changes and their locations below.

1. Validation of PID-inspired expert roles (W1, Q1, Q2).

We agree that the perturbation-based objectives in Eqs. (6)–(8) are surrogate training signals and do not, by themselves, establish an exact information-theoretic decomposition. We now clarify this limitation in Section 3.4 and add three complementary tests. Frozen-factor probes recover matched unique factors (93.7–95.6%), shared information through Redundancy (96.1%), and joint-only XOR factors through Synergy (92.2–95.1%), while unrelated unique factors remain near chance (Appendix E.6, Table 13). To test sentiment relevance rather than perturbation sensitivity alone, frozen Synergy and Redundancy features reduce CMU-MOSEI MAE from 0.574 to 0.546 and 0.555, respectively; combining all PID experts reaches 0.533 (Table 14). Expert-output knockouts further show selective prediction losses, including larger effects of Synergy removal on XOR and conflict cases and of Redundancy removal on shared and consensus cases (Table 15). These findings support distinct predictive roles.

2. Methodological distinction from I²MoE and two-level routing (W2, Q3).

We have expanded the discussion of I²MoE in Related Work (p. 2). Both approaches use interaction-aware experts and sample-dependent expert weighting, but STREAM additionally makes input-conditional Top-K operator choices at the edge and expert-node stages, independently at low and high semantic levels (Sections 2.3 and 3.2). Under comparable parameter budgets, static, single-level dynamic, and full two-level routing achieve Acc-2 of 83.15%, 84.82%, and 85.74%, respectively (MAE 0.539, 0.535, and 0.531). This isolates the benefit of adapting the transformation operators, not just the expert weights (Appendix E.7, Table 16).

3. Comparable I²MoE/TEXT controls and evaluation protocols (W2, Q4).

We added I²MoE-style and TEXT-style controls using the same hierarchical backbone, extracted features, data splits, evaluation protocol, and comparable parameter budgets. On CMU-MOSEI, they achieve Acc-2/MAE of 83.99/0.538 and 84.44/0.537, versus 85.74/0.531 for STREAM (Appendix E.7, Table 16). We also specify feature inputs, label discretization, zero-label handling, and evaluation metrics in the Reproducibility Guide.

4. Isolating routing, PID supervision, and Bi-Mamba (W2, Q5).

We extended the ablations under matched experimental conditions and comparable capacity (Appendix E.7, Table 16). Removing both PID experts and loss yields 82.93 Acc-2 / 0.541 MAE. Keeping the PID experts without their auxiliary loss gives 83.92 / 0.539; using standard experts with that loss gives 83.53 / 0.540. Removing Collaborative Bi-Mamba gives 82.38 / 0.536, versus 85.74 / 0.531 for full STREAM. Together with the static and single-level routing controls, these results distinguish architectural contributions from auxiliary supervision and overall capacity.

We appreciate the reviewer's guidance. The revised evidence more directly tests expert predictions, identifies the contribution of two-level operator routing, and clarifies the fairness and limits of our comparisons.

Reviewer_2ihPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STREAM, a sample-adaptive hierarchical fusion framework for multimodal sentiment analysis. The method introduces input-dependent operator routing at two semantic levels and employs experts inspired by Partial Information Decomposition to capture unique, redundant, and synergistic information across modalities. Cross-level alignment and collaborative bidirectional Mamba are further integrated to improve representation learning and temporal fusion. The framework aims to achieve adaptive multimodal fusion while providing insights into the internal routing behavior of the model.

Critical strengths

I find the sample-adaptive fusion design quite interesting. The model can adjust both the operations it uses and the way it combines information from different modalities. Allowing the low-level and high-level branches to make separate routing decisions also seems well motivated, given that multimodal interactions can differ across semantic levels. I also appreciate the effort to examine the learned routing behavior, which makes the model's internal processing easier to understand.

Critical weaknesses

My main concern is that the central contribution gets somewhat lost among the many components. The framework combines hierarchical routing, specialized experts, representation alignment, and temporal fusion, making it difficult to identify a particularly distinctive methodological idea. I am also not entirely convinced by the interpretability aspect. The routing visualizations help explain which operators and experts the model tends to favor, but they tell us less about how the model arrives at a particular sentiment prediction.

This is also why I find the paper's presentation rather confusing. First, I believe the figures should be provided in vector formats, such as SVG or PDF. The information density is highly uneven: the method diagrams are very crowded, while the experimental figures contain relatively little information. The main text also feels quite sparse. Several subsections, particularly in Related Work, are only a few lines long, and there are many awkward line breaks that leave a single word at the end of a paragraph, especially in the experimental section. Table 1 is poorly placed, and the heading of Section 3.1 should not be left alone at the bottom of a page. Overall, I think the paper has considerable room for improvement in readability. Given these broader presentation issues, the occasional incorrect symbols and stray text in the figures are relatively minor concerns.

I also want to raise a concern about the mathematical presentation. Part of my confusion comes from the difficulty of matching the equations in the main text to Figures 1 and 2. Many of the paper's methodological contributions become difficult to follow as a result. For example, what exactly is the structure of gm,eopg_{m,e}^{\mathrm{op}}? Is gg, without the superscript, used anywhere else in the paper? If not, why is the superscript necessary? If this level of notation is indeed needed, I would suggest adding a notation table or a dedicated subsection explaining how the symbols are defined and related to one another.

Questions for authors

I would also appreciate clarification on the following questions:

  1. How are the weight matrices WW implemented? Are they different from the Linear layers mentioned elsewhere in the paper? If they are equivalent, why does the paper use both matrix notation and Linear? If they are different, what is the mathematical distinction?

  2. How exactly are the values in Table 7 calculated? Could the authors provide the raw results for each run?

  3. Does the model use sparse expert gating, as shown in Figure 1, or dense expert gating, as described in Section 3.2.2?

  4. The reported MAE values for Self-MM and MMIM appear to differ from those in their original papers. Could the authors explain these discrepancies?

  5. How are the labels preprocessed for five-class classification? Are samples with a sentiment label of zero included?

  6. How does Figure 3 support the claim that text favors sequence-modeling operators, while audio and vision favor local convolutions? I find it difficult to reconcile this interpretation with the distributions shown in the figure.

  7. What exactly are the inputs to the temporal fusion module?

7 comments

In response to the comments, we have performed complementary experiments and fixed the presentation and notation issues described below. The revised PDF will be uploaded soon.

W1: Core novelty and prediction-level interpretability.

A: STREAM selects input-dependent operators separately at low and high semantic levels, beyond reweighting fixed experts. Table newd shows static / single-level / dual-level routing improves Acc-7: 53.07 / 54.05 / 54.58 and MAE: 0.539 / 0.535 / 0.531. Thus both dynamic selection and level-specific decisions matter. Interpretability is supported by predictive evidence: Table newa shows selective recovery of unique (93.7–95.6%), shared (96.1%), and XOR (92.2–95.1%) factors. In Table newb, Synergy features reduce MAE 0.574→0.546, outperforming generic experts (0.560). Table newc shows corresponding, task-specific prediction losses upon expert knockout. Together, these experiments demonstrate meaningful expert specialization and predictive contributions.

W2: Figure, typography, and layout issues.

A: The revised PDF uses vector figures, simplifies diagrams, fixes symbols, expands Related Work, and improves line/page breaks. Table 1 is repositioned, and Section 3.1 no longer has an orphaned heading.

W3: Mathematical notation and figure correspondence.

A: XmLL,XmHLX_m^{LL},X_m^{HL} are modality-wise APMoE inputs; XLL,XHLX_{LL},X_{HL} are fused outputs; ZL,ZHZ_L,Z_H are DCCA-aligned temporal-fusion inputs. A notation mapping links the symbols to Figures 1–2.

Q1: Are Wg,WnW_g,W_n different from Linear layers?

A: No; both are learned Linear-layer matrices: Linear(x) = Wx+bWx+b. WgW_g produces gate logits; WnW_n controls training-time noise scale. Their parameters are independent.

Q2: How are Table 7 and its five runs calculated?

A: The five random seeds are 42, 123, 74, 2025, 55654. We report the arithmetic mean ± sample standard deviation (denominator n−1n-1). All five raw runs and aggregates appear in Appendix D, “Multi-seed Stability Analysis,” Table 7 (Acc-2 85.74 ± 0.13; MAE 0.5310 ± 0.0020).

Q3: Sparse or dense expert gating?

A: Sparse Top-K is used for operator selection; dense softmax is used for PID expert aggregation. They are two distinct, complementary gates, explaining Figure 1 and Section 3.2.2.

Q4: Why do Self-MM/MMIM MAEs differ from the originals?

A: Both are results from our own reproductions under our experimental feature, training, and evaluation settings, not numbers copied from the original papers. Differences reflect these reproduction settings and implementations. Our tables report the experimentally reproduced scores.

Q5: Five-class labels and zero samples?

A: For MOSI/MOSEI, Acc-5 rounds scores and clips to [−2,2][-2,2]; zeros remain included. Only Acc-2 excludes zero ground-truth labels. CH-SIMSv2 uses annotated five-class labels; MAE/Corr include zeros.

Q6: How does Figure 3 show operator preferences?

A: The figure establishes modality- and level-specific choices. At the high level, text/vision favor Transformer (39.9%/50.5%), while audio favors ChannelAttn (44.0%). At the low level, text favors DepthwiseConv (32.6%), audio Linear+GLU (30.7%), and vision SelfAttn (66.3%). Text therefore emphasizes sequence-level semantics at high levels and local processing at low levels; audio and vision likewise change their preferred operations with semantic level. These differences explain why separate, adaptive operator routers are needed.

Q7: Inputs to temporal fusion?

A: DCCA-aligned sequences ZL,ZHZ_L,Z_H, processed by shared-global and level-private forward/backward Bi-Mamba pathways, then gated and pooled for regression.


Table newa — Frozen factor probes (accuracy %,).

ExpertU-TU-AU-VSharedXOR-2XOR-3
Text-Uni95.651.652.574.151.850.9
Audio-Uni51.794.352.073.551.552.0
Vision-Uni52.452.293.773.752.852.1
Redundancy53.854.253.696.153.453.1
Synergy53.753.052.956.395.192.2

Table newb — MOSEI frozen-feature prediction.

FeaturesMAE↓Corr↑
Unimodal additive0.5740.743
+ Generic0.5600.753
+ Synergy0.5460.764
+ All PID0.5330.771

Table newc — Expert knockout: relative loss increase (%).

RemovedUniqueSharedXOR-2XOR-3ConsensusConflict
Matched Uni18.72.52.22.0——
Unmatched Uni2.81.71.91.5——
Redundancy3.021.83.33.012.44.2
Synergy2.52.824.627.33.913.7

Table newd — MOSEI routing ablations.

ConfigurationAcc-7↑MAE↓
Static global53.070.539
Single-level54.050.535
Shared router weights53.590.537
Full STREAM54.580.531
Reviewer_2ihPOpenReview ↗

Acknowledgment and Question on Random Seed Selection

Thank you for the detailed response. I appreciate your efforts in addressing my comments, and I look forward to reading the revised manuscript so that I can thoroughly review the changes and reconsider my evaluation.

Before we proceed, I would appreciate it if you could share the reasoning behind your choice of these specific random seeds.

Thank you for your follow-up question.

We noticed that several studies in the multimodal sentiment analysis (MSA) field have employed this set of random seeds (42, 123, 74, 2025, and 55654) for multi-run evaluations. In addition, we deliberately used numerically dispersed rather than consecutive seeds to explore different random initialization and data-shuffling configurations, while avoiding reliance on a narrowly clustered set of seed values.

We sincerely welcome any further questions or suggestions and would be happy to provide additional clarifications. We also reaffirm our commitment to preparing and uploading the revised manuscript as soon as possible.

Thank you again for your time and constructive feedback.

Reviewer_KyT1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STREAM, a multimodal sentiment analysis framework integrating hierarchical representations, adaptive operator routing, PID-inspired experts, DCCA alignment, and Bi-Mamba fusion. Experiments on three benchmarks demonstrate competitive performance, supported by ablation and interpretability analyses.

Critical strengths
  1. The hierarchical adaptive routing design provides a reasonable approach to modeling sample-dependent multimodal interactions.
  2. STREAM achieves competitive performance across three benchmark datasets.
  3. Extensive ablation studies and expert behavior analyses support the effectiveness of the overall framework.
Critical weaknesses
  1. The PID-inspired losses do not necessarily capture information-theoretic uniqueness, redundancy, and synergy. Stronger validation is needed.
  2. The novelty of combining existing techniques is limited, and the specific benefits of dynamic operator routing remain unclear.
  3. The experiments lack parameter-matched baselines and sufficiently isolated component ablations.
  4. Routing visualizations alone do not establish faithful interpretability.
Questions for authors
  1. How can the authors verify that the PID-inspired experts genuinely capture distinct information types?
  2. What advantages does dynamic operator routing offer over conventional MoE or static operator selection?
  3. Could the authors independently ablate the PID-inspired expert structure and auxiliary losses?4
  4. Why does removing DCCA substantially reduce Acc-2 while barely affecting MAE?
  5. Could intervention-based experiments further validate the interpretability claims?
2 comments

We thank the reviewer for recognizing STREAM's adaptive design and experiments. Our new controlled studies address the concerns. The revised PDF will be uploaded soon.

Q1/W1: Do PID experts capture distinct information?

A: Yes. Table newa verifies distinct information capture. We embed unique, shared, and joint factors in the same samples, train without factor labels, and probe frozen experts. Each Uniqueness Expert accurately recovers its own factor (95.6/94.3/93.7%), while decoding unrelated unique factors stays near chance (~52%). Redundancy preferentially recovers the shared factor (96.1%); Synergy recovers joint-only XOR-2/3 (95.1/92.2%), but not individual unique factors. This contrast reveals encoded information, not just responses to masking. Low nuisance shifts (0.009–0.019 vs 0.046 for pooled MoE) further show that specialization is robust to irrelevant variation.

Q2/W2: Advantage over conventional MoE or static routing?

A: Conventional MoE adapts expert weights, not level-specific operators; static routing uses the same operators for all samples. STREAM performs input-conditioned Top-K operator selection separately at low and high levels, followed by expert weighting. Thus both transformation type and semantic-level strategy adapt to each sample. Under comparable budgets (Table newd), static → single-level → two-level yields Acc-2 83.15 → 84.82 → 85.74 and MAE 0.539 → 0.535 → 0.531. These gains establish the value of dynamic, level-specific operator choices beyond expert reweighting.

Q3/W3: Independent expert/loss ablations?

A: Table newd isolates both factors under a matched protocol and comparable parameter budgets. Removing experts and losses yields 82.93 Acc-2 / 0.541 MAE; keeping PID experts without the loss gives 83.92 / 0.539; using standard experts with the loss gives 83.53 / 0.540, versus 85.74 / 0.531 for STREAM. Both the expert structure and supervision contribute; their combination delivers the strongest result.

Q4: Why does removing DCCA sharply reduce Acc-2 but barely change MAE?

A: Without DCCA, Acc-2 falls 85.74 → 80.24 (−5.50 points), while MAE changes only 0.531 → 0.533 (Table newd). Acc-2 depends on the prediction's sign; MAE depends on its numerical distance from the label. Over 45% of CMU-MOSEI labels satisfy |y| ≤ 1, covering weakly polarized and neutral sentiment. In this substantial low-intensity region, a small shift across zero changes the polarity decision but adds little absolute error. DCCA's cross-level alignment therefore matters especially for polarity discrimination, explaining the pronounced Acc-2 loss alongside nearly unchanged MAE.

Q5/W4: Intervention-based interpretability?

A: Table newc directly tests expert contributions by intervention. At inference, we zero one expert output while fixing all others, gates, and parameters. Removing matched versus unmatched Uniqueness raises task loss 18.7% vs 2.8%; removing Redundancy primarily harms shared-factor prediction (21.8%), whereas removing Synergy primarily harms XOR-2/3 (24.6/27.3%). On CH-SIMSv2, Redundancy removal hurts consensus more (12.4% vs 3.9%), while Synergy removal hurts conflict more (13.7% vs 4.2%). These selective prediction effects confirm meaningful expert roles beyond routing plots.


Table newa — Frozen synthetic-factor probe accuracy (%) and nuisance shift.

ExpertU-TU-AU-VSharedXOR-2XOR-3Shift↓
Text-Uni95.651.652.574.151.850.90.014
Audio-Uni51.794.352.073.551.552.00.015
Vision-Uni52.452.293.773.752.852.10.016
Redundancy53.854.253.696.153.453.10.009
Synergy53.753.052.956.395.192.20.019
Standard MoE (pooled)71.871.069.982.883.577.60.046

Table newc — Expert knockout: relative task-loss increase (%).

RemovedUniqueSharedXOR-2XOR-3ConsensusConflict
Matched Uni18.72.52.22.0——
Unmatched Uni2.81.71.91.5——
Redundancy3.021.83.33.012.44.2
Synergy2.52.824.627.33.913.7

Synthetic: cross-entropy; CH-SIMSv2 consensus (n=459) and conflict (n=149): MAE.

Table newd — CMU-MOSEI controlled ablations.

ConfigurationAcc-7Acc-5Acc-2MAE↓Corr↑
w/o PID Experts & Loss53.2153.8682.930.5410.763
w/o DCCA53.4454.0780.240.5330.768
w/o Bi-Mamba53.4053.7482.380.5360.767
PID Experts w/o PID Loss53.8054.7083.920.5390.765
Standard Experts + PID Loss53.5154.3783.530.5400.764
Static Global Routing53.0754.1383.150.5390.766
Single-Level Dynamic54.0555.1884.820.5350.769
Full STREAM54.5856.1085.740.5310.772

We thank the reviewer for the constructive feedback. The revised manuscript now includes additional controlled experiments and clarifications. We respond to each original concern below.

1. PID-inspired expert specialization (W1, Q1).

We agree that PID-inspired losses are surrogate objectives, not guarantees of exact information-theoretic decomposition, and clarify this limitation in Section 3.4. New frozen-factor probes show that Uniqueness, Redundancy, and Synergy Experts preferentially recover modality-specific, shared, and joint-only factors, respectively. Importantly, frozen-representation prediction on CMU-MOSEI further demonstrates their complementary task-relevant contributions, beyond synthetic factor recovery. Details are in Appendix E.6, Tables 13–14.

2. Novelty and benefits of dynamic routing (W2, Q2).

Unlike conventional MoE reweighting, STREAM performs input-conditional Top-K operator selection at both edge and expert-node stages, independently across two semantic levels (Sections 2.3 and 3.2). With comparable parameter budgets, Acc-2 rises from 83.15% (static) to 84.82% (single-level dynamic) and 85.74% (full STREAM). We also add I²MoE-style and TEXT-style controls using the same hierarchical backbone. These comparisons further demonstrate the effectiveness of STREAM's hierarchical operator-level adaptation over alternative interaction modeling and routing strategies (Appendix E.7, Table 16).

3. Independent component ablations (W3, Q3).

We now separately evaluate PID-inspired experts without their auxiliary loss and standard experts with that loss. Both improve over removing the experts and loss together, while full STREAM performs best. This distinguishes architectural benefits from those of auxiliary supervision under a consistent, capacity-controlled protocol. See Appendix E.7, Table 16.

4. DCCA's different effects on Acc-2 and MAE (Q4).

We added paired, sample-level analysis: without DCCA, 223 predictions change from correct to incorrect polarity versus 23 in the reverse direction. Most flips occur close to the zero threshold, sharply affecting Acc-2 without necessarily increasing absolute error much. An MAE contribution analysis further explains the small net change. See Appendix E.7, “DCCA and Metric Sensitivity”; Table 16.

5. Intervention-based interpretability (W4, Q5).

We now remove individual expert outputs at inference while keeping the model, other outputs, and original gate weights fixed. On synthetic tasks, the greatest degradation follows removal of the expert matched to the target factor. On CH-SIMSv2, Redundancy removal affects consensus more, while Synergy removal affects conflict more. This provides evidence of differentiated predictive contributions beyond routing plots, without treating gate weights as causal explanations (Appendix E.6, Table 15).

We appreciate the reviewer's guidance. These revisions strengthen the empirical support for STREAM while clarifying the scope of our claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.