acceptodds
Under review as a conference paper at ICLR 2027

MSSD: Multimodal Fusion as Shared-State Dynamics

Abstract

Multimodal sequence models commonly organize temporal processing around modality specific representations and introduce cross source interaction through separate fusion mechanisms. This leaves recurrent memory either partitioned by source or shared only after source observations have already been combined. We study an alternative organization in which heterogeneous observations remain distinct at the interface but act on one evolving multimodal memory. Multimodal Shared-State Dynamics (MSSD) realizes this view with a single modality free matrix state shared by all sources. Source conditioned write and read factors preserve distinct observation interfaces, while Dual-Axis dynamics jointly evolve latent feature and temporal memory structure within the same recurrent state. The same formulation accommodates different observation times without changing the shared recurrent organization. Across four text audio visual benchmarks, MSSD performs competitively with general and task specific multimodal models, while controlled comparisons and representation analyses support the roles of shared recurrent storage and Dual-Axis dynamics. MSSD thus formulates multimodal fusion as the evolution of one shared recurrent memory rather than as an auxiliary interaction stage around separate source trajectories.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal learning, multimodal sequence modeling, state space models, multimodal fusion, dynamical systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4bitOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multimodal Shared-State Dynamics (MSSD), which formulates multimodal fusion as the evolution of a single shared recurrent state. Unlike separate-stream and joint-input architectures, MSSD keeps the source interfaces distinct while all sources update one recurrent state without a modality-specific state dimension. The authors further introduce Dual-Axis Dynamics to jointly model dynamics along the latent feature and temporal memory dimensions. Experimental results show the effectiveness of the proposed method.

Critical strengths
  1. The proposed MSSD framework is presented in a relatively clear and organized manner.

  2. The paper focuses on an interesting problem.

Critical weaknesses
  1. The comparison of the baseline model is outdated and fails to demonstrate the competitiveness of the proposed model compared to the latest methods. Moreover, in the table, MSSD does not outperform the existing dedicated multimodal methods in many indicators. Therefore, the current experiments are insufficient to prove the competitiveness relative to the latest methods.

  2. How does MSSD adapt to the changes in the importance of modalities in different samples? Can some experimental verification methods effectively handle these variations?

  3. Although the paper discussed the relevant multimodal state-space methods, it did not conduct a direct comparison with these methods under a unified experimental setup. Therefore, it is still unclear whether the proposed approach has an advantage over the existing methods.

  4. The impact of model capacity differences on performance gains remains unclear. Moreover, the gains from Dual-Axis dynamics are not consistent across all evaluation metrics.

Questions for authors

Please refer to the weakness section.

2 comments

We appreciate the reviewer's detailed comments. We address the four questions below.

W1/W3. Recent multimodal SSMs

We additionally evaluated a Coupled Mamba-core adaptation under common feature and training settings. On MOSI, MSSD achieves 0.723 ± 0.004 MAE versus 0.752 ± 0.007 for the adaptation. This provides a controlled comparison, although it does not reproduce the original method's complete training recipe.

W2. Dynamic modality importance

MSSD uses input-dependent, source-conditioned write/read interfaces, allowing different modalities to contribute through distinct transformations. This supports adaptive source interactions but does not constitute explicit modality reliability estimation.

W4. Capacity and efficiency

We report parameter counts and inference latency alongside performance. MSSD uses 1.148M non-BERT parameters versus 0.837M for Joint Mamba-3, making the additional capacity and computational cost explicit.

W4. Dual-Axis effectiveness

Dual-Axis combines temporal evolution and latent-axis interactions, with dataset-dependent benefits. Temporal-only performs slightly better on MOSI (0.720 vs. 0.723 MAE), while Full MSSD performs better on CH-SIMS (0.399 vs. 0.400). We have adjusted the discussion accordingly.

Thanks for the response. The rebuttal has addressed some of my concerns. However, some issues remain unresolved. For example, regarding Weakness 1, although the authors report evaluating a Coupled Mamba-core adaptation under common feature and training settings, the comparisons still do not adequately demonstrate the proposed method’s competitiveness against recent state-of-the-art approaches. Therefore, I will maintain my current rating.

Reviewer_ukVSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multimodal Shared-State Dynamics (MSSD), a multimodal sequence model in which modality-conditioned write and read interfaces operate on a single shared recurrent state. Building on the rank-RR multi-input, multi-output formulation of Mamba-3, MSSD introduces Dual-Axis dynamics that evolve both the latent-feature and temporal-memory dimensions of the state. The framework also supports different observation times through active updates for newly arriving observations and analytically integrated held updates for cached source inputs. The authors evaluate MSSD on MOSI, MOSEI, CH-SIMS, and CH-SIMS v2, reporting competitive performance in multimodal sentiment prediction. Architecture comparisons, component ablations, representation analyses, and observation-schedule perturbations examine the proposed recurrent organization, cross-modal interactions, and handling of observation gaps.

Critical strengths

The model reported performance across four multimodal sentiment benchmarks. MSSD achieves lower MAE and higher binary classification accuracy than the general sequence backbones evaluated in Table 1 on all four datasets, providing evidence of its practical potential across English and Chinese sentiment analysis tasks.

Critical weaknesses

This paper does not clearly explain how its shared-state design improves upon existing methods, nor why such improvements are important. RAM itself uses shared memory, and the joint Mamba-3 baseline model also maintains a shared cyclic state. MSSD changes how modalities read and write this memory and introduces additional state dynamics, but the paper does not explicitly explain which limitations of existing models these changes aim to address. Table 2 reports performance improvements, but the description of the differences between the compared architectures is insufficient to determine the true cause of these improvements. In particular, comparing two models that both share memory does not demonstrate that shared storage itself explains the performance improvements. The authors need to clearly identify the actual new design choices and provide a comparison that directly verifies their value.

Questions for authors
  1. What exact aggregation operations are used for ΓΔ\Gamma_{\Delta}, Γtime\Gamma_{\mathrm{time}}, and Γlat\Gamma_{\mathrm{lat}} in Eqs. (5) and (10)? Please specify whether each uses summation, averaging, or a learned operation.

  2. How are the per-modality outputs in Eq. (21) combined across modalities and time to produce the final sentiment prediction? Please specify the operations and prediction layers used.

  3. In Sections 3.1 and 3.3, what rule determines whether a cached observation ``remains valid'' for a held update? Does it remain valid until the next observation arrives, or is another validity rule applied?

  4. How is ``synchronous sparsification'' implemented in Figure 5? Please specify which observations are removed, at which times the state is subsequently updated, and when held updates are applied.

2 comments

We appreciate the reviewer's detailed comments. We address the four questions below.

Q1. Aggregation

Temporal controllers average the latest cached source projections before applying the shared nonlinearity. The latent-axis controller averages source-specific transformed contributions from currently active sources. Both use arithmetic averaging, without learned aggregation weights.

Q2. Prediction head

After final normalization, outputs are averaged across the three source positions, followed by masked temporal averaging. The resulting representation passes through LayerNorm–Linear–GELU–Dropout–Linear for prediction.

Q3. Cached observations

Each cached observation remains valid until the next observation from the same source or the end of the sequence. No additional expiration mechanism is used.

Q4. Synchronous sparsification

We retain every kth feature position and the endpoints, then reconstruct the event schedule. Removed positions do not trigger updates. Held contributions arise when another source generates an event while earlier observations remain cached.

These details clarify the existing implementation without changing the model equations.

Thank you for the clarifications and additional experiments. The response addresses my implementation questions and makes the aggregation, prediction head, and update rules clearer. My main concern about the contribution, however, remains only partially resolved. As the authors acknowledge, Joint Mamba-3 also uses shared recurrent storage, so the additional value needs to come from the source-conditioned interfaces and Dual-Axis dynamics. The new controls are helpful, but the mixed Dual-Axis results still leave the benefit of the added dynamics unclear. I appreciate the more precise framing, but the evidence has not sufficiently changed my assessment of the contribution. I therefore maintain my weak-reject recommendation.

Reviewer_xZPbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MSSD proposes multimodal sequence fusion through a single modality-free recurrent matrix state, extending Mamba-3's rank-R MIMO recurrence. Each source has its own write factors (X, B) and read factors (C, z). All writes are summed into one state under a common transition. "Dual-Axis" dynamics add damped-rotation dynamics along the latent-feature axis alongside the temporal axis. "Held" drives integrate cached observations between arrivals via a φ₁ exponential integrator. The paper evaluates on four multimodal sentiment benchmarks (MOSI, MOSEI, CH-SIMS, CH-SIMS v2) against general backbones and MMSA baselines. It also reports organization and component ablations, source-removal and probe analyses, and observation-schedule perturbations.

Critical strengths
  1. Clean formulation. Separating source identity (write/read interfaces) from recurrent storage is stated precisely. The unimodal reduction to Mamba-3 (Eq. 22) is a useful sanity check, and the held-drive φ₁ integral is a principled way to handle asynchronous sources without per-source states.
  2. Controlled comparison design (Table 2). Comparing independent, cross-attention, joint, shared-temporal, and full variants under matched settings is the right experiment for an "organization matters" claim, even though the results do not support that claim strongly (see below).
Critical weaknesses
  1. The reported gains are within seed noise, and the claims overstate them.

    • MSSD's own three-seed standard deviations are about the size of its margins: MOSI MAE 0.721 ± 0.011 and Acc2 85.26 ± 0.69, versus Joint Transformer at 0.732 and Joint Mamba-3 at 0.736. CH-SIMS Acc2 is 80.38 ± 1.46, versus Joint BiLSTM at 80.31. Baseline standard deviations are not reported, and no significance tests are given.
    • The margins are not consistent:
      • On MOSI, MSSD has the lowest Corr among the general backbones (0.784 vs. 0.792).
      • On CH-SIMS v2, Independent + concat beats MSSD on MAE (0.294 vs. 0.298) and Corr (0.712 vs. 0.702).
      • Dual-Axis lowers MOSI Corr (0.798 → 0.784).
      • On clean MOSEI, naive persistence beats analytic held on MAE and Corr.
    • The conclusion that "controlled comparisons associate the main gain with shared recurrent storage" therefore rests on differences of about 0.005–0.015 MAE that the evidence cannot separate from noise.
  2. The conceptual distinction from joint-input SSMs is unclear and not isolated.

    • Joint Mamba-3 also has a modality-free state with one transition. Summing per-source rank-R writes, ΣXₘBₘᵀ, is effectively a block-structured, rank-MR projection of the concatenated input.
    • The real differences appear to be higher effective write rank, per-source read heads, and extra parameters. None of these is controlled for.
    • No parameter counts or compute budgets are given. Tuning budgets for the baselines are also unspecified, while MSSD uses per-dataset rank, d_state, and depth settings.
  3. The mechanism analyses do not test the claims.

    • Source-removal influence (Fig. 3) is tautological. Any shared state shows nonzero off-diagonal influence, and the independent-stream control is zero by construction. The informative comparison would be against Joint Mamba-3, together with evidence that the influence matters for the task.
    • The probes (Fig. 4) are weak. Vision is not linearly decodable on either dataset (negative R²). MOSI text R² is about 0.06. Several "writes-removed" bars fall below the shuffled control. This does not support "source-conditioned accessibility."
    • The held-drive results are inconsistent.
      • MOSI and MOSEI use aligned inputs, so by Sec. 3.4 every source is active at every event and Hₜ = ∅. Held variants should then be identical to active-only on clean data, yet Table 3 and Fig. 6 (MOSI) show nonzero held writes and different scores.
      • Under perturbation, the shaded regions in Fig. 5 overlap heavily. On MOSEI (Fig. 7b), analytic held is worst under temporal shift, which the appendix glosses as "closer."
      • There is no comparison against established irregular-sampling methods (GRU-D, neural CDE/ODE).
  4. The scope is narrow for the claim. Only multimodal sentiment analysis with pre-extracted features is tested, on small and near-saturated benchmarks. Yet the paper frames MSSD as a general reformulation of multimodal fusion. The MMSA baselines are taken from a repository rather than rerun under the same BERT and feature setup. The reproducibility statement mentions LaTeX tables and figures but no code.

Questions for authors
  1. Please report the standard deviation for every baseline in Tables 1–3, along with paired significance tests. Do the MSSD vs. Joint Mamba-3 and Shared-temporal vs. Full differences survive? (This would most change my assessment.)
  2. How do held writes arise on aligned MOSI/MOSEI if all sources are active at every event? If they do not arise, what explains the clean-data differences between the held variants?
  3. Please add a Joint Mamba-3 control with matched write rank (M·R), per-source read heads, and matched parameter count. Please also report parameter counts and FLOPs for all models.
  4. Please run the Fig. 3 influence analysis on Joint Mamba-3 and Shared-temporal, and show whether influence magnitude correlates with accuracy.
  5. Does MSSD help on a task where asynchrony is real (e.g., raw-rate audio/video, sensor fusion) and against GRU-D or CDE baselines?
1 comment

We appreciate the reviewer's detailed comments. We address the five questions below.

Q1. Statistical reliability

All original experiments used three random seeds. We further compared MSSD, temporal-only, and Joint Mamba-3 under matched settings. On MOSI, MSSD achieves 0.723 ± 0.004 MAE versus 0.726 ± 0.003 for Joint Mamba-3; on CH-SIMS, the results are 0.399 ± 0.003 and 0.415 ± 0.006. We report the variability explicitly and avoid claims of statistical significance unsupported by these tests.

Q2. Held updates

The reviewer correctly identified an inconsistency. In clean aligned MOSI/MOSEI, the Held set is empty. Fixed-checkpoint tests confirm identical predictions with Held enabled or disabled. We have corrected the original interpretation, which incorrectly attributed differences between independently trained variants to Held updates.

Q3. Joint Mamba-3 and capacity

Both models use shared recurrent storage. MSSD differs in its source-conditioned write/read interfaces and Dual-Axis dynamics. Additional capacity controls include a parameter-adjusted Joint variant with 1.139M non-BERT parameters, compared with MSSD's 1.148M. These comparisons address capacity differences, although not every architectural factor is jointly matched.

Q4. Cross-modal influence

Fixed-network write-removal tests show that removing text writes increases MSSD's MOSI MAE by 0.078 on average, confirming their contribution to prediction. Since Joint Mamba-3 also exhibits cross-modal dependence, we interpret Figures 3–4 as evidence of learned interactions rather than a mechanism exclusive to MSSD.

Q5. Asynchrony and scalability

Our evaluation uses relative feature clocks, not physical sensor timestamps. Real-world asynchronous data and GRU-D/CDE comparisons remain outside the current evaluation. Another potential limitation is scaling to many simultaneous modalities (e.g., ten), where competition for finite shared-state capacity may cause interference. Our conclusions therefore focus on the evaluated three-modality settings.

Comments on the paper

1 comment

We have uploaded a revised manuscript with the following updates:

  1. Method clarification: We clarified the source-conditioned write/read operations, aggregation, prediction head, and asynchronous update rules.
  2. Additional experiments: Appendix E now includes three-seed comparisons, Rank/Read and capacity controls, parameter counts, inference latency, and a Coupled Mamba comparison.
  3. Mechanism analysis: We added fixed-factor write intervention results and clarified the Held-update behavior on aligned data.
  4. Discussion: We refined the claims regarding shared-state dynamics and Dual-Axis effectiveness to better reflect the experimental evidence.

The original benchmark results, main tables, and model equations remain unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.