acceptodds
Under review as a conference paper at ICLR 2027

Channel-Aware Turbo Multimodal Fusion: Learning under Noisy and Missing Modalities

Abstract

Multimodal systems in the wild must operate when modalities are corrupted by noise or entirely missing. Most existing methods address the two failure modes separately and fuse modalities with attention that is blind to signal quality. We draw an analogy between multimodal fusion and multi-channel communication, and propose Channel-Aware Turbo Multimodal Fusion (CAT-MF), which treats noise and missingness uniformly as channel impairments. CAT-MF has three components: (1) channel state estimation, which scores every modality by an existence probability, a spectral proxy for the inverse signal-to-noise ratio, and a learned damage-type embedding, all obtained without clean reference signals; (2) iterative joint decoding, a Turbo-inspired procedure that exchanges soft cross-modal information over a small number of rounds and accumulates it in a persistent state; and (3) channel-modulated attention, which gates cross-modal transfer by the estimated states, zeroing missing channels, down-weighting noisy ones, and suppressing transfer between channels sharing correlated damage. We prove that the decoding iterations converge absolutely and linearly to a limit once the steps are summable, that the accumulated extrinsic evidence yields non-negative information gain per round bounded below by the donor modalities' quality, and that the reconstruction error decays with the availability of high-quality correlated modalities. CAT-MF exceeds competitive baselines on multiple datasets and is markedly more robust under noisy and missing modalities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Fusion, Robust Learning, Channel Model, Missing Modality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9akmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats noisy and missing modalities as two forms of "channel impairment" and proposes Channel-Aware Turbo Multimodal Fusion (CAT-MF). CAT-MF first estimates a reference-free channel state for each modality consisting of an existence probability α\alpha, an inverse-SNR proxy β\beta, and a learned damage-type embedding γ\gamma. These estimates are then used to modulate cross-modal attention so that missing or noisy modalities contribute less and modalities with similar estimated damage types exchange less information. Fusion is performed through a weight-tied iterative decoder that accumulates cross-modal messages over several rounds using a geometrically decaying update schedule, motivated by Turbo decoding. The model also includes selective reconstruction of low-quality modalities and reliability-weighted final fusion. The paper provides theoretical analyses of decoder convergence, information accumulation, and reconstruction error, and evaluates CAT-MF across seven multimodal benchmarks under complete, missing-modality, and noisy settings, where it generally achieves strong or state-of-the-art performance.

Critical strengths
  • The parameter-matched comparison in Table 9 is the most consequential positive result. A single weight-shared block iterated three times outperforms an independent three-layer stack with 3×3\times the parameters, and channel-aware gating adds a further gain at the same parameter budget. This is a clean and controlled experiment with a useful takeaway for fusion architecture design, independent of the paper's broader communication-theoretic framing.

  • The core framing of separating reliability from relevance in cross-modal attention is well motivated. The evaluation is broad, covering seven benchmarks spanning sentiment, humor, sarcasm, movie-genre classification, and digit classification, including both missing- and noisy-modality settings. The attempt to model not only whether a modality is unreliable but also the type of degradation is an interesting direction for robust multimodal fusion.

Critical weaknesses
  • The inverse-SNR proxy β\beta is not fully convincing for vector modalities. Several vector pipelines apply LayerNorm before the CSE, which may constrain β=Var(z)/E[z2]\beta = \mathrm{Var}(z)/\mathbb{E}[z^2] more by normalization than by corruption. Also, a constant nonzero vector gives β=0\beta=0, which seems inconsistent with the paper's description of flat, unstructured vectors as noisy.

  • The noisy-modality protocol is unclear. The CSE is described as operating on pre-pooling features zm(0)z_m^{(0)}, while Eq. 58 injects noise into pooled features hm(0)h_m^{(0)}. If taken literally, the estimator would not observe the corruption used in Table 4.

  • The interpretation of Table 4 is not fully convincing. Since all modalities are corrupted with the same family and rate, common reliability or damage factors may largely cancel after row normalization. This makes it difficult to attribute the gains specifically to the proposed anti-corroboration mechanism.

  • Some results are difficult to reconcile with the iterative-decoding explanation. Figure 5 shows updates becoming nearly negligible after the first round, yet Table 8 reports sizable gains from T=3T=3 over T=1T=1. This is especially unclear in single-modality settings, where no cross-modal messages should remain to accumulate.

  • The theoretical guarantees appear weaker than the presentation suggests. Theorem 1 mainly follows from bounded updates and summable step sizes, while Theorem 2 is proved for an evidence-history representation that differs from the implemented summed buffer. The appendix itself notes that the donor-level guarantee does not directly carry over to the actual implementation.

  • The empirical reporting could be stronger. Results are averaged over five seeds, but no variance is reported. There are also some inconsistencies between the text and tables; for example, Section 5.3 states that CAT-MF is best at every Gaussian-noise level, while Table 4 does not support this for every MOSI Acc2 setting.

Questions for authors
  • In single-modality configurations, what are the realized gate weights given soft α\alpha, row normalization, and ε\varepsilon? What explains the T=1T=1 versus T=3T=3 gap there?

  • Which step schedule was actually used? Appendix C.3 describes a "sigmoid warm-up of Eq. 11," which conflicts with the geometric schedule that Theorem 1 relies on.

  • If the model is trained with T=3T=3 but evaluated with T=1T=1, does performance hold? If so, the benefit of TT would seem to be a training effect rather than iterative refinement. How do the authors reconcile this with Figure 5?

  • What are the donor weights when one modality is corrupted and two are clean? Clean modalities share the "clean" prototype, so γm⊤γn≈1\gamma_m^\top\gamma_n \approx 1 and g≈σ(−1)g \approx \sigma(-1). Could the γ\gamma factor therefore favor the corrupted donor over the clean one?

  • Can the authors isolate the effect of γ\gamma in Figure 4 with a β\beta-matched control? How does CAT-MF compare against DEQ fusion (Ni et al., 2023) and COCOON (Cho et al., 2025), the closest prior work on iterative and uncertainty-aware fusion?

11 comments
AuthorsOpenReview ↗

Response to Weakness 1

W1: βm\beta_m for vector modalities; the constant-vector edge case.

Response: (1) We agree on both the LayerNorm issue and that a constant non-zero vector gives βm=0\beta_m = 0 in Eq.5 under the old dispersion statistic. These are properties of the vector branch only, which is confined to AVMNIST, MM-IMDb and the two vector modalities of CH-SIMS-v2. Therefore, we have replaced the vector βm\beta_m in Eq.5 with a self-contained likelihood-ratio test (LRT), which is well defined for constant and LayerNorm inputs. Specifically, each affected modality keeps blind EMA running statistics of its clean pooled features, a mean μm\mu_m and a diagonal variance σm2\sigma_m^2, and scores the per-sample log likelihood ratio between the clean channel N(μm,diagσm2)N(\mu_m, diag \sigma_m^2) and the sensor noise N(0,I)N(0, I). Writing the per-sample statistic

(z_j – \mu _ {m,j})^2 / \sigma _ {m,j}^2 - z_j^2 ],$$ We calibrate it against the analytic expectations $\ell_m^0$ and $\ell_m^1$ under clean and pure-noise inputs (no fitted scaling) respectively, and report $$\beta_m = clip( (\ell_m - \ell_m^0) / (\ell_m^1 - \ell_m^0), 0, 1 ).$$ The implementation uses Bessel-corrected variance, a variance floor and a clean-sample count, and withholds the score until enough clean examples are seen. This is well defined for zero-mean/LayerNorm and constant inputs, and it is monotone under the deployed blend impairment (a plain Mahalanobis distance is not, because the blend first contr\$acts the variance). (2) We also wish to clarify that the zero-mean/LayerNorm failure is specific to the vector modalities, which appears only in AVMNIST, MM-IMDb and the two vector modalities of CH-SIMS-v2. AVMNIST and MM-IMDb are presented in Appendix B.1 explicitly as transfer experiments "to test generalization." Hence the main results (MOSI/MOSEI, ablation, noise experiments, etc.) use sequence modalities and are not implicated, and the main claims of the proposed method remain valid. **We have rerun the affected vector datasets with the new $\beta_m$**. **The performance of new vector $\beta_m$ is slightly better than the original vector $\beta_m$** (gains on AVMNIST and MM-IMDb, mixed but better MAE/Corr on CH-SIMS-v2; Table Q1-1), and the conclusions on them are unchanged. Table Q1-1: Results with the original vs. the repaired vector $\beta_m$. | Dataset | metric | original $\beta_m$ | repaired $\beta_m$ | | --- | --- | --- | --- | | AVMNIST | I | 91.33 | **92.00** | | AVMNIST | A | 98.00 | **98.67** | | AVMNIST | I+A | **99.33** | **99.33** | | AVMNIST | Avg | 96.22 | **96.67** | | MM-IMDb | T | 59.63 | **60.23** | | MM-IMDb | I | 46.02 | **46.04** | | MM-IMDb | T+I | 62.06 | **62.69** | | MM-IMDb | Avg | 55.90 | **56.32** | | CH-SIMS-v2 | Acc5 | 56.7 | **56.8** | | CH-SIMS-v2 | Acc3 | **75.8** | 75.0 | | CH-SIMS-v2 | Acc2 | **83.2** | 83.1 | | CH-SIMS-v2 | F1 | **83.2** | 83.1 | | CH-SIMS-v2 | MAE | 0.280 | **0.278** | | CH-SIMS-v2 | Corr | 0.742 | **0.752** | (3) Additionally, **we have conducted experiments to report the relationship between $\beta_m$ and actual corruption severity**, on both sequence and vector modalities, all on one trained CH-SIMS-v2 model. With the two peers fixed, one modality is corrupted at increasing noise ratio NR and $\beta_m$, the gate and the final fusion weight $\pi_m$ are recorded. Table Q1-2: $\beta_m$ values for sequence and vector modalities (mean over samples) as modality becomes increasingly corrupted. | Modality | NR=0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | Spearman | | --- | --- | --- | --- | --- | --- | --- | --- | | text (sequence) | 0.002 | 0.010 | 0.051 | 0.144 | 0.239 | 0.270 | 1.00 | | audio (vector) | 0.000 | 0.041 | 0.164 | 0.348 | 0.664 | 0.904 | 1.00 | | video (vector) | 0.001 | 0.038 | 0.155 | 0.363 | 0.629 | 0.885 | 1.00 | Table Q1-3: Realized weight $\pi_m$ in the final state-weighted fusion. | Modality | NR=0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | Spearman | | --- | --- | --- | --- | --- | --- | --- | --- | | text | 0.333 | 0.331 | 0.322 | 0.301 | 0.280 | 0.272 | -1.00 | | audio | 0.334 | 0.325 | 0.298 | 0.260 | 0.204 | 0.167 | -1.00 | | video | 0.333 | 0.325 | 0.300 | 0.257 | 0.210 | 0.170 | -1.00 | As shown in Table Q1-2, **$\beta_m$ is near zero for every clean input and increases strictly with severity (Spearman = 1.0 for all three)**. The text $\beta_m$ (spectral cue) saturates at the predicted white-noise level 0.25 (finite-frame 0.270), while the vector $\beta_m$ (LRT) spans almost the full range (0.904, 0.885). The fusion weight $\pi_m$ falls strictly as the modality is corrupted (Spearman = -1.0), so the damaged modality's actual contribution to the prediction is reduced, not merely its internal statistic (see Table Q1-3).
AuthorsOpenReview ↗

Response to Weakness 2, Weakness 3, Weakness 4, and Question 1

W2: Noisy-modality protocol (where is corruption applied?).

Response: We are sorry for the typo. In the implementation, corruption is applied to the unimodal sequences before mean pooling, and the CSE consumes those same corrupted sequences, so the estimator does observe the corruption used in Table 4. Eq. (58) is a pooled-level shorthand that obscures this. We have corrected the protocol to state the true operation.

W3: Table 4 attribution when all modalities share one damage family.

Response: We thank the reviewer and agree that the fully symmetric setting of Table 4 cannot by itself isolate anti-corroboration: when every modality is corrupted by the same family and rate, both the reliability factor and the pairwise damage-type factor are common across donors, and they cancel after row normalization. We re-scope the claim accordingly. Table 4 is an end-task robustness result under uniform corruption, whose gains come from the reliability term αme−βm\alpha_m e^{-\beta_m} down-weighting noisy channels together with the iterative exchange and selective reconstruction, rather than from anti-corroboration. We have stated this explicitly in the main text.

W4 / Q1: In single-modality configurations, what are the realized gate weights given soft αm\alpha_m, row normalization, and ϵ\epsilon? What explains the T=3T=3 versus T=1T=1 gap there?

Response: We thank the reviewer for the valuable comments. The number of decoder tokens MM is fixed at construction, so a "single-modality" pattern means one real token coexisting with learnable miss tokens for the absent modalities, not a literal one-token model. We have measured the realized states on the trained MOSI model for each choice of the real modality. For the zeroed miss channels, the CSE outputs a near-zero αm\alpha_m (a small positive value below 5e-5). The realized gate matrix is shown in Table W4-1.

Table W4-1: Realized gate matrix with text as the real modality (w[b,m,n]w[b, m, n], donor n into target m).

target \ donortextvisualacoustic
text (real)-0.50920.4903
visual (miss)1-0
acoustic (miss)10-

The miss target rows point to the real donor with weight 1. The real token's row is a normalized relative distribution over the two miss donors, not an absolute transfer: their absolute reliability is about 1e-5, and the gate and block normalize the row, so the ~0.5 only describes how this tiny residual is split, not its magnitude. ϵ\epsilon only prevents division by zero.

Why T=3T = 3 can still outperform T=1T = 1 in Table 8. Every model variant in Table 8 is trained and evaluated at its own TT, so this is not a truncation effect. The single-modality Acc2 numbers are:

Single-modality split (Acc2)T = 3 (Full)T = 1gap
{l} language only89.385.8+3.5
{a} acoustic only63.761.6+2.1
{v} visual only65.566.5-1.0

What changes with TT is the depth of the weight-tied computation: each round still applies the per-token FFN and the Wutanh(WmL)W_u tanh(W_m L) refinement to the real token, so the unrolled T=3T = 3 model is a deeper recurrent network. Training a deeper unrolled graph shapes the shared weights so the same representation is progressively self-refined and better mapped to the decision, the same reason a deeper nonlinear model can beat a shallower one on a single input. The effect is modality-dependent and not a guaranteed gain: visual modality reverses by 1.0, which is the signature of a depth/optimization effect. T=3T = 3 advantage is consistent with the deeper recurrent trajectory shaping the learned weights during training.

Additionally, before decoding, the αm<τmiss\alpha_m < \tau_{miss} rule replaces each absent input with one learnable token hmissh_{miss} instead of zero-padding. This hmissh_{miss} is a learned marginal prior: it carries the generic, average representation of an absent modality as seen in training, and it is refined from peers in later rounds, providing informative modality priors.

AuthorsOpenReview ↗

Response to Weakness 5, Weakness 6, Question 2, and Question 3

W5 / Q2: Which step schedule is actually used?

Response: We are sorry for the typo. The deployed schedule is the geometric decay: η(t)=η0rt,0<r<1\eta(t) = \eta_0 r^t, 0 < r < 1, matching Eq.11 (Eq.13 of the revised paper) and Theorem 1. We have corrected Appendix C.3 so it no longer references a sigmoid warm-up. With the common values η0=0.5,r=0.5\eta_0 = 0.5, r = 0.5, the realized steps are η(0,1,2)=0.500,0.250,0.125\eta(0,1,2) = 0.500, 0.250, 0.125, and the total schedule mass is Sη=η0/(1−r)=1.0S_{\eta} = \eta_0/(1-r) = 1.0.

Because the schedule is summable, Part I of the rewritten Theorem 1 gives absolute convergence and the linear rate directly, with no finite-horizon workaround. Part II then shows the steady-state map TT is a contraction with a unique fixed point when q=CSηℓB<1q = C S_\eta \ell_B < 1, and bounds the small gap between the implemented limit HinfH_{inf} and that fixed point; the deterministic zero initialization makes the path unique. The measured one-step Jacobian spectral radius ρ 0.007<<1\rho ~ 0.007 << 1 is consistent with q<1q < 1 and a negligible gap. After these edits, Eq.11 (Eq.13 of the revised paper), Appendix C.3 and Theorem 1 describe one and the same schedule.

W6: Variance is not reported and text/table inconsistencies.

Response: Thank you for the valuable comment. We have added multi-seed standard deviations across the five runs (mean ±\pm std below) and correct the Sec. 5.3 claim: CAT-MF achieves the best MAE at every noise level, while Acc2 is best at most levels.

Table W6-1: Five-seed mean ±\pm std on CMU-MOSI and CMU-MOSEI.

DatasetAcc7Acc2F1MAECorr
MOSI51.5 ±\pm 0.889.6 ±\pm 0.689.6 ±\pm 0.60.571 ±\pm 0.0160.878 ±\pm 0.009
MOSEI55.0 ±\pm 0.688.6 ±\pm 0.488.5 ±\pm 0.40.501 ±\pm 0.0090.817 ±\pm 0.007

Table W6-2: Five-seed mean ±\pm std on CH-SIMS-v2.

Acc5Acc3Acc2F1MAECorr
56.8 ±\pm 0.575.0 ±\pm 0.483.1 ±\pm 0.783.1 ±\pm 0.70.278 ±\pm 0.0060.752 ±\pm 0.008

As shown in Table W6-1 and Table W6-2, the standard deviations are small (at most 0.8 on the accuracy metrics and 0.016 on MAE), so the point estimates in the main tables are stable and the rankings do not change across seeds.

Q3: T=3T = 3-trained model evaluated at T=1T = 1; is the gain only a training effect? How do the authors reconcile this with Figure 5?

Response: We thank the reviewer for the valuable comments.

(1) We have run exactly this control on the trained MOSI model with no retraining: all weights are fixed and only the number of decoding rounds is reduced from T=3T = 3 to T=1T = 1. Performance does not hold, so the gain is not merely a training effect (see Table Q3-1).

Table Q3-1: T=3T = 3-trained model evaluated at T=1T = 1.

MOSI (Acc2)lavavallvlva
T = 3 (full model)89.667.889.089.689.365.563.7
T = 1 (same weights, truncated)88.762.387.288.388.760.760.1
gap-0.9-5.5-1.8-1.3-0.6-4.8-3.6

Because the identical trained weights are used, the drop is caused directly by removing the iterative rounds, not by optimization. The pattern is informative. With three reliable modalities (lav), one round already captures most of the gain and truncation costs only 0.9, since a single average over safe channels is already good. The cost grows as redundancy disappears: -5.5 on {a,v}, -4.8 on single visual and -3.6 on single acoustic, where the weak channel must be repeatedly corrected by its peer and re-routed across rounds. This is precisely the work that one-shot fusion cannot do.

(2) Reconciliation with Figure 5. Figure 5 plots the per-round update norm on the full-modality trajectory, where it becomes small after the first round under the geometric η\eta, and on that same lav split the truncation gap is only -0.9, so the two are consistent. A small token-motion norm does not mean the rounds are unused: the norm measures how far each token moves, not the routing decision or the accumulated evidence. On the weak and missing splits, the gate keeps reallocating which donor each token listens to, and these small-norm changes move decisions near the boundary, which is why truncation changes accuracy by up to 5.5 even though the plotted norm is small. The truncation test is therefore the decisive evidence: the T=3T = 3 benefit is genuine iterative refinement, and it is largest where the model has the least redundancy.

Reviewer_3qfgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Channel-Aware Turbo Multimodal Fusion (CAT-MF) to improve multimodal learning when input modalities are corrupted by noise or entirely missing. The main idea is to form multimodal fusion as a multi-channel communication problem. Each modality represents a communication channel with potentially different reliability. CAT-MF comprises three components: CSE aims to distinguish missing, noisy, and similarly corrupted modalities without requiring clean reference signals at inference. Channel modulated cross attention aims to exchange information between modalities and suppress missing or noisy modalities and reduce information transfer between modalities exhibiting similar corruption patterns. Turbo inspired iterative joint decoding iteratively refines modality representations through accumulated cross-model messages. Theoretical analyses concerning convergence, information accumulation, etc., are provided. Experimental results cover seven datasets report improvements over existing methods under complete, missing, and noisy modality settings.

Critical strengths

This paper addresses an important and meaningful challenge in multimodal learning. It proposes to treat modality missing and modality corruption within a unified framework is a useful direction, especially for real applications using noisy sensors, audio-visual learning and other available modalities. The combination of channel estimation, reliability-aware attention, iterative feature refinement, and selective reconstruction forms a coherent architectural pipeline. The experiments cover complete-modality performance, different missing-modality configurations, varying noise levels, and additional corruption types. The results are generally favorable compared with the selected baselines. The attempt to connect architectural choices with both theoretical analysis and empirical diagnostics is valuable, even though several aspects of these analyses require further justification.

Critical weaknesses

There are several concerns regarding the central claims and method.

  • The proposed inverse-SNR estimator βm\beta_m lacks a convincing mathematical justification. βm\beta_m is based on the proportion of high-frequency spectral energy for sequence modalities, or normalized coordinate variance for vector modalities. However, whether these statistics reliably represent the actual noise level is unsure. From equation 5, it seems that , for zero-mean feature vectors, $\beta_m \approx1$ regardless of whether the feature is clean or corrupted. Besides, the architecture explicitly uses layer normalization for several vector-based modality encoders. The resulting statistic may be nearly constant and therefore unable to distinguish clean from noisy observations. An empirical study is necessary to report the relationship between βm\beta_m and actual corruption severity or ground-truth SNR, independently for each modality and dataset.

  • The channel-modulated attention multiplies the standard attention weights by these routing weights and renormalizes. However, the proposed noise and damage-type modulation has essentially no effect on normalized cross-attention when there is only one available donor.

  • The convergence analysis relies primarily on two properties: the decoder produces bounded messages, and the geometrically decaying accumulation schedule has a finite sum. Under these conditions, convergence of the accumulated state follows naturally. This does not establish that the decoder is a contraction mapping or that it converges to a unique fixed point independent of initialization.

Questions for authors
  1. In Section 3.2, the authors claim that the inverse-SNR proxy βm\beta_m approaches 1 for noise-dominated observations. However, Equation (4) defines βm\beta_m as the fraction of spectral energy in the highest 25% of frequencies. For white Gaussian noise with approximately uniform spectral power, this ratio should approach 0.25 rather than 1. Could the authors clarify this discrepancy and provide empirical validation of the relationship between βm\beta_m and the actual noise level?

  2. How can βm\beta_m reliably estimate modality noise, given its behavior under white noise and layer-normalized vector features?

  3. Does Theorem 2 apply to the implemented compressed evidence buffer, rather than only an idealized history of donor-specific messages?

  4. How does channel-modulated attention suppress a noisy donor when only one donor is available?

7 comments
AuthorsOpenReview ↗

Response to Weakness 1, Question 1 and Question 2

W1 / Q1–Q2: For white Gaussian noise, sequence βm\beta_m in Eq.4 should approach 0.25 rather than 1; for zero-mean feature vectors/ feature vectors after layer normalization, vector βm\beta_m in Eq.5 cannot distinguish clean from noisy observations, and empirical study is necessary to report the relationship between βm\beta_m and actual corruption severity or ground-truth SNR.

Response: (1) We appreciate the reviewer's valuable comment. Both criticisms are correct. For βm\beta_m of sequence modality (Eq.4), the high-band ratio under white noise tends to 0.25, not 1, and we have corrected the relevant statement in the revised paper. For zero-mean feature vectors/ vectors after layer normalization, the old dispersion ratio (Eq.5) is unable to distinguish clean from noisy observations. Therefore, we have replaced the vector βm\beta_m in Eq.5 with a self-contained likelihood-ratio test (LRT): Each affected modality keeps blind EMA running statistics of its clean pooled features, a mean μm\mu_m and a diagonal variance σm2\sigma_m^2, and scores the per-sample log likelihood ratio between the clean channel N(μm,diagσm2)N(\mu_m, diag \sigma_m^2) and the sensor noise N(0,I)N(0, I). Writing the per-sample statistic

(z_j – \mu _ {m,j})^2 / \sigma _ {m,j}^2 - z_j^2 ],$$ we calibrate it against the analytic expectations $\ell_m^0$ and $\ell_m^1$ under clean and pure-noise inputs (no fitted scaling), and report $$\beta_m = clip( (\ell_m - \ell_m^0) / (\ell_m^1 - \ell_m^0), 0, 1 ).$$ The implementation uses Bessel-corrected variance, a variance floor and a clean-sample count, and withholds the score until enough clean examples are seen. This is well defined for zero-mean/LayerNorm and constant inputs, and it is monotone under the deployed blend impairment (a plain Mahalanobis distance is not, because the blend first contracts the variance). We also wish to clarify that the zero-mean/LayerNorm failure is specific to vector modalities, which appears only in AVMNIST, MM-IMDb and the two vector modalities of CH-SIMS-v2. AVMNIST and MM-IMDb are presented in Appendix B.1 as transfer experiments to test generalization. Hence the main results (MOSI/MOSEI/UR-Funny/MUStRAD, ablation, noise experiments, etc.) use sequence modalities and are not implicated, and the main claims of CAT-MF remain valid. We have rerun the affected vector datasets with new $\beta_m$. The performance of new vector $\beta_m$ is slightly better than the original vector $\beta_m$ (gains on AVMNIST and MM-IMDb, mixed but better MAE/Corr on CH-SIMS-v2; see Table Q1-1), and the conclusions on them are unchanged. Table Q1-1: Results with the original vs. the repaired vector $\beta_m$. |Dataset|metric|original $\beta_m$|repaired $\beta_m$| |-|-|-|-| | AVMNIST | I | 91.33 | **92.00** | | AVMNIST | A | 98.00 | **98.67** | | AVMNIST | I+A | **99.33** | **99.33** | | AVMNIST | Avg | 96.22 | **96.67** | | MM-IMDb | T | 59.63 | **60.23** | | MM-IMDb | I | 46.02 | **46.04** | | MM-IMDb | T+I | 62.06 | **62.69** | | MM-IMDb | Avg | 55.90 | **56.32** | | CH-SIMS-v2 | Acc5 | 56.7 | **56.8** | | CH-SIMS-v2 | Acc3 | **75.8** | 75.0 | | CH-SIMS-v2 | Acc2 | **83.2** | 83.1 | | CH-SIMS-v2 | F1 | **83.2** | 83.1 | | CH-SIMS-v2 | MAE | 0.280 | **0.278** | | CH-SIMS-v2 | Corr | 0.742 | **0.752** | (2) As suggested by the reviewer, we have conducted experiments to report the relationship between $\beta_m$ and actual corruption severity, on both sequence and vector modalities, all on one CH-SIMS-v2 model. With the two peers fixed, one modality is corrupted at increasing noise ratio NR, and $\beta_m$ and the final fusion weight $\pi_m$ are recorded. Table Q1-2: Inverse-SNR $\beta_m$ for sequence and vector modalities as modality becomes increasingly corrupted. |Modality|NR=0|0.2|0.4|0.6|0.8|1.0|Spearman| |-|-|-|-|-|-|-|-| | text (sequence) | 0.002 | 0.010 | 0.051 | 0.144 | 0.239 | 0.270 | 1.00 | | audio (vector) | 0.000 | 0.041 | 0.164 | 0.348 | 0.664 | 0.904 | 1.00 | | video (vector) | 0.001 | 0.038 | 0.155 | 0.363 | 0.629 | 0.885 | 1.00 | Table Q1-3: Realized weight $\pi_m$ in the final state-weighted fusion. |Modality|NR=0|0.2|0.4|0.6|0.8|1.0|Spearman| |-|-|-|-|-|-|-|-| | text | 0.333 | 0.331 | 0.322 | 0.301 | 0.280 | 0.272 | -1.00 | | audio | 0.334 | 0.325 | 0.298 | 0.260 | 0.204 | 0.167 | -1.00 | | video | 0.333 | 0.325 | 0.300 | 0.257 | 0.210 | 0.170 | -1.00 | As shown in Table Q1-2, **$\beta_m$ is near zero for every clean input and increases strictly with severity (Spearman = 1.0 for all three)**. The text $\beta_m$ (spectral cue) saturates at the predicted white-noise level 0.25 (finite-frame 0.270), while vector $\beta_m$ (LRT) spans almost the full range (0.904, 0.885). The fusion weight $\pi_m$ falls strictly as the modality is corrupted (Spearman = -1.0), so the damaged modality's actual contribution to the prediction is reduced, not merely its internal statistic (see Table Q1-3).
AuthorsOpenReview ↗

Response to Weakness 2, Weakness 3, and Question 4

W2 / Q4: Channel modulation with a single donor.

Response: We appreciate the valuable comment of the reviewer. Actually, in our implementation, when a target has a single donor (M=2M = 2), we do not normalize the routing: the gate keeps its raw scalar wn→m=αne−βnσ(−γmTγn/τγ),w _ {n\to m} = \alpha_n e^{-\beta_n} \sigma(-\gamma_m^T \gamma_n / \tau_{\gamma}), and the block applies it directly to the donor message, msg=wn→mVmsg = w_{n\to m} V, without the post-gating renormalization. A single-donor attention has no relative distribution to normalize, so the reliability acts as a message gate (gated injection), the standard gated-residual semantics. A noisy donor with low αm\alpha_m or high βm\beta_m then produces a small weight and an attenuated message, and the state-weighted fusion and selective reconstruction continue to provide additional suppression. We have clarified it in Section 3.3.

W3: Convergence does not establish that the decoder is a contraction mapping or that it converges to a unique fixed point

Response: We agree the earlier argument only had bounded messages plus a summable schedule, which gives series convergence, not contraction. We have reorganized Theorem 1 so every claim is stated honestly, and we have removed an auxiliary lemma that was not valid. Part I is the summable-series result, explicitly labelled as convergence only: the buffers converge absolutely and the tokens reach a path-dependent limit HinfH_{inf} at a linear rate, ∣∣Ht–Hinf∣∣<=Kr(t+1)||H^t – H_{inf}|| <= K r^{(t+1)}; it is not called a contraction. Part II is a separate small-gain statement. On the bounded token region let ℓB\ell_B be the block Lipschitz and C=ℓLN∣∣Wu∣∣2∣∣Wm∣∣2C = \ell_{LN} ||W_u|| _ 2 ||W_m|| _ 2. The steady-state self-map T(H) = LN(H0 + W_u tanh(W_m (S_{\eta} B(H)))$, with $ S_{\eta} = \sum_t \eta^t, obeys ∣∣T(H)−T(H′)∣∣<=CSηℓB∣∣H−H′∣∣=q∣∣H−H′∣∣.||T(H) - T(H')|| <= C S_{\eta} \ell_B ||H - H'|| = q ||H - H'||. When q<1q < 1, Banach gives a unique fixed point H∗H* of the map TT. We do not claim the implemented accumulating trajectory is TT-iteration or that it reaches H∗H* exactly: the buffer sums the time-varying messages of the early rounds, so the actual limit HinfH_{inf} is an approximate fixed point. Let D0=∣∣Hinf−H(0)∣∣D_0 = ||H_{inf} - H^{(0)}||, a finite initial distance because both tokens lie in the invariant set. We prove the explicit residual and gap bounds,

|| H_{inf} - H*|| <= \frac{C \ell_B}{1-q} \big[\eta^{(1)} D_0 + K \eta_0 r^4 / (1 - r^2)\big],$$ which are small constants. The $\tau=1$ term is kept explicit because the initial token $H^{(0)}$ is not an output of $g$ and is not covered by the rate bound; only $\tau\ge2$ terms use the geometric rate. The buffers are initialized deterministically at zero, so for a fixed input the path is unique. The empirical margin is large: the measured one-step Jacobian spectral radius (power iteration on $J^T J$, double backprop) is $\rho = 0.0072$ on clean and 0.0071 on corrupted inputs, i.e. $\rho << 1$, indicating $q$ is small and the gap is numerically negligible. The finite-horizon contraction condition $q_T<1$ uses only the finite mass $S_{\eta}^{(T)}$; the sharp residual constants are written through $K$ and $D_0$ under the summability assumption, and a purely finite bound $C\ell_B\,\mathrm{diam}(\mathcal K)S_\eta^{(T)}$ needs no infinite-summability.
AuthorsOpenReview ↗

Response to Question 3

Q3: Does Theorem 2 apply to the compressed summed buffer?

Response: We thank the reviewer for the important question. We have rewritten Theorem 2 so that the claims which hold on the actual compressed summed buffer are proved directly on it, with no slot assumption. Specifically, Part I applies verbatim to the implementation: the per-round non-decrease ΔIm(t)=I(pm(t);s∣Em(t−1))≥0\Delta I_m^{(t)} = I(p_m^{(t)}; s \mid E_m^{(t-1)}) \ge 0 uses only the chain rule and holds for every injected message, and data processing gives the model-free bound I(Em(t);s)≤I(H(0);s)I(E_m^{(t)}; s) \le I(H^{(0)}; s). We do not write a bound by the entropy H(s)H(s), since for a continuous source differential entropy can be negative while mutual information is nonnegative. Part II is proved directly on the summed buffer Lm(t)=∑τ≤tη(τ)pm(τ)L_m^{(t)} = \sum_{\tau\le t}\eta^{(\tau)}p_m^{(\tau)} under the linear Gaussian observation model: the buffer is a single merged Gaussian observation, and under the explicit admission condition a2/b≥Jm(t−1)a^2/b \ge J_m^{(t-1)} and aA~m(t−1)≥0a\widetilde A_m^{(t-1)} \ge 0 its posterior precision and hence I(Lm(t);s)I(L_m^{(t)};s) are non-decreasing. The gate does not force this condition, because it leaves the single-observation ratio a2/b=Am(t)2/Rm(t)a^2/b = A_m^{(t)2}/R_m^{(t)} unchanged. Its role is to skip hard-missing channels, which leaves the buffer unchanged at equality, and to attenuate low-quality donors by αme−βm\alpha_m e^{-\beta_m} together with the EMA reconstruction, which reduces the magnitude of an admission-failing write rather than making it admissible. The donor-level lower bound genuinely does not transfer to the aggregate sum, since the map from donor histories to S=∑nsnS = \sum_n s_n is many-to-one and donor identities are information-theoretically unrecoverable. We therefore state it as the slot-resident Part III under (A2), where the donor messages occupy separately addressable slots, and we do not claim it on the compressed buffer. The empirical evidence below confirms the buffer on the implementation is the operative quantity.

Empirically, we have added a controlled comparison under identical rounds and training settings that replaces the accumulated buffer Lm(t)=Lm(t−1)+η⋅pm(t)L_m^{(t)} = L_m^{(t-1)} + \eta \cdot p_m^{(t)} with a current-round state Lm(t)=η⋅pm(t)L_m^{(t)} = \eta \cdot p_m^{(t)} (no carry-over) to demonstrate the effectiveness of the summed buffer.

Table Q3-1: Current-round-only comparison on MOSI (Acc2).

Variantlavavallvlva
CAT-MF89.667.889.089.689.365.563.7
current-round-only89.266.588.888.588.962.862.4

As shown in Table Q3-1, removing carry-over is nearly free when all three modalities are present (-0.4) but increasingly costly as redundancy disappears: -1.3 on {a,v} and {a}, -1.1 on {l,v}, and -2.7 on the single-visual split. The weak single channels rely most on the accumulated evidence, so the buffer has a distinct, measurable value that is not reproduced by a one-shot state. The current-round-only ablation gives the empirical confirmation for the implemented buffer.

Reviewer_XGx8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Channel-Aware Turbo Multimodal Fusion (CAT-MF) for learning with noisy and missing modalities. The method estimates modality presence, a noise-level proxy, and damage-type embeddings, and uses these states to modulate cross-modal attention. Inspired by iterative turbo decoding, a shared fusion block repeatedly exchanges information between modalities and accumulates messages to refine their representations. A selective reconstruction module updates modalities identified as low quality, followed by state-weighted fusion for prediction. The paper also presents theoretical analyses of convergence, information accumulation, and reconstruction. Experiments cover seven benchmarks across several multimodal tasks, including evaluations with missing modalities and feature corruption. The reported results, component ablations, and comparisons with shared-weight and independently parameterized attention stacks provide empirical evidence for the proposed architecture.

Critical strengths
  1. The architecture follows a clear practical motivation. Channel-state estimates modulate attention during cross-modal interaction, while selective reconstruction targets modalities identified as low quality. Shared parameters allow repeated refinement without introducing a separate fusion block for each round.

  2. The empirical evaluation provides meaningful support for the proposed architecture. Experiments cover multiple tasks and both noisy and missing modality settings, with component ablations examining the contributions of the main modules. These results support the practical value of the fusion design and strengthen my assessment.

Critical weaknesses
  1. The noise score has not been directly validated as a measure of modality quality. The authors use spectral statistics to estimate the severity of corruption, but this interpretation is inconsistent with Eqs. (4)--(5). It also remains unclear how the score behaves as an input becomes increasingly corrupted. Overall performance gains in the presence of noise do not establish that this score correctly identifies unreliable inputs. I would like the authors to clarify its interpretation and, while keeping the other modalities unchanged, progressively corrupt one modality and show how its noise score and corresponding attention and fusion weights change. This would directly test whether the score provides a useful basis for quality-dependent fusion.

  2. The additional value of accumulating messages across rounds remains unclear. I understand the motivation for repeated interaction, however, the method also maintains a separate state that accumulates messages from all preceding rounds. The existing comparisons support repeated interaction and the complete architecture, but do not isolate the benefit of this additional accumulation. I would like to see a comparison that keeps the same interaction rounds and training settings, while replacing the accumulated message state with only the current round's message. Comparing their performance under clean and corrupted inputs would clarify what the accumulation mechanism contributes and when it is useful.

Questions for authors
  1. Eq. (4) defines βm\beta_m as the energy fraction in the highest quarter of the frequency range, but the text states that βm→1\beta_m \to 1 for noise-dominated signals. Spreading noise energy across the whole spectrum does not imply this limit. What justifies the stated limit? Does the formula or its interpretation need correction?

  2. In Sec. 3.3, reconstruction is triggered solely by the target modality being low quality. However, the text describes the availability of ``present, correlated, cleaner peers'' as the exact situation in which Eq. (13) is activated. These are different conditions: a degraded target does not establish that suitable peers are available. Does the method additionally check these peer conditions? If not, how is this statement justified?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

W1 / Q1: Direct validation of beta; the beta -> 1 limit.

Response: Thank you for the valuable comment.

(1) We agree the βm\beta_m was not directly validated and the limit was inconsistent with Eq.4. We have corrected the limit wording as $ \beta_m \rightarrow 0.25$, and run the exact progressive-corruption study requested, on both sequence and vector modalities, all on one trained CH-SIMS-v2 model. With the two peers fixed, one modality is corrupted at increasing noise ratio NR, and βm\beta_m and the final fusion weight πm\pi_m are recorded.

Table Q1-1: Inverse-SNR βm\beta_m for sequence and vector modalities (mean over samples) as modality becomes increasingly corrupted.

ModalityNR=00.20.40.60.81.0Spearman
text (sequence modality, spectral)0.0020.0100.0510.1440.2390.2701.00
audio (vector modality, LRT)0.0000.0410.1640.3480.6640.9041.00
video (vector modality, LRT)0.0010.0380.1550.3630.6290.8851.00

Table Q1-2: Realized weight πm\pi_m in the final state-weighted fusion.

ModalityNR=00.20.40.60.81.0Spearman
text0.3330.3310.3220.3010.2800.272-1.00
audio0.3340.3250.2980.2600.2040.167-1.00
video0.3330.3250.3000.2570.2100.170-1.00

As shown in Table Q1-1, βm\beta_m is near zero for every clean input and increases strictly with severity (Spearman = 1.0 for all three). The text βm\beta_m (spectral cue) saturates at the predicted white-noise level 0.25 (finite-frame 0.270), while the vector βm\beta_m (LRT) spans almost the full range (0.904, 0.885). The fusion weight πm\pi_m falls strictly as the modality is corrupted (Spearman = -1.0), so the damaged modality's actual contribution to the prediction is reduced, not merely its internal statistic (see Table Q1-2).

(2) Please note that for vector modalities, the new likelihood-ratio score (LRT) replaces the old dispersion ratio (Eq.5) as the old dispersion ratio might fail in specific cases (such as zero-mean vectors): with blind EMA statistics (μm,σm2\mu_m, \sigma_m^2),

(z_j – \mu _ {m,j})^2 / \sigma _ {m,j}^2 - z_j^2 ],$$ we calibrate it against the analytic expectations $\ell_m^0$ and $\ell_m^1$ under clean and pure-noise inputs (no fitted scaling), and report $$\beta_m = clip( (\ell_m - \ell_m^0) / (\ell_m^1 - \ell_m^0), 0, 1 ),$$ so the clean/corrupted distinction is restored and the statistic is calibrated without fitted scaling. We have rerun the affected vector datasets (MM-IMDb, AV-MNIST, CH-SIMS-v2) with the new score. The end-task numbers are slightly better than the original vector $\beta_m$ (gains on AVMNIST and MM-IMDb, mixed but better MAE/Corr on CH-SIMS-v2; Table Q1-3), and the conclusions on them are unchanged. Other results use sequence modalities, and the results and findings are unchanged. For more details, please refer to the response to other reviewers. Table Q1-3: Results with the original vs. the repaired vector $\beta_m$. | Dataset | metric | original $\beta_m$ | repaired $\beta_m$ | | --- | --- | --- | --- | | AVMNIST | I | 91.33 | **92.00** | | AVMNIST | A | 98.00 | **98.67** | | AVMNIST | I+A | **99.33** | **99.33** | | AVMNIST | Avg | 96.22 | **96.67** | | MM-IMDb | T | 59.63 | **60.23** | | MM-IMDb | I | 46.02 | **46.04** | | MM-IMDb | T+I | 62.06 | **62.69** | | MM-IMDb | Avg | 55.90 | **56.32** | | CH-SIMS-v2 | Acc5 | 56.7 | **56.8** | | CH-SIMS-v2 | Acc3 | **75.8** | 75.0 | | CH-SIMS-v2 | Acc2 | **83.2** | 83.1 | | CH-SIMS-v2 | F1 | **83.2** | 83.1 | | CH-SIMS-v2 | MAE | 0.280 | **0.278** | | CH-SIMS-v2 | Corr | 0.742 | **0.752** |
AuthorsOpenReview ↗

Response to Weakness 2 and Question 2

W2: Isolated value of accumulating messages.

Response: We appreciate the precious comment of the reviewer. We have added a controlled comparison under identical rounds and training settings that replaces the accumulated buffer Lm(t)=Lm(t−1)+η⋅pm(t)L_m^{(t)} = L_m^{(t-1)} + \eta \cdot p_m^{(t)} with a current-round state Lm(t)=η⋅pm(t)L_m^{(t)} = \eta \cdot p_m^{(t)} (no carry-over).

Table W2-1: Current-round-only comparison on MOSI (Acc2).

Variantlavavallvlva
CAT-MF (full)89.667.889.089.689.365.563.7
current-round (no carry-over)89.266.588.888.588.962.862.4

As shown in Table W2-1, removing carry-over is nearly free when all three modalities are present (-0.4) but increasingly costly as redundancy disappears: -1.3 on {a,v} and {a}, -1.1 on {l,v}, and -2.7 on the single-visual split. The weak single channels rely most on the accumulated evidence, so the buffer has a distinct, measurable value that is not reproduced by a one-shot state.

Q2: Reconstruction trigger vs. peer conditions.

Response: We thank the reviewer for the precious comment. The implementation triggers from the target modality's own state (βm>τβ\beta_m > \tau_{\beta} or αm<τα\alpha_m < \tau_{\alpha}) and does not explicitly check peer availability; the sentence linking this to the "present, correlated, cleaner peers" condition was imprecise, and we have corrected the statement accordingly. We distinguish the trigger from the Theorem 3 benefit condition and add a peer-availability guard so the reconstruction fires only when the maximum peer reliability exceeds a threshold; otherwise the target is retained. As shown in Table Q2-1, the guard is neutral on the full split (89.6) and within +/-0.7 on the two-modality splits, while it helps the single-visual channel (+3.1), so preventing a reconstruction when no reliable peer exists does not weaken the model.

Table Q2-1: Peer-availability guard comparison on MOSI (Acc2).

Variantlavavallvlva
CAT-MF (full)89.667.889.089.689.365.563.7
peer guard89.667.489.789.389.468.662.1

Thank you for both responses and the additional experiments. The corrected spectral interpretation, the revised vector-modality score with rerun results, and the current-round-only ablation address much of my original concern. In particular, the ablation provides clearer evidence for the benefit of accumulation under missing modalities. These results strengthen my confidence in the method’s practical value. I still consider it important to keep the theoretical claims within their stated assumptions and clearly document which implementation changes are included in the final method. Overall, I will maintain my weak-accept recommendation.

Dear Reviewer XGx8,

We sincerely thank you for the positive evaluation of our work, the constructive suggestions, and for recognizing that the corrected spectral interpretation, the rerun vector-modality results, and the current-round-only ablation address the original concerns. We have carefully addressed the two points you emphasize, which are included in the revised manuscript.

Theoretical claims are kept within their stated assumptions. The analysis is organized so that every statement is explicitly scoped. The chain-rule monotonicity is presented as an unconditional identity that needs no model assumption, while the compressed-buffer gain is a conditional result stated under the linear-Gaussian, independent-channel, and martingale-noise assumptions together with the admission condition. The donor-level lower bound is explicitly limited to the slot-addressable setting and is not used for the main claim. The implementation change includes the βm\beta_m for vector modalities, which is included in the revised manuscript.

We are grateful for the time and care you invested in this work, which has materially improved both the rigor and the clarity of the paper. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.