- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper treats noisy and missing modalities as two forms of "channel impairment" and proposes Channel-Aware Turbo Multimodal Fusion (CAT-MF). CAT-MF first estimates a reference-free channel state for each modality consisting of an existence probability , an inverse-SNR proxy , and a learned damage-type embedding . These estimates are then used to modulate cross-modal attention so that missing or noisy modalities contribute less and modalities with similar estimated damage types exchange less information. Fusion is performed through a weight-tied iterative decoder that accumulates cross-modal messages over several rounds using a geometrically decaying update schedule, motivated by Turbo decoding. The model also includes selective reconstruction of low-quality modalities and reliability-weighted final fusion. The paper provides theoretical analyses of decoder convergence, information accumulation, and reconstruction error, and evaluates CAT-MF across seven multimodal benchmarks under complete, missing-modality, and noisy settings, where it generally achieves strong or state-of-the-art performance.
Critical strengths
-
The parameter-matched comparison in Table 9 is the most consequential positive result. A single weight-shared block iterated three times outperforms an independent three-layer stack with the parameters, and channel-aware gating adds a further gain at the same parameter budget. This is a clean and controlled experiment with a useful takeaway for fusion architecture design, independent of the paper's broader communication-theoretic framing.
-
The core framing of separating reliability from relevance in cross-modal attention is well motivated. The evaluation is broad, covering seven benchmarks spanning sentiment, humor, sarcasm, movie-genre classification, and digit classification, including both missing- and noisy-modality settings. The attempt to model not only whether a modality is unreliable but also the type of degradation is an interesting direction for robust multimodal fusion.
Critical weaknesses
-
The inverse-SNR proxy is not fully convincing for vector modalities. Several vector pipelines apply LayerNorm before the CSE, which may constrain more by normalization than by corruption. Also, a constant nonzero vector gives , which seems inconsistent with the paper's description of flat, unstructured vectors as noisy.
-
The noisy-modality protocol is unclear. The CSE is described as operating on pre-pooling features , while Eq. 58 injects noise into pooled features . If taken literally, the estimator would not observe the corruption used in Table 4.
-
The interpretation of Table 4 is not fully convincing. Since all modalities are corrupted with the same family and rate, common reliability or damage factors may largely cancel after row normalization. This makes it difficult to attribute the gains specifically to the proposed anti-corroboration mechanism.
-
Some results are difficult to reconcile with the iterative-decoding explanation. Figure 5 shows updates becoming nearly negligible after the first round, yet Table 8 reports sizable gains from over . This is especially unclear in single-modality settings, where no cross-modal messages should remain to accumulate.
-
The theoretical guarantees appear weaker than the presentation suggests. Theorem 1 mainly follows from bounded updates and summable step sizes, while Theorem 2 is proved for an evidence-history representation that differs from the implemented summed buffer. The appendix itself notes that the donor-level guarantee does not directly carry over to the actual implementation.
-
The empirical reporting could be stronger. Results are averaged over five seeds, but no variance is reported. There are also some inconsistencies between the text and tables; for example, Section 5.3 states that CAT-MF is best at every Gaussian-noise level, while Table 4 does not support this for every MOSI Acc2 setting.
Questions for authors
-
In single-modality configurations, what are the realized gate weights given soft , row normalization, and ? What explains the versus gap there?
-
Which step schedule was actually used? Appendix C.3 describes a "sigmoid warm-up of Eq. 11," which conflicts with the geometric schedule that Theorem 1 relies on.
-
If the model is trained with but evaluated with , does performance hold? If so, the benefit of would seem to be a training effect rather than iterative refinement. How do the authors reconcile this with Figure 5?
-
What are the donor weights when one modality is corrupted and two are clean? Clean modalities share the "clean" prototype, so and . Could the factor therefore favor the corrupted donor over the clean one?
-
Can the authors isolate the effect of in Figure 4 with a -matched control? How does CAT-MF compare against DEQ fusion (Ni et al., 2023) and COCOON (Cho et al., 2025), the closest prior work on iterative and uncertainty-aware fusion?
11 comments
Response to Weakness 2, Weakness 3, Weakness 4, and Question 1
W2: Noisy-modality protocol (where is corruption applied?).
Response: We are sorry for the typo. In the implementation, corruption is applied to the unimodal sequences before mean pooling, and the CSE consumes those same corrupted sequences, so the estimator does observe the corruption used in Table 4. Eq. (58) is a pooled-level shorthand that obscures this. We have corrected the protocol to state the true operation.
W3: Table 4 attribution when all modalities share one damage family.
Response: We thank the reviewer and agree that the fully symmetric setting of Table 4 cannot by itself isolate anti-corroboration: when every modality is corrupted by the same family and rate, both the reliability factor and the pairwise damage-type factor are common across donors, and they cancel after row normalization. We re-scope the claim accordingly. Table 4 is an end-task robustness result under uniform corruption, whose gains come from the reliability term down-weighting noisy channels together with the iterative exchange and selective reconstruction, rather than from anti-corroboration. We have stated this explicitly in the main text.
W4 / Q1: In single-modality configurations, what are the realized gate weights given soft , row normalization, and ? What explains the versus gap there?
Response: We thank the reviewer for the valuable comments. The number of decoder tokens is fixed at construction, so a "single-modality" pattern means one real token coexisting with learnable miss tokens for the absent modalities, not a literal one-token model. We have measured the realized states on the trained MOSI model for each choice of the real modality. For the zeroed miss channels, the CSE outputs a near-zero (a small positive value below 5e-5). The realized gate matrix is shown in Table W4-1.
Table W4-1: Realized gate matrix with text as the real modality (, donor n into target m).
| target \ donor | text | visual | acoustic |
|---|---|---|---|
| text (real) | - | 0.5092 | 0.4903 |
| visual (miss) | 1 | - | 0 |
| acoustic (miss) | 1 | 0 | - |
The miss target rows point to the real donor with weight 1. The real token's row is a normalized relative distribution over the two miss donors, not an absolute transfer: their absolute reliability is about 1e-5, and the gate and block normalize the row, so the ~0.5 only describes how this tiny residual is split, not its magnitude. only prevents division by zero.
Why can still outperform in Table 8. Every model variant in Table 8 is trained and evaluated at its own , so this is not a truncation effect. The single-modality Acc2 numbers are:
| Single-modality split (Acc2) | T = 3 (Full) | T = 1 | gap |
|---|---|---|---|
| {l} language only | 89.3 | 85.8 | +3.5 |
| {a} acoustic only | 63.7 | 61.6 | +2.1 |
| {v} visual only | 65.5 | 66.5 | -1.0 |
What changes with is the depth of the weight-tied computation: each round still applies the per-token FFN and the refinement to the real token, so the unrolled model is a deeper recurrent network. Training a deeper unrolled graph shapes the shared weights so the same representation is progressively self-refined and better mapped to the decision, the same reason a deeper nonlinear model can beat a shallower one on a single input. The effect is modality-dependent and not a guaranteed gain: visual modality reverses by 1.0, which is the signature of a depth/optimization effect. advantage is consistent with the deeper recurrent trajectory shaping the learned weights during training.
Additionally, before decoding, the rule replaces each absent input with one learnable token instead of zero-padding. This is a learned marginal prior: it carries the generic, average representation of an absent modality as seen in training, and it is refined from peers in later rounds, providing informative modality priors.
Response to Weakness 5, Weakness 6, Question 2, and Question 3
W5 / Q2: Which step schedule is actually used?
Response: We are sorry for the typo. The deployed schedule is the geometric decay: , matching Eq.11 (Eq.13 of the revised paper) and Theorem 1. We have corrected Appendix C.3 so it no longer references a sigmoid warm-up. With the common values , the realized steps are , and the total schedule mass is .
Because the schedule is summable, Part I of the rewritten Theorem 1 gives absolute convergence and the linear rate directly, with no finite-horizon workaround. Part II then shows the steady-state map is a contraction with a unique fixed point when , and bounds the small gap between the implemented limit and that fixed point; the deterministic zero initialization makes the path unique. The measured one-step Jacobian spectral radius is consistent with and a negligible gap. After these edits, Eq.11 (Eq.13 of the revised paper), Appendix C.3 and Theorem 1 describe one and the same schedule.
W6: Variance is not reported and text/table inconsistencies.
Response: Thank you for the valuable comment. We have added multi-seed standard deviations across the five runs (mean std below) and correct the Sec. 5.3 claim: CAT-MF achieves the best MAE at every noise level, while Acc2 is best at most levels.
Table W6-1: Five-seed mean std on CMU-MOSI and CMU-MOSEI.
| Dataset | Acc7 | Acc2 | F1 | MAE | Corr |
|---|---|---|---|---|---|
| MOSI | 51.5 0.8 | 89.6 0.6 | 89.6 0.6 | 0.571 0.016 | 0.878 0.009 |
| MOSEI | 55.0 0.6 | 88.6 0.4 | 88.5 0.4 | 0.501 0.009 | 0.817 0.007 |
Table W6-2: Five-seed mean std on CH-SIMS-v2.
| Acc5 | Acc3 | Acc2 | F1 | MAE | Corr |
|---|---|---|---|---|---|
| 56.8 0.5 | 75.0 0.4 | 83.1 0.7 | 83.1 0.7 | 0.278 0.006 | 0.752 0.008 |
As shown in Table W6-1 and Table W6-2, the standard deviations are small (at most 0.8 on the accuracy metrics and 0.016 on MAE), so the point estimates in the main tables are stable and the rankings do not change across seeds.
Q3: -trained model evaluated at ; is the gain only a training effect? How do the authors reconcile this with Figure 5?
Response: We thank the reviewer for the valuable comments.
(1) We have run exactly this control on the trained MOSI model with no retraining: all weights are fixed and only the number of decoding rounds is reduced from to . Performance does not hold, so the gain is not merely a training effect (see Table Q3-1).
Table Q3-1: -trained model evaluated at .
| MOSI (Acc2) | lav | av | al | lv | l | v | a |
|---|---|---|---|---|---|---|---|
| T = 3 (full model) | 89.6 | 67.8 | 89.0 | 89.6 | 89.3 | 65.5 | 63.7 |
| T = 1 (same weights, truncated) | 88.7 | 62.3 | 87.2 | 88.3 | 88.7 | 60.7 | 60.1 |
| gap | -0.9 | -5.5 | -1.8 | -1.3 | -0.6 | -4.8 | -3.6 |
Because the identical trained weights are used, the drop is caused directly by removing the iterative rounds, not by optimization. The pattern is informative. With three reliable modalities (lav), one round already captures most of the gain and truncation costs only 0.9, since a single average over safe channels is already good. The cost grows as redundancy disappears: -5.5 on {a,v}, -4.8 on single visual and -3.6 on single acoustic, where the weak channel must be repeatedly corrected by its peer and re-routed across rounds. This is precisely the work that one-shot fusion cannot do.
(2) Reconciliation with Figure 5. Figure 5 plots the per-round update norm on the full-modality trajectory, where it becomes small after the first round under the geometric , and on that same lav split the truncation gap is only -0.9, so the two are consistent. A small token-motion norm does not mean the rounds are unused: the norm measures how far each token moves, not the routing decision or the accumulated evidence. On the weak and missing splits, the gate keeps reallocating which donor each token listens to, and these small-norm changes move decisions near the boundary, which is why truncation changes accuracy by up to 5.5 even though the plotted norm is small. The truncation test is therefore the decisive evidence: the benefit is genuine iterative refinement, and it is largest where the model has the least redundancy.
Response to Weakness 1
W1: βm for vector modalities; the constant-vector edge case.
Response: (1) We agree on both the LayerNorm issue and that a constant non-zero vector gives βm=0 in Eq.5 under the old dispersion statistic. These are properties of the vector branch only, which is confined to AVMNIST, MM-IMDb and the two vector modalities of CH-SIMS-v2. Therefore, we have replaced the vector βm in Eq.5 with a self-contained likelihood-ratio test (LRT), which is well defined for constant and LayerNorm inputs. Specifically, each affected modality keeps blind EMA running statistics of its clean pooled features, a mean μm and a diagonal variance σm2, and scores the per-sample log likelihood ratio between the clean channel N(μm,diagσm2) and the sensor noise N(0,I). Writing the per-sample statistic
(z_j – \mu _ {m,j})^2 / \sigma _ {m,j}^2 - z_j^2 ],$$ We calibrate it against the analytic expectations $\ell_m^0$ and $\ell_m^1$ under clean and pure-noise inputs (no fitted scaling) respectively, and report $$\beta_m = clip( (\ell_m - \ell_m^0) / (\ell_m^1 - \ell_m^0), 0, 1 ).$$ The implementation uses Bessel-corrected variance, a variance floor and a clean-sample count, and withholds the score until enough clean examples are seen. This is well defined for zero-mean/LayerNorm and constant inputs, and it is monotone under the deployed blend impairment (a plain Mahalanobis distance is not, because the blend first contr\$acts the variance). (2) We also wish to clarify that the zero-mean/LayerNorm failure is specific to the vector modalities, which appears only in AVMNIST, MM-IMDb and the two vector modalities of CH-SIMS-v2. AVMNIST and MM-IMDb are presented in Appendix B.1 explicitly as transfer experiments "to test generalization." Hence the main results (MOSI/MOSEI, ablation, noise experiments, etc.) use sequence modalities and are not implicated, and the main claims of the proposed method remain valid. **We have rerun the affected vector datasets with the new $\beta_m$**. **The performance of new vector $\beta_m$ is slightly better than the original vector $\beta_m$** (gains on AVMNIST and MM-IMDb, mixed but better MAE/Corr on CH-SIMS-v2; Table Q1-1), and the conclusions on them are unchanged. Table Q1-1: Results with the original vs. the repaired vector $\beta_m$. | Dataset | metric | original $\beta_m$ | repaired $\beta_m$ | | --- | --- | --- | --- | | AVMNIST | I | 91.33 | **92.00** | | AVMNIST | A | 98.00 | **98.67** | | AVMNIST | I+A | **99.33** | **99.33** | | AVMNIST | Avg | 96.22 | **96.67** | | MM-IMDb | T | 59.63 | **60.23** | | MM-IMDb | I | 46.02 | **46.04** | | MM-IMDb | T+I | 62.06 | **62.69** | | MM-IMDb | Avg | 55.90 | **56.32** | | CH-SIMS-v2 | Acc5 | 56.7 | **56.8** | | CH-SIMS-v2 | Acc3 | **75.8** | 75.0 | | CH-SIMS-v2 | Acc2 | **83.2** | 83.1 | | CH-SIMS-v2 | F1 | **83.2** | 83.1 | | CH-SIMS-v2 | MAE | 0.280 | **0.278** | | CH-SIMS-v2 | Corr | 0.742 | **0.752** | (3) Additionally, **we have conducted experiments to report the relationship between $\beta_m$ and actual corruption severity**, on both sequence and vector modalities, all on one trained CH-SIMS-v2 model. With the two peers fixed, one modality is corrupted at increasing noise ratio NR and $\beta_m$, the gate and the final fusion weight $\pi_m$ are recorded. Table Q1-2: $\beta_m$ values for sequence and vector modalities (mean over samples) as modality becomes increasingly corrupted. | Modality | NR=0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | Spearman | | --- | --- | --- | --- | --- | --- | --- | --- | | text (sequence) | 0.002 | 0.010 | 0.051 | 0.144 | 0.239 | 0.270 | 1.00 | | audio (vector) | 0.000 | 0.041 | 0.164 | 0.348 | 0.664 | 0.904 | 1.00 | | video (vector) | 0.001 | 0.038 | 0.155 | 0.363 | 0.629 | 0.885 | 1.00 | Table Q1-3: Realized weight $\pi_m$ in the final state-weighted fusion. | Modality | NR=0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | Spearman | | --- | --- | --- | --- | --- | --- | --- | --- | | text | 0.333 | 0.331 | 0.322 | 0.301 | 0.280 | 0.272 | -1.00 | | audio | 0.334 | 0.325 | 0.298 | 0.260 | 0.204 | 0.167 | -1.00 | | video | 0.333 | 0.325 | 0.300 | 0.257 | 0.210 | 0.170 | -1.00 | As shown in Table Q1-2, **$\beta_m$ is near zero for every clean input and increases strictly with severity (Spearman = 1.0 for all three)**. The text $\beta_m$ (spectral cue) saturates at the predicted white-noise level 0.25 (finite-frame 0.270), while the vector $\beta_m$ (LRT) spans almost the full range (0.904, 0.885). The fusion weight $\pi_m$ falls strictly as the modality is corrupted (Spearman = -1.0), so the damaged modality's actual contribution to the prediction is reduced, not merely its internal statistic (see Table Q1-3).