- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the phenomenon of "forced fusion" in multimodal models. While such fusion may possess some predictive value, it can also interfere with final predictions when the source of the fused information is misaligned with the target task. The authors first analyze the performance degradation observed when fusion models trained under aligned conditions that are presented with misaligned inputs. This phenomenon is empirically validated through synthetic tasks, five audiovisual datasets, and pre-trained general-purpose multimodal models. Drawing inspiration from Bayesian causal inference, the authors propose a module that estimates fusion weights by explicitly supervising the correspondences between modalities. This approach achieves a balance between the fused prediction and the independent predictions derived from individual modalities. Furthermore, the authors derive an upper bound for a "mixed log-loss." The experimental evaluation encompasses both task-specific fusion architectures and large-scale models such as Qwen2.5 and Qwen3, with a particular focus on scenarios involving conflicts between audio and text modalities.
Critical strengths
-
The motivation is clear, and the phenomenon of "mismatch" is supported by robust empirical evidence. The paper examines the differences in predictive performance between matched and mismatched conditions across synthesis tasks, five audiovisual datasets, and general-purpose multimodal models. The significant drop in accuracy observed across multiple datasets indicates that this issue is not confined to any single fusion architecture. These findings help shift the focus of multimodal evaluation, moving beyond standard accuracy metrics toward assessing whether a model can determine when auxiliary evidence is actually applicable.
-
The authors attempt to link the identification of correspondences to predictive outcomes, rather than merely reporting final accuracy. In addition to presenting target prediction performance, the paper analyzes the sensitivity of gating mechanisms to mismatches.
Critical weaknesses
-
There remains a gap between the semantics of the labels used for correspondence supervision and the core claims of the paper. The task-specific model employs auxiliary modality permutation to construct correspondence labels; however, the authors acknowledge that such permutations may inadvertently preserve the original pairings or class labels. Consequently, the supervisory labels do not necessarily reflect the true underlying source relationships or task states. While the paper’s upper bound on the loss function holds for the assigned binary labels, this does not prove that the learned gating mechanism is capable of identifying the actual source correspondences. Furthermore, the current experiments primarily rely on mismatched pairs drawn from different classes, failing to adequately distinguish between source inconsistency, class inconsistency, and variations in task relevance.
-
The independent contribution and statistical reliability of the newly introduced modules have not been sufficiently established. In the task-specific model, the "Mixed+BCI" configuration yields only a 1.44 percentage-point improvement in the unweighted average accuracy compared to the "Mixed" baseline, while the matching accuracy actually declines by 2.40 percentage points. For the general-purpose model, the unweighted average target accuracy drops from 79.26% to 78.77%. Although the balanced correspondence accuracy for the general-purpose model shows an increase, the model’s performance in correctly identifying mismatched samples has deteriorated. The authors should therefore provide a plausible explanation for these findings or conduct further analytical experiments.
Questions for authors
-
How can the performance contributions of correspondence supervision and the additional classification branches be disentangled? Compared to the "Mixed" baseline, "Mixed+BCI" not only incorporates a correspondence loss but also introduces classification losses for target unimodal, auxiliary unimodal, and mixed-modal predictions. The authors should provide results for the target prediction task while keeping the model architecture and classification losses constant, varying only the strength of the correspondence supervision.
-
Do the gated supervision labels truly correspond to the source relationships as defined in the paper? The task-specific model uses as the correspondence label; however, the authors acknowledge that permutations might inadvertently preserve the original pairings or class labels. Please quantify this label noise and report its scale across the various datasets.
5 comments
Response to Reviewer pkQ6 part[2/2]
W2. Accuracy trade-offs and statistical reliability
We examine variability and training exposure, then distinguish judgment quality from its use in General Purpose classification. Avg equally weights matched and mismatched accuracy.
First, we report mean Avg gains and standard deviations at 50% exposure across seven methods and five seeds. For SDs, we first average methods within each dataset and seed; all entries are pp:
| Dataset | Avg gain over Mixed | SD: Mixed | SD: Mixed+BCI |
|---|---|---|---|
| AV-MNIST | +3.8 | 5.13 | 2.98 |
| AVE | +0.8 | 1.67 | 0.63 |
| Kinetics-Sounds | +0.2 | 0.65 | 0.66 |
| CREMA-D | +1.6 | 0.86 | 0.77 |
| RAVDESS | +0.7 | 2.01 | 1.85 |
Mean Avg gains are positive on all five datasets, and Mixed+BCI has lower SDs on four. These results show consistent improvements in dataset means alongside reduced variability on most datasets.
We next examine the matched cost in Fig. 5(a). At 50% exposure, the +5.3pp mismatched gain is partly offset by −2.4pp matched accuracy, leaving +1.44pp Avg. At 5%, the Avg gain is +5.8pp, with only −0.2pp matched change. Correspondence AUROC already reaches 0.77 at 1% and 0.79 at 5%, then plateaus near 0.80. Thus, among the tested exposures, the largest predictive gain occurs as correspondence discrimination develops with limited mismatch exposure, while matched performance remains closer to Mixed.
For General Purpose models, we use the three fine-tuning repetitions available for both recipes on four clip datasets. Differences are Mixed+BCI minus Mixed:
| Model | Avg difference ± s.e. (pp) | Judgment balanced-accuracy gain (pp) |
|---|---|---|
| Qwen2.5 | −2.1±0.8 | +6.2 |
| Qwen3 | +0.3±0.5 | +6.2 |
Both models improve mean judgment balanced accuracy in all three repetitions, averaging +6.2pp. Avg decreases for Qwen2.5 and changes little for Qwen3.
The class-specific recalls explain the submitted Table 3's judgment result. Mean mismatch recall falls from 86.16% to 81.22%, but false rejection of matched pairs falls from 22.1% to 9.2% for Qwen2.5 and from 26.9% to 6.5% for Qwen3. AUROC also rises from 0.89/0.88 to 0.92/0.94. Thus, fewer false rejections of matched pairs account for the balanced-accuracy gain despite lower mismatch recall, while higher AUROC shows improved ranking.
To test a direct link to classification, we evaluated an additional judgment-before-classification variant in one conversation. It retains all 4,000 Mixed classification instructions (1,923 auditory). Both conditions receive the same correct judgment as training context and differ only in judgment-token loss. Across two models, four clip datasets, and three seeds:
| Measure | Judgment loss on | Judgment loss off |
|---|---|---|
| Judgment AUROC | 0.93 | 0.81 |
| Avg after own judgment | 71.6 | 68.3 |
| Avg with correct judgment supplied | 75.3 | 75.5 |
Supervision adds +3.3pp after the model's own judgment, improving 20/24 model–dataset–seed comparisons and 7/8 model–dataset means. Supplying correct judgments brings the conditions to similar accuracy, linking the gain to judgment quality. Forcing “no”→“yes” increases image-following by 48.7/45.2pp with loss on/off, showing that both conditions use the judgment. Thus, in this added design, correspondence supervision improves classification at fixed classification exposure.
We report reliability and judgment controls in Secs. 5.3, E.7 and F, and clarify Fig. 5(a) and Table 3 in Sec. 5.2.
Dear Reviewer pkQ6,
Thank you again for your helpful suggestions.
To address your concerns about correspondence labels and predictive improvements, we added collision and pairing controls, comparisons with architecture and classification losses held fixed, and repeated-adaptation analyses. We would appreciate any further thoughts on these results and would be glad to discuss any remaining concerns.
If our response has resolved the issues raised, we would sincerely appreciate your consideration in your final evaluation.
Best regards, The Authors
Response to Reviewer pkQ6 part[1/2]
We are grateful for the detailed review and helpful suggestions, and respond to each point below.
W1/Q2. Correspondence labels and label ambiguity
We quantify expected reassignment-collision rates, then test source identity, category agreement, and task relevance separately. Equation (15) labels retained pairs positive and reassigned pairs negative. The following percentages are among reassigned items; the same-class row includes unchanged pairs:
Original-pair collisions arise from permutation fixed points and resampled records. Same-class/different-recording pairs preserve category agreement; relabeling them changes the supervised definition of correspondence.
At 50% exposure, relabeling same-class pairs as positive changes Avg by +0.3pp across five datasets. Fixed-point-free permutations change Avg by +0.6pp on AV-MNIST/RAVDESS. Aggregate accuracy is stable across both label controls.
With independently sampled partners at 50% training exposure, accuracies are 26.4/48.6/53.1% for Matched-only/Mixed/Mixed+BCI. The 4.5pp gain over Mixed extends the result beyond different-class evaluation.
On same-class/different-recording pairs from four clip datasets, gate AUROC against original pairs is 0.53, including after training with these negatives. The fused branch exceeds the target-only branch by 17pp, showing the value of category-compatible auxiliary evidence for classification.
To isolate event visibility, the AVE control retained event audio and used off-event frames from the same 111 eligible clips. Mixed+BCI accuracy changes from 57.8% on-event to 53.1% off-event; mean gate weight falls from 0.50 to 0.40, versus 0.12 for different-class frames. This control separates event visibility from class disagreement and shows a larger gate response to the latter.
Proposition 2 bounds mixture loss for the assigned labels. BCE alone has population optimum P(r=1∣xt,xa). With source-model labels and sampling, this equals the source posterior P(C=1∣xt,xa). Supervision raises gate/reference logit correlation with known synthetic c⋆ from 0.49 to 0.93, and with the real-data class-head proxy from 0.28 to 0.81. This supports synthetic posterior tracking and useful category-level correspondence on real data.
We have added the label controls and theoretical analysis in Secs. D.2–D.3 and D.5 (Tables 19, 21–23).
Q1. Separating classification branches, mixture structure, and supervision
To isolate contributions, we add unimodal heads/losses, then mixture prediction and its classification loss. The last two columns fix architecture and classification losses, varying only correspondence supervision. Entries are matched/mismatched accuracy (%):
Correspondence BCE uses matched and mismatched pairs in every row, including Matched-only classification.
At 50%, heads add 2.3pp Avg, the unsupervised mixture adds 0.1pp, and supervision changes Avg by −0.9pp. Fig. 5(b) likewise shows mismatched accuracy within 52.7–53.2% as λ increases, while AUROC rises from 0.588 to 0.798. The additional classification components explain the 50% gain; supervision principally improves correspondence discrimination at that exposure.
At 5%, the corresponding increments are +2.1, +2.7, and +0.9pp. The last two columns isolate supervision's predictive contribution while architecture and classification losses remain fixed.
We further vary supervision strength under Matched-only classification, retaining mixed pairs for correspondence BCE:
Increasing λ improves mismatched accuracy throughout this sweep, with λ=1 adding +10.0pp Avg over λ=0. Together with the 5% result, this establishes supervision's independent predictive contribution when classification sees few or no mismatched pairs.
Finally, we fix each model's branch predictions and replace matched-counterpart weights with mismatched-input weights. At 50% exposure, relative mismatched-error reduction is 2.4% with supervision versus 0.5% for λ=0. Constant gates use identical weights for both inputs, leaving predictions unchanged. Together, these controls establish supervision's predictive contribution at low exposure and its benefit through input-dependent weighting.
We report component and weighting controls in Secs. 5.3 and E.3–E.4 (Tables 4, 25–27).