acceptodds
Under review as a conference paper at ICLR 2027

Forced Fusion: Multimodal Models Do Not Know When Not to Bind

Abstract

Multimodal models benefit from complementary observations, but unrelated evidence can redirect predictions away from the intended target. We call this failure Forced Fusion. We demonstrate it in seven task-specific fusion architectures on a synthetic task and five audio-visual datasets, and in pretrained general-purpose multimodal models on audio-visual and audio-text tasks. Our analysis of matched training motivates learning source correspondence to determine when auxiliary evidence should influence prediction. We propose a module based on Bayesian causal inference (BCI) that explicitly learns correspondence and uses it to balance fused and target-only predictions. Evaluations show that our approach mitigates Forced Fusion and improves average target accuracy over the implicit Mixed baseline in most audio-visual settings. Furthermore, analyses of task-specific and adapted general-purpose models show that explicit supervision helps models distinguish matched from mismatched inputs, even when gains in target accuracy are modest. Controlled interventions keep the module's branch predictions fixed and vary only its mixing weight, showing that correspondence-dependent weighting improves mismatched prediction on average. These findings highlight the importance of evaluating target accuracy alongside the ability to recognize source correspondence and use it to guide multimodal fusion.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Multimodal Learning, Forced Fusion, Selective Integration, Cross-Modal Correspondence, Bayesian Causal Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pkQ6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the phenomenon of "forced fusion" in multimodal models. While such fusion may possess some predictive value, it can also interfere with final predictions when the source of the fused information is misaligned with the target task. The authors first analyze the performance degradation observed when fusion models trained under aligned conditions that are presented with misaligned inputs. This phenomenon is empirically validated through synthetic tasks, five audiovisual datasets, and pre-trained general-purpose multimodal models. Drawing inspiration from Bayesian causal inference, the authors propose a module that estimates fusion weights by explicitly supervising the correspondences between modalities. This approach achieves a balance between the fused prediction and the independent predictions derived from individual modalities. Furthermore, the authors derive an upper bound for a "mixed log-loss." The experimental evaluation encompasses both task-specific fusion architectures and large-scale models such as Qwen2.5 and Qwen3, with a particular focus on scenarios involving conflicts between audio and text modalities.

Critical strengths
  1. The motivation is clear, and the phenomenon of "mismatch" is supported by robust empirical evidence. The paper examines the differences in predictive performance between matched and mismatched conditions across synthesis tasks, five audiovisual datasets, and general-purpose multimodal models. The significant drop in accuracy observed across multiple datasets indicates that this issue is not confined to any single fusion architecture. These findings help shift the focus of multimodal evaluation, moving beyond standard accuracy metrics toward assessing whether a model can determine when auxiliary evidence is actually applicable.

  2. The authors attempt to link the identification of correspondences to predictive outcomes, rather than merely reporting final accuracy. In addition to presenting target prediction performance, the paper analyzes the sensitivity of gating mechanisms to mismatches.

Critical weaknesses
  1. There remains a gap between the semantics of the labels used for correspondence supervision and the core claims of the paper. The task-specific model employs auxiliary modality permutation to construct correspondence labels; however, the authors acknowledge that such permutations may inadvertently preserve the original pairings or class labels. Consequently, the supervisory labels do not necessarily reflect the true underlying source relationships or task states. While the paper’s upper bound on the loss function holds for the assigned binary labels, this does not prove that the learned gating mechanism is capable of identifying the actual source correspondences. Furthermore, the current experiments primarily rely on mismatched pairs drawn from different classes, failing to adequately distinguish between source inconsistency, class inconsistency, and variations in task relevance.

  2. The independent contribution and statistical reliability of the newly introduced modules have not been sufficiently established. In the task-specific model, the "Mixed+BCI" configuration yields only a 1.44 percentage-point improvement in the unweighted average accuracy compared to the "Mixed" baseline, while the matching accuracy actually declines by 2.40 percentage points. For the general-purpose model, the unweighted average target accuracy drops from 79.26% to 78.77%. Although the balanced correspondence accuracy for the general-purpose model shows an increase, the model’s performance in correctly identifying mismatched samples has deteriorated. The authors should therefore provide a plausible explanation for these findings or conduct further analytical experiments.

Questions for authors
  1. How can the performance contributions of correspondence supervision and the additional classification branches be disentangled? Compared to the "Mixed" baseline, "Mixed+BCI" not only incorporates a correspondence loss but also introduces classification losses for target unimodal, auxiliary unimodal, and mixed-modal predictions. The authors should provide results for the target prediction task while keeping the model architecture and classification losses constant, varying only the strength of the correspondence supervision.

  2. Do the gated supervision labels truly correspond to the source relationships as defined in the paper? The task-specific model uses ri=1−bir_i = 1 - b_i as the correspondence label; however, the authors acknowledge that permutations might inadvertently preserve the original pairings or class labels. Please quantify this label noise and report its scale across the various datasets.

5 comments

Response to Reviewer pkQ6 part[1/2]

We are grateful for the detailed review and helpful suggestions, and respond to each point below.

W1/Q2. Correspondence labels and label ambiguity

We quantify expected reassignment-collision rates, then test source identity, category agreement, and task relevance separately. Equation (15) labels retained pairs positive and reassigned pairs negative. The following percentages are among reassigned items; the same-class row includes unchanged pairs:

Retained property (%)AV-MNISTAVEKinetics-SoundsCREMA-DRAVDESS
Original pair0.41.11.00.81.7
Same class10.44.94.417.418.6

Original-pair collisions arise from permutation fixed points and resampled records. Same-class/different-recording pairs preserve category agreement; relabeling them changes the supervised definition of correspondence.

At 50% exposure, relabeling same-class pairs as positive changes Avg by +0.3pp across five datasets. Fixed-point-free permutations change Avg by +0.6pp on AV-MNIST/RAVDESS. Aggregate accuracy is stable across both label controls.

With independently sampled partners at 50% training exposure, accuracies are 26.4/48.6/53.1% for Matched-only/Mixed/Mixed+BCI. The 4.5pp gain over Mixed extends the result beyond different-class evaluation.

On same-class/different-recording pairs from four clip datasets, gate AUROC against original pairs is 0.53, including after training with these negatives. The fused branch exceeds the target-only branch by 17pp, showing the value of category-compatible auxiliary evidence for classification.

To isolate event visibility, the AVE control retained event audio and used off-event frames from the same 111 eligible clips. Mixed+BCI accuracy changes from 57.8% on-event to 53.1% off-event; mean gate weight falls from 0.50 to 0.40, versus 0.12 for different-class frames. This control separates event visibility from class disagreement and shows a larger gate response to the latter.

Proposition 2 bounds mixture loss for the assigned labels. BCE alone has population optimum P(r=1∣xt,xa)P(r=1\mid x_t,x_a). With source-model labels and sampling, this equals the source posterior P(C=1∣xt,xa)P(C=1\mid x_t,x_a). Supervision raises gate/reference logit correlation with known synthetic c⋆c^\star from 0.49 to 0.93, and with the real-data class-head proxy from 0.28 to 0.81. This supports synthetic posterior tracking and useful category-level correspondence on real data.

We have added the label controls and theoretical analysis in Secs. D.2–D.3 and D.5 (Tables 19, 21–23).

Q1. Separating classification branches, mixture structure, and supervision

To isolate contributions, we add unimodal heads/losses, then mixture prediction and its classification loss. The last two columns fix architecture and classification losses, varying only correspondence supervision. Entries are matched/mismatched accuracy (%):

Classification trainingBaseline+heads+mixture, λ=0\lambda=0+supervision, λ=1\lambda=1
Matched-only69.8/22.569.9/23.769.8/25.466.4/48.9
Mixed (5%)71.4/31.970.6/36.970.7/42.371.2/43.6
Mixed (50%)65.1/47.466.1/51.064.5/52.762.7/52.7

Correspondence BCE uses matched and mismatched pairs in every row, including Matched-only classification.

At 50%, heads add 2.3pp Avg, the unsupervised mixture adds 0.1pp, and supervision changes Avg by −0.9pp. Fig. 5(b) likewise shows mismatched accuracy within 52.7–53.2% as λ\lambda increases, while AUROC rises from 0.588 to 0.798. The additional classification components explain the 50% gain; supervision principally improves correspondence discrimination at that exposure.

At 5%, the corresponding increments are +2.1, +2.7, and +0.9pp. The last two columns isolate supervision's predictive contribution while architecture and classification losses remain fixed.

We further vary supervision strength under Matched-only classification, retaining mixed pairs for correspondence BCE:

λ\lambda00.010.10.31
Mismatched accuracy (%)25.434.741.845.648.9

Increasing λ\lambda improves mismatched accuracy throughout this sweep, with λ=1\lambda=1 adding +10.0pp Avg over λ=0\lambda=0. Together with the 5% result, this establishes supervision's independent predictive contribution when classification sees few or no mismatched pairs.

Finally, we fix each model's branch predictions and replace matched-counterpart weights with mismatched-input weights. At 50% exposure, relative mismatched-error reduction is 2.4% with supervision versus 0.5% for λ=0\lambda=0. Constant gates use identical weights for both inputs, leaving predictions unchanged. Together, these controls establish supervision's predictive contribution at low exposure and its benefit through input-dependent weighting.

We report component and weighting controls in Secs. 5.3 and E.3–E.4 (Tables 4, 25–27).

Response to Reviewer pkQ6 part[2/2]

W2. Accuracy trade-offs and statistical reliability

We examine variability and training exposure, then distinguish judgment quality from its use in General Purpose classification. Avg equally weights matched and mismatched accuracy.

First, we report mean Avg gains and standard deviations at 50% exposure across seven methods and five seeds. For SDs, we first average methods within each dataset and seed; all entries are pp:

DatasetAvg gain over MixedSD: MixedSD: Mixed+BCI
AV-MNIST+3.85.132.98
AVE+0.81.670.63
Kinetics-Sounds+0.20.650.66
CREMA-D+1.60.860.77
RAVDESS+0.72.011.85

Mean Avg gains are positive on all five datasets, and Mixed+BCI has lower SDs on four. These results show consistent improvements in dataset means alongside reduced variability on most datasets.

We next examine the matched cost in Fig. 5(a). At 50% exposure, the +5.3pp mismatched gain is partly offset by −2.4pp matched accuracy, leaving +1.44pp Avg. At 5%, the Avg gain is +5.8pp, with only −0.2pp matched change. Correspondence AUROC already reaches 0.77 at 1% and 0.79 at 5%, then plateaus near 0.80. Thus, among the tested exposures, the largest predictive gain occurs as correspondence discrimination develops with limited mismatch exposure, while matched performance remains closer to Mixed.

For General Purpose models, we use the three fine-tuning repetitions available for both recipes on four clip datasets. Differences are Mixed+BCI minus Mixed:

ModelAvg difference ± s.e. (pp)Judgment balanced-accuracy gain (pp)
Qwen2.5−2.1±0.8+6.2
Qwen3+0.3±0.5+6.2

Both models improve mean judgment balanced accuracy in all three repetitions, averaging +6.2pp. Avg decreases for Qwen2.5 and changes little for Qwen3.

The class-specific recalls explain the submitted Table 3's judgment result. Mean mismatch recall falls from 86.16% to 81.22%, but false rejection of matched pairs falls from 22.1% to 9.2% for Qwen2.5 and from 26.9% to 6.5% for Qwen3. AUROC also rises from 0.89/0.88 to 0.92/0.94. Thus, fewer false rejections of matched pairs account for the balanced-accuracy gain despite lower mismatch recall, while higher AUROC shows improved ranking.

To test a direct link to classification, we evaluated an additional judgment-before-classification variant in one conversation. It retains all 4,000 Mixed classification instructions (1,923 auditory). Both conditions receive the same correct judgment as training context and differ only in judgment-token loss. Across two models, four clip datasets, and three seeds:

MeasureJudgment loss onJudgment loss off
Judgment AUROC0.930.81
Avg after own judgment71.668.3
Avg with correct judgment supplied75.375.5

Supervision adds +3.3pp after the model's own judgment, improving 20/24 model–dataset–seed comparisons and 7/8 model–dataset means. Supplying correct judgments brings the conditions to similar accuracy, linking the gain to judgment quality. Forcing “no”→“yes” increases image-following by 48.7/45.2pp with loss on/off, showing that both conditions use the judgment. Thus, in this added design, correspondence supervision improves classification at fixed classification exposure.

We report reliability and judgment controls in Secs. 5.3, E.7 and F, and clarify Fig. 5(a) and Table 3 in Sec. 5.2.

Dear Reviewer pkQ6,

Thank you again for your helpful suggestions.

To address your concerns about correspondence labels and predictive improvements, we added collision and pairing controls, comparisons with architecture and classification losses held fixed, and repeated-adaptation analyses. We would appreciate any further thoughts on these results and would be glad to discuss any remaining concerns.

If our response has resolved the issues raised, we would sincerely appreciate your consideration in your final evaluation.

Best regards, The Authors

Reviewer_xGVHOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Forced Fusion and proposes Bayesian causal inference (BCI)-inspired gated prediction with correspondence supervision. The problem is relevant, but the theoretical contribution, benchmark validity, and evidence for the proposed mechanism remain insufficiently established. The present results do not demonstrate a reliable advantage over simpler training and gating controls. I recommend rejection.

Critical strengths
  1. The distinction between reliable observations and relevant auxiliary evidence identifies a meaningful problem in multimodal prediction.
  2. Experiments cover seven fusion methods, five audio-visual datasets, and pretrained and adapted Qwen models.
  3. The synthetic task, supervision sweep, and fixed-branch interventions provide useful diagnostic evidence beyond aggregate accuracy.
Critical weaknesses
  1. The BCI-specific contribution is insufficiently established. Eq. (6) is a generic gated mixture; Proposition 1 bounds risk changes on rare inputs, and Proposition 2 holds for arbitrary routing labels. These valid bounds neither identify the learned gate as a source-correspondence posterior nor establish correspondence as optimal routing. Appendix A.2 acknowledges these limitations, but its sufficient improvement condition is not empirically tested. 2.The operational evidence for Forced Fusion is incomplete. Table 1 measures matched-to-mismatched accuracy loss, which can reflect removal of useful auxiliary support rather than performance below using the target alone. Tables 1–2 omit standalone audio-only models and the added target-only branch. Without these references on identical target inputs, the severity of harmful fusion and the benefit over simply ignoring auxiliary evidence remain unclear.
  2. The real-data mismatch protocol does not implement independent sources as in Eq. (1). Different-class negative pairing induces label dependence: the auxiliary class excludes a possible target class. It also permits correspondence detection through class disagreement. Appendix C.2 still uses different-class pairs and does not resolve this issue. Independently sampled negatives and same-class, different-recording pairs, with correspondence explicitly defined, are needed. 4.Gate supervision contains avoidable label ambiguity. In Eq. (15), reassignment receives a negative label even when a permutation fixed point leaves the pair unchanged. Same-class swaps may also preserve task-level semantic correspondence. The paper acknowledges construction labels but does not quantify collisions or test corrected labels. These controls are needed to separate correspondence learning from fitting sampling artifacts. 5.The predictive benefit of explicit supervision is not demonstrated. Eq. (14) adds branches and losses relative to Mixed. The same-architecture control in Fig. 5(b) raises AUROC from 0.588 to 0.798, yet barely changes mismatched accuracy and lowers matched accuracy. Fig. 4(c) does not compare against constant or zero-supervision gates. Report the existing control by method and dataset, alongside constant-gate comparisons, to identify the source of improvement. 6.The Qwen experiments do not establish the selective-fusion mechanism. Correspondence and classification are queried in separate conversations, so improved yes/no accuracy does not show that correspondence regulates prediction. Appendix B.2.3 also replaces one third of classification instructions with correspondence questions under a fixed budget, reducing expected auditory examples from 2,000 to about 1,333. Controls preserving classification exposure and interventions on correspondence processing are needed. 7.Accuracy gains are inconsistent and statistically unresolved. Table 2(b) falls from 79.26% to 78.77% overall and from 66.00% to 59.00% on CREMA-D; the task-specific Kinetics-Sounds gain is only 0.20 percentage points. Table 2 aggregates architectures without seed uncertainty, and LoRA uses one run per condition (Table 14). Per-model results, repeated adaptation runs, and appropriate confidence intervals are needed to assess reliability.
Questions for authors
  1. Can you report audio-only models, the added target-only branch, and constant-gate controls on identical target inputs? Without these references, Table 1 cannot distinguish harmful interference from losing matched-information benefits. Please also report the missing no-description condition and replace substring scoring (Appendix B.2.2) with unambiguous answer parsing.

  2. What evidence links Qwen correspondence judgments to classification decisions, given the separate evaluation conversations? Under the 4,000-instruction budget, expected auditory classification exposure falls from 2,000 to about 1,333 examples (Appendix B.2.3). Can you hold classification exposure fixed and intervene on correspondence processing to distinguish selective integration from auxiliary-task learning?

3 comments

Response to Reviewer xGVH part[1/2]

We appreciate the constructive feedback and address each point below.

W1. BCI, gate identification, and the sufficient condition

Under the BCI source model in Eqs. (1–2), Eq. (5) is the Bayes posterior. With its reference branches, E[ℓF−ℓS∣x,C]\mathbb{E}[\ell_F-\ell_S\mid x,C] is −KL(pF⋆∥pS⋆)-\mathrm{KL}(p_F^\star\Vert p_S^\star) for C=1C=1 and +KL(pS⋆∥pF⋆)+\mathrm{KL}(p_S^\star\Vert p_F^\star) for C=2C=2. Thus oracle r=1(C=1)r=\mathbf{1}(C=1) minimizes expected hard-routing loss under the source model and Bayes branches.

BCE alone has population minimizer P(r=1∣x)P(r=1\mid x). With source-model labels and sampling, this equals Eq. (4), with π=1−pswap\pi=1-p_{\rm swap}. Gate/reference logit correlations are:

Referenceλ=1\lambda=1λ=0\lambda=0
Real data: Eq. (4) proxy from class heads, 50% exposure0.810.28
Synthetic: known c⋆c^\star, 50% exposure0.930.49
Synthetic: known c⋆c^\star, matched-only fused branch0.930.30

Supervision improves agreement with the synthetic posterior and real-data proxy.

To test A.2, let G=ρE[ℓX−ℓt∣r=0]G=\rho\mathbb{E}[\ell_X-\ell_t\mid r=0]; then Rmix−RX≤Rcorr−GR_{\rm mix}-R_X\le R_{\rm corr}-G. A negative bound certifies improvement. On balanced test pairs (nats):

Classification trainingMatched-only*Mixed (5%)Mixed (50%)
Actual Rmix−RXR_{\rm mix}-R_X, λ=1\lambda=1−3.85−0.33−0.05
Bound Rcorr−GR_{\rm corr}-G, λ=1\lambda=1−2.56+0.28+0.73
Bound, λ=0\lambda=0+2.94+2.03+2.51

*Classification uses matched pairs only; correspondence BCE still uses matched and mismatched pairs.

Supervision lowers the bound in all three settings. Under Matched-only classification, its change from +2.94 to −2.56 certifies the observed loss reduction.

We have added these derivations and tests in Secs. D.1–D.3.

W2/Q1. Audio-only baselines and audio-text scoring

To distinguish interference from lost support, we fix target inputs. Audio-only has the fusion budget and five seeds; Mixed uses 50% exposure. Mat/Mis are matched/mismatched accuracies (%), and Avg their equal-weight mean:

PredictorMatMisAvg
Audio-only58.157.958.0
Matched-only fusion69.822.546.1
Mixed65.147.456.3
Mixed+BCI62.752.757.7
Its target-only branch54.154.154.1

Of the 47.3pp matched-to-mismatched drop, 35.5pp falls below audio-only: auxiliary input causes harmful interference. Mixed+BCI reduces this deficit to 5.2pp and retains a 4.6pp matched advantage.

With exposure selected by validation prevalence-weighted accuracy and independent test partners, Mixed+BCI achieves 71.0%/65.5% at 5%/25% test mismatch, exceeding audio-only by 12.9/7.4pp. Selective fusion retains useful auxiliary information at these prevalences.

For audio-text interference, we evaluated the same first 300 MCR-Bench items/task without descriptions. Parsing takes the first standalone yes/no (AQA) or first word-boundary label/synonym (SER/VSC) on the first response line, without consulting the reference; unparsed answers are wrong.

Model/taskNo descriptionUnrelatedConflicting
Qwen2.5 AQA94.069.026.7
Qwen2.5 SER44.040.00.0
Qwen2.5 VSC90.088.06.0
Qwen3 AQA90.783.750.0
Qwen3 SER52.347.030.7
Qwen3 VSC92.090.354.7

Both description types underperform no description, establishing harmful interference under explicit parsing.

We report the baselines in Secs. 3.2 and E.1–E.2 (Table 24), and the parsing for Table 1(b) in Secs. B.2.2 and F.5.

W3/W4. Pairing, correspondence, and label collisions

To test whether class exclusion drives the result, we evaluate independent partners:

Test partners; 50% training exposureMatched-onlyMixedMixed+BCI
Different class22.547.452.7
Independent26.448.653.1

The 4.5pp gain with independent partners extends the result beyond different-class evaluation.

We quantified expected collision rates among reassigned pairs, all labeled negative:

Retained property (%)AV-MNISTAVEKinetics-SoundsCREMA-DRAVDESS
Original pair0.41.11.00.81.7
Same class10.44.94.417.418.6

Relabeling same-class pairs changes Avg by +0.3pp; fixed-point-free permutations give +0.6pp on two datasets. Aggregate accuracy is stable across both label controls.

To test recording identity, we used same-class/different-recording pairs on four clip datasets: AUROC is 0.53 against original pairs, including after training with such negatives. At 50% exposure, fusion exceeds the target-only branch by 17pp. Category-compatible auxiliary evidence remains useful for prediction.

We have added these pairing controls in Sec. D.5.

Response to Reviewer xGVH part[2/2]

W5. Branches, supervision, and weighting

To isolate contributions, we add heads/losses, then a mixture. Entries are Avg; the last two columns fix architecture and classification losses. Gate supervision uses both pair types:

Classification trainingBaseline+ heads+ mixture, λ=0\lambda=0+ supervision, λ=1\lambda=1
Matched-only*46.146.847.657.6
Mixed (5%)51.753.756.557.4
Mixed (50%)56.358.558.657.7

Supervision adds +10.0pp Avg under Matched-only classification and +0.9pp at 5% exposure. At 50%, it changes Avg by −0.9pp while raising AUROC from 0.588 to 0.798 (Fig. 5(b)). Thus, added heads account for the high-exposure accuracy gain, and supervision improves discrimination and contributes independently to prediction at low exposure.

The same-architecture Avg change for λ=1\lambda=1 minus 0 at 50% exposure is (pp; five seeds):

MethodAV-MNISTAVEKinetics-SoundsCREMA-DRAVDESS
Concat+0.4−2.1+0.2−0.9+0.1
Cross-attention+0.3−2.0−1.1+0.5−5.6
TMC−10.1+0.4+0.6+0.3−1.5
QMF−0.4−1.7−2.2+0.4−0.4
PDF+0.8−0.6−0.9−0.8−0.1
CoRiM+0.2−1.40.0−0.2−1.4
DCR−0.1−1.1+0.1−0.4−0.3

Across seven methods, Fig. 4(c)'s intervention fixes each model's branch outputs and replaces matched-counterpart with mismatched-input weights. Relative mismatched-error reductions (%):

Classification trainingMatched-only*Mixed (5%)Mixed (50%)
Supervised gate31.49.42.4
λ=0\lambda=0 gate−3.83.70.5

Constant gates assign both pairs the same weight, so the intervention leaves predictions unchanged. Supervised weights improve prediction with branch outputs fixed.

Finally, we test simpler readouts at the module's validation-selected exposure, fixing branches:

Weight on fused predictionMatMisAvg
c=1c=170.543.256.8
c=0c=054.554.454.5
c=0.5c=0.567.150.558.8
Validation-selected constant67.550.859.1
Learned gate68.150.559.3
Correspondence oracle70.554.462.4

These constants read the same trained branches; selecting their own exposure gives 59.4% Avg. Tuned constants achieve similar aggregate accuracy alongside useful input-dependent weighting.

We report these controls in Secs. 5.3 and E.3–E.4 (Tables 4, 26–27).

W6/Q2. General Purpose exposure and judgment interventions

To test how judgments guide prediction, we evaluated an additional variant that judges correspondence before classification in the same conversation. We retained all 4,000 Mixed classification instructions (1,923 auditory). Both conditions use the same ground-truth judgment as training context and differ only in judgment-token loss. Results cover two models and four clip datasets, averaged over three seeds:

MeasureJudgment loss onJudgment loss off
Judgment AUROC0.930.81
Image-following increase, “no”→“yes” (pp)48.745.2
Avg after own judgment71.668.3
Avg after correct judgment is supplied75.375.5

Supervision raises own-judgment Avg by 3.3pp. Supplying correct judgments yields similar Avg (75.3%/75.5%), linking the gain to judgment quality. Both conditions respond to forced judgments. Thus, this added design links correspondence supervision to classification gains at fixed classification exposure.

We have added this control in Sec. F.4 (Table 31).

W7. Statistical reliability and repeated fine-tuning

We assess the submitted +1.44pp gain across seven methods and five seeds. CIs resample paired seeds within method–dataset cells; dataset SDs average methods per seed first. All entries are pp:

Dataset50% gain95% CISD: Mixed / Mixed+BCI
Overall+1.440.32 to 2.50—
AV-MNIST+3.8−1.4 to 8.65.13 / 2.98
AVE+0.8−0.4 to 1.91.67 / 0.63
Kinetics-Sounds+0.2−0.4 to 0.80.65 / 0.66
CREMA-D+1.61.1 to 2.20.86 / 0.77
RAVDESS+0.7−0.6 to 2.12.01 / 1.85

Mean Avg gains are positive on all five datasets; the 5%-exposure gain is +5.8pp (CI: 4.9 to 6.5). Resampling cells and seeds gives a 50%-exposure CI of −0.07 to 3.40.

We report three fine-tuning repetitions per recipe on four clip datasets. Differences are Mixed+BCI minus Mixed:

ModelAvg difference ± s.e. (pp)Judgment balanced-accuracy gain (pp)
Qwen2.5−2.1±0.8+6.2
Qwen3+0.3±0.5+6.2

Both models improve four-dataset mean judgment balanced accuracy in all three repetitions (+6.2pp). W6 also improves classification in 20/24 model–dataset–seed comparisons and 7/8 model–dataset means (+3.3pp; item-bootstrap 95% CI: 2.5 to 4.2). These results support repeatable judgment gains and classification benefits in most conditions when judgments guide prediction.

We report reliability in Secs. 5.3, E.7, F.1 and F.4 (Tables 4, 29 and 31).

Dear Reviewer xGVH,

Thank you again for your constructive comments.

To address your concerns about the theoretical grounding and source of the gains, we clarified the connection to BCI and added controls separating the contributions of classification components, correspondence supervision, and learned weighting. We would welcome your feedback on whether these analyses address your concerns.

If our response has resolved the issues raised, we would sincerely appreciate your consideration in your final evaluation.

Best regards, The Authors

Reviewer_i9CyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies the limitation of forced fusion, which is when unrelated modalities distract a multimodal model from the actual target. The authors explore various ways of remedying this scenario with several empirical and theoretical analyses.

Critical strengths
  1. The authors identify an understudied but crucial scenario in multimodal fusion which is that the input modalities do not corroborate or correspond to each other. This can be potentially a significant failure mode of such models. Therefore, uncovering it and studying it as a mainstream problem setting is necessary.

  2. As a result of studying the above setting, the authors reveal that multimodal models often try to force-fuse input modalities even if they are not matching, leading to harmed target performance.

  3. The authors perform several interesting experiments which clarifies the role of diverse training with respect to diverse training with explicit correspondence supervision. It uncovers an important phenomenon that correctly identifying correspondence between modalities is a distinct problem from performing well in the target task.

Critical weaknesses
  1. In the tasks considered here, the target modality, i.e., audio, is known. What about scenarios where the target modality is not known? What if the downstream task could depend on any one of the available modalities and not on any fixed known one, i.e., the true target modality changes depending on the input? The proposed prediction rule (equation 6) assumes that the target modality is known.

  2. The contribution to mean target-accuracy gains from explicit correspondence supervision through BCI over simple mixed training are marginal on average (less than 1.5%). Given this, it is important to know the error bars / standard deviation in these regards.

  3. Mixed and Mixed + BCI cause significant hits to Matched accuracies. However, Matched and Mismatched cases are assigned equal weight while averaging while no comment on their actual distribution is provided. It is advisable that the authors look at their actual frequencies of occurrence in real-world scenarios of multimodal inference and assign weights on a case-by-case basis depending on the likelihood of occurrence of Matched / Mismatched instances of the particular scenario.

  4. Jointly considering the results from Tables 2 and 3, it is implied that higher correspondence judgement capability need not translate to better target accuracy. Given that, is it not better to have a design where correspondence predictions and the actual task are independently trained as separate modules (the latter taking into account information from the former)?

  5. The work is strongly reminiscent of ideas that have been studied for a long time in the modality collapse literature. Specifically, the idea that a particular modality can dominate over a multimodal model [a] and its deleterious effects are maximally visible when that modality goes missing during inference [b] is a well studied problem. The scenario dealt with in this paper has close resemblance to the missing modality setting, where instead of a modality going entirely missing, an incorrect (non-corresponding) input is provided. In both scenarios, having a unimodal model is better than a multimodal model that exhibits the said pathologies. The approach of training with modality dropout [c] to deal with missing modalities is in spirit similar to the Mixed training strategy discussed here. Due to these close similarities, a detailed discussion on the relationship of this work with modality collapse and missing modality literature is necessary.

References:

[a] Javaloy et al., "Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization", ICML 2022.
[b] Chaudhuri et al., "A Closer Look at Multimodal Representation Collapse", ICML 2025.
[c] Neverova et al., "ModDrop: Adaptive Multi-Modal Gesture Recognition", T-PAMI 2016.

Questions for authors

Refer to the Weaknesses section.

3 comments

Response to Reviewer i9Cy part[1/2]

We thank the reviewer for the detailed and constructive comments, and we respond to each point below.

W1. Known, changing, and unobserved target modalities

Eq. (6) describes selective fusion when the task or query specifies the target modality. We test the rule with a different queried modality, then extend it to infer the target from context when the target varies across inputs.

For a visual query, we retained the audio-trained module's gate and fused branch and used its Eq. (14) visual classifier as the fallback, without retraining. At 50% exposure, entries give matched/mismatched accuracy (%) for each queried modality:

QueryMatched-onlyMixedMixed+BCI
Visual69.8/47.765.4/17.068.5/66.6
Audio69.8/22.565.4/47.462.7/52.7

These results show that the same gate and fused branch can support a visual target by changing the fallback branch.

We then extended the rule to infer the target from the input by adding a learnable gate ww:

p(y∣x)=cpX(y∣x)+(1−c)[wpt(y∣xt)+(1−w)pa(y∣xa)].p(y\mid x)=c p_X(y\mid x)+(1-c) [w p_t(y\mid x_t)+(1-w) p_a(y\mid x_a)].

Setting w=1w=1 or 00 recovers the known-target rule. We randomly assigned audio or visual targets, indicated by an image marker, and learned ww from classification loss alone (three seeds).

ReadoutAlways audio, w=1w=1Learned wwGround-truth ww
Mat/Mis64.5/34.267.3/58.864.9/60.3

The learned ww selected the intended modality on 94% of mismatched pairs and raised mismatched accuracy from 34.2% with a fixed audio fallback to 58.8%, close to 60.3% with ground-truth ww. These results support extending the proposed fusion principle to targets that vary across inputs and are inferred from observed context: cc determines whether to fuse, while ww selects the appropriate unimodal fallback.

We have added the extension in Sec. 4.2 and its tests in Sec. D.4 (Table 20).

W2. Effect size, standard deviations, and exposure

We assess the reliability of the Mixed+BCI gains through variability across random seeds and examine how these gains depend on training exposure.

We report Avg SDs (pp). Task-specific results use five seeds. General Purpose SDs average both models per repetition, using three Mixed and four Mixed+BCI repetitions:

DatasetTask-specific: Mixed / Mixed+BCIGeneral Purpose: Mixed / Mixed+BCI
AV-MNIST5.13 / 2.980.01 / 0.01
AVE1.67 / 0.630.96 / 0.53
Kinetics-Sounds0.65 / 0.661.04 / 0.41
CREMA-D0.86 / 0.771.21 / 1.21
RAVDESS2.01 / 1.851.08 / 2.32

Across both model families, Mixed+BCI shows low standard deviations on most datasets, supporting stable performance across training repetitions.

To examine how the gains depend on training exposure, we summarize the numerical results from Fig. 5(a), where Mixed and Mixed+BCI use the same reassignment probability:

Training exposure1%5%10%25%50% (submitted)
Avg gain, pp+4.6+5.8+4.7+3.3+1.44
Mixed+BCI gate AUROC0.770.790.790.800.80

Mixed+BCI achieves a gate AUROC of 0.77 with only 1% exposure and 0.79 at 5%, where its Avg advantage over Mixed peaks at 5.8pp. With further exposure, gate AUROC levels off near 0.80 while the accuracy advantage decreases to 1.44pp at 50%. Thus, among the tested exposures, Mixed+BCI provides its largest additional predictive benefit when mismatched training pairs are limited, while maintaining strong correspondence discrimination at higher exposure.

We report variability in Secs. 5.3, E.7 and F.1 (Tables 4 and 29), and clarify Fig. 5's interpretation in Sec. 5.2.

W3. Matched accuracy and deployment prevalence

The relative weighting of matched and mismatched inputs should reflect how often each occurs in the intended application. Accordingly, we evaluate performance at test mismatch rates informed by real-world observations, with training exposure selected for each rate.

AudioScope reports that 25.9% of 10,000 unfiltered five-second clips contain only off-screen sounds (Tzinis et al., 2020). Using this reported rate as a reference, we evaluate a test mismatch rate of 25% and a comparatively low rate of 5%.

We next evaluated A(q)=(1−q)Amat+qAmisA(q)=(1-q)A_{\rm mat}+qA_{\rm mis}. Each Mixed strategy selected exposure per method–dataset using validation A(q)A(q) with different-class partners. Test evaluation used independent partners, including same-class pairs:

Test prevalence qqMatched-onlyMixedMixed+BCI
0.0567.670.371.0
0.2558.963.865.5

Mixed+BCI exceeded Mixed in weighted accuracy by +0.7pp at 5% test mismatch prevalence and +1.7pp at 25%. Its matched accuracy was 71.2–72.3%, versus 69.8% for Matched-only. Thus, with training exposure selected for each test prevalence, Mixed+BCI improves weighted accuracy over Mixed while maintaining matched accuracy above the Matched-only baseline.

We have added this prevalence analysis in Sec. E.2 (Table 24).

Response to Reviewer i9Cy part[2/2]

W4. Separately trained correspondence and task modules

We test whether a separately trained correspondence module can guide prediction and whether separation improves accuracy.

First, we compare adaptation of a frozen base with joint optimization under Matched-only and Mixed classification training. Separate adaptation freezes the trained base encoders and fused predictor while learning the added encoders, unimodal heads, and gate; joint training optimizes the base and added module together. Both supervise correspondence with matched and mismatched pairs. Entries are AUROC / Avg:

Classification trainingSeparate adaptationJoint training
Matched-only0.76 / 53.90.79 / 57.6
Mixed (50% exposure)0.77 / 55.90.80 / 57.7

Joint training improved Avg by +3.7pp under Matched-only and +1.8pp under Mixed classification training. Separate adaptation is feasible; joint optimization improves prediction and correspondence discrimination in these controls.

To isolate a separate judge's utility, we held Qwen3's Matched-only adapter fixed and used each judge to select between its audio-visual and audio-only answers:

JudgePretrainedMixedMixed+BCI
Correspondence AUROC0.770.880.94
Routed adapter Avg73.876.077.5

Mixed+BCI achieved the highest correspondence AUROC (0.94) and the highest Avg (77.5%) with the fixed Matched-only predictor, compared with 73.8% and 76.0% using the Pretrained and Mixed judges. Thus, Mixed+BCI is also effective as a separately trained judgment module.

We additionally combined the Mixed adapter with the Mixed+BCI judge, obtaining 78.4% Avg across five datasets. The Mixed+BCI adapter, trained on both classification and correspondence, achieved 80.0% Avg. Thus, the adapter trained jointly on classification and correspondence achieved higher accuracy than the separately combined prediction and judgment modules in this comparison.

We have added these comparisons in Secs. E.5 and F.3 (Tables 27 and 30).

W5. Modality collapse, missing modalities, and unimodal prediction

Modality imbalance can contribute to mismatch failures. The revised Introduction relates our setting to modality collapse (Javaloy et al., 2022; Chaudhuri et al., 2025) and robustness to missing inputs through ModDrop (Neverova et al., 2015). We test whether preserving modality information or training with missing inputs also handles valid but unrelated evidence.

First, we compared mismatch training for Concat with two controls under Matched-only training: adding unimodal classification losses and additionally applying OGM-style modulation (Peng et al., 2022), which adaptively scales encoder gradients to balance learning across modalities.

Concat trainingMismatched accuracy (%)
Matched-only25.2
Matched-only + unimodal classification losses25.1
Matched-only + unimodal losses + OGM-style modulation26.0
Mixed, 50%50.9
Mixed+BCI, 50%53.0

These preservation controls left mismatched accuracy near 25%; mismatch training produced the larger recovery. Preserving modality information alone did not resolve this interference.

We then compared missing and unrelated inputs. A modality-dropout control zeroed visual inputs in 50% of training and was tested with zeroed inputs or independent visual partners:

TrainingZeroed visualIndependent visual partner
Modality dropout51.331.7
Mixed, 50%49.248.6
Mixed+BCI, 50%53.153.1

The dropout model handled absence better than unrelated content. Both training schemes seek robustness to changes in auxiliary input, but missing-input robustness does not fully transfer to valid non-corresponding evidence.

Finally, following your suggestion, we evaluated unimodal baselines on the same mismatched inputs:

Target labelsMatched-only fusion (%)Unimodal reference (%)
Audio22.557.9
Visual47.768.9

The lower accuracy of Matched-only fusion than the unimodal baselines shows that non-corresponding inputs can interfere with prediction. Under the prevalence-based evaluation in W3, with training exposure selected for each test mismatch rate, Mixed+BCI improves weighted accuracy over audio-only prediction by +12.9pp at 5% and +7.4pp at 25%. These results favor learning correspondence to use auxiliary information selectively over discarding the auxiliary modality entirely: selective fusion retains the benefits of multimodal information and achieves higher overall accuracy in these settings.

We have added these controls in Secs. E.1 and E.6 (Tables 24 and 28).

Dear Reviewer i9Cy,

Thank you again for your thoughtful feedback.

To address your questions about target flexibility and practical robustness, we added visual- and context-dependent target experiments, evaluations at different mismatch rates, and variability analyses. We would appreciate your thoughts on these additions when you have a chance, and would be happy to clarify any remaining questions.

If our response has resolved the issues raised, we would sincerely appreciate your consideration in your final evaluation.

Best regards, The Authors

Comments on the paper

1 comment

General Response

We thank the reviewers for their careful and constructive feedback. The revised manuscript addresses the shared concerns through controlled comparisons, clearer theoretical assumptions, and analyses of how correspondence affects prediction. All revisions in the revised manuscript are shown in red text.

Establishing harmful interference and the value of selective fusion. We compare multimodal predictions with standalone unimodal models on the same target inputs and add the no-description reference with explicit answer parsing. These controls establish interference beyond the loss of useful auxiliary support. Evaluation at different test mismatch prevalences shows how selecting training exposure can preserve matched performance while retaining useful auxiliary information. Modality-balancing and missing-input controls further distinguish robustness to missing evidence from robustness to valid but unrelated evidence.

Separating the contributions of classification components and correspondence supervision. We hold architecture and classification losses fixed to isolate supervision, and fix branch outputs to isolate the effect of the gate. The classification components improve prediction at high training exposure, while correspondence supervision provides an independent predictive gain at low exposure and strengthens correspondence discrimination. The weight intervention shows useful prediction changes driven by correspondence. We clarify this interpretation of the existing exposure and supervision figures and add the requested method–dataset breakdown and simpler-gate comparisons.

Connecting the theory to the learned correspondence. We state the source-model and reference-predictor assumptions under which oracle correspondence gives optimal hard routing, and empirically evaluate the sufficient improvement condition. Label-collision controls and independent, same-class, and within-clip comparisons distinguish construction labels, category agreement, and event visibility. Together with posterior-reference comparisons, these analyses support synthetic posterior tracking and useful category-level correspondence in real data. An additional fallback gate extends selective fusion to changing targets inferred from observed context.

Testing repeatability and the use of judgments in General Purpose prediction. Repeated adaptation establishes consistent improvements in correspondence judgment. We separately evaluate an additional design that places a judgment before classification, retaining the same classification exposure and training context while varying judgment supervision. Its improvements across most tested conditions, together with interventions on the supplied judgment, connect better correspondence estimation to prediction in this design. Separate-judge comparisons also show that correspondence can guide a fixed predictor.

The individual responses contain the evidence needed to assess each question and identify the corresponding locations in the revised manuscript. Existing experiments are retained, with additional interpretation where sufficient.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.