acceptodds
Under review as a conference paper at ICLR 2027

From Capability to Utility: Reliability Configuration Calibration for Multimodal Fusion

Abstract

Existing research on modality imbalance mainly aims to improve the learning and classification capabilities of individual modalities. However, stronger unimodal capability does not directly ensure that correct evidence is used in fusion. We identify and characterize the capability-to-utility gap (C2U gap): when one modality predicts correctly and the other incorrectly, the fusion model may still make an incorrect prediction. By examining the joint correctness patterns of unimodal predictions, we find a mismatch in reliability configurations between in-sample and held-out data. In particular, critical configurations in which the dominant modality is wrong and the other modality is correct are underrepresented in in-sample predictions. Based on this diagnosis, we propose Reliability Configuration Calibration (RCC). RCC uses configuration deficits between in-sample and held-out calibration data to determine the directions and probabilities of feature replacement. Reliability-weighted counterfactual feature replacement provides additional fusion supervision for critical cases, while unimodal anchoring and adoption preservation help maintain unimodal capability and limit interference with previously reliable configurations. Extensive experiments show improved fusion accuracy across six benchmarks and higher visual conditional utility on three audio–visual datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Modality Imbalance, Configuration Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xKJwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why accurate unimodal predictors do not necessarily translate into effective multimodal fusion. It distinguishes unimodal capability from conditional utility: whether the fused prediction remains correct when only one modality is correct. The proposed Reliability Configuration Calibration (RCC) compares the joint correctness configurations of the modalities on training data and an internal held-out split, then uses the observed deficits to set directional feature-replacement probabilities.

During training, RCC constructs counterfactual feature pairs by replacing one modality with a feature from a different class while preserving the other modality and the original sample label. Reliability-based weighting and auxiliary supervision guide this intervention. A calibration stage estimates the probabilities; a fresh model is then trained on the full training set using the resulting fixed probabilities. The authors report improvements on five bimodal benchmarks and a trimodal sentiment task, supported by ablations, common-support utility comparisons, and experiments with shared frozen encoders. The inference architecture is unchanged.

Critical strengths
  1. A useful diagnostic perspective with meaningful controls. The capability-to-utility distinction addresses a concrete failure mode: fusion can overlook the correct modality when the modalities disagree. The paper also recognizes that conditional metrics can change when different models define different evaluation subsets. The pairwise common-support comparisons in Table 10 and the shared frozen-encoder experiments in Table 12 strengthen the evidence that RCC changes fusion behavior. With encoder features held fixed, fusion accuracy increases from 80.94 to 82.42 on CREMA-D and from 84.98 to 87.03 on AVSBench (Table 12). These controls make the proposed explanation more informative than overall accuracy gains alone.

  2. Consistent reported gains across several tasks. Table 8 reports the highest mean fusion accuracy among the compared methods on all five bimodal datasets, with results from three runs, and the paper extends the method to trimodal sentiment analysis. RCC also preserves the inference architecture and parameter count (Table 14). This breadth is encouraging for a training intervention, although the additional calibration stage adds training cost and the model-selection concern below must be resolved before drawing firm conclusions about generalization.

Critical weaknesses
  1. The released runner selects checkpoints using the test split. In the released CREMA-D and AVSBench runner, stage_command() evaluates test_loader after each epoch and saves best.pth and summary.json when test fusion accuracy improves during the final stage, after the configured selection threshold (rcc_guard.py, lines 403–414). The dataset loaders confirm that this is the test split, rather than an internal validation split (dataset/cremad.py and dataset/avsbench.py).

If the paper's reported results follow this rule, the test set also serves as a model-selection set, which can make the reported performance optimistic. It is currently unclear whether the same selection procedure produced the main tables, how it was applied to the baselines, and whether other datasets use an independent validation protocol. This concern is about checkpoint selection; RCC's probability calibration itself uses a held-out subset of the original training data. A test-independent selection protocol is needed to establish the strength of the reported gains.

  1. The central calibration and restart design need more direct evidence. The method estimates directional probabilities using one internal 90/10 split per run, updates them at epochs 30, 40, and 50, and transfers the last estimates to a freshly initialized model trained on all training data. The final-stage probabilities remain fixed (Appendix B and the released calibration runner). The paper does not directly characterize how sensitive the final estimates are to the held-out split or the calibration checkpoint, or whether they remain suitable after reinitialization and training on the full dataset.

Table 6 provides useful comparisons with validation-selected fixed strategies and event-count-matched allocations. However, these comparisons do not directly characterize sensitivity to the scaling coefficient or isolate the value of restarting rather than continuing training. For example, the CREMA-D scaling coefficient of 0.35 is a hyperparameter, but the stated parameter sweeps do not provide enough detail to assess its selection or sensitivity. Final-accuracy variation across seeds also does not reveal how much the estimated probabilities vary. Focused stability and scaling-sensitivity results, together with a comparison against continued training under a comparable budget, would clarify the value of the specific two-stage protocol relative to its extra training cost.

Questions for authors
  1. Checkpoint selection. Which rule produced the results for each dataset and baseline? If the released test-based selection rule was used, please report results with an independent validation set or a prespecified checkpoint and assess whether the main comparisons remain unchanged.

  2. Split stability. How much do the estimated directional probabilities change across different random 10% held-out splits? A focused comparison using 5%, 10%, and 20% held-out data on a representative benchmark would help assess whether the calibration depends strongly on this choice.

  3. Training-time stability. What are the probability estimates at epochs 30, 40, and 50, and why is the last estimate representative? Please distinguish the stability of the calibrated probabilities from the stability of the final test accuracy.

  4. Transfer between stages. Does a probability calibrated on the first-stage model remain suitable after reinitialization and training on the full dataset? With comparable training budgets and the same calibrated probabilities, how does restarting compare with continuing from the calibration-stage checkpoint?

  5. Probability mapping and scaling. How were the scaling coefficients, including 0.35 for CREMA-D, selected? Please specify the search range, selection data, and criterion, and provide a focused sensitivity analysis that complements the fixed-strategy comparisons in Table 6.

  6. Necessity of adaptive calibration. To what extent do the improvements come from gap-driven probability calibration rather than feature replacement and additional supervision themselves? Although Table 6 includes fixed-strategy comparisons, a more controlled ablation using matched replacement rates and training budgets would help isolate the contribution of adaptive calibration. In particular, does RCC consistently outperform simple fixed-probability replacement under otherwise identical settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xKJw (Part 1/2, Q1–Q3)

We thank the reviewer #xKJw for the suggestions on calibration stability and experimental controls. Appendix K.1 now examines sensitivity to calibration splits and scheduled update epochs, while Appendix J.2 provides controlled comparisons of replacement probabilities under the same supervision mechanisms and training budget. Checkpoint selection, restarting versus continuing training, and scaling-coefficient selection are addressed in the responses below.

Q1. Checkpoint selection.

A1: The original main-table results for RCC and all baselines use checkpoints selected by test fusion accuracy, following public implementations such as InfoReg and IBML; some datasets also lack standardized validation splits. A shared protocol does not eliminate the effects of test-based selection. We therefore compare RCC and IBML on AVSBench using its independent validation split for checkpoint selection, alongside test-selected results. Both selection rules use fusion accuracy, and the table reports the corresponding test accuracies. Validation-based selection lowers both methods’ test accuracy, but RCC retains its advantage over IBML in this comparison.

ModelAcc. (val-selected)Acc. (test-selected)
IBML91.8992.97
RCC92.4493.35

Q2. Split stability.

A2: On CREMA-D, we compare 5%, 10%, and 20% calibration held-out ratios, fixing the model seed and using three random splits per ratio. The table reports means ± standard deviations across splits for the second-stage replacement probabilities and final test accuracy. At the default 10% ratio, the standard deviation of pa→vp_{a\to v} is 0.0097. Across all nine runs, calibration selects the same direction for additional supervision, despite variation in probability magnitude and final accuracy. This supports consistent directional guidance across the tested ratios and splits.

Held-out ratiopa→vp_{a\to v}pv→ap_{v\to a}Acc.
5%0.1000 ± 0.00760.0000 ± 0.000082.97 ± 0.31
10%0.1195 ± 0.00970.0000 ± 0.000084.24 ± 0.51
20%0.1140 ± 0.00410.0000 ± 0.000082.30 ± 0.39

Q3. Training-time stability.

A3: Algorithm 1 updates directional probabilities at calibration epochs 30, 40, and 50, then uses the last estimate for second-stage training. The epoch-50 choice uses later-stage configuration statistics and is prescribed by the schedule, not selected using final test accuracy.

On CREMA-D with the default 10% held-out ratio, the final estimate retains the same direction and remains close to earlier estimates, supporting the last scheduled update as a representative late-stage estimate. This table examines probability variation during calibration; Q2 examines variation in final test accuracy across splits when using the last probability estimate.

Calibration epochpa→vp_{a\to v}pv→ap_{v\to a}
300.1230 ± 0.01470.0000 ± 0.0000
400.1115 ± 0.00900.0000 ± 0.0000
500.1195 ± 0.00970.0000 ± 0.0000
AuthorsOpenReview ↗

Response to Reviewer xKJw (Part 2/2, Q4–Q6)

Q4. Transfer between stages.

A4: On CREMA-D, we compare fresh initialization with continuation from the calibration checkpoint using the same calibrated probabilities. Both strategies use the full training set, identical second-stage schedules and update budgets, and evaluation at epoch 100 of the second stage. Continuation retains the calibration model’s parameters and SGD momentum.

Restarting achieves higher accuracy, supporting the use of first-stage calibrated probabilities after reinitialization in this setting. These probabilities remain fixed during second-stage training, while sample reliability weights are recomputed from the current model’s predictions, allowing supervision weights to adapt to its state.

StrategyAcc.
Restart (RCC)84.66
Continue82.47

Q5. Probability mapping and scaling.

A5: On CREMA-D, we searched α∈0.15,0.25,0.35\alpha\in \\{0.15,0.25,0.35 \\} and selected α=0.35\alpha=0.35 using the highest test fusion accuracy in seed 0 runs. The table varies only this scaling coefficient, keeping all other hyperparameters at the original manuscript’s settings.

Accuracy increases over the tested range, with similar results for 0.25 and 0.35. This isolates the effect of probability scaling while retaining RCC’s configuration-gap rule and supervision terms, complementing the fixed-policy comparisons in Table 6.

Scaling coefficient α\alphaAcc.
0.1583.33
0.2584.41
0.3584.68

Q6. Necessity of adaptive calibration.

A6: Table 6 shows that fixed-probability replacement can itself improve performance. To isolate the contribution of probability calibration, we vary only replacement probabilities on CREMA-D, keeping the replacement mechanism, reliability weighting, auxiliary losses, initialization, and second-stage training budget unchanged. The symmetric rate-matched control preserves RCC’s sum of directional probabilities and divides it equally.

RCC outperforms both the rate-matched control and the same-direction fixed policy. This supports an additional benefit from configuration-guided probability allocation under the tested controls. It does not establish superiority over every fixed policy or random seed.

Policypa→vp_{a\to v}pv→ap_{v\to a}Acc.
Calibrated RCC0.1172083.66
Symmetric, rate-matched0.05860.058681.05
Fixed a→va\to v0.1000081.99
Fixed symmetric0.05000.050080.51
Fixed symmetric0.10000.100082.53

Dear Reviewer #xKJw,

Thank you again for your review. We have responded to your concerns point by point and added experiments and clarifications supporting our central contribution: diagnosing the capability-to-utility gap and improving fusion’s use of correct unimodal evidence through RCC.

We would appreciate your reassessment in light of our responses and revisions. Could you please let us know whether our responses address your concerns? We would be happy to discuss any remaining issues and provide further clarification.

Best regards, The Authors

Reviewer_9zRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates what the authors call the “capability-to-utility (C2U) gap,” where multimodal models fail to generate correct predictions in settings where one unimodal model gives an accurate prediction and the other does not. In contrast to prior approaches using optimization or reweighting methods to improve multimodal performance, the proposed method Reliability Configuration Calibration (RCC) is a two-stage, metric-driven approach to feature replacement. The authors run experiments on several datasets in audio-visual and audio-visual-text classification and include an ablation study on loss components (Section 4.5) and replacement strategies (Section 4.6). Broadly, the work is interesting and the experiment results seem to support the authors’ core argument. However, the limited discussion around and positioning against modality interaction quantification, exploration of the validity of the proposed metric and method on unique- (e.g. labels generated solely from one modality) and synergistic-information tasks (e.g. datasets like NLVR2), and a more detailed discussions of the tradeoffs in two-stage training and other design decisions limit this work. As it stands, I recommend weak rejection for this work.

Critical strengths
  • Problem setting and approach. The problem setting the paper investigates is a known issue in multimodal learning, and the perspective the authors take seems to be relatively novel.
  • Experimental results. The classification accuracy results in Tables 1 and 2 (and Table 8) support the empirical utility of the proposed approach.
Critical weaknesses
  • Distinguishing proposed method from modality interaction quantification. The authors do a good job of surfacing the literature on improving joint modality learning (e.g. calibration, reweighting) in the Introduction and Related Works section. However, the manuscript does not discuss modality interaction quantification (see Section 8 of [1]) at length. This reads as an oversight: the author’s problem setting seems directly related to the interactions of modalities in multimodal learning. A discussion on why information theoretic approaches like partial information decomposition (PID)-based ones (e.g. see [2]) are insufficient and require the introduction of the utility metric is necessary.
  • Validity of measure and method. The authors use six datasets (CREMA-D, Kinetics-Sounds, AVSBench, VGGSound50, UPMC Food-101, and CMU-MOSI) for their experiments, and Table 1 demonstrates how multimodal models outperform unimodal ones across several methods on these datasets. While the experimental results support the authors’ main arguments, the validity of both the C2U gap metric and RCC method should also be tested on where all the task-relevant information is in one modality [3] or requires both modalities like in NLVR2 [4]. If evaluating their metric and method on VLM settings in [3,4] are too much, the authors should consider synthetic experiments (I suggest looking at the synthetic experiments in [2] for inspiration).
  • Performance metric decisions and limitations discussions. The proposed method is mostly evaluated on classification accuracy (binary or class-wise), F1 scores, and the authors’ proposed metric in conditional utilities (Table 1 and Table 2). The authors should consider including additional performance metrics (particularly around calibration) and have a discussion on the tradeoffs of their model as appropriate (for example, are there comparator methods that are train-free or one-stage training? Are there calibration trade-offs involved?).

[1] Liang, Paul Pu, Amir Zadeh, and Louis-Philippe Morency. "Foundations & trends in multimodal machine learning: Principles, challenges, and open questions." ACM computing surveys 56.10 (2024): 1-42.

[2] Liang, Paul Pu, et al. "Quantifying & modeling multimodal interactions: An information decomposition framework." Advances in Neural Information Processing Systems 36 (2023): 27351-27393.

[3] Zhang, Yuhui, et al. "Why are visually-grounded language models bad at image classification?." Advances in Neural Information Processing Systems 37 (2024): 51727-51753.

[4] Suhr, Alane, et al. "A corpus for reasoning about natural language grounded in photographs." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019.

Questions for authors

Please refer to the Critical Weaknesses section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9zRM

We thank the reviewer #9zRM for the suggestions. Section 2 clarifies how conditional utility relates to PID. Appendices M.1 and M.2 evaluate the metric and objective on synthetic tasks; Section 5 discusses limitations and future work. Appendix L adds predictive calibration and probability-quality evaluations, complementing the cost and training-budget analyses in Appendix I.

W1. Distinguishing the proposed method from modality interaction quantification.

A1: PID decomposes information about task labels or model predictions into redundant, unique, and synergistic components, and can characterize interactions captured by a model. Conditional utility measures fusion accuracy when one unimodal predictor is correct and the other is wrong. Here, correctness refers to a specific predictor’s agreement with the ground-truth label; it is not equivalent to PID-unique information. PID components do not directly report this conditional accuracy.

Our contribution links this diagnosis to configuration-guided supervision. Train–held-out differences in joint correctness configurations identify underrepresented single-correct cases. RCC uses these deficits to determine replacement directions and probabilities, then adds reliability-weighted fusion supervision where correct unimodal evidence remains available.

Table 8 provides empirical support: with shared frozen encoders and fixed reference configurations, RCC improves both conditional utilities and overall accuracy on CREMA-D and AVSBench. Thus, supervision allocation can improve fusion’s use of existing representations, supporting separate diagnoses of unimodal capability and conditional utility. C2U analysis complements PID by connecting a specific fusion failure to a targeted training intervention. Section 2 now clarifies this relationship.

W2. Validity of the metric and method.

A2: We evaluate three synthetic tasks: Unique A, where label information comes only from A; Mixed A, with biased-XOR labels; and Pure synergy, which requires both modalities and has an optimal unimodal accuracy of 50%.

For the metric, full RCC’s A branch is perfectly accurate on Unique A, yet fusion fails on some A-correct/B-incorrect samples. Conditional utility identifies this failure to convert existing unimodal capability into correct fusion decisions. The reverse configuration is empty, so we report UBU_B as N/A. In Pure synergy, a unimodal prediction can be correct by chance without independent label information. Conditional utility describes fusion performance within correctness configurations; it does not distinguish whether success relies on unique or synergistic information and should be interpreted alongside the task’s information structure.

TaskA Acc.B Acc.UAU_AUBU_B
Unique A100.0049.4694.69N/A
Mixed A74.2749.7683.3474.77
Pure synergy50.5449.2277.7078.89

For the method, we compare full RCC with objective ablations; Uni denotes auxiliary unimodal supervision. In Unique A, full RCC improves over Plain + Uni but remains below Plain. It also underperforms Plain on Mixed A and Pure synergy, where removing Uni improves performance. These results expose limits of the full objective: unimodal supervision and its interaction with other losses can impair fusion. RCC’s benefit in converting capability to utility depends on how its supervision objectives fit the task’s information structure.

ObjectiveUnique A Acc.Mixed A Acc.Pure synergy Acc.
Plain100.00100.00100.00
Plain + Uni90.5380.7678.27
RCC w/o Uni100.0099.3297.12
Full RCC97.3183.9485.35

W3. Performance metrics and discussion of limitations.

A3: We evaluate ECE, NLL, and the Brier score on CREMA-D and AVSBench. ECE assesses agreement between confidence and empirical accuracy, while NLL and Brier measure overall probabilistic prediction quality. The table below shows that RCC improves accuracy and reduces all three metrics relative to single-stage Plain on both datasets. RCC targets configuration-related training supervision rather than directly optimizing predictive probability calibration.

Table 18 reports computational costs on CREMA-D: Plain trains for 100 epochs, while RCC uses 60 calibration epochs followed by 100 training epochs. Both have 23M parameters; full-test-set inference takes 7.13 and 7.15 seconds, respectively. RCC preserves the fusion architecture and uses neither replacement nor reliability weighting at inference. Its trade-off is additional training cost with essentially unchanged deployment cost; it does not offer the cost advantage of training-free approaches.

DatasetsModelAcc. ↑ECE ↓NLL ↓Brier ↓
CREMA-DPlain79.9314.691.0370.3363
RCC84.775.310.5240.2573
AVSBenchPlain91.085.810.4260.1396
RCC93.715.280.4130.1356

Dear Reviewer #9zRM,

Thank you again for your review. We have responded to your concerns point by point and added experiments and clarifications supporting our central contribution: diagnosing the capability-to-utility gap and improving fusion’s use of correct unimodal evidence through RCC.

We would appreciate your reassessment in light of our responses and revisions. Could you please let us know whether our responses address your concerns? We would be happy to discuss any remaining issues and provide further clarification.

Best regards, The Authors

Reviewer_EfQZOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reliability Configuration Calibration (RCC) to address the Capability-to-Utility Gap in multimodal fusion, where correct unimodal predictions may not translate into correct fusion decisions. The authors identify mismatches in joint unimodal correctness configurations between training and held-out data. RCC uses these configuration gaps to guide counterfactual feature replacement, while reliability-weighted supervision encourages the fusion model to utilize reliable evidence. Unimodal anchoring and adoption preservation are introduced to maintain individual modality performance and mitigate directional bias. Experiments on six multimodal classification benchmarks demonstrate improved overall fusion accuracy and enhanced conditional utility in selected reliability configurations.

Critical strengths

1.The paper introduces the Capability to Utility Gap (C2U Gap), distinguishing unimodal predictive capability from effective evidence utilization in multimodal fusion. The analysis of joint prediction correctness provides a valuable perspective on modality imbalance, with clear motivation and logical method design. 2. The paper proposes a reliability configuration calibration strategy to guide counterfactual feature replacement. Reliability weighted supervision improves evidence utilization, while unimodal anchoring and adoption preservation help maintain predictive capability and reduce modality bias. Each component has a clear purpose. 3.The paper presents extensive experiments across six multimodal benchmarks, including ablation studies and comparisons of feature replacement strategies. Theoretical analysis, evaluations on common reliability configurations, and computational efficiency analysis further support the effectiveness of the proposed method.

Critical weaknesses
  1. The paper identifies reliability configuration mismatch as a potential cause of the C2U Gap. However, this mismatch may also reflect unimodal generalization errors. Further controlled experiments are needed to clarify its causal relationship with fusion utility degradation.
  2. Feature replacement across different classes does not guarantee the intended reliability configurations. Although Appendix H provides diagnostic results, additional analysis of target configuration generation rates across datasets and their relationship with model performance would strengthen the justification for this strategy.
  3. RCC improves visual conditional utility but may reduce audio conditional utility compared with some baselines on CREMA-D and KS. Although this tradeoff is acknowledged, extending evaluations on identical samples to more datasets would provide a more comprehensive assessment of modality evidence utilization.
  4. RCC requires an additional calibration stage and uses fixed replacement probabilities during subsequent training. Although computational efficiency is reported, the sensitivity of calibration results to different data splits remains unclear. Further stability analysis and comparisons under equal total training budgets would strengthen the evaluation of its practical benefits. 5.The calibration rule lacks sufficient theoretical justification. Appendix A.4 acknowledges that the deficit normalization and probability mapping are heuristic design choices. Comparisons with simpler strategies based on unimodal accuracy gaps would help establish the necessity of joint reliability configurations. 6.The theoretical analysis mainly provides conditional risk comparisons for fixed fusion rules rather than guarantees for RCC training. The assumptions underlying the positive counterfactual risk advantage are not empirically verified. This limits the theoretical support for the claimed improvements in fusion utility.
Questions for authors

1.The reliability configuration mismatch may also result from unimodal generalization errors. Can the authors provide further experimental evidence to distinguish its specific contribution to the C2U Gap from generalization effects? 2.In Eq. (6), feature replacement across different classes does not guarantee the intended reliability configuration. What proportion of replaced samples satisfy the target configuration, and how does this proportion affect RCC performance? 3.How sensitive are the estimated configuration gaps and replacement probabilities to different calibration set sizes and random splits? Does RCC maintain its advantage over standard baselines under equal total training budgets? 4.Table 3 shows that improving visual conditional utility may reduce audio conditional utility. Does this tradeoff persist on common configuration subsets across other datasets? How can RCC mitigate this imbalance? 5.In Eq. (1), zeroing out modality features may introduce distribution shifts. Have the authors compared this reliability estimation strategy with independent unimodal classifiers? How do different estimation strategies affect configuration statistics and calibration performance? 6.Would a replacement policy based solely on unimodal accuracy gaps achieve similar performance to RCC? What additional benefit does modeling joint reliability configurations provide? 7.Appendix A.3 derives sufficient conditions for a positive counterfactual risk advantage. Have the authors empirically verified these conditions during training? How closely do the theoretical assumptions reflect the actual optimization process?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EfQZ (Part 1/3, Q1–Q2)

We thank the reviewer #EfQZ for the constructive suggestions. Section 4.7 and Appendix G.3 now present controlled studies of configuration-guided fusion supervision. Appendix H.2 examines target-configuration rates and reliability weighting, while Appendices K.1 and K.2 assess calibration stability and sensitivity to reliability estimation. Appendix I.2 reports comparisons under matched total training budgets, and Appendix A.5 provides empirical checks of the theoretical sufficient condition.

Q1. Reliability configuration mismatch may also arise from unimodal generalization errors.

A1: We agree that configuration mismatch can arise from unimodal generalization gaps, as acknowledged in the Introduction. Our further question is whether configuration-level supervision affects fusion even when unimodal predictions remain fixed. Table 8 shows improved fusion with shared frozen encoders; we further isolate the role of configuration supervision through a controlled experiment on CREMA-D.

We freeze the encoders and reference unimodal predictors and vary the training weight assigned to single-correct configurations, holding each class’s total training weight, within-class weighted unimodal accuracies, and training budget constant. As shown below, mean visual conditional utility and overall accuracy both increase from reduced coverage through the original distribution to increased coverage. With unimodal predictions and marginal accuracies unchanged, these results support an additional role for configuration-level supervision in translating existing unimodal capability into correct fusion decisions.

Training conditionSingle-correct weightUarefU_a^{ref}UvrefU_v^{ref}Acc.
Reduced coverage39.85 ± 4.7586.21 ± 3.0965.34 ± 11.3278.14 ± 1.69
Original distribution50.97 ± 2.5287.20 ± 3.2868.21 ± 11.7678.67 ± 1.67
Increased coverage63.56 ± 2.7087.81 ± 0.4670.85 ± 8.7280.18 ± 1.53

Q2. What proportion of cross-class replacements satisfies the target reliability configuration, and how does this affect RCC’s performance?

A2: Cross-class replacement does not guarantee the target configuration: the retained modality predicts the original label correctly while the replaced modality predicts it incorrectly. RCC therefore weights supervision by the retained modality’s support for that label (Eq. 7). We report the raw and reliability-weighted proportions of this joint event during training (Raw/Weighted). A paired ablation sets only the main replacement-loss reliability weights to 1 (Unweighted), keeping all other settings unchanged.

On both datasets, raw proportions remain nearly unchanged, while weighting increases the weighted share of target configurations in the direction that retains vision and improves accuracy. These results support reliability weighting as a way to concentrate supervision on target configurations, helping fusion use the retained correct evidence.

DatasetMain-loss weightinga→va\to v: Raw / Weightedv→av\to a: Raw / WeightedAcc.
CREMA-DUnweighted61.93 / 61.93N/A82.91
CREMA-DReliability-weighted61.89 / 80.84N/A84.18
AVSBenchUnweighted87.95 / 87.9599.41 / 99.4191.22
AVSBenchReliability-weighted87.85 / 96.3699.45 / 99.4592.62
AuthorsOpenReview ↗

Response to Reviewer EfQZ (Part 2/3, Q3–Q4)

Q3. How sensitive is calibration to held-out size and random splits? Do gains persist under matched total training budgets?

A3: On CREMA-D, we fix the model seed and use three random splits per held-out ratio, reporting means ± standard deviations across splits. All nine runs identify a positive gap for vision-correct/audio-incorrect configurations and a negative reverse gap; the mean positive gap ranges from 15.12 to 17.05 percentage points across ratios. Calibration consistently identifies the same direction for additional supervision across the tested sizes and splits.

Held-out ratiopa→vp_{a\to v}pv→ap_{v\to a}Acc.
5%0.1000 ± 0.00760.0000 ± 0.000082.97 ± 0.31
10%0.1195 ± 0.00970.0000 ± 0.000084.24 ± 0.51
20%0.1140 ± 0.00410.0000 ± 0.000082.30 ± 0.39

We then fix the default 10% calibration split and use three model seeds. Plain’s training budget is matched to either RCC’s total updates or its measured total training time, including calibration. RCC outperforms Plain under both controls, indicating that its gains in this setting cannot be explained solely by additional training budget.

MethodTraining budgetTest Acc. (%)
PlainMatched total updates79.93 ± 1.09
PlainMatched total training time79.70 ± 0.47
RCCCalibration + final training81.77 ± 1.03

Q4. Does the utility trade-off persist on common configurations across datasets, and how does RCC mitigate it?

A4: The trade-off is not inevitable. Table 13 shows that it persists on common configurations in CREMA-D. For AVSBench, we evaluate Plain and RCC on the intersections of their audio-only-correct and vision-only-correct configurations, ensuring identical samples within each direction. The comparison below shows that RCC improves mean conditional utility in both directions.

On CREMA-D and KS, additional supervision for vision-correct/audio-incorrect configurations may shift fusion toward vision and affect audio utility. This motivates adoption preservation (AP; Section 3.2.3), which adds reverse-direction replacement supervision weighted by the retained modality’s reliability. Table 5 shows that AP improves both conditional utilities on CREMA-D and KS when evaluated on each model’s own configurations. Our AVSBench ablation finds no further gain in mean utility. RCC supplements underrepresented configurations while using AP to mitigate interference with evidence utilization in the opposite direction; AP’s benefits depend on the setting.

ModelUa∩U_a^{\cap}(%)Uv∩U_v^{\cap} (%)
Plain88.86 ± 0.7193.25 ± 3.91
RCC89.87 ± 1.3294.42 ± 2.78
AuthorsOpenReview ↗

Response to Reviewer EfQZ (Part 3/3, Q5–Q7)

Q5. Could zero masking cause distribution shift? How do independent estimators affect configuration gaps and RCC performance?

A5: Equation (10) directly supervises the masked unimodal predictions in Eq. (1), so these input patterns are seen during training. We compare two alternatives: independent unimodal heads trained on the original model’s frozen encoders, and independently trained unimodal models with their own encoders and heads. Each estimator determines configuration statistics and replacement probabilities; downstream RCC training changes only these probabilities. We define configuration gaps as Δq=qheld−qtrain\Delta q=q_{\mathrm{held}}-q_{\mathrm{train}} , where positive values indicate insufficient training coverage.

Although gap magnitudes differ, all estimators identify the same underrepresented single-correct configurations on both datasets. The masked estimator also yields the highest mean RCC accuracy. These results support a configuration-mismatch diagnosis beyond zero masking and the practical use of the current model’s supervised unimodal predictions to guide fusion training, without separately training additional classifiers.

DatasetReliability estimatorΔq(A×V✓)Δq(A✓V×)Acc.
CREMA-DMasked (RCC)14.57 ± 2.68−17.63 ± 3.6984.67 ± 0.47
Independent heads18.31 ± 0.86−10.91 ± 1.2181.81 ± 1.28
Independent models15.07 ± 0.94−17.42 ± 2.6581.72 ± 0.94
AVSBenchMasked (RCC)8.99 ± 1.5116.88 ± 5.3392.85 ± 0.34
Independent heads9.18 ± 1.4623.69 ± 1.9491.58 ± 0.56
Independent models8.99 ± 1.7615.30 ± 5.7091.35 ± 0.62

Q6. Can replacement based on unimodal accuracy gaps match RCC? What do joint configurations add?

A6: On both datasets, RCC achieves higher mean accuracy than replacement based on unimodal accuracy gaps, including its rate-matched variant. We replace joint configuration gaps with each modality’s own training–held-out accuracy gap, keeping all other training settings unchanged. The rate-matched variant also matches RCC’s total expected replacement rate.

The additional information is how prediction errors overlap across modalities: increased errors in one modality may occur when the other remains correct or when both are wrong. Marginal accuracy gaps do not capture this distinction. Joint configurations identify underrepresented cases where correct unimodal predictions remain available to fusion, enabling more targeted allocation of replacement supervision. The advantage retained after rate matching supports the practical value of this information.

Replacement policyCREMA-DAVSBench
RCC (joint configurations)84.54 ± 1.2292.44 ± 0.51
Unimodal accuracy gap82.23 ± 1.4591.44 ± 0.81
Unimodal accuracy gap, rate-matched82.14 ± 0.0891.40 ± 0.87

Q7. Are Appendix A.3’s sufficient conditions supported during training, and how well do they reflect actual optimization?

A7: We examine training checkpoints on CREMA-D and AVSBench across three seeds per dataset, covering all directions with nonzero calibrated probabilities. At each checkpoint, we compare a prediction rule using only the replaced source modality with full fusion, retaining RCC’s trained features, reliability weights, replacement sampling, and batch-wise loss normalization. Using pilot-derived bounds and independent samples, we check the conditional-loss bounds and configuration-proportion threshold in Appendix A.3. Entries report passed/assessable checks based on point estimates; N/A indicates insufficient samples outside the target configuration for a complete check.

Of the 19 assessable cases, 18 satisfy the checked sufficient conditions. All 36 evaluated cases exhibit a positive counterfactual risk advantage, with each bootstrap 95% interval lying above zero. These results support the risk analysis at the examined training states: the counterfactual loss consistently favors full fusion, providing a risk-based rationale for RCC’s directional supervision to promote the use of correct evidence.

EpochCREMA-D a→va\to vAVSBench a→va\to vAVSBench v→av\to aPositive risk advantage
303/33/3N/A9/9
503/33/3N/A9/9
753/31/1N/A9/9
1002/3N/AN/A9/9
Total11/127/7N/A36/36

Dear Reviewer #EfQZ,

Thank you again for your review. We have responded to your concerns point by point and added experiments and clarifications supporting our central contribution: diagnosing the capability-to-utility gap and improving fusion’s use of correct unimodal evidence through RCC.

We would appreciate your reassessment in light of our responses and revisions. Could you please let us know whether our responses address your concerns? We would be happy to discuss any remaining issues and provide further clarification.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.