- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why accurate unimodal predictors do not necessarily translate into effective multimodal fusion. It distinguishes unimodal capability from conditional utility: whether the fused prediction remains correct when only one modality is correct. The proposed Reliability Configuration Calibration (RCC) compares the joint correctness configurations of the modalities on training data and an internal held-out split, then uses the observed deficits to set directional feature-replacement probabilities.
During training, RCC constructs counterfactual feature pairs by replacing one modality with a feature from a different class while preserving the other modality and the original sample label. Reliability-based weighting and auxiliary supervision guide this intervention. A calibration stage estimates the probabilities; a fresh model is then trained on the full training set using the resulting fixed probabilities. The authors report improvements on five bimodal benchmarks and a trimodal sentiment task, supported by ablations, common-support utility comparisons, and experiments with shared frozen encoders. The inference architecture is unchanged.
Critical strengths
-
A useful diagnostic perspective with meaningful controls. The capability-to-utility distinction addresses a concrete failure mode: fusion can overlook the correct modality when the modalities disagree. The paper also recognizes that conditional metrics can change when different models define different evaluation subsets. The pairwise common-support comparisons in Table 10 and the shared frozen-encoder experiments in Table 12 strengthen the evidence that RCC changes fusion behavior. With encoder features held fixed, fusion accuracy increases from 80.94 to 82.42 on CREMA-D and from 84.98 to 87.03 on AVSBench (Table 12). These controls make the proposed explanation more informative than overall accuracy gains alone.
-
Consistent reported gains across several tasks. Table 8 reports the highest mean fusion accuracy among the compared methods on all five bimodal datasets, with results from three runs, and the paper extends the method to trimodal sentiment analysis. RCC also preserves the inference architecture and parameter count (Table 14). This breadth is encouraging for a training intervention, although the additional calibration stage adds training cost and the model-selection concern below must be resolved before drawing firm conclusions about generalization.
Critical weaknesses
- The released runner selects checkpoints using the test split. In the released CREMA-D and AVSBench runner, stage_command() evaluates test_loader after each epoch and saves best.pth and summary.json when test fusion accuracy improves during the final stage, after the configured selection threshold (rcc_guard.py, lines 403–414). The dataset loaders confirm that this is the test split, rather than an internal validation split (dataset/cremad.py and dataset/avsbench.py).
If the paper's reported results follow this rule, the test set also serves as a model-selection set, which can make the reported performance optimistic. It is currently unclear whether the same selection procedure produced the main tables, how it was applied to the baselines, and whether other datasets use an independent validation protocol. This concern is about checkpoint selection; RCC's probability calibration itself uses a held-out subset of the original training data. A test-independent selection protocol is needed to establish the strength of the reported gains.
- The central calibration and restart design need more direct evidence. The method estimates directional probabilities using one internal 90/10 split per run, updates them at epochs 30, 40, and 50, and transfers the last estimates to a freshly initialized model trained on all training data. The final-stage probabilities remain fixed (Appendix B and the released calibration runner). The paper does not directly characterize how sensitive the final estimates are to the held-out split or the calibration checkpoint, or whether they remain suitable after reinitialization and training on the full dataset.
Table 6 provides useful comparisons with validation-selected fixed strategies and event-count-matched allocations. However, these comparisons do not directly characterize sensitivity to the scaling coefficient or isolate the value of restarting rather than continuing training. For example, the CREMA-D scaling coefficient of 0.35 is a hyperparameter, but the stated parameter sweeps do not provide enough detail to assess its selection or sensitivity. Final-accuracy variation across seeds also does not reveal how much the estimated probabilities vary. Focused stability and scaling-sensitivity results, together with a comparison against continued training under a comparable budget, would clarify the value of the specific two-stage protocol relative to its extra training cost.
Questions for authors
-
Checkpoint selection. Which rule produced the results for each dataset and baseline? If the released test-based selection rule was used, please report results with an independent validation set or a prespecified checkpoint and assess whether the main comparisons remain unchanged.
-
Split stability. How much do the estimated directional probabilities change across different random 10% held-out splits? A focused comparison using 5%, 10%, and 20% held-out data on a representative benchmark would help assess whether the calibration depends strongly on this choice.
-
Training-time stability. What are the probability estimates at epochs 30, 40, and 50, and why is the last estimate representative? Please distinguish the stability of the calibrated probabilities from the stability of the final test accuracy.
-
Transfer between stages. Does a probability calibrated on the first-stage model remain suitable after reinitialization and training on the full dataset? With comparable training budgets and the same calibrated probabilities, how does restarting compare with continuing from the calibration-stage checkpoint?
-
Probability mapping and scaling. How were the scaling coefficients, including 0.35 for CREMA-D, selected? Please specify the search range, selection data, and criterion, and provide a focused sensitivity analysis that complements the fixed-strategy comparisons in Table 6.
-
Necessity of adaptive calibration. To what extent do the improvements come from gap-driven probability calibration rather than feature replacement and additional supervision themselves? Although Table 6 includes fixed-strategy comparisons, a more controlled ablation using matched replacement rates and training budgets would help isolate the contribution of adaptive calibration. In particular, does RCC consistently outperform simple fixed-probability replacement under otherwise identical settings?
5 comments
Response to Reviewer xKJw (Part 2/2, Q4–Q6)
Q4. Transfer between stages.
A4: On CREMA-D, we compare fresh initialization with continuation from the calibration checkpoint using the same calibrated probabilities. Both strategies use the full training set, identical second-stage schedules and update budgets, and evaluation at epoch 100 of the second stage. Continuation retains the calibration model’s parameters and SGD momentum.
Restarting achieves higher accuracy, supporting the use of first-stage calibrated probabilities after reinitialization in this setting. These probabilities remain fixed during second-stage training, while sample reliability weights are recomputed from the current model’s predictions, allowing supervision weights to adapt to its state.
| Strategy | Acc. |
|---|---|
| Restart (RCC) | 84.66 |
| Continue | 82.47 |
Q5. Probability mapping and scaling.
A5: On CREMA-D, we searched and selected using the highest test fusion accuracy in seed 0 runs. The table varies only this scaling coefficient, keeping all other hyperparameters at the original manuscript’s settings.
Accuracy increases over the tested range, with similar results for 0.25 and 0.35. This isolates the effect of probability scaling while retaining RCC’s configuration-gap rule and supervision terms, complementing the fixed-policy comparisons in Table 6.
| Scaling coefficient | Acc. |
|---|---|
| 0.15 | 83.33 |
| 0.25 | 84.41 |
| 0.35 | 84.68 |
Q6. Necessity of adaptive calibration.
A6: Table 6 shows that fixed-probability replacement can itself improve performance. To isolate the contribution of probability calibration, we vary only replacement probabilities on CREMA-D, keeping the replacement mechanism, reliability weighting, auxiliary losses, initialization, and second-stage training budget unchanged. The symmetric rate-matched control preserves RCC’s sum of directional probabilities and divides it equally.
RCC outperforms both the rate-matched control and the same-direction fixed policy. This supports an additional benefit from configuration-guided probability allocation under the tested controls. It does not establish superiority over every fixed policy or random seed.
| Policy | Acc. | ||
|---|---|---|---|
| Calibrated RCC | 0.1172 | 0 | 83.66 |
| Symmetric, rate-matched | 0.0586 | 0.0586 | 81.05 |
| Fixed | 0.1000 | 0 | 81.99 |
| Fixed symmetric | 0.0500 | 0.0500 | 80.51 |
| Fixed symmetric | 0.1000 | 0.1000 | 82.53 |
Dear Reviewer #xKJw,
Thank you again for your review. We have responded to your concerns point by point and added experiments and clarifications supporting our central contribution: diagnosing the capability-to-utility gap and improving fusion’s use of correct unimodal evidence through RCC.
We would appreciate your reassessment in light of our responses and revisions. Could you please let us know whether our responses address your concerns? We would be happy to discuss any remaining issues and provide further clarification.
Best regards, The Authors
Response to Reviewer xKJw (Part 1/2, Q1–Q3)
We thank the reviewer #xKJw for the suggestions on calibration stability and experimental controls. Appendix K.1 now examines sensitivity to calibration splits and scheduled update epochs, while Appendix J.2 provides controlled comparisons of replacement probabilities under the same supervision mechanisms and training budget. Checkpoint selection, restarting versus continuing training, and scaling-coefficient selection are addressed in the responses below.
Q1. Checkpoint selection.
A1: The original main-table results for RCC and all baselines use checkpoints selected by test fusion accuracy, following public implementations such as InfoReg and IBML; some datasets also lack standardized validation splits. A shared protocol does not eliminate the effects of test-based selection. We therefore compare RCC and IBML on AVSBench using its independent validation split for checkpoint selection, alongside test-selected results. Both selection rules use fusion accuracy, and the table reports the corresponding test accuracies. Validation-based selection lowers both methods’ test accuracy, but RCC retains its advantage over IBML in this comparison.
Q2. Split stability.
A2: On CREMA-D, we compare 5%, 10%, and 20% calibration held-out ratios, fixing the model seed and using three random splits per ratio. The table reports means ± standard deviations across splits for the second-stage replacement probabilities and final test accuracy. At the default 10% ratio, the standard deviation of pa→v is 0.0097. Across all nine runs, calibration selects the same direction for additional supervision, despite variation in probability magnitude and final accuracy. This supports consistent directional guidance across the tested ratios and splits.
Q3. Training-time stability.
A3: Algorithm 1 updates directional probabilities at calibration epochs 30, 40, and 50, then uses the last estimate for second-stage training. The epoch-50 choice uses later-stage configuration statistics and is prescribed by the schedule, not selected using final test accuracy.
On CREMA-D with the default 10% held-out ratio, the final estimate retains the same direction and remains close to earlier estimates, supporting the last scheduled update as a representative late-stage estimate. This table examines probability variation during calibration; Q2 examines variation in final test accuracy across splits when using the last probability estimate.