- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper addresses the modality imbalance problem in multimodal learning by proposing a novel method called Max Confidence Regularization (MaxCR). The method introduces a nonlinear sparsity measure to dynamically monitor modality confidence and employs max suppression and max excitation to adjust the predictions of strong and weak modalities, respectively. Experiments on six multimodal datasets demonstrate that the proposed method achieves competitive performance compared with existing multimodal rebalancing methods.
Critical strengths
- The paper investigates the modality imbalance problem in multimodal learning from the perspective of prediction confidence. The proposed framework is intuitive, and its key components are clearly described.
- The proposed MaxCR method is simple yet effective and modality-agnostic, making it easy to integrate into existing multimodal learning frameworks.
- The proposed method is evaluated on six multimodal datasets, demonstrating competitive performance.
Critical weaknesses
- MaxCR is trained under the idealized assumption that all modalities are available. However, its performance under noisy or missing settings is not investigated.
- The confidence calibration results in Table 2 are primarily limited to KSounds. The authors are encouraged to report calibration results on additional datasets to provide a more comprehensive analysis.
- Although the performance gains are consistent, the improvements over strong baselines such as LFM and AUG are sometimes modest.
Questions for authors
- How does MaxCR distinguish genuine uncertainty from underconfidence? Can the authors provide per-modality calibration results across more datasets?
- How does the proposed theoretical analysis directly justify the effectiveness of MaxCR? Can the authors clarify the relationship between the certainty-transfer ratio and the measured gradient cosine similarity?
3 comments
Q.1.1. Uncertainty and Underconfidence
MaxCR does not explicitly distinguish uncertainty from underconfidence. It identifies potential underconfidence through cross-modal confidence discrepancies and uses cross-entropy supervision to constrain confidence enhancement. As shown in Appendix B.2, even for incorrect predictions caused by genuine ambiguity, cross-entropy preserves the corrective direction toward the ground-truth class, preventing unconstrained reinforcement of incorrect predictions.
The results in Tables C and D demonstrate that MaxCR improves prediction quality and calibration, rather than indiscriminately increasing predictive confidence.
Q.2. Theoretical Analysis and Gradient Cosine Similarity
Our theoretical analysis reveals a potential decoupling between task optimization and increasing prediction certainty, which motivates the design of MaxCR. Gradient cosine similarity measures the directional factor governing certainty transfer. The near-zero cosine similarity of the weaker modality in Figure 2 provides empirical evidence of this decoupling at the gradient direction.
This observation motivates us to directly regulate prediction certainty. Appendix B analyzes the local effects of Max Suppression and Max Excitation on prediction certainty, while the ablation results in Figure 4 further validate their effectiveness. Thus, our theoretical analysis provides a clear mechanistic motivation for MaxCR rather than a direct guarantee of its global performance.
Dear Reviewer gbvm,
Thank you for engaging with our response and for updating your score. We appreciate your time and valuable feedback.
We provided additional robustness and calibration experiments, clarified the theoretical motivation, and further demonstrated MaxCR’s effectiveness and advantages over existing methods.
We sincerely appreciate your positive assessment of our work and constructive suggestions.
W.1. Evaluation under Missing or Noisy Modality Settings
Our method is adaptable to scenarios with missing or noisy modalities. We comprehensively evaluate its robustness under test-time modality missingness and noise. Specifically, test-time missingness refers to the setting where one or more modalities are missing at inference. Test-time noise refers to the setting where additional noise is introduced at inference. We conduct experiments on CREMAD under different missing rates and noise levels. We compare against Naive MML, ReconBoost, and AUG.
Table A. Performance comparison in scenario with modality missing.
Table A reports the results at missing rates of 20% and 50%, where r denotes the missing rate. As expected, the performance of all methods decreases as the missing rate increases. Nevertheless, our method consistently achieves the best performance across all missing rates, demonstrating its effectiveness under modality missingness.
Table B. Performance comparison under modality noise scenario.
Table B further compares performance under modality noise, where μ denotes the noise rate. Our method again consistently outperforms the baselines, confirming its robustness to noisy modalities.
We believe MaxCR may benefit from two factors: the underlying decoupled framework preserves independent predictions from each modality, while MaxCR's dynamic confidence regulation may reduce overreliance on a single high-confidence modality.
The relevant discussion and experimental results have been added to Appendix E.10.
W.2. & Q.1.2 More Calibration Results
We conducted additional calibration experiments on CREMAD and report the unimodal calibration metrics for CREMAD and KSounds in Tables C and D, respectively. MaxCR achieves higher accuracy and lower ECE across all prediction branches, with consistent improvements in NLL and Brier Score. These results indicate that the performance gains are not solely due to final fusion but are also accompanied by improved unimodal prediction quality.
Table C. Calibration metrics on CREMAD dataset.
Table D. Calibration metrics on KSounds dataset.
The relevant discussion and experimental results have been added to Appendix E.11.
W.3. Performance Comparison
Regarding the improvements over strong baselines, our method achieves substantial gains on both audio-visual and trimodal datasets. The gains on image-text datasets are relatively limited, possibly because of the weaker cross-modal complementarity between image and text. Indeed, we observe that, compared with the best unimodal performance, all multimodal methods generally provide only marginal improvements on these datasets. Notably, on Twitter2015, most multimodal methods even underperform the best unimodal baseline. Addressing this limitation is a promising direction for future research.
More specifically, we compare the performance gains of our method with those of AUG, the strongest baseline. AUG improves accuracy by 1.53% on CREMAD, but only by 0.10% on KSounds, 0.27% on VGGSound, and 0.36% on NVGesture. In comparison, our method improves accuracy by 0.16% on CREMAD, 2.59% on KSounds, 1.15% on VGGSound, and 1.50% on NVGesture. Thus, our method provides substantially larger improvements on the other three datasets and demonstrates stronger overall gains than the best baseline.