acceptodds
Under review as a conference paper at ICLR 2027

Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification

Abstract

Multimodal learning (MML) falls into the optimization dilemma due to the modality imbalance phenomenon, leading to suboptimal overall performance in practice. While many attempts primarily focus on balancing the optimization dynamics across modalities to address this issue, we identify a subtle yet critical flaw: optimization yields asymmetric gains in predictive certainty, with the strong modality more confident than the weak one, driving imbalanced modality contributions. In this paper, our analysis reveals that this flaw stems from unimodal characteristics rather than multimodal learning, and this confidence discrepancy can be corrected by positive cross-modal intervention. Based on this insight, we propose multimodal Max Confidence Regularization (MaxCR) to dynamically intervene in modality semantic confidence. Specifically, the semantic confidence of each modality is tracked using a nonlinear sparsity measure. We then design max suppression and max excitation based on this measure to regularize strong and weak modalities, respectively. They penalize and encourage the top-1 confidence, thereby constraining multimodal prediction. To this end, strong and weak modalities are expected to make calibrated confidence, thereby improving the overall performance. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines. The source code is available at https://anonymous.4open.science/r/MaxCR-A07A.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Imbalaced Multimodal Learning, Multimodal Classification, Confidence Regularization

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_gbvmOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper addresses the modality imbalance problem in multimodal learning by proposing a novel method called Max Confidence Regularization (MaxCR). The method introduces a nonlinear sparsity measure to dynamically monitor modality confidence and employs max suppression and max excitation to adjust the predictions of strong and weak modalities, respectively. Experiments on six multimodal datasets demonstrate that the proposed method achieves competitive performance compared with existing multimodal rebalancing methods.

Critical strengths
  1. The paper investigates the modality imbalance problem in multimodal learning from the perspective of prediction confidence. The proposed framework is intuitive, and its key components are clearly described.
  2. The proposed MaxCR method is simple yet effective and modality-agnostic, making it easy to integrate into existing multimodal learning frameworks.
  3. The proposed method is evaluated on six multimodal datasets, demonstrating competitive performance.
Critical weaknesses
  1. MaxCR is trained under the idealized assumption that all modalities are available. However, its performance under noisy or missing settings is not investigated.
  2. The confidence calibration results in Table 2 are primarily limited to KSounds. The authors are encouraged to report calibration results on additional datasets to provide a more comprehensive analysis.
  3. Although the performance gains are consistent, the improvements over strong baselines such as LFM and AUG are sometimes modest.
Questions for authors
  1. How does MaxCR distinguish genuine uncertainty from underconfidence? Can the authors provide per-modality calibration results across more datasets?
  2. How does the proposed theoretical analysis directly justify the effectiveness of MaxCR? Can the authors clarify the relationship between the certainty-transfer ratio and the measured gradient cosine similarity?
3 comments
AuthorsOpenReview ↗

We provided additional robustness and calibration experiments, clarified the theoretical motivation, and further demonstrated MaxCR’s effectiveness and advantages over existing methods.

We sincerely appreciate your positive assessment of our work and constructive suggestions.

W.1. Evaluation under Missing or Noisy Modality Settings

Our method is adaptable to scenarios with missing or noisy modalities. We comprehensively evaluate its robustness under test-time modality missingness and noise. Specifically, test-time missingness refers to the setting where one or more modalities are missing at inference. Test-time noise refers to the setting where additional noise is introduced at inference. We conduct experiments on CREMAD under different missing rates and noise levels. We compare against Naive MML, ReconBoost, and AUG.

Table A. Performance comparison in scenario with modality missing.

Methodr=0r=0r=20r=20r=50r=50
Naive MML0.63610.58490.5242
ReconBoost0.75570.63210.5568
AUG0.85150.75040.6008
MaxCR0.85310.75810.6102

Table A reports the results at missing rates of 20% and 50%, where rr denotes the missing rate. As expected, the performance of all methods decreases as the missing rate increases. Nevertheless, our method consistently achieves the best performance across all missing rates, demonstrating its effectiveness under modality missingness.

Table B. Performance comparison under modality noise scenario.

Methodμ=0\mu=0μ=20\mu=20μ=50\mu=50
Naive0.63610.64250.6331
ReconBoost0.75570.72150.7031
AUG0.85150.83330.8253
MaxCR0.85310.83870.8320

Table B further compares performance under modality noise, where μ\mu denotes the noise rate. Our method again consistently outperforms the baselines, confirming its robustness to noisy modalities.

We believe MaxCR may benefit from two factors: the underlying decoupled framework preserves independent predictions from each modality, while MaxCR's dynamic confidence regulation may reduce overreliance on a single high-confidence modality.

The relevant discussion and experimental results have been added to Appendix E.10.

W.2. & Q.1.2 More Calibration Results

We conducted additional calibration experiments on CREMAD and report the unimodal calibration metrics for CREMAD and KSounds in Tables C and D, respectively. MaxCR achieves higher accuracy and lower ECE across all prediction branches, with consistent improvements in NLL and Brier Score. These results indicate that the performance gains are not solely due to final fusion but are also accompanied by improved unimodal prediction quality.

Table C. Calibration metrics on CREMAD dataset.

MethodModalityAcc ↑ECE ↓NLL ↓Brier ↓
NaiveMultimodal0.63310.24331.75320.5975
Audio0.58600.29061.99720.6721
Video0.26480.15571.79170.8387
MaxCRMultimodal0.85310.01390.43270.2124
Audio0.68550.19231.31390.4911
Video0.68280.08770.90290.4297

Table D. Calibration metrics on KSounds dataset.

MethodModalityAcc ↑ECE ↓NLL ↓Brier ↓
NaiveMultimodal0.64550.16071.52060.5167
Audio0.48130.34163.66540.8303
Video0.35330.39704.00260.9935
MaxCRMultimodal0.75220.03840.91950.3466
Audio0.55700.22242.13410.6435
Video0.56050.21052.08850.6340

The relevant discussion and experimental results have been added to Appendix E.11.

W.3. Performance Comparison

Regarding the improvements over strong baselines, our method achieves substantial gains on both audio-visual and trimodal datasets. The gains on image-text datasets are relatively limited, possibly because of the weaker cross-modal complementarity between image and text. Indeed, we observe that, compared with the best unimodal performance, all multimodal methods generally provide only marginal improvements on these datasets. Notably, on Twitter2015, most multimodal methods even underperform the best unimodal baseline. Addressing this limitation is a promising direction for future research.

More specifically, we compare the performance gains of our method with those of AUG, the strongest baseline. AUG improves accuracy by 1.53% on CREMAD, but only by 0.10% on KSounds, 0.27% on VGGSound, and 0.36% on NVGesture. In comparison, our method improves accuracy by 0.16% on CREMAD, 2.59% on KSounds, 1.15% on VGGSound, and 1.50% on NVGesture. Thus, our method provides substantially larger improvements on the other three datasets and demonstrates stronger overall gains than the best baseline.

Q.1.1. Uncertainty and Underconfidence

MaxCR does not explicitly distinguish uncertainty from underconfidence. It identifies potential underconfidence through cross-modal confidence discrepancies and uses cross-entropy supervision to constrain confidence enhancement. As shown in Appendix B.2, even for incorrect predictions caused by genuine ambiguity, cross-entropy preserves the corrective direction toward the ground-truth class, preventing unconstrained reinforcement of incorrect predictions.

The results in Tables C and D demonstrate that MaxCR improves prediction quality and calibration, rather than indiscriminately increasing predictive confidence.

Q.2. Theoretical Analysis and Gradient Cosine Similarity

Our theoretical analysis reveals a potential decoupling between task optimization and increasing prediction certainty, which motivates the design of MaxCR. Gradient cosine similarity measures the directional factor governing certainty transfer. The near-zero cosine similarity of the weaker modality in Figure 2 provides empirical evidence of this decoupling at the gradient direction.

This observation motivates us to directly regulate prediction certainty. Appendix B analyzes the local effects of Max Suppression and Max Excitation on prediction certainty, while the ablation results in Figure 4 further validate their effectiveness. Thus, our theoretical analysis provides a clear mechanistic motivation for MaxCR rather than a direct guarantee of its global performance.

Dear Reviewer gbvm,

Thank you for engaging with our response and for updating your score. We appreciate your time and valuable feedback.

Reviewer_6kehOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at modality imbalance in multimodal classification from the angle of predictive confidence. It observes that the strong modality ends up much more confident than the weak one. A gradient-alignment analysis is used to argue that this comes from the weak modality being intrinsically ambiguous, so that optimization stops turning into certainty gains. The proposed method, MaxCR, tracks each modality's confidence with a sparsity score. It then penalizes the top-1 probability of the more confident modality (max suppression) and pushes up that of the less confident one (max excitation), with the strength set by the confidence gap. Experiments cover six datasets against 16 baselines.

Critical strengths
  1. Looking at imbalance through unimodal confidence, rather than through gradients or learning speed, is a fairly fresh angle, and Fig. 1 makes the confidence gap easy to see. This is the main thing that makes the paper interesting, since most recent work in this area keeps reshaping gradients.

  2. The method is simple and cheap to add, and it is evaluated broadly, including a trimodal dataset. The gains on Kinetics-Sounds and NVGesture are clear. That breadth makes the empirical side more credible than a single-benchmark result would be.

Critical weaknesses
  1. Max suppression, λ(max p − 1/C), is essentially MaxSup (Zhou et al., NeurIPS 2025) written in probability space, and max excitation is the same term with the sign flipped. MaxSup is cited only in passing and never credited as the source, and Table 4 compares against label smoothing rather than MaxSup itself. The theory doesn't really back the story either. The vanishing certainty gain in Theorem 3 doesn't depend on H_w and holds for any modality with nonzero Bayes entropy, so it shows at most a difference in rate, not an asymmetry. Corollary 1 says the gap is set by Bayes entropy and is independent of the optimizer, which makes the weak modality's low confidence look like correct behavior rather than underconfidence. The over/underconfidence labels are always relative to the other modality and never checked against each modality's own accuracy. This seems fundamental: the contribution reduces to a known regularizer plus a switch. MaxSup should at least be credited and compared against, and per-modality calibration should support the underconfidence claim. Calibrating Multimodal Learning (Ma et al., ICML 2023) and the uni-modal ensemble results of Du et al. (ICML 2023) are closely related and missing.

  2. The ablation in Fig. 4 removes only one loss at a time, and there's no result for the underlying decoupled framework (unimodal CE plus averaged predictions) on its own. Even with one loss removed, CREMA-D accuracy is 0.82–0.84, close to or above LFM, so much of the gain may come from the framework rather than from MaxCR. Against AUG, the margin is 0.16 points on CREMA-D with a std of 0.44 and 0.08 points on Sarcasm. MaxCR trails AMSS+ on Twitter and AUG on VGGSound MAP. Only Kinetics-Sounds and NVGesture show clear gains. This is fixable by reporting the base framework on all datasets and adding simple baselines such as per-modality temperature scaling, MaxSup on the strong modality, and CML.

  3. Figs. 3–5 are hard to read because the font is very small. Fig. 1 (left) puts confidence, candidate counts and accuracy on one unlabeled axis.

Questions for authors
  1. The baseline numbers match Table 1 of AUG exactly. Were they re-run, or taken from AUG? The CREMA-D unimodal rows also look mismatched compared with AUG's table (audio Acc .6317 paired with MAP .5879, video .4583 with .6861), so please double-check them.

  2. On CREMA-D, the video branch under MaxCR reaches about 0.68, and about 0.66 even without L_exc, while unimodal video is 0.4583. Was the unimodal baseline trained under the same setup (frames, epochs)? If so, what explains such a large gap?

  3. What does the decoupled framework alone, with neither L_sup nor L_exc, achieve on each dataset? This largely decides how much of the gain can be attributed to MaxCR.

  4. Is the weak modality actually underconfident relative to its own accuracy? Per-modality ECE or reliability diagrams for Naive and MaxCR would answer this.

5 comments
AuthorsOpenReview ↗

We clarified key concerns, corrected reporting issues, and provided additional experiments to validate MaxCR’s effectiveness and superiority.

Thank you very much for your thorough review of our work and your insightful suggestions.

W.1.1. & W.2.3. Comparison with MaxSup, temperature scaling, CML, and UMT

We acknowledge the conceptual similarity between our proposed Max Suppression and MaxSup in terms of maximum-value regularization, but the two are not equivalent. Specifically, MaxSup penalizes zm−1C∑jzjz_m-\frac{1}{C}\sum_j z_j, whereas our proposed Lsup\mathcal{L}_{sup} penalizes pm−1Cp_m-\frac{1}{C}. Their gradients w.r.t. the logits are α(em−1/C)\alpha(e_m-\mathbf{1}/C) and λpm(em−p)\lambda p_m(e_m-p), respectively, leading to different optimization behaviors. More importantly, the core contribution of MaxCR lies in dynamic bidirectional regulation driven by cross-modal discrepancies. Specifically, MaxCR monitors modality-wise prediction certainty through nonlinear sparsity, dynamically determines regularization strength based on cross-modal discrepancies, and applies Suppression and Excitation to relatively strong and weak modalities, respectively.

Furthermore, we conducted additional comparisons with MaxSup, temperature scaling, CML, and UMT on CREMAD to evaluate the effectiveness of MaxCR. Specifically, we apply MaxSup to the dominant modality, adaptively scale logits based on modality discrepancies for temperature scaling, and use the official implementations of CML and UMT. As shown in Table A, our method achieves the best performance. Unidirectional suppression of strong modality cannot explicitly promote the learning of weaker modality and performs worse than our proposed Max Suppression (0.8387). In contrast, MaxCR jointly considers all modalities through discrepancy-aware bidirectional regulation, achieving better overall performance.

Table A. Comparison with MaxSup, temperature scaling, CML, and UMT.

MethodACCMAP
MaxSup0.80380.8815
temperature0.78090.8529
CML0.72720.8060
UMT0.70970.8292
MaxCR0.85310.9216

We have added a discussion of MaxSup at Line 289 in the paper. More relevant discussion and experimental results have been added to Appendix E.8.

W.1.2. Theoretical Asymmetry

We agree that Theorem 3's asymptotic conclusion may also apply to other modalities with nonzero conditional Bayes entropy under the corresponding assumptions. Moreover, the certainty gap determined by Bayes entropy in Corollary 1 may reflect genuine differences in modality informativeness rather than model underconfidence.

We would like to clarify that our theoretical analysis provides a mechanistic motivation rather than a direct guarantee of modality asymmetry or the global effectiveness of MaxCR. Specifically, the theoretical results reveal a potential decoupling between task optimization and prediction certainty, while empirical observations show differences in gradient-direction alignment across modalities. Motivated by these findings, we propose an adaptive confidence regulation method MaxCR.

W.2.1 & Q.1. & Q.2. & Q.3. Clarification of the baseline and decoupled framework

Our baseline results are directly taken from LFM, AMSS+, and AUG. We correct the mismatched unimodal MAP metric on CREMAD.

As stated in Lines 228–229, we adopt a decoupled multimodal learning framework following MLA, ReconBoost, and DI-MML. As noted in AMSS+, the unimodal baselines on CREMAD follow earlier studies, whereas subsequent methods and decoupled learning approaches, such as ReconBoost, use more iterations. This discrepancy, which does not occur on other datasets, explains the relatively poor performance of the video baseline on CREMAD. Moreover, Lexc\mathcal{L}_{exc} is not assigned to a fixed modality but adapts at each step. In particular, we observed that it affects video learning during the early stages on CREMAD. Thus, ablating Lexc\mathcal{L}_{exc} cannot be directly interpreted as independent video learning.

Finally, we report the performance of the standalone decoupled framework in Table B. It outperforms Naive by eliminating cross-modal conflicts but underperforms MLA due to the lack of cross-modal interactions. In contrast, our method achieves better results.

Table B. Comparison with decoupled multimodal learning framework.

MethodAudioVideoMultimodal
Naive0.58600.26480.6361
MLA0.57270.64910.7943
MLA*0.58470.64110.7930
Decoupled0.59950.63310.7863
MaxCR0.68550.68280.8531

MLA* denotes our reproduction of MLA, which is close to the originally reported results.

We have corrected the MAP metric error in Table 1.

W.2.2. Performance Comparison

Regarding the improvements over strong baselines, our method achieves substantial gains on both audio-visual and trimodal datasets. The gains on image-text datasets are relatively limited, possibly because of the weaker cross-modal complementarity between image and text. Indeed, we observe that, compared with the best unimodal performance, all multimodal methods generally provide only marginal improvements on these datasets. Notably, on Twitter2015, most multimodal methods even underperform the best unimodal baseline. Addressing this limitation is a promising direction for future research.

More specifically, we compare the performance gains of our method with those of AUG, the strongest baseline. AUG improves accuracy by 1.53% on CREMAD, but only by 0.10% on KSounds, 0.27% on VGGSound, and 0.36% on NVGesture. In comparison, our method improves accuracy by 0.16% on CREMAD, 2.59% on KSounds, 1.15% on VGGSound, and 1.50% on NVGesture. Thus, our method provides substantially larger improvements on the other three datasets and demonstrates stronger overall gains than the best baseline.

Q.4. Low Confidence, Uncertainty, Statistical Underconfidence, and Modality-Specific Calibration Metrics

We agree that lower predictive confidence does not necessarily indicate statistical underconfidence or irreducible uncertainty. MaxCR's sparsity score captures differences in the concentration of predictive distributions but cannot directly establish underconfidence. Neither low confidence on correctly classified samples nor ECE alone is sufficient to make such a claim. Nevertheless, lower confidence may also reflect insufficient representation learning rather than solely intrinsic uncertainty. For a modality that is inherently harder to learn, limited model capacity may prevent the model from fully extracting discriminative features, resulting in less confident predictions. Thus, relative confidence differences can still provide useful signals for adaptive regularization.

We report the accuracy and calibration metrics of each modality for both Naive and MaxCR in the paper (Table 2). As shown in Table C, MaxCR improves both classification accuracy and calibration-related metrics, suggesting that confidence regularization can enhance predictive quality rather than merely sharpen probability distributions.

Table C. Comparison of calibration metrics.

MethodModalityAcc ↑ECE ↓NLL ↓Brier ↓
NaiveMultimodal0.64550.16071.52060.5167
Audio0.48130.34163.66540.8303
Video0.35330.39704.00260.9935
MaxCRMultimodal0.75220.03840.91950.3466
Audio0.55700.22242.13410.6435
Video0.56050.21052.08850.6340

The relevant discussion and experimental results have been added to Appendix E.11.

W.3. Small font sizes in Figures 3–5 and the mixed y-axis scales in Figure 1

We enlarge the font sizes in Figures 3–5, and present accuracy, confidence, and the number of candidate classes separately in Figure 1.

We have adjusted the y-axis in Fig. 1 and the font sizes in Figs. 3–5.

Dear Reviewer 6keh,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request that you review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns.

Thank you again for your time and effort!

Reviewer_CjjQOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates modality imbalance in multimodal classification from a confidence calibration perspective and identifies an overlooked mechanism: optimization can yield asymmetric gains in predictive certainty across modalities. The authors analyze this phenomenon theoretically via a certainty-transfer ratio and further validate it empirically. Building on this, they propose Max Confidence Regularization (MaxCR), a lightweight plug-in that dynamically monitors per-modality confidence with a nonlinear sparsity metric. It applies max suppression to the strong modality and max excitation to the weak one, improving overall performance across six benchmarks.

Critical strengths
  1. The paper offers a fresh and useful perspective on multimodal imbalance by focusing on predictive certainty.
  2. The theoretical lens is insightful, while the proposed method is easy to implement and empirically effective across diverse datasets, with clear calibration benefits.
  3. The experimental evaluation is well executed, with ablation studies that support the design choices and comparisons with closely related regularizers.
Critical weaknesses
  1. Although the authors compare MaxCR with related regularization methods (label smoothing and entropy minimization), they do not provide corresponding comparisons in terms of calibration metrics.
  2. The paper includes experiments on a trimodal dataset, but it lacks relevant ablation studies.
  3. A natural baseline would be to directly align the task gradient with the entropy-reduction direction. The authors should provide a discussion and comparison with this baseline.
Questions for authors

Please refer to "Critical Weaknesses".

2 comments
AuthorsOpenReview ↗

We conducted additional experiments on calibration, trimodal ablation, and gradient alignment, further validating MaxCR's effectiveness and superiority.

We sincerely appreciate your positive assessment of our work and constructive suggestions.

W.1. Comparison of calibration metrics for label smoothing and entropy minimization.

We supplement Table 2 in the paper with ECE, NLL, and Brier Score for label smoothing (LS) and entropy minimization (EM), while also reporting the corresponding accuracy. Additional experiments are conducted on KSounds. Consistent with the experimental setup in the paper, we apply label smoothing to the strong modality and entropy minimization to the weak modality.

Table A. Comparison of calibration metrics.

MethodAccECENLLBrier
Naive0.64550.16071.52060.5167
OGM0.65820.14981.45610.5009
LS0.72860.06800.99090.4104
EM0.72480.05010.99970.3805
MaxCR0.75220.03840.91950.3466

As shown in Table A, both LS and EM improve performance and calibration compared with Naive and OGM, demonstrating the benefits of confidence regularization for multimodal learning. MaxCR achieves the best results across all four metrics. This may be because LS and EM do not explicitly model cross-modal confidence discrepancies, whereas MaxCR adaptively suppresses higher confidence and enhances lower confidence based on dynamic cross-modal discrepancies, thereby improving prediction quality.

The relevant discussion and experimental results have been added to Appendix E.6.

W.2. Ablation study in the trimodal setting.

We further include ablation experiments on the trimodal NVGesture dataset, reporting both unimodal and multimodal performance in Table B.

Table B. Ablation study on NVGesture dataset.

MethodRGBOFDepthMultimodal
MaxCR0.74690.80080.81540.8651
w/o Lexc\mathcal{L}_{exc}0.75310.79880.78220.8423
w/o Lsup\mathcal{L}_{sup}0.75520.78630.80290.8485

Removing the regularization components slightly improves RGB accuracy, possibly due to reduced suppression of correct predictions or reinforcement of incorrect ones. However, it degrades the performance of other modalities and overall performance.

The relevant discussion and experimental results have been added to Appendix E.7.

W.3. Entropy-direction Alignment Baseline

We thank the reviewer for the suggestion. Gradient alignment is a straightforward and feasible strategy. To evaluate its effectiveness, we conducted an additional experiment on CREMAD. Specifically, for each modality, we compute the cross-entropy gradient gce\boldsymbol g_{ce} and prediction entropy gradient ge\boldsymbol g_{e}. When their cosine similarity falls below a threshold ρ\rho, we adjust the task gradient as g~ce=∣gce∣(ρge∣ge∣+1−ρ2gce⊥∣gce⊥∣)\widetilde{\boldsymbol g}_{ce}=|\boldsymbol g_{ce}|\left(\rho\frac{\boldsymbol g_{e}}{|\boldsymbol g_{e}|}+\sqrt{1-\rho^2}\frac{\boldsymbol g_{ce\perp}}{|\boldsymbol g_{ce\perp}|}\right), where gce⊥=gce−⟨gce,ge⟩∣ge∣2ge\boldsymbol g_{ce\perp}=\boldsymbol g_{ce}-\frac{\langle \boldsymbol g_{ce}, \boldsymbol g_{e}\rangle}{|\boldsymbol g_{e}|^2}\boldsymbol g_{e}. This preserves the task gradient norm while increasing its cosine similarity with the entropy gradient to ρ\rho. Gradients already satisfying the threshold remain unchanged. We then update the corresponding branch using −g~ce-\widetilde{\boldsymbol g}_{ce}, aligning the task update direction with entropy reduction. We tune ρ∈{0.2,0.4,0.6,0.8}\rho\in \lbrace0.2,0.4,0.6,0.8\rbrace and report the best result. As shown in Table C, the results show that our method achieves superior performance.

Table C. Comparison with entropy-direction alignment.

MethodACCMAP
Gradient Alignment0.77150.8206
MaxCR0.85310.9216

Direct gradient alignment explicitly modifies the original task optimization direction without distinguishing between beneficial increases in prediction certainty and potentially harmful confidence amplification. In contrast, MaxCR provides a more effective strategy by adaptively regulating confidence based on cross-modal confidence discrepancies.

The relevant discussion and experimental results have been added to Appendix E.9.

Dear Reviewer CjjQ,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request that you review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns.

Thank you again for your time and effort!

Comments on the paper

1 comment

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have carefully revised the manuscript to address the concerns raised in the reviews. The main revisions are summarized below:

  1. Strengthened comparisons with related methods (Section 4.3 and Appendices E.8, E.9). We clarified the differences between MaxCR and MaxSup, and added experimental comparisons with MaxSup, temperature scaling, CML, UMT, and entropy-direction gradient alignment to further demonstrate the effectiveness of our method.

  2. Expanded calibration evaluations and ablation studies (Appendices E.6, E.7 and E.11). We supplemented calibration comparisons with label smoothing and entropy minimization, provided modality-specific calibration results on CREMAD and KSounds, and extended the ablation study to the trimodal NVGesture dataset.

  3. Added robustness evaluations under missing and noisy modalities (Appendix E.10). We conducted additional experiments under different modality missing rates and noise levels, demonstrating the robustness of MaxCR compared with existing multimodal learning methods.

Additionally, we corrected the unimodal MAP values in Table 1 and improved the presentation of several figures.

The changes in the revised manuscript are highlighted in red for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.