- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies multimodal imbalance from the perspective of post-training information utilization. The authors point out that useful class information may already exist in the intermediate layer representations of each modality, but the final prediction head does not fully utilize this information. Based on this observation, the paper proposes LAMF. The authors conduct extensive theoretical analysis and experiments to demonstrate the effectiveness of the method proposed in this paper.
Critical strengths
-
The research content and perspective are innovative. Multimodal imbalanced learning is a key technical approach to enhancing the performance of multimodal models. This study's approach from the perspective of post-training information utilization is also innovative.
-
The authors conducted thorough theoretical derivations and comprehensive experimental validation to demonstrate the algorithm's effectiveness.
Critical weaknesses
-
In my opinion, one of the most significant issues with this manuscript is that the paper’s overall narrative is built upon the concept of modal imbalance, and it posits that modal imbalance may prevent the learned modal information from being fully utilized in the final prediction. However, Propositions 1–3 do not explicitly mention modal imbalance, modal competition, dominant modalities, or weak modalities. From a mathematical perspective, these theories only consider a limited number of prediction sources and then investigate whether adding a new predictor can improve the current convex combination. This theory may also apply to other scenarios, such as different intermediate layers in a single-modal network. Therefore, could the authors provide further clarification on how to distinguish between the view currently supported in the manuscript—that “intermediate-layer predictions may contain complementary information not yet utilized by the final prediction head” (the view supported by the current theory)—and the common view in multimodal imbalanced learning that “modal imbalance leads to underutilization of already learned modal information”?
-
There are some theoretical details that the author needs to clarify further. Please see the “Questions” section for more information.
Questions for authors
-
Regarding Proposition 3: Equation (11) is used to express the deployment risk bound, where represents the difference between the fold models estimated using OOF and the final full-data model. I would like the authors to further explain why should be small during training. Is there a theoretical basis to support this view? Additionally, the risk consistency in Appendix A.5 requires that . Could the authors further clarify under what conditions we can expect ? Can they provide a finite-sample bound and a stability analysis?
-
Proposition 2 shows that when the training samples are fitted with high confidence, a bounded score perturbation may only cause a very small change in the training CE. Here, I would like the authors to further explain: Why does this phenomenon specifically support OOF estimation, rather than merely indicating that the CE on the training set has reached a plateau? There may be different reasons for the small response on the training set. For example, one possibility is that the intermediate-layer prediction does indeed contain valuable complementary information, but it cannot be identified by the CE due to an overly strong training fit; another possibility is that the prediction source itself is indeed redundant. Is there a theoretical criterion that can distinguish between these two scenarios?
-
The accuracy metrics reported in Table 2 differ significantly from those reported in existing papers (such as OGM, MMPareto, DGL, etc.). Therefore, I reviewed the code uploaded by the authors. I found numerous inconsistencies in the model architecture compared to the code in existing works, such as additional Dropout layers. Therefore, when comparing the performance of different methods, did the authors ensure that the model architectures were standardized? If the model architectures differ, the reliability of the algorithm performance comparison will be significantly compromised.
8 comments
Response to Reviewer JxD5 (Part 2/5, W1: Scope and Mechanism)
2. The theory applies to general prediction sources; the multimodal analysis examines modality and layer differences. The propositions hold for finitely many prediction sources and also apply to unimodal networks; being multimodal is not an assumption required for their validity. This general formulation separates additional combination value, its identification through in-sample estimation, and its preservation at deployment. In multimodal settings, we examine whether predictions from different modalities and layers complement the original combination, how modality-specific fitting differences affect their utilization, and whether fixed-model interventions improve their use in final decisions.
3. The existence of complementary value and the mechanism of underutilization are distinct.
Unused complementary information describes a phenomenon in which final predictions underutilize learned representations; attributing that underutilization to modality imbalance additionally specifies a cause. Prior work analyzes how optimization imbalance and modality competition during joint training limit the learning of some modalities (Peng et al., 2022; Huang et al., 2022). We further study information already learned: whether complementary predictions from different modalities enter the final combination and how fitting differences affect their utilization through in-sample combination estimation. Accordingly, decision-stage utilization asymmetry is an outcome we seek to explain and mitigate, with an analysis of how it may persist.
For unused complementary information, we provide both existence and realized-gain evidence. Figure 1(a) identifies samples that no convex combination of the original unimodal and fusion head outputs can classify correctly, but layer predictions can. This establishes complementary value relative to that combination family. Table 4 freezes OGM-GE, DGL, and TCMax models and reports accuracy gains of 0.54–1.08% from layer predictions over Output only, showing that this value can enter actual decisions. Proposition 1 provides conditions for decreasing expected CE; classification complementarity is tested experimentally, not inferred from a CE derivative alone.
For decision-stage modality imbalance, we analyze how fitting differences affect modality utilization through combination estimation. Proposition 2 implies that sufficiently fitting one modality can yield low training loss for the original combination and constrain its response to additional predictions. In-sample combination estimation may consequently underrepresent remaining out-of-sample complementary value. To test an intervenable step, original Appendix E.2 fixes fold models, layer predictor configurations, and scales, changes only the estimation records, and evaluates on the same independent set. The following are accuracy gains (%) from layer predictions over the respective Output only combinations.
| Combination estimation records | CREMA-D | KS |
|---|---|---|
| In-sample | +0.231 ± 0.201 | 0.000 ± 0.000 |
| Out-of-fold | +1.697 ± 0.694 | +1.503 ± 0.203 |
The same layer predictions produce larger gains with out-of-sample estimation, distinguishing available complementary information from its utilization by the combination. Our progression is to establish unused predictive value, characterize conditions under which in-sample responses can mask it under asymmetric fitting, and improve its utilization through out-of-sample combination estimation. Certificates and fixed-model results establish complementary value; mathematical analysis and controlled interventions test parts of the underutilization mechanism, rather than deriving a causal conclusion from general ensemble gains.
Response to Reviewer JxD5 (Part 3/5, W2 / Q1)
W2 / Q1: Stability conditions and finite-sample bounds for transfer error
1. Prediction-source stability controls the size of .
The original proposition does not require to decrease automatically with training epochs. In Equation (11), measures the expected-loss difference between fold models and the full model under the same combination parameters, not a model's training error. Further reducing training loss does not directly control it. The relevant theoretical basis is stability of prediction sources under refitting. With identically distributed training and test samples and all models and probes fixed, the oscillation bound for cross entropy gives
\mathbb E\max_j\operatorname{osc}\!\left(s_j(X;F_k)-s_j(X;F)\right), \qquad \omega_k=|\mathcal D_{3-k}|/N.$$ Here $\operatorname{osc}(v)=\max_c v_c-\min_c v_c$, and $s_j$ includes the deployed scales. This holds uniformly over convex combinations because their score changes are bounded by the maximum oscillation across sources. More stable corresponding sources therefore yield a smaller upper bound on $\Gamma$, directly connecting stability to Proposition 3. **2. $\Gamma\to0$ requires convergence of corresponding prediction sources.** This concerns increasing sample size, not increasing training epochs. A sufficient condition is a fixed number of sources, identically distributed training and test samples, and corresponding score functions trained on half and full datasets converging to a common limit with vanishing expected oscillation above. With frozen encoders, probe refitting can satisfy such consistency under nondegenerate standardization, appropriate moment conditions, and fixed positive ridge regularization. Retraining encoders additionally requires prediction stability and scale control. Thus, Appendix A.5 is a consistency result under these conditions; OOF does not replace the stability assumptions. **3. Independent evaluation provides a conditional finite-sample bound.** A conditional upper bound follows from the same stability quantity. Define $$g(X)=\sum_{k=1}^{2}\omega_k\max_j\operatorname{osc}\!\left(s_j(X;F_k)-s_j(X;F)\right).$$ If models, probes, and scales are fixed before evaluation and a valid population bound $0\le g(X)\le B$ holds, one-sided Hoeffding gives, on $n$ independent identically distributed evaluation samples, with probability at least $1-\delta$, $$\Gamma\le \mathbb E g(X) \le \frac1n\sum_{i=1}^{n}g(x_i) +B\sqrt{\frac{\log(1/\delta)}{2n}}.$$ This directly bounds $\Gamma$ through empirical source differences and sampling error. It requires independent evaluation and a valid population bound; an observed sample maximum cannot simply be substituted for that bound. **4. Experimental evidence on transfer stability.** The conditions above specify when $\Gamma$ can be controlled. Experimentally, we test whether combination gains on fold models survive full model refitting. Even if worst-case $\Gamma$ is large, loss changes shared by two combinations can cancel. Let $R(q)=L(q)-\widehat L_{\mathrm{OOF}}(q)$, let $\Delta_0$ contain original-head combinations, and let $q_0^\star\in\arg\min_{q\in\Delta_0}L(q)$. Denote by $\widehat G_{\mathrm{OOF}}$ the optimal OOF loss for original heads minus that for all sources. The original proof's optimization conditions yield $$L(\widehat q)-L_0^\star \le \xi-\widehat G_{\mathrm{OOF}} +R(\widehat q)-R(q_0^\star).$$ The original worst-case bound is therefore a sufficient guarantee, while realized gains can also be analyzed through relative transfer between combinations. The additional fixed-model diagnostic directly tests whether convex-combination gains are retained. CE gain below is Output only CE minus LAMF CE in nats; positive values indicate improvement. Both full-model combinations use OOF-estimated parameters. Intervals are per-model paired bootstrap 95% intervals. | Dataset | Seed | Fold-model CE gain | Full-model CE gain | Full-model gain 95% interval | | --- | ---: | ---: | ---: | --- | | CREMA-D | 1001 | 0.0682 | 0.0483 | [0.0232, 0.0755] | | CREMA-D | 1002 | 0.0638 | 0.0360 | [0.0110, 0.0630] | | CREMA-D | 1003 | 0.0691 | 0.0591 | [0.0311, 0.0875] | | KS | 1001 | 0.1534 | 0.1114 | [0.0807, 0.1433] | | KS | 1002 | 0.1606 | 0.1012 | [0.0711, 0.1345] | | KS | 1003 | 0.1657 | 0.1121 | [0.0804, 0.1448] |
Response to Reviewer JxD5 (Part 1/5, W1: Propositions)
Thank you for recognizing the perspective of post-training information utilization and identifying the theoretical and architectural points requiring clarification. We address the mechanism, transfer guarantees, in-sample responses, and comparison fairness below.
W1: Scope of the theory and its connection to modality imbalance
1. The propositions connect to modality imbalance through asymmetric fitting. The three propositions characterize information utilization in an imbalance setting, rather than the process by which a dominant modality emerges. Consider asymmetric fitting: one modality has very low training loss, while another has not reached the same degree of fit but may still contain information useful for final predictions. The propositions address whether this information has combination value, why in-sample estimation may underutilize it, and when it can improve deployment predictions.
Proposition 1: Weaker fitting does not imply an absence of complementary value. For an original-head combination z0, source sj has contribution A(sj;z0)=E[(eY−p0(X))⊤(sj(X)−z0(X))]. This depends on its correction of the current combination's residual, rather than the modality's standalone accuracy or convergence speed. Even for a modality that fits less well during training, a layer prediction satisfying A(sj;z0)>0 decreases expected cross entropy along that direction. This gives an explicit condition under which information learned by a weaker modality remains useful, without assuming that every weaker-modality prediction is effective.
Proposition 2: Loss-response attenuation can sustain asymmetric decision-stage utilization. Let sm be the original head of the earlier-fitting modality, q0 minimize training cross entropy over convex combinations of original heads, and z0 denote the corresponding scores. Write Ltr for mean training cross entropy. Since using sm alone is feasible, Proposition 2 and 1−p≤−logp imply, for a layer prediction satisfying osc(sj(xi)−z0(xi))≤Bj,
\le B_j\widehat L_{\mathrm{tr}}(q_0) \le B_j\widehat L_{\mathrm{tr}}(s_m).$$ With $B_j$ bounded, as this modality's training loss becomes small, the layer prediction's in-sample response is bounded by a correspondingly small quantity, **without requiring the other modality to fit equally well**. In this asymmetric-fitting setting, attenuated in-sample responses can underrepresent other modalities' out-of-sample complementary value characterized by Proposition 1, allowing asymmetric utilization to persist at the decision stage. **Proposition 2 rigorously characterizes the response-attenuation step of this pathway and provides a conditional explanation for persistent utilization asymmetry.** **Proposition 3: Sufficient conditions for modality information to yield deployment gains.** For the full model, a layer prediction satisfying Proposition 1's positive-contribution condition at the optimal original-head combination gives a positive ideal risk gain $G$. Proposition 3 further requires this gain to exceed estimation, transfer, and optimization errors before guaranteeing improvement over the optimal original-head convex combination. It thus distinguishes and connects available modality information and realized deployment improvements, rather than equating additional sources with guaranteed gains. **Together, the propositions characterize how complementary modality information reaches the final prediction through residual correction, loss conditions associated with asymmetric fitting, and deployment-error conditions.** Proposition 2 supplies the direct mathematical connection to asymmetric fitting; Propositions 1 and 3 characterize the remaining predictive value and the conditions for realizing it.