acceptodds
Under review as a conference paper at ICLR 2027

LAFE: Layer Augmented Fusion Estimation

Abstract

Multimodal learning frequently suffers from modality imbalance, which existing methods primarily study during model training. However, we find that modality imbalance may prevent final predictions from fully exploiting the class information already learned in modality representations. We observe that predictions from some intermediate layers correctly classify examples that no convex combination of the original unimodal and fusion head outputs can classify correctly. Nevertheless, we show that high confidence training fits attenuate training loss responses to these predictions under bounded score changes. These observations suggest that fully exploiting learned modality information requires considering both which predictions to fuse and how to estimate their combination. We propose Layer Augmented Multimodal Fusion (LAMF), which fits linear probes to intermediate representations of frozen encoders to complement the original classifier outputs. LAMF applies base models and their probes to samples excluded from their fitting and uses the resulting predictions to estimate a global combination of all prediction sources. The method preserves the base architecture and training objective, and inference uses only one full model and its probes. Extensive experiments demonstrate that LAMF outperforms current state-of-the-art methods on multiple tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal learning, Modality imbalance, Stacked generalization, Intermediate representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JxD5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies multimodal imbalance from the perspective of post-training information utilization. The authors point out that useful class information may already exist in the intermediate layer representations of each modality, but the final prediction head does not fully utilize this information. Based on this observation, the paper proposes LAMF. The authors conduct extensive theoretical analysis and experiments to demonstrate the effectiveness of the method proposed in this paper.

Critical strengths
  1. The research content and perspective are innovative. Multimodal imbalanced learning is a key technical approach to enhancing the performance of multimodal models. This study's approach from the perspective of post-training information utilization is also innovative.

  2. The authors conducted thorough theoretical derivations and comprehensive experimental validation to demonstrate the algorithm's effectiveness.

Critical weaknesses
  1. In my opinion, one of the most significant issues with this manuscript is that the paper’s overall narrative is built upon the concept of modal imbalance, and it posits that modal imbalance may prevent the learned modal information from being fully utilized in the final prediction. However, Propositions 1–3 do not explicitly mention modal imbalance, modal competition, dominant modalities, or weak modalities. From a mathematical perspective, these theories only consider a limited number of prediction sources and then investigate whether adding a new predictor can improve the current convex combination. This theory may also apply to other scenarios, such as different intermediate layers in a single-modal network. Therefore, could the authors provide further clarification on how to distinguish between the view currently supported in the manuscript—that “intermediate-layer predictions may contain complementary information not yet utilized by the final prediction head” (the view supported by the current theory)—and the common view in multimodal imbalanced learning that “modal imbalance leads to underutilization of already learned modal information”?

  2. There are some theoretical details that the author needs to clarify further. Please see the “Questions” section for more information.

Questions for authors
  1. Regarding Proposition 3: Equation (11) is used to express the deployment risk bound, where Γ\Gamma represents the difference between the fold models estimated using OOF and the final full-data model. I would like the authors to further explain why Γ\Gamma should be small during training. Is there a theoretical basis to support this view? Additionally, the risk consistency in Appendix A.5 requires that Γ→0\Gamma \to 0. Could the authors further clarify under what conditions we can expect Γ→0\Gamma \to 0? Can they provide a finite-sample bound and a stability analysis?

  2. Proposition 2 shows that when the training samples are fitted with high confidence, a bounded score perturbation may only cause a very small change in the training CE. Here, I would like the authors to further explain: Why does this phenomenon specifically support OOF estimation, rather than merely indicating that the CE on the training set has reached a plateau? There may be different reasons for the small response on the training set. For example, one possibility is that the intermediate-layer prediction does indeed contain valuable complementary information, but it cannot be identified by the CE due to an overly strong training fit; another possibility is that the prediction source itself is indeed redundant. Is there a theoretical criterion that can distinguish between these two scenarios?

  3. The accuracy metrics reported in Table 2 differ significantly from those reported in existing papers (such as OGM, MMPareto, DGL, etc.). Therefore, I reviewed the code uploaded by the authors. I found numerous inconsistencies in the model architecture compared to the code in existing works, such as additional Dropout layers. Therefore, when comparing the performance of different methods, did the authors ensure that the model architectures were standardized? If the model architectures differ, the reliability of the algorithm performance comparison will be significantly compromised.

8 comments
AuthorsOpenReview ↗

Response to Reviewer JxD5 (Part 1/5, W1: Propositions)

Thank you for recognizing the perspective of post-training information utilization and identifying the theoretical and architectural points requiring clarification. We address the mechanism, transfer guarantees, in-sample responses, and comparison fairness below.

W1: Scope of the theory and its connection to modality imbalance

1. The propositions connect to modality imbalance through asymmetric fitting. The three propositions characterize information utilization in an imbalance setting, rather than the process by which a dominant modality emerges. Consider asymmetric fitting: one modality has very low training loss, while another has not reached the same degree of fit but may still contain information useful for final predictions. The propositions address whether this information has combination value, why in-sample estimation may underutilize it, and when it can improve deployment predictions.

Proposition 1: Weaker fitting does not imply an absence of complementary value. For an original-head combination z0z_0, source sjs_j has contribution A(sj;z0)=E[(eY−p0(X))⊤(sj(X)−z0(X))]A(s_j;z_0)=\mathbb E[(e_Y-p_0(X))^\top(s_j(X)-z_0(X))]. This depends on its correction of the current combination's residual, rather than the modality's standalone accuracy or convergence speed. Even for a modality that fits less well during training, a layer prediction satisfying A(sj;z0)>0A(s_j;z_0)>0 decreases expected cross entropy along that direction. This gives an explicit condition under which information learned by a weaker modality remains useful, without assuming that every weaker-modality prediction is effective.

Proposition 2: Loss-response attenuation can sustain asymmetric decision-stage utilization. Let sms_m be the original head of the earlier-fitting modality, q0q_0 minimize training cross entropy over convex combinations of original heads, and z0z_0 denote the corresponding scores. Write L^tr\widehat L_{\mathrm{tr}} for mean training cross entropy. Since using sms_m alone is feasible, Proposition 2 and 1−p≤−log⁡p1-p\le-\log p imply, for a layer prediction satisfying osc⁡(sj(xi)−z0(xi))≤Bj\operatorname{osc}(s_j(x_i)-z_0(x_i))\le B_j,

\le B_j\widehat L_{\mathrm{tr}}(q_0) \le B_j\widehat L_{\mathrm{tr}}(s_m).$$ With $B_j$ bounded, as this modality's training loss becomes small, the layer prediction's in-sample response is bounded by a correspondingly small quantity, **without requiring the other modality to fit equally well**. In this asymmetric-fitting setting, attenuated in-sample responses can underrepresent other modalities' out-of-sample complementary value characterized by Proposition 1, allowing asymmetric utilization to persist at the decision stage. **Proposition 2 rigorously characterizes the response-attenuation step of this pathway and provides a conditional explanation for persistent utilization asymmetry.** **Proposition 3: Sufficient conditions for modality information to yield deployment gains.** For the full model, a layer prediction satisfying Proposition 1's positive-contribution condition at the optimal original-head combination gives a positive ideal risk gain $G$. Proposition 3 further requires this gain to exceed estimation, transfer, and optimization errors before guaranteeing improvement over the optimal original-head convex combination. It thus distinguishes and connects available modality information and realized deployment improvements, rather than equating additional sources with guaranteed gains. **Together, the propositions characterize how complementary modality information reaches the final prediction through residual correction, loss conditions associated with asymmetric fitting, and deployment-error conditions.** Proposition 2 supplies the direct mathematical connection to asymmetric fitting; Propositions 1 and 3 characterize the remaining predictive value and the conditions for realizing it.
AuthorsOpenReview ↗

Response to Reviewer JxD5 (Part 2/5, W1: Scope and Mechanism)

2. The theory applies to general prediction sources; the multimodal analysis examines modality and layer differences. The propositions hold for finitely many prediction sources and also apply to unimodal networks; being multimodal is not an assumption required for their validity. This general formulation separates additional combination value, its identification through in-sample estimation, and its preservation at deployment. In multimodal settings, we examine whether predictions from different modalities and layers complement the original combination, how modality-specific fitting differences affect their utilization, and whether fixed-model interventions improve their use in final decisions.

3. The existence of complementary value and the mechanism of underutilization are distinct.

Unused complementary information describes a phenomenon in which final predictions underutilize learned representations; attributing that underutilization to modality imbalance additionally specifies a cause. Prior work analyzes how optimization imbalance and modality competition during joint training limit the learning of some modalities (Peng et al., 2022; Huang et al., 2022). We further study information already learned: whether complementary predictions from different modalities enter the final combination and how fitting differences affect their utilization through in-sample combination estimation. Accordingly, decision-stage utilization asymmetry is an outcome we seek to explain and mitigate, with an analysis of how it may persist.

For unused complementary information, we provide both existence and realized-gain evidence. Figure 1(a) identifies samples that no convex combination of the original unimodal and fusion head outputs can classify correctly, but layer predictions can. This establishes complementary value relative to that combination family. Table 4 freezes OGM-GE, DGL, and TCMax models and reports accuracy gains of 0.54–1.08% from layer predictions over Output only, showing that this value can enter actual decisions. Proposition 1 provides conditions for decreasing expected CE; classification complementarity is tested experimentally, not inferred from a CE derivative alone.

For decision-stage modality imbalance, we analyze how fitting differences affect modality utilization through combination estimation. Proposition 2 implies that sufficiently fitting one modality can yield low training loss for the original combination and constrain its response to additional predictions. In-sample combination estimation may consequently underrepresent remaining out-of-sample complementary value. To test an intervenable step, original Appendix E.2 fixes fold models, layer predictor configurations, and scales, changes only the estimation records, and evaluates on the same independent set. The following are accuracy gains (%) from layer predictions over the respective Output only combinations.

Combination estimation recordsCREMA-DKS
In-sample+0.231 ± 0.2010.000 ± 0.000
Out-of-fold+1.697 ± 0.694+1.503 ± 0.203

The same layer predictions produce larger gains with out-of-sample estimation, distinguishing available complementary information from its utilization by the combination. Our progression is to establish unused predictive value, characterize conditions under which in-sample responses can mask it under asymmetric fitting, and improve its utilization through out-of-sample combination estimation. Certificates and fixed-model results establish complementary value; mathematical analysis and controlled interventions test parts of the underutilization mechanism, rather than deriving a causal conclusion from general ensemble gains.

AuthorsOpenReview ↗

Response to Reviewer JxD5 (Part 3/5, W2 / Q1)

W2 / Q1: Stability conditions and finite-sample bounds for transfer error

1. Prediction-source stability controls the size of Γ\Gamma.

The original proposition does not require Γ\Gamma to decrease automatically with training epochs. In Equation (11), Γ\Gamma measures the expected-loss difference between fold models and the full model under the same combination parameters, not a model's training error. Further reducing training loss does not directly control it. The relevant theoretical basis is stability of prediction sources under refitting. With identically distributed training and test samples and all models and probes fixed, the oscillation bound for cross entropy gives

\mathbb E\max_j\operatorname{osc}\!\left(s_j(X;F_k)-s_j(X;F)\right), \qquad \omega_k=|\mathcal D_{3-k}|/N.$$ Here $\operatorname{osc}(v)=\max_c v_c-\min_c v_c$, and $s_j$ includes the deployed scales. This holds uniformly over convex combinations because their score changes are bounded by the maximum oscillation across sources. More stable corresponding sources therefore yield a smaller upper bound on $\Gamma$, directly connecting stability to Proposition 3. **2. $\Gamma\to0$ requires convergence of corresponding prediction sources.** This concerns increasing sample size, not increasing training epochs. A sufficient condition is a fixed number of sources, identically distributed training and test samples, and corresponding score functions trained on half and full datasets converging to a common limit with vanishing expected oscillation above. With frozen encoders, probe refitting can satisfy such consistency under nondegenerate standardization, appropriate moment conditions, and fixed positive ridge regularization. Retraining encoders additionally requires prediction stability and scale control. Thus, Appendix A.5 is a consistency result under these conditions; OOF does not replace the stability assumptions. **3. Independent evaluation provides a conditional finite-sample bound.** A conditional upper bound follows from the same stability quantity. Define $$g(X)=\sum_{k=1}^{2}\omega_k\max_j\operatorname{osc}\!\left(s_j(X;F_k)-s_j(X;F)\right).$$ If models, probes, and scales are fixed before evaluation and a valid population bound $0\le g(X)\le B$ holds, one-sided Hoeffding gives, on $n$ independent identically distributed evaluation samples, with probability at least $1-\delta$, $$\Gamma\le \mathbb E g(X) \le \frac1n\sum_{i=1}^{n}g(x_i) +B\sqrt{\frac{\log(1/\delta)}{2n}}.$$ This directly bounds $\Gamma$ through empirical source differences and sampling error. It requires independent evaluation and a valid population bound; an observed sample maximum cannot simply be substituted for that bound. **4. Experimental evidence on transfer stability.** The conditions above specify when $\Gamma$ can be controlled. Experimentally, we test whether combination gains on fold models survive full model refitting. Even if worst-case $\Gamma$ is large, loss changes shared by two combinations can cancel. Let $R(q)=L(q)-\widehat L_{\mathrm{OOF}}(q)$, let $\Delta_0$ contain original-head combinations, and let $q_0^\star\in\arg\min_{q\in\Delta_0}L(q)$. Denote by $\widehat G_{\mathrm{OOF}}$ the optimal OOF loss for original heads minus that for all sources. The original proof's optimization conditions yield $$L(\widehat q)-L_0^\star \le \xi-\widehat G_{\mathrm{OOF}} +R(\widehat q)-R(q_0^\star).$$ The original worst-case bound is therefore a sufficient guarantee, while realized gains can also be analyzed through relative transfer between combinations. The additional fixed-model diagnostic directly tests whether convex-combination gains are retained. CE gain below is Output only CE minus LAMF CE in nats; positive values indicate improvement. Both full-model combinations use OOF-estimated parameters. Intervals are per-model paired bootstrap 95% intervals. | Dataset | Seed | Fold-model CE gain | Full-model CE gain | Full-model gain 95% interval | | --- | ---: | ---: | ---: | --- | | CREMA-D | 1001 | 0.0682 | 0.0483 | [0.0232, 0.0755] | | CREMA-D | 1002 | 0.0638 | 0.0360 | [0.0110, 0.0630] | | CREMA-D | 1003 | 0.0691 | 0.0591 | [0.0311, 0.0875] | | KS | 1001 | 0.1534 | 0.1114 | [0.0807, 0.1433] | | KS | 1002 | 0.1606 | 0.1012 | [0.0711, 0.1345] | | KS | 1003 | 0.1657 | 0.1121 | [0.0804, 0.1448] |
Reviewer_MStjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether multimodal predictors fully exploit the class information already present in their learned representations. It distinguishes two issues: which prediction sources are available for fusion, and how their combination is estimated. LAMF fits linear probes to frozen modality representations and combines their predictions with existing classifier outputs. Two-fold out-of-fold (OOF) predictions are used to select probe configurations, calibrate score scales, and estimate global combination parameters. The full model and probes are then refitted on all training data; inference requires only this full model and its probes. The paper provides conditional loss and deployment-risk analyses, evaluates eight tasks, and includes fixed-model comparisons and controls for alternative explanations. The MOSI/MOSEI main results use input-feature predictors rather than the same intermediate-layer construction used elsewhere.

Critical strengths
  1. A useful diagnostic distinction supported by controlled evidence. The paper examines whether limitations arise from the available predictions or their combination, rather than attributing all shortcomings to encoder training. Intermediate-layer probes correctly classify some examples that no convex combination of the original heads can classify correctly. More importantly, fixed-model comparisons show realized gains beyond output-only combination: Table 4 reports improvements of 0.54–1.08 percentage points across several base training methods, and Table 12 reports paired three-seed layer-source gains of 1.22 ± 0.10 pp on Food-101 and 3.60 ± 1.28 pp on MVSA (mean ± sample standard deviation). This makes the contribution more substantive than an improvement in aggregate benchmark scores alone.

  2. An implementable approach with substantial validation of its interpretation. The procedure preserves the base training objective and deploys a single full model with probes. The independent evaluation, calibration controls, correction-versus-destruction analysis, and refitting checks strengthen the empirical case that additional prediction sources can provide useful information. Although the components are familiar, their controlled investigation offers a reusable way to analyze and improve multimodal predictors.

Critical weaknesses
  1. The methodological novelty is limited, and the comparison with the closest approach requires careful attribution. LATES already combines intermediate-layer probes with the original classifier using weights learned on held-out data, while OOF combination estimation follows classical stacking. The paper appropriately cites these connections and includes comparisons. However, the LATES adaptation in Appendix C.2 changes several choices together, including probe fitting, combination constraints, and estimation records. Its overall performance gap cannot establish the individual value of OOF estimation. Appendix D.1 provides more targeted evidence, but the paper should distinguish which factors are controlled and which remain confounded. If these experiments cannot separately identify the benefit of OOF, the contribution should be attributed to the complete procedure rather than to OOF alone. This limits the strength of the algorithmic contribution.

  2. The evidence supports a narrower mechanism than a general explanation of modality imbalance. Unused intermediate-layer information and unreliable in-sample combination estimation can also occur in single-modality models; the reported findings do not by themselves establish modality imbalance as their cause. The convex-combination certificate also applies only to a restricted output-combination family. Table 23 attributes 0.18/0.22 pp of net accuracy gain to the certified subset on CREMA-D/KS, versus 1.25/1.17 pp overall. Since the certificate provides a sufficient rather than necessary condition for identifying complementary intermediate-layer information, its limited coverage does not imply that this mechanism is unimportant outside the certified subset. It limits only how much of the observed gain is directly explained by the certificate; this is an interpretation boundary, not evidence against the mechanism. The MOSI/MOSEI main settings also use input-feature predictors, so they should not be counted as identical evidence for the intermediate-layer mechanism. These limitations do not invalidate the gains, but they constrain the breadth of the paper’s explanatory contribution.

Questions for authors
  1. Clarification of incremental contribution: Please clarify which factors are controlled and which remain confounded in Appendices C.2 and D.1 when comparing OOF and held-out estimation, including prediction sources, probe configurations, score scaling, and combination rules. If the existing experiments cannot separately identify the benefit of OOF, please state this limitation and distinguish it from the benefit of the complete procedure.

  2. Clarification of the central claim: Is modality imbalance intended as a demonstrated cause of the observed information-utilization problem, or as motivation for investigating it? If a causal interpretation is intended, which existing evidence distinguishes it from the more general benefits of intermediate-layer ensembles? The narrower interpretation is supported more directly by the reported evidence.

5 comments
AuthorsOpenReview ↗

Response to Reviewer MStj (Part 1/3, W1 / Q1)

Thank you for recognizing the diagnostic distinction, fixed-model evidence, and deployment procedure, and for the specific suggestions on attribution and interpretation.

W1 / Q1: Contribution relative to LATES and stacking, and controlled comparison factors

Our contribution distinguishes two aspects of multimodal information utilization: the available prediction sources and the estimation of their combination. We analyze conditions governing in-sample responses and deployment gains, and use fixed-model comparisons to examine how these aspects jointly affect final predictions. LAMF implements this analysis without permanently reserving a calibration set and deploys only one full model and its probes. LATES combines intermediate layer probes and final classifier outputs using an independent calibration set to improve image classification and calibration; stacking estimates a higher-level combination from out-of-sample predictions. We further study the predictive value of different modalities and layers and its dependence on combination estimation. The original related work and Section 3.3 acknowledge these connections and distinctions with LATES, stacked generalization, and Super Learner.

Appendix C.2 compares the complete performance of two closely related methods, rather than isolating a single component. Layer positions, source types, and the evaluation set are matched. Probe fitting, configurations and scales, combination constraints, and estimation records follow the respective methods, so this comparison evaluates the complete procedures.

Appendices D.1 and E.2 provide additional controlled evidence on OOF estimation. First, out-of-sample estimation produces larger realized gains from the same layer predictions. Original Appendix E.2 fixes fold models, predictor configurations, and scales and evaluates on the same independent set. Adding layer predictions gives accuracy gains of only +0.23%/0.00% on CREMA-D/KS with in-sample estimation, versus +1.70%/+1.50% with out-of-sample estimation. Thus, improvement depends not only on adding predictors but also on estimating their combination. Second, Appendix D.1 compares OOF and independent calibration on the same base models and probes, fixing source definitions, probe configurations, scales, and the combination family, with both original outputs and audio/visual layer predictions included. Original Table 14 reports the following test accuracy differences (%, OOF minus independent calibration).

CombinationCREMA-DKS
Convex+0.69 ± 0.58+0.43 ± 0.52
Linear stacking+2.42 ± 1.50+1.98 ± 0.15

OOF achieves higher mean accuracy in these comparisons, supporting LAMF's combination estimation procedure. Appendix E.2 further shows that realized gains from identical layer predictions depend on the estimation records. LAMF also avoids permanently reserving a calibration set while retaining single-model deployment with probes.

AuthorsOpenReview ↗

Response to Reviewer MStj (Part 2/3, W2 / Q2: Mechanism)

W2 / Q2: Modality imbalance and the distinction from general intermediate layer ensemble gains

Unused intermediate layer information and unreliable in-sample combination estimation can also occur in unimodal models, depending on the model and task. We study how these issues affect the utilization of different modalities in multimodal settings. Existing imbalance methods primarily improve training, whereas our analyses show that even with these methods, final predictions can favor one modality's decisions and underutilize useful information already learned by others. We therefore treat decision-stage asymmetry in modality utilization as an outcome to explain and address, rather than presupposing modality imbalance as the cause of these phenomena.

Our proposed mechanism is that a mismatch between the depth-wise distribution of modality information and the final combination can sustain asymmetric utilization. Useful predictions need not be concentrated at the final layer. If the original combination already fits training samples with high confidence, training loss responses to additional layer predictions are attenuated, making their complementary value difficult to capture through in-sample records. This is a possible pathway, not a claim that different layer preferences necessarily cause imbalance. The following evidence examines prediction sources, estimation records, and final decisions.

First, useful combination information is not concentrated at the same depth across modalities. Original Appendix E.2 fixes models, configurations, and scales and measures accuracy gains (%) from adding a single layer predictor to Output only on the same independent evaluation set.

Added prediction sourceCREMA-DKS
Audio L2+1.255 ± 0.595+0.505 ± 0.314
Audio L4+0.030 ± 0.783−0.444 ± 0.245
Visual L4+0.964 ± 0.291+1.042 ± 0.158

Audio L2 and visual L4 provide positive mean gains, whereas audio L4 provides little or negative gain. Layer value must therefore be assessed by modality; adding arbitrary predictors is not equally effective.

Second, estimation records affect whether the same layer predictions realize gains. Original Appendix E.2 fixes fold models, predictor configurations, and scales and uses the same independent evaluation set. Relative to the respective Output only combinations, the gains on CREMA-D/KS are +0.23%/0.00% with in-sample estimation and +1.70%/+1.50% with out-of-sample estimation. Realized value thus depends on the records used to estimate the combination, not merely on the availability of layer predictions.

AuthorsOpenReview ↗

Response to Reviewer MStj (Part 3/3, W2 / Q2: Evidence and Scope)

Third, decision-stage intervention reduces asymmetry in modality utilization. The following excerpt from revised Table 7 reports accuracy (%) on fixed-model test subsets defined by the original branches: AO means audio only correct, and VO means visual only correct.

CombinationCREMA-D AOCREMA-D VOOverallKS AOKS VOOverall
Native fusion head95.6117.8871.5595.3348.4269.24
OOF Output only77.3787.2884.4185.7183.3676.57
LAMF82.3087.2385.6688.4983.3077.74

Compared with native fusion, the OOF output combination narrows the accuracy gap between the two subsets; adding layer predictions further improves overall accuracy. Additional checks on the frozen OGM-GE, DGL, and TCMax models in Table 4 likewise show smaller subset gaps and higher overall accuracy.

Beyond a general ensemble explanation based on overall accuracy, we test dependence on estimation records and the utilization of different modalities in final predictions. With layer predictions fixed, out-of-sample records yield larger mean gains. Modality-disagreement subsets show that the complete procedure reduces the native fusion head's asymmetric use of correct modality predictions. Output only already accounts for substantial rebalancing, and layer predictions further improve overall performance. These findings support the specific mechanism that prediction sources and combination estimation jointly affect modality information utilization; this can coexist with general ensemble benefits.

W2: Scope of the convex-combination certificate

The certificate establishes evidence unavailable from mean accuracy alone: some samples cannot be correctly classified by any convex combination of the original outputs, yet can be correctly classified by layer predictions. It directly establishes complementary predictive value relative to this combination family. Original Table 23 separates potential opportunities from realized utilization: the certified subset contributes net accuracy gains of 0.18%/0.22% on CREMA-D/KS. Fixed-model source additions and independent evaluation examine realized gains more broadly. Certification is sufficient, not necessary, so uncertified samples need not lack complementary value, and the certified contribution is not an estimate of the total effect of layer predictions. We have clarified this distinction in revised Appendix E.1, following Table 25: the certificate establishes particular correction opportunities; combination experiments evaluate realized gains. Both support the method without requiring the certificate to explain every gain. Its strict scope is the convex-combination family of original outputs.

W2: Evidence from CMU-MOSI and CMU-MOSEI

The sentiment tasks test the applicability of additional prediction sources and out-of-sample combination estimation beyond a single network setting. Original Appendices B.3 and C.1 describe the fixed input-feature predictors with the fusion head retained; Appendix E.3 separately evaluates Transformer intermediate layer predictions. In the same-model CMU-MOSI comparison, the input-feature variant gains 4.81 ± 2.15% over Output only, versus 2.53 ± 3.00% for intermediate layer augmentation, supporting this task adaptation. We use the sentiment tasks to support applicability of the overall procedure, and the other six tasks together with the intermediate layer experiments in E.3 to support the layer prediction mechanism. This separates the roles of the evidence without treating different predictor constructions as identical mechanism tests.

Reviewer_8HDDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Layer Augmented Multimodal Fusion (LAMF) to improve the utilization of information learned by multimodal models. It fits linear probes to intermediate representations of frozen encoders and combines their predictions with the existing unimodal and fusion heads. Two-fold out-of-fold (OOF) predictions are used to select probe configurations and estimate prediction scales and combination weights. The probes are subsequently refitted on the full training data, while inference uses one full model and its probes. Experiments cover eight classification tasks and include analyses of intermediate-layer complementarity, combination estimation, and deployment transfer.

Critical strengths

An interesting approach to a meaningful multimodal learning problem. LAMF examines whether the final prediction fully exploits information already learned at different depths, rather than addressing imbalance solely through encoder optimization. Intermediate-layer predictors can provide complementary evidence without redesigning the base encoders, making the approach potentially useful alongside existing training methods.

Useful controlled analyses and a practical deployment procedure. The fixed-model comparisons separate gains from output reweighting and from adding layer predictions. The supplementary independent-evaluation and calibration controls further support the value of aligned layer predictions. Deployment retains only one full model and its probes, avoiding the need to run both fold models at inference.

Critical weaknesses

The robustness and extent of the performance advantage need more careful assessment. The method is competitive, but does not consistently outperform the strongest baselines. On AVSBench, convex LAMF achieves 92.39%, only 0.05 percentage points above IBML at 92.34%, and below DGL at 94.14%. Appendix Tables 10–11 already report standard deviations: 0.21 for convex LAMF and 0.39 for IBML in this comparison. These values do not by themselves establish statistical significance or its absence, but show why the mean difference should not be emphasized without a paired analysis. The main tables should include uncertainty or directly reference it, and the authors should distinguish substantive improvements from near-ties.

The connection between prediction improvements and mitigation of modality imbalance remains insufficiently explained. On the CMU-MOSEI dataset, Text and Audio outperform all imbalance-mitigation methods, which is inconsistent with the results on other datasets. The authors are encouraged to discuss this discrepancy.

Minor presentation issues:

  • Lines 34–35 place alternating learning alongside gradient modulation and decoupled optimization without making the organizing criterion clear. Alternating versus joint learning describes a training schedule or paradigm, whereas gradient modulation describes an optimization mechanism; these are not mutually exclusive categories. Please clarify the taxonomy rather than presenting them as directly parallel categories.
  • Some references should be added to the introduction of alternating learning on Line 34.
  • Figure 2 could use a schematic face or a generic visual-modality icon instead of a face photograph. This would convey the same information without requiring an identifiable example.
  • Line 364 uses “MOSI,” whereas Table 3 uses “CMU-MOSI.” Please standardize the dataset naming throughout the manuscript.
Questions for authors
  • On CMU-MOSEI, how much do audio and visual predictors contribute beyond the text predictor and fusion head? Please explain the modality-specific performance gap and distinguish the input-feature replacement variant from intermediate-layer augmentation when interpreting these results.
  • The authors should provide a comparative analysis against the baselines, as the current results do not sufficiently validate the effectiveness of the proposed method.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 8HDD (Part 1/3, W1 / Q2)

Thank you for recognizing our decision-stage perspective on modality information utilization, the fixed-model analyses, and the single-model deployment procedure. We respond to each concern below.

W1 / Q2: Robustness of the performance advantage and comparisons with baselines

We agree that improvements, near-ties, and statistical significance should be distinguished. The following table combines the three AVSBench seeds in the original submission with two additional seeds and reports the five-seed mean ± sample standard deviation. Baseline results remain those reported in the original submission.

MethodSeedsAccuracy (%)
LAMF (convex)591.78 ± 1.39
LAMF (linear stacking)592.57 ± 0.83
IBML392.34 ± 0.39
DGL394.14 ± 0.08

Across five seeds, LAMF with linear stacking performs similarly to IBML and below the higher-performing baseline DGL, consistent with the original observations. The original accuracy difference of 0.05% over IBML is a near-tie, not evidence of a robust advantage. In the revised manuscript, the Table 2 caption now directs readers to Appendix C.1 for variability estimates, and Table 13 reports the five-seed results above. Comparisons of summaries with different seed counts are not used to claim paired statistical significance.

Performance on AVSBench. In Table 2, the visual branch accuracy is 54.00% for convex LAMF, versus 59.64% for IBML and 64.14% for DGL. LAMF improves the utilization of fixed representations without reoptimizing the encoders, so it need not close representation gaps produced by different training methods. This is an explanation consistent with the results, rather than an isolated causal finding, and illustrates how LAMF can complement training-stage improvements.

The advantages on other tasks remain. In original Tables 2 and 3, the two LAMF implementations collectively attain the highest mean fusion accuracy on seven tasks: linear stacking leads on five, and convex aggregation leads on UPMC-Food101 and CMU-MOSI. The accuracy differences are 1.97% over AUG on CREMA-D and 1.41% over IBML on UPMC-Food101. Thus, LAMF's performance advantage does not depend on the small AVSBench mean difference: it is competitive across tasks and achieves state-of-the-art performance on multiple datasets among the compared methods.

The original submission also provides direct comparisons on the same models. Table 4 freezes models trained by OGM-GE, DGL, and TCMax on CREMA-D and Kinetics-Sounds (KS); adding layer predictions improves on combining only the original outputs in each case. Original Table 12 reports paired three-seed layer gains of 1.22 ± 0.10% and 3.60 ± 1.28% on UPMC-Food101 and MVSA-Single, respectively, with the main-table models fixed. These controls directly establish decision-stage gains rather than substituting cross-method rankings for component validation.

AuthorsOpenReview ↗

Response to Reviewer 8HDD (Part 2/3, W2)

W2: Independently trained unimodal models outperform the corresponding branches on CMU-MOSEI

Yes, this occurs for several compared methods, including MMPareto, AUG, and LAMF. The relevant unimodal comparison in Table 3 concerns branch accuracy, not final fusion accuracy. Independently trained unimodal models outperform the corresponding branches of these multimodal methods.

Our explanation concerns the input setting and training objectives. As described in original Appendix B.1, CMU-MOSEI uses fixed, pre-extracted features, including BERT text representations. These methods modify downstream training but cannot update the upstream feature extractors; gradient-balancing strategies therefore do not guarantee that existing differences in modality predictive ability disappear. Moreover, independently trained models optimize unimodal classification, whereas joint training also serves the fusion objective. Better fusion need not improve every branch. These factors help explain why this task differs from tasks with end-to-end audio–visual representation learning.

LAMF aims to improve decision-stage information utilization, not to make each branch outperform independent training. Under this shared input setting, convex LAMF and linear stacking achieve final accuracies of 75.90% and 76.02%, both above the independent text model's 75.00%. The fixed-model ablations below further show positive mean gains from audio and visual predictors beyond the text predictor and fusion head.

AuthorsOpenReview ↗

Response to Reviewer 8HDD (Part 3/3, Q1 and Minor Issues)

Q1: Modality-specific contributions and the input-feature predictor variant

1. Audio and visual contributions beyond the text predictor and fusion head.

On CMU-MOSEI, we fix the three model states used for main-table linear stacking and refit the combination for each prediction set, using the requested text predictor plus fusion head as the reference. The table reports paired accuracy differences (%).

Added predictorsConvexLinear stacking
Audio only+0.36 ± 0.04+0.16 ± 0.50
Visual only+0.56 ± 0.25+0.43 ± 0.27
Audio and visual+0.59 ± 0.15+0.71 ± 0.11

The combined audio–visual gain is positive for all three seeds under both combination methods. Adding visual predictors alone gives the larger mean gain. Final linear stacking accuracy increases from 75.32% to the main-table result of 76.02%. Since the fusion head already uses all modalities, these are incremental contributions of the additional predictors, not the total contributions of audio and vision.

2. Modality-specific performance differences. On CMU-MOSEI, the standalone text input-feature predictor reaches 73.73%, indicating stronger individual predictive ability, but lower standalone accuracy does not imply an absence of complementary value. Visual predictors provide a larger mean incremental gain than audio predictors; audio-only addition under linear stacking is more variable, while their joint gain is the most stable. Both standalone performance and incremental value relative to the existing combination therefore matter; equal gains across modalities are not required.

On the same CMU-MOSEI task, we also partition test samples by the correctness of the jointly trained branches. On samples where vision is correct but text is wrong, and where audio is correct but text is wrong, native fusion accuracy is 16.9% and 13.1%, respectively; LAMF with linear stacking raises these to 38.1% and 33.3%. Accuracy decreases on the opposite-disagreement subsets, but overall accuracy increases from 73.30% to 76.02%. This supports more effective decision-stage utilization of existing predictive evidence.

3. Rationale for the input-feature predictor variant. For CMU-MOSI and CMU-MOSEI, input-feature predictors directly access class information in pre-extracted representations without making every prediction source depend on the downstream Transformer transformations. Since the inputs are already modality features, we can fit linear predictors directly while retaining the fusion head's jointly learned predictions. These provide direct feature-based and downstream jointly learned prediction paths, whose combination is estimated from OOF records. Original Appendix B.3 describes this prediction set. The same-model CMU-MOSI comparison in Appendix E.3 supports its practical value: the input-feature variant yields the larger mean gain over Output only, as reproduced below.

Prediction setAccuracy gain (%)
Original heads + Transformer intermediate layer predictors+2.53 ± 3.00
Fusion head + input-feature predictors (our variant)+4.81 ± 2.15

Minor presentation issues

  1. Taxonomy of alternating learning and gradient modulation. In the first paragraph of revised Section 1, we distinguish alternating updates as a training-schedule choice from gradient modulation and decoupling as optimization mechanisms, and explicitly state that these choices can coexist rather than forming mutually exclusive categories.
  2. References for alternating learning. The corresponding passage in revised Section 1 cites ReconBoost (Hua et al., ICML 2024) and MLA (Zhang et al., CVPR 2024).
  3. Face photograph in Figure 2. We have replaced it with a generic video icon in the revised figure.
  4. Dataset naming. In the backbone and input description in revised Section 4.1, we use CMU-MOSI and CMU-MOSEI consistently with Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.