acceptodds
Under review as a conference paper at ICLR 2027

Efficient Multimodal Fusion for Data-Constrained Regimes

Abstract

Multimodal classification often requires substantial paired data to learn how to combine heterogeneous representations from different modalities. We consider a streamlined regime in which modality-specific encoders are frozen and adaptation operates directly within their pretrained representation spaces. Our method, Factorized Modality Evidence targets multimodal fusion in data-constrained settings through a factorized multimodality approach. We model class structure independently within each modality and combines the resulting evidence in the shared label space. Each modality contributes a Gaussian discriminant score, while a single scalar coefficient calibrates the modality evidence and class prior. This avoids representation-level alignment, encoder updates, and high-dimensional learned fusion, while naturally supporting missing modalities by omitting unavailable evidence terms. Across six datasets and eleven classification tasks, FaME outperforms the strongest competing baseline in 36 of 42 few-shot dataset–shot settings. Under test-time modality missingness, our method consistently outperforms standard imputation and modality-dropout baselines. When few-shot adaptation data contain missing modalities, FaME achieves the strongest performance across all 18 settings. These results show that lightweight factorized adaptation can effectively exploit complementary information from independently pretrained modalities without learning a high-capacity multimodal fusion model. Our code is available at: https://anonymous.4open.science/r/FaME-66F3/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal classification, gradient-free adaptation, few-shot learning, missing modalities

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GDxpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FaME, a lightweight classifier for multimodal tasks with limited labeled data. It fits a Gaussian discriminant model separately to each frozen modality representation and combines their class evidence through a prior-corrected product rule. Missing modalities are handled by omitting their evidence. The authors evaluate the method on six image–text datasets under few-shot supervision and simulated modality missingness.

Critical strengths
  1. The method is simple, inexpensive to adapt, and works with independently pretrained encoders whose representation spaces differ.
  2. The evaluation covers multiple datasets, encoder pairs, and data-scarcity settings. The few-shot results are strong relative to both learned fusion and encoder-adaptation baselines.
  3. The ablations include useful controls, particularly unimodal classifiers, alternative fusion rules, and covariance estimators.
Critical weaknesses
  1. The methodological contribution is limited. FaME combines established modality-specific GDA classifiers with a classical prior-corrected product rule. Under the shared-covariance assumption, its prediction reduces to summing modality-specific discriminant scores with a class-prior term. It introduces neither a learned cross-modal interaction nor a mechanism for estimating the relative reliability of different modalities. The contribution therefore appears primarily empirical, and it is unclear whether this combination constitutes a sufficiently substantial technical advance.
  2. The role of the shared coefficient α is not supported by the stated formulation. With a uniform multiclass prior, any positive shared α leaves the predicted class unchanged; for multilabel tasks, it also preserves the per-label rankings used to compute average precision. Nevertheless, Table 13 reports different results for the shared-α and α=1 variants. This suggests a mismatch between the mathematical formulation, experimental protocol, and implementation, and weakens the claimed role of α as a fusion calibration parameter.
  3. The CUB few-shot protocol cannot be implemented as currently described. CUB has 200 classes but only 5,000 training and 994 validation examples. Constructing 64-shot training or even 8-shot validation sets would require more examples than are available unless an unstated sampling or truncation rule is used. This ambiguity affects both the interpretation and reproducibility of the few-shot results.
Questions for authors
  1. What is the specific technical contribution of FaME beyond applying existing GDA estimators and a classical prior-corrected product rule to frozen multimodal representations?
  2. What causes the shared-α and α=1 variants to differ in Table 13 under the stated equations and evaluation metrics? Were they evaluated using exactly the same support sets and missingness masks?
  3. How are the CUB training and validation sets constructed when k times the number of classes exceeds the available examples? Please report the sampling rule and the number of distinct examples used at each budget.
  4. Could the authors reconcile the Appendix F.3 discussion with Table 24? The table entries give an average of approximately 0.484 for mean late fusion at r=0.75, rather than the stated 0.522, and FaME is higher on six rather than four encoder pairs.
  5. How robust is FaME to imbalanced few-shot support sets or label-prior shift between the support and test sets? The balanced few-shot protocol makes the empirical prior uniform and does not evaluate this dependence.
Reviewer_aA5VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Factorized Modality Evidence (FAME), a lightweight method for multimodal classification with independently pretrained, frozen encoders. FAME fits a regularized Gaussian discriminant classifier in each modality’s representation space and combines the resulting class evidence in label space using a prior-corrected product rule. A single coefficient, selected on the validation set, balances the combined evidence against the empirical class prior. The method requires no representation alignment, encoder updates, or high-dimensional fusion network. When a modality is missing, its evidence is simply omitted; incomplete adaptation data can also be used separately by each modality classifier. The evaluation covers six image-text datasets, eleven tasks, multiple encoder pairs, seven few-shot budgets, test-time missingness, and incomplete few-shot adaptation data. Results are strong compared with mean late fusion, learned fusion heads, LoRA, and full fine-tuning, particularly in low-data settings.

Critical strengths
  1. FAME is simple and computationally efficient. It combines heterogeneous frozen encoders without a shared representation space and handles missing modalities by omitting their evidence, avoiding imputation, retraining, or architectural changes. This is useful when paired data and computational resources are limited.

  2. The evaluation covers multiple datasets, tasks, encoder pairs, shot counts, and missingness rates. It also reports full-data and unimodal controls, per-encoder results, computational costs, covariance and fusion ablations, and degenerate few-shot cases. The appendix clarifies the implementation and the assumptions behind the probabilistic interpretation.

Critical weaknesses
  1. FAME combines established regularized Gaussian discriminant analysis with a prior-corrected product rule. Its contribution is mainly their application to frozen, independently pretrained encoders and the supporting experiments, rather than a new fusion principle. The paper should clarify how it differs from earlier Gaussian-classifier and product-rule methods.

  2. The experiments consider random missingness, remove at most one of two modalities, and use matched validation and test missingness rates. Although the paper acknowledges selection bias under informative missingness, it does not test class-dependent missingness, validation–test shifts, or a specialized incomplete-modality baseline under the same frozen-encoder and data-budget protocol. This limits the scope of its conclusions.

  3. FAME uses a shared coefficient and cannot down-weight an unreliable modality for individual samples. Despite comparisons with GMU and separate global coefficients, the paper does not analyze modality agreement and disagreement or test confidence-aware per-sample weighting. This would help identify when a confident but incorrect modality dominates the prediction.

Questions for authors
  1. The evaluation includes imputation, modality dropout, ASIF, and GMU. Could the authors also compare FAME with a specialized incomplete-modality method using the same frozen encoders and data budget?

  2. Since independent fitting does not address selection bias under informative missingness, could the authors test class-dependent missingness or different missingness distributions for validation and testing?

  3. Because the MIMIC-CXR labels are derived from the same reports used as text inputs, could the authors report results excluding this dataset?

Reviewer_PPmKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors proposed a method named FAME to use Gaussian Discriminant Analysis for multimodal data fusion. Specifically, the method fits a separate statistical bell-curve model to the image and to the text and gives an evidence score for each category. Then, they add the image and text scores and multiple the sum by a scalar alpha. The authors applied their model to 6 different datasets and demonstrate a better prediction performance compared to the other baseline models.

Critical strengths
  1. The method is simple, determinism, and computational efficient as it relies on closed-form Gaussian discriminant statistics for each modality. It does not require gradient optimization, learning-rate schedules.
  2. The method does not require data imputation. Since FAME trains a separate statistical model for images and separate one for text, it can learn from every available image and text during training and skip whichever is missing during testing.
Critical weaknesses
  1. Although the method is clean, interpretable, and efficient, I didn't see a large novelty of the method. The method overlaps with Kittler et al. (https://ieeexplore.ieee.org/document/667881) and Wang et al. (https://arxiv.org/abs/2402.04087). The difference between the proposed method and previous methods relies on the alpha. However, the main few-shot results do not really demonstrate a benefit from alpha. For example, in table 11, few-shot ablation, the FaME tuned on validation data shows the same performance as the ablation of alpha=1.
  2. The role of alpha in full data regimes is confounded by scaling artifact in eq. 3 in full-data multiclass settings with imbalanced priors, where tuning alpha improves performance from 0.756 to 0.807. But this gain is confounded by how the precision estimator in eq. 3 scales with training set size. Consequently, for alpha=1, the reduced evidence is dominated by the class prior on imbalanced datasets, and selecting alpha >>1 primarily restores the natural scale of the log-likelihood relative to log(π^c)log(\hat{\pi}_c).
Questions for authors

I'd ask the authors to address the above two critical weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.