acceptodds
Under review as a conference paper at ICLR 2027

When More Foundation Models Mean Less: Diagnosing and Addressing Multi-View Fusion Failure

Abstract

Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in time independent of encoder dimension, and admits a conditional prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by , , and points, respectively. The same selection effect extends to image retrieval and frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multi-view learning, encoder selection, kernel alignment, foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M5mtOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method for MULTI-VIEW FUSION The paper claims that “downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline.” And, the authors introduce KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. The paper claims that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.

Critical strengths
  1. This paper is well organized and easy to follow.
  2. The quantitative results nicely complement the paper.
Critical weaknesses
  1. For the main experimental setup, M=15M=15 is used—is this a standard practice? Does it effectively substantiate the paper’s claim that "more foundation models mean less"? What would happen if MM were even larger?
  2. While the mean values across five seeds are provided, the table lacks standard deviations or confidence intervals. Some improvements are quite marginal (e.g., +0.6 for Pets, +2.5 for DTD). I suggest including standard deviations or the results of a paired test in the main tables—at the very least, it should be clarified whether the observed improvements are robust enough to exceed the variance caused by different random seeds.
Questions for authors

Please refer to Weaknesses.

1 comment

I recheck this paper, and find the experimental results are not convincing to support the conclusions. Thus, I decide to lower the score to weak reject.

Reviewer_vA2COpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies encoder subset selection for frozen multi-view fusion. KAGES greedily ranks encoders using centered kernel–target alignment and optionally selects a prefix of the resulting ordering.

Critical strengths

Its motivation, namely deciding which pretrained encoders to combine and how many to retain, is meaningful.

Critical weaknesses
  1. The method combines established kernel alignment with standard forward greedy selection to rank encoder subsets. The theoretical results are essentially a conditional application of an existing weak-submodular greedy approximation bound, while the information-theoretic framing basically provides motivation but does not yield a new optimization principle or substantive problem-specific theoretical insights. Overall, the technical advance over existing alignment-based kernel learning and feature selection frameworks appears rather limited.
  2. The paper does not establish a convincing theoretical connection between its information-theoretic motivation and the actual selection criterion. The relevance–redundancy decomposition is essentially an algebraic characterization of the CKA surrogate, which is insufficient to establish that its gains in the surrogate reflect increased task-relevant information or reduced predictive risk.
  3. The reported comparisons do not clearly isolate the benefit of set-level selection from simply prioritizing individually strong encoders. Comparative validation is also narrow: the named CLIP-adaptation baselines are restricted to four methods from 2022–2023, leaving competitiveness against strong, directly relevant alternatives insufficiently established. More broadly, the experimental analysis provides little mechanistic evidence supporting the proposed relevance–redundancy interpretation.
  4. Poor organization and incomplete exposition. Figures and tables are poorly integrated with the surrounding text, frequently disrupting the logical flow. Key mathematical symbols and experimental settings are introduced without sufficient explanation, leaving important aspects of the formulation and evaluation ambiguous and difficult to assess.
Questions for authors

Refer to weakness section.

Reviewer_vpKvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies performance degradation when combining increasing numbers of frozen encoders. It proposes KAGES, which greedily orders encoders by their marginal contribution to kernel-target alignment without training downstream classifiers during selection. Classification experiments show improved average performance across prefixes, with additional results on image retrieval and classification using frozen LLM representations. The paper also provides conditional theoretical guarantees for greedy selection.

Critical strengths
  1. Practical and efficient selection strategy. KAGES evaluates each encoder relative to the selected set, accounting for task relevance and overlap without repeatedly training downstream classifiers.

  2. Consistent empirical improvements. Results across five classification datasets, multiple data regimes, and five seeds support improved ordering quality. The retrieval experiments also illustrate that the useful number of encoders depends on the downstream task.

Critical weaknesses
  1. Validation of the selector remains concentrated on one fusion architecture. Although the paper includes gated-fusion diagnostics, the main KAGES comparisons use Concat+MLP and pools of at most 15 encoders. Its advantages with learned fusion and larger pools remain insufficiently established.

  2. The benefit of set-level selection is not fully isolated. DPP and facility-location baselines do not use the label kernel. A baseline that ranks individual encoders by the same kernel-target alignment would help separate the benefits of supervision from those of modeling set-level interactions. The optional score-based subset-size rule also remains unvalidated, although the paper acknowledges this limitation.

Questions for authors
  1. Do the existing results establish KAGES’s advantage under learned weighting or gating? The Frobenius-normalization ablation reports a small change on Protocol B; does this conclusion hold across the other protocols and fusion-head settings?

  2. How does KAGES compare with ranking individual encoders by kernel-target alignment? From the existing runs, how does the optional score-selected prefix compare with the hindsight-best prefix in size and accuracy?

  3. How sensitive are the LLM linear-probe results to the train/test split? Given the small held-out subsets, uncertainty estimates or consistency across available splits would help assess the reported improvements.

Comments on the paper

1 comment

Dear Reviewers and Authors,

As a quick reminder, the author-reviewer discussion period will close on November 18.

@Reviewers: Please make sure to read the authors' rebuttals, acknowledge their responses, and engage in further discussion if any points remain unclear. While final decisions are not made at this stage, we ask that you carefully re-evaluate your ratings in light of the new clarifications and experimental results:

Increase your score if the authors have satisfactorily resolved your concerns and strengthened the paper.

Decrease your score if new, unaddressed issues have emerged during the discussion.

Maintain your score if critical concerns remain unresolved.

Please note that you are under no obligation to change your rating, provided your final score accurately reflects the current state of the manuscript. To ensure a fair evaluation, please ground your assessment in concrete evidence rather than personal preference, and consider whether the work offers meaningful scientific value to the ICLR community.

Thank you very much for your time, thoughtful engagement, and contributions to the review process.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.