acceptodds
Withdrawn from ICLR 2027

FFM-CP: Few-Shot Fusion of Foundation Models for Computational Pathology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pathology vision-language foundation models vary in performance across diseases and tasks, with no single model consistently performing best. The high cost of expert pathology annotation can also limit the labeled data available for task-specific adaptation. Combining complementary pretrained representations is a potential approach to these limitations, although learning an effective fusion from few labeled examples remains challenging. We introduce Few-shot Fusion Foundation Models of Computational Pathology (FFM-CP), which is a framework that combines multiple pathology vision-language models in the few-shot learning setting. The framework first aligns heterogeneous representations using a closed-form Orthogonal Procrustes transformation estimated from corresponding support images. This alignment preserves within-model feature geometry without training an additional alignment network. Within the aligned space, a unified graph enables information exchange across backbones by jointly refining support-image features and visual and textual class prototypes. These refined representations support complementary text-prototype and case-retrieval branches that capture semantic class knowledge and within-class visual variation, respectively. Each branch learns to combine predictions from all ordered backbone pairs, allowing queries encoded by one model to draw on evidence represented by another. We evaluate three backbone combinations on six histopathology datasets at 4, 8, and 16 shots per class. FFM-CP achieves higher mean macro-F1 than the strongest individually adapted member of each fused set in 50 of 54 comparisons. These findings suggest that combining complementary pretrained representations can improve histopathological classification when annotations are limited.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
few-shot learning, fusion multiple foundation models, multi-modal learning, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vWCxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FFM-CP, a few-shot fusion framework for pathology vision-language foundation models. It aligns frozen encoders' embeddings using support-set-derived Orthogonal Procrustes maps, then jointly refines support-image features and visual and textual class prototypes through a multi-relation graph. Predictions combine text-prototype and case-retrieval streams across ordered backbone pairs. Experiments evaluate three model combinations on six patch/ROI classification datasets at 4, 8, and 16 shots per class.

Critical strengths
  1. The paper addresses a useful problem: exploiting complementary pathology VLMs when labeled examples are scarce. Its closed-form Procrustes alignment matches heterogeneous embedding spaces while preserving within-model geometry, and the unified graph jointly refines support-image features and visual and textual class prototypes across backbones. Cross-backbone pair fusion allows queries encoded by one model to draw on evidence represented by another. Together, these components provide a coherent approach to using cross-model information during both adaptation and prediction.

  2. The full framework achieves higher mean macro-F1 than its strongest constituent in 50 of 54 comparisons. In Table 4, Procrustes also achieves higher mean macro-F1 than zero-padding and the learned MLP mapping on all six datasets.

Critical weaknesses
  1. The experiments do not yet establish whether the full design justifies its added complexity. All Table 4 variants retain graph refinement, so its contribution is not isolated, and the diagonal variant still uses the joint graph. A straightforward ensemble of independently adapted models is missing. The ablation covers only one model combination at 16 shots and provides no paired statistical analysis. The computational and memory costs of using three VLMs are not quantified.

  2. Reproducibility and backbone selection require clarification. The prediction and fusion computations and basic training settings are insufficiently specified. For datasets other than BRACS, the use of original splits and patient/slide separation for any new partitions is unclear. The three fusion configurations are reported without an explained selection rule, and the evaluation omits CONCH. These gaps limit confidence in the interpretation and reproducibility of the gains.

Questions for authors

Please check the Critical Weaknesses section first.

  1. The current comparisons do not isolate graph refinement or establish an advantage over simple ensembling. Table 4 also covers only one backbone combination at 16 shots. Please include a controlled graph ablation and an ensemble of independently adapted models under the same few-shot protocol, extend the ablation to representative combinations and shot settings, and provide paired uncertainty estimates or statistical analysis for the differences in Table 4.

  2. The rationale and data used to select the three fusion configurations are unclear. Please report the selection criteria and any other combinations explored, assess sensitivity to backbone composition, and explain how to select models for a new few-shot task. Please also include CONCH [1] as an individual baseline and in a representative fusion configuration, or explain its exclusion.

  3. The prediction procedure, training settings, and computational cost are insufficiently documented. Please provide an algorithm table with training and inference pseudocode, specify how branch scores are computed and fused, and report the optimizer, learning rate, and number of epochs. Report feature-precomputation and adaptation times, end-to-end inference time, and peak memory use relative to the strongest individual model and a simple ensemble under comparable hardware and settings. Please also correct the duplicated wording immediately before Equation (2), on page 4, line 184.

  4. For datasets other than BRACS, the use of original splits is unclear. Please state whether each dataset's original training, validation, and test partitions were used where available. For any created or modified splits, explain whether and how patient- and slide-level separation was enforced to prevent related tiles or ROIs from appearing across partitions.

  5. The evaluation is limited to tile/ROI classification with image-text VLMs. Please discuss the modifications needed for slide-level inference, vision-only encoders, and broader multimodal pathology backbones. As further validation, include a representative slide-level analysis, using MI-Zero [2] as an example of MIL-based VLM aggregation, to assess whether fusion gains extend to whole-slide classification.

[1] Lu, Ming Y., et al. "A visual-language foundation model for computational pathology." Nature medicine 30.3 (2024): 863-874. [2] Lu, Ming Y., et al. "Visual language pretrained multiple instance zero-shot transfer for histopathology images." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

Reviewer_HSuCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FFM-CP combines frozen pathology vision-language models for few-shot classification through Procrustes alignment, joint graph-based representation refinement, and dual-branch cross-backbone prediction fusion. Evaluation covers six datasets, three shot settings, and three backbone combinations.

Critical strengths
  1. The problem is practically relevant, and the framework provides a reasonable way to exploit complementary pathology representations with limited annotations.

  2. Experiments cover diverse datasets and report five-run variability. Fusion improves over the strongest constituent in 50 of 54 evaluated settings.

Critical weaknesses
  1. A genuine simple ensemble baseline is missing. Table 4(a) retains the full cross-backbone graph and therefore does not represent independently adapted models followed by prediction averaging. Without this comparison, it is unclear whether the proposed alignment and graph refinement provide meaningful benefits beyond ordinary ensembling.

  2. The core prediction modules are incompletely defined. Equation (10) introduces branch logits and pair-specific predictions without sufficiently specifying their computation. The case-retrieval aggregation and the parameterization and application of the pair-fusion matrices also require explicit definitions.

Questions for authors
  1. Can the authors compare against independently adapted constituent models combined through uniform probability averaging and a validation-tuned weighted ensemble, using identical support sets?

  2. Can the authors provide complete equations or pseudocode for both prediction branches, including retrieval selection, aggregation, and pair-fusion weights?

  3. Are the training, validation, and test partitions separated at the patient or slide level for datasets containing multiple related tiles?

Reviewer_rrXvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FFM-CP, a framework for few-shot classification that combines heterogeneous pathology vision-language foundation models while keeping their encoders frozen. A closed-form Orthogonal Procrustes map aligns the representations using paired support images. A unified seven-relation graph then jointly refines support-image features and visual and textual class prototypes. Text-prototype and case-retrieval branches combine predictions from all ordered backbone pairs. Experiments cover five backbones, three fusion sets, six histopathology datasets spanning four organs, and 4-, 8-, and 16-shot settings. The method achieves higher mean macro-F1 than the strongest individually adapted constituent of each fusion set in 50 of 54 comparisons. Ablations favor Procrustes alignment over zero-padding and learned MLP alignment, while off-diagonal prediction pairs provide a smaller, task-dependent improvement.

Critical strengths
  1. The paper addresses foundation-model selection when backbone rankings vary across pathology tasks and annotations are scarce. Its focus on task-specific fusion without separate large-scale fusion pretraining is practically relevant. The alignment, joint refinement, and prediction stages are coherently motivated by this problem.

  2. The evaluation covers six datasets, four organs, three shot counts, and three fusion sets, with means and standard deviations over five runs. Fusion improves mean macro-F1 in 50 of 54 comparisons against the strongest individually adapted member of the same fusion set. The range of datasets and model combinations provides useful evidence that the reported benefits are not confined to a single benchmark.

  3. For the KEEP + QuiltNet + PLIP (KQP) combination at 16 shots, Procrustes alignment outperforms both zero-padding and learned MLP alignment on all six datasets, with average macro-F1 gains of 1.73 and 2.12 percentage points, respectively. These comparisons provide direct empirical support for the selected alignment method in the evaluated setting.

Critical weaknesses
  1. Equation 10 introduces the fused logits ℓt\ell^t and ℓc\ell^c and the diagonal pair logits Zm,mtZ^t_{m,m} and Zm,mcZ^c_{m,m}, but the manuscript does not define how these are calculated. It also does not explain how the learned matrices FtF^t and FcF^c combine the ordered backbone pairs. The case-retrieval scoring and aggregation rules are missing as well. Figure 1 gives only a schematic, and Appendix B lists prompt templates. Since prediction fusion is a central contribution, these omissions prevent faithful reconstruction of the method.

  2. The late-fusion baseline is not consistently defined. Section 4.1 describes variant (a) as a late-fusion ensemble in which each model uses only its own text prototypes and support features, whereas Section 4.3 and Table 4 state that variant (a) retains the full cross-backbone graph and restricts only the prediction pairs to m=n. Its representations therefore already benefit from joint cross-model refinement. The reported comparisons against M=1 models also do not establish whether FFM-CP outperforms a simple ensemble of independently adapted models. Consequently, the benefit of the proposed cross-model interactions over standard ensembling remains unclear.

  3. Table 4 evaluates alignment alternatives and off-diagonal prediction pairs for one fusion set at 16 shots. It does not isolate the unified graph from independent per-model refinement or establish the separate contributions of the text-prototype and case-retrieval branches. These are central elements of the proposed framework. The existing ablations therefore leave the source of the full-model gains partly unresolved. A graph comparison and separate branch results would address this directly.

  4. BRACS uses its released partitions, but the partitioning unit for the other five datasets is not specified. Appendix A describes multiple images or tiles from shared patients or specimens. The statement that image partitions are disjoint does not establish patient-level or slide-level separation. The protocol should identify the grouping unit and provide the split counts or identifiers. Clarifying whether any patients or source slides are shared across partitions is necessary to assess evaluation independence.

Questions for authors
  1. Please provide the complete definitions of the pairwise logits, the case-retrieval scoring and aggregation rules, and the fusion through FtF^t and FcF^c. Please also specify whether a support image remains in its own retrieval pool when used as a training query, and how the corresponding training logits are computed.
  2. Can you compare the full method with probability averaging of independently adapted M = 1 models under the same support and validation protocol? Since variant (a) retains the joint graph, please also provide controlled comparisons that isolate joint graph refinement and the contributions of the text-prototype and case-retrieval branches. These results would clarify whether the gains come from the proposed interactions or from simpler ensembling.
  3. For each dataset, what are the train, validation, and test split counts and the partitioning unit? Where patient or source-slide identifiers are available, are these groups kept separate across partitions? Please clarify any overlap or unavailable grouping information, since this affects the interpretation of the reported generalization performance.
  4. The overview figure is difficult to read due to small text and unclear visual connections between components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.