- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper brings the PFN paradigm to multiple instance learning (ICMIL). In a Perceiver-style model, learnable bag tokens cross-attend to their own instances, then TabPFN-v2 column-row attention runs across bags and labels. The model is pretrained on synthetic bags from factorized joint priors. These priors have complementary strengths (as shown in Table 1). A scaled model trained on a 0.70/0.15/0.15 mixture gets the best mean AUROC (84.17) and rank (4.67) on twelve ~100-bag benchmarks.
Critical strengths
- Brings in-context learning to label-scarce MIL is novel to me. It offers a useful taxonomy of synthetic priors as expected with PFN.
- The empirical results are promising and demonstrated the effectiveness of the in-context learning.
- The paper is is easy to follow. It is appreciated that the code and checkpoints are released by authors.
Critical weaknesses
-
ICMIL leads ACMIL by only 0.36 AUC and beats all baselines on just one task (Pos/Neg). It trails on SMIL (−13.3), Fox (−5.5), Letters (−3.5) and TCGA (−2.8). Also, TabPFN-Concat shows almost zero improvement on all tasks.
-
All features, including 1,536-d UNI2 and 2,048-d ResNet-50 embeddings, are PCA-reduced to 25 dimensions, and TCGA bags have 10 tiles. This contradicts the common practice in pathology, which involves of thousands of instances (i.e., also Challenge 1 the paper tries to tackle). These limits come from a small pretraining setup: synthetic tasks with ≤20 instances, 70–125 bags and 25 features, trained for ~24 h on one A100 (App. A). Although the authors acknowledge the gap, they do not test inputs beyond this range.
-
I think the benefit of PFN is to pretrain on a way larger synthetic tasks, which the authors fail to show in this paper.
Questions for authors
-
What is the "± SE" in Table 1 computed over, and why do not TabPFN-Concat improve?
-
Is the 25-dimension PCA a hard limit of the model? How does ICMIL behave on bags larger than the pretraining range?
4 comments
Response to Reviewer RyQH (Part II)
Is the 25-dimension PCA a hard limit of the model? How does ICMIL behave on bags larger than the pretraining range?
The method is general and can be used to train models on higher dimensional data. That being said, we already observe surprising generalization. We add additional experiments below:
Larger bags: While ICMIL is pretrained on comparably small bag sizes, the attention mechanism can by construction attend to more instances. We observe that this works well as indicated by the Musk2 benchmark (max. bag size 1,044, 90.8 ±1.1 vs. 91.3 ±0.4 for best baseline). To add further evidence, we evaluate the same model on three recent benchmarks [3,4,5] with larger bags:
- ICB: single-cell RNA-seq. 57 bags with up to 243 instances each.
- TAD: traffic anomaly detection in videos. 512 bags, subsampled to 1,000 instances each.
- COVID: single-cell RNA-seq. 50 bags with up to 1,000 instances each.
ICMIL compares favorably on ICB and COVID and is within three standard errors of the best baseline on TAD. (Table R3.1). This data indicates that ICMIL is able to generalize well beyond its pretraining range.
More features: Supporting more features requires pretraining on wider synthetic datasets, with no change to the method. To show this, we pretrained ICMIL models on up to 50 features and evaluated them on ICB-50, TAD-50 and COVID-50, the same benchmarks with PCA to 50 dimensions (Table R3.1). ICMIL is the best method on ICB-50 and COVID-50, and it is within 0.2 points of ACMIL on TAD-50. Scaling the feature dimension further requires additional compute or smart curriculum approaches as the millions of synthetically generated datasets grow in size. Current PFNs are generally able to perform competitively into the hundreds of features. This also addresses the reviewer’s point that the benefit of PFNs lies in pretraining on larger synthetic tasks. Doubling the maximum feature count of the synthetic prior from 25 to 50, allows ICMIL to process more features and we observe its AUROC rises from 73.0 to 80.3 on ICB and from 91.0 to 93.3 on TAD.
Table R3.1: AUROC (%) ± SE over three seeds. The 50-feature columns use the ICMIL model pretrained on up to 50 features.
| Method | Avg | ICB | TAD | COVID | ICB-50 | TAD-50 | COVID-50 |
|---|---|---|---|---|---|---|---|
| MeanLogReg | 82.1 | 66.6 ± 0.49 | 87.1 ± 0.83 | 87.0 ± 0.00 | 73.9 ± 0.36 | 90.8 ± 0.42 | 87.1 ± 0.02 |
| ACMIL | 79.2 | 63.2 ± 0.70 | 91.2 ± 0.20 | 80.2 ± 1.06 | 67.6 ± 0.59 | 93.5 ± 0.60 | 79.3 ± 1.11 |
| ICMIL | 85.8 | 73.0 ± 0.63 | 91.0 ± 0.38 | 88.5 ± 0.09 | 80.3 ± 0.83 | 93.3 ± 0.84 | 88.7 ± 0.52 |
| ICMIL (CE) | 86.8 | 73.3 ± 0.36 | 91.9 ± 0.36 | 89.2 ± 0.06 | 81.1 ± 0.57 | 95.3 ± 0.19 | 90.0 ± 0.55 |
On COVID and ICB we use the same splits and protocol as [6], so our numbers can be compared with published results (Table 2 in their paper). Using just 50 features and no task-specific gradient updates, ICMIL (CE) is within the reported error of their best model on COVID (90 vs. 90) and compares favorably on ICB (81.1 vs. 77).
Response to Reviewer RyQH (Part III)
More bags: While ICMIL is designed for the low-label MIL regime, we observe that it can be competitive up to a few hundred bags, but supervised methods often overtake it in the many-label regime. To illustrate this, we sample fractions of 0.25x to 4x the number of training bags used in the paper for the Pos/Neg and RSNA ICH benchmarks and subsample the full training split (408 bags) of the newly added TAD-50 benchmark:
Table R1.4. Performance on Pos/Neg for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).
| Method | 22 | 45 | 90 | 180 | 360 |
|---|---|---|---|---|---|
| ICMIL | 76.7 ± 0.9 | 83.8 ± 0.4 | 87.0 ± 0.2 | 88.2 ± 0.3 | 88.9 ± 0.1 |
| ABMIL | 62.7 ± 2.8 | 69.8 ± 2.1 | 74.6 ± 1.6 | 79.7 ± 0.3 | 86.7 ± 0.5 |
| ACMIL | 73.0 ± 1.7 | 77.3 ± 0.6 | 80.8 ± 0.4 | 83.5 ± 1.1 | 86.7 ± 0.3 |
| Δ | +3.7 | +6.5 | +6.2 | +4.7 | +2.2 |
Table R1.5. Performance on RSNA ICH for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).
| Method | 25 | 50 | 100 | 200 | 400 |
|---|---|---|---|---|---|
| ICMIL | 67.3 ± 0.1 | 71.7 ± 0.1 | 75.0 ± 0.1 | 76.6 ± 0.1 | 77.7 ± 0.2 |
| ABMIL | 62.8 ± 1.1 | 65.5 ± 1.5 | 71.2 ± 0.3 | 75.4 ± 0.7 | 80.1 ± 1.0 |
| ACMIL | 65.4 ± 0.7 | 69.9 ± 0.6 | 74.8 ± 1.0 | 79.8 ± 1.1 | 84.4 ± 0.4 |
| Δ | +1.9 | +1.8 | +0.2 | -3.2 | -6.7 |
Table R1.6. Performance on TAD-50 for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).
| Method | 25 | 50 | 100 | 200 | 408 |
|---|---|---|---|---|---|
| ICMIL | 75.9 ± 0.1 | 85.0 ± 0.6 | 90.3 ± 0.7 | 92.5 ± 0.6 | 93.3 ± 0.8 |
| ABMIL | 73.1 ± 2.2 | 78.7 ± 1.1 | 82.1 ± 2.7 | 86.3 ± 3.2 | 87.2 ± 3.9 |
| ACMIL | 77.4 ± 1.4 | 83.5 ± 0.7 | 88.1 ± 0.6 | 92.3 ± 0.2 | 93.5 ± 0.6 |
| Δ | -1.5 | +1.5 | +2.2 | +0.2 | -0.2 |
We hope these additional experiments and clarifications address the reviewer's concerns and strengthen their assessment of the paper. The anonymous repository now contains the code and checkpoints needed to reproduce the new experiments and benchmarks reported here.
References
[1] Prior Labs Team, "TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models", 2025
[2] Prior Labs Team, "TabPFN-3: Technical Report", 2026
[3] Ziegler et al., Impaired local intrinsic immunity to SARS-CoV-2 infection in severe covid-19, 2021
[4] Gondal MN, Cieslik M, Chinnaiyan AM. Integrated cancer cell-specific single-cell RNA-seq datasets of immune checkpoint blockade-treated patients, 2024
[5] Lv et al., Localizing Anomalies from Weakly-Labeled Videos, 2020
[6] Do et al., Incorporating hierarchical information into multiple instance learning for patient phenotype prediction with single-cell RNA-sequencing data, 2025
Response to Reviewer RyQH
Dear Reviewer,
As the discussion period is drawing to a close, we wanted to follow up on our rebuttal. In response to your review, we have (i) clarified the paper's contribution and (ii) added experiments on datasets with larger bag sizes and more bags, which demonstrate that ICMIL is able to generalize beyond its pretraining data.
We look very much forward to hearing your reaction to the responses we have provided above.
Thank you very much for your service,
the authors
Response to Reviewer RyQH (Part I)
We thank the reviewer for the positive assessment and are glad that they found our paper novel and informative. Below we address questions raised, including new experiments on larger bags, more features and more training bags.
We agree that the average lead over ACMIL is small and that ICMIL does not win uniformly. Two points put this in context. First, ICMIL reaches this performance without any training on the target dataset. Secondly and more importantly, we would like to highlight that the exact size of the lead is not central to the contribution of our paper.
We see the value of our paper in introducing the method that allows to train an in-context learner for bag-structured data and that matches or outperforms supervised methods in the low-bag regime. This opens up many different avenues for future work (e.g. richer priors, fine-tuning, post-training, harnesses) which we expect to be very fruitful for the community.
To illustrate this, we show that ICMIL can perform context ensembling. That is, we average predictions over multiple forward passes, each with a different transformation of the dataset and a different subsample of bags, without any weight updates. We use one fixed configuration for all datasets, and it increases ICMIL’s average AUROC by 1.08 percentage points (see table R1.1). Ensembling over transformed versions of the dataset is standard for tabular in-context learners [1,2]. Since this procedure requires no training cost, it is not directly applicable to supervised methods, which require weight updates on each dataset.
Table R1.1. AUROC (%) ± SE over three seeds.
Context ensembling narrows the gap to the best baseline on all four tasks the reviewer highlights, though a clear deficit remains on SMIL, which we are happy to discuss in the limitations. We hope that this addresses the reviewer’s performance concerns.
SE is the standard error over three seeds. For ICMIL, all three checkpoints are available in the anonymized repository. For TabPFN-Concat we performed concatenation and then a forward pass through the TabPFN backbone. This has no standard error as no training or weight updates happen and the ±0.0 is correct. We are happy to adjust the table caption: "± standard error for each model" → "± standard error for each model over three seeds".