acceptodds
Under review as a conference paper at ICLR 2027

In-Context Multiple Instance Learning

Abstract

Multiple Instance Learning (MIL) addresses problems where supervision is available at the level of bags of instances and has been successfully applied in fields ranging from computational pathology to satellite imagery. Nevertheless, existing algorithms struggle in the low-label regime that characterizes many real-world applications. Flexible models overfit and rigid ones fail to adapt to the task at hand. We show that pretraining an in-context learner with a Perceiver-style architecture on synthetic data yields a model that can solve new tasks from a handful of labeled bags. At inference time, classification happens in a single forward pass and requires no gradient updates. We propose and investigate synthetic data generators for bag-structured data and find that they capture complementary inductive biases. A model pretrained on a mixture of these generators inherits their per-task strengths and achieves the best average performance across twelve MIL benchmarks, outperforming supervised baselines that require task-specific training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multiple Instance Learning, In-Context Learning, Prior Fitted Neural Networks, Meta Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RyQHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper brings the PFN paradigm to multiple instance learning (ICMIL). In a Perceiver-style model, learnable bag tokens cross-attend to their own instances, then TabPFN-v2 column-row attention runs across bags and labels. The model is pretrained on synthetic bags from factorized joint priors. These priors have complementary strengths (as shown in Table 1). A scaled model trained on a 0.70/0.15/0.15 mixture gets the best mean AUROC (84.17) and rank (4.67) on twelve ~100-bag benchmarks.

Critical strengths
  • Brings in-context learning to label-scarce MIL is novel to me. It offers a useful taxonomy of synthetic priors as expected with PFN.
  • The empirical results are promising and demonstrated the effectiveness of the in-context learning.
  • The paper is is easy to follow. It is appreciated that the code and checkpoints are released by authors.
Critical weaknesses
  • ICMIL leads ACMIL by only 0.36 AUC and beats all baselines on just one task (Pos/Neg). It trails on SMIL (−13.3), Fox (−5.5), Letters (−3.5) and TCGA (−2.8). Also, TabPFN-Concat shows almost zero improvement on all tasks.

  • All features, including 1,536-d UNI2 and 2,048-d ResNet-50 embeddings, are PCA-reduced to 25 dimensions, and TCGA bags have 10 tiles. This contradicts the common practice in pathology, which involves of thousands of instances (i.e., also Challenge 1 the paper tries to tackle). These limits come from a small pretraining setup: synthetic tasks with ≤20 instances, 70–125 bags and 25 features, trained for ~24 h on one A100 (App. A). Although the authors acknowledge the gap, they do not test inputs beyond this range.

  • I think the benefit of PFN is to pretrain on a way larger synthetic tasks, which the authors fail to show in this paper.

Questions for authors
  • What is the "± SE" in Table 1 computed over, and why do not TabPFN-Concat improve?

  • Is the 25-dimension PCA a hard limit of the model? How does ICMIL behave on bags larger than the pretraining range?

4 comments
AuthorsOpenReview ↗

Response to Reviewer RyQH (Part I)

We thank the reviewer for the positive assessment and are glad that they found our paper novel and informative. Below we address questions raised, including new experiments on larger bags, more features and more training bags.


ICMIL leads ACMIL by only 0.36 AUC and beats all baselines on just one task (Pos/Neg). It trails on SMIL (−13.3), Fox (−5.5), Letters (−3.5) and TCGA (−2.8)


We agree that the average lead over ACMIL is small and that ICMIL does not win uniformly. Two points put this in context. First, ICMIL reaches this performance without any training on the target dataset. Secondly and more importantly, we would like to highlight that the exact size of the lead is not central to the contribution of our paper.

We see the value of our paper in introducing the method that allows to train an in-context learner for bag-structured data and that matches or outperforms supervised methods in the low-bag regime. This opens up many different avenues for future work (e.g. richer priors, fine-tuning, post-training, harnesses) which we expect to be very fruitful for the community.

To illustrate this, we show that ICMIL can perform context ensembling. That is, we average predictions over multiple forward passes, each with a different transformation of the dataset and a different subsample of bags, without any weight updates. We use one fixed configuration for all datasets, and it increases ICMIL’s average AUROC by 1.08 percentage points (see table R1.1). Ensembling over transformed versions of the dataset is standard for tabular in-context learners [1,2]. Since this procedure requires no training cost, it is not directly applicable to supervised methods, which require weight updates on each dataset.


Table R1.1. AUROC (%) ± SE over three seeds.

ModelAvg AUROCMusk1Musk2HEPMASSLettersSMILAdj. PairsPos/NegRSNA ICHElephantFoxTigerTCGA
ACMIL83.8189.0 ±1.383.9 ±2.581.1 ±3.998.7 ±1.387.7 ±1.072.5 ±0.680.8 ±0.474.8 ±1.094.1 ±0.264.4 ±1.788.0 ±1.290.7 ±0.7
ICMIL84.1793.3 ±0.690.8 ±1.184.5 ±0.795.7 ±0.774.4 ±0.377.6 ±0.587.0 ±0.275.0 ±0.194.2 ±0.160.7 ±0.888.9 ±0.687.9 ±0.1
ICMIL (CE)85.2594.7 ±0.0892.2 ±0.6085.1 ±0.2797.3 ±0.2776.2 ±0.4877.9 ±0.3287.9 ±0.3575.3 ±0.0594.4 ±0.2363.4 ±0.7690.2 ±0.1788.5 ±0.19

Context ensembling narrows the gap to the best baseline on all four tasks the reviewer highlights, though a clear deficit remains on SMIL, which we are happy to discuss in the limitations. We hope that this addresses the reviewer’s performance concerns.


What is the "± SE" in Table 1 computed over, and why do not TabPFN-Concat improve?


SE is the standard error over three seeds. For ICMIL, all three checkpoints are available in the anonymized repository. For TabPFN-Concat we performed concatenation and then a forward pass through the TabPFN backbone. This has no standard error as no training or weight updates happen and the ±0.0 is correct. We are happy to adjust the table caption: "± standard error for each model" → "± standard error for each model over three seeds".

AuthorsOpenReview ↗

Response to Reviewer RyQH (Part II)


Is the 25-dimension PCA a hard limit of the model? How does ICMIL behave on bags larger than the pretraining range?


The method is general and can be used to train models on higher dimensional data. That being said, we already observe surprising generalization. We add additional experiments below:

Larger bags: While ICMIL is pretrained on comparably small bag sizes, the attention mechanism can by construction attend to more instances. We observe that this works well as indicated by the Musk2 benchmark (max. bag size 1,044, 90.8 ±1.1 vs. 91.3 ±0.4 for best baseline). To add further evidence, we evaluate the same model on three recent benchmarks [3,4,5] with larger bags:

  • ICB: single-cell RNA-seq. 57 bags with up to 243 instances each.
  • TAD: traffic anomaly detection in videos. 512 bags, subsampled to 1,000 instances each.
  • COVID: single-cell RNA-seq. 50 bags with up to 1,000 instances each.

ICMIL compares favorably on ICB and COVID and is within three standard errors of the best baseline on TAD. (Table R3.1). This data indicates that ICMIL is able to generalize well beyond its pretraining range.

More features: Supporting more features requires pretraining on wider synthetic datasets, with no change to the method. To show this, we pretrained ICMIL models on up to 50 features and evaluated them on ICB-50, TAD-50 and COVID-50, the same benchmarks with PCA to 50 dimensions (Table R3.1). ICMIL is the best method on ICB-50 and COVID-50, and it is within 0.2 points of ACMIL on TAD-50. Scaling the feature dimension further requires additional compute or smart curriculum approaches as the millions of synthetically generated datasets grow in size. Current PFNs are generally able to perform competitively into the hundreds of features. This also addresses the reviewer’s point that the benefit of PFNs lies in pretraining on larger synthetic tasks. Doubling the maximum feature count of the synthetic prior from 25 to 50, allows ICMIL to process more features and we observe its AUROC rises from 73.0 to 80.3 on ICB and from 91.0 to 93.3 on TAD.


Table R3.1: AUROC (%) ± SE over three seeds. The 50-feature columns use the ICMIL model pretrained on up to 50 features.

MethodAvgICBTADCOVIDICB-50TAD-50COVID-50
MeanLogReg82.166.6 ± 0.4987.1 ± 0.8387.0 ± 0.0073.9 ± 0.3690.8 ± 0.4287.1 ± 0.02
ACMIL79.263.2 ± 0.7091.2 ± 0.2080.2 ± 1.0667.6 ± 0.5993.5 ± 0.6079.3 ± 1.11
ICMIL85.873.0 ± 0.6391.0 ± 0.3888.5 ± 0.0980.3 ± 0.8393.3 ± 0.8488.7 ± 0.52
ICMIL (CE)86.873.3 ± 0.3691.9 ± 0.3689.2 ± 0.0681.1 ± 0.5795.3 ± 0.1990.0 ± 0.55

On COVID and ICB we use the same splits and protocol as [6], so our numbers can be compared with published results (Table 2 in their paper). Using just 50 features and no task-specific gradient updates, ICMIL (CE) is within the reported error of their best model on COVID (90 vs. 90) and compares favorably on ICB (81.1 vs. 77).

AuthorsOpenReview ↗

Response to Reviewer RyQH (Part III)

More bags: While ICMIL is designed for the low-label MIL regime, we observe that it can be competitive up to a few hundred bags, but supervised methods often overtake it in the many-label regime. To illustrate this, we sample fractions of 0.25x to 4x the number of training bags used in the paper for the Pos/Neg and RSNA ICH benchmarks and subsample the full training split (408 bags) of the newly added TAD-50 benchmark:


Table R1.4. Performance on Pos/Neg for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).

Method224590180360
ICMIL76.7 ± 0.983.8 ± 0.487.0 ± 0.288.2 ± 0.388.9 ± 0.1
ABMIL62.7 ± 2.869.8 ± 2.174.6 ± 1.679.7 ± 0.386.7 ± 0.5
ACMIL73.0 ± 1.777.3 ± 0.680.8 ± 0.483.5 ± 1.186.7 ± 0.3
Δ+3.7+6.5+6.2+4.7+2.2

Table R1.5. Performance on RSNA ICH for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).

Method2550100200400
ICMIL67.3 ± 0.171.7 ± 0.175.0 ± 0.176.6 ± 0.177.7 ± 0.2
ABMIL62.8 ± 1.165.5 ± 1.571.2 ± 0.375.4 ± 0.780.1 ± 1.0
ACMIL65.4 ± 0.769.9 ± 0.674.8 ± 1.079.8 ± 1.184.4 ± 0.4
Δ+1.9+1.8+0.2-3.2-6.7

Table R1.6. Performance on TAD-50 for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).

Method2550100200408
ICMIL75.9 ± 0.185.0 ± 0.690.3 ± 0.792.5 ± 0.693.3 ± 0.8
ABMIL73.1 ± 2.278.7 ± 1.182.1 ± 2.786.3 ± 3.287.2 ± 3.9
ACMIL77.4 ± 1.483.5 ± 0.788.1 ± 0.692.3 ± 0.293.5 ± 0.6
Δ-1.5+1.5+2.2+0.2-0.2

We hope these additional experiments and clarifications address the reviewer's concerns and strengthen their assessment of the paper. The anonymous repository now contains the code and checkpoints needed to reproduce the new experiments and benchmarks reported here.


References

[1] Prior Labs Team, "TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models", 2025

[2] Prior Labs Team, "TabPFN-3: Technical Report", 2026

[3] Ziegler et al., Impaired local intrinsic immunity to SARS-CoV-2 infection in severe covid-19, 2021

[4] Gondal MN, Cieslik M, Chinnaiyan AM. Integrated cancer cell-specific single-cell RNA-seq datasets of immune checkpoint blockade-treated patients, 2024

[5] Lv et al., Localizing Anomalies from Weakly-Labeled Videos, 2020

[6] Do et al., Incorporating hierarchical information into multiple instance learning for patient phenotype prediction with single-cell RNA-sequencing data, 2025

AuthorsOpenReview ↗

Response to Reviewer RyQH

Dear Reviewer,

As the discussion period is drawing to a close, we wanted to follow up on our rebuttal. In response to your review, we have (i) clarified the paper's contribution and (ii) added experiments on datasets with larger bag sizes and more bags, which demonstrate that ICMIL is able to generalize beyond its pretraining data.

We look very much forward to hearing your reaction to the responses we have provided above.

Thank you very much for your service,

the authors

Reviewer_db6xOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a prior-data fitted network can do multiple instance learning with about a hundred labeled bags, where supervised aggregators tend to overfit. In ICMIL, a learnable token per bag cross-attends to that bag's instances. The bag tokens and label embeddings then pass through attention across bags. The two steps alternate, so the instance readout can depend on the context labels. The model is pretrained once on synthetic bag tasks from factorized and joint priors built on TabICL's SCMs. On twelve small benchmarks, all PCA-reduced to 25 features, a scaled model attains the best average AUROC (84.17 vs 83.81 for ACMIL). Ablations cover the number of iterations, inter-bag attention and the prior mixture.

Critical strengths

S1. A reasonable way to approach low-label MIL, with a useful prior study. To my knowledge this is the first in-context learner built for bag-structured data. Per-bag cross-attention bounds the cost over instances, and the alternation with inter-bag attention lets the readout depend on the context labels. Table 2 supports this (T=1 gives 65.5 mean AUROC, T=6 gives 83.3), and the prior mixture beats every single prior on average.

S2. A training-free model that is competitive on average and fully reproducible. Without per-dataset training, ICMIL reaches the best average AUROC and the best average rank. Every baseline fails badly somewhere, for example SVM-Summ on HEPMASS (42.1). The anonymous repository provides the prior generators, training and evaluation code, all checkpoints and the exact splits.

Critical weaknesses

W1. The headline claim of outperforming supervised baselines is not supported by the evidence; at most it shows competitiveness. The average margin over ACMIL is 0.36 AUROC (84.17 vs 83.81), and no significance analysis is reported. The ± in Table 1 is undefined, and TabPFN-Concat is ±0.0 on all twelve benchmarks, which cannot be a standard error over splits. Variance across pretraining runs is not reported. Against the full baseline pool, ICMIL is best only on Pos/Neg and clearly behind on SMIL (74.4 vs 87.7). The extended training schedule of the scaled model was chosen from learning curves on Fox and Musk2, two evaluation benchmarks, with no benchmark held out. The abstract and conclusion overstate the results.

W2. Performance depends heavily on the prior-target match, and this dependence is not analyzed: the evaluation neither commits to a target domain nor tests the final model as the match degrades. Changing only the pretraining prior moves the average AUROC from 51.3 to 82.8 (Table 1). There is no study of the final mixed model beyond the pretraining ranges (6 to 20 instances per bag, 70 to 125 bags, 25 features), and no diagnostic of prior coverage. A user cannot know in advance whether ICMIL will land near the top or the bottom of Table 1. Most benchmarks are classic or semi-synthetic, and the two realistic tasks are cut down (10 tiles per TCGA patient). Text, remote sensing and standard pathology benchmarks such as CAMELYON16 are absent, and the custom PCA-reduced splits cannot be compared with published results.

W3. The comparison with the field is incomplete: it omits pretrained MIL transfer, newer PFN backbones and recent aggregators, and the efficiency evidence is narrow. The introduction names supervised MIL pretraining and transfer (Shao et al., ICML 2025) as the low-label alternative, yet no transferred aggregator is evaluated. All PFN baselines use TabPFN-v2, although TabPFN-2.5 and TabICL were available before submission. ACMIL is the only recent aggregator; RRT-MIL and MambaMIL (2024) are absent. No PFN is run on the pooled bag vectors used by MeanLogReg, so in-context aggregation is not separated from the backbone. Figure 5 times only three simple baselines on Pos/Neg and never measures cost as bag size grows.

W4. The novelty over existing PFNs is limited to the composition and the priors, and the evidence for what is new is thin. The architecture combines Perceiver-style cross-attention, TabPFN-v2 feature grouping, column-row attention and TabICL's SCM generators. The case for label-conditioned iterative readout rests only on a single-seed iteration sweep (Table 2) and one constructed task (Appendix C). No matched-capacity ablation feeds a label-agnostic learned pooler into the same in-context stack. The theoretical support is standard PFN theory, with no analysis of what iterative readout can represent that one-shot pooling cannot.

Minor points, listed separately from the weaknesses above and not weighed in the assessment:

  1. Table 1 has other ±0.0 entries (MeanLogReg on RSNA, TabPFN-Cluster on Fox) that need explaining.
  2. The prose after Table 1 omits that ICMIL also trails the best baseline on Fox, HEPMASS and RSNA.
  3. Section 4.4 gives the scaled MLP hidden size as 1054; presumably 1024.
  4. The best average rank includes the internal prior ablations; please also report the rank among external baselines only.
  5. In Table 2, the inter-bag-attention-off variant is a sanity check rather than a component ablation.
  6. A title that names the contribution would help readers position the work.

Further requests, not decisive for the assessment:

  • TransMIL should be run rather than excluded on the argument that it overfits.
  • Please state whether zero-padded instances are masked, since Musk2 bags of up to 1,044 instances are padded.
  • Please report ICMIL on TCGA with many more tiles per patient or more PCA components.
  • Please time ACMIL, DSMIL and the TabPFN baselines on matched hardware and state the hardware used for ICMIL.
Questions for authors

Q1 (W1). What is the ± in Table 1 computed over, and why is TabPFN-Concat ±0.0 everywhere? Please report paired tests against ACMIL, variance across at least three pretraining seeds, and results on benchmarks not used to choose the training schedule. Without a margin that survives a seed-inclusive uncertainty, the claim should be stated as parity, not outperformance.

Q2 (W2). Please evaluate the final mixed model beyond the pretraining ranges (larger bags, larger contexts, more features), state the intended target domain, and report a measure of prior coverage that predicts ICMIL's rank. Without this, the dependence on the prior remains an open risk for any practical use.

Q3 (W3). Please run a PFN, with TabPFN-v2 and with TabPFN-2.5 or TabICL, on the same pooled bag vectors used by MeanLogReg and SVM-Summ. Parity here would attribute the gain to the backbone rather than to the proposed architecture.

Q4 (W3). On TCGA, how does ICMIL compare with a transferred MIL-Lab aggregator (Shao et al., 2025) on the same 100 patients, and how do RRT-MIL and MambaMIL compare across the benchmarks? A clear loss to any of them would confirm that the comparison in the paper is incomplete.

10 comments
AuthorsOpenReview ↗

Response to Reviewer db6x (Part I)

We thank the reviewer for the detailed review and are happy that they found our paper easy to follow, the prior study useful and recognize ICMIL as the first in-context learner built for bag-structured data. Given the number of points raised, we focus below on the reviewer's core questions (Q1–Q4) and the related weaknesses.


The headline claim of outperforming supervised baselines is not supported by the evidence; at most it shows competitiveness. … The abstract and conclusion overstate the results… the claim should be stated as parity, not outperformance


We are happy to change the phrasing to "competitive with the best supervised baseline" (Avg AUROC: ICMIL 84.17 versus 83.81 ACMIL). The following two points put this into context. Firstly, ICMIL reaches this performance without any training on the target dataset. Secondly and more importantly, the exact size of the lead is not central to the contribution of our paper.

In our paper we show that it is possible to train a single model that can be used unchanged and without any gradient updates across datasets and that matches or outperforms supervised methods like ACMIL or ABMIL on average in the low-bag regime. Establishing how this can be done and that this works at all is the point of our contribution. In addition, our paper opens up many avenues for future work (e.g. richer priors, fine-tuning, post-training, harnesses), which we expect to be fruitful for the community. Neither depends on the exact margin in any specific comparison against a supervised model.

To illustrate the potential of ICMIL for further improvement, we show that it can perform context ensembling. That is, we average predictions over multiple forward passes, each with a different transformation of the dataset and a different subsample of bags, without any weight updates. We use one fixed configuration for all datasets, and it increases ICMIL’s average AUROC by 1.08 percentage points (see Table R2.1). Ensembling over transformed versions of the dataset is standard for tabular in-context learners [1,2]. Since this procedure requires no further training, it is not directly applicable to supervised methods.


Table R2.1. Benchmark performance of ICMIL with context ensembling (CE) and the best supervised baseline (ACMIL). AUROC (%) ± SE over three seeds.

ModelAvg AUROCMusk1Musk2HEPMASSLettersSMILAdj. PairsPos/NegRSNA ICHElephantFoxTigerTCGA
ACMIL83.8189.0 ±1.383.9 ±2.581.1 ±3.998.7 ±1.387.7 ±1.072.5 ±0.680.8 ±0.474.8 ±1.094.1 ±0.264.4 ±1.788.0 ±1.290.7 ±0.7
MambaMIL77.9989.1 ±1.890.1 ±1.768.7 ±6.699.5 ±0.570.8 ±1.365.9 ±0.672.5 ±0.767.1 ±0.591.8 ±0.860.0 ±1.484.1 ±0.476.4 ±1.2
ICMIL84.1793.3 ±0.690.8 ±1.184.5 ±0.795.7 ±0.774.4 ±0.377.6 ±0.587.0 ±0.275.0 ±0.194.2 ±0.160.7 ±0.888.9 ±0.687.9 ±0.1
ICMIL (CE)85.2594.7 ±0.0892.2 ±0.6085.1 ±0.2797.3 ±0.2776.2 ±0.4877.9 ±0.3287.9 ±0.3575.3 ±0.0594.4 ±0.2363.4 ±0.7690.2 ±0.1788.5 ±0.19

The ± in Table 1 is undefined, and TabPFN-Concat is ±0.0 , which cannot be a standard error over splits."


It is indeed the standard error over three seeds. For ICMIL all three checkpoints for the three pretraining seeds are available in the repository. TabPFN-Concat performs concatenation and then a forward pass through the TabPFN backbone. This has no standard error as no training or weight updates happen and the ±0.0 is correct. We are happy to adjust the table caption: "± standard error for each model" → "± standard error for each model over three seeds".

AuthorsOpenReview ↗

Response to Reviewer db6x (Part II)


The extended training schedule of the scaled model was chosen from learning curves on Fox and Musk2, two evaluation benchmarks, with no benchmark held out


The schedule was not selected using benchmark results. We made a single change to the training schedule, doubling training from 20,000 to 40,000 steps, with no sweep and no comparison of alternatives. There was no selection step for which held-out benchmarks would be needed. We are happy to explicitly clarify this in the paper.


Most benchmarks are classic or semi-synthetic... Please evaluate the final mixed model beyond the pretraining ranges (larger bags, larger contexts, more features), state the intended target domain.."


The method is general and can be used to train models that work with higher-dimensional data. The experiments in the paper were designed to show that in-context MIL is feasible, and did not push the limits of what is computationally possible. We do believe that the method is especially competitive for low-label MIL problems and that is the intended target domain.

Larger bags and larger contexts: While ICMIL is pretrained on comparably small bag sizes, the attention mechanism can by construction attend to more instances. We observe that this works well as indicated by the Musk2 benchmark (max bag size 1,044, 90.8 ±1.1 vs. 91.3 ±0.4 for best baseline). To add further evidence, and to address the concern that most benchmarks are classic or semi-synthetic, we evaluated the model's from the paper on three recent benchmarks [3,4,5] with larger bags:

  • ICB: single-cell RNA-seq. 57 bags with up to 243 instances each.
  • TAD: traffic anomaly detection in videos. 512 bags, subsampled to 1,000 instances each.
  • COVID: single-cell RNA-seq. 50 bags with up to 1,000 instances each.

ICMIL compares favorably on ICB and COVID and is within three standard errors of the best baseline on TAD (Table R2.2). This data indicates that ICMIL is able to generalize well beyond its pretraining range.

More features: Supporting more features requires pretraining on wider synthetic datasets, with no change to the method. To show this, we pretrained a ICMIL models on up to 50 features and evaluated them on ICB-50, TAD-50 and COVID-50, the same benchmarks with PCA to 50 dimensions (Table R2.2). It compares favorably on ICB-50 and COVID-50, and with context ensembling also on TAD-50. Scaling the feature dimension further requires additional compute or smart curriculum approaches as the millions of synthetically generated datasets grow in size. Current PFNs are generally able to perform competitively into the hundreds of features.


Table R2.2: AUROC (%) ± SE over three seeds. The 50-feature columns use the ICMIL model pretrained on up to 50 features.

MethodAvgICBTADCOVIDICB-50TAD-50COVID-50
MeanLogReg82.166.6 ± 0.4987.1 ± 0.8387.0 ± 0.0073.9 ± 0.3690.8 ± 0.4287.1 ± 0.02
Summ-TabPFN-3.583.968.6 ± 0.1591.2 ± 0.1585.9 ± 0.1779.0 ± 0.0694.4 ± 0.1384.3 ± 0.16
ACMIL79.263.2 ± 0.7091.2 ± 0.2080.2 ± 1.0667.6 ± 0.5993.5 ± 0.679.3 ± 1.11
ICMIL85.873.0 ± 0.6391.0 ± 0.3888.5 ± 0.0980.3 ± 0.8393.3 ± 0.8488.7 ± 0.52
ICMIL (CE)86.873.3 ± 0.3691.9 ± 0.3689.2 ± 0.0681.1 ± 0.5795.3 ± 0.1990.0 ± 0.55

On COVID and ICB we use the same splits and protocol as [6], so our numbers can be compared with published results (Table 2 in their paper). Using just 50 features and no task-specific gradient updates, ICMIL (CE) is within the reported error of their best model on COVID (90 vs. 90) and compares favorably on ICB (81.1 vs. 77).

AuthorsOpenReview ↗

Response to Reviewer db6x (Part III)


Changing only the pretraining prior moves the average AUROC from 51.3 to 82.8 .. open risk for any practical use."


This spread is the expected behaviour and is what the experiment was designed to show. The single-prior models in Table 1 are deliberately restricted to one specific function class, in order to understand how the function classes perform individually. A model pretrained on a narrow function class is expected to perform poorly on many tasks that lie outside it, and the 51.3 result comes from one such narrow prior.

The broadest prior is the joint prior, which by construction covers the tasks that can be generated by an MLP-SCM over the full bag, thus a very broad class of functions. This is the same principle that tabular PFNs rely on and that has made them successful and robust on tabular data, which is why it makes up 70% of ICMIL’s pretraining mixture. On its own, the joint prior achieves the best average rank among the individual priors and leads on the correlated and interaction-driven benchmarks (Figure 4). That being said, one contribution of our paper is to show that performance on MIL-specific uncorrelated witness tasks can be improved further by designing priors that follow the standard MIL assumption. The mixture of both outperforms every individual prior on average (Table 1 in the paper) and is insensitive to the exact weights (Table 3 in the paper). Extending the prior families is a natural avenue for future work.

For practical purposes we recommend, as for any method, evaluating on held-out bags before deployment and choosing the method that performs best on the specific dataset. For ICMIL this check is particularly cheap, since it requires no training or tuning.


"prior coverage that predicts ICMIL's rank"


We would be grateful if the reviewer could clarify what form such a measure would take. If the request is for a quantity that predicts, before evaluation, how ICMIL will rank against other methods on a new dataset, we believe this is an open problem that is orthogonal to our contribution. To our knowledge, no such measure exists for tabular PFNs or for supervised MIL methods. What we do provide is a characterisation of where the individual priors work: Section 4.2 and Figure 4 relate prior performance to the label rule and the within-bag feature correlation of the task. If the reviewer has a specific diagnostic in mind, we are happy to compute it.


The comparison with the field is incomplete: it omits pretrained MIL transfer…"


We discuss supervised MIL pretraining (Shao et al., 2025) in the introduction and explicitly distinguish our methodological contribution from it. It requires a labelled in-domain source corpus (thousands of expert-annotated whole-slide images) and a feature extractor matching the one used in pretraining, so among our twelve benchmarks it could be applied only to TCGA. In contrast, the question we investigate is how to pretrain a model for MIL problems in settings where no large task-relevant pretraining corpus is available. ICMIL is pretrained purely on synthetic data and is applied unchanged to molecules, natural images, CT, and histopathology.


[...] the efficiency evidence is narrow


To further assess the efficiency of ICMIL, we compare the wall time to obtain predictions on the test set of Pos/Neg (Table R2.3). We find that ICMIL generates predictions about three orders of magnitude faster than ACMIL, underlining the advantage of needing only a single forward pass and no supervised training or hyperparameter tuning per task.


Table R2.3. Wall time in seconds per data split on Pos/Neg. Mean ± standard error over data splits and seeds. Both methods were run on a single NVIDIA A100 GPU.

MethodTime (s)Speed-up
ACMIL28.4 ± 0.61×
ICMIL0.018 ± 0.004≈ 1,500x


ACMIL is the only recent aggregator; RRT-MIL and MambaMIL (2024) are absent.


We have added MambaMIL, trained and tuned with the same protocol as ABMIL, DSMIL and ACMIL and reported over three seeds (see table R2.1). MambaMIL reaches an average AUROC of 77.99, compared with 84.17 for ICMIL and 85.25 for ICMIL (CE). This supports the claim of our paper that ICMIL is competitive on low-bag datasets with methods that require task-specific training.

Reviewer_No6mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces In-Context Multiple Instance Learning (ICMIL), a Prior-data Fitted Network for learning bag-level classification tasks from a small set of labeled bags without gradient-based adaptation. The proposed Perceiver-style architecture alternates between within-bag instance aggregation and cross-bag attention, allowing aggregation to depend on the labels and examples supplied in context while preserving permutation invariance.

The authors also develop synthetic task priors for pretraining: factorized priors encode conventional instance-to-bag aggregation assumptions, whereas joint priors permit correlations and interactions among instances. Experiments suggest that these priors have complementary strengths and that training on their mixture produces the most robust model. Across twelve MIL benchmarks, the scaled model obtains an average AUROC of 84.17, slightly above the strongest supervised baseline, ACMIL, at 83.81. Architectural ablations, prior comparisons, a synthetic aggregation task, and runtime measurements provide additional validation.

Critical strengths
  • A compelling formulation of few-shot MIL as amortized in-context inference. The alternating instance-aggregation and cross-bag attention architecture addresses an important limitation of fixed, label-agnostic bag representations. It is technically well motivated and could be useful well beyond the particular benchmarks studied here.

  • Thoughtful synthetic-prior design and broad empirical evaluation. The comparison between joint and factorized priors reveals meaningful complementary inductive biases. Evaluating on twelve benchmarks spanning witness, interaction, correlated, and uncorrelated regimes is substantially stronger than demonstrating the method on only one application domain.

Critical weaknesses
  • The claimed advantage over supervised methods is small and not statistically significant. ICMIL’s average AUROC is 84.17 versus 83.81 for ACMIL, a difference of only 0.36 points, while ICMIL does not consistently dominate individual benchmarks. No paired confidence interval or statistical test is reported for the aggregate comparison. Consequently, the evidence does not yet establish that ICMIL is reliably better rather than approximately competitive.

  • The experimental methodology may favor ICMIL and risks benchmark-level overfitting. The same twelve benchmarks appear to inform the choice of prior components, mixture, scaling schedule, and final assessment. Moreover, neural baselines select among many hyperparameter configurations using a validation subset of roughly nine bags, whereas ICMIL can use all labeled context bags. It is unclear whether the selected baselines are subsequently retrained on all training data. These issues are consequential because the paper’s principal claim rests on a very narrow aggregate performance margin.

Questions for authors
  • Were the prior family, mixture components, mixture weights, training duration, and checkpoint selected using performance on these same twelve benchmarks or their test splits? Were any downstream tasks held out entirely during method selection?

  • How does ICMIL perform with 5, 10, 20, and 50 labeled context bags across several natural as well as constructed benchmarks? This would substantiate the central low-label claim and test generalization outside the 70–125-bag pretraining range.

5 comments
AuthorsOpenReview ↗

Response to Reviewer No6m (Part I)

We thank the reviewer for their constructive and valuable feedback and are happy that they find our formulation compelling, and well motivated. We address each of the points below.

The claimed advantage over supervised methods is small and not statistically significant. [...] ICMIL does not consistently dominate individual benchmarks.

We are happy to change the phrasing to "competitive with the best supervised baseline" (Avg AUROC: ICMIL 84.17 versus 83.81 ACMIL). The following two points put this into context. Firstly, ICMIL reaches this performance without any training on the target dataset. Secondly and more importantly, the exact size of the lead is not central to the contribution of our paper.

In our paper we show that it is possible to train a single model that can be used unchanged and without any gradient updates across datasets and that matches or outperforms supervised methods like ACMIL or ABMIL on average in the low-bag regime. Establishing how this can be done and that this works at all is the point of our contribution. In addition, our paper opens up many avenues for future work (e.g. richer priors, fine-tuning, post-training, harnesses), which we expect to be fruitful for the community. Neither depends on the exact margin in any specific comparison against a supervised model.

To illustrate the potential of ICMIL for further improvement, we show that it can perform context ensembling. That is, we average predictions over multiple forward passes, each with a different transformation of the dataset and a different subsample of bags, without any weight updates. We use one fixed configuration for all datasets, and it increases ICMIL’s average AUROC by 1.08 percentage points (see Table R1.1). Ensembling over transformed versions of the dataset is standard for tabular in-context learners [1,2]. Since this procedure requires no further training, it is not directly applicable to supervised methods.

Table R1.1. Benchmark performance of ICMIL with context ensembling (CE) and the best supervised baseline (ACMIL).

ModelAvg AUROCMusk1Musk2HEPMASSLettersSMILAdj. PairsPos/NegRSNA ICHElephantFoxTigerTCGA
ACMIL83.8189.0 ±1.383.9 ±2.581.1 ±3.998.7 ±1.387.7 ±1.072.5 ±0.680.8 ±0.474.8 ±1.094.1 ±0.264.4 ±1.788.0 ±1.290.7 ±0.7
ICMIL84.1793.3 ±0.690.8 ±1.184.5 ±0.795.7 ±0.774.4 ±0.377.6 ±0.587.0 ±0.275.0 ±0.194.2 ±0.160.7 ±0.888.9 ±0.687.9 ±0.1
ICMIL (CE)85.2594.7 ±0.0892.2 ±0.6085.1 ±0.2797.3 ±0.2776.2 ±0.4877.9 ±0.3287.9 ±0.3575.3 ±0.0594.4 ±0.2363.4 ±0.7690.2 ±0.1788.5 ±0.19

The experimental methodology may favor ICMIL and risks benchmark-level overfitting. The same twelve benchmarks appear to inform the choice of prior components, mixture, scaling schedule, and final assessment. [...] Were the prior family, mixture components, mixture weights, training duration, and checkpoint selected using performance on these same twelve benchmarks or their test splits? Were any downstream tasks held out entirely during method selection?

The schedule was not selected using benchmark results. We made a single change to the training schedule, doubling training from 20,000 to 40,000 steps, with no sweep and no comparison of alternatives. All other hyperparameters, including the learning-rate schedule, were unchanged. We are happy to explicitly clarify this in the paper.

We did not tune the prior weights. We note that mixing the priors broadens the learned function class and makes the model more general and less benchmark specific. Within the mixture, we assigned a larger weight (0.7) to the SCM prior as it arguably represents a larger function space compared to the factorized priors that follow the standard MIL assumptions. We used this fixed weighting for all experiments and performed an ablation on the prior weight afterwards which can be found in App. C of the paper. The results indicate that better prior mixtures exist and prior curation is a promising direction for future work.

AuthorsOpenReview ↗

Response to Reviewer No6m (Part II)

To further confirm that ICMIL is not overfitting to the twelve benchmarks, we added three new benchmarks[3,4,5]:

  • ICB: single-cell RNA-seq. 57 bags with up to 243 instances each.
  • TAD: traffic anomaly detection in videos. 512 bags, subsampled to 1,000 instances each.
  • COVID: single-cell RNA-seq. 50 bags with up to 1,000 instances each.

ICMIL compares favorably on ICB and COVID and is within 0.2 points of ACMIL on TAD (Table R1.2), showing that ICMIL generalizes beyond the twelve benchmarks in the paper.

Table R1.2: AUROC (%) ± SE over three seeds, including the strongest paper baseline ACMIL.

MethodAvgICBTADCOVID
MeanLogReg80.266.6 ± 0.4987.1 ± 0.8387.0 ± 0.00
ACMIL78.263.2 ± 0.7091.2 ± 0.2080.2 ± 1.06
ICMIL84.273.0 ± 0.6391.0 ± 0.3888.5 ± 0.09
ICMIL (CE)84.873.3 ± 0.3691.9 ± 0.3689.2 ± 0.06

Moreover, neural baselines select among many hyperparameter configurations using a validation subset of roughly nine bags, whereas ICMIL can use all labeled context bags. It is unclear whether the selected baselines are subsequently retrained on all training data.


We thank the reviewer for bringing this to our attention. Indeed, ICMIL used all context bags while supervised MIL baselines used 10% of the bags as a validation set for hyperparameter selection. We therefore added a new protocol that subsequently retrains these methods on all training data (Table 1.3). While ABMIL did not profit from retraining on average, ACMIL’s average performance increased by roughly a half percentage point. Overall, ICMIL (CE) attains the best average performance.

Table R1.3. Benchmark performance when retraining baselines on all training data. Bold numbers are within SE of the best.

ModelAvg AUROCMusk1Musk2HEPMASSLettersSMILAdj. PairsPos/NegRSNA ICHElephantFoxTigerTCGA
ABMIL79.9785.7 ±3.575.4 ±2.875.7 ±1.797.6 ±0.885.4 ±0.565.5 ±1.174.6 ±1.671.2 ±0.394.1 ±0.259.8 ±1.683.9 ±1.590.7 ±1.6
ABMIL + retrain79.8886.7 ±2.174.3 ±1.777.6 ±3.797.1 ±0.787.1 ±0.464.0 ±1.873.4 ±0.870.9 ±0.394.5 ±0.457.7 ±1.284.4 ±2.090.8 ±1.6
ACMIL83.8189.0 ±1.383.9 ±2.581.1 ±3.998.7 ±1.387.7 ±1.072.5 ±0.680.8 ±0.474.8 ±1.094.1 ±0.264.4 ±1.788.0 ±1.290.7 ±0.7
ACMIL + retrain84.3290.1 ±2.884.2 ±2.581.5 ±1.5100.0 ±0.089.3 ±0.472.7 ±1.781.5 ±1.374.7 ±0.894.4 ±0.565.6 ±0.487.6 ±1.090.3 ±0.6
ICMIL84.1793.3 ±0.690.8 ±1.184.5 ±0.795.7 ±0.774.4 ±0.377.6 ±0.587.0 ±0.275.0 ±0.194.2 ±0.160.7 ±0.888.9 ±0.687.9 ±0.1
ICMIL (CE)85.2594.7 ±0.0892.2 ±0.6085.1 ±0.2797.3 ±0.2776.2 ±0.4877.9 ±0.3287.9 ±0.3575.3 ±0.0594.4 ±0.2363.4 ±0.7690.2 ±0.1788.5 ±0.19
AuthorsOpenReview ↗

Response to Reviewer No6m (Part III)


How does ICMIL perform with 5, 10, 20, and 50 labeled context bags across several natural as well as constructed benchmarks? This would substantiate the central low-label claim and test generalization outside the 70–125-bag pretraining range.


We thank the reviewer for this suggestion. To assess performance in the low-label regime and outside the pretraining range, we sample fractions of 0.25x to 4x the number of training bags used in the paper for the Pos/Neg and RSNA ICH benchmarks. We do not evaluate lower numbers of training bags because the supervised baselines require a 10% validation split, which would contain fewer labels than classes at ten bags. For the newly added TAD-50 benchmark, we used a variant of ICMIL trained on 50 dimensions (details in answer to RyQH) on subsamples of the full training split (408 bags). ICMIL keeps improving with more context bags on all three benchmarks, including beyond the pretraining range. Relative to the best supervised MIL baseline, ICMIL leads throughout on Pos/Neg, leads up to 50 bags on RSNA ICH before being overtaken and is on par with ACMIL on TAD-50. Overall, ICMIL is most useful when labeled bags are scarce, and supervised methods catch up or overtake as the number of labels increases.

Table R1.4. Performance on Pos/Neg for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).

Method224590180360
ICMIL76.7 ± 0.983.8 ± 0.487.0 ± 0.288.2 ± 0.388.9 ± 0.1
ABMIL62.7 ± 2.869.8 ± 2.174.6 ± 1.679.7 ± 0.386.7 ± 0.5
ACMIL73.0 ± 1.777.3 ± 0.680.8 ± 0.483.5 ± 1.186.7 ± 0.3
Δ+3.7+6.5+6.2+4.7+2.2

Table R1.5. Performance on RSNA ICH for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).

Method2550100200400
ICMIL67.3 ± 0.171.7 ± 0.175.0 ± 0.176.6 ± 0.177.7 ± 0.2
ABMIL62.8 ± 1.165.5 ± 1.571.2 ± 0.375.4 ± 0.780.1 ± 1.0
ACMIL65.4 ± 0.769.9 ± 0.674.8 ± 1.079.8 ± 1.184.4 ± 0.4
Δ+1.9+1.8+0.2-3.2-6.7

Table R1.6. Performance on TAD-50 for different numbers of training bags. Mean test AUROC (%) ± standard error over three seeds. Δ is the difference between ICMIL and the best of the two baselines shown (positive means ICMIL is better).

Method2550100200408
ICMIL75.9 ± 0.185.0 ± 0.690.3 ± 0.792.5 ± 0.693.3 ± 0.8
ABMIL73.1 ± 2.278.7 ± 1.182.1 ± 2.786.3 ± 3.287.2 ± 3.9
ACMIL77.4 ± 1.483.5 ± 0.788.1 ± 0.692.3 ± 0.293.5 ± 0.6
Δ-1.5+1.5+2.2+0.2-0.2

We thank the reviewer for helping to improve our paper and hope these additional experiments and clarifications address the reviewer's remaining concerns. The anonymous repository now contains the code and checkpoints needed to reproduce the new experiments, including the ensemble, and benchmarks reported here.


[1] Prior Labs Team, "TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models"

[2] Prior Labs Team, "TabPFN-3: Technical Report"

[3 ]Ziegler et al., Impaired local intrinsic immunity to SARS-CoV-2 infection in severe covid-19, 2021

[4] Gondal et al.. Integrated cancer cell-specific single-cell RNA-seq datasets of immune checkpoint blockade-treated patients, 2024

[5] Lv et al., Localizing Anomalies from Weakly-Labeled Videos, 2020

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion for the Area Chair

Dear Area Chair,

Thank you for handling our submission. As the reviewer-AC discussion begins, we would like to briefly summarize where the discussion stands.


Current Status. The ratings are 3 (RyQH), 2 (No6m) and 2 (db6x, raised from 1). Only Reviewer db6x has responded to the rebuttal. Reviewers No6m and RyQH have not yet reacted, so their ratings still reflect the original submission. We would be grateful if you could invite them to consider the rebuttal, which answers each of their questions and adds the experiments they asked for.


Strengths. All three reviewers regard the approach as novel and report that they learned something from the paper. Reviewer No6m calls it “a compelling formulation of few-shot MIL as amortized in-context inference”, and Reviewer db6x recognizes ICMIL as “the first in-context learner for bag-level functions” and notes that it is “fully reproducible”.


Concerns that we addressed during the rebuttal:

(1.) The reviewers found the wording of the paper too strong, as it stated that ICMIL outperforms supervised baselines without task-specific training. We addressed this by changing the wording throughout the revised paper to “competitive with the best supervised baselines”.

(2.) The reviewers (No6m, db6x) were unsure whether the training schedule or the prior weights were selected on the benchmarks. Furthermore, there were concerns (No6m) that the baselines were disadvantaged by their validation split. We addressed this by clarifying that neither the training schedule nor the prior weights were tuned on benchmark results and by re-evaluating the strongest baseline under an adjusted protocol (ACMIL + retrain). We furthermore added three new benchmarks (ICB, TAD, COVID) to demonstrate that ICMIL generalizes beyond the benchmarks in the paper. Over all 15 benchmarks, ICMIL reaches 84.17 average AUROC vs. 83.09 for retrained ACMIL.

(3.) All three reviewers asked for additional experiments that investigate how ICMIL generalizes beyond the pretraining range. To address this we have (i) added experiments where we evaluate different methods while varying the number of available training bags, (ii) included benchmarks with a large number of instances per bag (up to 1,000), and (iii) trained an ICMIL model with 50 features to show that the method is general and can be used to train models with higher-feature dimensions.

(4.) Reviewer db6x asked for MambaMIL and additional tabular foundation model baselines. To address this we have (i) added MambaMIL as a further recent aggregator, (ii) evaluated TabPFN on pooled bag vectors.

Reviewer db6x, the only reviewer to respond so far, confirms that the rebuttal “resolves much of my original review”.


Remaining disagreement. Reviewer db6x’s final rating rests on a single comparison, the margin over Summ-TabPFN 3.5. We would like to ask the AC to weigh the following:

(1.) Our main contribution is methodological and we show how to learn a bag aggregation function in-context. Summ-TabPFN is based on a fixed aggregation (mean, max,...). They are thus methodologically clearly distinct. On some tasks fixed aggregation works well (e.g. Fox, Elephant) while on others ICMIL outperforms Summ-Tabpfn by several percent (ICB: 73.3 vs. 68.6; COVID: 89.2 vs. 85.9).

(2.) We added this baseline ourselves during the rebuttal. TabPFN 3.5 was released in September 2026, days before the submission deadline, and is not peer-reviewed nor reproducible. We would consider it concurrent work. We included it for transparency.


For the camera-ready version we will include the retrained baselines and the three new benchmarks in Table 1. Code and checkpoints for all new experiments are in the anonymous repository. We hope that the paper opens up a new direction of interest to the ICLR community.

We thank the area chair for their consideration.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.