acceptodds
Withdrawn from ICLR 2027

Cell-level representations in pathology foundation models are a question of read-out

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pathology foundation models (PFMs) are trained and evaluated on tiles or slides, yet many biologically relevant tasks are defined at the level of individual cells. A cell's identity depends both on its own morphology and on its surroundings, so a cell embedding should be specific to the cell while integrating its context. Current read-outs trade one for the other. One approach crops a small window around the cell, resizes it to the encoder's input size, and keeps the CLS token. This isolates the cell, but it discards the surrounding tissue and presents the image at a magnification far from the one used in pretraining. Keeping the CLS token of a wider tile retains context, but the embedding summarizes the whole tile and is shared by many, often heterogeneous, cells. We study spatial indexing, which removes this trade-off: a tile is centered on the cell, and the patch tokens at its location are read out. The tile size sets how much context the encoder integrates, the token position sets which cell is represented, and DINOv2's token-level iBOT objective makes these tokens informative. To evaluate read-outs at scale, we curate 111 public Xenium samples with registered H&E, comprising 20.75 million cells with gene expression and expression-derived cell-type labels, the largest cell-level annotated pathology dataset to date. Across four PFMs, spatially indexed tokens outperform the CLS token of the same tile by 0.10 macro-F1 on cell-type classification, crop-and-resize is the weakest read-out, and the gains extend to single-cell gene-expression prediction. Linear probes and iBOT-token masking confirm that a single token holds both sides of the trade-off and that both pieces of information are complementary for classification. Spatial indexing is also cheaper: reading out every cell of a tile from a single forward pass needs 1.4% of the forward passes at a small cost in accuracy, and because the tokens already carry context, the neighbor-aggregation module of a gene-expression model can be removed, cutting its compute 26. We release the resource to support the development of cell-level pathology models.

Primary area:
datasets and benchmarks
Keywords:
Computational pathology, Pathology foundation models, Single-cell representation learning, token readout, Vision transformers, Spatial transcriptomics, Xenium, H&E whole-slide imaging, Cell-type classification, Benchmark dataset, UNI2, Virchow2, HOptimus, Phikon, Automated cell-type annotation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_Qws4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to extract cell-level representations from existing pathology foundation models. It proposes spatial indexing: encode a cell-centered tile and average the patch tokens at the cell’s location, preserving both cell-specific information and surrounding context without retraining the encoder. The authors curate 111 H&E-registered Xenium samples and evaluate four foundation models on cell-type classification and single-cell gene-expression prediction. Spatially indexed tokens improve classification by approximately 0.10 macro-F1 over the CLS token of the same tile and also improve expression prediction. The paper further shows that multiple cells can be represented in one encoder pass and that explicit neighbor aggregation can be removed in its expression-prediction setup with little loss of performance.

Critical strengths
  1. A consistent and consequential finding across models. Spatially indexed tokens outperform the CLS token by about 0.10 macro-F1 across four pathology foundation models, with gains also observed in gene-expression prediction. This shows that read-out choice materially affects cell-level performance without requiring a new encoder.

  2. Practical value and a substantial data resource. The authors curate 111 H&E-registered Xenium samples and show that reading multiple cells from one encoder pass reduces forward passes by roughly 70× with only a small classification drop. Both the resource and the efficiency result make the finding useful beyond the reported benchmarks.

Critical weaknesses
  1. Limited methodological novelty. Reading cell embeddings from spatially indexed ViT tokens has appeared in prior cell-level pipelines, and concurrent work reports advantages over tile-level pooling. This paper provides a valuable systematic evaluation, but its central method is a simple read-out choice rather than a new representation-learning approach. That limits its contribution at ICLR.

  2. Narrow evaluation relative to the paper’s claims. Although the curated resource contains 111 samples, the main benchmarks use only 7 pan-cancer, 5 colon, and 15 lung slides, with classification restricted to five broad cell types. The evidence is therefore limited for claiming that this read-out should be a general default across tissues and finer cell-level tasks.

Questions for authors
  1. Novelty (clarification): How does spatial indexing differ from token-based cell feature extraction in CellViT++, CellSymphony, and CellPath-Bench? Which part of the proposed read-out is new, rather than newly evaluated here?

  2. Evaluation scope: How were the 7 pan-cancer, 5 colon, and 15 lung slides selected from the 111-sample resource? Do the read-out gains hold on additional available slides and with finer-grained cell-type labels?

  3. Additional baselines: Could the authors add a matched comparison with the coordinate-sampling read-out used in CellPath-Bench, using the same frozen encoders, data splits, and downstream classifier? This would help establish whether the proposed four-token read-out offers a benefit over the closest existing approach.

Reviewer_7iAjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper benchmarks the read-out options in pathology foundation models for cell-level tasks, including the CLS token, central tokens around the cell, and nucleus tokens obtained by masking the cell, as well as a mix of central tokens and the CLS token. The preferred readout, named spatial indexing, demonstrates stronger performance. The authors conduct their benchmarking on 111 public Xenium and H&E samples.

Critical strengths

The question is interesting and relevant to the community since pathology foundation models are indeed trained on either the patch level or the slide level and how to use their embeddings for cell level tasks is an open question.

Critical weaknesses

The reviewer is critically concerned about (1) novelty and (2) completeness each detailed below.

  1. Novelty: The paper neither proposes a new model nor a new algorithm. Even though the question at hand is intriguing, the reviewer struggles to see an innovative contribution to the field.
  2. Completeness: The proposed benchmark is incomplete and certain design choices are confusing due to the following reasons.
  • First, the authors assemble 111 public Xenium samples with registered H&E samples. Given that they have access to fine-grained cell-type labels through the Xenium transcripts, it is unclear to me, why the authors chose only five coarse labels for their benchmark. I believe analysing the performance of different readouts in more difficult scenarios, for instance, in large label spaces, would strengthen the paper.
  • Accordingly, as mentioned by the authors, there are multiple well-benchmarked cell-level datasets, such as, but not limited to, PathoCell and NuCLS. A complete benchmark is expected to include these datasets. It is challenging to accept spatial indexing to be the best readout depending on results from a single dataset.
  • Second, the authors use only four pathology foundation models, which limits the diversity in their benchmark.
  • Third, there is no discussion regarding recent cell-level foundation models, including VOLTA [1], which is expected in a benchmark about cell-level performance.
  • Finally, given the reported standard deviations, the benefit of spatial indexing is unclear. Please test statistical significance.

[1] Nakhli, R., et al. "VOLTA: an enVironment-aware cOntrastive ceLl represenTation leArning for histopathology." Nature Communications. 2024.

Questions for authors
  • What is the motivation behind curating paired Xenium and H&E datasets while using five coarse cell-type labels?
  • What is the performance of well-established cell-level foundation models, e.g. VOLTA, in your dataset?
  • Given the high standard deviations reported in the tables, are your results statistically significant?
Reviewer_WnU7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to extract cell-level representations from pathology foundation models (PFMs) pretrained on image tiles. The authors argue that existing read-out strategies struggle to balance cell specificity and contextual information: cropping and resizing individual cell regions discards surrounding tissue context, whereas extracting the CLS token from a larger image tile produces a global representation that lacks specificity to the target cell.

To address this trade-off, the paper investigates spatial indexing, which extracts spatially localized patch tokens corresponding to a target cell from a cell-centered image tile. This approach directly leverages token-level representations learned through DINOv2/iBOT pretraining, without modifying or retraining the underlying encoder. The authors compare several read-out strategies, including CLS tokens, central patch tokens, nucleus-overlapping tokens, and their combinations. These strategies are evaluated across four PFMs on cell-type classification, single-cell gene-expression prediction, and computational efficiency. In addition, the authors curate a resource comprising 111 publicly available Xenium samples with paired H&E images, gene-expression measurements, and cell-type annotations, and use linear probing and masking experiments to investigate the contributions of cell-intrinsic and contextual information.

Critical strengths
  1. The paper addresses a practically important problem in computational pathology: how to obtain cell-specific yet context-aware representations from foundation models pretrained at the image-tile level.

  2. The authors systematically compare multiple read-out strategies across four PFMs and different downstream tasks. The relatively consistent performance improvements provide useful empirical guidance for applying existing foundation models to cell-level analysis.

  3. The paper employs linear probing, masking experiments, and expression-divergence analysis to investigate the contributions of cell-intrinsic and contextual information. It also explores potential computational benefits through multi-cell feature extraction and reduced neighborhood aggregation.

Critical weaknesses
  1. The methodological novelty is limited. The core approach essentially extracts and pools spatially localized patch tokens from existing ViT feature maps, a strategy closely related to prior work (CellViT++, Hörst et al., 2025; CellPath-Bench, Zhao et al., 2026; CELLO, Gao et al., 2026). Moreover, the ability of patch tokens to encode local semantic information is inherited from DINOv2/iBOT pretraining. The paper introduces no new representation learning mechanism.

  2. The proposed mechanistic explanation is insufficiently validated. Changing the image crop size simultaneously alters the amount of contextual information and the input scale, making their respective contributions difficult to disentangle. The masking experiments are conducted only with Phikon-v2, and no controlled ablation is provided to establish the specific role of iBOT pretraining.

  3. The reliability and reproducibility of the curated dataset are insufficiently established. The 111 samples are assembled from existing public datasets, yet the released code lacks complete pipelines for image registration, cell-type annotation, and dataset construction. The cell counts reported in the README also differ from those in the manuscript. Furthermore, the claimed high-confidence cell-type annotations are primarily derived from majority voting among three computational annotation methods, without sufficient independent validation.

  4. The benchmark evaluation is limited in scope. Although the paper reports a resource containing 20.75 million cells, the main experiments involve only approximately 2.62 million cell entries across three benchmarks, with some overlap between datasets. The number of independent test slides for classification is limited, only five coarse-grained cell categories are considered, and no independent evaluation on established external benchmarks is reported.

  5. The released gene-expression prediction pipeline raises concerns about potential test-set leakage. When no independent validation set is provided, the DeepCell implementation uses the test set as the validation set and subsequently relies on its performance for early stopping and hyperparameter selection. This may compromise the validity of the evaluation reported in Table 3. Although it remains unclear whether the reported results were generated using this exact configuration, the authors should provide the original experimental configurations and execution logs to clarify this issue.

  6. Discrepancies between the released analysis code and the manuscript raise concerns about the reliability of some mechanistic findings. The analysis underlying Figure 3b does not properly match masked and unmasked predictions by cross-validation split, resulting in duplicated observations and potentially biased statistics. In addition, the code associated with Table 10 uses central tokens, whereas the manuscript describes nucleus-overlapping tokens. These inconsistencies directly affect the credibility of the corresponding mechanistic analyses and warrant corrected implementations and re-evaluation.

Questions for authors

See Weaknesses

Reviewer_6cSbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at how to get one embedding per cell out of a pathology foundation model. It compares crop-and-resize with the CLS token, the CLS token of a cell-centered wider tile, and "spatial indexing", which reads out the patch tokens at the cell's location. The authors build a resource of 111 Xenium samples with registered H&E (20.75M cells, expression-derived cell types) and evaluate four foundation models on cell-type classification and single-cell expression prediction. They add linear probes and an iBOT mask-token experiment to explain the gain, and test multi-cell extraction and removal of a neighbor aggregation module for compute savings.

Critical strengths

The paper asks a practical question and answers it with a controlled design. The encoder is fixed, only the read-out and field of view change, and results are paired within leave-one-slide-out splits, which matters because per-slide difficulty varies a lot. The result is consistent across four foundation models and two benchmarks: central tokens beat the CLS token of the same tile by about 0.10 macro-F1, and crop-and-resize is the weakest. Practitioners using these models for cell-level work can act on that directly.

The mechanism analysis is the part I like most. The masking experiment, where the cell is replaced by the iBOT mask token, shows that context alone keeps about 93% of the performance and that the cell adds a further 0.04, mostly for cells that differ from their surroundings. The efficiency results are also useful, since reading many cells from one forward pass at about 1.4% of the cost for a loss of 0.018 macro-F1 is a good trade for large slides.

Critical weaknesses

The headline comparison against crop-and-resize is less convincing than it looks. The crop-and-resize baseline is a single 100 px crop upsampled by 2.24 times, and the paper itself notes that field of view and scale are confounded. A fairer baseline would tune the crop size and resolution, and also include token mean pooling over the whole tile or a small trained head on the token map, which are the obvious simple alternatives. Without that, "spatial indexing beats the common practice" partly reflects a poorly set up baseline. Prior and concurrent work already reports that token-level sampling beats CLS and pooling, so the new contribution rests on the crop-and-resize comparison, the mechanism analysis, and the dataset.

The labels and the mechanism evidence also limit the claims. Cell types come from a majority vote over three expression-based methods, collapsed into five coarse classes, and the paper itself points to transcript spillover in Xenium as a source of noise. Absolute macro-F1 is low, often 0.4 to 0.7, so differences are being measured against noisy targets and the unfiltered ceiling is unclear. The masking analysis uses only Phikon-v2 on colon, and the probes use one model on one tissue, so the statement that a single token holds both sides of the trade-off is shown in a narrow setting. For expression prediction the gain is small (0.02 Pearson on a base of 0.18). The resource is described as the largest annotated cell-level dataset, but I could not see validation of its labels against pathologist annotation or a clear release plan in the main text.

Questions for authors
  1. Can you add a tuned crop-and-resize baseline (several crop sizes, resized to the pretraining scale) and a whole-tile token mean pooling baseline, and show whether the 0.10 gap over CLS and the gap to crop-and-resize hold?

  2. How does the ranking of read-outs change if you evaluate only on cells where the three annotation methods agree with high confidence, or against a small pathologist-annotated set such as PanNuke or Lizard?

  3. For the 26 times compute reduction from dropping the neighbor module, what is the accuracy cost on expression prediction at equal K, and does it hold on the colon benchmark and with other foundation models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.