acceptodds
Withdrawn from ICLR 2027

PathRet: Benchmarking Whole-Slide Image Retrieval Beyond Label Agreement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Whole-slide image (WSI) retrieval returns prior cases that pathologists can directly compare, but conventional label-based evaluation measures diagnostic agreement with the query rather than which slides are actually returned. Replacing a retrieved slide with another assigned the same diagnostic relevance at the same rank leaves the evaluation unchanged. Such metrics can therefore measure whether the right kind of case was retrieved, but cannot distinguish which instances were selected within that diagnostic group. We compare seven retrieval systems from different methodological families and analyze queries for which a pair of systems each returns five slides with the query’s exact subtype. In these cases, label-based evaluation regards both systems as fully successful. Yet among 474 such queries for PRISM and Yottixel-K, 72.8% have no slide in common between the two returned top-five sets. Thus, retrievals that appear equally successful under diagnostic labels can present substantially different cases to the user. To address this limitation, we introduce PathRet, a benchmark of 4,249 WSIs from 13 cancer types that evaluates both label-defined clinical relevance and visual relatedness of the retrieved slides under a fixed retrieval protocol. Visual relatedness is measured by embedding WSI patches with a fixed pathology foundation encoder and comparing the query and retrieved slides at the slide level; an auxiliary measure compares their visual-phenotype distributions. When applied to real retrieval systems, the two views can lead to different system comparisons. PRISM scores higher than Prov-GigaPath on subtype and stage relevance, whereas Prov-GigaPath scores higher on both slide-level representation similarity and phenotype-composition similarity. This shows that a system retrieving more diagnostically relevant cases does not necessarily select cases that are more visually related to the query. WSI retrieval should therefore distinguish retrieving diagnostically appropriate cases from selecting visually related cases within those diagnostic groups, rather than evaluating retrieval through a single view alone.

Primary area:
datasets and benchmarks
Keywords:
whole-slide image, retrieval, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YejkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PathRet, a two-tier evaluation benchmark comprising 4,249 tissue slides from 3,980 patients across 13 cancer types. The first tier assesses correlations with subtypes and staging using rank-based correlation metrics and Normalized Discounted Cumulative Gain, while the second tier evaluates visual relevance within the fixed feature space of CONCH v1.5. The authors compared seven prominent retrieval systems using a shared dataset of 4,240 slides. A key finding from this analysis is that relying solely on label consistency to evaluate these systems can mask significant differences in the quality of instance-level retrieval.

Critical strengths

(1) The motivation is clear, and the hierarchical retrieval approach makes sense and is intuitive.

(2) As a benchmark, the study evaluates a variety of popular methods. The results are comprehensive and well-documented, further validating the necessity of a two-tiered evaluation framework.

Critical weaknesses

(1) While retrieving images based on visual similarity, independent of explicit labels, is intuitively appealing, it presents a challenge: such retrieval is heavily dependent on the source model and its specific training data. Consequently, different models may yield vastly different sets of "visually similar" results, making their rankings difficult to compare, especially in the absence of calibration by pathologists. Consistency across multiple encoders merely demonstrates that these models share similar preferences. It does not prove that these preferences constitute a valid or appropriate evaluation objective. Furthermore, beyond the issue of reliable evaluation, the dataset size for this benchmark appears somewhat limited. These are two particularly challenges.

(2) Section 5.2 compares slide identities for queries where both systems successfully retrieve the correct subtype, while Section 5.3 compares the clinical and visual scores of PRISM and Prov-GigaPath across the entire evaluation set. In the latter case, the diagnostic relevance of the results returned by the two systems is not held constant. This means that the visual discrepancies shown in Table 2 conflate two factors: which diagnostic categories are retrieved, and which specific slides are selected within those categories. Because the second stage lacks the cancer-type gating present in the first, results that are diagnostically incorrect, yet possess high encoder-based similarity, can still inflate the overall visual score. Therefore, while an inversion in the overall ranking demonstrates that the two metrics capture different attributes, it does not definitively prove that there is a consistent, directional difference in the quality of instance selection between the actual systems when diagnostic accuracy is held constant.

Questions for authors

See weaknesses above.

Reviewer_pu99OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PathRet, a benchmark for whole-slide image retrieval that distinguishes diagnostic label agreement from instance-level visual relatedness. The authors argue that conventional label-based retrieval metrics cannot differentiate systems that return different slides with identical diagnostic relevance. They formalize this limitation through substitution invariance and demonstrate its practical prevalence by analyzing retrieval-set overlap across existing WSI retrieval systems.

PathRet contains 4,249 WSIs from 13 cancer types and evaluates retrieval along two separate dimensions. The first measures clinical relevance using subtype- and stage-based NDCG. The second measures visual relatedness using a fixed CONCH v1.5 patch encoder, with an auxiliary phenotype-composition similarity metric. Seven retrieval systems are evaluated under a common protocol. The authors additionally investigate label-equivalent retrievals, alternative visual evaluators, source-site effects, and preprocessing sensitivity.

The results demonstrate that diagnostic relevance and representation-based visual similarity can produce different system comparisons, motivating their separate evaluation.

Critical strengths
  1. The paper identifies an important limitation of conventional WSI retrieval evaluation. Its distinction between class recovery and instance selection is well motivated by the intended use of retrieval systems, where users inspect actual retrieved cases rather than diagnostic labels alone. The empirical analysis of retrieval-set disagreement provides convincing evidence that label-based metrics cannot fully characterize retrieval behavior.

  2. The benchmark is carefully designed and extensively validated. It uses a common retrieval pool, patient-level exclusions, source-site controls, graded clinical relevance, and patient-cluster bootstrap confidence intervals. The evaluation of seven retrieval systems, together with cross-encoder robustness analyses and detailed documentation of data provenance, makes the benchmark a potentially useful resource for future pathology retrieval research.

Critical weaknesses
  1. The novelty of the proposed benchmark is relatively limited. The limitation of label-based retrieval metrics in distinguishing instances with identical relevance labels is already well understood in information retrieval. The proposed framework primarily combines existing techniques, including foundation-model embeddings, cosine similarity, and phenotype-distribution comparison. The paper does not sufficiently demonstrate what fundamentally new evaluation capability is introduced beyond these established approaches.

  2. The clinical validity of the proposed visual-relatedness metrics remains insufficiently established. Visual similarity in a pretrained feature space does not necessarily correspond to diagnostically meaningful morphological similarity or retrieval usefulness. Without independent pathologist assessments, it is difficult to determine whether the additional metrics provide a clinically relevant evaluation rather than an alternative representation-based ranking.

  3. The visual evaluation is highly dependent on the selected feature encoder. Although the authors evaluate multiple pathology foundation models, the overall rankings of retrieval systems change substantially across evaluators. For example, TITAN ranks first under CONCH v1.5 but fifth under UNI and Prov-GigaPath features. This sensitivity raises concerns about the reliability and general applicability of the proposed visual-relatedness benchmark.

  4. The evaluation is restricted to a single public pathology archive, limiting evidence of generalizability across institutions and acquisition conditions. Although the authors introduce source-site exclusions and patient-level controls, these do not substitute for evaluation on independent external cohorts. The robustness of the proposed evaluation framework under domain shifts therefore remains unclear.

Questions for authors
  1. Given that instance-level differences within the same relevance class are a known limitation of conventional retrieval metrics, what is the principal methodological or evaluation contribution of PathRet beyond combining existing label-based metrics with foundation-model similarity measures? How does it differ fundamentally from prior retrieval evaluation frameworks addressing relevance, diversity, and instance selection?

  2. Have the authors conducted any evaluation involving independent pathologists to determine whether the proposed visual-relatedness metrics correlate with expert judgments of morphological similarity or retrieval usefulness? Such evidence would be important to establish the practical significance of the additional evaluation dimension.

  3. The overall rankings of the seven retrieval systems change substantially across different visual encoders. How should benchmark users interpret improvements under CONCH v1.5 when the same retrieval system may rank considerably differently under other foundation models? What evidence supports the selection of CONCH v1.5 as the canonical evaluator?

  4. Have the authors evaluated the proposed benchmark or its visual-relatedness measures on an independent pathology dataset outside the TCGA-derived cohort? If not, what evidence supports their reliability across different institutions, staining variations, scanners, and tissue preparation protocols?

Reviewer_xvdgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines a limitation of label-based evaluation for whole-slide image retrieval: replacing a retrieved slide with another assigned the same diagnostic relevance at the same rank leaves the evaluation unchanged, even when the slides differ substantially in morphology. Label-based metrics therefore measure diagnostic agreement but cannot distinguish instance selection within the same relevance category.

The authors introduce PathRet, a benchmark containing 4,249 diagnostic WSIs from 3,980 patients across 13 TCGA projects. Layer 1 evaluates subtype and stage relevance separately. Layer 2 supplements these assessments with cosine similarity between mean-pooled CONCH v1.5 patch representations and similarity between visual-phenotype compositions. The evaluation demonstrates that systems can return different instances despite equivalent diagnostic relevance and that label-based and representation-based comparisons can favor different systems.

The evaluation problem is meaningful. My main concern is the limited independent validation of Layer 2’s interpretation as morphological relatedness or reference-case usefulness for pathologists. Evaluator dependence, the information retained by the visual summaries, and sensitivity to cohort selection also limit the broader interpretation of the benchmark results.

Critical strengths

S1: The paper identifies a meaningful blind spot in label-based evaluation. Cases with the same diagnosis can still have important morphological differences. The distinction between class recovery and instance selection clearly shows what label-based metrics miss.

S2: The benchmark provides a standardized comparison protocol. Shared cohorts, candidate pools, exclusion rules, scoring definitions, and patient-level uncertainty estimates improve comparability across retrieval systems.

S3: The authors conduct several useful robustness analyses. Cross-encoder evaluations, Source Site audits, and patch-sampling checks help characterize the measurements. The paper also acknowledges that complete visual rankings depend on the scoring representation and that Layer 2 has not been calibrated against pathologist judgments.

Critical weaknesses

W1. Layer 2 lacks independent calibration of its morphological interpretation. It measures similarity in specified learned representations, but it has not been calibrated against blinded pathologist judgments or independently annotated morphological relatedness. The paper explicitly acknowledges this limitation and does not establish clinical reference-case utility. Given its motivation of helping pathologists compare concrete cases, this validation gap limits how confidently differences in Layer 2 scores can be interpreted as meaningful differences for the intended users.

W2. The complete visual ranking depends on the evaluator, and the summaries may omit relevant local information. Appendix B.4 transparently reports that the PRISM–Prov-GigaPath comparison retains its direction across the tested representations, while the complete seven-system ordering changes. For example, TITAN moves from first under CONCH v1.5 to fifth under UNI and the Prov-GigaPath tile representation. This limits the interpretation of the complete ranking beyond the specified scoring space. In addition, mean-pooled features and phenotype-composition summaries can lose localized evidence and do not explicitly preserve spatial arrangement. The paper does not directly establish whether these omissions affect morphological judgments among candidates with equivalent diagnostic relevance.

W3. Sensitivity to cohort inclusion criteria remains unclear. The paper explains its primary-tumour label extraction and the reduction from 4,315 metadata-eligible slides to 4,249 usable slides. However, requiring stage information also restricts the cohort used for subtype and visual evaluation, neither of which inherently requires stage. An account of stage missingness across projects and Source Sites, together with a focused sensitivity analysis where feasible, would clarify whether the principal comparisons depend on this selection.

Questions for authors

Q1. What independent evidence supports interpreting Layer 2 as morphological relatedness? A focused, blinded assessment of equally diagnostically relevant candidates would help establish this interpretation.

Q2. Do the mean and phenotype-composition summaries miss localized morphology that changes the interpretation of same-label retrievals? Representative cases would help clarify this limitation.

Q3. What determined the selection of the 13 projects and the requirement for stage availability? How does stage missingness vary across projects and Source Sites?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.