- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PathRet, a two-tier evaluation benchmark comprising 4,249 tissue slides from 3,980 patients across 13 cancer types. The first tier assesses correlations with subtypes and staging using rank-based correlation metrics and Normalized Discounted Cumulative Gain, while the second tier evaluates visual relevance within the fixed feature space of CONCH v1.5. The authors compared seven prominent retrieval systems using a shared dataset of 4,240 slides. A key finding from this analysis is that relying solely on label consistency to evaluate these systems can mask significant differences in the quality of instance-level retrieval.
Critical strengths
(1) The motivation is clear, and the hierarchical retrieval approach makes sense and is intuitive.
(2) As a benchmark, the study evaluates a variety of popular methods. The results are comprehensive and well-documented, further validating the necessity of a two-tiered evaluation framework.
Critical weaknesses
(1) While retrieving images based on visual similarity, independent of explicit labels, is intuitively appealing, it presents a challenge: such retrieval is heavily dependent on the source model and its specific training data. Consequently, different models may yield vastly different sets of "visually similar" results, making their rankings difficult to compare, especially in the absence of calibration by pathologists. Consistency across multiple encoders merely demonstrates that these models share similar preferences. It does not prove that these preferences constitute a valid or appropriate evaluation objective. Furthermore, beyond the issue of reliable evaluation, the dataset size for this benchmark appears somewhat limited. These are two particularly challenges.
(2) Section 5.2 compares slide identities for queries where both systems successfully retrieve the correct subtype, while Section 5.3 compares the clinical and visual scores of PRISM and Prov-GigaPath across the entire evaluation set. In the latter case, the diagnostic relevance of the results returned by the two systems is not held constant. This means that the visual discrepancies shown in Table 2 conflate two factors: which diagnostic categories are retrieved, and which specific slides are selected within those categories. Because the second stage lacks the cancer-type gating present in the first, results that are diagnostically incorrect, yet possess high encoder-based similarity, can still inflate the overall visual score. Therefore, while an inversion in the overall ranking demonstrates that the two metrics capture different attributes, it does not definitively prove that there is a consistent, directional difference in the quality of instance selection between the actual systems when diagnostic accuracy is held constant.
Questions for authors
See weaknesses above.