- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes XPathAgent, a concept-driven, multi-tool collaborative reasoning framework for computational pathology, aiming to address the lack of explicit diagnostic evidence organization in existing pathology agents and the difficulty of assessing the reliability of outputs from different specialized tools. The authors evaluate the framework on 13 benchmark-task combinations covering cancer grading, subtyping, metastasis detection, stromal tumor-infiltrating lymphocyte (sTIL) scoring, and survival prediction, and validate its effectiveness through component ablations, leave-one-tool-out analyses, and in-context learning experiments. Overall, this paper presents a clinically reasonable framework for evidence organization and tool collaboration, with relatively broad experimental coverage. However, whether the output stability measured by ICR can represent diagnostic correctness, and the extent to which the framework depends on predefined medical knowledge and tool configurations, require further justification.
Critical strengths
-
XPathAgent decomposes pathology tasks into explicit morphological concepts, measurement tools, and evidence integration rules through the Diagnostic Atlas and Concept–Evidence Tree, allowing the reasoning process to be traced back to specific tissue regions and tool outputs. This explicit organization helps improve the transparency and auditability of the reasoning process.
-
ICR does not assume that all tool outputs are equally reliable. Instead, it adjusts evidence weights by considering image-quality perturbations, tool output stability, and cross-tool correlations, providing a structured mechanism for integrating heterogeneous pathology models.
Critical weaknesses
-
ICR measures stability using Spearman correlations between tool outputs before and after controlled perturbations and determines evidence weights accordingly. However, stable outputs are not necessarily correct, and models with systematic diagnostic biases may also remain consistent under perturbations. Furthermore, correlations between cross-tool perturbation responses do not directly represent correlations between diagnostic errors. The paper lacks direct validation of the relationship between ICR scores and actual tool prediction accuracy, leaving the effectiveness of ICR as a reliability assessment mechanism insufficiently supported.
-
The paper constructs offline stability curves using a small number of unlabeled images from each benchmark and then estimates tool reliability based on the severity of image-quality degradation in test regions. However, how the severity of resolution degradation, defocus, and staining deviations is automatically determined in test images is not sufficiently explained. Furthermore, each benchmark requires a separate calibration set, making it unclear whether the method can be directly applied to new data distributions.
-
Although XPathAgent is positioned as a plug-and-play framework, its task decomposition, concept selection, tool matching, and final evidence aggregation depend on the manually curated Diagnostic Atlas. New organs or diagnostic tasks still require additional construction and review of relevant concepts and rules. Furthermore, the organ set defined in the methodology does not include the lung, bladder, and endometrium, which are involved in the experiments. These configurations require further clarification.
-
The paper emphasizes its ability to construct reliable and traceable concept-to-evidence reasoning chains, but the main experiments still focus on final classification, grading, and survival prediction performance. Independent evaluations of ROI localization accuracy, tool selection appropriateness, concept measurement accuracy, and the clinical validity of evidence chains are lacking. Therefore, improvements in final prediction performance are insufficient to demonstrate that the intermediate evidence itself is more reliable.
-
The paper claims that XPathAgent achieves the best performance across all 13 benchmark-task combinations. However, Table 2 reports a QWK of 0.638 on TCGA-UCEC, which is lower than HistoSelect's 0.641. Furthermore, the gains from co-failure correction in Table 13 are only 0.002–0.008, and the paper does not provide sufficient confidence intervals or significance tests to establish the stability of these small improvements. This weakens the credibility of some experimental conclusions.
-
XPathAgent requires invoking multiple specialized tools across different tissue regions while performing reliability assessment, tool routing adjustments, and LLM reasoning. The paper does not sufficiently report end-to-end inference time, the number of tool calls, GPU memory usage, or computational cost comparisons with baselines. Therefore, its practical efficiency for large-scale WSI analysis remains unclear.
Questions for authors
-
Could the authors provide a correlation analysis between ICR scores and actual tool prediction accuracy, and explain how the framework handles tools that produce stable but diagnostically incorrect outputs?
-
How is the severity of image-quality degradation automatically estimated during inference? When the model is applied to unseen datasets, is it necessary to reconstruct the ICR calibration set?
-
How does XPathAgent support new organs and tasks not covered by the Diagnostic Atlas? Furthermore, why does the organ set defined in the methodology not fully match the organs included in the experiments?
-
Could the authors directly validate the correctness of ROI localization, concept measurements, and final evidence chains using pathologist annotations or independent reference standards?
-
Could the authors explain the inconsistency between the TCGA-UCEC results in Table 2 and the claim of achieving the best performance across all benchmarks, and provide statistical significance analyses for the improvements introduced by ICR?
-
Could the authors report the inference time, GPU memory consumption, and tool invocation costs of the complete system and compare them with the main baselines?
Response to Reviewer dFWw
We thank the reviewer for recognizing the transparency of our concept-to-evidence design and the structured reliability mechanism in ICR. We address each concern below.
[W1/Q1] ICR stability versus diagnostic correctness
We agree that stability does not guarantee correctness. ICR estimates measurement stability/informativeness, not intrinsic diagnostic accuracy. We therefore added ground-truth validation on 91 SICAPv2 biopsies from 63 patients using CONCH (Table 5(b); Appendix E.2). For cancer detection, where CONCH has strong clean performance (AUROC=0.92), lower ICR admissibility is significantly associated with more perturbation-induced errors (Kendall τ=0.67, Holm-adjusted p=0.002). Across 48 acquisition-condition comparisons, ICR ordering agrees with diagnostic-error ordering in 83% of comparisons versus 65% for nominal severity. At region level (1,584 pairs), lower admissibility is associated with higher error rates (Spearman ρ=-0.43, 95% CI: -0.52 to -0.33), and rejected measurements exhibit 25.9 percentage points more new errors than admitted measurements. For Gleason grading, CONCH is near chance on clean images (AUROC=0.56), so that analysis is inconclusive. Thus, ICR cannot rescue a diagnostically weak but stable tool; it identifies degradation susceptibility when the underlying specialist is competent.
[W2/Q2] Severity estimation and recalibration
We clarified inference-time severity estimation in Appendix C.2. Resolution uses slide micrometers-per-pixel metadata relative to the reference-set median; focus uses median log10 Laplacian variance mapped to an equivalent calibrated blur level; and stain deviation uses the RMS z-score of mean H/E/D concentrations mapped to an equivalent calibrated stain shift. Values are interpolated within the calibrated range and clipped/flagged outside it. Calibration is intentionally dataset-specific because acquisition distributions differ. For an unseen dataset, ICR requires a representative unlabeled reference set, but no diagnostic labels, specialist fine-tuning, or parameter updates.
[W3/Q3] Diagnostic Atlas and organ/task extension
The Diagnostic Atlas is constructed offline from established pathology literature/reporting conventions and subsequently clinician-reviewed (Appendix D.3); it is fixed during inference. New organs/tasks therefore require offline extension and clinical review of relevant concepts/mappings. “Plug-and-play” refers to integrating compatible specialist tools without fine-tuning them, not automatic ontology creation. We also corrected the organ inventory to include lymph node, lung, bladder, and endometrium (Lines 160–161), matching the experiments.
[W4/Q4] Intermediate evidence validity
We agree that final performance alone does not establish clinical correctness of every intermediate evidence chain. Direct ROI/concept-chain validation would require task-specific spatial annotations unavailable across our benchmarks. We therefore add complementary evidence: component ablations (Fig. 3), leave-one-tool-out analysis (Table 12), measurement-level ICR validation (Table 5(b), Appendix E.2), and complete qualitative traces including a failure case (Figs. 5–7). Appendix E.2 explicitly states that these analyses do not establish native ROI-localization accuracy or clinical correctness of full concept-to-region chains.
[W5/Q5] TCGA-UCEC inconsistency and statistical significance
We thank the reviewer for identifying the inconsistency. HistoSelect is slightly higher on TCGA-UCEC (0.641 vs. 0.638); we will correct the claim to state that XPathAgent achieves the strongest point performance on 12/13 benchmark-task combinations. We also added statistical analysis of ICR (Table 16): mean±SD over five runs, 95% bootstrap CIs, and paired permutation tests (10,000 permutations) with Holm correction. Full ICR improves over no reliability by +0.013 on SICAP (pHolm=0.028) and +0.013 on CAMELYON16 (pHolm=0.036), with gains of +0.005 on TCGA-RCC and +0.006 on TCGA-BRCA. Co-failure correction adds +0.006/+0.008/+0.002/+0.002 across the four benchmarks.
[W6/Q6] Computational efficiency
We added end-to-end profiling on TCGA-RCC using one NVIDIA A100 80GB GPU (Table 19). XPathAgent requires 17.0 min/WSI, 45 GB peak memory, and 34 LLM calls, achieving 0.939 BACC. PRISM, TITAN, and HistoSelect require 2.9/3.3/4.7 min, 12/15/19 GB, and achieve 0.615/0.663/0.875 BACC, respectively. XPathAgent provides relative BACC gains of 52.7%, 41.6%, and 7.3%, at additional compute cost. All specialist execution and route refinement are included in the end-to-end timing.