- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper points out that existing pathology VQA models may obtain correct answers by relying on textual priors rather than genuinely using visual evidence. To address this issue, the authors construct the PathoArgus dataset and propose the ESG counterfactual evaluation mechanism, which tests whether model predictions change with visual evidence by moving or removing the target case from candidate sets. Methodologically, PathoArgus selects question-relevant evidence from large numbers of WSI patches through question-conditioned hierarchical evidence allocation, tissue-aware patch selection, and source-aware answer generation. Overall, this paper introduces a valuable approach for evaluating visual evidence grounding in pathology and designs a corresponding evidence selection framework. However, the validity of the benchmark evidence, the rigor of the counterfactual experiments, and the method's advantages in real-world multi-slide pathology reasoning require further validation.
Critical strengths
-
The paper points out that conventional VQA accuracy cannot sufficiently demonstrate whether models truly utilize visual evidence. It proposes ESG and QExact to evaluate models through counterfactual changes in evidence. This design helps identify cases where models rely on textual priors or fixed answers.
-
The paper constructs a dataset covering six pathology capabilities and 15 TCGA projects. It also provides text-only controls, counterfactual evaluations, and multiple component ablations, offering a relatively systematic experimental foundation for analyzing the visual evidence utilization capabilities of pathology models.
Critical weaknesses
-
ESG constructs counterfactual samples by moving or removing target cases, but approximately half of the evaluations use cross-organ donor cases. In these settings, models may identify the target set simply by recognizing organ types rather than specific pathological features. Furthermore, for same-organ donors, it remains unclear whether cases containing the same target pathological findings are strictly excluded. Therefore, the current ESG results may reflect both organ recognition and pathological evidence localization capabilities, making it difficult to distinguish between them.
-
The paper primarily generates questions and answers from diagnostic fields in pathology reports. However, not all information in these reports can be directly determined from the provided WSIs. Although the authors conduct data integrity checks and report expert performance on a subset of questions, systematic WSI-level evidence validation is still lacking. Furthermore, the paper does not directly evaluate the pathological relevance or completeness of the selected patch evidence. This limits the reliability of the benchmark as a resource for evaluating evidence-grounded pathology VQA.
-
The paper emphasizes extending from single WSIs to multi-slide cases, but the dataset contains 4,809 cases and 5,400 WSIs, averaging only approximately 1.12 slides per case. Therefore, the proportion of cases containing multiple WSIs needs further clarification. Moreover, the Cross-Slide Integration (CSI) results do not clearly establish whether answering these questions genuinely requires integrating information from different slides. Although the paper states that CSI uses multiple slides from the same case, providing multiple slides does not necessarily imply that cross-slide information integration is required. If a single slide already contains sufficient evidence, the task may essentially remain one of single-slide recognition or evidence retrieval.
-
The paper compares 24 baseline systems, but PathoArgus is jointly trained on a specifically constructed training set, whereas most existing models are evaluated primarily using their original pretrained capabilities. Although the paper provides an SFT baseline and component ablations, it still lacks comparisons with strong evidence selection methods retrained under the same data, backbone, and training budget, as well as simpler question-conditioned retrieval methods. Therefore, it is difficult to determine whether the performance improvements primarily come from the proposed hierarchical selection mechanism or from task-specific supervision and training adaptation.
-
The paper emphasizes that only 512 patches are selected, retaining approximately 1.51% of the available features. However, the method still requires LongNet to encode all patch features from candidate slides, while the 512-patch limit mainly applies to the visual inputs passed to the Reader. Therefore, the current analysis primarily reflects Reader-side token consumption rather than the computational efficiency of the entire system. The lack of comparisons in end-to-end inference time, GPU memory consumption, and feature processing costs leaves the practical deployment efficiency unclear.
-
There is an inconsistency in the overall accuracy reported for the PathoArgus–Lingshu configuration between Table 2 and Figure 5a. Table 2 reports an overall accuracy of 61.39%, whereas Figure 5a reports 62.05% for the corresponding WSI-input setting. Could the authors clarify the reason for this discrepancy?
Questions for authors
-
Could the authors report ESG performance separately for same-organ and cross-organ donor settings and explain how they ensure that donors do not contain the same target pathological findings?
-
How do the authors verify that report-derived answers are genuinely supported by WSI visual evidence? Could they provide expert annotations or patch relevance evaluations?
-
How many cases contain multiple WSIs, and how many questions genuinely require cross-slide reasoning? Could the authors report performance specifically on these tasks?
-
Could the authors compare PathoArgus with stronger evidence selection methods under matched training data, backbone models, and training budgets?
-
Could the authors report end-to-end inference time, GPU memory usage, and computational costs, including complete WSI feature encoding and evidence selection?