- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes MedVESTA, a medical agent framework that combines a multimodal large language model (MLLM) and pathology specialist tools. This method uses the prediction heatmap from multiple instance learning (MIL) and find high weight patches to acquire evidence. Besides, the method supports local inspection and knowledge retrieval. In experiments on classification and report generation tasks, the system can link claims to evidence and use a verifier to flag content for re-inspection and revision. What's more, the authors proposed PathScore, which works by comparing 15 clinical fields between the generated report and the reference report.
Critical strengths
- The writing is moderate and relatively easy to understand.
- The paper is well-organized.
- The authors have conducted extensive experiments.
Critical weaknesses
-
The proposed MedVISTA has several different components that share similar ideas proposed by existing works. For example, PathPocket [1] combines multimodal pathology analysis and traceable knowledge retrieval. This idea is similar to the proposed combination of pathology specialist tools and diagnostic knowledge retrieval. PathAgent [2] conducts visual reasoning via iterative region selection, adjusting observation scales and aggregating visual evidence. It is quite similar to the proposed method using the heatmap of the specialist models to select the region, and using region-viewing tools to supplement visual evidence. QCAgent [3] uses report inspection, region retrieval, and report revision as a loop to handle tasks. This idea is similar to the proposed verifier to re-check the claims and use the executor to inspect evidence and make modifications.
-
I also find that there are some existing works that proposed scoring method PathScore that are similar to the proposed PathScore. For example, CheXbert [4], also mentioned in the related work section in this manuscript, uses a structured label extraction method for radiology. HARE [5] uses entity and relation extraction methods to help to evaluate generated pathology reports. QCAgent [3] uses field recall to evaluate the coverage of key clinical fields. PathReportEval [6] proposed Clinical Report Quality Score (CRQS) to make distinctions among clinical fact coverage, key information recall, reference-unsupported content, and clinical discordance, which I believe is a metric more carefully designed than the proposed PathScore. Therefore, it is unclear why the authors wanted to propose this new metric and what the advantage of this metric is compared with all existing works.
Based on all the above information, I think this work is more about a combination of different existing modules rather than a new and novel method like what the authors wrote in the manuscript.
References
[1] Xu, Z., Zhang, Z., Cai, Z., et al. A Multimodal Agentic Pathology Co-pilot via Evidence Grounded Reasoning. arXiv preprint arXiv:2606.08093, 2026.
[2] Chen, J., Cai, L., Wang, Z., et al. PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning. arXiv preprint arXiv:2511.17052, 2025.
[3] Wang, R., Ba, W., Zhou, Y., et al. QCAgent: An Agentic Framework for Quality-Controllable Pathology Report Generation from Whole Slide Image. arXiv preprint arXiv:2603.01647, 2026.
[4] Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., and Lungren, M. Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. In Proceedings of EMNLP, pp. 1500–1519, 2020.
[5] Kim, Y., Ong, M. W. S., Shavick, A., Wu, H., and Levine, A. P. HARE: An Entity and Relation Centric Evaluation Framework for Histopathology Reports. arXiv preprint arXiv:2509.16326, 2025.
[6] Singh, S., Majumder, S., Knudsen, B., Saltz, J., and Prasanna, P. PathReportEval: A Systematic Benchmark for Pathology Report Generation. arXiv preprint arXiv:2607.18448, 2026.
Questions for authors
None
3 comments
Thanks for the authors for their response. The new results have addressed parts of my concerns. Currently, I still have some concerns for the following points:
-
The practical value of PathScore comparing with CRQS. The clinical fact coverage (CFC) of CRQS and the reference-field recall of PathScore shared the similar per-report recall calculation while the field inventory and aggregation are different. Besides, according to Appendix G.2, NA and INDETERMINATE, though recorded separately, have the same effect to the final report evaluation metrics. Could authors leverage real generated reports to illustrate what specific evaluation errors in CRQS are these designs used to correct?
-
Expert validation of report-quality assessment. Do authors have pathologist ratings or pairwise quality judgments to validate the report scores and rankings of PathScore, and comparing it with clinical evaluation metrics?
Thank you for the follow-up and for acknowledging that the additional results addressed part of your concerns. We agree that the remaining questions concern the practical benefit and validation of PathScore, and we clarify the scope of our evidence below.
- Practical value of PathScore compared with CRQS
We agree that CRQS’s clinical fact coverage and PathScore’s reference-field recall share the same underlying principle: measuring recovery of reference clinical facts. CRQS also explicitly evaluates clinical discordance and reference-unsupported content. We therefore do not claim that structured factual evaluation or contradiction detection is unique to PathScore.
PathScore’s intended contribution is a consistent evaluation protocol for our cross-cohort experiments: a fixed inventory of 15 clinical fields, explicit field-value extraction, and field-balanced Macro-F1 alongside reference-field recall. The fixed inventory makes the evaluated dimensions consistent across cohorts, while Macro-F1 gives each evaluated field equal weight. These choices address comparability and aggregation; by themselves, they do not demonstrate that CRQS makes incorrect judgments on individual reports.
Your observation about NA and INDETERMINATE is also correct. Although recorded separately, they have the same effect on the reported aggregate metrics under the current scoring rules. Their separate representation supports inspection of the extracted outputs, but we should not present this distinction as an independently demonstrated advantage in report scoring.
Regarding real generated reports, the evidence presented so far does not establish specific CRQS evaluation errors that PathScore corrects. In particular, the constructed contrasts in Tables 13–14 evaluate QCAgent’s field recall, and should not be interpreted as evidence against CRQS. Demonstrating an advantage over CRQS would require scoring the same real reports with both methods and adjudicating their disagreements. We will clarify this evidential boundary and position PathScore as a complementary, consistently specified evaluation protocol, without claiming established superiority over CRQS.
- Expert validation of report-quality assessment
Our reference reports were written by pathologists. PathScore therefore measures agreement with expert-authored clinical content, providing expert grounding for the target findings against which generated reports are evaluated.
Separately, Appendix G.2 reports 95.9% agreement between automatically extracted field values and human-verified values. This supports the reliability of the extraction step. However, we agree that extraction agreement and expert-authored references do not, on their own, validate the aggregate scoring function or its ranking of generated reports.
We do not currently report an independent study in which pathologists directly rate generated reports or provide pairwise quality judgments, followed by a comparison of their judgments with PathScore and competing clinical metrics. Consequently, our results support PathScore as a measure of structured agreement with pathologist-written references; they do not establish that it best reproduces pathologists’ overall quality preferences.
A blinded expert assessment of the same generated reports, comparing metric–expert agreement for PathScore, CRQS, and other clinical metrics, would provide the additional validation for the next version, but it is not main scope of this study.
Thank you for the pointers to related work. The revised PDF discusses all six works in Section 2 and adds a point-by-point comparison in the new Appendix G.6 (Tables 21–22).
Two of the six works are already part of the submission: CheXbert is cited, and QCAgent is cited and evaluated as a baseline in Table 3 (PS-R 0.3161, against 0.4435–0.5539 for MedVESTA). The other four are now cited and discussed as well. We compare with all six below.
For the agent systems, the similarity holds at the level of topic (an agent that inspects regions and retrieves knowledge), not of mechanism. MedVESTA rests on three properties, and none of the cited systems combines them. (i) Predictions of task-trained specialists (labels, probabilities, attention, grading readouts) are registered as evidence that the executor inspects and may overturn. (ii) Findings are recorded as claims that cite registered case evidence by ID. (iii) The verifier only flags claims for re-inspection and never writes the answer; the executor re-inspects the slide and writes the final answer itself.
Table 21 sets the systems side by side. PathPocket does not use specialist predictions as evidence: slide-encoder attention pre-selects regions once. Its citations point to retrieved literature rather than to case evidence, it has no verifier, and it is evaluated on multiple-choice diagnostic questions and a reader study. PathAgent selects regions by text–patch relevance, does not cite case evidence by ID, relies on self-reflection on evidence sufficiency, and is evaluated on VQA. QCAgent starts from a PRISM draft and does not cite case evidence by ID; its QC model audits the report, retrieves patches and rewrites the report itself, and it is evaluated on gastric reports. In MedVESTA, specialist predictions are evidence, claims cite case evidence by ID, the verifier flags claims only while the executor re-inspects and writes, and the evaluation covers classification, VQA and reports.
As for PathScore and existing metrics, CheXbert is a chest X-ray report labeler; we cite it for the principle of structured extraction. HARE scores entity and relation overlap, and its paper states that it "does not yet explicitly handle negation or uncertainty". PathScore compares field values, including explicit negatives, and records INDETERMINATE statements. QCAgent's field recall checks whether a field is mentioned, not whether its value is right. On the ten constructed contrasts of Tables 13–14, its released implementation gives the faithful and the altered statement the same score in all ten; PathScore separates all ten.
CRQS (PathReportEval) is concurrent work and the closest in spirit. It uses dataset-specific schemas, and its released code extracts fields with hand-written rules. PathScore uses one fixed field inventory across cohorts, an explicit INDETERMINATE outcome and a field-balanced Macro-F1, and its extraction is validated: 95.9% agreement with human-verified values (Appendix G.2). PathScore has its own penalty for reference-absent content, the optional F1_halluc (defined and reported in Appendix G.3); it is not a primary metric because reference reports are not complete inventories: negative findings, or findings unrelated to the diagnosis, are often simply left unwritten, which is not an error of the generated report.
A new metric is needed because Tables 13–16 show that BLEU, BERTScore and embedding similarity rank clinically wrong statements above faithful ones. At submission, no validated pathology metric compared field values on a fixed clinical inventory. MedVESTA also ranks first under QCAgent's own metric: under its field recall, Claude Opus 5 + MedVESTA and Qwen3.5-27B + MedVESTA rank first and second among the 14 systems of Table 3 (0.92 and 0.88; best single-shot model 0.80; Appendix G.6).
On the comment that this is "a combination of different existing modules", the paper does not claim a new model. Its stated contributions are the specialist-guided workflow, evidence-linked claims with verification-guided revision, and PathScore. What is new is the protocol in (i)–(iii) and the controlled evidence for it: specialist evidence improves every evaluated backbone in VQA, the full workflow improves every evaluated backbone in reporting, and in classification a general MLLM exceeds its MIL specialist in balanced accuracy only with it (Table 1).