acceptodds
Under review as a conference paper at ICLR 2027

Towards A Verifiable Medical Agent Powered by Specialist Tools

Abstract

Medical reasoning requires selective image inspection, specialized computation, and reconciliation of evidence across sources. We introduce MedVESTA, a medical agent that connects general-purpose multimodal large language models (MLLMs) to specialist tools. Predictions, attention maps, and selected patches guide evidence acquisition; for classification and reporting, a verifier flags evidence-linked claims for re-inspection and revision. The toolbox combines pathology with available radiological and molecular evidence and diagnostic-knowledge retrieval. We also introduce PathScore, which evaluates report content through 15 clinical fields. On six classification tasks, Qwen3.5-27B with MedVESTA reaches 69.66% balanced accuracy with self-verification and 70.25% with a separate verifier, compared with 68.90% for its MIL specialist. VQA and report generation improve over the evaluated single-shot configurations of the same backbones. These results support specialist-guided evidence acquisition within an inspectable reasoning workflow.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
medical agents, multimodal large language models, specialist tools, evidence verification, computational pathology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yTbGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes MedVESTA, a medical agent framework that combines a multimodal large language model (MLLM) and pathology specialist tools. This method uses the prediction heatmap from multiple instance learning (MIL) and find high weight patches to acquire evidence. Besides, the method supports local inspection and knowledge retrieval. In experiments on classification and report generation tasks, the system can link claims to evidence and use a verifier to flag content for re-inspection and revision. What's more, the authors proposed PathScore, which works by comparing 15 clinical fields between the generated report and the reference report.

Critical strengths
  1. The writing is moderate and relatively easy to understand.
  2. The paper is well-organized.
  3. The authors have conducted extensive experiments.
Critical weaknesses
  1. The proposed MedVISTA has several different components that share similar ideas proposed by existing works. For example, PathPocket [1] combines multimodal pathology analysis and traceable knowledge retrieval. This idea is similar to the proposed combination of pathology specialist tools and diagnostic knowledge retrieval. PathAgent [2] conducts visual reasoning via iterative region selection, adjusting observation scales and aggregating visual evidence. It is quite similar to the proposed method using the heatmap of the specialist models to select the region, and using region-viewing tools to supplement visual evidence. QCAgent [3] uses report inspection, region retrieval, and report revision as a loop to handle tasks. This idea is similar to the proposed verifier to re-check the claims and use the executor to inspect evidence and make modifications.

  2. I also find that there are some existing works that proposed scoring method PathScore that are similar to the proposed PathScore. For example, CheXbert [4], also mentioned in the related work section in this manuscript, uses a structured label extraction method for radiology. HARE [5] uses entity and relation extraction methods to help to evaluate generated pathology reports. QCAgent [3] uses field recall to evaluate the coverage of key clinical fields. PathReportEval [6] proposed Clinical Report Quality Score (CRQS) to make distinctions among clinical fact coverage, key information recall, reference-unsupported content, and clinical discordance, which I believe is a metric more carefully designed than the proposed PathScore. Therefore, it is unclear why the authors wanted to propose this new metric and what the advantage of this metric is compared with all existing works.

Based on all the above information, I think this work is more about a combination of different existing modules rather than a new and novel method like what the authors wrote in the manuscript.

References

[1] Xu, Z., Zhang, Z., Cai, Z., et al. A Multimodal Agentic Pathology Co-pilot via Evidence Grounded Reasoning. arXiv preprint arXiv:2606.08093, 2026.

[2] Chen, J., Cai, L., Wang, Z., et al. PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning. arXiv preprint arXiv:2511.17052, 2025.

[3] Wang, R., Ba, W., Zhou, Y., et al. QCAgent: An Agentic Framework for Quality-Controllable Pathology Report Generation from Whole Slide Image. arXiv preprint arXiv:2603.01647, 2026.

[4] Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., and Lungren, M. Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. In Proceedings of EMNLP, pp. 1500–1519, 2020.

[5] Kim, Y., Ong, M. W. S., Shavick, A., Wu, H., and Levine, A. P. HARE: An Entity and Relation Centric Evaluation Framework for Histopathology Reports. arXiv preprint arXiv:2509.16326, 2025.

[6] Singh, S., Majumder, S., Knudsen, B., Saltz, J., and Prasanna, P. PathReportEval: A Systematic Benchmark for Pathology Report Generation. arXiv preprint arXiv:2607.18448, 2026.

Questions for authors

None

3 comments

Thank you for the pointers to related work. The revised PDF discusses all six works in Section 2 and adds a point-by-point comparison in the new Appendix G.6 (Tables 21–22).

Two of the six works are already part of the submission: CheXbert is cited, and QCAgent is cited and evaluated as a baseline in Table 3 (PS-R 0.3161, against 0.4435–0.5539 for MedVESTA). The other four are now cited and discussed as well. We compare with all six below.

For the agent systems, the similarity holds at the level of topic (an agent that inspects regions and retrieves knowledge), not of mechanism. MedVESTA rests on three properties, and none of the cited systems combines them. (i) Predictions of task-trained specialists (labels, probabilities, attention, grading readouts) are registered as evidence that the executor inspects and may overturn. (ii) Findings are recorded as claims that cite registered case evidence by ID. (iii) The verifier only flags claims for re-inspection and never writes the answer; the executor re-inspects the slide and writes the final answer itself.

Table 21 sets the systems side by side. PathPocket does not use specialist predictions as evidence: slide-encoder attention pre-selects regions once. Its citations point to retrieved literature rather than to case evidence, it has no verifier, and it is evaluated on multiple-choice diagnostic questions and a reader study. PathAgent selects regions by text–patch relevance, does not cite case evidence by ID, relies on self-reflection on evidence sufficiency, and is evaluated on VQA. QCAgent starts from a PRISM draft and does not cite case evidence by ID; its QC model audits the report, retrieves patches and rewrites the report itself, and it is evaluated on gastric reports. In MedVESTA, specialist predictions are evidence, claims cite case evidence by ID, the verifier flags claims only while the executor re-inspects and writes, and the evaluation covers classification, VQA and reports.

As for PathScore and existing metrics, CheXbert is a chest X-ray report labeler; we cite it for the principle of structured extraction. HARE scores entity and relation overlap, and its paper states that it "does not yet explicitly handle negation or uncertainty". PathScore compares field values, including explicit negatives, and records INDETERMINATE statements. QCAgent's field recall checks whether a field is mentioned, not whether its value is right. On the ten constructed contrasts of Tables 13–14, its released implementation gives the faithful and the altered statement the same score in all ten; PathScore separates all ten.

CRQS (PathReportEval) is concurrent work and the closest in spirit. It uses dataset-specific schemas, and its released code extracts fields with hand-written rules. PathScore uses one fixed field inventory across cohorts, an explicit INDETERMINATE outcome and a field-balanced Macro-F1, and its extraction is validated: 95.9% agreement with human-verified values (Appendix G.2). PathScore has its own penalty for reference-absent content, the optional F1_halluc (defined and reported in Appendix G.3); it is not a primary metric because reference reports are not complete inventories: negative findings, or findings unrelated to the diagnosis, are often simply left unwritten, which is not an error of the generated report.

A new metric is needed because Tables 13–16 show that BLEU, BERTScore and embedding similarity rank clinically wrong statements above faithful ones. At submission, no validated pathology metric compared field values on a fixed clinical inventory. MedVESTA also ranks first under QCAgent's own metric: under its field recall, Claude Opus 5 + MedVESTA and Qwen3.5-27B + MedVESTA rank first and second among the 14 systems of Table 3 (0.92 and 0.88; best single-shot model 0.80; Appendix G.6).

On the comment that this is "a combination of different existing modules", the paper does not claim a new model. Its stated contributions are the specialist-guided workflow, evidence-linked claims with verification-guided revision, and PathScore. What is new is the protocol in (i)–(iii) and the controlled evidence for it: specialist evidence improves every evaluated backbone in VQA, the full workflow improves every evaluated backbone in reporting, and in classification a general MLLM exceeds its MIL specialist in balanced accuracy only with it (Table 1).

Thanks for the authors for their response. The new results have addressed parts of my concerns. Currently, I still have some concerns for the following points:

  1. The practical value of PathScore comparing with CRQS. The clinical fact coverage (CFC) of CRQS and the reference-field recall of PathScore shared the similar per-report recall calculation while the field inventory and aggregation are different. Besides, according to Appendix G.2, NA and INDETERMINATE, though recorded separately, have the same effect to the final report evaluation metrics. Could authors leverage real generated reports to illustrate what specific evaluation errors in CRQS are these designs used to correct?

  2. Expert validation of report-quality assessment. Do authors have pathologist ratings or pairwise quality judgments to validate the report scores and rankings of PathScore, and comparing it with clinical evaluation metrics?

Thank you for the follow-up and for acknowledging that the additional results addressed part of your concerns. We agree that the remaining questions concern the practical benefit and validation of PathScore, and we clarify the scope of our evidence below.

  1. Practical value of PathScore compared with CRQS

We agree that CRQS’s clinical fact coverage and PathScore’s reference-field recall share the same underlying principle: measuring recovery of reference clinical facts. CRQS also explicitly evaluates clinical discordance and reference-unsupported content. We therefore do not claim that structured factual evaluation or contradiction detection is unique to PathScore.

PathScore’s intended contribution is a consistent evaluation protocol for our cross-cohort experiments: a fixed inventory of 15 clinical fields, explicit field-value extraction, and field-balanced Macro-F1 alongside reference-field recall. The fixed inventory makes the evaluated dimensions consistent across cohorts, while Macro-F1 gives each evaluated field equal weight. These choices address comparability and aggregation; by themselves, they do not demonstrate that CRQS makes incorrect judgments on individual reports.

Your observation about NA and INDETERMINATE is also correct. Although recorded separately, they have the same effect on the reported aggregate metrics under the current scoring rules. Their separate representation supports inspection of the extracted outputs, but we should not present this distinction as an independently demonstrated advantage in report scoring.

Regarding real generated reports, the evidence presented so far does not establish specific CRQS evaluation errors that PathScore corrects. In particular, the constructed contrasts in Tables 13–14 evaluate QCAgent’s field recall, and should not be interpreted as evidence against CRQS. Demonstrating an advantage over CRQS would require scoring the same real reports with both methods and adjudicating their disagreements. We will clarify this evidential boundary and position PathScore as a complementary, consistently specified evaluation protocol, without claiming established superiority over CRQS.

  1. Expert validation of report-quality assessment

Our reference reports were written by pathologists. PathScore therefore measures agreement with expert-authored clinical content, providing expert grounding for the target findings against which generated reports are evaluated.

Separately, Appendix G.2 reports 95.9% agreement between automatically extracted field values and human-verified values. This supports the reliability of the extraction step. However, we agree that extraction agreement and expert-authored references do not, on their own, validate the aggregate scoring function or its ranking of generated reports.

We do not currently report an independent study in which pathologists directly rate generated reports or provide pairwise quality judgments, followed by a comparison of their judgments with PathScore and competing clinical metrics. Consequently, our results support PathScore as a measure of structured agreement with pathologist-written references; they do not establish that it best reproduces pathologists’ overall quality preferences.

A blinded expert assessment of the same generated reports, comparing metric–expert agreement for PathScore, CRQS, and other clinical metrics, would provide the additional validation for the next version, but it is not main scope of this study.

Reviewer_VjdvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The core contribution of this paper is not a new foundational model, but rather MedVESTA's workflow of "specialized tool guidance+evidence registration+claim level verification+targeted retesting", as well as the PathScore clinical field level report evaluation metric. It provides a reusable template for designing the trustworthiness of medical agents, but its performance gains are limited, causal attribution and clinical validation are insufficient, making it more suitable for feasibility studies rather than clinical effectiveness evidence.

Critical strengths
  1. Both tool output and observation have persistent evidence IDs, and conclusions are bound to evidence in the form of claims; The validator only marks suspicious claims, and the executor returns to the original slice and tool output for retesting, rather than directly providing the answer by the validator.

  2. Transitioning from text similarity to comparing the values of 15 clinical fields, it is possible to distinguish between correct, incorrect, omitted, and uncertain values; Controlled experiments have shown that BLEU/BERTScore ranks clinical error reports higher, while PathScore maintains the correct ranking.

Critical weaknesses
  1. Limited innovation: Basic MLLM, UNI2-h, ABMIL, and validators are all existing components; The novelty of the paper mainly lies in system combination, evidence protocol, and evaluation indicators, rather than new models or algorithms.
  2. Insufficient causal attribution and statistical rigor: The report generation compares the complete workflow and cannot isolate the contribution of the validation module; The gradual improvement in classification is not independent causal melting; VQA only runs once per configuration without confidence intervals or significance tests, and VQA itself does not use validators, resulting in inconsistent task protocols.
  3. Insufficient data quality assurance: The newly organized Full-653 and TCGA-209 were not reported to pathology experts for case by case review, dual labeling, or expert arbitration; The specialized model tags are automatically derived from keyword rules, and the paper clearly states that there is no additional manual judgment; The VQA issue also largely inherits existing benchmarks and has not been re audited.
Questions for authors

Please refer to 'Weaknesses'.

1 comment

Thank you for the precise assessment. The revised PDF adds Appendix G (G.1–G.7, Tables 18–23) with the analyses below.

W1. Novelty. We agree with the reviewer's summary of what the contribution is; we disagree that it is limited. The contribution is a methodology for medical AI. It joins the strengths of general MLLMs (a natural-language interface for clinicians and the ability to call other modules) with those of specialist models (strong performance on specific tasks), and it uses the specialists' outputs to guide where the MLLM looks. This matters because general MLLMs are far behind specialists in pathology: they trail ABMIL by 8–31 points BA [1, 2], and GPT-4o stays at chance level on slide-level tasks [3]. The results come from this method, not from the components.

The same specialist, used differently. Handing a general MLLM the specialist's prediction, heatmap and patches in a single shot does not reach the specialist: Qwen3.5-27B obtains 57.28% BA and the best of eight MLLMs 68.40%, against 68.90% for ABMIL (Table 1, S3). With MedVESTA the same backbone reaches 69.66% and exceeds its specialist.

Three mechanisms produce this. (1) Specialist outputs are search cues and evidence that the executor inspects and may overturn, not answers (Section 3.2). (2) Every tool output is registered under an evidence ID, and conclusions are claims bound to those IDs (Eq. 1). (3) The verifier only flags claims, so the executor must return to the slide (Eq. 4). No related pathology agent combines them (Appendix G.6, Table 21).

MedVESTA deliberately introduces no new model. Its contribution is the protocol that the review lists as a strength.

PathScore is a second contribution, now validated against human judgment (Appendix G.2: 95.9% agreement on 591 audited cells). Under QCAgent's field recall, an independent metric, MedVESTA still ranks first and second among the 14 systems of Table 3 (Appendix G.6).

W2. Attribution and statistical rigor. For reports, verification is isolated in Appendix G.4 (Table 19): the first pass and the revision use the same executor, initial evidence and tool budget. For Claude Opus 5 + MedVESTA, revision raises the share of cited evidence IDs that resolve from 94.7% to 99.3% on TCGA-209 (EAGLE-48: 98.9% to 99.5%; TCGA-242: 95.3% to 99.3%) and lowers the share of claims without type-compatible evidence from 1.3% to 0.6% (1.4% to 0.3%, 0.7% to 0.1%).

For classification, the initial and verified answers in Table 1 use the same executor and inputs, so their difference also isolates verification: +2.86 points BA with self-verification and +3.45 with the separate verifier (Appendix G.5). Figure 3b isolates the heatmap likewise: with the same random patches, only the heatmap changes (true, absent or randomized).

For VQA, both arms answer the same questions, so we report Holm-corrected exact McNemar tests (Table 20): all 15 same-backbone gains are positive and 10 are significant. VQA has no verifier by design: its answer is a single option letter and is often a direct conclusion, so there is little evidence to verify and verification would be high-variance. VQA therefore isolates the contribution of specialist evidence.

W3. Data quality. Full-653 and TCGA-209 carry expert labels at source: TCGA diagnoses and grades were assigned by pathologists, and we take them without relabelling. Subtype labels are TCGA project and diagnosis codes; grade labels are the grades stated in the TCGA pathology reports (Appendix A.2). TCGA-209 references are the TITAN slide reports, derived from the same pathologist-written reports. EAGLE-48 and TCGA-242 use native human-written reports, and the human audit finds 98.0% agreement for reference-side extraction.

The specialist training labels come from REG2026 annotations with a fixed vocabulary, obtained by direct matching with no LLM judgment. We manually inspected sampled cases and found no errors (Appendix B.1). They are training targets only: the classification specialist is evaluated on full-653, whose TCGA labels are independent of REG2026.

We use the public, peer-reviewed VQA benchmarks unmodified, for comparability with prior work. Our conclusions rest on paired same-backbone comparisons on identical questions, where question noise affects both arms equally, and on transfer to BCNB and CPTAC.

We agree that this is a methodological contribution, not clinical-effectiveness evidence; the paper makes no clinical claim (Ethics Statement).

[1] Sun et al., CPath-Omni, CVPR 2025. [2] Sun et al., CPathAgent, NeurIPS 2025. [3] Neidlinger et al., Nat. Commun. 17, 5740 (2026).

Reviewer_K1LJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MedVESTA connects multimodal language models to pathology specialists, selective image inspection, and evidence-linked revision. It evaluates classification, VQA, and report generation, and introduces PathScore for agreement on clinical report fields. Same-backbone results generally improve, including on additional report and VQA cohorts.

Critical strengths
  • The study includes multiple backbones, specialist-information controls, and external cohorts, giving a useful picture of where specialist guidance helps (Tables 1-4).
  • The explicit evidence registry and careful disclosure of development use, baseline overlap, and remaining report errors improve transparency and make the workflow inspectable.
Critical weaknesses
  • PathScore is central to the report conclusions, but the production field extractor lacks a reported human-validation study. The constructed examples test scoring with supplied values, not extraction accuracy. Fields absent from references are ignored (Appendix E), so unsupported additions can escape both reported metrics. This limits conclusions about report reliability.
  • The evidence does not isolate the benefit of verification from extra evidence and inference. VQA has no verifier, and report comparisons change the complete workflow. Classification exceeds the MIL specialist by only 0.76-1.35 BA points without paired uncertainty. Matched-input revision controls and claim-support validation are needed to substantiate the central verification contribution.
Questions for authors
  1. Which model and prompts implement PathScore extraction, and is any existing human agreement or extraction-error estimate available?
  2. Did report baselines receive the same auxiliary evidence, and do existing logs separate first-pass quality from the effect of revision?
1 comment

We thank the reviewer for the careful reading. The revised PDF adds a new Appendix G (G.1–G.7, Tables 18–23) with the analyses below.

W1(a). Human validation of the extractor (Appendix G.2, Table 18). For 20 randomly sampled TCGA-209 patients (5 per organ), we audited all 15 fields of the reference report and of the Claude Opus 5 + MedVESTA report. Values were pre-annotated by an LLM and then verified by an author. Extracted values agree with the human-verified values in 95.9% of cells, and using the human-verified values changes PS-R on these slides only from 0.529 to 0.536. We also ran a second extractor from another model family (Claude Opus 5) on all 209 reports. It assigns the same outcome at 98.0% of all 3,135 field positions (Cohen's κ = 0.970) and at 97.0% of the 1,239 reference-supported positions (κ = 0.945).

W1(b). Fields absent from the reference. A reference report is not a complete inventory of findings: "not written" is not "not present". Reports are concise (TCGA-209 references average 46 words), and peripheral or negative findings are often omitted. Content beyond the reference therefore cannot by itself be called hallucination; penalising it would punish reports that are correct and more complete. PathScore is a protocol for extracting, comparing and scoring clinical fields, and the summary metric can be chosen for the evaluation purpose: PS-R for coverage of reference content, PS-F1 for mismatch-sensitive agreement, and Macro-F1 for field-balanced agreement. To count statements on reference-absent fields as false positives, F1_halluc can be used (Appendix G.3). On TCGA-209, MedVESTA reaches 0.42 with Qwen3.5-27B, 0.45 with Occamy-1.0 and 0.49 with Claude Opus 5.

W2. Verification in reports (Appendix G.4, Table 19). As a matched-input control, the first pass and the revision use the same executor, initial evidence and tool budget. For Claude Opus 5 + MedVESTA, revision improves two claim-support measures. First, the share of cited evidence IDs that resolve rises from 94.7% to 99.3% on TCGA-209 (98.9% to 99.5% on EAGLE-48, 95.3% to 99.3% on TCGA-242). For example, a claim "lymphovascular invasion: not identified [zoom:004]" is unresolved when only zoom:001–003 were registered, because the cited record does not exist. Second, the share of claims without type-compatible evidence falls from 1.3% to 0.6% (1.4% to 0.3%, 0.7% to 0.1%). For example, in "Gleason 4+4, Grade Group 4 [rag:002]", rag:002 exists, but it is a retrieved textbook passage on pattern 4, not an observation of this slide, so it cannot support the grade; [zoom:002], a view of this slide, can. This is a direct claim-support validation, judged over all claims by an LLM (Claude Opus 5.5 in Claude Code); an author checked 73 sampled type judgments and agreed with all.

W2. Verification in classification. An MIL specialist is a very strong reference. General MLLMs trail ABMIL by 8–31 points BA [1, 2], and GPT-4o stays at chance level on slide-level tasks [3]; in [1, 2], pathology-trained models 1.3–2.9 points above ABMIL are presented as matching or exceeding it. Table 1 agrees: ABMIL (68.90%) is above every pathology-specific model and every single-shot MLLM, even GPT-5.5 given its heatmap, patches and prediction (68.40%). Exceeding the specialist without fine-tuning the executor is therefore a strong result. Table 1 also gives a matched-input control (Appendix G.5): with the same executor and inputs, BA rises from 66.80% to 69.66% with self-verification and to 70.25% with the separate verifier.

Q1. Generated reports are extracted by Qwen3.5-27B (temperature 0, thinking disabled) with one fixed schema prompt, now given verbatim in Appendix G.1. Extracted values agree with human-verified values in 95.9% of audited cells, and using the human-verified values changes PS-R only from 0.529 to 0.536.

Q2. Yes. The same-backbone baselines received the same auxiliary evidence as MedVESTA: the slide overview, the same radiological and molecular evidence, the same example reports, and also the MIL heatmap with top-20 patches and the grading-head readouts (Table 23, Appendix G.7). They do not receive the MIL predicted labels, diagnostic criteria, or MedVESTA's inspection and retrieval tools: which regions are inspected, which measurements are requested and which passages are retrieved is decided by the executor in each run, so the evidence a run accumulates has no fixed counterpart that could be handed to a single-shot model. The first-pass vs. revised comparison is described under W2; Table 19 gives more details.

[1] Sun et al., CPath-Omni, CVPR 2025. [2] Sun et al., CPathAgent, NeurIPS 2025. [3] Neidlinger et al., Nature Communications 17, 5740 (2026). All three are now cited in Section 5.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.