Towards A Verifiable Medical Agent Powered by Specialist Tools

Medical reasoning requires selective image inspection, specialized computation, and reconciliation of evidence across sources. We introduce MedVESTA, a medical agent that connects general-purpose multimodal large language models (MLLMs) to specialist tools. Predictions, attention maps, and selected patches guide evidence acquisition; for classification and reporting, a verifier flags evidence-linked claims for re-inspection and revision. The toolbox combines pathology with available radiological and molecular evidence and diagnostic-knowledge retrieval. We also introduce PathScore, which evaluates report content through 15 clinical fields. On six classification tasks, Qwen3.5-27B with MedVESTA reaches 69.66% balanced accuracy with self-verification and 70.25% with a separate verifier, compared with 68.90% for its MIL specialist. VQA and report generation improve over the evaluated single-shot configurations of the same backbones. These results support specialist-guided evidence acquisition within an inspectable reasoning workflow.