MEDAGENT-X: An Agentic Framework for Retrieval-Grounded and Auditable Medical Image Classification

Automated chest radiograph classification can support large-scale medical image analysis, but uncertainty, noisy report-derived supervision, and limited interpretability remain important challenges. We introduce MEDAGENT-X, a retrieval-grounded agentic framework for explainable and auditable multi-label chest X-ray classification. MEDAGENT-X maintains a strict separation between visual prediction, evidence-based intervention, and auditing. A fine-tuned RAD-DINO vision encoder serves as the primary predictor for 12 radiographic findings, while label-specific thresholds identify uncertain predictions within predefined gray zones. Only these cases are routed through image-to-report retrieval, where reports from visually similar training studies provide contextual evidence for a deterministic label-fusion policy. A constrained LLM reviews only proposed fusion changes, while a second read-only LLM independently assesses the evidence supporting finalized predictions without modifying diagnostic outputs. The framework is evaluated on a quality-controlled, patient-disjoint cohort of 5,299 CheXpert Plus studies. On the 906-study test set, MEDAGENT-X achieves a macro-F1 of 0.7246, improving the vision-only baseline by 0.0275, with particularly strong performance within the gray-zone subset (macro-F1 0.7613). Beyond predictive performance, MEDAGENT-X produces structured, case-level evidence records that preserve retrieved support, intervention history, and audit provenance. These results demonstrate a practical approach for incorporating agentic AI into medical imaging while preserving explicit decision boundaries between prediction, intervention, and evidence verification.