acceptodds
Under review as a conference paper at ICLR 2027

MEDAGENT-X: An Agentic Framework for Retrieval-Grounded and Auditable Medical Image Classification

Abstract

Automated chest radiograph classification can support large-scale medical image analysis, but uncertainty, noisy report-derived supervision, and limited interpretability remain important challenges. We introduce MEDAGENT-X, a retrieval-grounded agentic framework for explainable and auditable multi-label chest X-ray classification. MEDAGENT-X maintains a strict separation between visual prediction, evidence-based intervention, and auditing. A fine-tuned RAD-DINO vision encoder serves as the primary predictor for 12 radiographic findings, while label-specific thresholds identify uncertain predictions within predefined gray zones. Only these cases are routed through image-to-report retrieval, where reports from visually similar training studies provide contextual evidence for a deterministic label-fusion policy. A constrained LLM reviews only proposed fusion changes, while a second read-only LLM independently assesses the evidence supporting finalized predictions without modifying diagnostic outputs. The framework is evaluated on a quality-controlled, patient-disjoint cohort of 5,299 CheXpert Plus studies. On the 906-study test set, MEDAGENT-X achieves a macro-F1 of 0.7246, improving the vision-only baseline by 0.0275, with particularly strong performance within the gray-zone subset (macro-F1 0.7613). Beyond predictive performance, MEDAGENT-X produces structured, case-level evidence records that preserve retrieved support, intervention history, and audit provenance. These results demonstrate a practical approach for incorporating agentic AI into medical imaging while preserving explicit decision boundaries between prediction, intervention, and evidence verification.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Agentic AI, Medical Image Analysis, Chest X-Ray Classification, Retrieval-Augmented Generation, Explainable AI, Auditable AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sN2kOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies study-level chest X-ray classification across 12 findings. A fine-tuned RAD-DINO classifier produces the initial predictions. For predictions near label-specific thresholds, the system retrieves reports from similar training studies and applies deterministic rules to revise the labels. An LLM reviews proposed changes, and a second invocation produces read-only evidence assessments. Experiments use a selected CheXpert Plus cohort and an additional five-condition expert-labeled CheXpert cohort. The reported classification improvement mainly comes from deterministic retrieval fusion.

Critical strengths

1、Restricting label changes to predefined cases and separating prediction from read-only verification are sensible design choices. The stored intervention traces make the pipeline easier to inspect.

2、Deterministic retrieval fusion improves macro-F1 over the same vision-only model on both evaluation cohorts. The patient-disjoint split and training-only retrieval index are also appropriate safeguards.

Critical weaknesses

1、The methodological novelty is not convincingly established. The framework combines an existing visual encoder, similarity retrieval, hand-designed fusion rules, and constrained LLM calls. The claimed novelty mainly lies in separating prediction, intervention, and auditing. However, the paper does not position this design against relevant medical-agent frameworks such as MedRAX and MedAgent-Pro, which already integrate medical tools or retrieval into structured reasoning workflows. I therefore find it difficult to identify the substantive methodological advance beyond this particular combination of components.

2、The experiments do not establish the added value of the proposed framework. Table 3 shows that deterministic retrieval fusion accounts for most of the classification improvement, but there is no comparison with simpler retrieval-based alternatives. The only external baseline is zero-shot GLoRIA, which is not evaluated under comparable supervision. Meanwhile, the paper does not assess whether the LLM reviewer catches incorrect label changes or whether the verifier’s evidence judgments are accurate. The results therefore support adding retrieval to the classifier, but do not sufficiently establish the benefit of the proposed agentic design.

Questions for authors

1、What specific methodological contribution distinguishes MEDAGENT-X from existing medical-agent and retrieval-augmented approaches?

2、Why is zero-shot GLoRIA the only external baseline, rather than classifiers evaluated under comparable supervision?

3、What evidence shows that the proposed fusion rules outperform simpler ways of combining predictions from retrieved cases?

4、How are the accuracy and usefulness of the LLM review and verification outputs evaluated?

Reviewer_GNhCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MEDAGENT-X, a framework for multi-label chest X-ray classification that selectively revises uncertain predictions using retrieved radiology reports. It combines a RAD-DINO classifier, rule-based evidence fusion, and constrained LLM-based review and verification. Experiments on CheXpert Plus show improvements over the vision-only baseline, particularly for predictions near decision thresholds.

Critical strengths
  1. The paper addresses a meaningful problem: improving uncertain medical image predictions while keeping decisions traceable.

  2. The separation between visual prediction, evidence-based intervention, and LLM verification is clear and practically motivated.

  3. The reported improvements are encouraging, and the ablations help identify deterministic retrieval fusion as the main source of performance gains. The additional expert-labeled evaluation is also useful.

Critical weaknesses
  1. The contribution of retrieval itself is not sufficiently isolated. Most improvements come from promoting negative predictions to positive ones. It is unclear whether similar gains could be achieved through simpler threshold adjustments without retrieval. This weakens the central claim about retrieval-grounded intervention.
  2. The evaluation protocol raises concerns about the generality of the results. Only around 15% of study-label pairs are scored in the main evaluation, and the scoreable labels are heavily positive-skewed. Given that the method mainly increases positive predictions, the reported gains may depend substantially on this evaluation setup.
Questions for authors
  1. How does the method compare with a simpler gray-zone threshold adjustment, especially when matched for the number of positive predictions?

  2. Would the improvements persist if randomly retrieved reports were used instead of visually similar ones?

  3. Can the authors clarify how the label-aware cohort selection and low evaluation coverage affect the reported results, particularly the precision-recall trade-off?

Reviewer_RkWzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an intelligent agent framework for multi-label diagnosis of chest X-rays. The RAD-DINO visual model initially outputs 12 labels, predictions falling within the threshold ±δ (δ=0.15) gray area trigger a two-step process: (1) kNN report retrieval (K=10, retrieving only the training set and excluding the same patient), and then label modification using rule-based negative extraction and deterministic fusion. (2) a constrained LLM reviews the results, and a read-only verifier generates an audit log. On the CheXpert Plus patient-split test set, the macro F1 score increased from 0.6971 to 0.7246, and the F1 score within the gray area increased from 0.6310 to 0.7613.

Critical strengths
  • The responsibilities of the prediction, intervention, and audit modules are separated. Neither the LLM nor the verifier can directly modify the diagnosis, ensuring clear safety boundaries. This design is suitable for clinical deployment.

  • Data is segmented by patient, the retrieval index only includes the training set and excludes patients from the same set, and the judge is independent of the inference process.

  • When all gray areas are changed to uncertain, the F1 score drops to 0.6126, indicating that the gain comes from retrieval fusion, not from abandoning the judgment. The paper provides a complete threshold vector and QC rules, demonstrating good reproducibility.

Critical weaknesses
  • The positioning as an ''agentic'' framework could be better aligned with the empirical contribution. CheXpert labels are extracted from reports, so deterministic fusion is essentially equivalent to voting on the kNN nearest neighbor reports, and almost all the gains come from this step. Adding LLM only increases the F1 score from 0.7246 to 0.7257. The results of the main configuration Qwen3-14B are identical to deterministic fusion without LLM, showing no substantial effect from LLM and agent orchestration.

  • There lacks sufficient baseline and statistical evidence. The paper does not compare it with the following simple schemes: kNN label voting, retrieval-enhanced classification, label-based threshold adjustment or calibration, and conformal selective prediction. External evaluation only compares it to zero-shot GLoRIA, where the absolute macro F1 score is only 0.398, after adding fusion on the real scale (0.1700), it is actually lower than vision-only (0.1719), and also inferior to GLoRIA in terms of edema. All experiments used only seed 42, without confidence intervals or significance tests.

  • The core claim of ''auditability'' lacks evaluation. The paper contains 2,989 evidence records, of which 387 changed the final label, but the accuracy, fidelity, and clinical usability of the verifier output were not manually assessed or quantified. Fusion thresholds such as δ=0.15, c+≥3, and c−≥1 lacked sensitivity analysis. Furthermore, the gain was concentrated in pleural effusion, lung shadows, edema, and atelectasis, while it was impaired in pneumothorax, a point that requires further discussion.

Questions for authors
  • Please supplement the following baselines: (a) kNN voting; (b) a vision-only model with threshold adjustment or calibration based on labels; (c) conformal or selective prediction for gray area samples. This is necessary to demonstrate the incremental improvement of this framework compared to simpler retrieval methods.

  • Please explain why the main configuration (Qwen3-14B) and the deterministic fusion results without LLM are identical, and report how many predictions LLM actually modified and whether the modifications were correct.

  • Please conduct a small-scale radiologist evaluation of the verifier records, e.g., sampling 100 records, to assess whether the evidence supports the conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.