acceptodds
Under review as a conference paper at ICLR 2027

X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning

Abstract

Chest X-ray (CXR) interpretation often requires integrating visual findings with task-specific evidence rather than relying on direct visual recognition alone. While tool-augmented medical agents can supply such evidence via external modules, they often lack explicit mechanisms for learning when auxiliary evidence is needed and for preventing inapplicable quantitative evidence from entering diagnostic reasoning. To address this, we propose X-Ragent, a post-training framework for tool-augmented reinforcement learning that decouples learned auxiliary-evidence seeking from validity-gated quantitative measurement acquisition for evidence-grounded CXR reasoning. We construct Rad-CoT, a 7.7K-sample corpus combining causally ordered tool-augmented trajectories with report-style diagnostic reasoning, and use it to initialize Qwen3-VL-8B via agentic supervised fine-tuning (SFT). We then optimize the policy with Group Relative Policy Optimization (GRPO) using an accuracy-gated reward that applies process-level terms for measurement consistency and auxiliary-tool cost only to correct trajectories, preventing plausible process behavior from compensating for an incorrect diagnosis. Across three CXR benchmarks, X-Ragent achieves the strongest results, including an 8.6-point improvement on the fully held-out ChestAgentBench. By seeking evidence selectively and admitting only valid measurements, X-Ragent supports more faithful and evidence-grounded CXR reasoning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Vision Language Model, Agentic Reinforcement Learning, Medical Agents, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8FC5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes X-Ragent, a tool-augmented reinforcement learning framework for evidence-grounded chest X-ray reasoning. It introduces Rad-CoT, a 7.7K-sample corpus of diagnostic reasoning trajectories, and combines agentic SFT with GRPO and accuracy-gated rewards. The method achieves strong results across three CXR benchmarks.

Critical strengths
  1. The motivation is reasonable, and the proposed corpus provides useful supervision for tool use and evidence-grounded reasoning.

  2. The method achieves strong results across three benchmarks.

Critical weaknesses
  1. The novelty is limited, as the method primarily combines existing techniques, including SFT, GRPO, and tool augmentation.

  2. Ablation results suggest that the gains from RL over SFT are relatively modest, raising questions about the necessity and contribution of RL to the overall performance.

  3. Rad-CoT relies on LLM-generated reasoning trajectories filtered by rule-based checks, leaving their reliability insufficiently validated.

  4. The baseline comparison is insufficient, with limited coverage of recent medical VLMs and tool-augmented reasoning agents.

Questions for authors
  1. What is the substantive methodological novelty of X-Ragent compared with existing approaches?

  2. How do the authors validate the reliability of the LLM-generated reasoning trajectories in Rad-CoT?

  3. How does X-Ragent compare with recent medical VLMs and tool-augmented reasoning methods?

4 comments

Thank you for reviewing our work and for your suggestions on methodological contribution and data reliability. We hope these additions answer your questions; following your feedback, we will refine the relevant material and upload a revised manuscript.

W1 / Q1: What is the methodological novelty beyond combining existing techniques?

A1: We do not claim SFT, GRPO, learned tool calling, or correctness conditioning alone as novel. X-Ragent organizes evidence acquisition, applicability checks, and diagnostic process supervision around the different degrees of evidence verifiability.

Separate acquisition from applicability. Parsing, cropping, and disease knowledge have question-dependent value and are selected by the policy. CTR, LFA, and Thoracic Shift depend on projection and anatomical geometry; a separate backend checks these conditions and directly supplies accepted measurements without a policy-selected measurement call.

Supervise what can be checked. Visual/knowledge use is optimized through final task performance. Eligible measurements additionally permit checks of value, category, and direction interpretation against backend outputs. We combine this task-relevant R_CXR with auxiliary-call cost:

R(τ)=Rformat+Racc(1+λmRCXR−λtRtool)R(\tau)=R_{\mathrm{format}}+R_{\mathrm{acc}}(1+\lambda_m R_{\mathrm{CXR}}-\lambda_t R_{\mathrm{tool}})

Only correct answers receive measurement-consistency credit; call cost discourages unnecessary acquisition. Rad-CoT's causally ordered calls, actual observations, and diagnoses align supervision with this architecture. The design explicitly controls both evidence admissibility and interpretation, extending beyond tool selection alone. Corresponding controls are:

DesignControlled comparisonChestAgentBench / CheXbench
Independent measurement pathwayvs. Policy Routing + Gate (revised Table 12)+0.7 / +1.2 pp
Tool-interaction supervisionRemove tool trajectories (Table 5)−4.5 / −6.1 pp
Accuracy-gated optimizationvs. ungated reward combination (Table 3)+1.9 / +1.7 pp

These controls separately support the pathway, supervision, and reward design, providing component evidence for the methodological contribution. Reply 2/2 includes recent-agent comparisons.

W2: What does RL contribute beyond SFT?

A2: With the same tools, Measurement Gate, and inference protocol, RL improves held-out ChestAgentBench accuracy by 3.1 pp:

CheckpointWithout tools (%)With the same tools (%)
SFT64.468.6
SFT + RL67.471.7
RL gain+3.0 pp+3.1 pp

Tool-free results are from Table 2; evaluating the SFT checkpoint with the same tools is an additional experiment. ChestAgentBench is excluded from post-training in both settings. Table 3 further shows that the full reward exceeds Answer + Format Reward by 2.3 pp (69.4%→71.7%) and the ungated combination by 1.9 pp. RL therefore improves the capabilities established by SFT even when tool access is held fixed.

W3 / Q2: How is Rad-CoT reliability validated beyond rule-based filtering?

A3: We add reference-report claim audits and independent medical-model screening:

Report-style auditResult
Randomly sampled trajectories400/4,000
Claims supported by reference Findings97.3%
Contradicted / indeterminate claims0.6% / 2.1%
Trajectories with no detected contradiction97.5%

The audit detects 49/50 (98.0%) separately constructed contradictions involving disease presence, laterality, location, and severity. For 300/3,715 task-stratified routed trajectories, Lingshu-7B, distinct from the generation teacher, checks the CXR, ground-truth answer, and evidence, flagging 9 (3.0%). The audit permits image-based corrections to tool observations.

Rad-CoT contains 4,000 report-style trajectories generated by Qwen3.5-122B-A10B-FP8 from CXRs and reference Findings, retaining Findings as targets, plus 3,715 QA-derived routed trajectories with ground-truth answers and actual tool observations. Calls precede observations; teacher rescreening and format, length, similarity, and leakage filtering follow. The sampled claims' reference support and independent screening provide content-level evidence for supervision reliability beyond formatting checks.

W4 / Q3: How does X-Ragent compare with recent medical VLMs and agents?

A4: We add [CheXOne (Zhang et al., 2026)], [Lingshu-7B (Xu et al., 2025)], and mechanism adaptations of [ViTAR (Chen et al., 2025)] and [Ophiuchus (Jiang et al., ICML 2026)], whose preprint appeared in 2025.

MethodChestAgentBench (%)CheXbench (%)
CheXOne58.7271.27
Lingshu-7B63.6063.59
ViTAR-style67.873.5
Ophiuchus-inspired70.375.8
X-Ragent71.777.3

ChestAgentBench uses all 2,500 questions; CheXbench is the macro-average over three sources. CheXOne and Lingshu-7B use direct inference. ViTAR-style shares Qwen3-VL-8B-Instruct, CXR training tasks, and evaluation, retaining region marking, two-turn reinspection, and SFT/GRPO. Ophiuchus-inspired shares Rad-CoT SFT initialization, tool interfaces, and GRPO configuration, using its VQA-conditioned tool reward without self-reflection SFT. These are controlled mechanism adaptations, not full reproductions of the original methods.

X-Ragent gains 3.9/3.8 pp over ViTAR-style and 1.4/1.5 pp over Ophiuchus-inspired. Against the latter on ChestAgentBench, invocation rate drops from 93.1% to 53.2%, calls from 1.52 to 0.95/question (−37.5%), and latency from 15.8 to 12.4 s/question (−21.5%). Invocation rate counts questions with any call; averages use all questions.

Apologies for the delayed response. I've been quite busy recently.

Thank you for the rebuttal. However, even after the authors' explanations, I still don't quite see the novelty and significance of this work. I also don't find the significant performance gains over the newly added baselines. Currently, I will maintain my rating.

Thank you for clarifying your remaining concerns. We would like to explain more concretely the design contribution and the evidence supporting its value.

1. What does X-Ragent contribute?

X-Ragent separates two decisions that require different criteria. The policy learns which auxiliary information is useful for a question, including regional inspection, crops, and disease knowledge. A separate backend determines whether quantitative measurements are applicable based on projection and anatomical geometry, and supplies eligible measurements without requiring a policy call. For example, a question may concern heart size, while the image projection makes CTR unsuitable for interpretation.

Training follows this separation. Visual and knowledge tool use is optimized through answer performance. Measurement interpretations additionally receive consistency supervision against backend values, categories, and directions. Both measurement-consistency credit and auxiliary-call cost apply only to correctly answered trajectories. Rad-CoT provides causally ordered tool calls, actual observations, and diagnostic reasoning for learning this behavior.

SFT, GRPO, and correctness-conditioned rewards are established techniques. Our methodological contribution is the specific coordination of learned evidence acquisition, independent measurement applicability checks, and measurement-consistency supervision.

2. What demonstrates its practical significance?

The added comparisons show benefits in both accuracy and computational cost. X-Ragent exceeds ViTAR-style by 3.9/3.8 pp on ChestAgentBench/CheXbench. Against the closer Ophiuchus-inspired adaptation, it gains 1.4/1.5 pp while using 37.5% fewer auxiliary calls and 21.5% less end-to-end time on ChestAgentBench. The latter comparison shares the backbone, Rad-CoT SFT initialization, tool interfaces, and GRPO configuration, using the Ophiuchus-inspired tool reward without its self-reflection SFT. These results evaluate the specified CXR adaptations.

Further controls help identify where the benefit comes from. With the same checkpoint, toolbox, and 0.95 calls/question, learned routing reaches 71.7%, compared with 69.3% for budget-matched random routing. Relative to Static-All, it reduces latency by 35.1% and GPU-seconds by 35.8%, with observed accuracies of 71.7% and 71.0%. Separately, the Gate rejects 379/400 AP or lateral inputs in projection validation, directly supporting its role in screening measurement applicability.

Together, these experiments support the value of the proposed design through component gains, question-dependent tool selection, and lower inference cost.

Reviewer_ug6TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents X-Ragent, a tool-augmented model for chest X-ray reasoning. The model first selects auxiliary tools, then uses their outputs to produce an answer. Quantitative measurements are supplied separately, subject to an acquisition-validity check. Training combines SFT on the proposed Rad-CoT dataset with GRPO using answer accuracy, measurement consistency, and tool cost. The results are encouraging, especially the 8.6-point gain over MedRAX on the held-out ChestAgentBench. My main concerns are the distinction from prior work, the validity of the measurement-based supervision, and whether the experiments support the claimed improvement in reasoning faithfulness.

Critical strengths
  1. Separating auxiliary-tool selection from measurement applicability is a sensible choice. The policy decides what additional information to seek, while the backend checks whether quantitative measurements can be used. Treating unavailable measurements differently from normal findings is also useful.

  2. ChestAgentBench is excluded from post-training, and the reported accuracy improves from 63.1% for MedRAX to 71.7%, with gains across all seven capability categories. The documented image exclusions and report-level overlap checks strengthen this evaluation.

  3. The ablations cover training stages, reward components, corpus composition, and inference-time evidence modules. The comparison with static tool invocation is particularly relevant to the routing design. Observation masking and causally ordered tool trajectories are also appropriate choices.

Critical weaknesses
  1. Tool-integrated SFT followed by reinforcement learning for adaptive visual inspection is already explored in related medical agents. Ophiuchus also conditions its tool-use reward on answer correctness. X-Ragent introduces a separate measurement pathway, a measurement-consistency reward, and a tool penalty, but the experiments do not clearly establish their advantages over a closely related training framework. Ophiuchus and ViTAR are discussed without experimental comparisons. Using the same backbone, training data, and toolbox would make the methodological contribution easier to assess.

  2. Appendix C.3 computes CTR using the maximum-width heart row and the lung-field span at that same row. This differs from the cited method’s use of the horizontal extrema of the heart and lung masks, so applying the conventional 0.5 threshold requires validation. More generally, deterministic calculations from predicted masks are not necessarily accurate measurements. Since the backend supplies both the observations and the targets for the consistency reward, its errors could be reinforced during training. Agreement with expert measurements and an evaluation of gate errors are needed to support the description of this evidence as verified.

  3. The reported answer accuracies do not directly establish faithful evidence integration. A model could repeat the supplied measurement interpretation while reaching its answer through other cues. Table 4 shows that tools are called more frequently on questions the base model answers incorrectly, but this does not establish whether those calls improve the decision. The knowledge supplied by evidence_lookup also comes from another LLM rather than a verified reference source. Controlled tests with withheld, shuffled, and corrupted observations would help determine whether the policy uses evidence appropriately.

  4. The system combines an 8B policy with Qwen3.5-27B and specialized perception models. Removing auxiliary interpretation produces the largest individual module drop in Table 10, suggesting that access to the knowledge model contributes substantially. The static-all-tools comparison is useful, but it does not separate routing quality from the benefit of additional information and computation. Comparisons with the base policy using the same toolbox, fixed knowledge augmentation, and random routing at a matched call budget would help. End-to-end latency and resource measurements are also needed to assess practical efficiency.

  5. Table 6 reports a 0.24-point overall improvement from validity gating, with 17 questions corrected and 11 becoming incorrect. Table 10 gives learned routing a 0.7-point advantage over static invocation. These results favor the proposed design, but paired confidence intervals would help assess how much weight to place on the differences. The standalone SLAKE gain over MedRAX corresponds to approximately three additional correct answers among 114 questions. Repeated training runs and uncertainty estimates that account for questions sharing an image or case would make these comparisons more convincing.

  6. Several details remain insufficiently specified for reproduction. Appendix B.4 does not fully define the measurement–task relevance mapping, the scoring rules for s_k, or the treatment of omitted and contradictory measurement statements. Mask-quality thresholds and the thoracic-shift calculation also need clarification. Appendix D includes base-model prompting variants but omits the main agent, teacher-generation, and knowledge-tool prompts. For Rad-CoT, length and similarity filtering address formatting and copying, but do not independently establish visual correctness. An expert audit of sampled trajectories would strengthen confidence in the supervision.

Questions for authors
  1. Which gains remain when a closely related agentic training method uses the same backbone, corpus, and tools?

  2. How accurately do the measurements agree with expert annotations? How often does the gate admit unsuitable inputs or reject suitable ones?

  3. Does the model revise its answer appropriately when relevant observations are removed or replaced with incorrect evidence?

  4. How does learned routing compare with random routing at the same call budget, and what runtime saving does it provide?

2 comments

Thank you for your careful review and thoughtful suggestions. Your feedback helped us strengthen the paper, and we will follow up by uploading a revised manuscript that includes the new results and clarifications.

W1 / Q1: Which gains remain with the same backbone, data, and tools?

A1: SFT, GRPO, and correctness-conditioned tool rewards are established techniques. X-Ragent separates learned tool routing from independent measurement applicability checks and gates both measurement consistency and tool cost on answer correctness. The policy selects question-dependent parsing, crops, and knowledge; a separate backend checks projection/geometry and supplies applicable measurements without a policy call. Visual/knowledge use is optimized through task performance, while R_CXR checks eligible measurement interpretations against backend values, categories, and directions. Accuracy-gated R_CXR and R_tool reward consistent interpretation and efficient acquisition only with correct answers. Rad-CoT supplies causally ordered calls, actual observations, and diagnoses.

Recent [ViTAR (Chen et al., 2025)] emphasizes regional reinspection, and [Ophiuchus (Jiang et al., ICML 2026)] iterative tool-based visual reasoning. Our CXR adaptations yield:

MethodChestAgentBench (%)CheXbench (%)Calls/questionLatency (s/question)
ViTAR-style67.873.5——
Ophiuchus-inspired70.375.81.5215.8
X-Ragent71.777.30.9512.4

ViTAR-style uses the same Qwen3-VL-8B-Instruct and CXR protocol, retaining region marking and two-turn interaction. Ophiuchus-inspired fixes Rad-CoT SFT initialization, tool interfaces, and GRPO configuration, but substitutes its VQA-conditioned tool reward without self-reflection SFT. These are controlled mechanism adaptations, not full reproductions of the original methods. Efficiency is measured over all ChestAgentBench questions.

X-Ragent gains 1.4/1.5 pp, with 37.5% fewer calls and 21.5% lower latency, over Ophiuchus-inspired. With backend and Gate fixed, independent measurement routing gains 0.7/1.2 pp over Policy Routing + Gate (revised Table 12); the full reward gains 1.9/1.7 pp over the ungated combination (Table 3). These controls support benefits from the pathway and training objective beyond shared tool access.

W2 / Q2: Validate CTR, expert agreement, and Gate false acceptances/rejections.

A2 — CTR implementation. Thank you for flagging the CTR description. We verified that the original code and Fig. 2 visualization both use CTR = W_heart,max / W_thorax,max, with the widths maximized independently. Heart and internal thoracic widths come from the heart and bilateral rib masks, respectively. We corrected the same-row description in Appendix C.3, Eqs. (12)–(13). Training and all reported results used the original implementation.

Validation against external references

CTR and LFA. We compare CTR with expert measurements and LFA with expert contours.

Measurement/referenceNMAE (95% CI)ICC(2,1)
CTR / Wingspan expert CTR2210.0417 [0.0349, 0.0495]0.545
LFA / Montgomery expert contours1380.0711 [0.0591, 0.0836]0.672

CTR Spearman correlation is 0.700. At the 0.5 threshold, accuracy, sensitivity, and specificity are 80.54%, 78.29%, and 85.51%. LFA Pearson correlation and CCC are 0.736 and 0.671. The predefined categories (<0, 0–0.26, >0.26) yield 87.7% accuracy (121/138, 95% CI [81.9%, 92.8%]) and κ=0.626. Directional agreement is 94.2% (130/138, 95% CI [89.9%, 97.8%]). All 17 errors are between adjacent categories. CIs use 10,000 case-level bootstrap replicates.

Thoracic shift. We used report labels as references for 887 patient-deduplicated MIMIC-CXR PA images and set thresholds on the development set only. Of 532 locked test cases, 304 passed geometry checks (57.1% coverage, 95% CI [52.8%, 61.5%]). Among accepted cases, AUROC is 0.686 [0.626, 0.741] and AUPRC is 0.792. Directional agreement is 90.18% [84.82%, 95.54%] among definite positives that passed the checks and were detected.

Projection and rotation screening. On 600 independently labeled MIMIC-CXR/CheXpert images, the Gate accepted 186/200 PA and rejected 379/400 AP or lateral images. PA false rejection was 7.0%, non-PA false acceptance was 5.25%, and overall accuracy was 94.17% [92.17%, 96.00%]. The Gate rejected all 200 lateral images, and all false acceptances were AP. It rejected all 300 inputs in the 90° stress test. In 21 paired cases with computable CTR, bypassing the Gate raised MAE from 0.0409 to 0.4876, showing that computable CTR can still be highly erroneous on rotated inputs.

Implications for training. R_CXR uses backend outputs as consistency references. External comparisons quantify measurement error. Errors in accepted measurements can still affect the consistency reward on correctly answered examples.

W3 / Q3: Is evidence used appropriately under interventions?

A3 — Measurement interventions (ChestAgentBench). On Gate-accepted samples only, we change the second-turn measurement bundle: Withheld marks it unavailable; Shuffled permutes values across accepted samples, preserving their distribution; Corrupted reverses directions and moves CTR across 0.5.

ConditionAcc (%)Δ (pp)Correct→WrongWrong→CorrectFlip rate (%)
Original72.41————
Withheld71.13−1.2843214.70
Shuffled71.42−0.9937204.17
Corrupted70.72−1.6852235.62

Flip rates include changes between two incorrect answers. On questions that passed the Gate, accuracy was highest with the original measurements and fell by 0.99–1.68 percentage points when measurements were withheld, shuffled, or corrupted. These results show that some final answers depend on measurement content and that corrupted measurements can mislead the model.

Knowledge interventions. With first-turn reasoning/plans and other inputs fixed, we withhold/corrupt evidence_lookup knowledge. Across all 2,500 questions, Original/Withheld/Corrupted accuracy is 71.7/69.8/68.6%, with 85/38 and 123/46 correct→wrong/wrong→correct changes. The knowledge is Qwen3.5-27B-generated. Both interventions support content-dependent decisions; their denominators differ.

W4 / Q4: Does learned routing improve matched-budget accuracy and runtime?

A4: At 0.95 calls/question, learned routing scored 2.4 pp higher than random routing. Against Static-All, it cut calls, latency, and GPU-seconds by 68.3%, 35.1%, and 35.8% (17.2 vs. 26.8 GPU-s/question).

RoutingAccuracy (%)Calls/questionLatency (s/question)
Learned71.70.9512.4
Budget-matched random69.3 ± 0.40.9512.6
Fixed knowledge + learned visual71.31.62—
Static-All71.03.0019.1

Random routing (five seeds) shuffles plans, preserves call counts, types, combinations, and dependencies, and regenerates parameters per question. Fixed knowledge always calls evidence_lookup.

Runs share the checkpoint, toolbox, hardware, resident services, and inference settings. Costs include backend/Gate runtime and Static-All parameter completion.

W5: Quantify uncertainty in the small accuracy gains.

A5: Gate CIs use 1M paired question-level bootstrap replicates. P-values are from exact McNemar tests.

ComparisonCorrections/regressionsΔ accuracy (pp), 95% CIp
Gate: overall17/11+0.24 [−0.16, +0.64]0.345
Gate: frontal8/80.00 [−0.42, +0.42]1.000
Gate: non-frontal/atypical9/3+0.98 [−0.16, +2.12]0.146
Learned vs. Static-All55/38+0.68 [−0.08, +1.44]0.097

Paired cluster bootstraps for the Gate (100K replicates each by shared-image group and case/article) yielded nearly unchanged 95% CIs. Under both groupings, the CIs were [−0.16, +0.64] pp overall and [−0.16, +2.11] pp for non-frontal/atypical inputs. Both include zero. All six net corrections came from the latter subset.

W6: Clarify scoring, geometry, prompts, and trajectory validation.

A6: Revised Appendices B.4, C.3, D.2–D.4 specify:

  • Scoring: Question/option concepts (including equivalents/negations) map CTR to heart size, LFA to lung-area asymmetry, Shift to tracheal/mediastinal displacement. Generic abnormality activates none. a_k=r_k b_k controls reward eligibility, not execution. s_k=1 requires correct, noncontradictory value/category/direction interpretation; omissions, number-only repetition, errors, ambiguity, or unparsable text score 0. Explicit corrections are allowed; injected observations are not scored. R_CXR=sum(a_k s_k)/max(1,sum(a_k)); missing is not normal. Tool cost counts repeated auxiliary calls, excluding backend routines.
  • Geometry: PA/orientation and mask checks precede measurements. CTR uses independent width maxima; LFA=ln(A_R/A_L) requires valid positive areas. Shift=median_y[(T−V)/W] uses valid rows in the upper 60% of rib-cage height. Checks: N_W≥5, largest tracheal-component area fraction ≥5×10^-5, nonempty vertebrae, N_valid≥max(5,0.15N_T), median_y |(V−C)/W|≤0.06. Sign gives patient left/right; 0.02/0.05 delimit minimal/moderate/marked magnitude. Failed projection/item checks withhold the bundle/item.
  • Prompts/audits: Agent, teacher, and knowledge prompts define roles, formats, and causal observation injection. In 400/4,000 report trajectories, claims are 97.3% supported, 0.6% contradicted, 2.1% indeterminate; 97.5% have no detected contradiction. The audit detects 49/50 controlled contradictions. Lingshu-7B independently flags 9/300 (3.0%) task-stratified routed trajectories using images, answers, and evidence; no radiologist image audit was conducted.
Reviewer_X6HzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a chest X-ray inference framework based on Qwen3-VL-8B. The model first selects optional tools for region examination and medical knowledge lookup, then integrates their outputs to generate the final answer, providing quantitative measurements only after acquisition and mask quality checks. The authors construct Rad-CoT for supervised fine-tuning and apply the GRPO algorithm based on accuracy gating. Experiments on three benchmark datasets show that X-Ragent achieves improved answer accuracy, reaching 71.7% on the reserved ChestAgentBench dataset and 63.1% on the MedRAX dataset.

Critical strengths

Extensive experiments were conducted on reward items, data composition, evidence modules, static use of all auxiliary tools, and measurement validity gating. X-Ragent achieved 71.7% accuracy on the ChestAgentBench dataset, outperforming its base model and the strongest baseline model in Table 1. Furthermore, this benchmark test is not included in the proposed post-training process, demonstrating the framework's applicability beyond training data.

Critical weaknesses

The value of these new designs relative to existing tool-based inference needs further proof. Medical agents like ViTAR and Ophiuchus have already explored tool selection and inference based on returned evidence. Besides the Rad-CoT dataset, the paper's technical innovations are primarily a separate measurement validity gating and a specific reward design for measurement consistency and tool usage costs. However, the measurement validity gating only increased the overall accuracy of ChestAgentBench from 71.44% to 71.68% (Table 6), but its contribution to overall accuracy is still small. The efficiency gains from learning-based tool selection are not yet sufficiently quantified. Learning-based routing achieved 71.7% on ChestAgentBench, while using all auxiliary tools resulted in 71.0% (Table 10). The 53.2% call rate in the report only accounts for the percentage of issues where an auxiliary tool was called at least once. It doesn't specify the actual computational or time savings. To demonstrate the actual benefits compared to using all tools, average call counts by tool type, along with end-to-end latency or computational cost, need to be provided.

Questions for authors

Compared to medical agents like ViTAR and Ophiuchus, what are the main technological innovations? Can a comparison be made with comparable data and tools, observing the benefits of the added design in isolation? Furthermore, measurement validity gating is touted as a key contribution, but Table 6 shows an overall accuracy increase of only 0.24 percentage points. Does this gating effectively prevent errors caused by inapplicable measurements? Regarding the comparison between learning-based routing and static routing using all auxiliary tools, it's also necessary to explain the average number of calls per tool type under the same conditions, as well as end-to-end latency or computational cost. Furthermore, how are the target regions and candidate conditions selected in the static baseline?

2 comments

We are grateful for your thorough and positive review. Your recognition of our work is highly encouraging. If the additions below address your concerns, we will incorporate them into the manuscript and upload an updated version. We address your concerns below.

W1 / Q1: What advances beyond recent agents remain under comparable conditions?

A1: We do not claim SFT, GRPO, tool calling, or correctness conditioning alone as novel. X-Ragent separates learned tool routing from independent measurement applicability checks and gates both measurement consistency and tool cost on answer correctness.

Parsing, cropping, and disease knowledge have question-dependent value, so the policy selects them. CTR, LFA, and Thoracic Shift depend on projection and anatomical geometry; an independent backend checks applicability and supplies accepted measurements without requiring a policy call. Auxiliary visual/knowledge use is optimized through task performance. For eligible measurements, R_CXR additionally checks interpretation of values, categories, and directions against backend outputs; it and R_tool are accuracy-gated, so measurement credit cannot compensate for an incorrect answer. Rad-CoT aligns training with this design through causally ordered calls, actual observations, and diagnoses.

We add CXR adaptations of recent agents ViTAR (Chen et al., 2025) and Ophiuchus (Jiang et al., ICML 2026):

MethodChestAgentBench (%)CheXbench (%)Calls/questionLatency (s/question)
ViTAR-style67.873.5——
Ophiuchus-inspired70.375.81.5215.8
X-Ragent71.777.30.9512.4

ViTAR-style shares the backbone, CXR training tasks, and evaluation protocol, retaining region marking, two-turn interaction, and SFT/GRPO. Ophiuchus-inspired shares Qwen3-VL-8B-Instruct, Rad-CoT SFT initialization, tool interfaces, and GRPO configuration, but uses its VQA-conditioned tool reward without self-reflection SFT. These are controlled mechanism adaptations, not full reproductions of the original methods. Calls and latency are measured on ChestAgentBench, with all questions as the denominator.

X-Ragent gains 1.4/1.5 pp over Ophiuchus-inspired while reducing calls by 37.5% and latency by 21.5%. Separately, fixing the backend and Gate, the independent pathway exceeds Policy Routing + Gate by 0.7/1.2 pp (revised Table 12); the full reward exceeds its ungated counterpart by 1.9/1.7 pp (Table 3); removing tool-trajectory supervision costs 4.5/6.1 pp (Table 5). These controls support contributions from the pathway, supervision, and training objective beyond shared tool access.

Q2: Can you provide broader quantitative evidence that the Gate prevents errors from inapplicable measurements?

A2: On 600 independently labeled CXRs from MIMIC-CXR and CheXpert, the combined Type + Rotation Gate gives:

Reference projectionAcceptedRejected
PA (200)18614
AP/lateral (400)21379

Projection screening. It retains 93.00% of PA inputs and rejects 94.75% of non-PA inputs: accuracy 94.17% (95% CI [92.17%, 96.00%]), PA false-rejection 7.0%, and non-PA false-acceptance 5.25%. All 200 lateral images are rejected; all false acceptances are AP. Accuracy is 93.0%/95.33% by source.

Rotation stress test. For a stress test, we rotate 100 images per measurement by 90°. The Gate rejects 300/300; bypassing it still yields CTR/LFA/Shift values for 21/3/2 images. In the 21 paired CTR cases, MAE rises from 0.0409 to 0.4876, with an increase of 0.4467 (95% CI [0.2475, 0.7408]). The Gate thus blocks computable but highly erroneous values under this condition.

Downstream accuracy. Table 6 shows +0.24 pp overall (17 corrections/11 regressions) and +0.98 pp in 613 non-frontal/atypical questions (9/3), with zero net change on frontal inputs. Paired 95% CIs are [−0.16, +0.64] and [−0.16, +2.12] pp, with exact McNemar p=0.345/0.146. These intervals include zero. The independent projection labels and Table 6 view groups use different definitions.

Projection validation and the stress test support the Gate's screening value on the evaluated inputs; downstream net corrections concentrate in non-frontal/atypical inputs.

W2 / Q3: Report calls, runtime, and Static-All region/condition selection.

A3: Learned Routing reduces auxiliary calls by 68.3%, end-to-end latency by 35.1%, and GPU-seconds by 35.8% relative to Static-All, using identical hardware, resident model services, and inference settings.

MetricLearned RoutingStatic-All
Accuracy (%)71.771.0
Invocation rate (%)53.2100.0
region_parse calls/question0.341.00
roi_crop calls/question0.281.00
evidence_lookup calls/question0.331.00
Total auxiliary calls/question0.953.00
MLLM latency (s/question)7.17.4
Tool latency (s/question)5.311.7
End-to-end latency (s/question)12.419.1
GPU-seconds/question17.226.8

Average calls use all ChestAgentBench questions as the denominator; invocation rate denotes the fraction receiving at least one call. Both configurations use the same independent Measurement Gate and backend. Their runtime is included in end-to-end latency, but backend calls are excluded from auxiliary-call counts. The paired accuracy difference is +0.68 pp (95% CI [−0.08, +1.44]; exact McNemar p=0.097). The practical advantage is lower measured runtime and compute cost with similar observed accuracy.

Static-All parameter selection. Both configurations share first-stage image observations. Static-All reuses parameters for tools selected by Learned Routing and fills in parameters for unselected tools from the current question and preliminary observations, without ground-truth answers:

  • Regions: Prefer the question-specified region, then the first localizable abnormal region in preliminary observations, otherwise both lungs. region_parse invokes CXAS/BiomedParse according to the structure or lesion; roi_crop uses the returned bounding box to crop the original image.
  • Conditions: Extract and deduplicate disease names from the question and options. If none are explicit, generate up to three relevant diseases or radiographic concepts from preliminary observations. All configurations use the same Qwen3.5-27B query template for morphology, distribution, and differential information.

Static-All executes all three auxiliary tool types for every question; parameter-completion overhead is included. The comparison therefore measures actual savings from selective evidence acquisition under a shared toolbox and measurement pathway.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.