acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Medical LLM Reasoning with AgentTool Reinforcement Learning

Abstract

Large language models (LLMs) have achieved strong performance on medical question answering, yet single-number accuracy obscures where they fail. We introduce a diagnostic evaluation framework that decomposes medical reasoning into three complementary abilities: , the possession of question-relevant medical knowledge; , the ability to retrieve such knowledge when answering a question; and , the ability to solve the question when the required knowledge is provided in context. We instantiate this framework on MedQA, MedMCQA, and MMLU-Med with dedicated probes for each ability. Evaluating six LLMs reveals that recall tracks downstream accuracy most consistently among the three measured abilities, whereas recognized knowledge alone shows weak and sometimes negative association with correctness. We further show that standard reinforcement learning (e.g., GRPO) improves primarily medium- and high-recall questions, leaving the low-recall regime largely untouched in our setting. Motivated by these findings, we propose , a reinforcement learning framework that addresses the low-recall bottleneck by making parametric recall an explicit, on-demand tool. The tool runs in an isolated context to generate relevant knowledge without disrupting final-answer reasoning. We further optimize reasoning and recall with decoupled group-wise objectives, using answer correctness for reasoning trajectories and a length-regularized recall-coverage reward for tool-generated recall. Across in-domain and out-of-distribution medical QA benchmarks, AgentTool RL outperforms standard GRPO and strong medical baselines on every dataset.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical LLMs, Agentic Reasoning, Medical Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sDagOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper decomposes medical LLM reasoning into three abilities: Knowledge, Recall, and Inference. The analysis finds that Recall is most consistently associated with accuracy, while standard RL provides limited improvements on low-recall samples. Based on these findings, the paper proposes AgentTool RL, which uses an isolated knowledge recall tool and decoupled rewards to optimize recall and reasoning separately. Experimental results show that AgentTool RL outperforms GRPO across five medical benchmarks.

Critical strengths
  1. The paper focuses on failure attribution in medical reasoning rather than merely improving benchmark accuracy, although the completeness of its capability decomposition requires further justification.
  2. AgentTool RL consistently outperforms standard GRPO across five medical benchmarks and two model backbones, covering both in-domain and out-of-distribution evaluations.
Critical weaknesses
  1. The paper does not justify why Knowledge, Recall, and Inference are sufficient to characterize medical reasoning, nor does it discuss or validate other potential capability dimensions.
  2. The correlation between Recall and Accuracy may be confounded by question difficulty, sampling randomness, and other factors. The current evidence is insufficient to establish recall as a primary bottleneck in medical reasoning.
  3. AgentTool RL mainly combines existing techniques, including context isolation, tool use, and decoupled rewards, with limited novelty in RL algorithms or optimization strategies.
  4. Traditional medical benchmarks such as MedQA, MedMCQA, and MMLU-Med are approaching saturation for some models, limiting their ability to assess complex medical reasoning. Moreover, Table 2 lacks the untuned Qwen3.5-9B baseline, making it difficult to assess the actual performance gains.
  5. The paper lacks a direct comparison of recall ability before and after training, leaving it unclear whether AgentTool RL genuinely improves parametric recall or merely benefits from additional reasoning steps or evidence reorganization.
  6. The recall reward relies primarily on semantic similarity to measure knowledge coverage, without explicitly evaluating factual correctness or contribution to the final decision. This may reward semantically relevant but medically incorrect knowledge.
  7. Given the complexity of AgentTool RL, why is RL-based parametric recall optimization preferable to simpler retrieval-augmented approaches such as RAG? Could the authors provide direct comparisons in terms of accuracy, computational cost, and inference efficiency?
Questions for authors

Please refer to Weaknesses.

1 comment
Reviewer_sDagOpenReview ↗

Official Comment by Reviewer sDag

I thank the authors for their detailed response. Most of my concerns have been adequately addressed, so I have decided to raise my score.

Reviewer_2smWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a diagnostic evaluation framework that decomposes medical reasoning into Knowledge, Recall, and Inference, and evaluates these abilities across six LLMs and three medical QA benchmarks. Experiments show that LLMs performs better when they can successfully retreive relevant medical knowledge during reasoning, rather than simply recognizing it in isolated questions. The authors also propose AgentTool RL, a reinforcement learning approach with separated rewards on answer correctness and knowledge coverage. Experiments on five medical QA benchmarks show consistent improvements of AgentTool RL over standard GRPO and other baselines.

Critical strengths

1.Interesting and well motivated research problem: The paper points out an interesting problem that even current LLMs may possess relevant medical knowledge, they still fail to recall these knowledge when answering complex questions. The decomposition into Knowledge, Recall, and Inference is intuitive and provides a useful perspective for understanding different failure modes of LLMs.

  1. Clear methodology with consistent experimental improvements: The proposed method is relatively simple and easy to follow. The authors first identify a strong correlation between Recall and downstream accuracy in Table 1, and then show the limitations of standard GRPO in the low-recall regime in Section 3.2. The proposed AgentTool RL approach is reasonable which optimize both the rewards for reasoning accuracy and knowledge coverage. Experiments in general show convincing improvements over standard GRPO.
Critical weaknesses
  1. Reliability of the recall score: The recall score is computed using BGE-M3 embedding similarity between generated and reference knowledge. It is unclear whether embedding similarity can reliably capture subtle but clinically important differences in medical knowledge, such as incorrect mechanisms or contradictory statements. Since this metric is central to both the diagnostic analysis and the RL reward, it would be helpful to provide more direct evidence of its reliability.
Questions for authors
  1. It is a bit unclear to what extent the performance gains depend on the knowledge provided by a stronger model such as GPT-5-mini. The authors seem to use GPT-5-mini to generate both the SFT trajectories and the Gold Knowledge used for the recall reward. For example, it would be helpful to report the performance of GPT-5-mini on the same benchmarks or more detailed discussions.

  2. Does AgentTool RL directly improve the model's knowledge recall ability? Figure 4 shows improvements in downstream accuracy across different recall regimes, but it is unclear whether the recall score itself improves after training. It would be nice to see the recall analysis before and after training.

Reviewer_5B5dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a diagnostic framework that measures medical LLM performance through Knowledge, Recall, and Inference probes. Motivated by the association between measured recall and QA accuracy, it introduces AgentTool RL, which performs optional knowledge recall in an isolated context and optimizes recall and answer generation with separate rewards. Experiments on five medical QA benchmarks show average improvements over standard GRPO of 3.0 percentage points on Qwen3-8B and 2.4 points on Qwen3.5-9B.

Critical strengths

The diagnostic framework offers a useful perspective beyond aggregate QA accuracy. The paper includes knowledge-quality checks, human calibration, and sensitivity analyses for the binning procedure, which strengthen the descriptive findings. Also, The method consistently improves accuracy across two backbones and five benchmarks. Self-Ask, tool-only, and forced-recall comparisons provide useful evidence about the proposed components.

Critical weaknesses
  1. The recall metric does not sufficiently distinguish factual recall from topical coverage. Appendix F.1 constructs concise knowledge points suitable as graph nodes, and the examples in Appendix J include concept names rather than explicit factual propositions. High embedding-based coverage may therefore reflect enumeration of relevant topics without establishing that the model has recalled the correct facts needed to answer the question. The length penalty does not directly address incorrect or contradictory content. Although the reported agreement with an LLM coverage judge supports measurement convergence, it does not resolve this distinction. This concern is consequential because the same metric underpins both the recall-bottleneck diagnosis and the auxiliary training reward.

  2. The evaluation does not sufficiently establish the necessity or cost-effectiveness of the proposed complexity under matched supervision and computation. Standard GRPO starts from 4,000 direct-only SFT trajectories, whereas AgentTool uses 8,000 paired trajectories containing additional recall and tool-conditioned answering supervision. The negligible improvement from direct-only SFT does not exclude benefits from this additional supervision. Furthermore, while the QA benchmarks are appropriate for measuring accuracy, aggregate scores do not establish when isolated recall and decoupled objectives are preferable to simpler knowledge-elicitation strategies. The tool is invoked on approximately 87% of MedQA and 77% of MedMCQA examples, making token usage and latency important considerations. Self-Ask and Forced Recall are useful controls, but do not isolate context separation under matched knowledge supervision, auxiliary rewards, and generation budgets. Consequently, the accuracy gains are promising, while their attribution to the proposed mechanism remains incomplete.

Questions for authors
  1. How does the recall metric distinguish concept-only lists, correct factual statements, and topically similar but incorrect statements when output length is controlled?
  2. Please provide a factuality audit of generated recall outputs and examine whether higher recall rewards correspond to fewer factual errors and better final-answer accuracy.
  3. Does AgentTool outperform a same-context “recall then answer” baseline with matched SFT supervision, the same recall reward, and comparable training and inference budgets? This comparison would help isolate the benefit of context separation.
  4. What are the total generated tokens, including tool outputs, and end-to-end latency for AgentTool, GRPO, and Self-Ask under identical evaluation conditions?
  5. Please break down accuracy gains and tool-invocation rates by factual recognition, single-step application, and multi-step reasoning, including uncertainty estimates for the gains over GRPO.
1 comment

Thank you for the detailed responses and for running all these extra ablations and audits. Some of my initial concerns on the experimental controls (like matching the 8K SFT data and comparing against the same-context baseline) have been addressed. However, my broader concern about the motivation and benchmark choice remains. These standard multiple-choice medical QA benchmarks are already very well addressed by existing LLMs/VLMs, so building an agent-tool RL pipeline just to answer static multiple-choice questions still feels somewhat weakly motivated to me (and as Q5 shows, the improvement on actual multi-step reasoning is still pretty marginal). It would be much more convincing to test this method in scenarios that genuinely call for agentic tool use and sequential decision-making, such as AgentClinic or open-ended clinical interaction benchmarks, rather than closed-ended QA. Thanks again for the effort, and I will factor your new results into the final discussion with the AC and other reviewers.

Comments on the paper

1 comment
Area_Chair_6paSOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.