- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper decomposes medical LLM reasoning into three abilities: Knowledge, Recall, and Inference. The analysis finds that Recall is most consistently associated with accuracy, while standard RL provides limited improvements on low-recall samples. Based on these findings, the paper proposes AgentTool RL, which uses an isolated knowledge recall tool and decoupled rewards to optimize recall and reasoning separately. Experimental results show that AgentTool RL outperforms GRPO across five medical benchmarks.
Critical strengths
- The paper focuses on failure attribution in medical reasoning rather than merely improving benchmark accuracy, although the completeness of its capability decomposition requires further justification.
- AgentTool RL consistently outperforms standard GRPO across five medical benchmarks and two model backbones, covering both in-domain and out-of-distribution evaluations.
Critical weaknesses
- The paper does not justify why Knowledge, Recall, and Inference are sufficient to characterize medical reasoning, nor does it discuss or validate other potential capability dimensions.
- The correlation between Recall and Accuracy may be confounded by question difficulty, sampling randomness, and other factors. The current evidence is insufficient to establish recall as a primary bottleneck in medical reasoning.
- AgentTool RL mainly combines existing techniques, including context isolation, tool use, and decoupled rewards, with limited novelty in RL algorithms or optimization strategies.
- Traditional medical benchmarks such as MedQA, MedMCQA, and MMLU-Med are approaching saturation for some models, limiting their ability to assess complex medical reasoning. Moreover,
Table 2lacks the untuned Qwen3.5-9B baseline, making it difficult to assess the actual performance gains. - The paper lacks a direct comparison of recall ability before and after training, leaving it unclear whether AgentTool RL genuinely improves parametric recall or merely benefits from additional reasoning steps or evidence reorganization.
- The recall reward relies primarily on semantic similarity to measure knowledge coverage, without explicitly evaluating factual correctness or contribution to the final decision. This may reward semantically relevant but medically incorrect knowledge.
- Given the complexity of AgentTool RL, why is RL-based parametric recall optimization preferable to simpler retrieval-augmented approaches such as RAG? Could the authors provide direct comparisons in terms of accuracy, computational cost, and inference efficiency?
Questions for authors
Please refer to Weaknesses.
Official Comment by Reviewer sDag
I thank the authors for their detailed response. Most of my concerns have been adequately addressed, so I have decided to raise my score.