- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CohortWorld, an interactive environment built from UK Biobank records for disease-risk prediction. It provides 29,884 cohorts derived from over 500,000 participants and introduces CohortCal to evaluate model predictions against observed incidence rates. It also proposes CS-GRPO to train models to select comparison cohorts, query their outcomes, and estimate risk adjustments. Experiments evaluate general-purpose and medical language models and report improvements in risk discrimination and calibration with CohortWorld.
Critical strengths
- The paper builds an agentic environment from over 500,000 UK Biobank participants. The query environment lets models use observed outcomes without releasing individual records. This is a useful resource for studying evidence-based risk prediction.
- Post-training improves risk discrimination and reduces tool use. For Qwen3.5-9B, CT AUROC increases from 0.516 to 0.638. Average tool calls decrease from 12.0 to 3.3. These results show that training helps agents use the environment more effectively.
Critical weaknesses
- The medical explanations used for training are generated by MedGemma, but no independent clinical validation is reported. The numerical adjustment can be computed from observed cohort rates. This does not verify the explanation for that difference. In particular, an association between a history feature and disease incidence does not establish the causal account given in the explanation. It is unclear whether these demonstrations teach reliable medical reasoning.
- Table 4 shows a substantial decline in CC after post-training. For Qwen3.5-9B, accuracy falls from 51.6% to 16.0% and reaches 16.3% with the adjustment reward. For Qwen3.6-27B, it falls from 61.1% to 20.5%, then to 18.1%. Section 4.1 interprets this as a trade-off between matching and contrast. However, it remains unclear why the adjustment reward does not recover CC performance, or how these results support improved risk-adjustment reasoning.
- Table 3 shows that agents with access to cohort incidence rates outperform models answering from medical history alone. However, this comparison changes both data access and the prediction procedure. An ablation study should remove the adjustment step while keeping the agent-selected cohorts fixed, returning their observed incidence rates directly. Comparing this variant with the full method, including absolute prediction error, would show whether adjustment improves prediction. A simple rule-based cohort-matching baseline would also help assess the value of LLM-based cohort selection.
- Figure 1 illustrates a patient asking whether existing symptoms will worsen. The task in Section 3 predicts a first recorded diagnosis within a fixed time horizon. The figure suggests a broader clinical task than the experiments evaluate and should be revised to match the actual setting.
- The training gains need further ablation. Table 4 should include an SFT-only model to separate the effects of SFT and RL. Comparing CS-GRPO with standard GRPO under the same training conditions would also help assess the value of the cross-sample reward in Section 3.3.
- The contribution of the generated explanations is not clear. An SFT ablation that retains the reference cohort and adjustment value but removes the medical explanation would show whether the explanation itself helps.
Questions for authors
-
How were the generated medical explanations checked for clinical correctness and consistency with the prescribed adjustment?
-
Why does CC decline for both models after post-training? Why does adding the adjustment reward fail to recover this performance?
-
How does the method compare with simple rule-based cohort matching? For the same agent-selected cohorts, does applying the adjustment improve on directly returning the queried incidence rate?
-
How much improvement does RL provide over an SFT-only model? How does CS-GRPO compare with standard GRPO under the same training conditions?
-
What happens when the SFT demonstrations retain the reference cohorts and adjustment values but omit the medical explanations?
4 comments
W3/Q3
We thank the reviewer for the suggested ablation and the rule-based baseline.
-
Regarding the ablation of the adjustment step, the adjustment improves both AUROC and MAE for both base LMs on CohortCal as listed in Table F3.
-
We define the rule-based baseline as a retrieval baseline by dropping history features up to the first existing cohort in CohortWorld. In Table F3, the complete version of Qwen3.6-27B with δ outperforms the rule-based baseline and all others. The smaller LM with 9B parameters shows no significant difference from the rule-based baseline due to a higher CC MAE and a lower CC accuracy as listed in Table F2. We will add this ablation study to the Appendix.
| Table F3. w/ CW | AUROC | MAE |
| Rule-based baseline | 0.625 | 0.057 |
| Qwen3.5-9B w/o δ | 0.569 | 0.069 |
| Qwen3.5-9B w/ δ | 0.618 | 0.062 |
| Qwen3.6-27B w/o δ | 0.551 | 0.063 |
| Qwen3.6-27B w/ δ | 0.668 | 0.049 |
W4
We thank the reviewer for the remark on Fig. 1. We will revise Fig. 1 to show the evaluated task. The patient in Fig. 1a will ask for the risk of a first recorded diagnosis of gastritis within five years as in the setting of Section 3.2.
W5/Q4
We thank the reviewer for the suggested ablation studies of the SFT-only and standard GRPO versions. We directly use the model after SFT and before CS-GRPO as the SFT-only version. The standard GRPO version is trained from the model after SFT. They are compared on all measurements reported in Table 3 and Table 4 of our paper. In Table F4 (using Qwen3.5-9B), RL improves all measurements except for the CC accuracy over the SFT-only model. The standard GRPO gives a lower CohortThink AUROC than CS-GRPO due to the low cohort matching accuracy.
| Table F4. | CM acc | CC acc | CC MAE | CohortThink | CohortCal |
| w/ CW | 29.6% | 51.6% | 0.122 | 0.516 [0.479, 0.553] | 0.618 [0.611, 0.625] |
| SFT only | 49.2% | 36.3% | 0.049 | 0.520 [0.483, 0.559] | 0.476 [0.470, 0.483] |
| GRPO | 22.0% | 55.6% | 0.057 | 0.521 [0.484, 0.557] | 0.635 [0.629, 0.642] |
| CS-GRPO | 80.0% | 16.3% | 0.040 | 0.638 [0.605, 0.670] | 0.663 [0.657, 0.670] |
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new retrieval baselines, adjustment ablation, and SFT/GRPO comparisons. These results strengthen the empirical evaluation and address several of my original concerns.
However, two important concerns remain.
1. Reliability of the medical reasoning. The authors acknowledge that the MedGemma-generated explanations were not clinically validated. While the reported sign consistency is useful, it does not establish clinical correctness. The CoT ablation also does not demonstrate that the generated explanations reflect medically sound reasoning. Given the paper's emphasis on trustworthy, clinician-like reasoning, I believe this remains a significant limitation.
2. Substantial degradation in risk-difference reasoning. The additional MAE results help explain the benefits of the adjustment reward. However, the dramatic decline in CC accuracy (from 51.6% to 16.3% for 9B and from 61.1% to 18.1% for 27B) remains concerning. A lower adjustment MAE does not necessarily imply more reliable reasoning about the direction of risk changes, particularly when the underlying risk differences are small. The response explains why the reward does not directly optimize CC, but does not fully address why the degradation is so substantial.
Overall, the additional experiments improve my assessment of the empirical contributions, especially the benefits of CS-GRPO and numerical adjustment. Nevertheless, I remain unconvinced by the stronger claims regarding trustworthy medical reasoning. I therefore maintain my current rating.
We thank the reviewer for the further comments and for the positive assessment of the new experiments. However, we respectfully disagree with the remaining concern about medical explanations given below reasons.
- Our paper makes no claim about the clinical correctness of generated medical text. The three contributions in Lines 121 to 134 on page 3 are the CohortWorld environment with CS-GRPO, the cohort collection method, and the CohortCal benchmark of disease risk prediction over diverse LMs. In fact, the word “explanation” does not occur in our paper.
- The MedGemma-generated text serves only as the CoT for the cold start of SFT (Lines 272 to 274 on page 6). No reward term or metric measures its medical content. Every metric in our paper measures the predicted disease risk against the observed incidence of real-world cohorts, rather than the explanation.
- The trustworthiness in our paper refers to the verifiable RWE. The terms “doctor-like” and “reason like a human doctor” refer to the reasoning path in the hypothesis of Section 1, i.e., matching a real cohort before the reasoning on the target disease. We do not refer to the clinical correctness of the generated text.
We thank the reviewer again for the detailed review and the discussion with us. We kindly invite the reviewer to reconsider the score in light of these reasons. We are happy to answer any further questions before the end of the discussion period.
W1
We thank the reviewer for the close reading of the medical explanation in our SFT stage. However, our paper benchmarks the disease risk prediction of LMs rather than plausible medical explanations. As a middle step towards disease risk prediction, the generated explanation serves as the Chain-of-Thought (CoT) of CS-GRPO (see Lines 272 to 274 on page 6), which is the same as the standard CoT with no label. In Table F1, removing the CoT tag <cohort-think> shows no significant decline in AUROC in a paired bootstrap due to the difficulty of risk-difference reasoning. The power of this CoT tag comes mainly from the numerical adjustment as listed in Table F3. We will revise Section 3.3 to explicitly state that the MedGemma-generated text has no clinical review.
Q1
The explanation generated by MedGemma has no clinical label. Therefore, we cannot measure the clinical correctness. Instead, we can measure its consistency with the adjustment δ. Table F1a lists that the prescribed adjustment agrees in sign with the direction stated in the generated explanation in over 97% of data points. This consistency is measured by comparing the last three direction terms (lower / higher / same) in the CoT and the adjustment number.
W6/Q5
We thank the reviewer for the suggested ablation that retains the adjustment number but removes the CoT content. We have re-trained CS-GRPO for Qwen3.5-9B without CoT (re-trained w/o CoT). In Table F1b, removing the generated explanations from CoT reduces AUROC after re-training. The performance drop in the ablated version is caused by the lower CM accuracy and the higher CC mean absolute error (CC MAE) due to the absence of reasoning for the adjustment number.
W2/Q2
We thank the reviewer for the close reading of the CC results in Table 4. Among the three CohortWorld tasks, CT is the only target task of RL post-training (Lines 288 to 289 on page 6 of our paper). We therefore do not expect RL post-training to raise the CC accuracy. The decline in CC after post-training is already discussed in Lines 423 to 424 of our paper, which shows a trade-off between CM and CC for a better disease risk prediction.
The reason for the trade-off: The reasoning on risk adjustment is hard even for physicians. Practitioners misjudge the change of disease probability after a test result for several common tests [1]. GPT-4 shows the same difficulty after a positive test result [2]. This difficulty is consistent with the small effect of adjustment reward (ψ) on CC in Table 4.
The reason that the adjustment reward (ψ) fails to recover CC: The ψ in Eq. (4) targets the error of δ against the true difference \hat{r} − r rather than the CC accuracy. CC in Eq. (6) measures the accuracy of the adjustment direction, i.e., whether the ground truth and δ have the same sign. Table F2 reports the error of δ as the mean absolute error (MAE) of the adjustment besides the CC accuracy, where ψ lowers the CC MAE for Qwen3.6-27B in a paired bootstrap. We will add this CC MAE column to Table 4.
[1] Morgan, Daniel J., et al. "Accuracy of practitioner estimates of probability of diagnosis before and after testing." JAMA internal medicine 181.6 (2021): 747-755.
[2] Rodman, Adam, et al. "Artificial intelligence vs clinician performance in estimating probabilities of diagnoses before and after testing." JAMA Network Open 6.12 (2023): e2347075.