acceptodds
Under review as a conference paper at ICLR 2027

CohortWorld: Aligning Real-World Data-Driven Diagnostics and Language Models

Abstract

Medical language models (LMs) have shown great potential in complex real-world healthcare tasks since they can act as agents in interactive virtual clinical environments with verifiable rewards as trustworthy feedback. However, the accuracy of data-driven diagnostics in clinic practice is challenged by label quality due to restrictive data use agreements, synthetic yet access-gated health environment, and overly narrow health context, which leads to the absence of real-world evidence (RWE) verifying LMs to avoid hallucination in medical LMs. To bridge this gap, we present CohortWorld, an in-silico environment generated from large-scale UK Biobank (UKB) cohort in which each state is a real-world cohort anchored in disjoint histories of disease, procedure, and medication. CohortWorld is an reinforcement learning (RL) suite based on disjoint anonymous cohorts (=29,884) with real-world data gathered from 500k UKB subjects for evaluating LMs on disease diagnosis and simulating RWE reasoning as a human doctor. Our empirical testing demonstrates that existing closed- and open-source LMs severely underperform on the RWE-calibrated benchmark, hovering at or below an AUROC of 0.5 with suboptimal slope metrics for disease risk ranking. In contrast, Qwen3.5-9B outperforms Sonnet 5 using our CohortWorld framework, demonstrating the benefits of RWE matching and counterfactual risk modification. Together, CohortWorld advances the trustworthiness of artificial diagnosis and enables LMs to reason like a human doctor based on verifiable RWE in a releasable and actionable in-silico environment.

Primary area:
datasets and benchmarks
Keywords:
world models, real-world evidence, electronic health records, clinical risk prediction, benchmarking, counterfactual reasoning, LLM agents, reinforcement learning, UK Biobank

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_feJBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CohortWorld, an interactive environment built from UK Biobank records for disease-risk prediction. It provides 29,884 cohorts derived from over 500,000 participants and introduces CohortCal to evaluate model predictions against observed incidence rates. It also proposes CS-GRPO to train models to select comparison cohorts, query their outcomes, and estimate risk adjustments. Experiments evaluate general-purpose and medical language models and report improvements in risk discrimination and calibration with CohortWorld.

Critical strengths
  1. The paper builds an agentic environment from over 500,000 UK Biobank participants. The query environment lets models use observed outcomes without releasing individual records. This is a useful resource for studying evidence-based risk prediction.
  2. Post-training improves risk discrimination and reduces tool use. For Qwen3.5-9B, CT AUROC increases from 0.516 to 0.638. Average tool calls decrease from 12.0 to 3.3. These results show that training helps agents use the environment more effectively.
Critical weaknesses
  1. The medical explanations used for training are generated by MedGemma, but no independent clinical validation is reported. The numerical adjustment can be computed from observed cohort rates. This does not verify the explanation for that difference. In particular, an association between a history feature and disease incidence does not establish the causal account given in the explanation. It is unclear whether these demonstrations teach reliable medical reasoning.
  2. Table 4 shows a substantial decline in CC after post-training. For Qwen3.5-9B, accuracy falls from 51.6% to 16.0% and reaches 16.3% with the adjustment reward. For Qwen3.6-27B, it falls from 61.1% to 20.5%, then to 18.1%. Section 4.1 interprets this as a trade-off between matching and contrast. However, it remains unclear why the adjustment reward does not recover CC performance, or how these results support improved risk-adjustment reasoning.
  3. Table 3 shows that agents with access to cohort incidence rates outperform models answering from medical history alone. However, this comparison changes both data access and the prediction procedure. An ablation study should remove the adjustment step while keeping the agent-selected cohorts fixed, returning their observed incidence rates directly. Comparing this variant with the full method, including absolute prediction error, would show whether adjustment improves prediction. A simple rule-based cohort-matching baseline would also help assess the value of LLM-based cohort selection.
  4. Figure 1 illustrates a patient asking whether existing symptoms will worsen. The task in Section 3 predicts a first recorded diagnosis within a fixed time horizon. The figure suggests a broader clinical task than the experiments evaluate and should be revised to match the actual setting.
  5. The training gains need further ablation. Table 4 should include an SFT-only model to separate the effects of SFT and RL. Comparing CS-GRPO with standard GRPO under the same training conditions would also help assess the value of the cross-sample reward in Section 3.3.
  6. The contribution of the generated explanations is not clear. An SFT ablation that retains the reference cohort and adjustment value but removes the medical explanation would show whether the explanation itself helps.
Questions for authors
  1. How were the generated medical explanations checked for clinical correctness and consistency with the prescribed adjustment?

  2. Why does CC decline for both models after post-training? Why does adding the adjustment reward fail to recover this performance?

  3. How does the method compare with simple rule-based cohort matching? For the same agent-selected cohorts, does applying the adjustment improve on directly returning the queried incidence rate?

  4. How much improvement does RL provide over an SFT-only model? How does CS-GRPO compare with standard GRPO under the same training conditions?

  5. What happens when the SFT demonstrations retain the reference cohorts and adjustment values but omit the medical explanations?

4 comments

W1

We thank the reviewer for the close reading of the medical explanation in our SFT stage. However, our paper benchmarks the disease risk prediction of LMs rather than plausible medical explanations. As a middle step towards disease risk prediction, the generated explanation serves as the Chain-of-Thought (CoT) of CS-GRPO (see Lines 272 to 274 on page 6), which is the same as the standard CoT with no label. In Table F1, removing the CoT tag <cohort-think> shows no significant decline in AUROC in a paired bootstrap due to the difficulty of risk-difference reasoning. The power of this CoT tag comes mainly from the numerical adjustment as listed in Table F3. We will revise Section 3.3 to explicitly state that the MedGemma-generated text has no clinical review.

Table F1. AUROCw/o <cohort-think>w/ <cohort-think>
Qwen3.5-9B0.623 [0.589, 0.655]0.638 [0.605, 0.670]
Qwen3.6-27B0.626 [0.593, 0.661]0.627 [0.594, 0.660]

Q1

The explanation generated by MedGemma has no clinical label. Therefore, we cannot measure the clinical correctness. Instead, we can measure its consistency with the adjustment δ. Table F1a lists that the prescribed adjustment agrees in sign with the direction stated in the generated explanation in over 97% of data points. This consistency is measured by comparing the last three direction terms (lower / higher / same) in the CoT and the adjustment number. 

Table F1a.Qwen3.5-9BQwen3.6-27B
% sign agrees with text97.6%97.7%

W6/Q5

We thank the reviewer for the suggested ablation that retains the adjustment number but removes the CoT content. We have re-trained CS-GRPO for Qwen3.5-9B without CoT (re-trained w/o CoT). In Table F1b, removing the generated explanations from CoT reduces AUROC after re-training. The performance drop in the ablated version is caused by the lower CM accuracy and the higher CC mean absolute error (CC MAE) due to the absence of reasoning for the adjustment number.

Table F1b. Qwen3.5-9B CM acc.CC acc.CC MAECohortThinkCohortCal
Complete version76.1%19.1%0.0410.638 [0.605, 0.670] 0.663 [0.657, 0.670] 
re-trained w/o CoT75.5% 21.4% 0.0500.632 [0.598, 0.663] 0.645 [0.638, 0.651] 

W2/Q2

We thank the reviewer for the close reading of the CC results in Table 4. Among the three CohortWorld tasks, CT is the only target task of RL post-training (Lines 288 to 289 on page 6 of our paper). We therefore do not expect RL post-training to raise the CC accuracy. The decline in CC after post-training is already discussed in Lines 423 to 424 of our paper, which shows a trade-off between CM and CC for a better disease risk prediction.

  1. The reason for the trade-off: The reasoning on risk adjustment is hard even for physicians. Practitioners misjudge the change of disease probability after a test result for several common tests [1]. GPT-4 shows the same difficulty after a positive test result [2]. This difficulty is consistent with the small effect of adjustment reward (ψ) on CC in Table 4.

  2. The reason that the adjustment reward (ψ) fails to recover CC: The ψ in Eq. (4) targets the error of δ against the true difference \hat{r} − r rather than the CC accuracy. CC in Eq. (6) measures the accuracy of the adjustment direction, i.e., whether the ground truth and δ have the same sign. Table F2 reports the error of δ as the mean absolute error (MAE) of the adjustment besides the CC accuracy, where ψ lowers the CC MAE for Qwen3.6-27B in a paired bootstrap. We will add this CC MAE column to Table 4.

[1] Morgan, Daniel J., et al. "Accuracy of practitioner estimates of probability of diagnosis before and after testing." JAMA internal medicine 181.6 (2021): 747-755.

[2] Rodman, Adam, et al. "Artificial intelligence vs clinician performance in estimating probabilities of diagnoses before and after testing." JAMA Network Open 6.12 (2023): e2347075.

Table F2. CC MAE (error)CC acc. (direction)
Qwen3.5-9B0.059 [0.055, 0.062] 59.8%
w/ CW0.122 [0.111, 0.134] 51.6%
+ CS-GRPO0.045 [0.042, 0.047]16.0%
+ ψ0.042 [0.039, 0.045]16.3%
Qwen3.6-27B0.052 [0.049, 0.055] 67.2%
w/ CW0.057 [0.054, 0.061] 61.1%
+ CS-GRPO0.041 [0.038, 0.044]20.5%
+ ψ0.040 [0.038, 0.043]18.1%

W3/Q3

We thank the reviewer for the suggested ablation and the rule-based baseline. 

  1. Regarding the ablation of the adjustment step, the adjustment improves both AUROC and MAE for both base LMs on CohortCal as listed in Table F3.

  2. We define the rule-based baseline as a retrieval baseline by dropping history features up to the first existing cohort in CohortWorld. In Table F3, the complete version of Qwen3.6-27B with δ outperforms the rule-based baseline and all others. The smaller LM with 9B parameters shows no significant difference from the rule-based baseline due to a higher CC MAE and a lower CC accuracy as listed in Table F2. We will add this ablation study to the Appendix.

Table F3. w/ CWAUROCMAE
Rule-based baseline0.6250.057
Qwen3.5-9B w/o δ0.5690.069
Qwen3.5-9B w/ δ0.6180.062
Qwen3.6-27B w/o δ0.5510.063
Qwen3.6-27B w/ δ0.6680.049

W4

We thank the reviewer for the remark on Fig. 1. We will revise Fig. 1 to show the evaluated task. The patient in Fig. 1a will ask for the risk of a first recorded diagnosis of gastritis within five years as in the setting of Section 3.2.

W5/Q4

We thank the reviewer for the suggested ablation studies of the SFT-only and standard GRPO versions. We directly use the model after SFT and before CS-GRPO as the SFT-only version. The standard GRPO version is trained from the model after SFT. They are compared on all measurements reported in Table 3 and Table 4 of our paper. In Table F4 (using Qwen3.5-9B), RL improves all measurements except for the CC accuracy over the SFT-only model. The standard GRPO gives a lower CohortThink AUROC than CS-GRPO due to the low cohort matching accuracy.

Table F4. CM accCC accCC MAECohortThinkCohortCal
w/ CW29.6%51.6%0.1220.516 [0.479, 0.553]0.618 [0.611, 0.625] 
SFT only49.2% 36.3% 0.0490.520 [0.483, 0.559] 0.476 [0.470, 0.483] 
GRPO22.0% 55.6% 0.0570.521 [0.484, 0.557]0.635 [0.629, 0.642] 
CS-GRPO80.0%16.3%0.0400.638 [0.605, 0.670] 0.663 [0.657, 0.670]  

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new retrieval baselines, adjustment ablation, and SFT/GRPO comparisons. These results strengthen the empirical evaluation and address several of my original concerns.

However, two important concerns remain.

1. Reliability of the medical reasoning. The authors acknowledge that the MedGemma-generated explanations were not clinically validated. While the reported sign consistency is useful, it does not establish clinical correctness. The CoT ablation also does not demonstrate that the generated explanations reflect medically sound reasoning. Given the paper's emphasis on trustworthy, clinician-like reasoning, I believe this remains a significant limitation.

2. Substantial degradation in risk-difference reasoning. The additional MAE results help explain the benefits of the adjustment reward. However, the dramatic decline in CC accuracy (from 51.6% to 16.3% for 9B and from 61.1% to 18.1% for 27B) remains concerning. A lower adjustment MAE does not necessarily imply more reliable reasoning about the direction of risk changes, particularly when the underlying risk differences are small. The response explains why the reward does not directly optimize CC, but does not fully address why the degradation is so substantial.

Overall, the additional experiments improve my assessment of the empirical contributions, especially the benefits of CS-GRPO and numerical adjustment. Nevertheless, I remain unconvinced by the stronger claims regarding trustworthy medical reasoning. I therefore maintain my current rating.

We thank the reviewer for the further comments and for the positive assessment of the new experiments. However, we respectfully disagree with the remaining concern about medical explanations given below reasons.

  1. Our paper makes no claim about the clinical correctness of generated medical text. The three contributions in Lines 121 to 134 on page 3 are the CohortWorld environment with CS-GRPO, the cohort collection method, and the CohortCal benchmark of disease risk prediction over diverse LMs. In fact, the word “explanation” does not occur in our paper.
  2. The MedGemma-generated text serves only as the CoT for the cold start of SFT (Lines 272 to 274 on page 6). No reward term or metric measures its medical content. Every metric in our paper measures the predicted disease risk against the observed incidence of real-world cohorts, rather than the explanation.
  3. The trustworthiness in our paper refers to the verifiable RWE. The terms “doctor-like” and “reason like a human doctor” refer to the reasoning path in the hypothesis of Section 1, i.e., matching a real cohort before the reasoning on the target disease. We do not refer to the clinical correctness of the generated text.

We thank the reviewer again for the detailed review and the discussion with us. We kindly invite the reviewer to reconsider the score in light of these reasons. We are happy to answer any further questions before the end of the discussion period.

Reviewer_UdnCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CohortWorld, an interactive environment grounded in real-world UK Biobank data, and the CohortCal benchmark for evaluating LLMs' ability to predict disease risks from medical histories. The authors also propose a reinforcement learning approach (CS-GRPO) that encourages evidence-grounded reasoning and improves cohort matching and overall disease-risk prediction.

Critical strengths
  • The paper introduces a novel interactive environment grounded in real-world clinical data.

  • The proposed framework demonstrates promising improvements in cohort matching and overall disease-risk prediction.

Critical weaknesses

[W1]: The authors ensure CohortWorld and CohortCal contain different cohorts, but they do not quantify how many underlying UK Biobank participants contribute to both. Because the method constructs multiple pseudo-subjects from longitudinal histories, cohort disjointness alone does not establish subject-level independence.

[W2]:The proposed reward function improves cohort matching and overall risk prediction but substantially degrades CohortContrast accuracy (Table 4), raising concerns about whether it encourages accurate risk-difference reasoning. The authors could provide ablation studies of the individual reward components to better understand the reward function components contribution.

[W3]:The experimental analysis is relatively limited. Additional analyses could investigate how fine-tuning affects performance on other medical tasks and preserves existing medical LLM capabilities, and stratify results by disease and prediction horizon, or perform a qualitative error analysis to better understand the models' strengths and limitations. (or the ablation study I mentioned in W2).

[W4]: Authors report that CohortCal contains train, seen and unseen splits, yet the reported leaderboard only uses the unseen split. Can the authors explain the purpose of the other splits?

Questions for authors

Please refer to the weaknesses

4 comments

W1

We thank the reviewer for the question on subject-level independence. The cohorts in the two sets share a small percentage of subject overlap, i.e., 13.4% of CohortCal subjects are also members of CohortWorld cohorts. Therefore, this results in a small effect on the benchmark as listed in Table 3 of our paper. To show subject-level independent results, we reproduced Table 3 in Table U1. The results repeat the same pattern as Table 3 on the unseen split, indicating that the gain of the CohortWorld agent does not depend on these shared subjects. This new table will be added to the Appendix.

Table U1 AUROCbeats bslope
Qwen3.5-2B0.503 [0.492, 0.514]0.2650.043
Qwen3.5-4B0.526 [0.516, 0.537]0.2690.044
Qwen3.5-9B0.539 [0.528, 0.549]0.2990.198
Qwen3.6-27B0.542 [0.532, 0.552]0.2590.194
Gemma 3 4B0.488 [0.478, 0.500]0.2740.013
Gemma 3 27B0.561 [0.551, 0.572]0.3060.186
MedGemma 4B0.498 [0.487, 0.509]0.2630.009
MedGemma 27B0.579 [0.568, 0.589]0.2980.204
Llama 3 8B0.545 [0.535, 0.555]0.3100.095
Mistral Small 24B0.585 [0.574, 0.595]0.2990.113
Claude Sonnet 50.575 [0.565, 0.585]0.3100.315
Gemini 3.5 Flash-Lite0.561 [0.550, 0.572]0.2530.553
Qwen3.5-9B w/ CW0.645 [0.635, 0.656]0.3970.408
Qwen3.6-27B w/ CW0.704 [0.694, 0.713]0.4310.388

W2

We thank the reviewer for the suggestion of ablation studies of the individual reward components. Table U2 ablates each term of Eq. (3) using Qwen3.5-9B as the base model. The complete version uses all reward terms. Each of the ablated versions removes one item at a time. In Table U2, the invocation reward λ drives the gain in AUROC. Without λ, the policy calls no tool in CohortWorld, resulting in the worst AUROC. Removing φ or g gives no significant change in AUROC against the complete version in a paired bootstrap. 

Regarding the degraded CC accuracy, which measures whether δ has the same sign as the ground truth, the reason is that the reward function Eq. (4) targets the error of δ rather than the sign consistency measured by CC accuracy. Therefore, CC accuracy is not boosted by the post-training. We report the error of δ as the mean absolute error (MAE) of the adjustment alongside the CC accuracy in Table U2, where the complete version has the lowest MAE.

Table U2CM acc.CC acc.CC MAECT AUROC
Complete version80.0%16.3%0.040 [0.038, 0.043] 0.638 [0.605, 0.670]
Without ψ76.1%19.1%0.047 [0.045, 0.050] 0.621 [0.588, 0.654]
Without φ74.6%20.4%0.059 [0.055, 0.062] 0.606 [0.574, 0.638]
Without λ0.0%68.6%0.433 [0.429, 0.437] 0.294 [0.260, 0.328]
Without g76.9%18.6%0.059 [0.055, 0.063] 0.627 [0.594, 0.660]

W3

We thank the reviewer for the suggested analyses. We have performed the ablation study you mentioned in W2 in Table U2.

  1. Fine-tuning preserves the medical knowledge of both base models on four medical QA benchmarks scored by the option with the highest probability. In Table U3a, no difference between a fine-tuned checkpoint and its base model survives a Bonferroni correction. We will add this Table to the Appendix.
Table U3a. AccuracyMedQAMedMCQAPubMedQAMMLU-med.
Qwen3.5-9B, base0.7380.6580.7760.855
Qwen3.5-9B, CS-GRPO0.7360.6550.7780.862
Qwen3.6-27B, base0.8320.7040.7990.900
Qwen3.6-27B, CS-GRPO0.8430.7110.7870.901
  1. In Table U3b, we stratify results by disease and horizon. Every method has a lower AUROC at 10 years than at 1 year on CohortCal. Using CohortWorld raises AUROC over Claude Sonnet 5 most on eye and ear disorders and least on digestive disorders. We will include every horizon and chapter in the Appendix.
Table U3b. CohortCal AUROC1 y10 yEye, earDigestive
Qwen3.5-9B0.6320.5180.5750.523
Qwen3.5-9B w/ CW0.6630.6060.7380.552
Qwen3.6-27B0.6090.5010.4890.502
Qwen3.6-27B w/ CW0.7420.6440.8230.557
Claude Sonnet 50.6790.5620.5540.522
Claude Sonnet 5.50.6600.5420.5850.494
Claude Opus 5.50.7130.6460.6790.582
  1. Regarding the qualitative error analysis, our paper has already performed this analysis by stratifying by the cohort size in Fig. 4, where single-turn LMs give a nearly flat prediction over the ground truth r and therefore over-predict low risks. We conduct the same analysis on the horizons and diseases in Fig. U3, where the single-turn LMs show the same flat over-prediction at all five horizons and in all disease chapters with a bin of at least 30 items. We will add Fig. U3 to the Appendix. 

Fig. U3. Predicted r^\hat{r} against ground truth r by horizon h and disease on CohortCal. Grey bars show the percentage of items in each bin. URL: https://anonymous.4open.science/r/cohort-world-E44D/data/fig_u3_y-vs-rhat_horizons-and-chapters.png

W4

We thank the reviewer for the question on the purpose of the other splits. The train and seen data splits serve supervised learning methods on CohortCal. The leaderboard reports scores on the unseen data split. Since we focused on benchmarking LMs in our paper, the train and seen splits remained unused. Here, we test a plain method on the subject-level independent CohortCal in Table U4. We intuitively put random items from the train split of CohortCal into the prompt for Qwen3.6-27B when testing the unseen items so that the LM uses in-context learning (ICL) with the train split. The comparison between the base LM, ICL, and CW shows that ICL is less effective than the proposed CW harness. We will state the purpose of each split in Section 3, and add this Table to the Appendix.

U4AUROC
Qwen3.6-27B base0.542 [0.532, 0.552]
Qwen3.6-27B ICL0.653 [0.643, 0.662]
Qwen3.6-27B w/ CW0.704 [0.694, 0.713] 

Dear Reviewer UdnC,

We thank the reviewer again for the constructive review. We would like to kindly ask whether our response addresses the four weaknesses in the review. We kindly invite the reviewer to update the score given these new results. We are happy to answer any further questions before the end of the discussion period. Thank you very much.

Dear Reviewer UdnC,

As the discussion period is coming to a close, we would like to kindly follow up on our responses, in which we addressed each of your comments with additional experiments. We would greatly appreciate your feedback. Thank you again for your time and valuable comments.

Best regards, The Authors

Reviewer_wySoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies disease risk estimation from coded medical history. The authors introduce CohortWorld, an RL environment whose states are de-identified UK Biobank cohorts labeled with observed first-diagnosis incidence. The agent moves between neighboring cohorts, observes a cohort's rate, and commits an estimate. Training applies CS-GRPO after an SFT cold start, with a reward for format, tool use, risk-difference accuracy, and a "cross-sample" contribution to batch AUROC and calibration slope. The authors evaluate on the CohortCal benchmark with single-turn open and closed models and two tool-using Qwen agents, and on CohortWorld tasks with the same models. The reported results show that single-turn models sit at or slightly above chance, CohortWorld tools raise AUROC, and CS-GRPO raises cohort-matching accuracy while cohort-contrast accuracy falls.

Critical strengths
  • A releasable source of real-world reward. Turning a credentialed biobank into a public collection of aggregated cohort-level outcomes is a useful idea. If the release is permitted and disclosure-safe, it gives medical-LM work a verifiable reward grounded in observed population outcomes rather than simulators or exam answer keys. This contribution survives even if the training-method claims fall, and it is the main reason the paper is not a clear reject.

  • Concrete evidence that single-turn LMs are poorly calibrated on population disease risk. Table 3 and Fig. 4 show ranking close to chance, and below it for some small models, together with flat predicted-vs-true curves, consistently across the single-turn models and many cohort sizes. The target is UK Biobank recorded incidence rather than general-population incidence, which qualifies how clean the result is. Subject to the baseline concerns below, this is a useful negative result, and its weight in my overall assessment is conditional on those concerns.

Critical weaknesses
  • The gains measure retrieval of a neighboring cohort's rate, not improved reasoning, and the key baselines are missing. The tool-using agent reads observed incidence from disjoint cohorts drawn from the same population, coding scheme, and follow-up as the ground truth, while single-turn models answer from memory under a prompt that forbids any explanation, with the closed models at low or minimal thinking effort. After CS-GRPO, matching accuracy jumps and contrast accuracy collapses, and in every shown trajectory (Fig. 5, A.2) the agent calls commit with no rates, so the scored answer equals the last rate it read and the stated δ is never applied. The ψ term does not change this consistently. The claims that LMs "reason like a human doctor" and Qwen-9B "outperforms Sonnet 5" rest on this comparison and remain unsupported until age/sex-stratified prevalence and a drop-features-until-published-rate heuristic are reported, which alone drives my Soundness score.
  • The reasoning being trained is decorative, and some of it is medically unsound. The SFT demonstrations compute the comparison and sign of δ from the data, and MedGemma-27B writes a mechanism for that predetermined direction. Some A.1 rationales are clinically dubious, e.g., treating infections protecting against nervous-system disease and ENT medication "nearly tripling" musculoskeletal risk. The model copies this disconnect: A.2 states a "substantially lower chance … by several hundred cases per thousand" from a reference rate that would make such a reduction negative, then writes a near-zero positive adjustment, and in Fig. 5 it argues the risk is higher yet writes a positive δ, which by the paper's convention means lower. This is consequential because the paper is framed around trustworthiness and reducing hallucination, and training fluent mechanistic justifications that are decoupled from the numeric output works against that stated goal.
Questions for authors
  • Committed versus read rates. How often does the committed answer differ from the last observed rate, and how often does the sign of the written δ agree with the direction stated in the text? If δ is usually applied and sign-consistent, I would consider raising my score.

  • Retrieval baselines. How do age/sex-stratified prevalence and a rule-based baseline that coarsens features until a published rate exists perform on CohortCal and CohortThink? If CohortWorld agents clearly outperform these baselines, I would consider raising my score; otherwise the central claim falls.

  • Disease labels versus scored codes. The tool output in A.2 shows "skin disorder" corresponds to L11, and Fig. 5 uses "musculoskeletal disorder [M0]" and "infectious disease [A4]", so what code set does each disease label cover? If the labels shown to models are substantially broader than the scored codes, single-turn over-prediction in Fig. 4 may reflect label mismatch rather than miscalibration, and I would lower my score.

  • Variance. Are the CS-GRPO results from a single seed, since ψ ablation differences sit within or near the bootstrap intervals? If gains hold across seeds, I would consider raising my score.

3 comments

W1/Q2

We thank the reviewer for the careful reading of our paper. 

1. We first extended our benchmark on the thinking effort of the single-turn models in Table R1a to show that multi-turn doesn't help on this benchmark. Qwen3.5-9B with 32k thinking tokens and Claude Haiku 5.5 at higher effort rank worse in AUROC than at their lowest setting.

Table R1a. CohortCalAUROC
Qwen3.5-9B, thinking off0.541 [0.534, 0.549]
Qwen3.5-9B, thinking, 32k tokens0.522 [0.515, 0.530]
Claude Haiku 5.5, low effort0.547 [0.541, 0.554]
Claude Haiku 5.5, medium effort0.542 [0.536, 0.549]
Claude Haiku 5.5, high effort0.542 [0.535, 0.549]
Claude Haiku 5.5, max effort0.527 [0.520, 0.534]
Claude Opus 5.5, low effort0.651 [0.644, 0.657]
Qwen3.6-27B w/ CW0.668 [0.662, 0.675] 

2. We agree with this reviewer on the larger impact of cohort matching (CM) than of the ψ term due to the trade-off between matching and adjustment prediction. This is already discussed in Lines 423-424 of our paper. Nevertheless, using δ is more accurate than the retrieval of a cohort's rate. Table R1b reports a higher AUROC for the answers with δ than for the retrieved disease risk without δ. We will add this ablation study to the Appendix in the final version.

Table R1b. AUROCQwen3.5-9B w/ CWQwen3.6-27B w/ CW
w/ δ0.6180.668
w/o δ0.5690.551

3. We have added both retrieval baselines to Table R1c, where the CW agent exceeds both baselines on CohortCal and CohortThink.

Table R1c. AUROCCohortCalCohortThink
Baseline 1 (age-stratified)0.6510.623 
Baseline 2 (sex-stratified)0.5650.634
Baseline 3 (coarsen features)0.6250.640
CW agent (Qwen3.6-27B)0.6680.642

Together, these new results support that the AUROC gain of the CW agent on CohortCal comes from the adjustment δ on top of cohort matching, not from the retrieval of a cohort’s rate alone.

W2

We thank the reviewer for the close reading of our CoT examples. We respectfully disagree with the two cited examples. The A.2 example is consistent with the prediction r^\hat{r} = max(r − δ, 0) defined in Lines 280 to 281 on page 6. Under this definition, a reduction beyond the reference rate gives r^\hat{r} = 0 rather than a negative risk. Its positive δ also matches the stated “lower chance”. In Fig. 5, the "..." omits three sentences of the reasoning. The last omitted sentence attributes "muscle aches and joint pain to the cardiovascular medication of the comparison group", which supports the final argument of a lower risk for the patient without this medication. This is consistent with the positive δ. 

We apologize for the confusing "..." in Fig. 5. We will make it clear by showing the last omitted sentence. We agree with the reviewer on the demonstration being decorative, since MedGemma-27B received both rates and the direction for the generation of SFT demonstration without a clinical review. Section 3.3 will explicitly state these mechanisms.

Q1

The committed answer of the CohortWorld agents in Table 3 differs from the last observed rate on over 88% of items as listed in Table R2, indicating that δ is mostly applied. Moreover, the written δ agrees in sign with the direction stated in the text over 97% of data points. The percentage of sign agreement with the text is measured by comparing the last of the three direction terms, "lower / higher / same", in the CoT and the adjustment number.

Table R2. Qwen3.5-9B w/ CWQwen3.6-27B w/ CW
% answer ≠ observed88.5%94.6%
% sign agrees with text97.6%97.7%

Q3

We are sorry for the confused code set definition of CohortCal items shown in Fig. 5. The question of CohortCal to all LMs in our paper contains the same code set as the disease label, i.e., two-character codes as shown in Fig. 5. The code set for CohortCal item has been explicitly defined in Lines 926-927 in A.3, where γ=1st-depth. We will move this information to the main text. There is no mismatch between the labels shown to models and the scored codes. 

Q4

We thank the reviewer for the question on seeds. Every CS-GRPO row of our paper comes from seed 0. We evaluate two more seeds here. In Table R3, the 9B gain of CS-GRPO over 9B w/ CW holds across seeds on CohortThink. 

Table R3. AUROC gainSeed 0Seed 1Seed 2
Qwen3.5-9B w/ CS-GRPO+0.091 [0.043, 0.140]+0.095 [0.046, 0.141]+0.092 [0.045, 0.138]

Dear Reviewer wySo,

We thank the reviewer again for the detailed review. We would like to ask whether our response addresses the questions in the review.

We genuinely invite the reviewer to reconsider the score given our answers to your questions. We are happy to answer any further questions before the end of the discussion period. Thank you very much!

Dear Reviewer wySo,

As the discussion period is coming to a close, we would like to kindly follow up on our responses, in which we addressed each of your comments with additional experiments. We would greatly appreciate your feedback. Thank you again for your time and valuable comments.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.