acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning

Abstract

Clinical decision-making is a sequential process of information collection, clinical inference, and forecasting under partial observability, often without a uniquely correct treatment decision. Large language models (LLMs) have demonstrated strong medical question-answering capabilities but extending these advances to clinical reasoning remains challenging because many clinical tasks are only partially verifiable. Recent clinical reinforcement learning (RL) work has therefore largely focused on diagnostic endpoints, where correctness can be more directly verified. Oncology treatment decision making is a particularly challenging instance of this problem that extends beyond medical question-answering in which multiple decision trajectories can be clinically defensible for the same partially observed patient state. We formulate oncology treatment planning as an interactive RL problem under partial observability and construct patient-specific, process-aware rewards from decision processes observed in real world clinical records. The agent must learn to sequentially acquire information from a structured patient representation before committing to a first-line treatment recommendation, and is evaluated on information gathering, treatment selection, and reasoning alignment. Using GRPO and GDPO to optimize Llama-3.1-8B-Instruct within this environment, RL substantially improves the base policy beyond frontier model capabilities. The GDPO-trained policy increases Mean@3 scalar clinical reward by over base Llama and exceeds GPT-5.2 by under the same interactive environment, while reducing query count during training. Overall, our results show that patient-specific, partially verifiable rewards can provide an effective learning signal for interactive clinical RL when both the patient state and the underlying clinical decision process are only partially observed.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Clinical Decision-Making, Oncology Treatment Planning, Partial Observability, Partially Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KhDdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates colon cancer treatment planning as an interactive RL problem under partial observability and partial verifiability. It constructs a patient-specific environment where agents sequentially acquire clinical information before recommending treatment, with rewards evaluating information gathering, treatment selection, and reasoning alignment. Experiments show that GRPO and GDPO improve clinical decision-making performance, with the GDPO-trained Llama-3.1-8B outperforming GPT-5.2 under the proposed evaluation framework.

Critical strengths

Interesting problem formulation. The paper addresses an important challenge in clinical decision-making, where multiple treatment plans and reasoning trajectories can be clinically valid rather than having a single correct answer. This provides a meaningful setting for exploring reinforcement learning under partial verifiability.

Critical weaknesses
  1. Limited novelty and insufficient discussion of related work. Active information acquisition through RL is not specific to colon cancer treatment planning and has been explored in prior work such as ProMed (ACL 2026). The paper does not sufficiently distinguish its approach from existing methods or demonstrate the advantages of its reward design.

  2. Questionable validity of partially verifiable rewards. Although the paper argues that multiple treatment plans can be clinically valid, its reward is largely based on a single clinician's historical decision. It remains unclear whether the LLM judge can reliably recognize reasonable alternatives that differ from the ground truth, raising concerns about the validity of the reward and the reported improvements.

Questions for authors
  1. How does the proposed framework differ from existing RL-based active clinical information acquisition methods, such as ProMed (ACL 2026)?

  2. Why is a trajectory-level LLM-judged reward preferred over explicit information-gain-based rewards? Could the authors provide comparisons to demonstrate its advantages?

  3. How does the LLM judge distinguish clinically reasonable treatment alternatives from incorrect predictions when they differ from the clinician's ground truth? Has this capability been validated through independent clinical expert evaluation?

3 comments

Thank you for your review. We appreciate your recognition of the problem formulation and your questions on the reward design.

Q1.

We agree that active information acquisition through RL is not specific to oncology and has been explored in ProMed (Ding et al., ACL 2026). The difference between ProMed and our work is in the clinical decision problem and in how the reward is constructed.

ProMed studies medical question answering, where a ground-truth answer is available, and rewards each question by its Shapley information gain toward predicting the correct answer. Treatment planning, on the other hand, can have multiple acceptable treatment options for the same patient. Since the medical records contain only the realized treatment plan, we cannot observe the other acceptable plans or the decision processes that could lead to them, which makes both the treatment decision and the decision process partially verifiable. We therefore grade the whole trajectory against the realized plan and the documented decision process instead of each question. We do not claim novelty in the RL algorithm or in active information acquisition itself, but in the interactive environment and the patient-specific reward for treatment planning under partial verifiability. We will add ProMed and this distinction to the related work.

Q2.

We conduct the ablation study to compare the trajectory-level LLM-judged reward with rule-based rewards for information acquisition and regimen selection. The rule-based information acquisition reward is the F1 score between the factors the agent asks about and those in the physician's documented decision process, matched by keyword over 26 factors such as stage, biomarkers and labs. While the rule-based information acquisition reward is not directly comparable with the Shapley information gain proposed in ProMed, it measures how much of the documented decision process the agent covers.

The ablation study is set up as follows. The judge is Qwen3.8-27B-FP8 with a LoRA adapter fine-tuned on the past judging completions from GPT-5.2, and the mediator is Qwen3-1.7B with a LoRA adapter. We train Llama-3.1-8B-Instruct with GDPO for 300 steps under six reward settings with group size 8, 5 turns, the same patient order and seed 42. We evaluate the checkpoint at step 300 on the validation split with 143 patients with one trajectory per patient with greedy decoding.

Treatment rewardProcess rewardClinical scoreCompleteQuestions per caseDecision-factor recall
LLM-judged regimen–0.5240.97917.80.369
LLM-judged regimenLLM-judged info + reasoning0.5110.95825.40.459
LLM-judged regimen + rule-based drug matchLLM-judged info + reasoning + rule-based decision factors0.4670.85338.50.537
Rule-based drug matchRule-based decision factors0.4150.80441.40.547

Table 1. Reward ablation on the validation split (143 patients, one greedy trajectory per patient at step 300, scored by the fine-tuned judge). In the LLM-judged setting, the policy is rewarded by the fine-tuned judge; in the rule-based setting, the reward is computed by deterministic rules. Decision-factor recall is the share of the physician's decision factors the agent asked about. The full table can be found at our comment above.

In the rule-based reward setting for information acquisition, the agent exploits the reward by asking more questions. With the rule-based decision-factor reward alone, the agent asks about more of the physician’s decision factors (recall 0.547), but it asks 41.4 questions per case and completes only 80.4% of the cases. Imposing the LLM-judged reward along with the rule-based reward still leads to 38.5 questions per case and 85.3% completed cases. On the other hand, the LLM-judged information and reasoning rewards raise the decision-factor recall from 0.369 to 0.459 at 25.4 questions per case, compared to 17.8 without them, while the agent still completes 95.8% of the cases. In principle, an information-gain reward can also be measured against the documented decision process, and the rule-based decision-factor reward is our closest version of it. We could not run ProMed's Shapley information gain within the rebuttal period, since it estimates the gain of each question with repeated model calls, and we will add this comparison to the discussion.

(continue)

Q3.

An independent clinician assessment shows that the regimen scores from the GPT-5.2 judge agree with the clinician's grades of the treatment plans (Spearman's rank correlation coefficient 0.80). We draw 30 patients from the held-out test set. Patients the clinician had already seen in the earlier 34-case study were excluded. The clinician reviews the treatment plans from GDPO-Llama and GPT-5.2 for those patients in blinded fashion along with the full patient representation and the original treatment plan from the physician as the comparison point. Each treatment plan is graded with one of the following choices:

  • Same regimen,
  • Clinically equivalent,
  • Reasonable alternative,
  • Not appropriate, or
  • Cannot judge.

The clinician judges the choice of drugs and ignores dose, schedule and number of cycles. We then compare each grade with the regimen score that the GPT-5.2 judge gave the same rollout.

ModelSame regimenClinically equivalentReasonable alternativeNot appropriateCannot judgeAcceptable (95% CI)
GDPO-Llama14447122 of 29, 76% (58-88%)
GPT-5.284103522 of 25, 88% (70-96%)

Table 2. Clinician grades of the plans from GDPO-Llama and GPT-5.2 for 30 held-out test patients, rated blind against the treating oncologist's plan. Acceptable plans include same regimen, clinically equivalent or reasonable alternative, excluding the plans that cannot be judged. We report the 95% Wilson intervals.

The clinician rated 76% of GDPO-Llama's plans and 88% of GPT-5.2's plans acceptable, i.e., same regimen, clinically equivalent or reasonable alternative (95% intervals 58-88% and 70-96%). GDPO-Llama more often chose the physician's regimen (14 against 8 plans), while GPT-5.2 more often gave a reasonable alternative (10 against 4).

Clinician gradeNumber of plansMean judge score
Same regimen224.68
Clinically equivalent83.50
Reasonable alternative142.64
Not appropriate102.20
Cannot judge63.17
Spearman's rank correlation coefficient between clinician and GPT-5.2 judge-0.80
Agreement on same or equivalent, % (Cohen's κ\kappa)-76% (0.48)

Table 3. The agreement between the clinician assessment and the GPT-5.2 judge's scoring on the regimen accuracy on plans from GDPO-Llama and GPT-5.2 for 30 patients. The rubric gives 5 to an exact match with the regimen from the original treatment plan, 3-4 to a clinically equivalent regimen, 2 to a regimen of the right class that is not equivalent, and 0-1 to a wrong class. We report the Spearman's rank correlation coefficient between the judge's scores and the clinician's grades. For the agreement, a plan counts as same or equivalent for the judge when it scores 3-5, and for the clinician when it is graded same regimen or clinically equivalent. Plans the clinician cannot judge are excluded from the rank correlation and the agreement.

The judge's regimen score is concordant with the clinician's grade; the mean judge score is 4.68 for plans graded as the same regimen, 3.50 for clinically equivalent, 2.64 for reasonable alternative and 2.20 for not appropriate. Of the six plans the clinician could not judge, 4 are GPT-5.2 plans that list several conditional options without choosing a single one, and 2 are for one patient whose original plan was unclear.

For the plans that are graded not appropriate by the clinician, five of GDPO-Llama's 7 plans are rated not appropriate because they leave out radiation or surgery from a multimodal curative plan for rectal cancer or anal squamous cell carcinoma although the plans suggest the right chemotherapy. The missing modality is a limit of the agent and the rubric, which focuses on the accuracy of the regimen, since our evaluation targets the chemotherapy plan.

Thank you for your responses, I decide to maintain my original score.

Reviewer_1fw6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper frames first-line colon cancer treatment planning as an interactive reinforcement learning task under partial observability. An agent queries patient records within a turn budget before recommending a treatment. The authors construct patient-specific, process-aware rewards from retrospective clinical records and optimize the agent using GRPO and GDPO.

Core Claim: Patient-specific, partially verifiable rewards derived from real clinical records can train interactive agents with information gathering, clinical inference, and treatment decision capabilities.

Critical strengths
  1. Important and realistic problem setting: Clinical treatment decisions require active information acquisition and rarely have a single ground-truth answer. Modeling this as an interactive POMDP is a step in the right direction compared to static QA benchmarks.

  2. Decomposing information gathering, treatment, and reasoning: Factoring clinical decision-making into information gathering, regimen selection, and reasoning alignment is a logical setup that facilitates trajectory analysis.

Critical weaknesses
  1. "Partial verifiability" is primarily a motivation rather than an algorithmic contribution: The paper frames clinical tasks as decision problems with varying degrees of verifiability, but the method directly uses standard GRPO/GDPO algorithms and an LLM rubric judge. There is no new RL formulation designed to handle reward noise, partial verification, or supervision uncertainty. As a result, it is unclear what new RL principles are discovered beyond applying off-the-shelf policy optimization to a custom medical simulator.

  2. Experiment results do not substantiate the claim of clinical reasoning: The patient-specific reference information is extracted from retrospective clinical notes rather than direct observations of step-by-step expert information acquisition and decision-making processes. Reconstructed references reflect historical choices and documented rationales, but do not naturally equate to real expert information-seeking strategies. Furthermore, training and evaluation rely heavily on reference information and scoring rubrics constructed by an LLM. A higher reward indicates better adaptation to the specific environment and scoring rules, but is insufficient to prove the acquisition of superior clinical decision principles. Notably, a substantial portion of the composite reward advantage of the GDPO-trained Llama over GPT-5.2 stems from the penalty difference for failing to provide a treatment recommendation before turns run out (Ic\mathbb{I}_c), whereas GPT-5.2 still leads in information gathering (2.977 vs 2.953) and reasoning scores (3.016 vs 2.791). Therefore, a "higher composite reward" cannot be directly translated into "stronger clinical reasoning".

Questions for authors
  1. What generalizable principle does this work contribute to RL in partially verifiable domains, beyond running standard GRPO/GDPO against an LLM rubric?

  2. What concrete evidence distinguishes genuine clinical reasoning from policy adaptation to the specific retrieval interface and judge prompt?

3 comments

Thank you for your review. We appreciate your careful reading and your detailed questions.

W1 and Q1.

We agree that our method uses standard GRPO and GDPO with an LLM judge, and we do not claim novelty in the RL algorithm itself. Our contribution is in the interactive environment and the patient-specific reward for treatment planning under partial verifiability. Treatment planning can have multiple acceptable treatment options for the same patient. Since the medical records contain only the realized treatment plan, we cannot observe the other acceptable plans or the decision processes that could lead to them, which makes both the treatment decision and the decision process partially verifiable. We therefore assume that the realized plan is one of the acceptable decisions, and we extract the realized plan and the documented decision process from the records as the reference for the LLM judge, which grades the whole trajectory.

To investigate the specific benefit of the process-aware reward, we ablate the reward into rule-based, LLM-judged and combined settings (Table 1). The rule-based information acquisition reward is the F1 score between the factors the agent asks about and those in the physician's documented decision process, matched by keyword over 26 factors such as stage, biomarkers and labs.

Treatment rewardProcess rewardClinical scoreCompleteQuestions per caseDecision-factor recall
LLM-judged regimen–0.5240.97917.80.369
LLM-judged regimenLLM-judged info + reasoning0.5110.95825.40.459
LLM-judged regimen + rule-based drug matchLLM-judged info + reasoning0.5190.99316.60.408
LLM-judged regimen + rule-based drug matchLLM-judged info + reasoning + rule-based decision factors0.4670.85338.50.537
Rule-based drug matchRule-based decision factors0.4150.80441.40.547

Table 1. Reward ablation on the validation split (143 patients, one greedy trajectory per patient at step 300, scored by the fine-tuned judge). Decision-factor recall is the share of the physician’s decision factors the agent asked about. The full table with six settings is in our response to Reviewer KiFX.

Rewarding information acquisition and reasoning with the LLM judge makes the agent ask about more of the decision factors the physician used. The decision-factor recall is 0.459 with these two LLM-judged rewards compared to 0.369 with the LLM-judged regimen only, while the agent still completes 95.8% of the cases. In the rule-based reward setting for information acquisition, on the other hand, the agent exploits the reward by asking more questions. With the rule-based decision-factor reward alone, the agent asks 41.4 questions per case and completes only 80.4% of the cases, and imposing it along with the LLM-judged reward still leads to 38.5 questions per case, compared to 16.6 without it.

For Q1, the principle we draw from these results is that when the decision process is only partially verifiable, the documented process is better used as the reference for an LLM judge than as the target of a rule-based reward, since the rule-based version is exploited. We will restate our contributions in the paper in this regard and revise the scope accordingly.

(continue)

W2.

We agree that a higher composite reward alone does not show stronger clinical reasoning, and we will narrow the claim to the treatment planning behavior in this environment. *As the reviewer points out, most of GDPO-Llama's lead over GPT-5.2 comes from the completion penalty. *On the trajectories that are completed, the clinical scores are level (0.615 for GDPO-Llama against 0.617 for GPT-5.2), with GDPO-Llama higher on the regimen and GPT-5.2 higher on the reasoning and the information acquisition (Table 2). We now report the clinical score without penalty next to the score with penalties (Table 1 in our response to Reviewer KiFX).

PolicyCompleted rolloutsClinical scoreInfoRegimenReasoning
Llama-3.1-8B base366 of 426 (85.9%)0.4152.1512.0761.996
GPT-5.2354 of 426 (83.1%)0.6173.0653.0953.100
GRPO-Llama409 of 426 (96.0%)0.5942.7623.4352.708
GDPO-Llama421 of 426 (98.8%)0.6152.9543.4662.799

Table 2. Scores on the completed rollouts only, i.e., the rollouts that end with a treatment recommendation within 5 turns, out of 426 per policy (142 test patients, 3 rollouts each). Rollouts without a recommendation are excluded. Info, regimen and reasoning are the GPT-5.2 judge’s scores (0-5); clinical score is the sum of the three axes divided by 15.

We also agree that a reference reconstructed from the notes reflects one documented decision and its rationale, not a step-by-step record of how the physician gathered information. The judge therefore grades the process against the documented decision process, instead of requiring the agent to ask the same questions.

Q2.

While we do not have a direct measure of clinical reasoning, we provide two measurements independent of the LLM judge. First, an independent clinician assessment shows that the regimen scores from the GPT-5.2 judge agree with the clinician’s grades of the treatment plans (Spearman’s rank correlation coefficient 0.80; Tables 3 and 4 in our response to Reviewer KiFX). Second, the decision-factor recall in the ablation is computed by keyword matching without any LLM, and it rises from 0.369 to 0.459 with the LLM-judged information and reasoning rewards (Table 1).

The clinician assessment also shows a limit of both the agent and the judge. For the plans that are graded not appropriate by the clinician, five of GDPO-Llama’s 7 plans leave out radiation or surgery from a multimodal curative plan for rectal cancer or anal squamous cell carcinoma although the plans suggest the right chemotherapy. The judge scores them 3 or 4, since the rubric focuses on the accuracy of the regimen, and we will discuss this limit in the paper.

For the retrieval interface, our current results do not separate the agent’s behavior from adaptation, since each policy is trained and evaluated with the same mediator. We will evaluate the policies with a mediator they were not trained with in the revision.

Ethics.

Thank you for raising this. Both the 34-case study with four oncologists and the new clinician assessment are covered by our institution's IRB protocol; we withhold the protocol number for anonymity and will add it in the camera-ready version. We will also correct the ethics statement in the paper accordingly.

Thank you to the authors for carefully addressing my questions and adding the reward ablation and independent clinical evaluation. I appreciate the value of these experiments, and I also thank the authors for clarifying the scope of the paper's contributions.

However, my two main concerns remain.

First, the new ablation shows that adding the LLM-judged process reward increases decision-factor recall from 0.369 to 0.459, while the average number of questions rises from 17.8 to 25.4 and the clinical score drops from 0.524 to 0.511. This indicates that the agent covers more decision factors, but it does not yet show that the process reward improves overall treatment decisions. I also do not see a clear, generalizable RL insight coming out of these results.

Second, the authors confirmed my earlier reading of the composite reward. On completed trajectories, GDPO-Llama achieves roughly the same clinical score as GPT-5.2 (0.615 vs. 0.617). The added clinician assessment supports the validity of the regimen scoring, but it does not yet demonstrate stronger clinical reasoning. Some clinically inappropriate treatment plans still receive high judge scores, and generalization experiments across different retrieval mediators have not yet been completed. As a result, the current evidence is not yet sufficient to separate genuine clinical reasoning from adaptation to a specific environment and evaluation rule.

Regarding the IRB explanation, I believe the authors have addressed my procedural questions.

Overall, I recognize that the rebuttal improves the experimental analysis and the paper's framing, but the core methodological contribution and its scientific significance are still not sufficiently demonstrated. Therefore, I will maintain my original score of 1.

Reviewer_KiFXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning for clinical treatment planning under incomplete patient information and partially verifiable feedback. It constructs an interactive environment from retrospective colon cancer records, where agents gather information and select treatments using patient-specific rewards for information acquisition, treatment choice, and reasoning. Applying GRPO and GDPO, the authors report that a specialized small language model outperforms its base model and GPT-5.2 on the environment’s aggregate reward, demonstrating the potential of retrospective clinical data to support interactive policy training.

Critical strengths
  1. The paper addresses a clinically meaningful problem by coupling active information acquisition with treatment selection under incomplete patient information. Patient-specific evaluation of both the decision process and final regimen provides a useful extension beyond static medical question answering.

  2. Reinforcement learning improves the reported reward across three model backbones, supporting the utility of the constructed environment. After training, the small model achieves higher reward than advanced models such as GPT-5.2. Patient-level data separation, validation-based checkpoint selection, and analyses of individual reward components and query behavior strengthen the empirical evidence.

Critical weaknesses
  1. The approach combines established GRPO/GDPO algorithms and embedding-based retrieval. Its main contribution is the clinical environment. It would be helpful to clarify how “partial verifiability” informs the proposed reward design and distinguishes it from existing instance-specific, rubric-based RL. Targeted ablations of the information-gathering and reasoning rewards could further demonstrate their added value over rewarding treatment accuracy alone.
  2. Independent clinician assessment would strengthen the evaluation, since GPT-5.2 provides both training rewards and test scores. Reporting clinical scores separately from completion penalties, together with supervised fine-tuning baselines and reward ablations, would help clarify the specific benefits of process-aware interactive RL.
Questions for authors

See Weakness.

4 comments

Thank you for your review. We appreciate your recognition of the strength of the paper and your advice on making the paper better.

W1.

Our reward design indeed belongs to the family of instance-specific, rubric-based RL. However, partial verifiability informs us where the instance-specific context should come from. Existing rubric-based rewards, such as Rubrics as Rewards (Gunjal et al., 2026), derive the criteria for each instance from a gold reference answer, which assumes that the correct answer is known. Treatment planning, on the other hand, is partially verifiable because there is only one realized trajectory, meaning that we are not able to observe the possible alternatives. At a given patient state, there can be multiple treatment alternatives. While we assume that the treatment plan that was realized in the medical records was one of the acceptable decisions, annotating the complete set of acceptable plans would require clinicians for every patient. Instead, we extract the realized trajectory from the medical records, including the decision and the decision-making process. This allows us to scale the training set along with the available records in a timely fashion.

We design the reward in three axes, which are regimen selection, information acquisition and reasoning, for the following considerations. For the regimen, instead of using exact match, we use an LLM judge to grade the regimen against the realized plan and the documented reasons, so that a clinically equivalent or acceptable alternative still receives credit. For the information acquisition and reasoning, the documented decision process provides the factors that the physician weighed, but other valid ways to reach the decision are not observed. The judge therefore grades the process against the documented decision process, instead of requiring the agent to ask the same questions. Since the agent is most rewarded when its plan and actions match the realized trajectory, the agent learns to adapt to the local practice among the acceptable options.

For the targeted ablations on the information-gathering and reasoning rewards, please see our response at W2.

(continue)

W2.

We report the clinical scores separately from the completion penalties as follows. We also include a supervised finetuning (SFT) baseline on Llama-3.1-8B-Instruct model. The table reports the scores of the policy on the held-out test set judged by GPT-5.2. Additionally, we also measure the exact drug match between the policy and the physician. The regimen is decomposed into its active ingredients according to RxNorm and HemOnc vocabularies. For example, FOLFOX with bevacizumab is decomposed to leucovorin, fluorouracil, oxaliplatin and bevacizumab. The exact match is 1 if the sets of drugs between the policy and the physician match exactly, 0 otherwise. We fine-tuned a base Llama-3.1-8B-Instruct on its own trajectories sampled using rejection sampling to retain the ones that matched the physician's regimen. We sampled 8 trajectories per patient for the 665 training patients; 632 of the 5,320 trajectories from base Llama had an exact drug match with the physician, and keeping at most 4 per patient gave 436 training and 49 validation trajectories from 266 patients. We trained a LoRA adapter with rank 32, alpha 32, same as the RL policy, for 3 epochs on the assistant turns only.

The clinical score without penalty from GDPO-Llama is 0.613, which still leads GPT-5.2 at 0.580. On exact drug match, GDPO-Llama reaches 0.526 against 0.326 for GPT-5.2. SFT baseline reached a clinical score of 0.411 not including penalties, close to base Llama at 0.395, and 0.224 on exact drug match, more than double base Llama at 0.100. Both stay below GDPO-Llama at 0.613 and 0.526.

PolicyClinical scoreClinical score with penaltiesPenaltyInfoRegimenReasoningCompleteExact drug match
Llama-3.1-8B base0.3950.339-0.0562.1241.8761.9270.8590.100
SFT0.4110.318-0.0932.2141.9392.0190.7680.224
GPT-5.20.5800.512-0.0682.9772.7023.0160.8310.326
GRPO Llama0.5870.571-0.0162.7543.3662.6880.9600.519
GDPO Llama0.6130.608-0.0052.9533.4532.7910.9880.526

Table 1. Scores on the held-out test set (142 patients, 3 trajectories each). Info, regimen and reasoning are the GPT-5.2 judge's scores (0-5); clinical score is the sum over three reward axes divided by 15, without penalties. Complete is the share of trajectories with a recommendation. Exact drug match is the share of trajectories whose recommended drugs are the same set as the physician's, after mapping both to active ingredients with RxNorm and HemOnc; dose, schedule, surgery and radiation are ignored.

To investigate the specific benefit of process-aware interactive RL, we ablate the reward into rule-based, LLM-judged and combined settings. In the rule-based settings, the policy is rewarded for drug match and completion, and in one setting also for information acquisition. Drug match rewards the chemotherapy backbone and the other drugs separately against the physician's regimen extracted from the medical record. The rule-based information acquisition reward is the F1 score between the factors the agent asks about and those in the physician's documented decision process, matched by keyword over 26 factors such as stage, biomarkers and labs. In the LLM-judged setting, the judge scores information, regimen and reasoning with the rubric of the paper.

(continue)

We conduct the ablation study with the following setup. The judge is Qwen3.8-27B-FP8 with a LoRA adapter fine-tuned on the past judging completions from GPT-5.2, and the mediator is Qwen3-1.7B with a LoRA adapter. We train Llama-3.1-8B-Instruct with GDPO for 300 steps under six reward settings with group size 8, 5 turns, the same patient order and seed 42. We evaluate the checkpoint at step 300 on the validation split with 143 patients with one trajectory per patient with greedy decoding. Since the judge, the mediator and the split differ from the main experiments, these scores are not comparable with the table above.

Treatment rewardProcess rewardClinical scoreInfoRegimenReasoningCompleteExact drug matchQuestions per caseDecision-factor recall
LLM-judged regimenLLM-judged info + reasoning0.5112.5662.6922.4130.9580.24825.40.459
LLM-judged regimen–0.5242.6012.8812.3780.9790.34017.80.369
LLM-judged regimen + rule-based drug matchLLM-judged info + reasoning0.5192.6292.7692.3920.9930.28416.60.408
LLM-judged regimen + rule-based drug matchLLM-judged info + reasoning + rule-based decision factors0.4672.3642.3852.2520.8530.18438.50.537
Rule-based drug match–0.4912.4902.5732.3010.9790.22713.30.330
Rule-based drug matchRule-based decision factors0.4152.2521.8812.0980.8040.07141.40.547

Table 2. Reward ablation on the validation split (143 patients, one greedy trajectory per patient at step 300, scored by the fine-tuned judge). In the LLM-judged setting, the policy is rewarded by the fine-tuned judge; in the rule-based setting, the reward is computed by deterministic rules. Decision-factor recall is the share of the physician's decision factors the agent asked about.

Rewarding information acquisition and reasoning with the LLM judge makes the agent ask about more of the decision factors the physician used. The decision-factor recall is 0.459 with these two LLM-judged rewards compared to 0.369 with the LLM-judged regimen only and 0.330 with the rule-based drug match only. However, the exact drug match does not improve, at 0.248 against 0.340 from the LLM-judged regimen only setting and 0.227 from the rule-based drug match only setting.

In the rule-based reward setting for information acquisition, the agent exploits the reward by asking more questions regardless of the LLM-judged reward. Imposing the rule-based decision-factor reward along with the LLM-judged reward leads to higher recall of clinical factors at 0.537, but the agent asks 38.5 questions per case on average, compared to 16.6 for the agent trained without the rule-based reward. Without the judge, the agent asks 41.4 questions per case, compared to 13.3 with the rule-based drug match reward alone. The higher decision-factor recall therefore comes at the cost of excessive questions and fewer completed cases (0.853 and 0.804).

To conclude, the information and reasoning rewards change the decision process of the agent, so that it asks about more of the factors the physician used, while the verifiable version of the information reward is exploited. Due to the partial verifiability of the process, the LLM-judged reward supplied with a realized process allows the model to learn more practical behaviors that suit the clinical reality.

(continue)

An independent clinician assessment shows that the regimen scores from the GPT-5.2 judge agree with the clinician's grades of the treatment plans (Spearman's rank correlation coefficient 0.80). We draw 30 patients from the held-out test set. Patients the clinician had already seen in the earlier 34-case study were excluded. The clinician reviews the treatment plans from GDPO-Llama and GPT-5.2 for those patients in blinded fashion along with the full patient representation and the original treatment plan from the physician as the comparison point. Each treatment plan is graded with one of the following choices:

  • Same regimen,
  • Clinically equivalent,
  • Reasonable alternative,
  • Not appropriate, or
  • Cannot judge.

The clinician judges the choice of drugs and ignores dose, schedule and number of cycles. We then compare each grade with the regimen score that the GPT-5.2 judge gave the same rollout.

ModelSame regimenClinically equivalentReasonable alternativeNot appropriateCannot judgeAcceptable (95% CI)
GDPO-Llama14447122 of 29, 76% (58-88%)
GPT-5.284103522 of 25, 88% (70-96%)

Table 3. Clinician grades of the plans from GDPO-Llama and GPT-5.2 for 30 held-out test patients, rated blind against the treating oncologist's plan. Acceptable plans include same regimen, clinically equivalent or reasonable alternative, excluding the plans that cannot be judged. We report the 95% Wilson intervals.

The clinician rated 76% of GDPO-Llama's plans and 88% of GPT-5.2's plans acceptable, i.e., same regimen, clinically equivalent or reasonable alternative (95% intervals 58-88% and 70-96%). GDPO-Llama more often chose the physician's regimen (14 against 8 plans), while GPT-5.2 more often gave a reasonable alternative (10 against 4).

Clinician gradeNumber of plansMean judge score
Same regimen224.68
Clinically equivalent83.50
Reasonable alternative142.64
Not appropriate102.20
Cannot judge63.17
Spearman's rank correlation coefficient between clinician and GPT-5.2 judge-0.80
Agreement on same or equivalent, % (Cohen's κ\kappa)-76% (0.48)

Table 4. The agreement between the clinician assessment and the GPT-5.2 judge's scoring on the regimen accuracy on plans from GDPO-Llama and GPT-5.2 for 30 patients. The rubric gives 5 to an exact match with the regimen from the original treatment plan, 3-4 to a clinically equivalent regimen, 2 to a regimen of the right class that is not equivalent, and 0-1 to a wrong class. We report the Spearman's rank correlation coefficient between the judge's scores and the clinician's grades. For the agreement, a plan counts as same or equivalent for the judge when it scores 3-5, and for the clinician when it is graded same regimen or clinically equivalent. Plans the clinician cannot judge are excluded from the rank correlation and the agreement.

The judge's regimen score is concordant with the clinician's grade; the mean judge score is 4.68 for plans graded as the same regimen, 3.50 for clinically equivalent, 2.64 for reasonable alternative and 2.20 for not appropriate. Of the six plans the clinician could not judge, 4 are GPT-5.2 plans that list several conditional options without choosing a single one, and 2 are for one patient whose original plan was unclear.

For the plans that are graded not appropriate by the clinician, five of GDPO-Llama's 7 plans are rated not appropriate because they leave out radiation or surgery from a multimodal curative plan for rectal cancer or anal squamous cell carcinoma although the plans suggest the right chemotherapy. The missing modality is a limit of the agent and the rubric, which focuses on the accuracy of the regimen, since our evaluation targets the chemotherapy plan.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.