acceptodds
Under review as a conference paper at ICLR 2027

SAGA-CDP: Large Language Models Can Learn Guideline-Adherent Clinical Decision-Making from Synthetic Data

Abstract

Reliable clinical decision-making with large language models requires models to adhere to specified clinical guidelines throughout the decision-making process. In this work, we systematically investigate whether guideline adherence can be learned as a transferable capability from both real-world and synthetic guideline data. These two data sources present distinct trade-offs in guideline-adherence training. To enable structured synthetic data generation, we introduce SAGA-CDP, a data generation pipeline that produces fictional clinical guidelines, clinical decision pathways (CDPs), and case vignettes from shared underlying decision logic, yielding 2,000 fictional guidelines and 76,584 corresponding clinical cases. Extensive experiments demonstrate that real-world and fictional guideline data are both effective training sources, and that combining them achieves the strongest overall performance. Scaling analyses reveal distinct behaviors across the two data sources, with real-world guideline data continuing to benefit from more training cases, while further gains from fictional data depend more strongly on broader guideline coverage. Cross-regional real-world studies and controlled synthetic interventions provide evidence of context-memory conflict in guideline-adherent clinical decision-making. These findings establish real-world guideline data as a clinically grounded and viable resource for guideline-adherence training, while highlighting synthetic data as a scalable complement that increases the amount of training data and reduces reliance on real-world guideline knowledge.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Clinical Decision-Making, Clinical Guideline Adherence, Synthetic Data Augmentation, Large Language Models, Clinical Decision Pathways

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZLUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a pipeline that learns clinical guideline-following behavior from synthetic "fictional guidelines": guidelines are generated by an LLM, refined via embedding-distance filtering, and used to fine-tune a model. The authors argue that real guidelines are unnecessary for acquiring such behavior and report large PCC gains on the MEGA-CDP benchmark.

Critical strengths

The problem setting is novel: demonstrating that guideline-following behavior can be acquired from fictional synthetic text opens a viable route for clinical scenarios lacking real guidelines, and the reported benchmark gains are substantial.

Critical weaknesses

All fine-tuning uses a single random seed with no repeats or significance testing, so the uncertainty of the headline comparisons cannot be assessed.

In addition, every experiment uses only one base model (Qwen3.5-9B) on one benchmark (MEGA-CDP), and the core PCC metric is scored automatically by another LLM (Qwen3-8B) with no human validation; together these choices leave the reality of the gains unsupported.

Questions for authors

How is leakage between the synthetic data and the evaluation set ruled out? What is the seed-to-seed variance? Can you provide agreement data between the LLM judge and human judgments?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Questions 1–2 of Reviewer ZLUB

Stability across training seeds

Weakness 1:
All fine-tuning uses a single random seed with no repeats or significance testing, so the uncertainty of the headline comparisons cannot be assessed.

Question 2:
What is the seed-to-seed variance?

  • We sincerely thank the reviewer for the constructive comments. To test whether the results were stable across random seeds, we repeated fine-tuning of Qwen3.5-9B under each of the three training configurations in Table 1 using seeds 40, 41, and 42. We evaluated all nine runs on the MEGA-CDP test set. The percentages in the training configurations refer to the fraction of each data pool used. We report the mean, sample standard deviation (SD), and sample variance across the three seeds. In the revised version, we report the individual scores for all nine runs in Appendix F.3, Table 10.

    Training ConfigurationSeedPCC (%) ↓Acc (%) ↑SR (%) ↑
    100% Real-World4040.1951.3099.73
    4140.6951.0499.85
    4240.4851.3499.81
    Mean ± SD40.45 ± 0.2551.23 ± 0.1699.80 ± 0.06
    Variance0.06300.02650.0037
    20% Fictional4043.7845.9799.91
    4143.7546.3999.93
    4243.6546.1699.97
    Mean ± SD43.73 ± 0.0746.17 ± 0.2199.94 ± 0.03
    Variance0.00460.04420.0009
    100% Real-World + 3% Fictional4039.5351.8599.75
    4140.0451.2699.58
    4239.5552.6499.65
    Mean ± SD39.71 ± 0.2951.92 ± 0.6999.66 ± 0.09
    Variance0.08340.47940.0073
  • Across the three training configurations, the sample variance over seeds ranged from 0.0046 to 0.0834 for PCC, from 0.0265 to 0.4794 for Acc, and from 0.0009 to 0.0073 for SR. These low variances indicate that performance was stable across the three seeds tested.

Disease-concept overlap with the evaluation set

Question 1:
How is leakage between the synthetic data and the evaluation set ruled out?

  • Thank you for these perceptive comments. We combined LLM-based review and manual inspection of the entire synthetic dataset, focusing on whether its disease concepts overlapped with those represented in the MEGA-CDP evaluation set. Neither check identified any overlap in disease concepts.

  • Generation starts from five categories of entirely fictional disease concepts in Appendix B: Fantasy, Science Fiction, Xuanhuan, Famous Fictional Works, and Realistic Medical Variants. The generation instructions require fictional disease identities and constrain synthesis to preserve the fictional setting. These choices are designed to avoid reproducing real-world disease entities and reduce disease-specific overlap. The LLM-based and manual checks provide additional empirical support for the finding that no leakage was detected.

AuthorsOpenReview ↗

Response to Weakness 2 & Question 3 of Reviewer ZLUB

Agreement between the LLM judge and human judgments

Weakness 2:
the core PCC metric is scored automatically by another LLM (Qwen3-8B) with no human validation

Question 3:
Can you provide agreement data between the LLM judge and human judgments?

  • We fully understand your concern about how closely the LLM judge’s assessments agree with human judgments. Our study focuses on guideline-adherence training and uses the existing evaluation procedure from MEGA-CDP, so a systematic assessment of human-model agreement falls outside its scope. Nevertheless, we can provide a set of results from an analysis of how closely the LLM judge’s assessments of MEGA-CDP pathways agree with human judgments. These results directly address your question.

  • In this study, two human judges independently reviewed the same 200 pathway pairs from the multi-turn setting. For each case, they assessed two predicted pathways against a common reference and indicated which followed it more closely. The experiment recorded agreement whenever a human judge selected the pathway with the lower PCC.

  • In this experiment, Group 1 contained 100 randomly sampled pairs from eligible outputs across 16 base models. Group 2 contained 100 pairs from Claude-Sonnet-4.6 and Qwen3-Next-80B-A3B-Instruct, allowing us to examine agreement when comparing two stronger models. Each group comprised 50 English and 50 Chinese cases. We report agreement for each group and across all 200 pairs.

    GroupComparisonAgreement Rate
    Group 1Combined Human–Model Agreement89.0%
    Inter-Human Agreement88.0%
    Group 2Combined Human–Model Agreement91.0%
    Inter-Human Agreement92.0%
    OverallCombined Human–Model Agreement90.0%
    Inter-Human Agreement90.0%
  • Across both groups, human–model agreement was 90.0%, the same rate observed between the two human judges. These results indicate that the LLM judge can reliably assess how closely a predicted pathway follows the reference pathway.

AuthorsOpenReview ↗

Response to Weakness 2 of Reviewer ZLUB (1/2)

Evaluation on a second model family

Weakness 2:
every experiment uses only one base model (Qwen3.5-9B)

  • We extended training to Gemma-3-12B-IT, using the same mixture of real-world and fictional guideline data. Comparing the fine-tuned model against its own baseline allows us to examine whether the approach also benefits a backbone outside the Qwen family. In the revised version, we report all three Gemma metrics in Appendix F.1, Table 8.

    ModelPCC (%) ↓Acc (%) ↑SR (%) ↑
    Baseline (Gemma-3-12B-IT)60.0510.8699.03
    Ours (Gemma-3-12B-IT)44.3446.6199.76
  • The Gemma model achieves a 15.71-percentage-point reduction in PCC and a 35.75-percentage-point increase in accuracy. Its success rate also increases from an already high baseline. These results provide evidence that the training benefits are not restricted to the Qwen model family.

Reviewer_wEqROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether adherence to a provided clinical guideline can be learned as a transferable skill. It introduces SAGA-CDP, which generates 1,000 fictional diseases in five narrative categories (each in English and Chinese), builds a reasoning tree per disease, and derives from the same tree a fictional guideline, reference clinical decision pathways (CDPs), and case vignettes (2,000 guidelines and 76,584 cases in total). An agentic reviewer-rewriter loop moves the guideline text closer to real guidelines, and a separate validator checks that the decision logic stays complete. LoRA fine-tuning of Qwen3.5-9B on real-world data (from the MEGA-CDP pipeline), fictional data, or both improves pathway consistency cost (PCC) and outcome accuracy on the MEGA-CDP test set, with the mixture best (PCC 39.55 and Acc 52.64, against 53.82 and 24.77 for the base model). Scaling analyses suggest that real data keep improving with more cases, while fictional data reach their best validation PCC at about 15K cases and then degrade; wider guideline coverage helps fictional data at a fixed budget. Cross-regional and controlled synthetic experiments show that training on one version of a disease's decision logic degrades adherence when the test guideline differs, which the authors interpret as context-memory conflict.

Critical strengths

S1. A clear and practical question: can adherence to a supplied guideline be trained, and what do synthetic and real guideline data each contribute? The research questions are well defined.

S2. Generating guideline text, pathways, and vignettes from one explicit reasoning tree is a sound design. It gives exact reference logic and allows controlled edits of individual decision nodes, which the paper uses for the conflict study.

S3. Fictional diseases avoid leakage from memorized medicine and sidestep licensing limits on real guidelines; the bilingual dataset could be a useful resource.

Critical weaknesses

W1. The evaluation is narrow and may reward format learning. All fine-tuning uses one base model (Qwen3.5-9B), all evaluation uses the MEGA-CDP protocol with one LLM judge (Qwen3-8B), and training uses one seed and one epoch of LoRA. Both training sources share the test format: the real-world data are the non-test MEGA-CDP guidelines, and the fictional vignettes follow MEGA-CDP's case construction procedure (Section 3.2). Part of the gain may therefore come from familiarity with MEGA-CDP's interaction format rather than from adherence. The success rate rises from 89.96% to between 99.65% and 99.97%, and on the intersection of successful dialogues (Appendix E) the PCC gains shrink by a third to a half (for example from 13.34 to 8.74 points for real-only training), although the accuracy gains mostly remain. Please add (a) a format-only control, for example fine-tuning on the same dialogues with the guideline removed or replaced by an unrelated one; (b) at least one interactive clinical benchmark not produced by this pipeline [1, 2, 3], or the guideline-adherence benchmarks the paper already cites; (c) a second base-model family; and (d) clinician validation of the judge's step matching, since a Qwen-family judge scores a Qwen-family student.

[1] Li et al. MediQ: question-asking LLMs and a benchmark for reliable interactive clinical reasoning. NeurIPS 2024. [2] Schmidgall et al. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv, 2024. [3] Johri et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nature Medicine, 2025.

W2. The quality of the synthetic data is not validated by experts. The refinement stopping rule compares embedding distances to the real-world and initial fictional guideline centroids, which measures surface similarity rather than clinical plausibility or internal consistency. Apart from an informal inspection of sampled trees during prompt development (Section 3.2), with no stated reviewers or results, there is no clinician review of a sample of trees, guidelines, or vignettes, and no count of contradictions or unreachable branches. The vignettes also contain, by construction, only facts on the reference pathway (Table 8, rules 12 and 15), with no distractors or irrelevant findings, which may make the pathway easier to recover than in real cases. A small expert audit would make the dataset much more credible.

Questions for authors

See above

7 comments
AuthorsOpenReview ↗

Response to Weakness 1(a) of Reviewer wEqR

Format learning alone cannot explain the gains in guideline adherence

Weakness 1(a):
a format-only control, for example fine-tuning on the same dialogues with the guideline removed or replaced by an unrelated one

  • We thank you for these perceptive comments and agree that a format-only control is necessary. We repeated fine-tuning under two conditions using the same dialogues, either removing the guideline or replacing it with an unrelated one. Both experiments followed the mixed training setting with all real-world training data plus 3% of the fictional pool. In the revised version, we report the results for both format controls and the Qwen3.5-9B baseline in Section 4.6, Table 4.

    Training ConditionPCC (%) ↓Acc (%) ↑SR (%) ↑
    Baseline (Qwen3.5-9B)53.8224.7789.96
    Ours39.5552.6499.65
    Same Dialogues, Guideline Removed55.6415.6397.51
    Same Dialogues, Unrelated Guideline96.755.5599.32
  • Both controls improve SR over Qwen3.5-9B (baseline), but neither improves PCC or accuracy. The unrelated-guideline control reaches 99.32% SR, close to our model's 99.65%, while its pathway consistency and accuracy deteriorate substantially. Guideline removal likewise increases SR to 97.51% while reducing pathway consistency and outcome accuracy. Only fine-tuning with the corresponding guideline improved both pathway consistency and outcome accuracy. This pattern suggests that our model learned to follow the supplied guideline.

AuthorsOpenReview ↗

Response to Weakness 1(b) of Reviewer wEqR

Evaluation in three settings across two external clinical benchmarks

Weakness 1(b):
at least one interactive clinical benchmark not produced by this pipeline [1, 2, 3], or the guideline-adherence benchmarks the paper already cites

  • We sincerely thank you for your constructive suggestion. To our knowledge, MEGA-CDP is the first benchmark to assess whether a model’s multi-turn clinical decision pathway follows a guideline provided in context. We could not identify an independent benchmark that directly evaluates the same task. Nevertheless, we compared the baseline and fine-tuned models in three settings across two external benchmarks. We used MediQ for interactive clinical question answering and evaluated longitudinal clinical decision-making on NPCBench L5 with and without added guideline context. In the revised version, we report the baseline and fine-tuned results across all three settings in Section 4.7, Table 6.

1. Interactive evaluation on MediQ

  • MediQ (Li et al., 2024) is a benchmark for evaluating how models gather missing patient information to answer clinical questions. We used the iMedQA dataset in MediQ’s BASIC interactive setting, where the model can ask the simulated patient follow-up questions before selecting its final answer. The table below reports final-answer accuracy using the official implementation.

    ModelAccuracy (%) ↑
    Baseline (Qwen3.5-9B)48.35 [45.60, 51.10]
    Ours50.79 [48.03, 53.54]
    Llama-3-8b†33.0
    Llama-3-70b†55.1
    GPT-3.5†42.2
    GPT-4†55.4
  • The results marked with † are taken directly from Table 2 of the MediQ paper for this task setting. For our evaluations, we estimated 95% confidence intervals using 20,000 bootstrap resamples and report them in brackets. Although differences in model size and underlying capabilities make comparisons with the reference models difficult, our model still achieves a clear improvement over its baseline.

2. Multi-turn longitudinal decisions on NPCBench L5

  • NPCBench (Wang et al., 2026) is a benchmark for evaluating clinical decision-making in nasopharyngeal carcinoma care. We used its L5 task, which focuses on longitudinal clinical decision-making across multiple turns. At each decision point, the model uses the available patient history to select the next management action from the options provided. The benchmark supplies these histories without model–patient interaction. Text-only inputs also let us assess transfer without adding visual perception requirements. Using the official implementation, we report turn-micro accuracy and all-turn patient success:

    ModelTurn-Micro Accuracy (%) ↑All-Turn Patient Success (%) ↑
    Baseline (Qwen3.5-9B)62.62 [57.72, 67.42]24.00 [16.00, 33.00]
    Ours72.13 [67.08, 77.11]39.00 [30.00, 49.00]
    Qwen3.5-9B†60.724.0
    HuatuoGPT-o1-7B†55.120.0
    Fleming-R1-7B†59.323.0
    HealthGPT-Pro-8B†38.79.0
    Hulu-Med-7B†47.216.0
    Lingshu-7B†49.214.0
    GPT-OSS-20B†48.913.0
  • Reference entries marked † come from NPCBench Table 1. For our evaluations, we estimated 95% confidence intervals using 20,000 patient-level bootstrap resamples and report them in brackets. Our model outperforms the baseline on both metrics, supporting generalization beyond MEGA-CDP.

3. Supplying guideline context to NPCBench L5

  • For the third setting, we inserted the CSCO guideline identified in the NPCBench paper into the L5 prompt and evaluated the same decision task with the guideline in context. Although this adaptation is an imperfect proxy for guideline adherence, it allows us to assess how well the model answers clinical questions when the relevant guideline is available.

    ModelTurn-Micro Accuracy (%) ↑All-Turn Patient Success (%) ↑
    Baseline (Qwen3.5-9B)50.16 [44.93, 55.45]15.00 [8.00, 22.00]
    Ours65.25 [60.45, 70.04]26.00 [18.00, 35.00]
  • For our evaluations, we report 95% confidence intervals in brackets. Because supplying the guideline changes the original task design, direct comparison with the standard NPCBench scores above is inappropriate. Within the adapted setting, our model outperforms the baseline on both measures when the guideline is supplied as context.

AuthorsOpenReview ↗

Response to Weakness 1(c) of Reviewer wEqR

Evaluation on a second model family

Weakness 1(c):
a second base-model family

  • To test whether the benefits extend beyond Qwen3.5-9B, we fine-tuned Gemma-3-12B-IT using mixed real-world and fictional guideline data. We compared it with its own baseline on MEGA-CDP. In the revised version, we report Gemma’s performance before and after fine-tuning in Appendix F.1, Table 8.

    ModelPCC (%) ↓Acc (%) ↑SR (%) ↑
    Baseline (Gemma-3-12B-IT)60.0510.8699.03
    Ours (Gemma-3-12B-IT)44.3446.6199.76
  • Although the baseline already has a high success rate, fine-tuning lowers PCC by 15.71 percentage points and raises accuracy by 35.75 percentage points. These findings show that the improvements extend across both evaluated model families and suggest that the gains are not merely due to format learning.

Reviewer_ULCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether clinical guideline adherence can be improved through post-training and transferred to unseen real-world guidelines. The authors propose SAGA-CDP, a synthetic data generation pipeline that constructs fictional diseases, reasoning trees, clinical guidelines, reference clinical decision pathways (CDPs), and corresponding case vignettes from shared decision logic. The synthetic guidelines are further refined to better match the structure and presentation of real clinical documents.

The resulting dataset contains 2,000 fictional guidelines and 76,584 cases. The authors fine-tune Qwen3.5-9B using real-world guideline data, fictional guideline data, and a mixture of the two, and evaluate the resulting models on MEGA-CDP. Both real-world and fictional supervision improve pathway consistency and final outcome accuracy, while mixed training achieves the best overall performance. The paper further studies data scaling, guideline diversity, and context-memory conflict.

Critical strengths
  1. The synthetic data construction provides a controlled setting for studying guideline-adherence training.

A notable strength of the paper is that the fictional guideline, reference CDP, and case vignette are all derived from the same underlying reasoning tree. This provides explicit control over the decision logic shared across the different components of each training instance. The fictional-only training results further show that supervision based on synthetic clinical rules can improve performance on held-out real-world guidelines, suggesting that the observed gains are not solely explained by memorization of real guideline content.

  1. The paper provides a reasonably broad empirical characterization of the proposed training setup.

Beyond the main comparison among real-world, fictional, and mixed supervision, the paper examines the effects of training scale and guideline diversity, and separately studies conflicts between training-time and test-time decision rules. These analyses help characterize both the benefits and limitations of guideline-adherence training rather than relying only on aggregate benchmark improvements.

Critical weaknesses
  1. The evidence for transfer remains limited to a narrow evaluation setting.

The main post-training results are evaluated within the MEGA-CDP setting, and Qwen3.5-9B is the only backbone used for fine-tuning. The paper does demonstrate generalization to held-out real-world guidelines, but these guidelines are still evaluated under the same multi-turn CDP formulation.

Since the paper motivates guideline adherence as a transferable capability, stronger evidence would require showing that the improvement persists under a different guideline-based clinical decision setting. This is particularly relevant because the proposed training improves both pathway consistency and final outcome accuracy. Evaluation on an independent benchmark with a different interaction or output format would help determine whether the improvement reflects a broader guideline-following capability or is partly specific to the MEGA-CDP formulation. In its current form, the evaluation does not fully resolve this distinction.

  1. The current comparisons do not sufficiently establish the contribution of the proposed post-training approach.

Table 1 includes a broad comparison with existing LLMs, but most of these models are evaluated in their existing form rather than serving as alternative post-training methods under comparable conditions. The real world only fine-tuning result is a useful control, but it does not establish whether the specific SAGA-CDP data construction is more effective than a simpler form of task-specific supervision.

This issue is important because the proposed pipeline contains several design choices, including the explicit reasoning-tree construction and the agentic refinement stage. The paper does not provide a downstream ablation of these components or a comparison with a simpler synthetic training strategy. As a result, it remains difficult to determine how much of the improvement should be attributed to the specific SAGA-CDP pipeline rather than to the additional supervised training data more generally.

Questions for authors
  1. Do the authors have results on another guideline-based clinical decision benchmark with a different task or output format from MEGA-CDP? Such evidence would materially affect my assessment of whether the learned guideline-adherence behavior transfers beyond the current evaluation setting.

  2. Do the authors have a comparison with a simpler post-training strategy using the same base model, or an ablation that quantifies the downstream contribution of the agentic refinement stage? Such results would help clarify which parts of the proposed pipeline are responsible for the reported improvements.

5 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1 of Reviewer ULCM (1/2)

Generalization of guideline-adherence training across clinical tasks

Weakness 1:
The evidence for transfer remains limited to a narrow evaluation setting.

Question 1:
Do the authors have results on another guideline-based clinical decision benchmark with a different task or output format from MEGA-CDP?

  • Thank you for your constructive feedback! To our knowledge, MEGA-CDP is the first benchmark to evaluate multi-turn clinical decision pathways against a guideline supplied in context. We did not find an independent public benchmark that directly evaluates whether a model’s clinical decision pathway follows the guideline supplied for that case. Nevertheless, we compared the baseline and fine-tuned models in three settings across two related benchmarks, using NPCBench L5 for multi-turn longitudinal clinical decision-making and MediQ for interactive clinical question answering. In the revised version, we report these results in Section 4.7, Table 6.

1. NPCBench L5

  • NPCBench L5 (Wang et al., 2026) evaluates multi-turn longitudinal clinical decisions in nasopharyngeal carcinoma care. At each turn, the model selects the next management action using the available patient history. The benchmark provides the patient history as text, allowing us to evaluate longitudinal decision-making without introducing a visual perception requirement. We used the official implementation. Turn-micro accuracy measures correctness across decisions. All-turn success requires every decision for a patient to be correct.

    ModelTurn-Micro Accuracy (%) ↑All-Turn Patient Success (%) ↑
    Baseline (Qwen3.5-9B)62.62 [57.72, 67.42]24.00 [16.00, 33.00]
    Ours72.13 [67.08, 77.11]39.00 [30.00, 49.00]
    Qwen3.5-9B†60.724.0
    HuatuoGPT-o1-7B†55.120.0
    Fleming-R1-7B†59.323.0
    HealthGPT-Pro-8B†38.79.0
    Hulu-Med-7B†47.216.0
    Lingshu-7B†49.214.0
    GPT-OSS-20B†48.913.0
  • Scores marked with † come from NPCBench Table 1. For our evaluations, we report 95% confidence intervals in brackets, estimated from 20,000 patient-level bootstrap resamples. Our model improves both turn-micro accuracy and all-turn patient success over the baseline and substantially outperforms similarly sized models on both metrics. These results show that the gains from fine-tuning also extend to multi-turn longitudinal clinical decision-making on NPCBench L5.

2. MediQ

  • We evaluated our model on MediQ (Li et al., 2024) using the iMedQA dataset in the BASIC interactive setting. The model initially receives a multiple-choice clinical question and limited patient information. At each turn, it can either ask the simulated patient a follow-up question or select a final answer. The patient responds based on the case record, so the additional clinical details available to the model depend on what it asks. Using the official implementation, we report final-answer accuracy, the percentage of cases in which the model selects the correct option.

    ModelAccuracy (%) ↑
    Baseline (Qwen3.5-9B)48.35 [45.60, 51.10]
    Ours50.79 [48.03, 53.54]
    Llama-3-8b†33.0
    Llama-3-70b†55.1
    GPT-3.5†42.2
    GPT-4†55.4
  • Scores marked with † are from MediQ Table 2 for this setting. For our evaluations, we estimated 95% confidence intervals using 20,000 bootstrap resamples and report them in brackets. Although the original MediQ study did not evaluate Qwen3.5-9B, the comparison with our own baseline shows that fine-tuning improved final-answer accuracy from 48.35% to 50.79%.

3. NPCBench L5 with guideline context

  • To evaluate our model on an external benchmark in a setting where the guideline is explicitly provided as context, we added the CSCO guideline specified in the NPCBench paper to the L5 prompt at each decision point.

    ModelTurn-Micro Accuracy (%) ↑All-Turn Patient Success (%) ↑
    Baseline (Qwen3.5-9B)50.16 [44.93, 55.45]15.00 [8.00, 22.00]
    Ours65.25 [60.45, 70.04]26.00 [18.00, 35.00]
  • For our evaluations, we report 95% confidence intervals in brackets. Adding guideline context is not fully aligned with the original task design, so these results are not directly comparable with the NPCBench results above. With the guideline supplied, our model outperforms the baseline on both metrics. This evaluation uses a simple adaptation of NPCBench L5 in which the guideline text is added to the existing prompt. We hope to see more independent benchmarks for evaluating guideline adherence in clinical decision-making. These could help advance LLM-based clinical decision-making toward greater trustworthiness and interpretability.

AuthorsOpenReview ↗

Response to Weakness 1 & Question 1 of Reviewer ULCM (2/2)

Generalization of guideline-adherence training across model families

Weakness 1:
Qwen3.5-9B is the only backbone used for fine-tuning.

  • Thank you for highlighting the need to evaluate a second model family. In response, we fine-tuned Gemma-3-12B-IT on mixed real-world and fictional guideline data. In the revised version, we report its MEGA-CDP results before and after fine-tuning in Appendix F.1, Table 8.

    ModelPCC (%) ↓Acc (%) ↑SR (%) ↑
    Baseline (Gemma-3-12B-IT)60.0510.8699.03
    Ours (Gemma-3-12B-IT)44.3446.6199.76
  • PCC decreases by 15.71 percentage points, and accuracy increases by 35.75 percentage points, despite an already high baseline SR. The benefits therefore extend across the two evaluated model families.

AuthorsOpenReview ↗

Response to Weakness 2 & Question 2 of Reviewer ULCM

Contribution of design choices to post-training gains

Weakness 2:
The paper does not provide a downstream ablation of these components (the explicit reasoning-tree construction and the agentic refinement stage) or a comparison with a simpler synthetic training strategy.

Question 2:
Do the authors have a comparison with a simpler post-training strategy using the same base model, or an ablation that quantifies the downstream contribution of the agentic refinement stage?

  • We sincerely thank the reviewer for the valuable suggestion regarding our post-training approach. In the revised version, we present and analyze the results of the tree-free and tree-guided generation comparison in Appendix G, Table 11 and the agentic refinement ablation in Section 4.6, Table 5.

1. Tree-guided post-training and a simpler tree-free baseline

  • An explicit reasoning tree is central to our data construction pipeline. Its branches encode the guideline’s decision logic, and each case and its reference pathway follow the same branch decisions. This gives us a straightforward and effective way to keep all three components consistent. To test whether this design improves downstream performance, we compared tree-guided generation against a tree-free direct generation baseline. In the tree-free condition, GPT-5.4 generated a guideline directly from a fictional disease definition. For each case, it then used that guideline to generate a vignette, a sequence of clinician–patient questions and answers, and a final decision. The generator had no access to the reasoning trees. In the tree-guided condition, we used explicit reasoning trees and their paths to construct the training examples. Both conditions omitted agentic refinement.

  • We matched the disease definition, language, and case quota for each of the 200 guidelines, yielding 6,486 synthetic cases per condition. In both runs, we fine-tuned Qwen3.5-9B on all real-world training data together with the corresponding synthetic cases. In the revised version, we report the results for both generation methods in Appendix G, Table 11.

    Fine-Tuning DataPCC (%) ↓Acc (%) ↑SR (%) ↑
    All Real-World Data (Baseline)40.4851.3499.81
    All Real-World Data + Tree-Free Cases41.8848.4399.97
    All Real-World Data + Tree-Guided Cases40.1952.4999.44
  • Adding tree-free cases to the real-world training data worsened pathway consistency and outcome accuracy relative to the real-world-only baseline. PCC rose by 1.40 percentage points and accuracy fell by 2.91 points. Adding the same number of tree-guided cases lowered PCC and raised accuracy relative to that baseline. In the matched comparison, tree-guided generation lowered PCC by 1.69 points and raised accuracy by 4.06 points relative to tree-free generation, suggesting that explicit reasoning paths provide more effective supervision for guideline adherence.

2. Agentic refinement ablation

  • We report results from an early ablation study of agentic refinement. In this experiment, we compared models trained on synthetic data with and without refinement, keeping all other experimental settings identical. In the revised version, we report the results with and without agentic refinement in Section 4.6, Table 5.

    Training Data ConditionPCC (%) ↓Acc (%) ↑SR (%) ↑
    Without Agentic Refinement47.8043.0698.91
    With Agentic Refinement45.0946.8999.85
  • We observe clear improvements across all three metrics. Refinement reduces PCC by 2.71 percentage points and increases accuracy by 3.83 points. Under matched training conditions, the refined guidelines provided more effective supervision for pathway consistency and outcome accuracy.

Comments on the paper

2 comments

Dear reviewers,

We’re about halfway through the discussion period, and the authors have put considerable time into responding to your feedback. Please read their rebuttal and leave a brief comment letting the authors know whether it addressed your questions or concerns and what, if anything, remains open.

Thank you.

AuthorsOpenReview ↗

Summary of Our Revisions and Responses

Dear Reviewers,

Thank you for your careful reading of our paper and for the questions you raised about SAGA-CDP. We have responded to each of your questions and highlighted all changes in blue in the revised version. Below, we summarize the key revisions and findings from our rebuttal:

Generalization across clinical tasks and base models, and robustness across judge models and training seeds (following Reviewers ULCM, wEqR, and ZLUB):

  • The results show that fine-tuning improved performance in all three settings across the two external benchmarks, NPCBench and MediQ, benefited both Qwen and Gemma models, and yielded consistent gains across two LLM judges and three training seeds (Section 4.7, Table 6 and Appendix F, Tables 8–10).

Ablation studies of guideline matching, explicit reasoning trees, and agentic refinement (following Reviewers ULCM and wEqR):

  • The ablations show that matching guidelines to training dialogues improved pathway consistency and outcome accuracy, and that explicit reasoning trees and agentic refinement each provided further gains in both metrics (Section 4.6, Tables 4–5 and Appendix G, Table 11).

Human validation covering synthetic data quality, potential leakage, and agreement between the LLM judge and human judgments (following Reviewers wEqR and ZLUB):

  • The expert audit found clear inconsistencies in fewer than 0.5% of the reviewed pathways and vignettes, LLM-based and manual checks found no disease-concept overlap with the evaluation set, and human–model agreement on pathway pairs reached 90%, matching the agreement rate between human annotators (Appendix D).

As the discussion period draws to a close, we kindly ask you to review the revised version of our paper and our responses to your comments. We would be very happy to answer any further questions. Thank you again for the time and care you have devoted to our work.

Sincerely,
Authors of Paper #24208

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.