- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds a pipeline that learns clinical guideline-following behavior from synthetic "fictional guidelines": guidelines are generated by an LLM, refined via embedding-distance filtering, and used to fine-tune a model. The authors argue that real guidelines are unnecessary for acquiring such behavior and report large PCC gains on the MEGA-CDP benchmark.
Critical strengths
The problem setting is novel: demonstrating that guideline-following behavior can be acquired from fictional synthetic text opens a viable route for clinical scenarios lacking real guidelines, and the reported benchmark gains are substantial.
Critical weaknesses
All fine-tuning uses a single random seed with no repeats or significance testing, so the uncertainty of the headline comparisons cannot be assessed.
In addition, every experiment uses only one base model (Qwen3.5-9B) on one benchmark (MEGA-CDP), and the core PCC metric is scored automatically by another LLM (Qwen3-8B) with no human validation; together these choices leave the reality of the gains unsupported.
Questions for authors
How is leakage between the synthetic data and the evaluation set ruled out? What is the seed-to-seed variance? Can you provide agreement data between the LLM judge and human judgments?
8 comments
Response to Weakness 2 & Question 3 of Reviewer ZLUB
Agreement between the LLM judge and human judgments
Weakness 2:
the core PCC metric is scored automatically by another LLM (Qwen3-8B) with no human validationQuestion 3:
Can you provide agreement data between the LLM judge and human judgments?
-
We fully understand your concern about how closely the LLM judge’s assessments agree with human judgments. Our study focuses on guideline-adherence training and uses the existing evaluation procedure from MEGA-CDP, so a systematic assessment of human-model agreement falls outside its scope. Nevertheless, we can provide a set of results from an analysis of how closely the LLM judge’s assessments of MEGA-CDP pathways agree with human judgments. These results directly address your question.
-
In this study, two human judges independently reviewed the same 200 pathway pairs from the multi-turn setting. For each case, they assessed two predicted pathways against a common reference and indicated which followed it more closely. The experiment recorded agreement whenever a human judge selected the pathway with the lower PCC.
-
In this experiment, Group 1 contained 100 randomly sampled pairs from eligible outputs across 16 base models. Group 2 contained 100 pairs from Claude-Sonnet-4.6 and Qwen3-Next-80B-A3B-Instruct, allowing us to examine agreement when comparing two stronger models. Each group comprised 50 English and 50 Chinese cases. We report agreement for each group and across all 200 pairs.
Group Comparison Agreement Rate Group 1 Combined Human–Model Agreement 89.0% Inter-Human Agreement 88.0% Group 2 Combined Human–Model Agreement 91.0% Inter-Human Agreement 92.0% Overall Combined Human–Model Agreement 90.0% Inter-Human Agreement 90.0% -
Across both groups, human–model agreement was 90.0%, the same rate observed between the two human judges. These results indicate that the LLM judge can reliably assess how closely a predicted pathway follows the reference pathway.
Response to Weakness 2 of Reviewer ZLUB (1/2)
Evaluation on a second model family
Weakness 2:
every experiment uses only one base model (Qwen3.5-9B)
-
We extended training to Gemma-3-12B-IT, using the same mixture of real-world and fictional guideline data. Comparing the fine-tuned model against its own baseline allows us to examine whether the approach also benefits a backbone outside the Qwen family. In the revised version, we report all three Gemma metrics in Appendix F.1, Table 8.
Model PCC (%) ↓ Acc (%) ↑ SR (%) ↑ Baseline (Gemma-3-12B-IT) 60.05 10.86 99.03 Ours (Gemma-3-12B-IT) 44.34 46.61 99.76 -
The Gemma model achieves a 15.71-percentage-point reduction in PCC and a 35.75-percentage-point increase in accuracy. Its success rate also increases from an already high baseline. These results provide evidence that the training benefits are not restricted to the Qwen model family.
Response to Weakness 1 & Questions 1–2 of Reviewer ZLUB
Stability across training seeds
We sincerely thank the reviewer for the constructive comments. To test whether the results were stable across random seeds, we repeated fine-tuning of Qwen3.5-9B under each of the three training configurations in Table 1 using seeds 40, 41, and 42. We evaluated all nine runs on the MEGA-CDP test set. The percentages in the training configurations refer to the fraction of each data pool used. We report the mean, sample standard deviation (SD), and sample variance across the three seeds. In the revised version, we report the individual scores for all nine runs in Appendix F.3, Table 10.
Across the three training configurations, the sample variance over seeds ranged from 0.0046 to 0.0834 for PCC, from 0.0265 to 0.4794 for Acc, and from 0.0009 to 0.0073 for SR. These low variances indicate that performance was stable across the three seeds tested.
Disease-concept overlap with the evaluation set
Thank you for these perceptive comments. We combined LLM-based review and manual inspection of the entire synthetic dataset, focusing on whether its disease concepts overlapped with those represented in the MEGA-CDP evaluation set. Neither check identified any overlap in disease concepts.
Generation starts from five categories of entirely fictional disease concepts in Appendix B: Fantasy, Science Fiction, Xuanhuan, Famous Fictional Works, and Realistic Medical Variants. The generation instructions require fictional disease identities and constrain synthesis to preserve the fictional setting. These choices are designed to avoid reproducing real-world disease entities and reduce disease-specific overlap. The LLM-based and manual checks provide additional empirical support for the finding that no leakage was detected.