- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a clinical AI system that uses clinical guidelines to guide decisions as patient information changes over time. It stores guideline knowledge in memory and selects the most relevant parts for each step of the decision process. The paper also introduces a benchmark for testing whether clinical AI systems make decisions that follow guidelines throughout a patient care process. The method performs better than most baselines on the cancer datasets they evaluate.
Critical strengths
1- The paper introduces a clear conceptual shift by treating clinical guidelines as state dependent control logic for sequential decision making. 2- The paper presents a novel evaluation framework by progressively revealing multimodal evidence and assessing complete clinical trajectories.
Critical weaknesses
1- The paper addresses a timely problem, and the different components of the proposed framework are combined in a coherent and technically sound way. However, the overall method feels more like a combination of existing techniques applied to a new problem than a fundamentally novel methodological contribution.
2- The evaluated cohorts are relatively small. For datasets of this size, reporting only point estimates is insufficient, and I would expect mean and standard deviation values across repeated runs or resampling.
3- A major concern is the reliance on LLMs both to construct the ground truth trajectories and to evaluate model performance. This creates a potential source of evaluator bias, especially in cases where the same or closely related models are involved in both prediction and evaluation.
4- The train, validation, and test separation is not described clearly enough. Given the use of learned guideline representations and benchmark construction procedures, the paper should make the data separation and potential sources of leakage explicit.
Questions for authors
1- Given the relatively small cohort sizes, could you report variability across runs or resampling, such as mean and standard deviation, rather than only single point estimates?
2- How were the ground truth trajectories generated and validated, and how much of this process relies on LLM generated outputs?
3- Could you clarify whether the same or closely related LLMs are used both for generating predictions and evaluating them? If so, how do you mitigate potential evaluator bias or self preference?
4- Could you provide a clearer description of the train, validation, and test splits, including whether any patient, guideline, or trajectory information can overlap across these partitions?
5 comments
Response to Reviewer sZSU: 2) Cohort Size and Statistical Variability
2) Cohort Size and Statistical Variability
Critical Weaknesses and Question:
2- The evaluated cohorts are relatively small. For datasets of this size, reporting only point estimates is insufficient, and I would expect mean and standard deviation values across repeated runs or resampling.
1- Given the relatively small cohort sizes, could you report variability across runs or resampling, such as mean and standard deviation, rather than only single point estimates?
Response:
We thank the reviewer for this suggestion. We first note that CPGTrajBench is comparable in scale to representative case-based multimodal clinical benchmarks, as summarized below.
| Benchmark | Cases | Clinical Modalities | Interaction |
|---|---|---|---|
| AgentClinic-NEJM [1] | 120 | Text, EHR, Pathology | Dialogue/Simulation |
| MTBBench [2] | 66 (26 multimodal + 40 longitudinal) | Text, Pathology, Molecular | Executable Environment |
| CPGTrajBench (Ours) | 109 | Text, EHR, Radiology, Pathology, Molecular | Executable Environment |
Unlike conventional static QA benchmarks, CPGTrajBench integrates five clinical modalities across three cancer types, requiring progressive evidence acquisition and multi-step clinical planning. Each reference trajectory is further validated by clinical experts. The construction of this multimodal, interactive, and guideline-grounded benchmark is itself a core contribution of our work.
Nevertheless, we acknowledge that 109 cases remain relatively limited for statistical estimation and generalizability. To address this concern, we have updated Table 2 to report mean ± bootstrap SD from 10,000 within-cohort patient-level resamples (seed 42).
Specifically, CPGAgent achieves 47.957 ± 1.808 on LUAD (30 cases), 49.663 ± 0.788 on UCEC (40 cases), and 55.519 ± 1.622 on NPC (39 cases), with an overall score of 51.289 ± 0.813. The same bootstrap procedure is applied to all baselines.
References
[1] Schmidgall, Samuel, et al. “AgentClinic: A Multimodal Benchmark for Tool-Using Clinical AI Agents.” npj Digital Medicine, vol. 9, 2026, article no. 499, doi:10.1038/s41746-026-02674-7.
[2] Vasilev, Kiril, et al. “MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology.” Advances in Neural Information Processing Systems 38, Datasets and Benchmarks Track, 2025, pp. 12217–12246, doi:10.52202/085713-0368.
Response to Reviewer sZSU: 3) Trajectories Generation
3) Trajectories Generation
Questions For Authors:
2- How were the ground truth trajectories generated and validated, and how much of this process relies on LLM generated outputs?
Response:
As detailed in Section 4.2 and Appendices G–H, reference trajectories are constructed through iterative LLM generation, step-level verification, and clinical expert review.
First, LLM-based construction and verification. At each step, a DeepSeek-V4-Pro Generator proposes guideline-grounded actions and state transitions based on the current patient state. A separately invoked DeepSeek-V4-Pro Verifier checks these proposals against the complete patient record, identifies inconsistencies, and supports state correction before the next step.
Second, clinical expert validation. As detailed in the newly added Appendix H, clinicians review complete trajectories for clinical appropriateness, guideline consistency, and cross-step coherence. Localized errors are corrected, while trajectories with major unresolved errors are rejected. Of 135 trajectories reviewed, 109 were accepted and 26 failed clinical expert review (Table 10).
Thus, while trajectory generation and initial verification rely on LLMs, the final reference trajectories undergo independent clinical expert validation instead of relying solely on LLM outputs.
Response to Reviewer sZSU: 1) Novelty
1) Novelty
Critical Weaknesses:
1- The paper addresses a timely problem, and the different components of the proposed framework are combined in a coherent and technically sound way. However, the overall method feels more like a combination of existing techniques applied to a new problem than a fundamentally novel methodological contribution.
Response:
Thank you for this suggestion. While our framework builds upon established techniques, its novelty lies in their integration for state-aware guideline control, particularly through Dynamic Anchor Attention (DAA). Unlike Q-Former, which uses fixed learned query embeddings, our DAA dynamically selects the most decision-relevant guideline memory as the anchor based on the current patient state. Then, we enable State-aware knowledge integration through two stages. First, the patient state guides the extraction of decision-relevant information from the anchor guideline memory, effectively querying what matters for the current patient state. The resulting state-aware representation then attends to other selected guideline memories to identify complementary information from additional sections. This allows the Planner to focus on key clinical decisions while incorporating relevant cross-section knowledge.