acceptodds
Under review as a conference paper at ICLR 2027

Latent Decision Flow Rewards for Clinical Guideline Reasoning

Abstract

Clinical practice guidelines encode evidence-based recommendations as conditional decision flows, which makes them a natural source of process supervision for training language models. Using this supervision is difficult for two reasons. A decision flow consistent with a guideline does not by itself determine the correct answer for a given patient, which we refer to as the path–answer gap, and several valid decision flows can support the same clinical decision, which we refer to as path multiplicity. Under these conditions, answer rewards provide no signal about the decision process, supervised training on annotated flows constrains the model to a single trajectory and reduces answer accuracy, and process rewards computed against one reference flow penalize alternative valid reasoning. We introduce GUIDER, a joint controller–policy optimization approach that learns guideline-adherent clinical reasoning while accounting for these challenges. The controller learns to generate plausible decision flows from guideline-derived supervision and provides process-level supervision to the policy, while the policy learns to generate clinical reasoning and final decisions. By aggregating process rewards over multiple controller-generated decision flows, \name accounts for path multiplicity, and by gating process rewards on answer correctness, it couples guideline adherence with correct final answers. Joint optimization allows process supervision to evolve with the policy instead of relying on a fixed reference trajectory. The policy is never conditioned on an oracle decision path, so GUIDER requires no access to guidelines at inference time. On MedGUIDE, \name with Qwen3-14B achieves 82.5 weighted accuracy and 73.5 guideline adherence, improving over the strongest clinical reinforcement learning baseline by 3.3 and 3.1 points, respectively. \name also improves over CRPO, MRPO, and FaithMed on the out-of-domain benchmarks AMEGA-LLM, RealMedQA, and CancerGUIDE with Qwen3-14B and Gemma4-12B backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
guideline reasoning, finetuning, reward shaping

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5sVSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how structured decision-making workflows from Clinical Practice Guidelines (CPGs) can provide process-level supervision for clinical reasoning in large language models (LLMs). The authors identify two key challenges: Path Multiplicity, where multiple valid reasoning paths may support the same clinical decision, and the Path–Answer Gap, where guideline-consistent reasoning does not necessarily lead to a correct answer. To address these issues, they propose GUIDER, a framework that jointly optimizes a Controller and a Policy. The Controller learns reference decision paths through supervised learning and generates multiple candidate paths for each case. These paths are weighted by their coverage of the reference path, and token-level Jaccard similarity is used to construct a Multi-Flow Process Reward. An Answer-Gated Reward further ensures that additional process rewards are granted only when the final answer is correct. Both components share model parameters and are jointly optimized using supervised learning and GRPO.

Experiments on MedGUIDE, AMEGA-LLM, RealMedQA, and CancerGUIDE, using Qwen3-14B and Gemma4-12B, demonstrate that GUIDER outperforms existing clinical reasoning reinforcement learning methods, including CRPO, MRPO, and FaithMed. On Qwen3-14B, GUIDER achieves 82.5 Weighted Accuracy and 73.5 Guideline Adherence on MedGUIDE. Overall, the paper addresses a practically important problem with a reasonably novel approach and promising empirical results. However, it remains unclear whether the proposed method truly resolves Path Multiplicity and how much the multi-path process reward improves performance compared with simpler reference-path supervision. More thorough theoretical analysis and experimental validation are needed to substantiate these claims.

Critical strengths
  1. This paper focuses on the correctness of the clinical reasoning process rather than solely on final answer accuracy. This distinction is particularly important in healthcare, where a model may arrive at the correct answer through incomplete reasoning or reasoning that does not align with clinical guidelines. The authors further identify two key challenges, Path Multiplicity and the Path–Answer Gap, providing a clear problem formulation for guideline-based process supervision.
  2. The paper integrates guideline-derived decision path supervision with reinforcement learning. Specifically, the Controller generates candidate decision paths, which are then used to construct process rewards for the Policy. Unlike approaches that require retrieving or executing clinical guidelines during inference, GUIDER incorporates guideline supervision only during training, eliminating the need for an additional Controller or guideline access at inference time. This design offers potential advantages in deployment efficiency.
  3. GUIDER achieves improvements on both Qwen3-14B and Gemma4-12B and outperforms the compared reinforcement learning baselines across multiple clinical reasoning benchmarks. Notably, on MedGUIDE, GUIDER improves both answer accuracy and guideline adherence, whereas direct Answer + Flow SFT improves guideline adherence but substantially reduces answer accuracy. These findings suggest that jointly optimizing answer correctness and process-level supervision is a promising direction for further research.
Critical weaknesses

The central motivation behind Path Multiplicity is that multiple clinically valid, guideline-consistent decision paths may exist for the same case, and therefore, a single reference path should not serve as the sole supervision target.

However, in the proposed method, each Controller-generated candidate path z(m)z^{(m)} is still scored based on its coverage of the single annotated reference path z∗z^*:

sm=λPRM1∣V(z)∣∑v∈V(z)match⁡(v,z(m))s_m = \lambda_{\mathrm{PRM}} \frac{1}{|V(z^)|} \sum_{v \in V(z^)} \operatorname{match}(v,z^{(m)})

These scores are then used to compute candidate path weights, which are combined with the Jaccard similarity between the Policy-generated path and each candidate path to construct the final process reward:

Rproc(z^)=∑m=1Mwm[sm]+J(z^,z(m))R_{\mathrm{proc}}(\hat{z}) = \sum_{m=1}^{M} w_m [s_m]_+ J(\hat{z},z^{(m)})

This means that even if a candidate path is clinically valid, it may receive a low reference-path support score simply because it shares relatively few nodes with the annotated reference path, thereby reducing its contribution to the final reward.

Although incorporating multiple candidate paths can partially alleviate the limitations of single-path matching, the current design does not guarantee that structurally different but equally clinically valid reasoning paths will be appropriately rewarded.

The paper uses token-set Jaccard similarity to measure the consistency between Policy-generated paths and Controller-generated candidate paths. Although this metric is simple to implement and computationally efficient, it remains unclear whether it can effectively capture the correctness of clinical decision-making processes.

Questions for authors

See the weaknesses.

3 comments

[1/2]:

W1: We thank the reviewer for this suggestion. We agree that demonstrating clinically valid alternative paths, rather than merely stochastic paraphrases, is important for validating the path-multiplicity motivation. To address this concern, we conduct two additional evaluations.

Alternative-flow recognition:

MedGUIDE is constructed from 55 NCCN clinical decision trees spanning 17 cancer types. The dataset creators convert these trees into structured JSON, enumerate and manually review root-to-leaf decision paths, and generate patient scenarios conditioned on these paths. Each question therefore has an underlying guideline-derived reference trajectory.

We select 200 MedGUIDE test cases and use their corresponding guideline trees to construct four trajectories: (i) the reference path (zAz_A), (ii) a valid reordering of independent decisions (zBz_B), (iii) a structurally distinct, guideline-consistent alternative applicable to the same patient (zCz_C), and (iv) an invalid condition–action flow (zDz_D). We construct alternative clinical flows by identifying cases that share a common decision path of more than two nodes and using the shared subpath as a clinically plausible alternative. While imperfect, this provides a practical approximation without requiring additional clinical expert annotation. We follow the same approach for (zC)(z_C), constructing an alternative flow from a shared subpath within the guideline decision tree. Its clinical validity is assumed following the NCCN decision tree structure.

We compare GUIDER against single-reference matching, Rsingle(z)=J(z,zA)R_{single}(z)=J(z,zA).

FlowSingle-reference reward ↑GUIDER reward ↑
Reference (zAz_A)1.000.86
Valid reordered (zBz_B)1.000.86
Distinct valid (zCz_C)0.430.72
Invalid (zDz_D)0.210.25

GUIDER assigns higher reward to the distinct valid alternative than single-reference matching (0.72 vs. 0.43). The invalid flow receives substantially lower reward (0.25). These results suggest that multi-flow aggregation better accommodates valid alternative reasoning paths, although reference-dependent weighting may still penalize alternatives with limited lexical overlap.

2. Coverage of clinically valid alternative paths

We further evaluate whether increasing the number of controller samples improves the coverage of structurally distinct, clinically valid alternatives on the 200 test samples. We introduce ValidAltCoverage@M, defined as the percentage of cases containing at least one clinically valid controller proposal that is structurally distinct from the annotated reference.

MMedGUIDE WAcc. (%) ↑Adherence (%) ↑ValidAltCoverage@M (%) ↑
180.969.218.4
281.771.129.6
482.573.541.8
882.673.748.3

Increasing M from 1 to 4 improves ValidAltCoverage from 18.4% to 41.8%, alongside improvements in weighted accuracy and guideline adherence. This provides evidence that additional controller samples increase the likelihood of identifying clinically valid alternatives rather than simply reproducing the reference trajectory. Together, these experiments demonstrate that GUIDER can assign meaningful process credit to distinct valid alternatives and that multi-flow sampling improves their coverage. We acknowledge that reference-anchored scoring may still underreward valid paths with limited lexical overlap, and that the current reward does not formally guarantee clinical validity. Nevertheless, these evaluations provide direct empirical support for the benefits of multi-flow supervision beyond stochastic paraphrasing.

[2/2]:

W2: We thank the reviewer for highlighting these concerns. We agree that token-set Jaccard similarity is a lightweight approximation of clinical reasoning compatibility and does not explicitly capture negation, medical synonyms, or temporal dependencies. However, GUIDER's process reward combines reference-node coverage with weighted agreement across multiple controller-generated trajectories, rather than relying on a single Jaccard comparison. To investigate these limitations, we conduct additional experiments examining the sensitivity of GUIDER's process reward to clinically incorrect trajectories and its ability to accommodate valid alternative decision flows.

Error discrimination analysis

MedGUIDE contains structured clinical decision flows derived from 55 NCCN decision trees spanning 17 cancer types. We select 200 test cases and introduce three types of corruption:

  1. Unsupported-node injection: Add a clinical recommendation not supported by the applicable guideline.
  2. Contradicted-action injection: Introduce an action conflicting with the patient's clinical conditions.
  3. Condition–action swap: Associate an action with an incorrect clinical condition while preserving the original clinical concepts.

We measure Error Discrimination Rate (EDR), the percentage of cases where the clean trajectory receives a strictly higher reward than its corrupted counterpart. Both are evaluated using identical controller proposals and weights, without retraining. We additionally replace GUIDER's token-set Jaccard similarity with a frozen Qwen3.5-4B judge that scores semantic compatibility, keeping the remaining reward components unchanged.

Corruption typeCoverage-only EDR (%) ↑GUIDER (Jaccard) EDR (%) ↑GUIDER (LLM judge) EDR (%) ↑
Unsupported node0.084.585.3
Contradicted action0.089.091.3
Condition–action swap0.012.558.5

Reference coverage cannot distinguish these errors because the original reference nodes remain present. GUIDER with Jaccard discriminates unsupported nodes (84.5%) and contradictory actions (89.0%) but struggles with condition–action swaps (12.5%). Jaccard-based GUIDER effectively detects unsupported recommendations (84.5%) and contradictory actions (89.0%) but exhibits limited sensitivity to condition--action swaps (12.5%). The LLM judge improves discrimination across all corruption types, with the largest gain for condition--action swaps (58.5%), highlighting the benefit of semantic compatibility scoring for capturing clinical dependencies.

Computational efficiency

We compare training time under identical data, architecture, and optimization settings, varying only the similarity function.

Similarity functionTraining time (hours) ↓
Jaccard10
Qwen3.5-4B judge25

The LLM judge requires 2.5× longer training time (25 vs. 10 hours). Thus, Jaccard offers a more computationally efficient alternative, although it is less sensitive to complex clinical dependencies.

Dear Reviewer 5sVS,

As the discussion deadline is approaching, we would like to kindly follow up on our rebuttal and the additional experiments addressing your concerns regarding path multiplicity, alternative-flow coverage, and the effectiveness and limitations of Jaccard-based process rewards.

We appreciate your positive assessment of our work and your constructive suggestions, which have helped us strengthen the empirical analysis and refine our claims.

We hope our responses have addressed your concerns and would greatly appreciate your feedback. We would be happy to provide any further clarifications before the discussion period concludes.

Thank you again for your time and consideration.

Reviewer_hddoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GUIDER, a reinforcement-learning framework for clinical guideline reasoning. It argues that guideline paths are latent and non-unique, while a guideline-consistent path may not uniquely determine the task answer. GUIDER trains a shared-parameter controller to generate candidate decision flows, rewards policy flows by their weighted token overlap with those samples, and gates process reward by answer correctness. Experiments on MedGUIDE and three transfer benchmarks report improved accuracy and guideline-adherence scores over clinical RL baselines across Qwen3-14B and Gemma4-12B.

Critical strengths
  1. The paper clearly articulates two meaningful difficulties in using guidelines as process supervision: multiple valid reasoning paths can support an answer, and path validity need not imply answer correctness. The answer-gated process-reward formulation is a sensible attempt to prevent rewarding guideline-like reasoning that does not produce the correct decision.

  2. The empirical study is reasonably broad: it uses two backbones, reports both answer and adherence metrics, and includes several external benchmarks. The comparison with flow-supervised SFT is particularly useful: it shows that directly training on one annotated flow can raise adherence while substantially reducing answer accuracy.

Critical weaknesses
  1. The main claim of addressing path multiplicity is not yet convincingly supported by the stated reward. Each controller proposal is weighted by its coverage of the single annotated reference flow, and a proposal with no matched reference nodes receives zero process credit. Moreover, the policy-controller agreement is token-set Jaccard similarity, which ignores order and may not recognize clinically equivalent but lexically different paths. Thus, the method appears to soften supervision around one reference path rather than validate or reward genuinely alternative guideline-consistent paths. Since controller and policy also share parameters and the controller has no guideline access beyond its training supervision, it is unclear why sampled flows should constitute independently valid alternatives rather than stochastic paraphrases or reward-hackable overlap.

  2. Important implementation and evaluation details prevent attributing the gains confidently. The manuscript does not precisely define match(v,z) or provide the value/selection procedure for λPRM\lambda_{\rm PRM}. Tables 1, 3, and 4 omit uncertainty despite the stated three-seed protocol, and it is unclear whether MM, flow length, and other choices were selected on a validation set rather than the reported test benchmarks. The closest internal baseline, answer-only SFT followed by answer-only GRPO, reaches 80.1 WAcc., making the direct improvement 2.4 points rather than the headline 3.3-point comparison to MRPO. Finally, several results rely on a single LLM judge, while potential guideline/data overlap between the training sources and transfer benchmarks is not analyzed.

Questions for authors
  1. How exactly is match(v,z) implemented (string match, embedding threshold, LLM judge, or another procedure), and how was λPRM\lambda_{\rm PRM} selected? Please provide enough detail to reproduce the reward.

  2. Can you directly demonstrate that the controller produces clinically valid alternatives rather than paraphrases of the annotated path? In particular, evaluate on cases with multiple independently verified valid guideline paths, and report whether the method rewards each alternative relative to a single-reference process reward.

  3. Please add three-seed uncertainty/statistical tests for Tables 1, 3, and 4; state the validation protocol used to choose MM, maximum flow length, and reward hyperparameters; and include the answer-only SFT→GRPO baseline in the main cross-benchmark comparison. Please also document dataset/guideline overlap checks and judge-robustness or human-evaluation results.

5 comments

[1/2]:

W1: We thank the reviewer for this careful analysis. We agree that GUIDER anchors process credit to the annotated reference. Proposals with zero reference coverage receive zero weight, and token-set Jaccard similarity is invariant to node order and sensitive to lexical variation in clinical terms. We therefore describe GUIDER as a relaxation of single-path supervision and interpret controller samples as guideline-supervised proposals, distinct from independently validated clinical pathways. The structure of the reward limits reward hacking through token overlap. The coverage score sₘ depends only on the controller proposal and the reference flow, so the policy influences the reward solely through the Jaccard term. Jaccard similarity decreases when a flow contains extra tokens, because these tokens enlarge its union with each proposal. Tokens outside a proposal therefore strictly lower J(ẑ, z⁽ᵐ⁾), whereas a recall measure would give equal or higher credit to the same flow. For tokens taken from a second proposal, the proposition in our response to Reviewer 1 (W4) shows that, for two proposals, the hybrid flow obtains at most the reward of the proposal with the larger weight. Policy completions are also limited to 256 tokens and N = 4 decision nodes, which bounds the length of any padded flow. The error discrimination analysis (Table 3b) confirms this behavior, since an injected unsupported node lowers the GUIDER reward in 84.5% of cases. Answer gating further restricts process credit to completions with correct final answers. Sampled flows also include valid alternatives to the reference. In the alternative-flow analysis (Table 3a), GUIDER gives a structurally distinct valid alternative a reward of 0.72, compared with 0.43 under single-reference matching. The fraction of cases with at least one structurally distinct, clinically valid controller proposal rises from 18.4% at M = 1 to 41.8% at M = 4 (response to Q2). Over the same range, MedGUIDE weighted accuracy increases from 77.1 to 82.5 and guideline adherence from 70.2 to 73.5 (Table 4). These results indicate that sampling several controller flows lets the process reward credit valid alternatives beyond the annotated path, within the limits of reference anchoring.

Q1: We thank the reviewer for requesting these clarifications.

  1. Node matching: match(v,z)match(v,z) uses deterministic lexical matching rather than embeddings or an LLM judge. Text is lowercased, tokenized, and filtered for stopwords and short tokens. We compute reference-node token recall: match(v,z)=∣Tok(v)∩Tok(z)∣/∣Tok(z)∣match(v,z) = |Tok(v) ∩ Tok(z)| / |Tok(z)| A reference node is considered matched when its recall is ≥ 0.28. The coverage score is the fraction of matched reference nodes.
  2. Reward scaling: We use a fixed λPRMλ_{PRM} = 0.5, giving each controller proposal a score of sm=0.5×coverages_m = 0.5 × coverage. This value was selected heuristically before final evaluation to keep process credit conservatively scaled relative to the unit answer reward. It was not tuned on the test set.
  3. Multi-flow reward: We sample M = 4 controller flows and aggregate their coverage-weighted agreement with the policy flow using token-set Jaccard similarity, as defined in Eqs. (3)–(4). Process credit is gated by answer correctness.

[2/2]:

Q2: We thank the reviewer for this suggestion. We agree that demonstrating clinically valid alternative paths, rather than merely stochastic paraphrases, is important for validating the path-multiplicity motivation. To address this concern, we conduct two additional evaluations.

Alternative-flow recognition:

MedGUIDE is constructed from 55 NCCN clinical decision trees spanning 17 cancer types. The dataset creators convert these trees into structured JSON, enumerate and manually review root-to-leaf decision paths, and generate patient scenarios conditioned on these paths. Each question therefore has an underlying guideline-derived reference trajectory.

We select 200 MedGUIDE test cases and use their corresponding guideline trees to construct four trajectories: (i) the reference path (zAz_A), (ii) a valid reordering of independent decisions (zBz_B), (iii) a structurally distinct, guideline-consistent alternative applicable to the same patient (zCz_C), and (iv) an invalid condition–action flow (zDz_D). We construct alternative clinical flows by identifying cases that share a common decision path of more than two nodes and using the shared subpath as a clinically plausible alternative. While imperfect, this provides a practical approximation without requiring additional clinical expert annotation. We follow the same approach for (zC)(z_C), constructing an alternative flow from a shared subpath within the guideline decision tree. Its clinical validity is assumed following the NCCN decision tree structure.

We compare GUIDER against single-reference matching, Rsingle(z)=J(z,zA)R_{single}(z)=J(z,zA).

FlowSingle-reference reward ↑GUIDER reward ↑
Reference (zAz_A)1.000.86
Valid reordered (zBz_B)1.000.86
Distinct valid (zCz_C)0.430.72
Invalid (zDz_D)0.210.25

GUIDER assigns higher reward to the distinct valid alternative than single-reference matching (0.72 vs. 0.43). The invalid flow receives substantially lower reward (0.25). These results suggest that multi-flow aggregation better accommodates valid alternative reasoning paths, although reference-dependent weighting may still penalize alternatives with limited lexical overlap.

2. Coverage of clinically valid alternative paths

We further evaluate whether increasing the number of controller samples improves the coverage of structurally distinct, clinically valid alternatives on the 200 test samples. We introduce ValidAltCoverage@M, defined as the percentage of cases containing at least one clinically valid controller proposal that is structurally distinct from the annotated reference.

MMedGUIDE WAcc. (%) ↑Adherence (%) ↑ValidAltCoverage@M (%) ↑
180.969.218.4
281.771.129.6
482.573.541.8
882.673.748.3

Increasing M from 1 to 4 improves ValidAltCoverage from 18.4% to 41.8%, alongside improvements in weighted accuracy and guideline adherence. This provides evidence that additional controller samples increase the likelihood of identifying clinically valid alternatives rather than simply reproducing the reference trajectory. Together, these experiments demonstrate that GUIDER can assign meaningful process credit to distinct valid alternatives and that multi-flow sampling improves their coverage. We acknowledge that reference-anchored scoring may still underreward valid paths with limited lexical overlap, and that the current reward does not formally guarantee clinical validity. Nevertheless, these evaluations provide direct empirical support for the benefits of multi-flow supervision beyond stochastic paraphrasing.

Q3: Thank you for the suggestion. We have added uncertainty estimates to Tables 1 and 2 in the main text of the revised manuscript.

Dear Reviewer hddo,

As the discussion deadline is approaching, we would like to kindly follow up on our rebuttal and the additional experiments addressing your concerns regarding path multiplicity, alternative-flow coverage, and reward formulation.

We appreciate your constructive suggestions, which have helped us better characterize the capabilities and limitations of GUIDER and improve the clarity of our manuscript.

We hope our responses have addressed your concerns and would greatly appreciate your feedback. We would be happy to provide any further clarifications before the discussion period concludes.

Thank you again for your time and consideration.

Reviewer_uN5zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes GUIDER, an SFT+RL framework designed to align LLMs with clinical practice guidelines (CPGs) without requiring guideline retrieval at inference. This work identifies two challenges in clinical reasoning decision training: (1) path multiplicity, where various different trajectories can lead to the same correct clinical outcome; and (2) path–answer gap, where learning CPG’s trajectories doesn’t guarantee the correct final answer. To address these issues, GUIDER utilizes a joint optimization approach with two branches: (1) SFT branch to train the LLM as a controller generating clinical trajectory candidates aligned with CPGs; and (2) GRPO branch to train the LLM as a policy with final answer reward and final-answer-gated process reward.

Experiments on Qwen3-14B and Gemma4-12B across MedGUIDE, AMEGA-LLM, RealMedQA, and CancerGUIDE show that while direct SFT on reference trajectories degrades answer accuracy, GUIDER improves both answer accuracy and LLM-judged guideline adherence over existing clinical RL baselines.

Critical strengths

Clear presentation. The paper is well written, logically structured, and easy to follow throughout.

Well-motivated challenges. The paper identifies critical and realistic challenges in training clinical LLMs, highlighting key bottlenecks that are practically important and theoretically sound. Moreover, the motivation is well formalized.

Broad evaluation. 4 benchmark against various SOTA RL baselines and frontier models. The

Critical weaknesses

W1. Coarse similarity measurement in process rewards. The token-set Jaccard index J(z^,z(m))J(\hat{z}, z^{(m)}) measures alignment between a rollout from the policy and a set of candidate trajectories from the controller, but relying purely on surface-level token overlap. This could cause the reward signal to be distorted:

• Semantic inversion: J(z^,z(m))J(\hat{z}, z^{(m)}) allows semantically opposed statements (e.g., “no evidence of metastasis” vs. “evidence of metastasis”) to receive high similarity scores despite opposite clinical meanings. This could happen regardless of the filtering.

• Medical synonyms: J(z^,z(m))J(\hat{z}, z^{(m)}) can not handle interchangeable ontology or synonyms. This could improperly discount correct reasoning steps, creating “false negatives”.

• Temporal order: Although Definition 1 treats clinical decisions as an ordered sequence z=(v1,…,vT)z = (v_1, \dots, v_T), which is good, J(z^,z(m))J(\hat{z}, z^{(m)}) overlooks this order. This could incorrectly reward clinically inappropriate trajectories, creating “false positives”.

W2. Exaggerated method. The controller and the policy are actually the same model; the difference is that the former is trained using SFT, while the latter is trained using GRPO, and they employ different prompts. In my view, from the perspective of implementation, the proposed method is a variant of an approach that combines SFT and GRPO in parallel, like [1,2]. Based on this perspective, I believe the current description of the method overlooks the fact that the controller and the policy learn shared information during the training process, like “Instead, the policy generates its own decision flow and receives guideline supervision only through scalar process rewards.” (line 111,112), and “Guideline supervision is transferred to the policy through our answer-gated process reward and GRPO, without conditioning the policy on an oracle trajectory.” (line 229-231). Therefore, the claim that the controller resolves path multiplicity may be untenable; under the branch of GRPO, candidate trajectories could still end up suffering from mode collapse [3,4].

W3. Pseudo-implementation of "path diversity". The design of the controller is to account for the path multiplicity; however, the controller’s candidate trajectory set is, in practice, learn a single reference trajectory (flow) z∗z^*. This may not truly resolve the issue of multiple paths. Furthermore, mode collapse from the RL branch could potentially exacerbate this problem (see W2).

W4. Data overlap reduces RL to post-hoc process alignment. Joint optimization (GUIDER) is initialized from the SFT (ans) checkpoint (line 391) and trained on the exact same full dataset. Because the model has already memorized the ground-truth answers, the subsequent GRPO stage does not genuinely learn to derive clinical answers through process reasoning, but degenerates into post-hoc process format alignment with already-known answers. Also, the SFT (ans) is missed in the Sec 3.2, which misleads readers into believing that GUIDER’s performance gain is solely from joint optimization.

W5. Insufficient ablation studies. Based solely on the current ablation studies, it is hard to independently determine the performance contribution of the proposed method. Important ablation baselines are missing: a baseline optimized solely via GRPO or joint optimization, without SFT.

W6. Overclaiming of "latent flows". The paper frames clinical reasoning using the probabilistic terminology of latent variable modeling and Monte Carlo to capture path multiplicity. But the "latent flows" are actually text-format clinical trajectories, not continuous latent states, and are explicitly optimized via the controller branch. In my view, the "process reward" in the proposed method is actually a lexical matching reward smoothed via the aggregation of multiple trajectories.

[1] Zhang, W., Xie, Y., Sun, Y., Chen, Y., Wang, G., Li, Y., ... & Zhou, J. (2026, April). On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. In International Conference on Learning Representations (Vol. 2026, pp. 120693-120726).

[2] Fu, Y., Chen, T., Chai, J., Wang, X., Tu, S., Yin, G., ... & Zhao, D. (2026, April). SRFT: A single-stage method with supervised and reinforcement fine-tuning for reasoning. In International Conference on Learning Representations (Vol. 2026, pp. 102430-102459).

[3] Cui, G., Zhang, Y., Chen, J., Yuan, L., Wang, Z., Zuo, Y., ... & Ding, N. (2025). The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617.

[4] He, A. W., Fried, D., & Welleck, S. (2025, November). Rewarding the unlikely: Lifting grpo beyond distribution sharpening. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 25559-25571).

Questions for authors

See weaknesses.

Please provide justifications for W1–6.

For W4, please clarify if the GUIDER is initialized from the SFT checkpoint. And please give justification of the difference between the pipepline design choices of GUIDER and the deepseek-R1-type post-training pipeline (light SFT with complete trajectories + heavy GRPO), which should be focused on the equivalent data coverage of the two training stages, and the choice of answer-only SFT.

For W5, please add ablation experiments.

10 comments

[1/3]:

W1: We thank the reviewer for highlighting these concerns. We agree that token-set Jaccard similarity is a lightweight approximation of clinical reasoning compatibility and does not explicitly capture negation, medical synonyms, or temporal dependencies. However, GUIDER's process reward combines reference-node coverage with weighted agreement across multiple controller-generated trajectories, rather than relying on a single Jaccard comparison. To investigate these limitations, we conduct additional experiments examining the sensitivity of GUIDER's process reward to clinically incorrect trajectories and its ability to accommodate valid alternative decision flows.

Error discrimination analysis

MedGUIDE contains structured clinical decision flows derived from 55 NCCN decision trees spanning 17 cancer types. We select 200 test cases and introduce three types of corruption:

  1. Unsupported-node injection: Add a clinical recommendation not supported by the applicable guideline.
  2. Contradicted-action injection: Introduce an action conflicting with the patient's clinical conditions.
  3. Condition–action swap: Associate an action with an incorrect clinical condition while preserving the original clinical concepts.

We measure Error Discrimination Rate (EDR), the percentage of cases where the clean trajectory receives a strictly higher reward than its corrupted counterpart. Both are evaluated using identical controller proposals and weights, without retraining. We additionally replace GUIDER's token-set Jaccard similarity with a frozen Qwen3.5-4B judge that scores semantic compatibility, keeping the remaining reward components unchanged.

Corruption typeCoverage-only EDR (%) ↑GUIDER (Jaccard) EDR (%) ↑GUIDER (LLM judge) EDR (%) ↑
Unsupported node0.084.585.3
Contradicted action0.089.091.3
Condition–action swap0.012.558.5

Reference coverage cannot distinguish these errors because the original reference nodes remain present. GUIDER with Jaccard discriminates unsupported nodes (84.5%) and contradictory actions (89.0%) but struggles with condition–action swaps (12.5%). Jaccard-based GUIDER effectively detects unsupported recommendations (84.5%) and contradictory actions (89.0%) but exhibits limited sensitivity to condition--action swaps (12.5%). The LLM judge improves discrimination across all corruption types, with the largest gain for condition--action swaps (58.5%), highlighting the benefit of semantic compatibility scoring for capturing clinical dependencies.

Computational efficiency

We compare training time under identical data, architecture, and optimization settings, varying only the similarity function.

Similarity functionTraining time (hours) ↓
Jaccard10
Qwen3.5-4B judge25

The LLM judge requires 2.5× longer training time (25 vs. 10 hours). Thus, Jaccard offers a more computationally efficient alternative, although it is less sensitive to complex clinical dependencies.

Alternative-flow recognition:

MedGUIDE is constructed from 55 NCCN clinical decision trees spanning 17 cancer types. The dataset creators convert these trees into structured JSON, enumerate and manually review root-to-leaf decision paths, and generate patient scenarios conditioned on these paths. Each question therefore has an underlying guideline-derived reference trajectory.

We select 200 MedGUIDE test cases and use their corresponding guideline trees to construct four trajectories: (i) the reference path (zAz_A), (ii) a valid reordering of independent decisions (zBz_B), (iii) a structurally distinct, guideline-consistent alternative applicable to the same patient (zCz_C), and (iv) an invalid condition–action flow (zDz_D). We construct alternative clinical flows by identifying cases that share a common decision path of more than two nodes and using the shared subpath as a clinically plausible alternative. While imperfect, this provides a practical approximation without requiring additional clinical expert annotation. We follow the same approach for (zC)(z_C), constructing an alternative flow from a shared subpath within the guideline decision tree. Its clinical validity is assumed following the NCCN decision tree structure.

We compare GUIDER against single-reference matching, Rsingle(z)=J(z,zA)R_{single}(z)=J(z,zA).

FlowSingle-reference reward ↑GUIDER reward ↑
Reference (zAz_A)1.000.86
Valid reordered (zBz_B)1.000.86
Distinct valid (zCz_C)0.430.72
Invalid (zDz_D)0.210.25

GUIDER assigns higher reward to the distinct valid alternative than single-reference matching (0.72 vs. 0.43). The invalid flow receives substantially lower reward (0.25). These results suggest that multi-flow aggregation better accommodates valid alternative reasoning paths, although reference-dependent weighting may still penalize alternatives with limited lexical overlap.

[2/3]:

W2: We thank the reviewer for this observation and for highlighting relevant prior work [1–4]. We agree that GUIDER's controller and policy share parameters, and therefore controller supervision can influence policy behavior through shared parameter updates. We will clarify this interaction and revise statements suggesting that guideline supervision reaches the policy exclusively through scalar rewards. However, GUIDER differs from simply combining SFT and GRPO. While [1,2] investigate the integration of supervised and reinforcement learning objectives, GUIDER introduces an additional interaction: the reference-supervised controller stochastically generates candidate clinical decision flows, which are scored and aggregated to construct the process reward used by GRPO. Thus, the controller does not merely provide a parallel supervised objective; its generated trajectories directly determine part of the policy's optimization signal. Importantly, the policy generates its own decision flow without being explicitly conditioned on the oracle trajectory during GRPO rollouts. Nevertheless, guideline supervision influences the shared parameters through controller training as well as the policy reward. We will clarify these two complementary mechanisms. Regarding mode collapse, we acknowledge the concerns raised in [3,4]. Stochastic sampling does not guarantee diverse trajectories, and GUIDER does not claim to eliminate GRPO-induced distribution sharpening. Rather, sampling multiple controller proposals allows reward construction to consider alternative trajectories instead of relying exclusively on a single fixed reference. The effectiveness of this mechanism depends on the diversity and quality of the sampled proposals. We will revise our claims to emphasize that GUIDER mitigates the rigidity of single-reference process supervision through multi-flow reward aggregation, rather than guaranteeing recovery of all valid pathways or preventing mode collapse.

W3: We thank the reviewer for this observation. Although the controller is supervised using a single annotated reference flow per case, it learns a conditional distribution over decision flows rather than deterministically reproducing the reference. GUIDER samples multiple candidate trajectories and aggregates their contributions to the process reward, relaxing strict single-reference matching. Importantly, Appendix A.2 (Table 3) provides empirical evidence supporting multi-flow supervision.

Number of flows (M)MedGUIDE WAcc. ↑MedGUIDE Adh. ↑AMEGA-LLM ↑RealMedQA ↑CancerGUIDE ↑
177.170.220.31.61.9
280.372.521.61.72.1
482.573.522.31.92.4
682.474.122.11.92.3
882.374.5**22.21.92.3

Increasing the number of sampled flows from M=1 to M=4 improves MedGUIDE weighted accuracy from 77.1 to 82.5 (+5.4 points) and guideline adherence from 70.2 to 73.5 (+3.3 points). We also observe improvements across AMEGA-LLM, RealMedQA, and CancerGUIDE, with performance largely saturating beyond M=4. These results demonstrate that aggregating multiple controller proposals provides meaningful benefits over single-flow supervision. However, stochastic sampling does not guarantee clinically distinct valid pathways or eliminate mode collapse. GUIDER therefore mitigates the limitations of single-reference supervision by allowing multiple sampled trajectories to contribute to policy optimization, rather than explicitly enforcing path diversity.

[3/3]:

W4: We thank the reviewer for this observation. GUIDER is initialized from the answer-only SFT checkpoint, as stated in the implementation details, and we will state this initialization explicitly in Section 3.2. All evaluations use the held-out MedGUIDE test split, so any gain at evaluation requires generalization beyond the training answers. Table 1 gives a controlled comparison from the same SFT (ans) checkpoint. Answer-only GRPO increases MedGUIDE weighted accuracy from 78.0 to 80.1 and guideline adherence from 53.1 to 54.5, while GUIDER reaches 82.5 and 73.5. If the second stage only aligned the flow format with known answers, the two methods would reach similar accuracy from this checkpoint. GUIDER instead improves accuracy by 2.4 points and adherence by 19.0 points over answer-only GRPO. The gains extend to out-of-domain benchmarks with different patient cases and task formats. Relative to answer-only GRPO from the same initialization, GUIDER improves AMEGA-LLM from 20.5 to 22.3, RealMedQA from 1.81 to 1.90, and CancerGUIDE from 1.75 to 2.40. The comparison without SFT, which starts from the pretrained backbone, shows the same pattern. GUIDER outperforms answer-only GRPO by 7.0 points in weighted accuracy (74.8 vs. 67.8) and 17.6 points in adherence (68.2 vs. 50.6), and it improves CancerGUIDE from 1.74 to 2.12.

These results indicate that the guideline-derived process reward improves answer accuracy and guideline adherence beyond answer-only optimization. Causal faithfulness of the generated flows to the final answers requires separate analysis and is an important direction for future work.

W5: We agree that evaluating GRPO and GUIDER without SFT initialization helps disentangle the contribution of SFT from the proposed joint optimization. Table 1 provides controlled comparisons under the same answer-only SFT initialization. For completeness, we reproduce the full results below, including two additional configurations trained directly from the pretrained backbone without SFT.

Training StrategyMedGUIDE WAcc. ↑MedGUIDE Adh. ↑AMEGA-LLM ↑RealMedQA ↑CancerGUIDE ↑
Base52.347.217.21.621.71
SFT (ans)78.053.120.31.731.73
SFT (ans) → GRPO (ans)80.154.520.51.811.75
SFT (ans+flow)64.071.318.81.821.80
SFT (ans+flow) → GRPO (ans+flow)68.271.619.11.851.82
GRPO (ans), no SFT67.850.618.71.691.74
GUIDER, no SFT74.868.220.61.792.12
GUIDER82.573.522.31.902.40

Under identical SFT initialization, GUIDER improves MedGUIDE weighted accuracy by 2.4 points and guideline adherence by 19.0 points over answer-only GRPO. Without SFT initialization, GUIDER also outperforms GRPO (ans) by 7.0 points in weighted accuracy and 17.6 points in adherence, suggesting that its benefits extend beyond SFT initialization. Furthermore, Appendix A.2 (Table 3) evaluates multi-flow supervision. Increasing controller samples from M=1 to M=4 improves weighted accuracy from 77.1 to 82.5 (+5.4 points) and adherence from 70.2 to 73.5 (+3.3 points), with corresponding improvements across out-of-domain benchmarks. Together, these results support the contribution of guideline-informed optimization and multi-flow aggregation, while demonstrating that SFT initialization further improves performance.

W6: We clarify that "latent" in GUIDER refers to unobserved decision trajectories rather than continuous latent representations. Latent variables need not be continuous; they can also be discrete structured objects, including sequences of clinical decisions.

In GUIDER, the observed patient case and final answer do not uniquely determine the underlying clinical decision flow. We therefore treat the flow as a latent structured variable and approximate the expectation of the process reward over controller-sampled candidate flows using Monte Carlo estimation. Although the controller is explicitly supervised using annotated reference flows, the alternative sampled trajectories are not individually annotated, and the policy does not observe these trajectories during rollout generation.

We agree that the current process reward uses lexical similarity, specifically token-set Jaccard overlap, and that aggregating multiple sampled flows does not introduce semantic or structural verification. However, the reward is not simply an unweighted average of lexical similarities: candidate contributions are weighted by their support relative to the guideline-derived reference flow, and the resulting process reward is gated by answer correctness. Thus, the contribution is not a continuous latent-state model or a learned semantic reward model. Rather, GUIDER introduces a sample-based, reference-weighted process reward over discrete candidate decision flows, jointly optimized with the reasoning policy.

Reviewer_712jOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a reinforcement learning framework for guideline-adherent clinical reasoning named GUIDER. Specifically, GUIDER consists of three core components: the decision-flow controller, the multi-flow process reward, and the joint controller–policy optimization. The decision-flow controller framework generates plausible decision flows through guideline-derived supervision and autoregressive candidate sampling. The multi-flow process reward encourages guideline adherence under answer-correctness gating via score-weighted Monte Carlo aggregation over multiple controller-generated flows. Finally, the joint controller–policy optimization framework updates shared model parameters through GRPO and supervised decision-flow learning. The experimental evaluation in this paper assesses the effectiveness of GUIDER across four benchmark datasets. Results demonstrate that GUIDER achieves higher answer accuracy while improving guideline adherence, outperforming several clinical reinforcement learning baselines.

Critical strengths
  • The idea of converting the decision flows in clinical guidelines into training signals so that LLMs can both provide correct answers and generate guideline-consistent reasoning processes is reasonable.
  • I concur with the authors’ view that transitions between decision nodes should be consistent with guideline logic and the patient’s state.
Critical weaknesses
  • In Definition 1, the authors define a decision flow as an ordered sequence of nodes and require transitions between nodes to be consistent with both clinical guidelines and the current patient’s state. That is, validity depends on conditions, actions, and the relationships between them. However, the actual process reward cannot distinguish certain critical errors in decision logic. Consider two flows in which the condition–action correspondences are swapped but the token sets are identical. Even if one flow follows the given rules while the other violates them, they still receive exactly the same process reward. This means that the reward function cannot distinguish certain flows that contain all the relevant keywords but have incorrect conditional relationships.
  • In my understanding, one motivation of the paper is that multiple reasonable paths may exist for the same case, and the sole annotated path z* should not be treated as the only correct answer. However, in the actual method, z* still governs key components. The Controller uses z* for autoregressive supervision and the support scores of all candidate paths are also determined by their coverage of the same z*. Therefore, multiple candidates are essentially still generated and weighted under a scoring system defined by a single reference path. GUIDER does relax the requirement to directly reproduce the sole reference text, but it lacks an effective way to assess a clinical decision flow that differs from the reference path yet remains valid.
  • The candidate support score defined in Eqn (3) primarily measures how many nodes in the reference path are matched in a candidate path. However, covering correct content does not mean that no incorrect content is present. If a candidate containing an erroneous node receives a high weight, the Policy’s reproduction of this erroneous node may increase its Jaccard similarity with that candidate. This means that a candidate path’s reference coverage cannot sufficiently guarantee the validity of the candidate itself.
  • A weighted average of paths may encourage invalid path concatenations. Even if all candidates are individually valid, the current aggregation scheme does not necessarily encourage the Policy to fully follow one of the valid paths. The current reward may favor invalid combinations of content.
Questions for authors
  • It seems that the process reward ultimately remains a score for the entire trajectory, lacking specific error attribution.
  • Answer gating only establishes reward priorities, without directly checking whether the flow supports the final answer, and also seems to discard differences in process quality among incorrect responses. Does this mean that, for difficult samples on which all responses generated by the Policy are incorrect, the within-group advantages computed from this task reward are all zero? This would mean that gating sacrifices the use of partially correct flows within incorrect responses while avoiding rewards for incorrect answers.
4 comments

[1/3]

W1: We thank the reviewer for this precise example. The reviewer is correct that the token-set Jaccard term gives identical scores to two flows with the same tokens and swapped condition–action pairs. Definition 1 specifies the target object, an ordered flow with valid transitions, and the process reward serves as a tractable surrogate for this object during training. We chose an order-invariant surrogate because clinicians often sequence independent assessment steps differently and still reach the same guideline-consistent decision. This choice trades sensitivity to the order of dependent steps for tolerance to the reordering of independent steps. The alternative-flow analysis (Table 3a) shows the benefit of this trade-off. GUIDER gives a valid reordering the same reward as the reference flow (0.86), and it gives a structurally distinct valid alternative a higher reward than single-reference matching (0.72 vs. 0.43). Flows from the trained policy score highly under an order-aware evaluation. We measure guideline adherence with a judge that scores clinical equivalence and ordering against the gold guideline path (Appendix A.4, Figure 5). Under this evaluation, GUIDER achieves 73.5 adherence on MedGUIDE, compared with 54.5 for answer-only GRPO from the same initialization and 70.4 for the strongest clinical RLVR baseline (Tables 1 and 2).

The error discrimination analysis (Table 3b) quantifies the limitation the reviewer describes. The full GUIDER reward detects unsupported nodes in 84.5% of cases and contradicted actions in 89.0% of cases, and condition–action swaps are the main failure mode at 12.5%. GUIDER treats the similarity function as a modular component. Replacing Jaccard with a frozen Qwen3.5-4B compatibility judge raises swap discrimination to 58.5% at 2.5 times the training cost (Appendix A.3.1). We agree that the process reward approximates guideline process alignment, and relation-aware rewards over typed conditions, actions, and transitions offer a direct extension toward verification of guideline logic.

W2: Alternative-flow recognition:

MedGUIDE is constructed from 55 NCCN clinical decision trees spanning 17 cancer types. The dataset creators convert these trees into structured JSON, enumerate and manually review root-to-leaf decision paths, and generate patient scenarios conditioned on these paths. Each question therefore has an underlying guideline-derived reference trajectory.

We select 200 MedGUIDE test cases and use their corresponding guideline trees to construct four trajectories: (i) the reference path (zAz_A), (ii) a valid reordering of independent decisions (zBz_B), (iii) a structurally distinct, guideline-consistent alternative applicable to the same patient (zCz_C), and (iv) an invalid condition–action flow (zDz_D). We construct alternative clinical flows by identifying cases that share a common decision path of more than two nodes and using the shared subpath as a clinically plausible alternative. While imperfect, this provides a practical approximation without requiring additional clinical expert annotation. We follow the same approach for (zC)(z_C), constructing an alternative flow from a shared subpath within the guideline decision tree. Its clinical validity is assumed following the NCCN decision tree structure.

We compare GUIDER against single-reference matching, Rsingle(z)=J(z,zA)R_{single}(z)=J(z,zA).

FlowSingle-reference reward ↑GUIDER reward ↑
Reference (zAz_A)1.000.86
Valid reordered (zBz_B)1.000.86
Distinct valid (zCz_C)0.430.72
Invalid (zDz_D)0.210.25

GUIDER assigns higher reward to the distinct valid alternative than single-reference matching (0.72 vs. 0.43). The invalid flow receives substantially lower reward (0.25). These results suggest that multi-flow aggregation better accommodates valid alternative reasoning paths, although reference-dependent weighting may still penalize alternatives with limited lexical overlap.

[2/3]

W3: We thank the reviewer for this observation. We agree that the support score in Eq. (3) measures reference-node coverage without explicitly penalizing incorrect content. However, GUIDER's process reward additionally considers agreement between the policy-generated trajectory and multiple controller proposals: Rproc(z^)=∑m=1Mwm[sm]+J(z^,z(m)).R_{\mathrm{proc}}(\hat z)=\sum_{m=1}^{M}w_m[s_m]_+J(\hat z,z^{(m)}). Here, sms_m measures how well candidate z(m)z(m) covers the annotated guideline flow, wmw_m is its normalized weight, and JJ measures token-set similarity with the policy trajectory zz. Thus, high reference coverage alone does not guarantee a high process reward. Answer gating further restricts process reinforcement to correct final answers, although it cannot guarantee intermediate clinical correctness.

Error discrimination analysis

MedGUIDE contains structured clinical decision flows derived from 55 NCCN decision trees spanning 17 cancer types. We select 200 test cases and introduce three types of corruption:

  1. Unsupported-node injection: Add a clinical recommendation not supported by the applicable guideline.
  2. Contradicted-action injection: Introduce an action conflicting with the patient's clinical conditions.
  3. Condition–action swap: Associate an action with an incorrect clinical condition while preserving the original clinical concepts.

We measure Error Discrimination Rate (EDR), the percentage of cases where the clean trajectory receives a strictly higher reward than its corrupted counterpart. Both are evaluated using identical controller proposals and weights, without retraining. We additionally replace GUIDER's token-set Jaccard similarity with a frozen Qwen3.5-4B judge that scores semantic compatibility, keeping the remaining reward components unchanged.

Corruption typeCoverage-only EDR (%) ↑GUIDER (Jaccard) EDR (%) ↑GUIDER (LLM judge) EDR (%) ↑
Unsupported node0.084.585.3
Contradicted action0.089.091.3
Condition–action swap0.012.558.5

Reference coverage cannot distinguish these errors because the original reference nodes remain present. GUIDER with Jaccard discriminates unsupported nodes (84.5%) and contradictory actions (89.0%) but struggles with condition–action swaps (12.5%). Jaccard-based GUIDER effectively detects unsupported recommendations (84.5%) and contradictory actions (89.0%) but exhibits limited sensitivity to condition--action swaps (12.5%). The LLM judge improves discrimination across all corruption types, with the largest gain for condition--action swaps (58.5%), highlighting the benefit of semantic compatibility scoring for capturing clinical dependencies.

Computational efficiency

We compare training time under identical data, architecture, and optimization settings, varying only the similarity function.

Similarity functionTraining time (hours) ↓
Jaccard10
Qwen3.5-4B judge25

The LLM judge requires 2.5× longer training time (25 vs. 10 hours). Thus, Jaccard offers a more computationally efficient alternative, although it is less sensitive to complex clinical dependencies.

W4: We thank the reviewer for this observation. We acknowledge that weighted aggregation does not guarantee path consistency. However, GUIDER does not concatenate candidate trajectories; it evaluates the policy-generated flow against multiple controller proposals.

Invalid-hybrid generation rate (IHGR)

MedGUIDE is constructed from structured NCCN clinical decision trees, where different branches represent clinical decisions under specific patient conditions. We select 200 test cases with two valid but mutually exclusive decision pathways and examine whether the trained policy incorrectly combines their recommendations into a single trajectory. We measure Invalid-Hybrid Generation Rate (IHGR), the percentage of generated flows that combine incompatible decisions from these pathways. We compare policies trained using single-reference matching, best-of-M selection (maximum candidate reward), and GUIDER's weighted aggregation.

MethodIHGR (%) ↓
Single-reference reward14.2
Best-of-M reward9.1
GUIDER6.8

GUIDER achieves the lowest IHGR (6.8%), compared with single-reference matching (14.2%) and best-of-M (9.1%). This suggests that weighted multi-flow aggregation can reduce invalid pathway combinations in practice, although it does not formally guarantee clinical consistency. We will clarify this limitation and discuss relation-aware rewards as future work.

[3/3]

Q1: We thank the reviewer for this observation. We agree that GUIDER produces a trajectory-level scalar reward rather than explicit step-wise error attribution. However, unlike outcome-only rewards, GUIDER incorporates guideline-derived process information through multiple controller proposals. As defined in the process reward formulation, each proposal contributes according to its reference support score and compatibility with the policy-generated trajectory.

This decomposition enables us to examine (i) which controller proposals contribute to the reward, (ii) their guideline-reference support scores, and (iii) their compatibility with the generated reasoning flow. Thus, GUIDER provides candidate-level diagnostics, although the policy is optimized using the aggregated scalar reward. Importantly, our objective is not to explicitly localize individual reasoning errors but to provide process-informed supervision that encourages guideline-compatible reasoning beyond final-answer correctness.

Q2 : We agree that answer gating prioritizes final-answer correctness without explicitly verifying whether the reasoning flow supports the final decision. As defined in our answer-gated reward formulation, when all policy rollouts in a GRPO group produce incorrect answers, their task rewards are zero, resulting in zero within-group advantages from this reward. This is an intentional design trade-off. In clinical guideline reasoning, assigning positive process rewards to trajectories that produce incorrect clinical decisions may reinforce misleading reasoning patterns. Therefore, GUIDER restricts process-based reinforcement to correct-answer trajectories, where guideline compatibility further distinguishes reasoning quality. Importantly, zero GRPO advantages do not necessarily imply zero learning in our joint training framework. The controller is additionally optimized through supervised cross-entropy loss on guideline-derived reference flows. Thus, while all-incorrect groups provide no policy-side learning signal from the gated task reward, controller supervision can still update the shared model parameters.

We acknowledge that this design sacrifices process-quality differences among incorrect responses in exchange for avoiding positive reinforcement of incorrect clinical decisions.

Dear Reviewer 712j,

As the discussion deadline is approaching, we would like to kindly follow up on our rebuttal and the additional experiments addressing your concerns regarding process reward reliability, alternative-flow recognition, and invalid pathway combinations.

We appreciate your thoughtful technical observations, which have helped us better understand and characterize the strengths and limitations of GUIDER. We hope our additional analyses and clarifications have addressed your concerns.

We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion period concludes.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback. During the rebuttal, we conducted additional experiments, clarified methodological details, and refined our claims.

All reviewers recognized the importance and motivation of guideline-grounded clinical reasoning. Reviewers also highlighted the clear problem formulation (uN5z, hddo, 5sVS), methodological novelty and inference-time efficiency (5sVS), motivation for process supervision (712j, hddo), and broad empirical evaluation across models and clinical benchmarks (uN5z, hddo, 5sVS).

Discussion and overlapping reviewer concerns

The concerns raised by Reviewers 712j, hddo, and uN5z share several common themes:

  • Process reliability and lexical reward limitations (712j: W1, W3; hddo: W1; uN5z: W1).
  • Path multiplicity and single-reference supervision (712j: W2; hddo: W1, Q2; uN5z: W2, W3).
  • Contribution of process supervision and answer correctness (712j: Q2; hddo: W2; uN5z: W4, W5).
  • Effectiveness of multi-flow supervision (712j: W2, W4; hddo: W1, Q2; uN5z: W3).

Following multiple rounds of discussion, Reviewer uN5z raised their rating from 2 (Weak Reject) to 3 (Weak Accept), acknowledging that our additional experiments and clarifications substantially eased their concerns. Together with Reviewer 5sVS, two reviewers now lean toward acceptance.

Reviewer hddo acknowledged that their concerns were partly addressed. We have responded to their follow-up questions on alternative-path validity, experimental protocols, and uncertainty estimates.

Reviewer 712j appreciated the alternative-flow recognition and error discrimination analyses but maintained their original rating due to concerns regarding condition–action relationships and reasoning–answer consistency. We acknowledge these limitations, while our experiments demonstrate the empirical benefits of multi-flow supervision and the potential of more expressive reward functions.

Additional experiments and key findings

  • No-SFT ablation: GUIDER improves weighted accuracy and adherence by 7.0 and 17.6 points over answer-only GRPO without preliminary SFT.

  • Alternative-flow recognition: GUIDER assigns higher rewards to structurally distinct, guideline-supported candidate alternatives than single-reference matching (0.72 vs. 0.43).

  • Alternative-flow coverage: ValidAltCoverage, a proxy for structural alternative coverage, increases from 18.4% to 41.8% with 1 to 4 controller samples, reaching 48.3% with 8 samples.

  • Error discrimination: GUIDER achieves 84.5% and 89.0% discrimination of unsupported nodes and contradicted actions, compared with 0% for reference coverage alone.

  • Semantic reward ablation: An LLM-based compatibility judge improves condition–action swap discrimination from 12.5% to 58.5%, at 2.5× the training cost.

  • Invalid-hybrid generation: GUIDER reduces incompatible pathway combinations to 6.8%, compared with 14.2% for single-reference matching and 9.1% for best-of-M aggregation.

  • Disjoint SFT–RL training: With non-overlapping 30% SFT and 70% RL subsets, GUIDER improves weighted accuracy and adherence by 6.5 and 19.9 points over answer-only GRPO, with consistent gains across all three out-of-domain benchmarks.

  • Multi-flow ablation: The existing controlled ablation shows that increasing M from 1 to 4 during training improves weighted accuracy and adherence by 5.4 and 3.3 points.

Manuscript revisions and concluding remarks

We clarified the optimization framework, SFT initialization, reward implementation, latent-flow formulation, answer-gating trade-offs, and experimental protocols, and added uncertainty estimates to Tables 1 and 2.

We refined our claims to emphasize that GUIDER mitigates, rather than fully resolves, path multiplicity. We will explicitly discuss reasoning faithfulness, reasoning–answer consistency, semantic inversion, temporal ordering, condition–action swaps, and independent clinical validation as limitations and future directions.

While these challenges remain open, our additional experiments provide complementary evidence supporting GUIDER's contributions beyond single-reference and answer-only optimization. We believe this represents substantive progress toward guideline-grounded clinical reasoning.

We respectfully request that the Area Chair consider the empirical contributions, additional experimental evidence, and methodological clarifications in the final assessment.

Thank you,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.