acceptodds
Under review as a conference paper at ICLR 2027

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

Abstract

Clinical practice guidelines (CPGs) encode evidence-based decision logic that clinicians apply by evaluating patient variables, conditional criteria, and recommendation rules. However, existing methods often use CPGs as free-text training data or retrieval sources, underutilizing their procedural decision structure. To better exploit this structure, we introduce a guideline-derived training pipeline that transforms CPG recommendations into executable clinical decision logic and uses it to generate factual and counterfactual question-answering data. These data teach models both guideline-supported decisions and how decisions change under different patient conditions. Post-training a medical LLM on the generated data yields MedGuideX. Across four clinical reasoning benchmarks, MedGuideX achieves a 10.42% relative improvement in average accuracy. Physician evaluation further shows that MedGuideX better recovers clinician-authored reasoning steps and produces physician-preferred rationales in faithfulness, validity, completeness, and clarity. Overall, our results show that executable decision logic from CPGs can be transformed into scalable supervision for building reliable medical LLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Reasoning, Clinical Guideline

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_mdd6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

MedGuideX converts clinical guidelines into executable decision functions and uses them to generate factual and counterfactual QA supervision. Mixed-data SFT followed by factual GRPO improves clinical reasoning performance across four benchmarks and two model scales.

Critical strengths
  1. Transforming guideline decision logic into executable supervision is an interesting and well-motivated approach. It provides a structured alternative to treating guidelines as unstructured training text or retrieval material.

  2. The improvements are consistent across benchmarks and model scales. Training-strategy ablations and a blinded physician evaluation provide additional supporting evidence.

Critical weaknesses
  1. The counterfactual formulation may be underdetermined. Multiple hidden-variable assignments can reproduce the same factual outcome while yielding different counterfactual outcomes. The paper does not explain how such ambiguity is excluded, yet Equation (7) requires recovery of the particular sampled hidden state.

  2. Executability and consistency with an extracted decision tree do not establish fidelity to the source guideline. Independent clinical auditing of the generated functions is needed to assess omitted conditions, exceptions, and unsupported rules.

  3. A matched baseline that generates QA directly from the same guidelines would better isolate the benefit of executable supervision. The physician study is useful but small and restricted to cases both models answered correctly.

Questions for authors
  1. Does counterfactual generation guarantee a uniquely identifiable hidden state, or at least an identical counterfactual outcome across all compatible hidden states? How are ambiguous examples handled?

  2. Can the authors provide an independent clinical audit of a representative sample of executable functions?

  3. How does the approach compare with direct guideline-to-QA synthesis under matched data volume, teacher model, and training budget?

2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer mdd6’s Comments

Thank you for the precise questions about label validity and matched supervision. References use the revised manuscript of 17 November 2026.

1. Hidden-state and action identifiability

The original generator guarantees neither a unique hidden state nor agreement across all compatible states. Your concern about Equation 7 is valid.

All 4,963 factual labels and both stored outputs of all 5,028 counterfactual records replay. Among 4,623 Boolean-hidden records, 1,123 admit both hidden values under exact factual-output matching. Both yield the same counterfactual action, but the verifier accepts only the sampled state. Thus, these action targets remain determinate while the state-matching condition is overly restrictive.

Numerical variables can also yield conflicting actions. In the ICU census rule, capacities of 12 and 13 both explain no action at census 30. At census 36, only capacity 12 triggers the three-times-capacity threshold. Appendix C.4, pages 16 and 17, reports these checks and 213 interventions that change only repetitions of identical action strings.

Appendix E.1, page 21, gives the validity criterion. A unique hidden state is unnecessary when all factual-compatible states agree on the intervened action. Any compatible state should then receive credit. Conflicting actions require ambiguity-aware targets or exclusion. Equivalent states call for a verifier correction, while conflicting actions require a target correction. Checkpoints retain the original corpus. The factual-only control below tests supervision without counterfactual abduction.

2. Clinical source fidelity

Function validation checks syntax, executability, and consistency with the extracted tree. These checks are automated. Physicians evaluated model rationales.

The diabetes source check confirms that repeat testing is gated by qualifying laboratory criteria. A pediatric urine-collection function contains an exception-precedence error. It can return both methods when the source requires catheterization if reliable clean catch is unavailable. Applying the exception changes a factual or intervened reference in four of its six counterfactual records. Two become action-preserving, while its two factual records remain unchanged. Appendix C.4 identifies the source and affected records.

This separates reproducible execution from faithful translation of clinical conditions and exceptions. Both factual and counterfactual labels depend on the latter.

3. Matched Direct QA comparison

Both arms use GPT-5.4 xhigh. Direct QA generates answers and rationales from guideline text and the question. Executable supervision obtains actions by running the function and verbalizes its trace. We match Qwen3.5-9B, source guidelines, questions, full student inputs, data volume, budgets, and GRPO scoring. Both students receive the same function and learn from their own references, fixing teacher quality and code exposure.

With mixed SFT followed by factual RL, execution gains 8.7 versus 6.5 points on MedCaseReasoning and 6.9 versus 5.6 on MedQA. It exceeds Direct QA in all 15 configurations. Factual-only RL retains advantages of 1.5 and 0.7 points without counterfactual training. Figure 3a, page 7, and Appendix F.1, page 21, report the comparison.

The held-out test supplies complete variables and includes both action-preserving and action-changing edits. MedGuideX answers 205/300 cases correctly, versus 193/300 for Direct QA and 176/300 for the backbone. GPT-6.1 Sol xhigh generates cases and action pseudolabels. Gains over Direct QA are 7.0 points on changing edits and 3.0 on preserving edits, each over 100 cases. Table 3, page 8, reports the results.

4. Physician preferences

Two physicians each assess 20 cases, yielding 40 annotations over 30 jointly correct cases with 10 shared cases. This compares rationale quality when final diagnoses agree. Selection favored MedGuideX rationales, while identities were hidden and order randomized. Appendix G, pages 22 to 24, specifies the sample.

Table 4, page 9, shows 36 versus 3 preferences for completeness and 17 versus 16 for conciseness. The pooled non-tie win rate is 76.9%. The preference within this sample is concentrated in clinical coverage. The benchmarks and matched Direct QA comparison provide the broader performance evidence.

Within the rebuttal time and compute budget, we prioritized these comparisons and audits. We will continue representative independent clinical validation and corrected-label evaluation beyond the rebuttal.

Reviewer_mdd6OpenReview ↗

Comments

Thanks for your relply. Some of my concerns have been solved. I keep my original score as it is already positive.

Reviewer_XeQEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a post-training pipeline that turns clinical practice guidelines (CPGs) into execution-verifiable supervision. An LLM extracts condition–action recommendations, converts each to a decision tree, and compiles each tree into a Python function ff. Sampling variable assignments XX and executing f(X)f(X) yields factual QA pairs. The sampling, intervening on one observed variable, and re-executing yields counterfactual QA candidates. Qwen3.5-4B and Qwen3.5-9B are post-trained with LoRA SFT on the factual and counterfactual mixture, then GRPO with a format reward and an execution-checked final-answer reward. The proposed MedGuideX-9B improves the 9B backbone average from 53.29 to 58.85.

Critical strengths
  1. Execution-verifiable supervision with a solid negative control. The paper’s most convincing point is the contrast between raw-text guidelines and executable QA: raw continued pretraining or fine-tuning actually degrades accuracy, whereas compiled QA plus execution-checked GRPO consistently improves it. Table 2 backs this up nicely by showing the gains come from fixing genuine backbone errors rather than random answer churn. This is the key takeaway since it cleanly isolates the text-to-code transformation as the actual engineering contribution rather than just feeding in more guideline text.

  2. Thorough and transparent evaluation. The experimental setup is thorough: 4 benchmarks with diverse formats and metrics, 2 model scales, a full data-type and training-phase ablation grid, 5 well-documented guideline baselines, and raw human study counts. This level of detail makes the MedGuideX gains easy to audit and genuinely trustworthy.

Critical weaknesses
  1. The core mechanism is claimed, not measured. Eq. 2 frames the decision path πf(X)\pi_f(X) as the central unit of guideline logic the model should internalize, and Section 3.2 mentions checking intermediate variable execution. Yet when you check the actual RL reward in Eq. 6, it only checks the final answer. There are no experiments on path-consistency rate, zero evaluation on held-out guidelines/functions, and no counterfactual accuracy numbers even in-distribution. The paper asserts that the model learns execution logic, but never actually measures it.

  2. Weak novelty positioning relative to cited literature. The moving parts here are mostly known: MedDM (Li et al., 2023a) already compiles clinical guidelines into executable decision trees. Vashishtha et al. (2025) already formalized the abduction–intervention–prediction framework over code with SFT vs RL. And Staniek et al. (2025) already turned guideline rules into SFT pairs. While these are cited, the authors never clearly articulate their precise delta.

  3. Missing critical negative controls. To prove that "executable decision logic" is what drives the gain, we need the exact same SFT + GRPO recipe applied to either: (1) a matched-size standard medical QA set (e.g., MedQA train or standard synthetic QA without code), or (2) guideline QA with shuffled/corrupted function outputs. Without this ablation, it’s impossible to distinguish guideline-derived executable logic from just the generic benefits of QA post-training with format rewards.

  4. Apples-to-oranges backbone comparison. In Appendix D, the base Qwen models have thinking mode disabled under a tight 1,024-token cap, while MedGuideX is explicitly trained via format rewards to produce long-form reasoning before answering. Touting a "26.64% relative improvement" against an artificially silenced baseline isn't convincing; the proper reference point for claiming guideline-induced reasoning is the base model with thinking mode turned on.

  5. Human study is underpowered, confounded, and over-interpreted. The setup is just 30 cases evaluated by two doctors, strictly filtered to cases both models already solved.

Minor:

  1. Over-reliance on relative percentages in headline claims. The abstract and text lead with relative gains (e.g., "10.42% relative improvement"), which masks baseline scale and tends to inflate the perceived impact. The authors should make the absolute percentage point gains (+8.7, +6.9, +3.6, +3.0 points) primary throughout.

  2. Several model references point to older technical reports: Qwen3.5 cites the Qwen3 report, DeepSeek-V4 cites DeepSeek-V3, and Kimi-2.6 cites Kimi K2. These citations need to be audited and corrected.

Questions for authors
  1. How do Qwen3.5-4B and 9B perform when thinking mode is enabled under a matched token budget?

  2. What fraction of Claude-Haiku-4.5 generations failed the MedQA regex parser? On MedCaseReasoning, what percentage of outputs from Kimi-2.6 and MedGemma-1.5-4B failed formatting or were scored as non-answers?

  3. Which specific LLM handled recommendation extraction, tree generation, validation, and Python compilation?

  4. Could the authors clarify the exact composition of the "held-out validation splits" referenced in Appendix E (e.g., unseen guidelines, or held-out QA pairs derived from the training guidelines)?

1 comment
AuthorsOpenReview ↗

Authors’ Response to Reviewer XeQE’s Comments

Thank you for the specific questions. References use the 17 November 2026 revision.

1. Decision behavior and reward

Your reading of Equation 6 is correct. Factual GRPO checks format and final-action correctness. Execution paths supply SFT traces. Hidden-state checks belong to the counterfactual verifier. Section 3.2, page 4, and Appendix E.1, page 21, now distinguish these roles.

On 100 triplets from EPFL MEDITRON source guidelines excluded from SFT and RL, MedGuideX answers 205/300 cases correctly, versus 193/300 for Direct QA and 176/300 for the backbone. Each triplet contains an original case and single-variable edits that preserve or change its action. Models receive identical guideline text and complete variables, with versions queried independently. GPT-6.1 Sol xhigh generates cases and action pseudolabels.

Gains over Direct QA are 7.0 points on changing edits and 3.0 on preserving edits, each over 100 cases. The advantage is largest when a modified condition requires a different action. Table 3, page 8, and Appendix F.4, pages 21 and 22, report individual action accuracy. Hidden-state recovery, paths, and paired consistency require distinct measurements.

2. Generic QA control

Direct QA matches Qwen3.5-9B, guidelines, questions, full student inputs, generation model and reasoning effort, data volume, budgets, and reward scoring. It generates answers and rationales from guideline text. Executable supervision runs the function for the action and verbalizes its trace. Both students receive the same function and use their own reference answers.

With mixed SFT followed by factual RL, execution adds 2.2 points over Direct QA on MedCaseReasoning and 1.3 on MedQA. It performs better in all 15 configurations. Factual-only RL retains advantages of 1.5 and 0.7 points. This controls guideline content and code exposure alongside QA training. Figure 3a, page 7, and Appendix F.1, page 21, give the comparison.

3. Prior work and contribution

MedDM and CPGPrompt use clinical decision structures for inference. Staniek et al. derive supervised examples from verbalized rules. Executable Counterfactuals develops program-based abduction, intervention, and prediction. Sections 2.2 and 2.3, pages 2 and 3, acknowledge these precedents. Our contribution is guideline execution as clinical training supervision. The matched comparison tests its added value, while the ablations identify mixed SFT followed by factual RL as the strongest use of counterfactual data.

4. Thinking mode

Both Qwen3.5 and MedGuideX use native thinking disabled, greedy decoding, and a 1,024-token cap. MedQA and MedCaseReasoning prompts request written reasoning from both. Appendix D, pages 17 and 18, gives the protocol. Direct QA also receives rationale supervision and the same format reward, controlling the training asymmetry you identified. The requested thinking-enabled 4B and 9B comparison is part of our follow-up evaluation.

5. Parsing and non-answers

MedQA uses deterministic option extraction. Open-diagnosis tasks use GPT-5.4 mini xhigh for semantic judging. Aggregate accuracy does not separate parser failures and non-answers for Claude-Haiku-4.5, Kimi-K2.6, and MedGemma-1.5-4B. We will report these model-specific rates through the follow-up output audit.

6. Physician evaluation

Two physicians each assess 20 cases, yielding 40 annotations over 30 jointly correct cases with 10 shared cases. This compares reasoning when final diagnoses agree. Selection favored MedGuideX rationales, so the findings characterize that sample. Table 4, page 9, shows 36 versus 3 preferences for completeness and 17 versus 16 for conciseness. The advantage centers on coverage, consistent with reference-step recall of 62.8% versus 50.8% for GPT-5.0. Appendix G, pages 22 to 24, gives the design.

7. Generation models

GPT-5.4 xhigh handles recommendation extraction, tree generation, and validation. A fixed program converts trees into Python functions. Appendix C.2, page 16, specifies these stages. Both supervision arms use GPT-5.4 xhigh for rationales.

8. Validation split

Checkpoint selection uses QA from source guidelines excluded from SFT and RL. Appendix E, page 20, specifies this split, separate from the new decision test.

9. Absolute gains and citations

Average accuracy rises from 53.29% to 58.85%, or 5.56 points. Section 4.2, pages 8 and 9, reports gains of 8.7, 6.9, 3.6, and 3.0 points. Table 2, page 6, corrects the Qwen3.5, DeepSeek-V4, and Kimi-K2.6 references. The introduction also corrects the Qwen3.5 citation.

Within the rebuttal time and compute budget, we prioritized the matched training and held-out decision tests. We will continue the inference checks above and path-level and hidden-state evaluation beyond the rebuttal.

Reviewer_J2F4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces MedGuideX, a post-training pipeline that converts clinical practice guidelines (CPGs) into executable decision logic. The authors extract guideline recommendations, represent them as decision trees, and compile them into Python functions. These functions are used to generate factual and counterfactual QA pairs with reasoning traces. The resulting data are used for supervised fine-tuning, followed by GRPO with verifiable rewards on factual QA.

Experiments on four clinical reasoning benchmarks show consistent improvements over the Qwen3.5-4B and 9B backbones. MedGuideX-9B improves average accuracy from 53.29% to 58.85%, a 10.42% relative gain. The paper also reports improved reasoning recall and physician preferences for its generated rationales.

Critical strengths
  1. A practical approach to guideline-based supervision.

The paper addresses a relevant problem: how to use the conditional decision rules in clinical guidelines to train medical LLMs. Converting guidelines into executable functions provides a systematic way to generate and verify training labels. The end-to-end pipeline is a potentially useful contribution.

  1. Consistent improvements across benchmarks.

MedGuideX improves over its backbone at both 4B and 9B scales across all four benchmarks. The results suggest that guideline-derived supervision can benefit medical QA, and the physician evaluation provides additional evidence of improved rationale quality.

Critical weaknesses
  1. Insufficient baselines to establish the contribution of executable decision logic.

The main contribution is the integration of executable clinical guidelines into the post-training pipeline. However, the experiments do not adequately isolate the benefit of this design from standard synthetic QA generation and post-training.

Related work has already explored executable clinical decision trees [1, 2] and program-generated counterfactual supervision [3]. The novelty of MedGuideX therefore lies primarily in combining these ideas for medical LLM training. This makes controlled comparisons particularly important.

The paper compares against raw CPG fine-tuning, but this is not equivalent to training on guideline-derived QA pairs without executable functions. A direct CPG-to-QA synthesis baseline using the same SFT and GRPO recipe, with matched data volume and training budget, is needed.

Furthermore, the CPG-derived process reward baseline in Appendix H starts from the same mixed-data SFT checkpoint and uses the same final-answer reward. Its main difference is the additional LLM-based process reward. Thus, the performance gap between this baseline and MedGuideX does not establish the benefit of executable supervision.

Without these controls, the reported gains could largely come from synthetic QA training and GRPO rather than the executable representation itself. This leaves the paper's central methodological claim insufficiently supported.

  1. Limited evidence that the model internalizes clinical decision logic.

The paper claims that MedGuideX learns the decision logic encoded in clinical guidelines, but the main evaluation focuses on final diagnostic accuracy. Improved accuracy does not necessarily imply that the model follows the intended decision rules.

The counterfactual data construction also raises concerns. The authors retain only interventions that change the outcome, potentially introducing shortcuts. In addition, the hidden variables required for counterfactual abduction may not be uniquely identifiable from the observed context. The paper does not explain how such ambiguous cases are handled.

A direct evaluation on held-out guidelines, covering decision-path correctness and both outcome-changing and outcome-preserving interventions, would better support the internalization claim.

Finally, the executable functions are mainly validated through automated checks. The paper does not report independent clinical validation of their fidelity to the source guidelines. Since these functions determine the training labels, their correctness is important to the claimed reliability of the supervision.

References

[1] Li et al. MedDM: LLM-Executable Clinical Guidance Tree for Clinical Decision-Making. arXiv, 2023.

[2] Deng et al. CPGPrompt: Translating Clinical Guidelines into LLM-Executable Decision Support. JAMIA, 2026.

[3] Vashishtha et al. Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code. ICLR, 2026.

Questions for authors
  1. Contribution of executable supervision.

Can the authors compare MedGuideX against a direct CPG-to-QA synthesis baseline without executable functions, using matched training data volume, reasoning traces, and SFT/GRPO budgets? This would help determine whether executable decision logic provides benefits beyond synthetic QA training.

  1. Decision logic and counterfactual reasoning.

Can the authors provide direct evaluation of decision-path correctness and counterfactual consistency on held-out guideline rules? How are cases handled when multiple hidden-variable assignments are consistent with the observed outcome?

  1. Clinical validation and physician evaluation.

Were the executable functions independently checked for clinical correctness against their source guidelines? Also, Figure 3 appears to include a case where one response gives a diagnosis different from the ground truth, despite the stated requirement that both models produce correct diagnoses. Could the authors clarify this discrepancy?

3 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer J2F4’s Comments

Thank you for identifying the missing control and the distinction between diagnostic gains and guideline application. References use the revised manuscript of 17 November 2026.

1. Contribution beyond synthetic QA and GRPO

Execution exceeds Direct QA in all 15 configurations on both benchmarks. With mixed SFT followed by factual RL, gains over the backbone are 8.7 versus 6.5 points on MedCaseReasoning and 6.9 versus 5.6 on MedQA. Thus, execution adds 2.2 and 1.3 points even when both arms use their strongest shared training schedule.

We match Qwen3.5-9B, guidelines, questions, full student inputs, generation model and reasoning effort, data volume, budgets, and GRPO scoring. Direct QA generates answers and rationales from guideline text. The executable arm runs the function for the action and verbalizes its trace. Both students receive the same function and learn from their respective answers, holding code exposure fixed.

You correctly identify the earlier process-reward baseline as a test of an additional judge from a shared checkpoint. Appendix H.5, page 25, clarifies that scope. The new attribution control appears in Figure 3a, page 7, and Appendix F.1, page 21.

2. Held-out decisions and the intervention filter

We test 100 triplets from EPFL MEDITRON source guidelines excluded from SFT and RL. Each contains an original case and single-variable edits that preserve or change its action. Models receive identical guideline text and complete variables, with versions queried independently. GPT-6.1 Sol xhigh generates the cases and action pseudolabels.

MedGuideX answers 205/300 cases correctly, versus 193/300 for Direct QA and 176/300 for the backbone. Its gains over Direct QA are 7.0 points on changing edits and 3.0 on preserving edits, each over 100 cases. The latter extends the benefit to the category excluded by the outcome-changing training filter.

These are individual action accuracies. Paired consistency requires joint decisions, and path correctness requires intermediate-condition checks. Table 3, page 8, and Appendix F.4, pages 21 and 22, specify the evaluation.

3. Hidden-state identifiability

The original pipeline does not guarantee action identifiability. Among 4,623 Boolean-hidden records, 1,123 admit two factual-compatible states with the same counterfactual output, yet the verifier rejects the alternative. In the numerical ICU rule, capacities of 12 and 13 both explain no action at census 30, but yield different actions at 36.

Appendix C.4, pages 16 and 17, reports the audit. Appendix E.1, page 21, states the criterion. All factual-compatible states must agree on the intervened action for single-answer supervision. Any compatible state should then be accepted. Conflicting actions require ambiguity-aware targets or exclusion. Checkpoints retain the original corpus. Factual-only RL separately adds 1.5 and 0.7 points over Direct QA without counterfactual training.

4. Clinical validation and the questionnaire discrepancy

Function validation was automated. A source check found an exception-precedence error affecting four of a pediatric urine-collection rule's six counterfactual records. Appendix C.4 identifies the source and affected references.

The acute-pancreatitis diagnosis in the original illustration was an error. Both archived responses give the reference diagnosis of hyperlipemic abdominal crisis. Figure 4, page 23, restores the full questionnaire with verified response excerpts and both diagnoses.

Section 4.6, page 10, and Appendix G, pages 22 to 24, specify 40 annotations from two physicians over 30 jointly correct cases, with 10 shared cases. Selection favored MedGuideX rationales, with identities blinded and order randomized. In Table 4, page 9, completeness receives 36 preferences versus 3, while conciseness is near parity at 17 versus 16. The preference in this sample centers on clinical coverage.

5. Novelty relative to prior work

MedDM and CPGPrompt use clinical decision structures for inference. Executable Counterfactuals develops program-based abduction, intervention, and prediction. Section 2.2, page 2, acknowledges these precedents and rule-derived supervised training. Our contribution is the training use of guideline execution, tested against matched Direct QA. The ablations also identify a stage-specific result, with counterfactual examples most effective in mixed SFT followed by factual RL.

Within the rebuttal time and compute budget, we prioritized the matched training and held-out decision tests. We will continue paired and path-level evaluation, independent clinical validation, and corrected-label evaluation beyond the rebuttal.

Thank you for the detailed rebuttal and additional experiments. The matched Direct QA comparison provides useful evidence that execution-derived supervision contributes beyond standard synthetic QA training. I also appreciate the new held-out decision evaluation.

However, the evaluation measures individual action accuracy rather than decision-path correctness or paired counterfactual consistency. The reliance on model-generated pseudolabels and the identified issues in function fidelity and counterfactual supervision also leave important reliability concerns unresolved.

I therefore raise my rating from 1 to 2.

Thank you for reconsidering the paper and for recognizing the value of the matched Direct QA comparison. You identified two questions that our original accuracy metric could not answer. Did the model get both decisions right for the same patient? Did its explanation identify the guideline condition that justified the decision? We revisited the same 100 held-out case triplets to answer both.

1. Paired counterfactual consistency. MedGuideX answered both versions correctly on 51 of 100 pairs that required a different action, compared with 41 for Direct QA. The distinction you raised matters here. Scoring only the edited case gives 61 versus 54. A model can earn that credit even if it was wrong on the original. Requiring both answers to be correct gives the 51 versus 41 comparison. On the 100 pairs where the action should stay the same, the result was 59 versus 52. This gain matters because our counterfactual training filter retained only edits that changed the action.

Table 3. Decisions on held-out guidelines. Each model saw the same guideline and answered each case separately. Each edit changed one patient variable. The first three columns score individual cases, and Overall pools all 300. The next two score pairs. The last two score the written decision path.

ModelOriginalPreserving editChanging editOverallPreserving pairChanging pairGPT-5.4 miniGPT-6.1 Sol
Qwen3.5-9B66.061.049.058.747.035.051.850.2
Direct QA 9B72.067.054.064.352.041.055.355.7
MedGuideX-9B74.070.061.068.359.051.058.960.1
Gain+2.0+3.0+7.0+4.0+7.0+10.0+3.6+4.4

Scores are percentages. Gain is MedGuideX minus Direct QA in percentage points, calculated before rounding. Reported path-score variances are 0.09315 for GPT-5.4 mini and 0.07276 for GPT-6.1 Sol.

2. Decision-path correctness. A correct action earns no path credit if the explanation uses the wrong branch or omits the decisive condition. Each judge received the guideline passage, case, reference path and one anonymous answer. A pass required the answer to identify the decisive patient facts, apply the relevant rule and exceptions, and derive the correct action. Both judges scored the same 300 answers per model in three runs at xhigh reasoning effort. MedGuideX exceeded Direct QA by 3.6 percentage points under GPT-5.4 mini and 4.4 under GPT-6.1 Sol. Section 4.5 reports the comparison, and Appendix K gives the full judge prompt.

3. Reliability of pseudolabels and path judgments. GPT-6.1 Sol generated the action pseudolabels used to score the first six columns. We separately checked the judgments behind the final column. Medical expert review agreed with 47 of 50 sampled GPT-6.1 Sol pass or fail verdicts on the written paths and disagreed with three. The 94% agreement reported in Appendix F.4 therefore applies to those path judgments. The action targets for the paired test remain model-generated.

4. Fidelity of executable functions. Comparing a pediatric urine-collection function with its source guideline revealed an exception that the function applied in the wrong order. Giving the exception precedence changes four of that rule's six counterfactual records. Appendix C.4 identifies the rule and the affected records. The check establishes a specific source error. Its prevalence across functions remains unknown. The reported checkpoints retain the original training data.

5. Ambiguity in counterfactual supervision. Among 4,623 training records with a Boolean hidden variable, 1,123 admit two hidden values consistent with the observed action. Both give the same action after the edit, yet the verifier rejects one value. Either value should earn credit in these cases. When compatible values imply different actions, the observed case does not support a unique target. Appendix E.1 gives this criterion. Figure 3a also tests a setting that avoids counterfactual training examples altogether. With factual RL alone, executable supervision still exceeds matched Direct QA by 1.5 points on MedCaseReasoning and 0.7 on MedQA.

We appreciate your suggestion to evaluate the same patient before and after an edit. It let us show that MedGuideX more often changes or retains the recommendation as the supplied rule requires, and both judges more often find that its explanation gives the reason for doing so.

Reviewer_AQjrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MedGuideX, which converts clinical guidelines into executable functions and uses them to generate factual and counterfactual QA for post-training. The developed models improve over their Qwen3.5 backbones on four medical question/diagnosis benchmarks.

Critical strengths
  1. The pipeline provides a systematic way to turn clinical guidelines into training cases with answers that can be reproduced by running the corresponding function.

  2. The evaluation covers both exam questions and clinical cases, with consistent accuracy gains across two model sizes.

Critical weaknesses
  1. The experiments do not isolate the contribution of executable supervision. The raw-guideline baseline learns to reproduce guideline text, while the process-reward baseline already uses the same function-derived QA and SFT checkpoint. It remains unclear how much of the improvement comes from the executable functions versus additional medical QA training.

  2. The evidence that the model learns the underlying decision rules is limited. The final GRPO reward checks format and answer correctness. The reasoning-recall metric and physician study assess the written explanations, but the paper does not directly test whether the model follows the correct guideline branch or changes its decision when a relevant patient variable changes.

  3. The paper does not report an independent clinical audit of the generated functions against the source guidelines. In Figure 1, the displayed code can recommend repeat testing even when neither lab value reaches its threshold. Errors at this stage would carry over to the training labels.

Questions for authors

How does MedGuideX compare with a baseline trained on QA generated directly from the same guidelines, with matched data volume and SFT/GRPO budgets?

Were any generated functions checked by clinicians against the original guidelines?

How does the counterfactual data pipeline handle cases where multiple hidden states fit the factual observation but imply different counterfactual answers?

1 comment
AuthorsOpenReview ↗

Authors’ Response to Reviewer AQjr’s Comments

Thank you for identifying the key attribution and validity questions. References below use the revised manuscript of 17 November 2026.

1. Does execution add value beyond medical QA training?

Execution improves over Direct QA in all 15 matched configurations on both benchmarks. With mixed SFT followed by factual RL, gains over Qwen3.5-9B are 8.7 versus 6.5 percentage points on MedCaseReasoning and 6.9 versus 5.6 on MedQA. Factual-only RL retains advantages of 1.5 and 0.7 points without counterfactual training.

Both arms share source guidelines, questions, student inputs, GPT-5.4 xhigh generation, data volume, budgets, and GRPO scoring. Direct QA generates answers and rationales from guideline text. Executable supervision obtains actions by running the function and verbalizes its trace. Both students receive the same function and learn from their own reference answers. The additional gain therefore tests supervision generation with code exposure and training resources fixed.

Raw CPG fine-tuning tests text exposure. The process-reward baseline tests an additional judge from a shared SFT checkpoint. The new Direct QA comparison addresses their attribution gap. Figure 3a, page 7, and Appendix F.1, page 21, give the results and protocol.

2. Does the model apply guideline rules under patient edits?

On 100 triplets from EPFL MEDITRON source guidelines excluded from SFT and RL, MedGuideX answers 205/300 cases correctly, versus 193/300 for Direct QA and 176/300 for the backbone. Each triplet contains an original case and two single-variable edits, one preserving the action and one changing it. Models receive identical guideline text and complete variables, with versions queried independently. GPT-6.1 Sol xhigh generates the cases and action pseudolabels used for scoring.

MedGuideX gains 7.0 points over Direct QA on changing edits and 3.0 on preserving edits, each over 100 cases. Its largest advantage occurs where the changed condition requires a different recommendation. The metric is individual action accuracy. Table 3, page 8, and Appendix F.4, pages 21 and 22, report the test.

Your reward observation is correct. Factual GRPO checks format and the final action. Execution traces supervise SFT, and hidden-state checks belong to the counterfactual verifier. Section 3.2, page 4, and Appendix E.1, page 21, now distinguish these roles.

3. Were functions checked against clinical sources?

Function validation was automated. Physicians evaluated model rationales. The diabetes source check confirms that repeat testing requires a qualifying laboratory criterion, with no action when none is met. Figure 2, page 3, corrects the original illustration.

A targeted check also found an exception-precedence error affecting a factual or intervened reference in four of a pediatric urine-collection rule's six counterfactual records. Appendix C.4, pages 16 and 17, identifies the source and affected records. Successful execution checks consistency with the program. Clinical fidelity additionally requires checking its conditions and exceptions against the source.

4. How are compatible hidden states handled?

Among 4,623 Boolean-hidden records, 1,123 admit two factual-compatible states with identical counterfactual outputs under exact list matching. Their action targets remain determinate, but the verifier accepts only the sampled state. A numerical counterexample also admits different counterfactual actions. The reported checkpoints retain the original corpus.

Appendices C.4 and E.1 give the validity criterion. A single target is justified when every factual-compatible state yields the same intervened action. Any compatible state should then receive credit. Conflicting actions require ambiguity-aware targets or exclusion. Equivalent hidden states call for a verifier correction, while conflicting actions require a target correction. Factual-only RL separately supports execution-derived supervision without these counterfactual targets.

Within the rebuttal time and compute budget, we prioritized these two experiments and audits. We will continue independent clinical validation, corrected-label evaluation, and paired and path-level guideline tests beyond the rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary for the Area Chair

Dear Area Chair,

Thank you for coordinating the review. The revision addresses two shared questions through matched Direct QA training and held-out guideline decisions. References use the revised manuscript of 17 November 2026.

1. Does execution contribute beyond synthetic QA?

All four reviewers questioned attribution. Both arms now share Qwen3.5-9B, guidelines, questions, full student inputs, generation model and reasoning effort, data volume, budgets, and GRPO scoring. Direct QA generates answers and rationales from guideline text. Executable supervision obtains actions by running the function and verbalizes its trace. Both students receive the same function, holding code exposure fixed.

With mixed SFT followed by factual RL, gains over the backbone are 8.7 versus 6.5 points on MedCaseReasoning and 6.9 versus 5.6 on MedQA. Execution adds 2.2 and 1.3 points and performs better across all 15 configurations. Factual-only RL retains advantages of 1.5 and 0.7 points without counterfactual training. Figure 3a, page 7, and Appendix F.1, page 21, give the comparison.

Direct QA captures substantial benefit from guideline-based training. Execution adds value with the source, questions, teacher, code exposure, and optimization budget fixed. Its advantage also persists under the strongest training schedule shared by both arms.

2. Do gains extend to guideline decisions under patient edits?

For AQjr, J2F4, and XeQE, we evaluate 100 triplets from EPFL MEDITRON source guidelines excluded from SFT and RL. Each contains an original case and single-variable edits preserving or changing its action. Models receive identical guideline text and complete variables, with versions queried independently. GPT-6.1 Sol xhigh supplies cases and action pseudolabels.

MedGuideX answers 205/300 cases correctly, versus 193/300 for Direct QA and 176/300 for the backbone. Gains over Direct QA are 2.0 points on original cases, 3.0 on preserving edits, and 7.0 on changing edits, each over 100 cases. The largest advantage occurs when a modified condition requires a different recommendation. Preserving edits extend the test to a category excluded by the outcome-changing training filter. Table 3, page 8, and Appendix F.4, pages 21 and 22, report individual action accuracy.

The two tests provide complementary evidence. The matched comparison attributes an additional gain to execution-derived supervision. The held-out test shows that this gain extends to applying new guideline rules under patient changes.

3. Validation and corrections

For AQjr, J2F4, and mdd6, Appendix C.4, pages 16 and 17, audits label validity. Among 4,623 Boolean-hidden records, 1,123 admit two factual-compatible states with the same intervened output, yet the verifier rejects the alternative state. A numerical counterexample admits different actions. These findings distinguish an overly strict state verifier from an ambiguous action target. Appendix E.1, page 21, specifies when all compatible states justify a single action.

A targeted source check found an exception-precedence error affecting four of a pediatric rule's six counterfactual records. Checkpoints retain the original corpus. The audit identifies concrete requirements for correcting labels and validating source fidelity.

Figure 2, page 3, corrects the diabetes illustration. Figure 4, page 23, retains the complete physician questionnaire with checked archived responses. Section 4.6 and Appendix G state the selection favoring MedGuideX rationales among jointly correct cases, with counts and ties in Table 4. Section 3.2, page 4, corrects intermediate-verification claims. Section 2.2, page 2, acknowledges prior decision structures and rule-derived training. Appendix E specifies source-guideline-level checkpoint selection.

The matched Direct QA comparison also fixes rationale supervision, format rewards, and decoding, addressing the reasoning-format training asymmetry raised by XeQE. Average benchmark accuracy rises from 53.29% to 58.85%, or 5.56 percentage points, under the reported protocol.

Within the rebuttal time and compute budget, we prioritized the two shared empirical concerns and these audits. We will continue independent clinical validation, corrected-label evaluation, thinking-enabled and output-failure checks, and paired and path-level tests beyond the rebuttal.

The revised evidence supports a practical contribution, turning guideline rules into training supervision that improves over matched Direct QA and better applies held-out rules. Thank you for considering the additional results and corrections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.