- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
MedGuideX converts clinical guidelines into executable decision functions and uses them to generate factual and counterfactual QA supervision. Mixed-data SFT followed by factual GRPO improves clinical reasoning performance across four benchmarks and two model scales.
Critical strengths
-
Transforming guideline decision logic into executable supervision is an interesting and well-motivated approach. It provides a structured alternative to treating guidelines as unstructured training text or retrieval material.
-
The improvements are consistent across benchmarks and model scales. Training-strategy ablations and a blinded physician evaluation provide additional supporting evidence.
Critical weaknesses
-
The counterfactual formulation may be underdetermined. Multiple hidden-variable assignments can reproduce the same factual outcome while yielding different counterfactual outcomes. The paper does not explain how such ambiguity is excluded, yet Equation (7) requires recovery of the particular sampled hidden state.
-
Executability and consistency with an extracted decision tree do not establish fidelity to the source guideline. Independent clinical auditing of the generated functions is needed to assess omitted conditions, exceptions, and unsupported rules.
-
A matched baseline that generates QA directly from the same guidelines would better isolate the benefit of executable supervision. The physician study is useful but small and restricted to cases both models answered correctly.
Questions for authors
-
Does counterfactual generation guarantee a uniquely identifiable hidden state, or at least an identical counterfactual outcome across all compatible hidden states? How are ambiguous examples handled?
-
Can the authors provide an independent clinical audit of a representative sample of executable functions?
-
How does the approach compare with direct guideline-to-QA synthesis under matched data volume, teacher model, and training budget?
2 comments
Comments
Thanks for your relply. Some of my concerns have been solved. I keep my original score as it is already positive.
Authors’ Response to Reviewer mdd6’s Comments
Thank you for the precise questions about label validity and matched supervision. References use the revised manuscript of 17 November 2026.
1. Hidden-state and action identifiability
The original generator guarantees neither a unique hidden state nor agreement across all compatible states. Your concern about Equation 7 is valid.
All 4,963 factual labels and both stored outputs of all 5,028 counterfactual records replay. Among 4,623 Boolean-hidden records, 1,123 admit both hidden values under exact factual-output matching. Both yield the same counterfactual action, but the verifier accepts only the sampled state. Thus, these action targets remain determinate while the state-matching condition is overly restrictive.
Numerical variables can also yield conflicting actions. In the ICU census rule, capacities of 12 and 13 both explain no action at census 30. At census 36, only capacity 12 triggers the three-times-capacity threshold. Appendix C.4, pages 16 and 17, reports these checks and 213 interventions that change only repetitions of identical action strings.
Appendix E.1, page 21, gives the validity criterion. A unique hidden state is unnecessary when all factual-compatible states agree on the intervened action. Any compatible state should then receive credit. Conflicting actions require ambiguity-aware targets or exclusion. Equivalent states call for a verifier correction, while conflicting actions require a target correction. Checkpoints retain the original corpus. The factual-only control below tests supervision without counterfactual abduction.
2. Clinical source fidelity
Function validation checks syntax, executability, and consistency with the extracted tree. These checks are automated. Physicians evaluated model rationales.
The diabetes source check confirms that repeat testing is gated by qualifying laboratory criteria. A pediatric urine-collection function contains an exception-precedence error. It can return both methods when the source requires catheterization if reliable clean catch is unavailable. Applying the exception changes a factual or intervened reference in four of its six counterfactual records. Two become action-preserving, while its two factual records remain unchanged. Appendix C.4 identifies the source and affected records.
This separates reproducible execution from faithful translation of clinical conditions and exceptions. Both factual and counterfactual labels depend on the latter.
3. Matched Direct QA comparison
Both arms use GPT-5.4 xhigh. Direct QA generates answers and rationales from guideline text and the question. Executable supervision obtains actions by running the function and verbalizes its trace. We match Qwen3.5-9B, source guidelines, questions, full student inputs, data volume, budgets, and GRPO scoring. Both students receive the same function and learn from their own references, fixing teacher quality and code exposure.
With mixed SFT followed by factual RL, execution gains 8.7 versus 6.5 points on MedCaseReasoning and 6.9 versus 5.6 on MedQA. It exceeds Direct QA in all 15 configurations. Factual-only RL retains advantages of 1.5 and 0.7 points without counterfactual training. Figure 3a, page 7, and Appendix F.1, page 21, report the comparison.
The held-out test supplies complete variables and includes both action-preserving and action-changing edits. MedGuideX answers 205/300 cases correctly, versus 193/300 for Direct QA and 176/300 for the backbone. GPT-6.1 Sol xhigh generates cases and action pseudolabels. Gains over Direct QA are 7.0 points on changing edits and 3.0 on preserving edits, each over 100 cases. Table 3, page 8, reports the results.
4. Physician preferences
Two physicians each assess 20 cases, yielding 40 annotations over 30 jointly correct cases with 10 shared cases. This compares rationale quality when final diagnoses agree. Selection favored MedGuideX rationales, while identities were hidden and order randomized. Appendix G, pages 22 to 24, specifies the sample.
Table 4, page 9, shows 36 versus 3 preferences for completeness and 17 versus 16 for conciseness. The pooled non-tie win rate is 76.9%. The preference within this sample is concentrated in clinical coverage. The benchmarks and matched Direct QA comparison provide the broader performance evidence.
Within the rebuttal time and compute budget, we prioritized these comparisons and audits. We will continue representative independent clinical validation and corrected-label evaluation beyond the rebuttal.