- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GEVHarness, a clinical reasoning framework that separates machine-checkable guideline rules from context-dependent semantic judgments. It uses a deterministic runtime to execute formalized rules, while LLMs interpret clinical evidence and maintain semantic beliefs. A verification and control module handles incomplete evidence and decision refinement.
Critical strengths
The paper studies an important problem of incorporating partially executable specifications into LLM reasoning. The distinction between semantic interpretation and deterministic runtime authority is interesting, and the authority-allocation experiments provide useful observations.
Critical weaknesses
- The methodology design seems very conventional. The framework mainly combines existing techniques, including guideline formalization, symbolic rule execution, explicit belief states, and iterative LLM verification. The paper does not sufficiently distinguish its core technical contributions from prior neuro-symbolic clinical reasoning and executable guideline systems.
- The central challenge of partial formalization is not fully addressed. The framework assumes that clinical guidelines can be correctly partitioned into executable and semantic predicates. However, determining this boundary, faithfully compiling guidelines, and accurately grounding semantic predicates are themselves major challenges.
- Insufficient ablation analysis over effectiveness. It introduces multiple components, including structured guidelines, iterative reasoning, verification, and evidence acquisition. The performance improvement of each components are not well analyzed. In particular, the matched one-step authority ablation shows a relatively modest improvement (+1.80 percentage points), while multi-step execution introduces much larger computational costs and performance gains.
Questions for authors
Please refer to the weakness part.
7 comments
Response to Reviewer RvZY (2/4)
W2. How does GEVHarness address predicate partitioning, faithful compilation, and semantic grounding?
We thank the reviewer for highlighting this important challenge. GEVHarness addresses partial formalization through predicate-level partitioning, source-grounded compilation into typed and traceable IR, and evidence-associated belief states. These mechanisms enable deterministic execution of explicit guideline logic while preserving context-dependent clinical judgments.
1. Predicate classification is based on operational clarity rather than data type. Conditions with explicit thresholds or logical criteria are represented as executable predicates, whereas those requiring interpretation of patient histories, symptoms, or ambiguous clinical descriptions remain semantic predicates. A single clinical recommendation can therefore contain both types without requiring the entire decision to be formalized.
2. Explicit numerical thresholds can be executed while their clinical inputs remain grounded in patient evidence. Consider the 2023 AGS Beers Criteria, which recommends avoiding nitrofurantoin in older adults with creatinine clearance (CrCl) below 30 mL/min.
| Guideline element | IR classification | Model's role | Runtime role |
|---|---|---|---|
| Patient age ≥ 65 years | Typed numeric input; executable comparison (≥ 65) | Extract age from patient records | Execute threshold comparison |
| CrCl < 30 mL/min | hybrid/number input; executable comparison (< 30) | Identify CrCl value and supporting evidence | Execute threshold comparison |
| Applicable care setting | Semantic predicate | Interpret clinical context | Read validated belief state |
| Target medication is nitrofurantoin | Semantic predicate | Identify current medication and decision target | Read validated belief state |
| Condition combination and avoidance recommendation | Executable condition/recommendation | Provide predicate values and evidence | Execute AND relationship and trigger recommendation |
The simplified guideline logic is:
(age >= 65) AND applicable_care_setting
AND target_medication_is_nitrofurantoin
AND (CrCl < 30 mL/min)
-> avoid_nitrofurantoin
This example illustrates how the model grounds patient-specific inputs while the runtime executes explicit thresholds and logical relationships.
Response to Reviewer RvZY (3/4)
W2 (continued)
3. Context-dependent clinical criteria remain semantic even when they participate in explicit logical relationships. Consider the NCCN Guidelines for Central Nervous System Cancers and their recommendations for recurrent primary central nervous system lymphoma (PCNSL). Treatment selection depends on prior high-dose methotrexate, whole-brain radiotherapy, and the duration of response to previous treatment. In particular, a response lasting approximately one year or longer requires clinical interpretation rather than an arbitrary 365-day threshold.
| Guideline element | IR classification | Model's role | Runtime role |
|---|---|---|---|
| Relapsed or refractory PCNSL | Semantic predicate | Interpret disease status and clinical records | Read validated belief state |
| Prior high-dose methotrexate | Semantic predicate | Interpret treatment history | Read validated belief state |
| Prior whole-brain radiotherapy | Semantic predicate | Determine prior treatment history | Apply explicit NOT relationship |
| Response lasted approximately one year or longer | Semantic predicate | Interpret treatment timeline in clinical context | Avoid introducing an unsupported exact threshold |
| Treatment suitability | Contextual recommendation | Interpret patient-specific treatment options | Record guideline support without asserting an unconditional recommendation |
The simplified logic is:
relapsed_or_refractory_PCNSL
AND prior_high_dose_methotrexate
AND NOT prior_whole_brain_radiotherapy
AND response_lasted_about_one_year_or_more
-> contextual_support_for_systemic_treatment
This example illustrates that deterministic logical operations can act on validated semantic judgments without converting the underlying clinical criteria into executable predicates. The model interprets complex treatment histories and contextual qualifiers, while the runtime applies explicit logical relationships.
4. Evidence-associated belief states preserve uncertainty and support runtime execution. During inference, the model records each predicate's value, evidence provenance, revision history, and assessment status (SUPPORTED, CONTRADICTED, UNCERTAIN, or UNASSESSED). Inputs that violate type, range, or evidence-reference requirements are rejected, while insufficiently supported conditions remain Unknown rather than being treated as False. The runtime executes guideline logic based on validated belief states; unresolved decision-critical conditions keep the corresponding branch unresolved. Verification and Bounded Control guide targeted re-evaluation. The system also separately records the model's final answer and the guideline's actual normative support.
5. Source fidelity and controlled experiments validate the formalization boundary. Codex (GPT-5.6-Luna) generates candidate Typed IR from versioned guideline sources, followed by automated checks of schema, identifiers, dependencies, source references, and VM execution. In a human audit of 300 decision units, two independent reviewers achieved 96.7% agreement (290/300). After adjudication, 283 units (94.3%) were faithful; 17 (5.7%) had substantive discrepancies: applicability omissions (n=4), logical or exception errors (n=7), and predicate misclassifications (n=6). An independent Codex review, without access to human review results, achieved 95.0% agreement with the adjudicated human results, the reference standard.
Granting runtime authority to executable structures improves performance from 23.83% to 46.63%, whereas semantic hardening in certain settings and incorrect authority allocation reduce it to 41.80% and 23.32%, respectively. These findings support enforcing explicit rules while preserving contextual clinical interpretation.
In summary, GEVHarness makes partial formalization actionable through predicate-level partitioning, source-grounded compilation into typed and traceable IR, and evidence-associated belief states, while preserving clinical interpretation. We have added the source-fidelity audit and a source-to-IR example in Appendix D.1 (Table 4), along with worked examples of predicate partitioning and grounding in Appendix D.3 (Tables 5 and 6).
Response to Reviewer RvZY (1/4)
We sincerely appreciate the reviewer’s recognition of the motivation behind our study of clinical reasoning under partially formalized specifications, GEVHarness’s separation of semantic interpretation from deterministic runtime authority, and the useful observations from our authority-allocation experiments. We sincerely hope that our responses can address your concerns.
W1. What is the core technical contribution beyond existing neuro-symbolic clinical reasoning and executable-guideline systems?
We sincerely thank the reviewer for the opportunity to clarify our contribution. We have also strengthened the discussion in the Related Work section to distinguish our approach more clearly from prior work.
Clinical decisions require patient-specific reasoning and reliable guideline adherence. Yet guidelines are only partially formalizable: some conditions have explicit rules, while others require contextual interpretation. The central challenge is to enforce formalizable constraints deterministically while preserving the flexibility required for patient-specific clinical reasoning. GEVHarness addresses this through an inference-time harness for partial formalization. Our contributions are threefold:
1. We identify the fundamental challenge of achieving reliable guideline adherence under partial formalization. Existing studies have advanced guideline-informed LLM reasoning through knowledge integration, structured guidance, and verification [1–5]. Meanwhile, executable-guideline systems [6–10] and neuro-symbolic approaches [11] incorporate symbolic logic into decision-making. However, providing guideline knowledge does not guarantee adherence, while attempting to formalize all clinical conditions risks restricting necessary clinical judgment. The key gap is therefore how to make partially formalized guidelines effectively constrain LLM reasoning while accommodating conditions that cannot be fully formalized. This is fundamentally different from simply providing guidelines to an LLM or converting all clinical decisions into rigid executable rules.
2. We introduce GEVHarness, an inference-time framework that makes partially formalized clinical guidelines actionable, rather than treating them as passive reference knowledge or requiring complete executable specifications. Its core principle, Selective Runtime Authority, selectively enforces explicit guideline requirements while preserving model interpretation for context-dependent judgments. For example, when a blocking exception remains unassessed, the model can still give a provisional answer, and the runtime records that guideline support for the affected treatment remains unresolved.
3. We validate GEVHarness through broad benchmark evaluations and controlled experiments to support this principle. GEVHarness achieves superior performance across all eight benchmarks in matched comparisons with both backbones. Controlled experiments show that enforcing executable guideline structures improves performance from 23.83% to 46.63%, whereas excessive semantic hardening and incorrect authority allocation reduce it to 41.80% and 23.32%, respectively. These findings show that effective guideline adherence depends not on maximizing formalization, but on enforcing explicit constraints while preserving context-dependent clinical reasoning.
In summary, GEVHarness makes partial guideline formalization actionable and verifiable, enabling explicit rules to constrain decisions while preserving patient-specific judgment. We hope this work contributes to more reliable clinical reasoning, where guideline adherence and patient-specific reasoning complement each other.
References
[1] Deng et al. CPGPrompt: Translating clinical guidelines into LLM-executable decision support. JAMIA, 2026.
[2] Ge et al. GuideTree: Guideline-induced review trees for long medical records. ACL Industry Track, 2026.
[3] Shen et al. MedGuideX: Internalizing decision logic from executable guidelines into LLMs. arXiv:2605.26567, 2026.
[4] Yun et al. Med-PRM: Medical reasoning models with guideline-verified process rewards. EMNLP, 2025.
[5] Zhang et al. GLEAN: Guideline-grounded evidence accumulation for high-stakes agent verification. ICML, 2026.
[6] Hripcsak et al. The Arden Syntax for Medical Logic Modules. SCAMC, 1990.
[7] Hripcsak. Writing Arden Syntax Medical Logic Modules. Comput Biol Med, 1994.
[8] Boxwala et al. GLIF3: A representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform, 2004.
[9] Sutton & Fox. The syntax and semantics of the PROforma guideline modeling language. JAMIA, 2003.
[10] Brandt et al. Cross-platform EHR-driven phenotyping using Clinical Quality Language. Learn Health Syst, 2020.
[11] Gorenshtein et al. Neuro-symbolic LLM integration in clinical medicine: A systematic review and taxonomy. Research Square, 2026.