acceptodds
Under review as a conference paper at ICLR 2027

When Specifications Are Only Partially Formalized: Towards Reliable Clinical Reasoning

Abstract

Modern coding agents succeed when their environments provide reliable, machine-checkable feedback that can constrain model behavior. Software development exemplifies this regime: executable artifacts, compilation, testing, and execution create dense feedback signals that make actions verifiable and correctable. Clinical reasoning presents a different challenge. Clinical guidelines provide rich specifications, but only part of their content can be converted into machine-checkable structure, while the remainder requires context-dependent semantic judgment. The challenge is therefore to operationalize the executable portions of clinical specifications without forcing inherently semantic decisions into rigid rules. We introduce GEVHarness, a guideline harness for reliable clinical reasoning under partially formalized specifications. GEVHarness partitions guidelines into machine-checkable structure, which receives deterministic runtime authority over explicit state, and contextual judgments, which remain model-mediated. The model interprets evidence, while the runtime decides what follows, withholds commitment when evidence is incomplete, and makes every trajectory verifiable by replay. Across eight frozen benchmark panels spanning workflow control, open clinical reasoning, and evidence control, GEVHarness is the strongest interface among matched text and model-guided structured baselines for both evaluated backbone blocks; it improves the cases whose correct decisions change with evidence, and Controlled interventions further show that the value of formalization depends not only on how much of the specification is executable, but on where runtime authority is placed. These results position partially formalized specifications as a distinct inference-harness regime: a specification need not be a complete execution oracle to govern inference. More broadly, clinical reasoning exposes a persistent tension between semantic flexibility and executable reliability; GEVHarness offers one concrete step toward understanding how models, specifications, and runtimes should share control.

Primary area:
foundation or frontier models, including LLMs
Keywords:
clinical reasoning, large language models, clinical guidelines

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RvZYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GEVHarness, a clinical reasoning framework that separates machine-checkable guideline rules from context-dependent semantic judgments. It uses a deterministic runtime to execute formalized rules, while LLMs interpret clinical evidence and maintain semantic beliefs. A verification and control module handles incomplete evidence and decision refinement.

Critical strengths

The paper studies an important problem of incorporating partially executable specifications into LLM reasoning. The distinction between semantic interpretation and deterministic runtime authority is interesting, and the authority-allocation experiments provide useful observations.

Critical weaknesses
  1. The methodology design seems very conventional. The framework mainly combines existing techniques, including guideline formalization, symbolic rule execution, explicit belief states, and iterative LLM verification. The paper does not sufficiently distinguish its core technical contributions from prior neuro-symbolic clinical reasoning and executable guideline systems.
  2. The central challenge of partial formalization is not fully addressed. The framework assumes that clinical guidelines can be correctly partitioned into executable and semantic predicates. However, determining this boundary, faithfully compiling guidelines, and accurately grounding semantic predicates are themselves major challenges.
  3. Insufficient ablation analysis over effectiveness. It introduces multiple components, including structured guidelines, iterative reasoning, verification, and evidence acquisition. The performance improvement of each components are not well analyzed. In particular, the matched one-step authority ablation shows a relatively modest improvement (+1.80 percentage points), while multi-step execution introduces much larger computational costs and performance gains.
Questions for authors

Please refer to the weakness part.

7 comments
AuthorsOpenReview ↗

Response to Reviewer RvZY (1/4)

We sincerely appreciate the reviewer’s recognition of the motivation behind our study of clinical reasoning under partially formalized specifications, GEVHarness’s separation of semantic interpretation from deterministic runtime authority, and the useful observations from our authority-allocation experiments. We sincerely hope that our responses can address your concerns.

W1. What is the core technical contribution beyond existing neuro-symbolic clinical reasoning and executable-guideline systems?

We sincerely thank the reviewer for the opportunity to clarify our contribution. We have also strengthened the discussion in the Related Work section to distinguish our approach more clearly from prior work.

Clinical decisions require patient-specific reasoning and reliable guideline adherence. Yet guidelines are only partially formalizable: some conditions have explicit rules, while others require contextual interpretation. The central challenge is to enforce formalizable constraints deterministically while preserving the flexibility required for patient-specific clinical reasoning. GEVHarness addresses this through an inference-time harness for partial formalization. Our contributions are threefold:

1. We identify the fundamental challenge of achieving reliable guideline adherence under partial formalization. Existing studies have advanced guideline-informed LLM reasoning through knowledge integration, structured guidance, and verification [1–5]. Meanwhile, executable-guideline systems [6–10] and neuro-symbolic approaches [11] incorporate symbolic logic into decision-making. However, providing guideline knowledge does not guarantee adherence, while attempting to formalize all clinical conditions risks restricting necessary clinical judgment. The key gap is therefore how to make partially formalized guidelines effectively constrain LLM reasoning while accommodating conditions that cannot be fully formalized. This is fundamentally different from simply providing guidelines to an LLM or converting all clinical decisions into rigid executable rules.

2. We introduce GEVHarness, an inference-time framework that makes partially formalized clinical guidelines actionable, rather than treating them as passive reference knowledge or requiring complete executable specifications. Its core principle, Selective Runtime Authority, selectively enforces explicit guideline requirements while preserving model interpretation for context-dependent judgments. For example, when a blocking exception remains unassessed, the model can still give a provisional answer, and the runtime records that guideline support for the affected treatment remains unresolved.

3. We validate GEVHarness through broad benchmark evaluations and controlled experiments to support this principle. GEVHarness achieves superior performance across all eight benchmarks in matched comparisons with both backbones. Controlled experiments show that enforcing executable guideline structures improves performance from 23.83% to 46.63%, whereas excessive semantic hardening and incorrect authority allocation reduce it to 41.80% and 23.32%, respectively. These findings show that effective guideline adherence depends not on maximizing formalization, but on enforcing explicit constraints while preserving context-dependent clinical reasoning.

In summary, GEVHarness makes partial guideline formalization actionable and verifiable, enabling explicit rules to constrain decisions while preserving patient-specific judgment. We hope this work contributes to more reliable clinical reasoning, where guideline adherence and patient-specific reasoning complement each other.

References

[1] Deng et al. CPGPrompt: Translating clinical guidelines into LLM-executable decision support. JAMIA, 2026.

[2] Ge et al. GuideTree: Guideline-induced review trees for long medical records. ACL Industry Track, 2026.

[3] Shen et al. MedGuideX: Internalizing decision logic from executable guidelines into LLMs. arXiv:2605.26567, 2026.

[4] Yun et al. Med-PRM: Medical reasoning models with guideline-verified process rewards. EMNLP, 2025.

[5] Zhang et al. GLEAN: Guideline-grounded evidence accumulation for high-stakes agent verification. ICML, 2026.

[6] Hripcsak et al. The Arden Syntax for Medical Logic Modules. SCAMC, 1990.

[7] Hripcsak. Writing Arden Syntax Medical Logic Modules. Comput Biol Med, 1994.

[8] Boxwala et al. GLIF3: A representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform, 2004.

[9] Sutton & Fox. The syntax and semantics of the PROforma guideline modeling language. JAMIA, 2003.

[10] Brandt et al. Cross-platform EHR-driven phenotyping using Clinical Quality Language. Learn Health Syst, 2020.

[11] Gorenshtein et al. Neuro-symbolic LLM integration in clinical medicine: A systematic review and taxonomy. Research Square, 2026.

AuthorsOpenReview ↗

Response to Reviewer RvZY (2/4)

W2. How does GEVHarness address predicate partitioning, faithful compilation, and semantic grounding?

We thank the reviewer for highlighting this important challenge. GEVHarness addresses partial formalization through predicate-level partitioning, source-grounded compilation into typed and traceable IR, and evidence-associated belief states. These mechanisms enable deterministic execution of explicit guideline logic while preserving context-dependent clinical judgments.

1. Predicate classification is based on operational clarity rather than data type. Conditions with explicit thresholds or logical criteria are represented as executable predicates, whereas those requiring interpretation of patient histories, symptoms, or ambiguous clinical descriptions remain semantic predicates. A single clinical recommendation can therefore contain both types without requiring the entire decision to be formalized.

2. Explicit numerical thresholds can be executed while their clinical inputs remain grounded in patient evidence. Consider the 2023 AGS Beers Criteria, which recommends avoiding nitrofurantoin in older adults with creatinine clearance (CrCl) below 30 mL/min.

Guideline elementIR classificationModel's roleRuntime role
Patient age ≥ 65 yearsTyped numeric input; executable comparison (≥ 65)Extract age from patient recordsExecute threshold comparison
CrCl < 30 mL/minhybrid/number input; executable comparison (< 30)Identify CrCl value and supporting evidenceExecute threshold comparison
Applicable care settingSemantic predicateInterpret clinical contextRead validated belief state
Target medication is nitrofurantoinSemantic predicateIdentify current medication and decision targetRead validated belief state
Condition combination and avoidance recommendationExecutable condition/recommendationProvide predicate values and evidenceExecute AND relationship and trigger recommendation

The simplified guideline logic is:

(age >= 65) AND applicable_care_setting
AND target_medication_is_nitrofurantoin
AND (CrCl < 30 mL/min)
-> avoid_nitrofurantoin

This example illustrates how the model grounds patient-specific inputs while the runtime executes explicit thresholds and logical relationships.

AuthorsOpenReview ↗

Response to Reviewer RvZY (3/4)

W2 (continued)

3. Context-dependent clinical criteria remain semantic even when they participate in explicit logical relationships. Consider the NCCN Guidelines for Central Nervous System Cancers and their recommendations for recurrent primary central nervous system lymphoma (PCNSL). Treatment selection depends on prior high-dose methotrexate, whole-brain radiotherapy, and the duration of response to previous treatment. In particular, a response lasting approximately one year or longer requires clinical interpretation rather than an arbitrary 365-day threshold.

Guideline elementIR classificationModel's roleRuntime role
Relapsed or refractory PCNSLSemantic predicateInterpret disease status and clinical recordsRead validated belief state
Prior high-dose methotrexateSemantic predicateInterpret treatment historyRead validated belief state
Prior whole-brain radiotherapySemantic predicateDetermine prior treatment historyApply explicit NOT relationship
Response lasted approximately one year or longerSemantic predicateInterpret treatment timeline in clinical contextAvoid introducing an unsupported exact threshold
Treatment suitabilityContextual recommendationInterpret patient-specific treatment optionsRecord guideline support without asserting an unconditional recommendation

The simplified logic is:

relapsed_or_refractory_PCNSL
AND prior_high_dose_methotrexate
AND NOT prior_whole_brain_radiotherapy
AND response_lasted_about_one_year_or_more
-> contextual_support_for_systemic_treatment

This example illustrates that deterministic logical operations can act on validated semantic judgments without converting the underlying clinical criteria into executable predicates. The model interprets complex treatment histories and contextual qualifiers, while the runtime applies explicit logical relationships.

4. Evidence-associated belief states preserve uncertainty and support runtime execution. During inference, the model records each predicate's value, evidence provenance, revision history, and assessment status (SUPPORTED, CONTRADICTED, UNCERTAIN, or UNASSESSED). Inputs that violate type, range, or evidence-reference requirements are rejected, while insufficiently supported conditions remain Unknown rather than being treated as False. The runtime executes guideline logic based on validated belief states; unresolved decision-critical conditions keep the corresponding branch unresolved. Verification and Bounded Control guide targeted re-evaluation. The system also separately records the model's final answer and the guideline's actual normative support.

5. Source fidelity and controlled experiments validate the formalization boundary. Codex (GPT-5.6-Luna) generates candidate Typed IR from versioned guideline sources, followed by automated checks of schema, identifiers, dependencies, source references, and VM execution. In a human audit of 300 decision units, two independent reviewers achieved 96.7% agreement (290/300). After adjudication, 283 units (94.3%) were faithful; 17 (5.7%) had substantive discrepancies: applicability omissions (n=4), logical or exception errors (n=7), and predicate misclassifications (n=6). An independent Codex review, without access to human review results, achieved 95.0% agreement with the adjudicated human results, the reference standard.

Granting runtime authority to executable structures improves performance from 23.83% to 46.63%, whereas semantic hardening in certain settings and incorrect authority allocation reduce it to 41.80% and 23.32%, respectively. These findings support enforcing explicit rules while preserving contextual clinical interpretation.

In summary, GEVHarness makes partial formalization actionable through predicate-level partitioning, source-grounded compilation into typed and traceable IR, and evidence-associated belief states, while preserving clinical interpretation. We have added the source-fidelity audit and a source-to-IR example in Appendix D.1 (Table 4), along with worked examples of predicate partitioning and grounding in Appendix D.3 (Tables 5 and 6).

Reviewer_WH9SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEVHarness separates clinical guidelines into executable rules and judgments that still require a language model. A deterministic runtime applies the rules to an explicit belief state, tracks unresolved evidence, and checks execution by replay. The paper evaluates this design on eight benchmark panels, with authority ablations and additional backbone-transfer tests. Its main contribution is a practical way to let formal rules constrain inference without treating all clinical judgments as mechanically decidable.

Overall assessment: The scoped runtime design and matched authority ablation justify acceptance, but the broader mechanism and reliability claims should be narrowed pending the clarifications above.

Critical strengths
  • The design makes a useful distinction between execution correctness and clinical correctness. Typed beliefs, explicit unknown states, and separately recorded normative support make the guarantee inspectable. The paper correctly limits Proposition 1 to the compiled program and recorded beliefs, rather than claiming clinical safety.
  • There is evidence beyond a simple retrieval comparison. The one-call authority ablation improves accuracy over receipt-only access by 1.80 percentage points, with a paired 95% interval of [0.42, 3.27] (Table 3). Improvements across eight panels and four additional Qwen3.5 backbones strengthen the case that the overall system is useful.
Critical weaknesses
  • The broad authority-versus-access conclusion remains partly confounded. Sec. 3.3 says the harness can acquire hidden evidence while passive baselines cannot, but Appendix E.4 says matched conditions have identical evidence access. Table 3 also shows that much of the improvement accompanies substantially more inference: 73.07% at 7.85 calls versus 61.88% at one call. The one-call ablation supports a narrower authority effect; it does not explain the full system's advantage over an equally iterative, equally informed baseline.
  • Compilation fidelity is not quantitatively established. The system can consistently enforce a mistranslated rule. Sec. 2.1 says source-to-IR fidelity is reported separately, but Appendix D describes checks and review records without a measured semantic error rate, audit sample, or reviewer agreement. Given the 568-source registry, this missing evidence limits confidence in the reliability claim, even though the paper appropriately disclaims clinical validation.
Questions for authors
  1. Clarification: Can every matched baseline request the same hidden resources? Please reconcile Sec. 3.3 with Appendix E.4 and report actual calls/tokens by method. Is there a multi-step receipt-only comparison with the same acquisition and revision budget?
  2. Clarification: What source-to-IR audit was performed, by whom, and on how many rules? What errors were found, and were the guideline release and experience bank developed without evaluation-case labels?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WH9S (1/2)

We sincerely appreciate the reviewer’s thoughtful comments, as well as their recognition of the conceptual contribution and broad empirical evaluation of GEVHarness. We hope that our responses below adequately address the questions raised and alleviate the reviewer’s concerns.

W1/Q1. Are GEVHarness’s gains simply due to more inference computation, and are they worth the extra cost?

We thank the reviewer for raising this important question. We clarify evidence-access conditions across methods, introduce matched multi-step and single-call comparisons to isolate the contribution of guideline enforcement, and report the associated computational costs.

1. We distinguish evidence access from guideline execution to ensure fair comparisons. All methods are evaluated on the same frozen cases and case-source resources, but their interfaces differ in their ability to request hidden evidence. For static benchmarks, all methods receive the same initially visible evidence. For interactive benchmarks such as ClinCtrl-Acquire, GEVHarness can request case-associated resources based on unresolved clinical conditions, whereas passive interfaces primarily rely on currently visible information. Codex and Claude Code can also request case-associated resources and serve as relevant comparators. We have clarified initial and subsequent evidence access in Section 3.2 and Appendices E.3–E.4.

2. GEVHarness outperforms a matched multi-step baseline under nearly identical computational budgets. Multi-step ReadOnly retains GEVHarness's clinical state representation, guideline selection and execution, verification, and targeted revision, but treats execution results as advisory rather than binding for the final decision. Both configurations use the same revision protocol, impose a 20-call limit per case, and disable Experience, Association, and hidden evidence acquisition. Under these matched conditions, GEVHarness achieves 73.07%, compared with 67.84% for Multi-step ReadOnly (+5.23 percentage points). Their average calls are 7.85 and 7.89, respectively, with token consumption of 65,775 and 65,063. This isolates the added benefit of making guideline execution results binding for final decisions, beyond structured iteration alone.

3. GEVHarness also improves decisions without additional iterations. In the single-call ablation, merely exposing execution results does not outperform NoVM (60.08% vs. 61.46%). In contrast, granting these results normative authority improves performance over ReadOnly by 1.80 percentage points (95% CI [0.42, 3.27]). This shows that the benefit of GEVHarness is not contingent on multi-step reasoning.

4. Multi-step reasoning improves accuracy at a measurable computational cost. Compared with single-step G+V, multi-step G+V improves accuracy from 61.88% to 73.07% (+11.19 percentage points), while increasing average calls from 1.00 to 7.85 and token consumption from 9,303 to 65,775. We have also reported mean calls and total tokens per case for all six Qwen3.8-Flash interfaces in Appendix E.3. Table 3 gives the following accuracy and cost results for the six ablation configurations.

ConfigurationMean accuracyMean calls/caseMean tokens/case
NoVM (single-call)61.46%1.008,120
ReadOnly (single-call)60.08%1.009,105
Core (G+V, single-step)61.88%1.009,303
Generic Iterative Refinement66.53%8.9754,788
Multi-step ReadOnly67.84%7.8965,063
Core (G+V, multi-step)73.07%7.8565,775

5. On the matched static cases, making guideline execution binding improved accuracy over advisory feedback even when both systems could revisit the same evidence at comparable cost. On ClinCtrl-Acquire, the interfaces differed in which records they could obtain. We have clarified these access conditions in Appendix E.3.

AuthorsOpenReview ↗

Response to Reviewer WH9S (2/2)

W2/Q2. Compilation Fidelity and Evaluation-Label Use

We appreciate the reviewer’s thoughtful questions. GEVHarness's reliability depends on both correct runtime execution and faithful translation of the original clinical guidelines into IR. We therefore assess compilation through source-grounded mapping, automated validation, and human audit.

1. We map each clinical decision unit into typed and traceable IR. Each unit is decomposed into predicates, derived conditions, decision branches, exceptions, recommendations, and entry points, represented as Typed IR G=(P,D,C,X,R,Q)G=(P,D,C,X,R,Q). Each element retains its source reference and applicability scope. The mapping follows these principles:

Source guideline contentIR representationPreserved meaning
Applicable population and settingGuideline Cards / Entry PointsScope and decision entry
Clinical conditionsTyped PredicatesDefinitions and types
Thresholds and logical relationsDerived Conditions / BranchesComparisons and AND/OR/NOT
ContraindicationsExceptionsBlocking conditions
RecommendationsRecommendationsTargets and qualifications
Original evidenceSource ReferencesProvenance

Explicit thresholds and logical relations become executable structures, while context-dependent conditions remain semantic predicates. The central principle is preserving the original clinical meaning rather than maximizing executability.

2. We validate compilation through automated checks and independent source-level review. Codex (GPT-5.6-Luna) generates candidate IR from versioned guideline sources. Automated checks cover schema, identifiers, dependencies, source references, VM execution, and consistency between guideline-card entry points and compiled programs. Human review examines applicability, clinical meaning, rule coverage, logic, exceptions, and recommendation fidelity. Source tracking and version freezing preserve the provenance of released programs.

To quantify Source-to-IR fidelity, we audited 300 compiled decision units against their original guideline sources. Two independent reviewers achieved 96.7% agreement (290/300). After adjudication, 283 units (94.3%) were classified as faithful; 17 (5.7%) contained substantive discrepancies: applicability omissions (n=4), logical or exception errors (n=7), and predicate misclassifications (n=6). An independent Codex review using the same criteria, without access to the human review results, achieved 95.0% agreement with the adjudicated human results, which served as the reference standard. This audit provides a quantitative assessment of source fidelity beyond structural validation and runtime correctness.

3. Guideline compilation remains independent of evaluation labels. Guideline sources are selected according to the clinical domains covered by each benchmark, not its test-case gold labels. IR compilation and revision rely on original guideline texts and source references, without using gold answers. The Experience Bank contains procedural guidance for case interpretation, guideline selection, and reasoning verification, but no patient-specific answers or clinical facts derived from evaluation labels. Both the Guideline Release and Experience Bank are frozen before formal evaluation, with gold labels and final scoring kept outside the runtime inference environment.

We have clarified the audit and the use of evaluation information during development in Appendices D.1 and F.

Dear Reviewer WH9S,

Thank you again for your careful review. Your comments were exceptionally constructive and have materially strengthened the manuscript. We would like to briefly highlight three additions in our response.

  • Matched inference cost. We added two variants with comparable inference costs to distinguish the benefit of GEVHarness from the effects of additional computation and iteration. GEVHarness achieved 73.07% accuracy (7.85 calls per case), compared with 66.53% (8.97 calls) for Generic Iterative Refinement and 67.84% (7.89 calls) for Multi-step ReadOnly.
  • Evidence access. We clarified how the evidence-access mechanism works, including which methods can request additional case records during inference.
  • Compilation fidelity and label use. We added a two-reviewer audit of 300 compiled decision units. After adjudication, 283 (94.3%) were judged faithful. We also clarified how the guideline release and experience bank were developed and frozen before formal evaluation.

We are very grateful for your positive assessment and would welcome any further comments or suggestions that could help us further improve the manuscript. We hope these revisions address your concerns.

Thank you sincerely for your invaluable feedback and thoughtful guidance!

Reviewer_jidFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces GEVHarness, a guideline-driven framework for improving the reliability of LLM-based clinical reasoning under partially formalized specifications. Its key contribution is separating machine-checkable clinical rules from context-dependent semantic judgments, combining deterministic execution, explicit belief states, and replay-based verification. The framework is evaluated across eight benchmarks covering clinical workflow, diagnostic reasoning, and evidence control, consistently outperforming matched LLM-only, Text-RAG, and structured-navigation baselines. Ablation and authority-allocation studies demonstrate that selective runtime control improves performance, while transfer studies confirm improvements across multiple model backbones.

Critical strengths
  1. Well-motivated system design. GEVHarness combines deterministic rule execution with flexible model-based interpretation. Its explicit state representation and replay-based verification make the reasoning process more structured and traceable.

  2. Thoughtful mechanism analysis. Beyond comparing end-task performance, the authors conduct controlled interventions on runtime authority allocation. These experiments provide useful evidence that the placement of authority matters, rather than simply increasing the amount of formalization.

  3. Broad experimental coverage. The evaluation spans eight benchmarks covering clinical workflow control, open clinical reasoning, and evidence control. The matched comparisons and additional ablation studies provide a reasonably comprehensive assessment of the proposed framework across different task settings.

Critical weaknesses
  1. The trade-off of computational cost is insufficiently analyzed. The multi-step configuration inTable 2 requires 65,775 tokens and 7.85 calls per case, introducing substantial inference overhead. In contrast, the single-step variant yields only a 1.80% improvement over the Read-Only variant. This raises concerns about whether the performance gains improvement from the additional computational cost.
  2. The performance gains vary substantially across benchmarks, but the paper provides limited analysis. As shown in Table 1, GEVHarness achieves substantial improvements on MedPIC, while the gains on MedGUIDE-P256 and TriageBench are relatively subtle, despite that all three benchmarks involves guideline-related clinical decisions. Conversely, GEVHarness achieves a surprisingly large improvement on DiagnosisArena, an open-ended diagnostic reasoning benchmark where executable guideline coverage is expected to be relatively limited. The paper does not sufficiently explain which components primarily contribute to these gains.
  3. Analysis of source-to-IR fidelity lacks. The proposed framework relies on deterministic execution of compiled guideline rules, making the correctness and completeness of the source-to-IR transformation essential. Although the author acknowledges that source-to-IR fidelity requires separate evaluation, the paper does not provide concrete quantitative fidelity metrics or agreement statistics.
Questions for authors

Could the author clarify whether there is a clear criterion for distinguishing executable predicates from semantic predicates? In real-world clinical scenarios, this boundary is often ambiguous.

6 comments
AuthorsOpenReview ↗

Response to Reviewer jidF (1/5)

We sincerely appreciate the reviewer’s thoughtful comments and recognition of GEVHarness’s well-motivated design, mechanism analysis, and comprehensive experimental evaluation. Below, we provide clarifications and additional analyses addressing the remaining concerns. We genuinely hope that our responses adequately address your questions and alleviate your concerns.

W1: Are GEVHarness’s gains simply due to more inference computation, and are they worth the extra cost?

We thank the reviewer for raising this important question. GEVHarness's gains do not simply arise from increased inference computation, but from directing computation toward resolving clinical uncertainty and enforcing explicit guideline constraints. We examine these contributions through matched iterative baselines, single-call ablations, and computational cost analysis.

1. Generic iterative refinement alone cannot explain GEVHarness's performance gains. We introduce a Generic Iterative Refinement baseline that retrieves the top-5 guideline passages using BM25 and performs four rounds of general critique-and-revision, without explicit clinical states, deterministic guideline execution, or dedicated verification. It achieves 66.53% accuracy with an average of 8.97 calls and 54,788 tokens. In contrast, GEVHarness achieves 73.07% accuracy with only 7.85 average calls. This shows that repeated critique-and-revision alone cannot substitute for structured clinical reasoning and guideline execution.

2. GEVHarness outperforms a matched multi-step baseline under nearly identical computational budgets. We introduce Multi-step ReadOnly, which retains GEVHarness's clinical state tracking, guideline selection and execution, verification, and targeted revision, but treats execution results as advisory rather than binding for the final decision. Under nearly identical computational budgets, GEVHarness achieves 73.07%, compared with 67.84% for Multi-step ReadOnly (+5.23 percentage points). Their average calls are 7.85 and 7.89, respectively, with token consumption of 65,775 and 65,063. This isolates the additional benefit of making verified guideline execution genuinely constrain final decisions, beyond structured iteration alone.

3. GEVHarness also improves decisions without additional iterations. The single-call ablation shows that merely exposing execution results to the model does not outperform NoVM (60.08% vs. 61.46%). In contrast, granting these results normative decision-making authority improves performance over ReadOnly by 1.80 percentage points (95% CI [0.42, 3.27]). This confirms that the benefit of guideline enforcement is not merely a byproduct of multi-step reasoning.

4. Multi-step reasoning improves accuracy at a measurable computational cost. Compared with the single-step G+V variant, multi-step G+V improves accuracy from 61.88% to 73.07% (+11.19 percentage points), while increasing average calls from 1.00 to 7.85 and token consumption from 9,303 to 65,775. We do not claim that multi-step reasoning is equally cost-effective across all tasks. Rather, its value lies in enabling targeted verification and iterative resolution when unresolved clinical conditions materially affect decisions.

ConfigurationMean accuracyMean calls/caseMean tokens/case
NoVM (single-call)61.46%1.008,120
ReadOnly (single-call)60.08%1.009,105
Core (G+V, single-step)61.88%1.009,303
Generic Iterative Refinement66.53%8.9754,788
Multi-step ReadOnly67.84%7.8965,063
Core (G+V, multi-step)73.07%7.8565,775

We have added the two iterative controls to Table 3 so readers can weigh the accuracy gain against the cost of additional calls. Even when both systems can revisit a case at similar cost, letting an applicable rule constrain the final answer still improves accuracy on the matched cases. Appendix G describes their revision protocols.

AuthorsOpenReview ↗

Response to Reviewer jidF (2/5)

W2: Why do the performance gains vary across benchmarks?

We thank the reviewer for this insightful observation. We have also strengthened the relevant analysis in the Results section (Section 3.3). The variation in performance gains reflects the extent to which different clinical benchmarks depend on explicit guideline execution versus context-dependent clinical interpretation. Our eight benchmarks span clinical workflow control, open-ended clinical reasoning, and evidence management.

1. Benchmarks with explicit guideline logic benefit from runtime enforcement. On Qwen3.8-Flash, GEVHarness improves over direct inference by 20.77 percentage points on MedPIC, 0.78 on MedGUIDE-P256, and 3.00 on TriageBench. The corresponding gains over IR Navigation are 2.15, 11.72, and 11.00 points. On MedPIC, giving the model access to the structured rules already raises accuracy from 62.96% to 81.58%, close to the 83.73% achieved by GEVHarness. On MedGUIDE and TriageBench, IR Navigation falls below direct inference. This explains how GEVHarness can substantially improve on navigation while showing a smaller gain over the direct baseline.

2. Benchmarks requiring contextual clinical interpretation benefit from preserving model flexibility. MedBullets and DiagnosisArena rely more heavily on complex clinical reasoning, where guidelines cover only part of the decision process. GEVHarness is the strongest matched interface on both benchmarks for both backbones. On DiagnosisArena, Qwen3.8-Flash improves from 19.45% with direct inference to 41.94% with GEVHarness. A guideline can identify a finding that warrants further assessment while leaving several diagnoses plausible. Choosing among them still requires interpreting the case as a whole. On ClinCtrl-Acquire, GEVHarness also achieves higher diagnostic success than Claude Code and Codex on both backbones.

3. Controlled experiments confirm the importance of selective formalization. Granting runtime authority to executable structures improves performance from 23.83% to 46.63%. However, extending constraints to semantic judgments reduces performance to 41.80% in certain settings, while incorrect authority allocation further degrades it to 23.32%. These results show that more formalization does not necessarily lead to better clinical reasoning; effective guideline adherence depends on enforcing explicit rules while preserving contextual clinical interpretation.

The gains depend on how much of the clinical decision an applicable rule can settle. Once a medication risk is established, a rule can directly guide the choice on MedPIC. In open diagnosis, the same finding may warrant further assessment while leaving several diagnoses plausible. Section 3.3 now uses this distinction to explain why rule execution and clinical interpretation matter to different degrees across benchmarks.

AuthorsOpenReview ↗

Response to Reviewer jidF (3/5)

W3. How do we make the compilation of clinical guidelines into IR faithful?

We appreciate the reviewer’s constructive comments. Our Source-to-IR process preserves explicit guideline logic while keeping context-dependent clinical judgments semantic. We compile clinical decision units into typed and traceable IR and assess fidelity through automated validation and human audit.

1. We map each decision unit into typed and traceable IR. Each unit is decomposed into predicates, derived conditions, branches, exceptions, recommendations, and entry points, represented as (G=(P,D,C,X,R,Q)). Each element retains its source reference and applicability scope. Explicit thresholds and logical relations become executable structures, while context-dependent conditions remain semantic predicates. The mapping preserves applicability, AND/OR/NOT relations, exceptions, and recommendation qualifiers without adding unsupported criteria.

Source guideline contentIR representationPreserved meaning
Applicable population and settingGuideline Cards / Entry PointsScope
Clinical conditionsTyped PredicatesDefinitions and types
Thresholds and logicDerived Conditions / BranchesComparisons and AND/OR/NOT
ContraindicationsExceptionsBlocking conditions
RecommendationsRecommendationsTargets and qualifications
Original evidenceSource ReferencesProvenance

2. An IR example illustrates the mapping. We use NCCN Pancreatic Adenocarcinoma, Version 2.2026. For locally advanced disease progression, the guideline specifies prerequisites for considering chemoradiation or stereotactic body radiation therapy (SBRT): good performance status, no prior chemoradiation or SBRT, and progression confined to the primary site.

Source requirementIR fieldPreserved meaning
Locally advanced pancreatic cancerP_LOCALLY_ADVANCEDApplicability
Disease progressionP_PROGRESSIONDecision trigger
Good performance statusP_GOOD_PSRequired condition
No prior chemoradiation or SBRTP_NO_PRIOR_CRT_OR_SBRTTreatment restriction
Progression confined to primary siteP_PRIMARY_SOLE_SITEProgression restriction
All conditions requiredANDLogical relation
Consider chemoradiation or SBRTR_LOCAL_PROGRESSConditional recommendation

The resulting logic is:

AND(P_LOCALLY_ADVANCED, P_PROGRESSION, P_GOOD_PS,
    P_NO_PRIOR_CRT_OR_SBRT, P_PRIMARY_SOLE_SITE)
    -> R_LOCAL_PROGRESS
decision_effect = "contextual"

Each predicate and recommendation retains its source reference. The contextual effect means the conditions support considering chemoradiation or SBRT, rather than mandating a single treatment. This example illustrates how required conditions, logical relationships, and recommendation qualifiers are preserved in traceable IR.

3. We validate compilation through automated checks and independent source-level review. Codex (GPT-5.6-Luna) generates candidate IR from versioned guideline sources. Automated checks cover schema, identifiers, dependencies, source references, VM execution, and consistency between guideline-card entry points and compiled programs. Human review checks applicability, clinical meaning, rule coverage, logic, exceptions, and recommendation fidelity. Source tracking and version freezing preserve the provenance of released programs.

To quantify Source-to-IR fidelity, we audited 300 compiled decision units against their original guideline sources. Two independent reviewers achieved 96.7% agreement (290/300). After adjudication, 283 units (94.3%) were classified as faithful; 17 (5.7%) had substantive discrepancies: applicability omissions (n=4), logical or exception errors (n=7), and predicate misclassifications (n=6). An independent Codex review using the same criteria achieved 95.0% agreement with the adjudicated human results, which served as the reference standard. This audit assesses source fidelity independently of structural validation and runtime correctness.

A compiled rule can execute exactly as written even when its meaning has changed in translation. We therefore evaluated fidelity against the original guideline text. Appendix D.1 now reports both agreement between the reviewers and the discrepancies they found, so readers can judge the reliability of this step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.