acceptodds
Under review as a conference paper at ICLR 2027

Code, Not Chains-of-Thought: Localizing Guideline-Adherence Risk by Compiling Guidelines into MedCode

Abstract

Clinical large language models can cite guidelines fluently while producing unsupported dosages, staging criteria, or treatment recommendations. We argue that this failure is partly architectural: when the same model both interprets evidence and emits the clinical action, there is no machine-checkable contract between a recommendation and the source guideline. We present MedCode, a model-agnostic framework that compiles oncology guidelines into executable, citation-bound Python rules and restricts the LLM to structured patient-state extraction. The rule engine, rather than the decoder, maps typed evidence to a closed ClinicalAction space with bounded verbatim citations and explicit abstention when required evidence is missing. This design does not eliminate clinical error; residual failures can still arise from extraction or rule authoring. Instead, it localizes guideline-adherence risk to two auditable choke-points, enforced by six static compliance gates and localized rule diffs under guideline updates. We instantiate MedCode on oncology staging and treatment tasks using NCCN-derived rule scaffolds and real clinical cohorts. On TNM staging, deterministic execution substantially improves smaller and open-weight models: Qwen3.6-27B rises from 25.95 to 54.91 TNM accuracy on LungCURE-EN, and Gemma-4-E4B-it rises from 17.31 to 58.70 on EsophagusPKUCH. On treatment recommendation, MedCode improves coverage of physician-specified gold-plan items across all evaluated models, although this rubric measures coverage rather than harmful or extraneous recommendations. Finally, the same executable rules serve as a verifiable GRPO reward for MedCode-LM: distilling a Qwen3.6-27B teacher into a Qwen3.5-4B student raises TNM accuracy from 35.64 to 50.34, approaching the teacher's 52.67 while preserving auditable decision semantics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
executable clinical guidelines, rule-based verification, oncology treatment recommendation, verifiable reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_wWKKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The main idea of the paper is to use EMR data to automatically fill out a document for the patient process it through a guideline constructing Python executable versions using LLMs.

Critical strengths

The paper addresses a timely problem.

Critical weaknesses

Major Issues

  1. The main contribution of the paper is to limit the LLMs output to machine readable format for EMR data; constructing Python executable decision graphs which then are checked by some predetermined steps. Training the smaller model and showing similar performance doesn't really have novelty as they are already widely known. I don't think the paper is novel enough for ICLR.
  2. The figure doesn't match up to ICLR standards.
  3. The manuscript reads more like a report rather then a scientific paper.
  4. The evaluation is being done by another LLM without any human validation
  5. The accuracy of the generated graphs are not evaluated.
  6. The accuracy of the generated EMR's are not reported.

Minor Issues

  1. The code type font on ``ClinicalAction'' should be removed.
  2. The usage of the word ``diff'' is informal.
  3. The third contribution is not mentioned at all in the abstract.
  4. The table presented should be self sufficient whereas the code type fonts makes it harder and the caption doesn't provide enough information.
  5. Phrases such as ``What is covered in this submission'' should be removed.
Questions for authors
  1. How accurate is the LLM at filling in the patient record on its own?
  2. How accurate is the LLM at writing rules from the guidelines?
  3. Did you check that the GPT-5.4 judge agrees with human doctors?
Reviewer_QTW8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MedCode compiles oncology guidelines into executable Python rules and restricts LLMs to structured patient-information extraction. The paper evaluates cancer staging and treatment-plan coverage and uses the rules to train a smaller model.

Critical strengths
  1. The paper addresses an important problem: making guideline-based recommendations easier to inspect and trace to their sources.

  2. The staging gains for several smaller models suggest practical value for tasks with clear rules and sufficient coverage.

Critical weaknesses
  1. It is unclear whether the added control is worth the loss of flexibility and rule-maintenance costs. Missing context and extraction errors can still lead to wrong decisions. These trade-offs need evaluation in cases with exceptions or conflicting evidence.

  2. The training contribution is not clearly distinguished from MedGuideX [1], which already uses executable guidelines to generate training data and rewards. The paper should explain what its training method adds.

  3. A key missing baseline is an LLM that can use the same rule tools, revisit the record, and explicitly abstain. The current comparison does not show why the LLM must be removed from decision-making.

  4. The ablation should separate extraction quality from execution, keeping the rules fixed and using both model-extracted and clinician-checked patient states. The performance drop for GPT-5.5 also needs error analysis. Saying that strong models have memorized the rules does not explain this loss.

[1] Shen, Y., Cao, L., Du, S., Wang, Y., Zhou, J., Peng, H., & Guo, Y. (2026). MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning. arXiv preprint arXiv:2605.26567.

Questions for authors

Please refer to the weaknesses above.

Reviewer_pWcqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MEDCODE, which restricts the LLM to structured patient-state extraction while deterministic, citation-bound Python rules map the extracted state to a closed set of clinical actions, with explicit abstention and six static compliance gates. It is evaluated on TNM staging (LungCURE-EN, EsophagusPKUCH) and treatment-plan coverage (a 202-case LungFAHGMU subset), with large gains for smaller open-weight models but not for frontier models. The rule engine is also used as a GRPO reward to post-train a Qwen3.5-4B student (MEDCODE-LM).

Critical strengths
  1. The evaluation uses real clinical datasets and popular model families. The experiments are conducted on real-world patient records and cover multiple widely used models, including GPT-5.5, DeepSeek-V4-Pro, Qwen3.6, and Gemma-4. The paper reports substantial gains for smaller and open-weight models, and candidly acknowledges that frontier models do not benefit and can even slightly degrade.
Critical weaknesses
  1. The novelty is limited. The core mechanism where an LLM extracts typed variables and deterministic code executes clinical rules already exists in the similar benchmarks. LungCURE's LCAgent extracts T, N, and M evidence separately and aggregates them with a deterministic rule-based node. CDR-Agent extracts variables from clinical notes and executes Python clinical-decision-rule scripts. However, neither method is included as a baseline.
  2. The effective scope of the method is limited, and its results are difficult to reproduce. On TNM staging, the gains are concentrated on smaller open-weight models, while the stronger models (GPT-5.5 and Qwen3.5-397B) show no improvement or even a decline. The authors attribute this to frontier models having "memorized the staging thresholds," but this explanation would predict no change rather than a decline, so the drop remains unexplained. In addition, two of the three datasets (LungFAHGMU and EsophagusPKUCH) appear to be private in-house cohorts, and all treatment results rely entirely on private data, which makes the main findings difficult to reproduce.
Questions for authors
  1. Can the authors compare against LCAgent on LungCURE TNM staging and against a simple baseline of structured extraction plus a standard AJCC staging calculator (or a PAL-style code baseline), to show what MEDCODE contributes beyond deterministic aggregation of extracted fields?
  2. Can the authors substantially revise the manuscript’s organization and exposition? Despite having some familiarity with medical LLMs, I found the paper difficult to follow for reasons beyond domain knowledge: clinical terminology, software contracts, evaluation settings, and training claims are introduced together without sufficient separation or definition. Given ICLR’s broad readership, the paper should clearly separate the clinical background and assumptions from the technical contributions, define terms such as TNM and NCCN at first use, and distinguish the 68-guideline scaffold from the actually authored executable rules, the runtime MEDCODE system, and MEDCODE-LM training.
Reviewer_ayGBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to address unsupported recommendations from clinical LLMs, which the authors attribute partly to the same model both interpreting the evidence and emitting the clinical action. They propose MedCode, which compiles NCCN guidelines into citation-bound Python rules, restricts the LLM to extracting a typed patient state, and lets a rule engine select an action from a closed set, abstaining when evidence is missing. The evaluation covers TNM staging on LungCURE and an in-house esophageal cohort, treatment-plan coverage on a 202-case lung subset scored by an LLM judge, and the distillation of a 27B teacher into a 4B student with SFT and GRPO using a rule-engine reward. The authors report large staging gains for several open-weight models and coverage gains for all evaluated models.

Critical strengths

I think the direction is reasonable. Keeping the reading of the record apart from the application of the guideline, and making the latter deterministic and tied to a verbatim citation, is a sensible route towards clinical recommendations that can be audited. The authors are also candid about what their gates do not certify and about the limits of their coverage metric.

Critical weaknesses

My main concern is that the contribution amounts to prompt engineering combined with hand-written rules, and that it is not evaluated against the alternatives that matter. The paper itself describes the design as restricting the LLM to structured extraction, with the decision left to the rule engine. Executable guideline representations have a long history, which the paper acknowledges, and the only learning component, the distillation, draws almost all of its gain from supervised fine-tuning on extractions, while the rule-based GRPO step that the paper calls decisive adds about one point (Table 6). At the same time, the comparison is thin: there is a single prompting condition whose prompt and guideline text are not specified, no other prompting strategy such as chain-of-thought, few-shot examples or code generation is tried, and none of the guideline-grounded systems discussed in the paper is actually run.

I am also not persuaded by the results that are reported. The method does not help a frontier model: for GPT-5.5 it is level with guideline prompting on lung staging and clearly worse on esophageal staging (75.90 vs. 68.27), and the paper's own ablation attributes the drop to restricting the model to typed extraction (Tables 3-4). The large gains in the abstract are measured against a no-guideline baseline and become much smaller against guideline prompting, and the treatment comparison uses a coverage-only metric judged by an LLM, on cases selected to be guideline-conformant, which does not penalize extraneous or harmful recommendations. More fundamentally, the properties the paper is built around, namely auditability, localized updates, abstention and consistent answers, are not measured. Runnable rules exist only for NSCLC, although the paper provides catalog scaffolds for 68 guidelines and evaluates esophageal staging, and it reports nothing about the number of rules or the effort of writing and maintaining them. When a new therapy enters a guideline, new rules, tests and citations would have to be written and verified by clinicians, and this authoring and maintenance burden, long the main obstacle for computable guidelines, is not addressed.

Beyond these points, the reporting needs substantial work. Several numbers cannot be traced to any table, statements in the text disagree with the tables, the paper contradicts itself on its data sources and on how the GRPO reward is computed, important results are deferred to a supplement that is not part of the paper, and variance and statistical tests are largely missing.

Questions for authors

How does MedCode compare with other prompting strategies (chain-of-thought, few-shot, code generation and execution) and with an existing guideline-grounded system on the same data?

Why does MedCode reduce GPT-5.5's accuracy, and what does it offer a strong model?

How many rules are there, which of them implement esophageal staging, and is there any evidence for the update and abstention claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.