acceptodds
Under review as a conference paper at ICLR 2027

GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Events

Abstract

Structured electronic health records (EHRs) contain a patient's clinical trajectory as a sequence of clinical codes. Answering clinical questions from such records requires both the context of the whole trajectory and the specific events that support the answer. We introduce GLoC-EHR, a multimodal language model that reads a contextual encoding of the record through a fixed-size global memory of the trajectory and a local memory of selected events. The model learns to generate hospital-course summaries from the global memory and descriptions of masked concepts from the local memory, aligning both with clinical text. It is then trained to cite evidence before answering, through rationale fine-tuning followed by group relative policy optimization (GRPO) with rewards for correct answers and record-supported evidence. On three MIMIC-IV outcome tasks, GLoC-EHR attains the highest macro AUROC among the compared models when it answers directly, whereas zero-shot LLMs reading the serialized record fall far behind. With evidence-cited reasoning, it stays close to its direct multi-task counterpart in macro AUROC, and the evidence terms of the objective reduce unsupported evidence at a similar macro AUROC. The local memory adds distinct supported findings, particularly under strict matching, without a detectable change in macro AUROC. Without retraining, GLoC-EHR transfers to EHRSHOT on par with EHR-BERT and answers two unseen laboratory questions better than zero-shot prompting of its own backbone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
electronic health records, clinical language models, multimodal learning, clinical reasoning, evidence grounding, reinforcement learning (GRPO), clinical outcome prediction, external validation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kj8hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GLoC-EHR, a framework that connects structured EHR representations with an LLM through global and local memory modules. The global memory captures patient-level clinical context, while the local memory preserves individual clinical events. The model is trained through text alignment, rationale SFT, and GRPO with evidence-aware rewards to generate evidence-cited clinical predictions. Experiments on MIMIC-IV demonstrate competitive prediction performance and reduced unsupported evidence, with additional evaluation on EHRSHOT.

Critical strengths

Interesting and practically relevant problem. The paper addresses the important challenge of grounding clinical predictions in evidence from structured EHRs, making model outputs more interpretable.

Critical weaknesses
  1. Limited evidence of faithful reasoning. The proposed evidence reward mainly relies on lexical matching with EHR records, which does not guarantee that the cited evidence actually supports the clinical conclusion or influences the prediction. The lack of independent clinical validation further weakens the claims of evidence-grounded reasoning.

  2. Unclear benefits of the global-local architecture. Removing the local memory has little impact on prediction performance, and the reasoning model performs worse than its direct prediction counterpart. Given the complexity of the architecture and multi-stage training, the necessity and practical benefits of these components remain insufficiently demonstrated.

Questions for authors
  1. How can the authors ensure that the cited evidence genuinely supports the clinical prediction rather than merely matching terms in the EHR? Could independent clinical expert evaluation help validate the faithfulness of the generated reasoning?

  2. Given that removing the local memory has little impact on prediction performance, what is its essential contribution to the reasoning process? Could the authors provide stronger evidence that the global-local architecture is necessary compared with simpler alternatives?

6 comments

Questions 1 and 2. Evidence support and the local route

Thank you for your thoughtful questions. Our lexical matcher only checks that a cited finding appears in the record. To check the evidence independently of it, we had gpt-oss-120b judge every evidence bullet against the full record, one bullet per prompt. The judge decides whether the record contains the stated finding, regardeless of the wording, so paraphrases and abbreviations count as matches. Under a strict binary criterion fixed before scoring, a bullet is supported only if every detail it states (level, quartile, drug exposure, dose, form) is correct.

The actual purpose of the local route is to let the model cite specific record details, such as the quartile of a lab value or the product details of a drug. The global-route-only variant, the simpler alternative in our ablations, writes the broad evidence well but often misses these details. Under the strict criterion, the local route gives 0.47 [0.46, 0.49] more distinct supported findings per response (3.88 vs. 3.41, +14%) and a slightly lower unsupported rate (0.528 vs. 0.537, a difference of −0.009 [−0.011, −0.007]). The global-only model states lab quartiles less often (3.6% vs. 8.9% of lab bullets) and less accurately (59% vs. 77% match the record), and more often gets drug product details wrong. This advantage grows as the record gets longer, that is, as the global-only model has to compress more events into its fixed-size memory. The local route gives 0.18 more distinct supported findings in the shortest quarter of records (≤45 entries) and 0.88 more (+18%) in the longest (>139 entries), where its unsupported rate is also 0.051 lower.

The lexical matcher shows the same once it is made strict. At the reward threshold (τ = 0.55), a bullet that names the right event matches even without such details, and the global-only model has the lower unsupported rate (0.470 vs. 0.508). At τ = 0.85, where a bullet must reproduce most of the event description, the local route cites 12.6% [11.5, 13.8] more distinct supported findings (Figure 2).

Also, when scoring the no-evidence model with same LLM judge under the standard criterion, which accepts a minor mismatch in details, it writes more unsupported bullets (4.41 vs. 3.64 per response), and the claims of its reasoning prose are more often unsupported (0.507 vs. 0.470, −0.037 [−0.052, −0.022], 1,002 cases). The cited evidence is also informative about the outcome. With the EHR withheld, the model's own bullets reach a macro AUROC of 0.711, against 0.665 when the sufficiency loss is removed.

We agree that independent clinical expert evaluation would help validate the generated reasoning. We are currently designing a clinician scoring study on a sample of the outputs and will add its results if it can be completed within the discussion period.

Thank you for your time and detail experiments and clarifications.

However, I still remain some concern about the reproducibility...I appreciate the plan to release code upon acceptance, but reproducibility is difficult to assess without it. Could the authors share an anonymized repository during review, including the training and evidence evaluation code?

Thank you for the helpful suggestion and we agree that reproducibility is crucial. Our code is available in an anonymized repository at link: "https://anonymous.4open.science/r/GLoC_EHR-8014/"

This is not the final version, since some baselines are not included yet. But it still contains the full training and evaluation code for GLoC-EHR.

Reviewer_SLazOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present GLoC-EHR, a language model that reads structured EHRs through a fixed-size resampled global memory and a top-k local memory of selected events. On three MIMIC-IV outcome tasks, the direct-answer variant matches or slightly exceeds fine-tuned EHR-BERT in macro AUROC. The reasoning variant lowers the share of unsupported evidence bullets at similar discrimination.

Critical strengths

The reward design is thought through carefully, with development runs that expose and close specific reward-gaming routes. The evaluation includes cited-slot masking with matched controls, a pre-registered replication of a readmission effect, and sensitivity analyses for chains without an answer.

Critical weaknesses
  1. The paper's own experiments do not support the dual-route design that motivates it. Removing the local route leaves macro AUROC unchanged. The global-only variant has a lower unsupported rate and a higher bullets-only AUROC, and gives up only 0.15 distinct supported findings per response at the reward threshold. Masking cited local slots changes neither the answer nor its AUROC relative to a control mask. The local route, its selector, three-phase concept alignment and two-phase SFT therefore add considerable complexity in exchange for a small change in lexical overlap at strict thresholds.
  2. The cited evidence and the answer seem to come largely from separate processes. Bullets-only AUROC is about 0.76, while the full model reaches about 0.87, so most of the discriminative signal behind the answer is absent from the cited findings. In the reasoning model, evidence functions as a parallel output rather than as the basis for the prediction, and this should be stated plainly in the main text.
  3. The reward and the evaluation share the same instrument. Support precision, unsupported rate and distinct supported findings are all computed with the matcher that the policy was optimized against. Re-scoring at stricter thresholds uses the same matcher family and does not break this circularity. Gains on these metrics may partly reflect learned surface agreement with OMOP description wording. Even after training, 44% of bullets match no event in the record. That level is high for a system presented as evidence-grounded, and the paper describes it mainly as an improvement over 58%.
  4. The leading gap (macro-auroc) is small. No significance tests are reported for Table 1. The closest generative EHR methods cited in the paper (Llemr, EHRMIND, EHR-R1) are not compared. Comparing a 1.7B-parameter model with a BERT-sized encoder also calls for a discussion of model size and training compute.
  5. The transfer and unseen-question results rest on weak references. On EHRSHOT, all models lose about 0.1 macro AUROC, and readmission prevalence differs threefold from MIMIC-IV, which points to a definitional mismatch rather than a clean transfer test. The share of events removed as out-of-vocabulary should be reported, since it directly determines how much of the external record the model sees. For the laboratory questions, the only comparator is the zero-shot backbone. A trivial baseline, such as whether the most recent sodium or platelet value was already low, would likely exceed AUROC 0.66 and is needed to judge whether these results show generalization.
Questions for authors

See weaknesses above.

3 comments

Thank you very much for the careful review. All numbers below use the corrected inputs (see our general comment), so some differ from those quoted in the review.

Questions 1 and 3. The local route and the shared matcher

We did not explain well why we also score evidence at a stricter matcher threshold, and this is tied to what the local route is for. The local route is not meant to change the answer but to let the model cite specific record details, such as the quartile of a lab value or the product details of a drug. The global-route-only model writes the broad evidence well but often misses these details, and under the lenient lexical matcher it has the lower unsupported rate (0.470 vs. 0.508). At the reward threshold (τ = 0.55), a bullet that names the right event matches even without such details, so we also counted distinct supported findings at τ = 0.85 (Table 2), where a bullet must reproduce most of the event description, including these details.

As the reviewer notes, this still uses the reward matcher. To break the circularity, we had gpt-oss-120b judge every evidence bullet against the full record, one bullet per prompt. Paraphrases and abbreviations count as matches. Under a strict binary criterion fixed before scoring, a bullet is supported only if every detail it states (level, quartile, drug exposure, dose, form) is correct. Under this criterion the local route gives 0.47 [0.46, 0.49] more distinct supported findings per response (3.88 vs. 3.41, +14%) and a slightly lower unsupported rate (0.528 vs. 0.537, a difference of −0.009 [−0.011, −0.007]). The global-only model states lab quartiles less often (3.6% vs. 8.9% of lab bullets) and less accurately (59% vs. 77% match the record), and more often gets drug product details wrong. This advantage grows as the record gets longer, that is, as the global-only model has to compress more events into its fixed-size memory. The local route gives 0.18 more distinct supported findings in the shortest quarter of records and 0.88 more in the longest, where its unsupported rate is also 0.051 lower. Scored by the same llm judge under the standard criterion, which accepts a minor mismatch in detail, the variant without the evidence terms writes more unsupported bullets (4.41 vs. 3.64 per response), and the claims of its reasoning prose are more often unsupported (0.507 vs. 0.470, −0.037 [−0.052, −0.022], 1,002 cases).

We also trained a Qwen3-4B variant with the same pipeline to see if scaling the backbone could reduce the unsupported rates. It reuses the EHR encoder, whose code-description embeddings were computed with Qwen3-1.7B. At step 2,400, the 4B variant writes 1.71 fewer unmatched bullets per response than the main model and about the same number of matched ones, at a similar AUROC (Table R3). This suggests that much of the remaining unmatched evidence reflects how well the backbone reads the EHR tokens, rather than a limit of the training objective.

Table R3. Evidence bullets per response at step 2,400 of 3,216.

Step 2,400Macro AUROCBulletsMatchedUnmatchedUnmatched rate
Main (1.7B)0.8398.404.174.230.507
4B variant0.8356.644.132.520.375

Question 2. Evidence as a parallel output

We agree and now state this plainly in the main text. With the corrected inputs, bullets-only AUROC is 0.711 against 0.842 for the full model, and the answer barely depends on the reasoning. Answering after an empty reasoning block lowers macro AUROC by only 0.004 [0.002, 0.006]. This follows from our training objective (details in our response to Reviewer 2yL1, Question 1-1). Without the answer loss, the label imbalance makes the policy drift toward "No", and down-weighting the loss 200-fold lowers test AUROC by about 0.09 on every task. The answer loss, however, lets the answer be read from the EHR directly. We therefore added the sufficiency loss, which trains the model to answer from its own bullets with the EHR withheld. Without it, bullets-only AUROC falls from 0.711 to 0.665, the level of the SFT model, while answer AUROC and evidence quality are unchanged. The reasoning is thus record-grounded evidence for the answer, not its cause.

Questions 4 and 5. Comparison with generative EHR models and transfer

We evaluated EHR-R1 in both its 1.7B and 8B variants. We did not add EHRMIND or Llemr. EHRMIND's code and weights are not publicly available, and its training, SFT warm-up followed by RL with an answer-only reward, is mostly same with our ablation without the evidence terms (Table 2). Since Llemr was trained mainly for question answering, we did not add Llemr as a new baseline .

Since our MIMIC-IV test tasks and data are aligned with our training setup, a direct comparison with EHR-R1 would not be fair, and the gap is large (macro AUROC 0.678 for EHR-R1-8B vs. 0.842 for GLoC-EHR). We therefore compare the models on EHRSHOT, an institution that neither model was trained on. EHR-R1 performed poorly on our record format, which gives value levels instead of raw values, so we gave it the same cases in its own format, with raw values, units and reference flags. For the two 24-hour tasks, we report this format both without and with the diagnosis codes of the current admission. With these diagnoses, EHR-R1 sees information that the other models do not (see our general comment on label leakage).

Table R5. EHRSHOT, the three tasks of Table 3 (AUROC).

ModelInputMortalityLong LOSReadmissionMacro
GLoC-EHR (Reasoning)Our record0.8080.6420.6420.698
EHR-R1-1.7BOur record0.7490.5380.4930.593
EHR-R1-1.7BEHR-R1 format, raw values0.8050.5400.5690.638
EHR-R1-1.7BEHR-R1 format, raw values + admission diagnoses0.8070.5420.5690.639
EHR-R1-8BOur record0.7080.6000.5890.632
EHR-R1-8BEHR-R1 format, raw values0.8480.6460.5650.686
EHR-R1-8BEHR-R1 format, raw values + admission diagnoses0.8630.6590.5650.696

Our record: the events and value levels that our model reads, given to EHR-R1 in its layout.

With raw values, EHR-R1-8B comes close to GLoC-EHR in macro AUROC (0.686 vs. 0.698). With the admission diagnoses as well, which GLoC-EHR does not see, it reaches 0.696, the same level as GLoC-EHR. EHR-R1-1.7B, the same size as our model, stays lower (0.638).

Regarding the vocabulary overlap, 75% of the clinical events in the EHRSHOT inputs are kept in our record, and the other 25%, whose codes are outside our MIMIC-IV training vocabulary, are removed (25.5% in the 24-hour inputs and 25.3% in the readmission inputs). We added these numbers to Section 4.5 and Appendix B.

Since our task definitions may differ from those of EHRSHOT, we also added eleven tasks defined by the EHRSHOT benchmark itself, with zero-shot Qwen3-1.7B and Llama-3.3-70B as LLM baselines (Table R6, Appendix C.5).

Table R6. Eleven additional EHRSHOT tasks, mean task AUROC per group [95% CI].

ModelLab abnormality (5 tasks)ICU transferNew diagnosis within 1 year (5 tasks)Macro (11 tasks)
GLoC-EHR (Reasoning)0.539 [0.528, 0.551]0.635 [0.597, 0.670]0.531 [0.514, 0.548]0.544 [0.535, 0.554]
EHR-R1-1.7B0.493 [0.481, 0.505]0.604 [0.563, 0.641]0.555 [0.540, 0.571]0.531 [0.522, 0.541]
EHR-R1-8B0.463 [0.452, 0.475]0.674 [0.638, 0.711]0.677 [0.662, 0.691]0.579 [0.570, 0.588]
Qwen3-1.7B (zero-shot)0.504 [0.493, 0.516]0.427 [0.389, 0.462]0.560 [0.544, 0.576]0.523 [0.513, 0.532]
Llama-3.3-70B (zero-shot)0.525 [0.513, 0.536]0.670 [0.633, 0.706]0.683 [0.667, 0.698]0.610 [0.600, 0.619]

On lab abnormality, only GLoC-EHR and Llama-3.3-70B are above chance, with our model slightly higher. On ICU transfer, an in-hospital outcome close to our training tasks, GLoC-EHR is close to EHR-R1-8B and Llama-3.3-70B. On new diagnoses within a year, however, the 1.7B models are all low and GLoC-EHR is the lowest (0.531), while EHR-R1-8B and Llama-3.3-70B reach about 0.68. This is likely because diagnosis codes appear only in the readmission inputs of our training data, so our model was trained mostly without them, and none of our training tasks asks a long-horizon question.

For the laboratory questions, we computed the suggested baseline on the same cases. Each positive case is the first low result of its admission (Appendix C.4), so the most recent value was already low in only 2% (sodium) and 6% (platelets) of cases, equally often before low and normal results. This baseline reaches AUROC 0.499 and 0.503, and GLoC-EHR (Reasoning) exceeds it by 0.109 [0.084, 0.133] and 0.089 [0.062, 0.115].

Thanks to the authors for their response. I'll keep my score as it is already positive.

Reviewer_2yL1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GLoC-EHR, a multimodal language model for evidence-cited clinical reasoning over structured electronic health records. Its main contribution is a global–local architecture that combines a fixed-size global memory summarizing the patient trajectory with a local memory of selected clinical events. The model aligns these representations with clinical text and employs supervised fine-tuning followed by group relative policy optimization, incorporating rewards for answer correctness and record-supported evidence, together with an evidence-sufficiency loss.

The model is evaluated on three clinical outcome prediction tasks using MIMIC-IV: in-hospital mortality, prolonged length of stay, and 30-day readmission. Results show that GLoC-EHR achieves comparable predictive performance to existing EHR models, with improvements in some tasks. Evidence-aware training reduces unsupported clinical findings while largely preserving outcome prediction performance. Additional experiments on EHRSHOT evaluate transferability without retraining, and two unseen laboratory prediction tasks assess generalization to new clinical questions. Ablation studies examine the respective contributions of global and local memory to prediction and evidence citation.

Critical strengths
  1. The paper includes ablation studies on the evidence-aware objectives, local memory, and different training stages. These analyses help clarify the effects of individual components and provide useful context for interpreting the experimental results.
Critical weaknesses
  1. The paper primarily uses lexical matching to evaluate evidence support, which does not fully capture clinical semantic correctness. In addition, masking cited local representations leads to no detectable change in predictive performance, although it does affect which findings are cited. These results leave some uncertainty about the extent to which the cited evidence meaningfully contributes to the model's predictions, a key aspect of the paper's proposed approach.
Questions for authors
  1. The current experiments do not isolate whether generating reasoning helps the model reach its answer. GLoC-EHR (Reasoning) and GLoC-EHR (MT) are different checkpoints. Moreover, GRPO includes cross-entropy supervision on the gold answer, so the improvement from rationale SFT (0.658) to GRPO (0.866) does not distinguish the contribution of this supervision from that of policy optimization. Table 8 also shows the same average answer sensitivity when masking cited or matched uncited local slots (|ΔpYes| = 0.013 in both cases). The reasoning prose is not checked for record support, while the evidence reward and evaluation use the same lexical matcher. Could the authors compare the same GRPO checkpoint with and without reasoning generation, and ablate the answer loss? An evaluation independent of the reward matcher, such as clinician review, would also help establish whether the generated explanations are useful.

  2. The evidence terms reduce the unsupported rate from 0.577 to 0.440 (Table 2), but the remaining proportion is high. With 7.83 bullets per response (Table 15), multiplying the reported averages gives roughly 7.83 × 0.44 ≈ 3.4 unsupported bullets per response. This is only an approximation, but it raises concerns about the usefulness of the evidence lists. The total bullet count without evidence terms is unreported, so the absolute reduction cannot be determined. As Appendix A.7 notes, some unmatched bullets may be valid paraphrases rather than hallucinations, making the percentage alone difficult to interpret. Could the authors report the actual average total and unsupported counts for both variants, and manually categorize a sample of unmatched bullets—for example, fabricated findings, valid paraphrases, general medical knowledge, or metadata?

  3. The main outcome evaluation covers only three tasks, which limits the assessment of the method’s generality. Could the authors evaluate the model on additional prediction tasks?

  4. EHR-R1, EHRMIND, and Llemr are not included in the experiments, leaving the gains over closely related EHR LLMs unclear. The introduction motivates the evidence objective through the absence of explicit record-support rewards in EHR-R1 and EHRMIND, but does not establish whether their outputs actually contain more unsupported findings. EHR-R1 includes a 1.7B variant trained on MIMIC-IV and reports EHRSHOT results; EHRMIND also evaluates on EHRSHOT, although its tasks and preprocessing differ. Could the authors compare these methods on compatible prediction tasks and, where possible, assess their outputs under the same evidence-support protocol? If differences in task or input format prevent a meaningful comparison, could they explain those constraints?

  5. The prompt templates, the teacher’s EHR serialization, and the “format and content checks” on teacher outputs are not specified. These details affect how the supervision was constructed and whether the results can be reproduced. Could the authors provide the exact prompts for BHC alignment, concept alignment, teacher rationale generation, direct prediction, and reasoning, along with the teacher’s decoding settings, filtering criteria, and fraction of outputs discarded?

4 comments

Thank you for the thoughtful suggestions. Running the requested experiments also gave us several new findings, summarized below. All numbers use the corrected inputs (see our general comment).

Question 1-1. Answer loss and direct-inference ablation

We have added the two losses (answer loss and sufficiency loss) for the following reasons. Without the answer loss, the label imbalance (about 72% of training prompts are "No") makes the policy drift toward "No" under the answer reward alone. In the training rollouts of a run without it, the Yes rate stays 0.11-0.13 below the label rate (0.03-0.05 with the answer loss), and the performance is 0.05-0.07 lower. Keeping the answer loss but down-weighting it (0.2 → 0.001) still degrades the model. At step 2,400 (75% of the single epoch; the full run did not fit in the discussion period), test AUROC is about 0.09 lower on every task, and 8.2% of chains give no valid answer (Table R1).

Table R1. Test AUROC at step 2,400 of 3,216.

Step 2,400MortalityLong LOSReadmissionMacroNo valid answer
Main (answer loss 0.2)0.9170.8020.7990.8390.2%
Answer loss 0.001, no sufficiency loss0.8230.7110.7020.7458.2%
Paired difference [95% CI]−0.090 [−0.115, −0.066]−0.088 [−0.100, −0.077]−0.097 [−0.118, −0.075]−0.092 [−0.104, −0.080]

As the reviewer points out, the answer loss lets the final answer be read from the EHR directly rather than through the generated reasoning. We therefore included the sufficiency loss, which trains the model to answer from its own evidence bullets with the EHR withheld. Removing it leaves answer AUROC and evidence quality unchanged, but bullets-only AUROC falls from 0.711 to 0.665, the level of the SFT model (Table R2). The two losses are thus complementary. The answer loss keeps the answer accurate, and the sufficiency loss trains answering from the evidence alone. However, even with our main configuration, answering after an empty reasoning block lowers macro AUROC by only 0.004 [0.002, 0.006], which aligns with Table 9.

Table R2. Test AUROC of the final checkpoints, by what the answer is computed from.

Final checkpointAnswer fromMortalityLong LOSReadmissionMacro
MainEHR + reasoning0.9170.8090.8020.843
MainEHR, empty reasoning0.9160.8050.7960.839
MainOwn bullets, EHR withheld0.8060.6540.6740.711
No sufficiency lossEHR + reasoning0.9170.8100.8030.843
No sufficiency lossEHR, empty reasoning0.9170.8080.7970.841
No sufficiency lossOwn bullets, EHR withheld0.7710.6220.6030.665

Question 1-2. Evaluation independent of the reward matcher

The local route is not meant to change the answer but to let the model cite specific record details, such as the quartile of a lab value. The global-route-only model rarely states such details, and under the lenient lexical matcher it has the lower unsupported rate (0.470 vs. 0.508).

To check this without the lexical matcher, we ran gpt-oss-120b locally to judge every evidence bullet against the full record, counting the paraphrases and abbreviations as matches. Under a strict binary criterion fixed before scoring, a bullet is supported only if every detail it states (level, quartile, drug exposure, dose, form) is correct. Under this criterion the local route gives 0.47 [0.46, 0.49] more distinct supported findings per response (3.88 vs. 3.41, +14%) and a slightly lower unsupported rate (0.528 vs. 0.537; −0.009 [−0.011, −0.007]). The global-route-only model states lab quartiles less often (3.6% vs. 8.9% of lab bullets) and less accurately (59% vs. 77% match the record), and more often gets drug product details wrong. This advantage grows as the record gets longer, that is, as the global-route-only model has to compress more events into its fixed-size memory. The local route gives 0.18 more distinct supported findings in the shortest quarter of records and 0.88 more in the longest, where its unsupported rate is also 0.051 lower.

Using the same LLM judge, we also checked the reasoning prose in 1,002 cases sampled by task and label. The judge first lists the claims the prose makes about the patient's recorded data (e.g., a lab level, a procedure). It then checks each claim against the full record; a claim counts as supported if the record contains the same finding, even when a minor detail differs. About 80% of the prose claims restate a finding already listed in the bullets, so the prose unsupported rate is close to that of the bullets (0.470 vs. 0.446). The evidence terms also lower the prose unsupported rate, from 0.507 without them to 0.470 (−0.037 [−0.052, −0.022]).

We are currently designing a clinician scoring study on a sample of the outputs and will add its results here if it can be completed within the discussion period.

Question 2. Bullet counts and categories of unmatched bullets

Thank you for pointing out the omission of detailed numbers and your estimate was close. The main model writes 8.36 bullets per response, of which 4.22 (50.8%) are not matched to the record. Without the evidence terms it writes 9.09, of which 5.86 (65.8%) are unmatched (Table R4). The evidence terms thus remove 1.6 unmatched bullets and add 0.9 matched ones per response, so the lower rate does not come only from writing fewer bullets.

Also, according to our additional experiments, the number of remaining unmatched bullets largely depends on the backbone scale. We trained a Qwen3-4B variant with the same pipeline, with reusing the EHR encoder, whose code-description embeddings were computed with Qwen3-1.7B (thus not fully optimized for 4B variant). At step 2,400, the 4B variant writes 1.71 fewer unmatched bullets per response than the main model and about the same number of matched ones, at a similar AUROC (Table R3). This suggests that much of the remaining unmatched evidence reflects how well the backbone reads the EHR tokens, even with an encoder built for the smaller model, rather than a limit of the training objective.

Table R3. Evidence bullets per response at step 2,400 of 3,216.

Step 2,400Macro AUROCBulletsMatchedUnmatchedUnmatched rate
Main (1.7B)0.8398.404.174.230.507
4B variant0.8356.644.132.520.375

We categorized every unmatched bullet with rules, and a sample of 450 per variant with the LLM judge. About 84% of the main model's unmatched bullets are errors (3.6 per response), either findings that are not in the record, or recorded findings stated with a wrong level (e.g., high/low) or a wrong detail. Valid paraphrases account for 0.65 per response, and general knowledge or metadata proportions are very low (Table R4). Of the 1.63 unmatched bullets that the evidence terms remove, about 60% are errors and 40% are paraphrases.

Table R4. Unmatched bullets per response by category, final checkpoints.

CategoryMainw/o evidence termsDifference
Not in the record (absent, a related but different entry, or an inferred diagnosis)2.412.69−0.29
Recorded finding with a wrong level (mostly the wrong direction) or a wrong detail1.151.83−0.68
Valid paraphrase0.651.30−0.65
General knowledge, risk statements, or metadata0.020.04−0.02
All unmatched4.225.86−1.63

Note that rows may not add up to the total because of rounding.

Questions 3 and 4. Comparison with EHR-R1 and EHRMIND, and additional tasks

We evaluated EHR-R1 in both its 1.7B and 8B variants. We did not add EHRMIND since its code and weights are not publicly available, and its training, SFT warm-up followed by RL with an answer-only reward, is also the same type of objective as our ablation without the evidence terms (Table 2), whose evidence is less often supported (Question 2).

Since our MIMIC-IV test tasks and data are aligned with our training setup, a direct comparison with EHR-R1 would not be fair, and the gap is large (macro AUROC 0.678 for EHR-R1-8B vs. 0.842 for GLoC-EHR). We therefore compare the models on EHRSHOT, an institution that neither model was trained on. We use the three tasks of Table 3 and eleven more tasks defined by the EHRSHOT benchmark. EHR-R1 was trained on records with raw lab values and diagnosis codes, and it performs poorly on our record format, which gives value levels instead of raw values (macro AUROC 0.632 for the 8B model). We therefore also give it the same cases in its own format, with raw values, units and reference flags. For the two 24-hour tasks, we report this format both without and with the diagnosis codes of the current admission. With these diagnoses, EHR-R1 sees information that the other models do not (see our general comment on label leakage). We did not score EHR-R1's evidence, because with reasoning enabled, neither variant does better, and the 8B model mostly answers with a number instead of yes or no (AUROC 0.535).

Table R5. EHRSHOT, the three tasks of Table 3 (AUROC).

ModelInputMortalityLong LOSReadmissionMacro
GLoC-EHR (Reasoning)Our record0.8080.6420.6420.698
EHR-R1-1.7BOur record0.7490.5380.4930.593
EHR-R1-1.7BEHR-R1 format, raw values0.8050.5400.5690.638
EHR-R1-1.7BEHR-R1 format, raw values + admission diagnoses0.8070.5420.5690.639
EHR-R1-8BOur record0.7080.6000.5890.632
EHR-R1-8BEHR-R1 format, raw values0.8480.6460.5650.686
EHR-R1-8BEHR-R1 format, raw values + admission diagnoses0.8630.6590.5650.696

Our record: the events and value levels that our model reads, given to EHR-R1 in its layout.

With raw values, EHR-R1-8B comes close to GLoC-EHR in macro AUROC (0.686 vs. 0.698). With the admission diagnoses as well, which GLoC-EHR does not see, it reaches 0.696, the same level as GLoC-EHR. EHR-R1-1.7B, the same size as our model, stays lower (0.638).

Table R6. Eleven additional EHRSHOT tasks, mean task AUROC per group [95% CI].

ModelLab abnormality (5 tasks)ICU transferNew diagnosis within 1 year (5 tasks)Macro (11 tasks)
GLoC-EHR (Reasoning)0.539 [0.528, 0.551]0.635 [0.597, 0.670]0.531 [0.514, 0.548]0.544 [0.535, 0.554]
EHR-R1-1.7B0.493 [0.481, 0.505]0.604 [0.563, 0.641]0.555 [0.540, 0.571]0.531 [0.522, 0.541]
EHR-R1-8B0.463 [0.452, 0.475]0.674 [0.638, 0.711]0.677 [0.662, 0.691]0.579 [0.570, 0.588]
Qwen3-1.7B (zero-shot)0.504 [0.493, 0.516]0.427 [0.389, 0.462]0.560 [0.544, 0.576]0.523 [0.513, 0.532]
Llama-3.3-70B (zero-shot)0.525 [0.513, 0.536]0.670 [0.633, 0.706]0.683 [0.667, 0.698]0.610 [0.600, 0.619]

On lab abnormality, only GLoC-EHR and Llama-3.3-70B are above chance, with our model slightly higher. On ICU transfer, an in-hospital outcome close to our training tasks, GLoC-EHR is close to EHR-R1-8B and Llama-3.3-70B. On new diagnoses within a year, however, the 1.7B models are all low and GLoC-EHR is the lowest (0.531), while EHR-R1-8B and Llama-3.3-70B reach about 0.68. This is likely because diagnosis codes appear only in the readmission inputs of our training data, so our model was trained mostly without them, and none of our training tasks asks a long-horizon question. We have added these results and limitation to Table 3 and Appendix C.5 (Table 19).

Question 5. Prompt templates and experimental settings

We have added all of these to the revised manuscript. The prompts are in Appendix D, and the teacher’s decoding settings, EHR serialization, format and content checks, and the number of discarded outputs per task in Appendix A.6 (Table 8).

Reviewer_2yL1OpenReview ↗

Comments

Thanks for the revision. Some of my concerns have been solved, I have raised my score.

Comments on the paper

2 comments

Major change: same-admission diagnosis codes removed from the 24-hour tasks

We thank all reviewers for their detailed and thoughtful feedback. We learned a lot from the new experiments that reviewers suggested. Before responding to each review, we would like to report one major change to our experiments.

According to Ramadan et al. [1], using diagnosis codes to predict outcomes of the same admission (e.g., in-hospital mortality) can cause label leakage. The final diagnoses are usually fixed at discharge, so they are future information for such tasks and can let models learn shortcuts. We therefore removed the diagnosis codes of the current admission from the inputs of the two 24-hour tasks (mortality and long length of stay) and re-ran all downstream experiments.

All models, including ours, lose AUROC on the 24-hour tasks, but the ranking by macro AUROC is nearly unchanged and our main claims still hold. GLoC-EHR (ST) remains the best in macro AUROC, and GLoC-EHR (Reasoning) drops the least (−0.024 vs. −0.030 to −0.032 for the other models in the table). We have updated all results in the revised PDF and reworded the statements that the new numbers no longer support. We will then respond to each reviewer's feedbacks individually.

AUROC on the MIMIC-IV test set (submission → revision).

ModelMortalityLong LOSReadmissionMacro
LightGBM (counts)0.9527 → 0.90550.8624 → 0.82080.7758 → 0.77580.8637 → 0.8340
EHR-BERT (ST)0.9607 → 0.91320.8701 → 0.82160.7817 → 0.78090.8708 → 0.8386
Llama-3.3-70B (zero-shot, direct)0.8307 → 0.77530.6663 → 0.62710.6056 → 0.60470.7009 → 0.6690
GLoC-EHR (ST)0.9561 → 0.91790.8740 → 0.81990.8075 → 0.80690.8792 → 0.8482
GLoC-EHR (Reasoning)0.9459 → 0.91580.8469 → 0.80740.8042 → 0.80200.8656 → 0.8417

[1] Ramadan, B., Liu, M.-C., Burkhart, M. C., Parker, W. F., & Beaulieu-Jones, B. K. (2025). Diagnostic codes in AI prediction models and label leakage of same-admission clinical outcomes. JAMA Network Open, 8(12), e2550454. https://doi.org/10.1001/jamanetworkopen.2025.50454

Dear Area Chair and Reviewers,

Thank you for your thoughtful and constructive feedback, which has substantially improved our paper. We have uploaded a revised manuscript incorporating the additional experiments, analyses, and clarifications discussed during the rebuttal. The main changes are as follows.

  • Abstract and Introduction. Updated to reflect the new results.
  • Section 4.3. LLM judge evaluation, strict-criterion analysis of the local route, and backbone size comparison.
  • Section 4.4. Controls for the answer loss and the evidence-sufficiency loss.
  • Section 4.5. EHR-R1 comparison on EHRSHOT and out-of-vocabulary statistics.
  • Section 5. Revised conclusion and limitations.
  • Appendix A. Corrected pretraining sampler (A.1) and teacher details (A.6).
  • Appendix B. Out-of-vocabulary stats for EHRSHOT.
  • Appendix C. Training-objective controls (C.1), LLM judge and larger-backbone results (C.2), a simple lab baseline (C.4), and the EHR-R1 comparison with eleven additional EHRSHOT tasks (C.5).
  • Appendix D. Complete set of prompts, including LLM-judge prompts.

The GLoC-EHR training and evaluation code is also available in our anonymized repository.

Thank you again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.