- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GLoC-EHR, a framework that connects structured EHR representations with an LLM through global and local memory modules. The global memory captures patient-level clinical context, while the local memory preserves individual clinical events. The model is trained through text alignment, rationale SFT, and GRPO with evidence-aware rewards to generate evidence-cited clinical predictions. Experiments on MIMIC-IV demonstrate competitive prediction performance and reduced unsupported evidence, with additional evaluation on EHRSHOT.
Critical strengths
Interesting and practically relevant problem. The paper addresses the important challenge of grounding clinical predictions in evidence from structured EHRs, making model outputs more interpretable.
Critical weaknesses
-
Limited evidence of faithful reasoning. The proposed evidence reward mainly relies on lexical matching with EHR records, which does not guarantee that the cited evidence actually supports the clinical conclusion or influences the prediction. The lack of independent clinical validation further weakens the claims of evidence-grounded reasoning.
-
Unclear benefits of the global-local architecture. Removing the local memory has little impact on prediction performance, and the reasoning model performs worse than its direct prediction counterpart. Given the complexity of the architecture and multi-stage training, the necessity and practical benefits of these components remain insufficiently demonstrated.
Questions for authors
-
How can the authors ensure that the cited evidence genuinely supports the clinical prediction rather than merely matching terms in the EHR? Could independent clinical expert evaluation help validate the faithfulness of the generated reasoning?
-
Given that removing the local memory has little impact on prediction performance, what is its essential contribution to the reasoning process? Could the authors provide stronger evidence that the global-local architecture is necessary compared with simpler alternatives?
6 comments
Thank you for your time and detail experiments and clarifications.
However, I still remain some concern about the reproducibility...I appreciate the plan to release code upon acceptance, but reproducibility is difficult to assess without it. Could the authors share an anonymized repository during review, including the training and evidence evaluation code?
Thank you for the helpful suggestion and we agree that reproducibility is crucial. Our code is available in an anonymized repository at link: "https://anonymous.4open.science/r/GLoC_EHR-8014/"
This is not the final version, since some baselines are not included yet. But it still contains the full training and evaluation code for GLoC-EHR.
Questions 1 and 2. Evidence support and the local route
Thank you for your thoughtful questions. Our lexical matcher only checks that a cited finding appears in the record. To check the evidence independently of it, we had gpt-oss-120b judge every evidence bullet against the full record, one bullet per prompt. The judge decides whether the record contains the stated finding, regardeless of the wording, so paraphrases and abbreviations count as matches. Under a strict binary criterion fixed before scoring, a bullet is supported only if every detail it states (level, quartile, drug exposure, dose, form) is correct.
The actual purpose of the local route is to let the model cite specific record details, such as the quartile of a lab value or the product details of a drug. The global-route-only variant, the simpler alternative in our ablations, writes the broad evidence well but often misses these details. Under the strict criterion, the local route gives 0.47 [0.46, 0.49] more distinct supported findings per response (3.88 vs. 3.41, +14%) and a slightly lower unsupported rate (0.528 vs. 0.537, a difference of −0.009 [−0.011, −0.007]). The global-only model states lab quartiles less often (3.6% vs. 8.9% of lab bullets) and less accurately (59% vs. 77% match the record), and more often gets drug product details wrong. This advantage grows as the record gets longer, that is, as the global-only model has to compress more events into its fixed-size memory. The local route gives 0.18 more distinct supported findings in the shortest quarter of records (≤45 entries) and 0.88 more (+18%) in the longest (>139 entries), where its unsupported rate is also 0.051 lower.
The lexical matcher shows the same once it is made strict. At the reward threshold (τ = 0.55), a bullet that names the right event matches even without such details, and the global-only model has the lower unsupported rate (0.470 vs. 0.508). At τ = 0.85, where a bullet must reproduce most of the event description, the local route cites 12.6% [11.5, 13.8] more distinct supported findings (Figure 2).
Also, when scoring the no-evidence model with same LLM judge under the standard criterion, which accepts a minor mismatch in details, it writes more unsupported bullets (4.41 vs. 3.64 per response), and the claims of its reasoning prose are more often unsupported (0.507 vs. 0.470, −0.037 [−0.052, −0.022], 1,002 cases). The cited evidence is also informative about the outcome. With the EHR withheld, the model's own bullets reach a macro AUROC of 0.711, against 0.665 when the sufficiency loss is removed.
We agree that independent clinical expert evaluation would help validate the generated reasoning. We are currently designing a clinician scoring study on a sample of the outputs and will add its results if it can be completed within the discussion period.