acceptodds
Under review as a conference paper at ICLR 2027

ReLMem: Learning Recurrent Memory for Longitudinal EHR Modeling

Abstract

Longitudinal electronic health record (EHR) modeling requires integrating new visits with an expanding patient history. Yet the continual accumulation of clinical information imposes increasing computational and memory costs on large language models (LLMs) when they process and retain complete patient histories. A practical alternative is visit-wise recurrent compression, which incorporates each incoming visit into a compact, continually updated patient memory. However, under a fixed memory budget, successive updates must integrate new information without progressively losing critical historical evidence needed to subsequent tasks. To address this challenge, we introduce *Re*current *L*ongitudinal *Mem*ory (*ReLMem*), a framework that learns to maintain fixed-capacity patient memory for efficient downstream prediction with a frozen LLM. ReLMem equips this LLM with lightweight compression adapters to recurrently update the memory from its previous state and each incoming visit, without rereading earlier records. Specifically, we develop a multi-granularity optimization strategy to preserve task-relevant information throughout recurrent updates and support downstream prediction from the final memory. The intermediate supervision aligns attention outputs from compressed memory and the full history under identical queries, while prediction supervision minimizes cross-entropy with ground truth answers conditioned on the final memory. On EHR-based medication prediction, ReLMem approaches the F1 scores of full-history baseline while reducing average retained historical storage by 97.1%. Under the same memory budget, it improves macro- and micro-F1 over the strongest compressed-memory baseline by 4.66 and 4.75 percentage points, respectively. These results highlight the value of learning recurrent patient memory for efficient longitudinal EHR modeling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Longitudinal EHR Modeling, Recurrent Memory, Context Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_55GNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a multi-granularity recurrent KV-memory compression framework for longitudinal EHR modeling that treats historical evidence in tiers (recent, intermediate, distant). On MIMIC-IV it approaches full-history accuracy on medication and diagnosis prediction while substantially reducing inference cost.

Critical strengths

The ablations are thorough: the contribution of each memory tier is systematically isolated, a level of completeness rarely seen in this area, and the efficiency claims are backed by a clear mechanism and direct measurements.

Critical weaknesses

Both tasks are evaluated only on a single institution and a test set of roughly 300 patients in MIMIC-IV, the main comparisons lack paired uncertainty estimates, and generalization therefore remains unknown.

The medication task also uses admission-level diagnoses and procedures to predict medications initiated within the first 24 hours, so target information may already be present in the input, and no ablation is provided that rules out this confound.

Questions for authors

Can you validate on a second dataset or institution? Can you run a dedicated ablation for the input-leakage concern in the medication task? What is the training-time compute comparison, given that the intermediate alignment requires full-history KV during training?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 55GN (Part1)

Thank you very much for recognizing the memory and inference efficiency analysis and for suggesting additional evaluations to strengthen the study. We appreciate your comments on external validation, paired uncertainty, potential leakage, and training cost. We have incorporated the revisions prompted by your suggestions into the manuscript and marked them in red. These insightful suggestions have substantially improved the quality of our manuscript, and we sincerely thank you again. We address each point below.

Weakness 1: Both tasks are evaluated only on a single institution and a test set of roughly 300 patients in MIMIC-IV, the main comparisons lack paired uncertainty estimates, and generalization therefore remains unknown.

Response: Thank you very much for raising these concerns. We agree that evaluation on an additional data source and paired uncertainty estimates are important for assessing the robustness of our findings.

External evaluation. We evaluated Qwen3-4B on 30-day readmission prediction using EHRSHOT [1] (NeurIPS 2023), which contains longitudinal EHRs from the Stanford Medicine Research Data Repository (STARR), a data source independent of MIMIC-IV. The test set contains 2,189 samples from 1,190 patients. As shown in the following table, ReLMem achieves comparable performance to Full History with only a difference of 0.14 percentage points, suggesting that ReLMem retains effectiveness on another data source and clinical task.

Full HistoryReLMem
Macro-F1 (%)60.7960.65

Paired uncertainty estimates. We further performed 10,000 paired bootstrap resamples over the same 300 test patients, as shown below. Each resample uses the same patient for a baseline and Full History, allowing us to estimate the confidence interval of their performance difference directly. ReLMem has the smallest estimated performance loss among the evaluated methods, and both confidence intervals include zero.

MethodsMacro-F1 (%)Micro-F1 (%)Δ Macro-F1 (pp, 95% CI)Δ Micro-F1 (pp, 95% CI)
Full History62.2263.32ReferenceReference
LLM-Rsum18.9719.21−43.25 (−45.19, −41.25)−44.11 (−45.99, −42.12)
SnapKV11.9111.95−50.31 (−52.32, −48.21)−51.37 (−53.33, −49.36)
KVzip52.5354.52−9.69 (−11.65, −7.73)−8.80 (−10.59, −7.04)
Attention Matching6.448.32−55.78 (−57.85, −53.72)−55.00 (−57.16, −52.81)
RMT47.2848.42−14.94 (−16.64, −13.20)−14.90 (−16.43, −13.33)
CCM-merge57.2758.49−4.94 (−6.27, −3.59)−4.83 (−6.02, −3.62)
ReLMem61.9363.24−0.28 (−1.58, 1.01)−0.08 (−1.35, 1.20)

Revisions in the manuscript. We have added the external evaluation to Section 5.5.6 (Lines 510-514)‾\underline{\text{Section 5.5.6 (Lines 510-514)}}, and reported the paired uncertainty estimates in Appendix G.5 (Lines 1674-1693)‾\underline{\text{Appendix G.5 (Lines 1674-1693)}}.

Weakness 2: The medication task also uses admission-level diagnoses and procedures to predict medications initiated within the first 24 hours, so target information may already be present in the input, and no ablation is provided that rules out this confound.

Response: Thank you for highlighting this potential confound. We agree that current-visit information, particularly procedures, may include information unavailable at the time of medication prescription. We would like to clarify that our task formulation follows established prior work. Nevertheless, we sincerely recognize this concern and have conducted an additional ablation study following your suggestion.

Task formulation. We followed the medication prediction formulation adopted in GAMENet [2] (AAAI 2019) and SARMR [3] (IJCAI 2021). Both studies use current-visit diagnoses and procedures, together with prior patient history, to predict medications prescribed during the first 24 hours of hospitalization.

Procedure ablation. We also removed procedure information from the current visit and repeated the evaluation, as shown below.

  • Removing procedures reduces average F1 by only 0.51 percentage points for Full History and 0.30 points for ReLMem, suggesting that current-visit procedure information has a limited effect on predictive performance in this evaluation.

  • ReLMem still remains close to Full History after removing procedures, indicating that the compression comparison is also robust to this input change.

Current-visit inputFull History (Average F1, %)ReLMem (Average F1, %)
With procedures62.7762.59
Without procedures62.2662.29

Revisions in the manuscript. We have added this ablation to Section 5.1 (Lines 330-332) and Appendix G.4 (Lines 1659-1673)‾\underline{\text{Section 5.1 (Lines 330-332) and Appendix G.4 (Lines 1659-1673)}} to clarify the impact of current-visit procedure information on medication prediction.

AuthorsOpenReview ↗

Response to Reviewer 55GN (Part2)

Question 1: Can you validate on a second dataset or institution? Can you run a dedicated ablation for the input-leakage concern in the medication task? What is the training-time compute comparison, given that the intermediate alignment requires full-history KV during training?

Response: Thank you for these helpful suggestions. We have added the external evaluation and potential input-leakage ablation, as detailed in our responses to Weakness 1 and Weakness 2, respectively. Below, we report the additional training cost of intermediate attention alignment.

Training cost of intermediate alignment. We agree that the full-history reference introduces additional computation during the recurrent memory learning stage. We therefore conducted a detailed comparison of the computational cost of intermediate alignment, as shown below. Intermediate alignment adds only 8.48 seconds per step, as caching the Full History KV states reduces recomputation across successive memory updates while increasing peak GPU memory by 5.49 GiB.

ConfigurationTraining time (s/step)Peak GPU memory (GiB)
Without LinterL_{\mathrm{inter}}38.7842.74
With LinterL_{\mathrm{inter}}47.2648.23

Revisions in the manuscript. We have added this cost comparison to Appendix G.2 (Lines 1613-1616)‾\underline{\text{Appendix G.2 (Lines 1613-1616)}} to provide a more comprehensive discussion of intermediate alignment.

References

[1] Wornow, Michael, et al. "Ehrshot: An ehr benchmark for few-shot evaluation of foundation models." Advances in Neural Information Processing Systems 36 (2023): 67125-67137.

[2] Shang, Junyuan, et al. "Gamenet: Graph augmented memory networks for recommending medication combination." proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[3] Wang, Yanda, et al. "Self-Supervised Adversarial Distribution Regularization for Medication Recommendation." IJCAI. 2021.

AuthorsOpenReview ↗

A Gentle Reminder of the Feedback

Dear Reviewer 55GN,

Thank you again for your constructive suggestions. Your comments have helped us substantially improve the quality of our manuscript. Following your comments, we have:

  • Added external evaluation on EHRSHOT.
  • Reported paired bootstrap confidence intervals for performance differences.
  • Added an ablation removing current-visit procedures to examine their influence on medication prediction.
  • Quantified the additional training time and GPU memory required by intermediate alignment.

As the discussion deadline approaches, we would appreciate your thoughts on whether our responses address your concerns, along with any further suggestions.

Best regards,

The authors

Reviewer_gYE1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a new framework for managing long-sequence data in EHR data by focusing on memory update and curriculum learning .

Critical strengths
  • The overall design of the framework is fairly straightforward and logical, combining compression techniques with curricular learning.
  • The study presents extensive baselines in their experiments, albeit many of them fall into a similar memory compression category.
Critical weaknesses
  • The key gap that the paper aims to address (i.e., the inability of LLMs in handling long context EHR sequences) needs better and stronger evidence. Currently, only a few general citations are provided. A reader would want to see explicit evidence showing that modern LLMs indeed would have difficulty, especially when handling only tabular EHR data. The paper mentions “key challenge of recurrent memory is not merely to compress individual visits but to integrate new information while preserving historical evidence for downstream prediction.” While this is logical, no strong evidence of this issue is presented.
  • The paper does include other types of LLMs beyond the main Qwen-4B model, but those results are presented separately (not in main Table 1) and with a different formatting (like with no variation reported).
  • The baselines are all from the category of compression techniques. When the downstream discriminative performance is used for evaluation, one would ask why not use other established EHR predictive methods for these tasks, including the non-LLM ones.
  • Mimic-IV does have longitudinal data, but being an ICU-centric dataset means it's less suitable for long-sequence analysis.
Questions for authors

Do the authors claim other types of baselines are not needed here?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gYE1 (Part1)

Thank you for your careful review and constructive suggestions. Your comments have helped us substantially strengthen the motivation for ReLMem, broaden the comparisons, and improve the presentation of our results. We have incorporated the revisions prompted by your suggestions into the manuscript and marked them in red. These insightful suggestions have substantially improved the quality of our manuscript, and we sincerely thank you again. We provide a point-by-point response below.

Weakness 1: The key gap that the paper aims to address (i.e., the inability of LLMs in handling long context EHR sequences) needs better and stronger evidence. Currently, only a few general citations are provided. A reader would want to see explicit evidence showing that modern LLMs indeed would have difficulty, especially when handling only tabular EHR data. The paper mentions “key challenge of recurrent memory is not merely to compress individual visits but to integrate new information while preserving historical evidence for downstream prediction.” While this is logical, no strong evidence of this issue is presented.

Response: Thank you for pointing this out. We apologize that the original manuscript did not provide sufficiently explicit evidence to support our motivation. First, we would like to clarify that the difficulty addressed in this work is primarily the increasing computational cost of processing patient histories as visits accumulate. Second, we evaluate performance using only tabular EHR data to clarify the role of clinical notes in EHR modeling for LLMs. Finally, we examine the importance of preserving historical evidence during recurrent memory updates for downstream prediction.

Computational cost of longer histories. In Transformer-based LLMs with dense attention, each new token attends to all preceding tokens. This supports effective modeling of longitudinal text but also incurs substantial computational costs as the history grows.

  • An intuitive example. In MIMIC-IV, a patient history comprising 10 visits contains approximately 38k tokens. For Qwen3-4B in BF16 precision, the corresponding KV states require approximately 5.27 GiB of GPU memory, equivalent to about 70% of the 7.49 GiB occupied by the model weights themselves.

  • Support from prior work. Previous studies have also reported substantial computational costs when applying LLMs to long EHR sequences. For example, Hegselmann et al. [1] (npj Digital Medicine 2026) reported that Qwen3-Emb-8B required 21.8 hours on eight H200 GPUs to encode approximately 1.16 million EHRSHOT prediction instances, with inputs capped at 8,192 tokens. MedTPE [2] (ICML 2026) reported that Llama3-8B inference on 5,815 MIMIC-IV ICU-mortality test cases required 6.3 hours on a single NVIDIA A5500 GPU.

  • Further evidence from experiments. We further sampled 110 patients with more than eight historical visits from the test set for evaluation, as shown below. As more visits are retained, the model achieves higher predictive performance, but its peak GPU memory and update latency both more than double. These results show that longer histories provide additional information for downstream prediction but also introduce substantial computational costs, which may limit the application of LLMs to longitudinal EHR modeling.

VisitsMacro-F1 (%)Micro-F1 (%)Peak GPU Memory (GiB)Update Latency (s/visit)
153.2055.668.690.2308
258.9161.019.620.2569
461.0462.0811.580.3369
861.2862.1915.390.4972
All61.4962.3117.660.5913

Analysis of tabular-only EHR data. Structured tabular data is a common form of EHR data and is widely used for clinical prediction. We therefore conducted an additional analysis using only tabular EHR data.

  • Experimental results on tabular-only EHR data. We removed clinical notes from the test set to construct tabular-only inputs for Qwen3-4B, as shown below. Using only tabular data reduces average F1 scores by 4.40 percentage points. This suggests that clinical notes contain information useful for downstream prediction and that LLMs can extract this information through their text modeling capabilities. Including clinical notes is therefore a natural choice for LLM-based EHR modeling.
DataMacro-F1 (%)Micro-F1 (%)
Only tabular data57.7858.97
With clinical notes62.2263.32
  • Further discussion about clinical notes. This finding is also consistent with prior works, where clinical notes contribute to downstream EHR prediction. For example, SERA [3] (Nature Communications 2021) improved sepsis AUROC by 0.10–0.15 by adding clinical notes for prediction 12–48 hours ahead. MSK-CHORD [4] (Nature 2024) achieved a C-index of 0.83 by integrating text-derived and structured features, compared with 0.78 using only cancer staging information, in stage I–III breast cancer.
AuthorsOpenReview ↗

Response to Reviewer gYE1 (Part2)

Importance of preserving historical evidence. We agree that the claim that recurrent memory must integrate new information while preserving historical evidence for downstream prediction requires direct empirical support. We therefore conducted the following experiments to examine the importance of patient history itself and of retaining key evidence during compression.

  • Impact of patient-specific history. We first examined how prediction with Full History depends on the patient’s own history. As shown below, replacing the patient-specific history with another patient’s reduces average F1 scores from 62.77% to 36.70%, while removing history further reduces it to 10.23%. These results demonstrate the importance of patient-specific history for downstream prediction.
History conditionMacro-F1 (%)Micro-F1 (%)
Patient-specific history62.2263.32
History from another patient36.2037.20
No history9.8110.65
  • Impact of historical evidence. We further examined how preserving historical evidence during recurrent memory updates affects downstream prediction. During recurrent memory learning, we introduce intermediate attention alignment (LinterL_{\mathrm{inter}}) to encourage the updated memory to produce attention readouts close to those of Full History, thereby promoting the retention of historical evidence. As shown below, the configuration without LinterL_{\mathrm{inter}} achieves an average F1 score 14.43 percentage points below the complete ReLMem framework. This result supports the importance of preserving key historical evidence for downstream prediction.
MethodMacro-F1 (%)Micro-F1 (%)
ReLMem w/o LinterL_{inter}47.6748.65
ReLMem (ours)61.9363.24
  • Support from prior work. These results are also consistent with prior studies on longitudinal EHR reasoning, which indicate the clinical importance of using evidence across visits and time. For example, TIMER [5] (npj Digital Medicine 2025) highlights the importance of integrating evidence across visits, and Kruse et al. [6] (EMNLP 2025) emphasize synthesizing relevant information from patient histories for clinical summarization and prediction.

Revisions in the manuscript. We have revised Section 1 (Lines 53, 70-72, Lines 86-89)‾\underline{\text{Section 1 (Lines 53, 70-72, Lines 86-89)}} and Section 2 (Lines 128-129)‾\underline{\text{Section 2 (Lines 128-129)}} to strengthen the motivation and related discussion, and added Section 5.1 (Lines 327-329) and Appendix G.3 (Lines 1646-1656)‾\underline{\text{Section 5.1 (Lines 327-329) and Appendix G.3 (Lines 1646-1656)}} to present the analysis of tabular-only EHR data. We have also added Section 5.5.7 (Lines 525-531) and Appendix G.7 (Lines 1717-1727)‾\underline{\text{Section 5.5.7 (Lines 525-531) and Appendix G.7 (Lines 1717-1727)}} to demonstrate the value of patient-specific history for downstream prediction, and these results, together with the intermediate attention alignment analysis, highlight the importance of preserving historical evidence.

Weakness 2: The paper does include other types of LLMs beyond the main Qwen-4B model, but those results are presented separately (not in main Table 1) and with a different formatting (like with no variation reported).

Response: Thank you for this suggestion. We apologize for the inconsistent presentation and agree that results across backbones should include uncertainty estimates in a consistent format. Below, we first explain the reason for separate presentation and then provide the missing uncertainty estimates.

Reason for the separate presentation. Given limited computational resources, our original manuscript included a complete comparison against all baselines only on Qwen3-4B to demonstrate ReLMem’s advantage in retaining the predictive performance of Full History. We then extended ReLMem to other backbones to examine whether its performance remains close to Full History across model scales and families.

Additional variation results on other backbones. We fully agree that omitting uncertainty estimates weakens the evidence for ReLMem’s generalizability across backbones. We therefore report the F1 scores (%) and their 95% CI below, where ReLMem achieves performance close to Full History across the evaluated backbones.

AuthorsOpenReview ↗

Response to Reviewer gYE1 (Part3)

BackboneMetricFull HistoryReLMem
Qwen3-4BMacro-F1 (95% CI, %)62.22 [60.46, 63.92]61.93 [60.21, 63.64]
Qwen3-4BMicro-F1 (95% CI, %)63.32 [61.69, 64.88]63.24 [61.62, 64.85]
Qwen3-8BMacro-F1 (95% CI, %)62.00 [60.16, 63.73]60.44 [58.67, 62.19]
Qwen3-8BMicro-F1 (95% CI, %)63.37 [61.74, 64.93]61.57 [59.97, 63.16]
Qwen3-32BMacro-F1 (95% CI, %)63.69 [61.84, 65.50]61.53 [59.72, 63.29]
Qwen3-32BMicro-F1 (95% CI, %)64.73 [63.05, 66.35]63.18 [61.53, 64.79]
Llama-3.1-8BMacro-F1 (95% CI, %)65.06 [63.40, 66.70]62.34 [60.53, 64.05]
Llama-3.1-8BMicro-F1 (95% CI, %)66.18 [64.66, 67.66]63.98 [62.40, 65.48]
MedGemma1.5-4BMacro-F1 (95% CI, %)53.80 [51.12, 56.40]54.26 [52.59, 55.86]
MedGemma1.5-4BMicro-F1 (95% CI, %)58.49 [56.56, 60.29]55.31 [53.77, 56.78]

Revisions in the manuscript. We have revised Section 5.5.5 (Line 498)‾\underline{\text{Section 5.5.5 (Line 498)}} and added detailed results to Appendix G.6 (Lines 1695-1714)‾\underline{\text{Appendix G.6 (Lines 1695-1714)}} to more clearly present ReLMem’s generalization across backbones.

Weakness 3: The baselines are all from the category of compression techniques. When the downstream discriminative performance is used for evaluation, one would ask why not use other established EHR predictive methods for these tasks, including the non-LLM ones.

Response: Thank you very much for this helpful suggestion. Our comparisons focused on compression methods because ReLMem mainly addresses the computational cost of LLM-based longitudinal EHR modeling. We also fully agree that comparisons with established EHR predictive methods, particularly non-LLM approaches, provide valuable support for the motivation of our work.

Rationale for comparing compression methods. ReLMem aims to reduce the computational cost of LLM-based longitudinal EHR modeling by recurrently compressing patient visit sequences into a fixed-capacity memory while maintaining downstream performance close to uncompressed Full History. We therefore compare various compression methods against the same uncompressed Full History reference.

Additional EHR baselines. Following your suggestion, we have implemented an MLP baseline and reproduced Doctor AI [7] (MLHC 2016), RETAIN [8] (NeurIPS 2016), and GAMENet [9] (AAAI 2019) for medication prediction, with the results shown below. Since these baselines do not encode clinical notes, Qwen3-4B also uses only tabular EHR data at inference in this comparison, but still outperforms all evaluated non-LLM baselines. These results support the effectiveness of LLMs in modeling complex EHR data and further motivate our study of LLM-based longitudinal EHR modeling.

MethodMacro-F1 (%)Micro-F1 (%)
MLP52.9153.86
Doctor AI54.3555.70
RETAIN54.6755.71
GAMENet53.7755.07
Qwen3-4B57.7858.97

Revisions in the manuscript. We have added this comparison to Appendix G.10 (Lines 1792-1807)‾\underline{\text{Appendix G.10 (Lines 1792-1807)}} to further support our motivation, which focuses on LLMs for longitudinal EHR modeling.

Reviewer_e3yNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReLMem, aiming to reduce the computation needed to remember the full history of the EHR. The framework consists of many existing components like a recurrent memory module and compression of KV by using LoRA with a memory token module. Curriculum learning and SFT are combined with a frozen LLM backbone. Testing on 300 patients shows similar accuracy with improved peak GPU memory and latency. Evaluations on Qwen3-4B, Qwen3-8B, and Llama-3.1-8B show that the framework can work on these LLMs.

Critical strengths
  • Like how GPs use records of our information and new percepts to decide how to give prescriptions and make decisions, longitudinal EHRs are indeed an important topic of research. And with LLMs, they could help doctors to provide useful information in a short time. So latency is a good motivation here.

  • The framework consists of many components, and an ablation study shows that the different loss combinations help the overall performance.

  • The framework is evaluated on different LLMs using different metrics like macro and micro F1, latency, and peak GPU memory.

Critical weaknesses
  • The framework is consisting of existing components. Many components are coming from CCM like recurrent memory design, token compression of KV, curriculum learning, SFT then freeze LLMs. All these are comes from existing techniques and I did not see a fundamental novelty here.

  • Limited evaluations on LLMs and datasets. The evaluation only used few small-scaled Qwen and Llama models. If the model originally performs poor, via some optimization it might still be poor. No powerful LLMs like GPT or Claude are evaluated. For open sourced models, parameters are also limited. Only 300 subjects mean the size of the dataset is too small.

  • In Table 3, remove LinterL_{inter} gives 47.67, which is lower than the baseline CCM-merge 57.27. This means ReLMem changes CCM's structure is failed.

  • SnapKV、KVzip、Attention Matching are training free, thus it is unfair to compare with the SFT of ReLMem.

Questions for authors
  • Except LinterL_{inter}, what is your core difference with CCM?

  • For your LinterL_{inter}, what is the core difference between existing attention reconstruction techniques like compressive transformer?

  • What are the performance when using SOTA LLMs like GPT-6.1 Sol and Claude 5.5 Opsus?

  • Can you provide the result of CCM-merge + $L_{inter}?

5 comments
AuthorsOpenReview ↗

Response to Reviewer e3yN (Part 1)

Thank you for recognizing the importance of efficient longitudinal EHR modeling and for identifying the comparisons needed to clarify our contribution. We have incorporated the revisions prompted by your suggestions into the manuscript and marked them in red. These insightful suggestions have substantially improved the quality of our manuscript, and we sincerely thank you again. We appreciate your suggestions and provide a point-by-point response below.

Weakness 1: The framework is consisting of existing components. Many components are coming from CCM like recurrent memory design, token compression of KV, curriculum learning, SFT then freeze LLMs. All these are comes from existing techniques and I did not see a fundamental novelty here.

Response: Thank you very much for raising this concern. We apologize that our original manuscript did not clearly distinguish ReLMem from CCM-merge or fully convey the novelty of our work. Below, we clarify the core difference, restate our contribution, and show that our training framework also benefits CCM-merge.

Core difference from CCM-merge. The fundamental distinction lies in how information from earlier visits is preserved across recurrent updates. Specifically, let Mt−1M_{t-1} denote the previous KV memory and M~t\widetilde M_{t} the newly generated memory, which is obtained by jointly encoding the current visit and Mt−1M_{t-1}. CCM-merge combines them through a predefined weighted average, MtCCM=t−1tMt−1+1tM~tM_t^{\mathrm{CCM}}=\frac{t-1}{t}M_{t-1}+\frac{1}{t}\widetilde M_t, whereas ReLMem directly uses the newly generated memory, MtReLMem=M~tM_t^{\mathrm{ReLMem}}=\widetilde M_t.

Novelty of ReLMem. Our contribution is a learning framework for efficient longitudinal EHR modeling that continually updates a fixed-capacity patient representation without re-encoding the full history at inference. Because even small information losses can accumulate across recurrent compression steps and lead to prediction errors, we introduce a multi-granularity optimization strategy. This strategy combines intermediate attention alignment (LinterL_{\mathrm{inter}}) to preserve historical evidence during memory updates and final prediction supervision (LpredL_{\mathrm{pred}}) to support accurate clinical prediction.

Effectiveness of ReLMem on CCM-merge. To further evaluate this training framework, we apply ReLMem’s intermediate attention alignment to CCM-merge. As shown below, adding LinterL_{\mathrm{inter}} yields an average gain of 2.26 percentage points, suggesting that LinterL_{\mathrm{inter}} helps preserve historical information useful for downstream clinical prediction.

MethodMacro-F1 (%)Micro-F1 (%)
CCM-merge57.2758.49
CCM-merge + LinterL_{inter}59.6060.68

Revisions in the manuscript. We have clarified these distinctions in Section 4.2 (Line 249) and Section 4.3 (Lines 286-288)‾\underline{\text{Section 4.2 (Line 249) and Section 4.3 (Lines 286-288)}} and added Appendix G.1 (Lines 1583-1598)‾\underline{\text{Appendix G.1 (Lines 1583-1598)}} to present the additional CCM-merge experiment as further evidence for the effectiveness of LinterL_{\mathrm{inter}}.

AuthorsOpenReview ↗

Response to Reviewer e3yN (Part 2)

Weakness 2: Limited evaluations on LLMs and datasets. The evaluation only used few small-scaled Qwen and Llama models. If the model originally performs poor, via some optimization it might still be poor. No powerful LLMs like GPT or Claude are evaluated. For open sourced models, parameters are also limited. Only 300 subjects mean the size of the dataset is too small.

Response: Thank you for this suggestion. We fully agree that broader evaluation is essential to establishing the effectiveness of ReLMem. Following your suggestions, we have expanded our evaluation to an external EHR data source, additional model families and scales, and stronger proprietary models.

External data source. We further evaluate ReLMem on EHRSHOT [1] (NeurIPS 2023), which is derived from the Stanford Medicine Research Data Repository (STARR), a data source independent of MIMIC-IV. Specifically, we use its 30-day readmission prediction task and evaluate on 2,189 test samples from 1,190 patients. As shown below, ReLMem is only 0.14 percentage points below Full History, supporting the effectiveness of our framework across data sources and prediction tasks.

Full HistoryReLMem
Macro-F1 (%)60.7960.65

Additional open source models. We have also evaluated ReLMem on MedGemma1.5-4B and Qwen3-32B to further examine its effectiveness across model families and scales. As shown below, ReLMem remains within 2 percentage points of Full History on both models.

BackboneFull History (Average F1, %)ReLMem (Average F1, %)
MedGemma1.5-4B56.1454.79
Qwen3-32B64.2162.35

Comparison with stronger proprietary models. As shown below, we additionally evaluate GPT-6.1 Sol and Claude Opus 5.5 on our test set, which both outperform Qwen3-4B. This result is consistent with our expectations of their relative capabilities, indicating that our test set can distinguish models with different levels of performance. However, the primary focus of ReLMem is whether maintaining a fixed-capacity patient state can approach the predictive performance of Full History at a lower cost in longitudinal EHR modeling. We therefore compare various compression methods with Full History while keeping the backbone fixed. Since proprietary models do not expose the internal states required by ReLMem, we could not apply ReLMem to them but included the results to demonstrate the effectiveness of our test set.

Model and configurationMacro-F1 (%)Micro-F1 (%)
GPT-6.1 Sol68.5669.62
Claude Opus 5.568.5469.89
Qwen3-4B, Full History62.2263.32
Qwen3-4B, ReLMem61.9363.24

Revisions in the manuscript. We have revised Section 5.5.6 (Lines 509-514)‾\underline{\text{Section 5.5.6 (Lines 509-514)}} to report ReLMem’s performance on the external data source, expanded Section 5.5.5 (Lines 495-499)‾\underline{\text{Section 5.5.5 (Lines 495-499)}} with the additional model evaluations, and added Appendix G.11 (Lines 1809-1822)‾\underline{\text{Appendix G.11 (Lines 1809-1822)}} to present the prediction performance of proprietary models on our test set.

AuthorsOpenReview ↗

Response to Reviewer e3yN (Part 3)

Weakness 3: In Table 3, remove LinterL_{inter} gives 47.67, which is lower than the baseline CCM-merge 57.27. This means ReLMem changes CCM's structure is failed.

Response: Thank you for this insightful comment. We respectfully clarify that this comparison needs to account for the different memory update strategies of ReLMem and CCM-merge. As discussed in our response to Weakness 1, CCM-merge combines the newly generated memory state with the previous memory through a weighted average, whereas ReLMem directly uses the newly encoded state. Therefore, for ReLMem, LinterL_{\mathrm{inter}} explicitly supervises the preservation of historical evidence in the memory update process, which plays an important role for downstream prediction. We also apply LinterL_{\mathrm{inter}} to CCM-merge and compare its performance with ReLMem, as shown below.

  • CCM-merge outperforms ReLMem without LinterL_{\mathrm{inter}}, highlighting the importance of retaining earlier information during recurrent updates in longitudinal EHR modeling.

  • Both CCM-merge and ReLMem perform better with LinterL_{\mathrm{inter}} than their respective variants without it, showing that explicit intermediate supervision can benefit different recurrent memory methods.

  • ReLMem outperforms the original CCM-merge, suggesting that explicitly supervising historical evidence preservation can be more effective than relying on weighted averaging alone.

  • Even with LinterL_{\mathrm{inter}}, CCM-merge still falls short of ReLMem. We think this is because averaging the previous memory with the newly generated state may dilute key information encoded in the new state.

MethodMacro-F1 (%)Micro-F1 (%)
ReLMem w/o LinterL_{inter}47.6748.65
CCM-merge57.2758.49
CCM-merge + LinterL_{inter}59.6060.68
ReLMem (ours)61.9363.24

Revisions in the manuscript. We have further clarified the differences between ReLMem and CCM-merge in Section 4.2 (Line 249)‾\underline{\text{Section 4.2 (Line 249)}} and Section 4.3 (Lines 286-288)‾\underline{\text{Section 4.3 (Lines 286-288)}} and added Appendix G.1 (Lines 1583-1598)‾\underline{\text{Appendix G.1 (Lines 1583-1598)}} to provide a more detailed comparison of the two methods.

Weakness 4: SnapKV、KVzip、Attention Matching are training free, thus it is unfair to compare with the SFT of ReLMem.

Response: Thank you for this comment. We agree that SnapKV, KVzip, and Attention Matching are training-free methods. In our experiments, we carefully controlled the comparison setup to make the evaluation as fair as possible, which also helps clarify the value of ReLMem for longitudinal EHR modeling.

The same task-adapted backbone. As summarized below, all baselines and ReLMem use the same LLM after task-specific adaptation. Their main difference is whether the memory compression mechanism is learned through additional training in the second stage.

MethodTask-specific adaptationRecurrent memory learning
Full History✓—
LLM-Rsum✓—
SnapKV✓—
KVzip✓—
Attention Matching✓—
RMT✓✓
CCM-merge✓✓
ReLMem✓✓

Additional training only focused on compression. ReLMem involves an additional stage of recurrent memory learning, but the LoRA adapter trained at this stage is used only during compression. It does not modify the weights of the task-adapted backbone and is not used when encoding the source text. Its purpose is only to compress the KV states produced by the LLM into a more compact representation.

Advantages of learned compression. Efficient recurrent modeling of longitudinal EHRs requires representing an ongoing sequence of visits within a fixed-capacity state while retaining the information needed for downstream prediction. A compressor trained with explicit supervision can learn to preserve this information under tight memory budgets. Below, we compare the average F1 scores of KVzip and ReLMem at different history budgets. KVzip, which does not use this explicit training supervision, requires substantially more memory capacity to achieve comparable downstream performance.

Methods12851210242048
KVzip13.5942.4453.5259.15
ReLMem59.6861.3762.5962.20

Revisions in the manuscript. We have clarified the comparison setup in Section 5.3 (Lines 363-365)‾\underline{\text{Section 5.3 (Lines 363-365)}} and expanded the discussion of the advantages of learned compression in Section 5.5.2 (Line 422)‾\underline{\text{Section 5.5.2 (Line 422)}}.

Question 1: Except LinterL_{inter}, what is your core difference with CCM?

Response: Thank you for this question, which helps us clarify our contribution. The core difference between ReLMem and CCM-merge lies in their memory update strategies and LinterL_{\mathrm{inter}} is designed to supervise the preservation of key information during ReLMem's recurrent memory learning. Please see our response to Weakness 1 for a more detailed explanation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Revisions

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely appreciate your time and effort in reviewing our manuscript and overseeing the review process. We are also grateful to the reviewers for their thoughtful suggestions, which have helped us substantially strengthen our work. As the discussion period draws to a close, we would like to summarize our core contribution and the key revisions.

Core contribution. ReLMem introduces a learning framework for maintaining fixed-capacity patient memory through recurrent compression in LLM-based longitudinal EHR modeling. By combining intermediate attention alignment with final prediction supervision, it preserves task-relevant historical evidence across memory updates and supports downstream prediction. Comparisons with general-purpose compression baselines demonstrate ReLMem’s stronger predictive performance. We further validate its effectiveness across data sources, clinical tasks, model scales, and backbone families.

Critical strengths

  1. Important research topic and clear motivation (Reviewer e3yN). The reviewer recognized longitudinal EHR modeling as “an important topic of research” and identified latency reduction as “a good motivation” for this work.

  2. Straightforward and logical framework (Reviewer gYE1). The reviewer highlighted the framework’s straightforward and logical design, combining compression with curriculum learning.

  3. Extensive baseline comparisons (Reviewer gYE1). The reviewer recognized the extensive baseline comparisons, covering a range of alternative approaches to memory compression.

  4. Well-supported efficiency results (Reviewer 55GN). The reviewer recognized that the efficiency results are supported by both a clear mechanism and direct computational measurements.

  5. Evaluation across multiple LLM backbones (Reviewer e3yN). The reviewer highlighted evaluations across multiple LLM backbones, covering different model scales and families.

  6. Thorough ablation studies (Reviewers e3yN and 55GN). The reviewers recognized that the ablations examine the contributions of individual components and training objectives. Reviewer 55GN specifically highlighted “a level of completeness rarely seen in this area.”

Key revisions and improvements

1. Clarified Contributions and Comparisons (Reviewer e3yN)

  • Clarified the distinctions from CCM-merge and Compressive Transformer.
  • Added experiments applying intermediate attention alignment to CCM-merge.
  • Examined how the choice of supervision reference affects intermediate attention alignment and downstream performance.
  • Clarified that all compression methods share the same task-adapted backbone, with ReLMem’s additional training dedicated solely to learning the compression adapter.
  • Expanded comparisons with training-free compression baselines across memory budgets, highlighting the benefits of learned compression under tight memory constraints.

2. Broader Evaluation Across Data Sources and Models (Reviewers e3yN, gYE1, and 55GN)

  • Added external evaluation on EHRSHOT for 30-day readmission prediction.
  • Extended ReLMem evaluation to MedGemma1.5-4B and Qwen3-32B.
  • Included prediction results from proprietary models to demonstrate that our test set distinguishes models with different predictive capabilities.

3. Strengthened Motivation for Longitudinal EHR Modeling (Reviewer gYE1)

  • Added an intuitive example and further references to illustrate the practical importance of efficient longitudinal EHR modeling.
  • Added experiments examining the predictive value of patient history and the computational cost of longer histories.
  • Broadened comparisons with established EHR methods, including MLP, Doctor AI, RETAIN, and GAMENet, to further support our focus on LLM-based longitudinal EHR modeling.

4. Expanded Statistical and Dataset Analyses (Reviewers gYE1 and 55GN)

  • Added paired bootstrap analyses of the baselines in Table 1, reporting confidence intervals for performance differences relative to Full History.
  • Reported 95% confidence intervals consistently across five backbones.
  • Added an ablation removing current-visit procedures to examine their potential shortcut effects on medication prediction.
  • Clarified the construction and longitudinal characteristics of our MIMIC-IV cohort, including visit counts and sequence lengths.
  • Added comparisons showing that clinical notes complement tabular-only data for EHR prediction.

5. Detailed Training Cost Analysis (Reviewer 55GN)

  • Quantified training time and peak GPU memory with and without intermediate attention alignment.

We have provided point-by-point responses and incorporated these revisions into the manuscript, with changes highlighted in red. We believe these additions further strengthen ReLMem. We would be grateful for your consideration of the revised manuscript and our responses. Thank you again for your time and guidance.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.