acceptodds
Under review as a conference paper at ICLR 2027

Separating Laboratory Values and Care Events in ICU Representation Learning

Abstract

Self-supervised models of health records typically merge laboratory results and care events (laboratory orders and medication starts) into a single sequence read by one transformer, mixing signals about the patient’s physiology with signals about the care process. Because care processes differ across hospitals, we study whether these two types of information are better represented separately. We propose a coupled model in which laboratory values and care events are encoded by separate encoders and aligned with a contrastive loss. Across three ICU databases (eICU, MIMIC-IV, HiRID), ten seeds, and eight downstream tasks, the coupled model achieves significantly higher AUROC than the single-stream baseline in 67 of 85 comparisons and significantly lower AUROC in only one, with a mean improvement of +0.019, including at held-out sites and after cross-database transfer. The coupled model also preserves more information about measured laboratory values: linear probes recover laboratory values substantially better from its representation, with R² gains of +0.042 to +0.108 in distribution, +0.053 and +0.072 at held-out sites, and +0.075 to +0.238 after transfer. Ablations show that separating the encoders, using a set encoder for laboratory values, and adding a laboratory–vital contrastive loss improve laboratory-value recovery, whereas the cross-encoder contrastive loss primarily improves downstream prediction and can reduce recovery on one database. These results suggest that separately encoding laboratory values and care events, while aligning their representations contrastively, can improve both predictive performance and transfer of ICU representations across sites and databases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
electronic health records, self-supervised learning, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_h7GJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether laboratory measurements and clinical care events should be represented separately in self-supervised learning from electronic health records (EHRs). The authors argue that conventional approaches may conflate information about patients' physiological states with information about clinical care processes, potentially limiting generalization across hospitals. To address this issue, they propose a coupled architecture with separate encoders for laboratory values and care events, aligned through contrastive learning objectives. The approach is evaluated on three ICU databases (MIMIC-IV, eICU, and HiRID), using eight downstream tasks and ten random seeds. The authors report statistically significant AUROC improvements over a single-stream baseline in 67 of 85 comparisons, with an average improvement of 0.019. Additional linear probing experiments demonstrate improved recoverability of laboratory values, including under cross-site and cross-database transfer. Ablation studies investigate the contributions of architectural separation and different contrastive objectives. Overall, the paper addresses a practically relevant problem and presents a comprehensive experimental evaluation.

Critical strengths
  1. The distinction between physiological measurements and clinical care processes is important for learning representations from observational healthcare data. Laboratory measurements provide information about patient physiology, whereas laboratory ordering patterns and medication initiation also reflect clinical decision-making, institutional policies, and healthcare workflows. The latter may differ substantially across hospitals and healthcare systems. Investigating whether these information sources should be modeled separately is therefore a relevant research question with potential implications for the robustness and transferability of EHR foundation models.

  2. A major strength is the extensive evaluation across three ICU databases: MIMIC-IV, eICU, and HiRID. The evaluation covers eight downstream prediction tasks, multiple random seeds, and cross-site and cross-database transfer settings. The reported improvements are consistent across most comparisons, suggesting that the proposed approach provides practical benefits over the selected single-stream baseline. The use of ten random seeds further strengthens the reliability of the experimental findings.

  3. The linear probing experiments are a particularly positive aspect of the work. Rather than evaluating representations exclusively through downstream predictive performance, the authors investigate how well laboratory values can be recovered from the learned embeddings. The reported improvements in laboratory-value recovery, including under cross-database transfer, provide complementary evidence about the information retained in the representations.

  4. The ablation experiments distinguish the contributions of encoder separation, laboratory set encoding, and different contrastive learning objectives. The observation that the cross-encoder contrastive objective primarily benefits downstream prediction, while not consistently improving laboratory-value recovery, is particularly interesting. It suggests that different architectural components influence representation properties in distinct ways.

Critical weaknesses
  1. My main concern relates to the methodological novelty and the positioning of the proposed approach relative to existing literature. In particular, the claim that previous EHR Transformer models serialize heterogeneous patient records into a single token sequence is too general and overlooks previously published approaches that explicitly distinguish different types of EHR information. For example, ExMed-BERT (Lentzen et al., IEEE Journal of Biomedical and Health Informatics, 2023, DOI: 10.1109/JBHI.2023.3288768) represents different EHR information types through separate input sequences and corresponding embeddings before combining them within a shared Transformer architecture. Although this differs from the two-encoder design proposed in the present paper, it demonstrates that heterogeneous EHR information has previously been handled through structured, type-specific representations rather than through a single undifferentiated tokenization scheme.

  2. The experimental comparison against a conventional single-stream Transformer is relevant but insufficient to establish the advantages of the proposed design over existing approaches for representing heterogeneous EHR data. In particular, the paper should consider architectures that employ type-specific embeddings followed by a shared Transformer, as well as other multimodal fusion strategies such as separate encoders with intermediate fusion or cross-attention. This distinction is important because the proposed approach changes several aspects of the architecture simultaneously: the separation of input streams, the encoding of laboratory values, the representation capacity, and the contrastive training objectives.

  3. Another concern is the apparent lack of systematic hyperparameter optimization. The proposed architecture introduces several design choices, including encoder configurations, representation dimensionality, the weighting of contrastive objectives, and potentially contrastive temperature parameters. The performance of contrastive representation-learning methods can be sensitive to these choices. It is therefore important to establish whether the proposed architecture and the single-stream baseline were optimized under comparable computational budgets. The extensive evaluation across ten random seeds is commendable, but repeated training with different random initializations does not substitute for hyperparameter optimization.

  4. The central motivation is that physiological measurements and clinical care events capture different aspects of the patient record, with care processes being more dependent on institutional practices. While the experiments demonstrate improved downstream prediction and cross-database transfer, they do not directly establish that these gains arise from reduced dependence on hospital-specific care processes. For example, it has not been tested whether learned representations allow to predict the hospital or site from which data was collected.

  5. The linear probing experiments show improved recoverability of laboratory values from the proposed representations. While this is encouraging, the interpretation requires some caution. Because laboratory values are explicitly processed by a dedicated encoder, improved recovery of these measurements is not entirely unexpected. It remains unclear to what extent the observed improvements reflect a more meaningful representation of patient physiology rather than the preservation of information that is directly supplied to the laboratory encoder.

  6. The ablation results suggest that different components contribute differently to laboratory-value recovery and downstream prediction. In particular, the cross-encoder contrastive objective appears to improve predictive performance without consistently improving laboratory-value recovery. This is an interesting finding that deserves further investigation.

Additional considerations:

  • Although the reported improvements are statistically significant in most comparisons, the mean absolute AUROC gain of 0.019 is relatively modest. The authors should contextualize its practical relevance and provide appropriate confidence intervals and effect-size analyses.

  • Given the 85 reported comparisons, the paper should clearly explain how statistical significance was assessed and whether appropriate corrections for multiple testing were applied. Dependencies between tasks, datasets, and seeds should also be considered.

  • The proposed architecture introduces additional encoders and contrastive objectives. Reporting parameter counts, training time, and computational requirements would help assess the practical trade-off between model complexity and predictive performance.

Questions for authors
  1. What is the precise methodological novelty of the proposed architecture relative to existing modality-specific and multi-view EHR representation-learning approaches?
  2. Can the authors compare their approach against a shared Transformer with modality-specific embeddings, as well as other established multimodal fusion architectures, under comparable model capacities and training budgets?
  3. How were the hyperparameters selected? Were the proposed model and the baseline subjected to comparable hyperparameter optimization procedures?
  4. Can the authors provide more direct evidence that the improved cross-site and cross-database performance results specifically from separating patient physiology and institutional care processes?
  5. To what extent are improvements in laboratory-value recovery explained by the dedicated laboratory encoder rather than by the contrastive learning objectives?
  6. How were statistical significance tests across the 85 comparisons performed, and were multiple comparisons appropriately accounted for?
  7. Will the code, preprocessing procedures, and experimental configurations be made publicly available?
2 comments
AuthorsOpenReview ↗

Response to Reviewer h7GJ Part 1

Thank you for the thorough review. Several of your questions exposed real limitations in our initial framing and led directly to the new experiments in Appendices G–P.

the claim that previous EHR Transformer models serialize ... is too general ... For example, ExMed-BERT

We now cite ExMed-BERT in the Introduction and Related Work and revised both passages: rather than saying prior models typically serialize the record, we state that they process laboratory values and care events within one encoder, whether by serializing events into a single sequence or by combining type-specific embeddings before a shared Transformer trunk.

Q1 (Capacity and confounds). See our response to Reviewer Bsjw (Q3) and Appendix O for the dropout, L2, 192-d readout-width, and probed-layer controls (the last of which shows a real sensitivity on MIMIC-IV now disclosed in Appendix O and Section 7). On capacity, matching the decoupled care-event encoder to the coupled model's exact 693,870 trained parameters (+33,024 parameters, Appendix B) leaves downstream AUROC unchanged (−0.0007 across 85 cells), and replacing the laboratory set encoder with a two-layer Transformer (Appendix M) adds 264,831 effective parameters (+40%, bringing decoupled to 1.40× its set-encoder total) without increasing recovery in either arm.

Q2 (Shared Transformer with modality-specific embeddings, and intermediate fusion). Our single-stream baseline is already a shared Transformer with modality-specific embeddings: separate embedders for laboratory values, vitals, care summary, and care-event sequence, plus a learned token-type embedding, before a shared two-layer Transformer (now explicit in Sections 2–3).

To test intermediate fusion, we implemented bidirectional cross-attention between the laboratory token set and care-event sequence after token embedding and before pooling, replacing the output-level contrastive loss so only the meeting point changes (Appendix N, 60 runs). On the 85 unrestricted downstream cells, cross-attention and the contrastive loss are indistinguishable (+0.0010 within-database, 8 up / 5 down; −0.0003 after transfer, 11 up / 17 down):

ContrastMean ΔAUROC95% CI > 095% CI < 0
Cross-attention − single-stream+0.018268 of 850
Cross-attention − coupled+0.00031922
Cross-attention − decoupled+0.0138623
Coupled − single-stream (reference)+0.0179612

On laboratory-value recovery, however, cross-attention drops to the single-stream baseline (0.6810, 0.7701, 0.7573 R² vs. single-stream 0.6945, 0.7921, 0.7564; Table 18), trailing the coupled model by −0.048, −0.079, and −0.113 R² (0 of 10 seeds above zero on each database). Comparing both axes (Table 19):

ArmCross-stream interactionSeparate representationsΔAUROCΔRecovery R²
DecoupledNoYes+0.004+0.084
Coupled (ours)Yes (output-level)Yes+0.018+0.068
Cross-attentionYes (token-level)No+0.018−0.012

Any cross-stream interaction buys the downstream margin; keeping the representations separate before pooling buys value retention. Because cross-attention has more parameters than all three main arms (818,222 allocated / 793,454 trained vs. 685,614 / 660,846 for decoupled), its lower recovery is not due to capacity.

AuthorsOpenReview ↗

Response to Reviewer h7GJ Part 2

Q3 (Hyperparameter selection). At submission all arms shared an unsearched default (lr = 2e-3). We ran an equal-budget sweep across four learning rates × three arms × three seeds on eICU and HiRID (72 cells: 54 new runs plus 18 at 2e-3), selecting rates on the in-distribution split only. Every arm preferred a lower rate than 2e-3; under tuned rates the coupled margin over single-stream holds (eICU: +0.0044 → +0.0123; HiRID: +0.0408 → +0.0363, 3/3 seeds each), while decoupled on eICU moves from −0.0213 to −0.0059. This sweep ran under the recovery-model convention on one hyperparameter across two databases, so it checks relative ordering rather than re-estimating the 85-cell +0.019 margin.

Q4 (Site invariance vs. institutional care processes). Promoted to Appendix K (Table 13): on eICU, a multinomial logistic classifier predicting which of the 19 held-out hospitals (with ≥ 100 evaluation rows) a row came from reaches macro AUROC 0.91 on single-stream and 0.93 on the coupled laboratory stream (0.87 on the coupled care stream). None of the representations are site-invariant. At the same time, the recovery gap is largest under cross-database transfer (+0.075 to +0.238 R², Table 2), where source and target hospital IDs come from different health systems and source site codes cannot help the probe. Section 7 and Appendix K state clearly that the design yields better transfer, not site invariance, and we do not attribute the gains to removing site-specific care information.

Q5 (Dedicated encoder vs. contrastive objectives). Appendix G (Table 10) adds a linear visible-input reference—a ridge (alpha = 10) on unmasked reconstruction-slot values, presence indicators, and vital-sign means—reaching 0.406, 0.501, and 0.465 R² (eICU, MIMIC-IV, HiRID) against 0.303, 0.475, and 0.467 for coupled (0.214, 0.412, 0.336 for single-stream). On eICU this reference exceeds both representations on 16 of 20 analytes, and each analyte's gap tracks its cross-predictability (Spearman +0.86, +0.81, +0.77; on eICU three red-cell indices carry 39% of the summed gap, though the correlation stays +0.77, p = 0.0003 without them). We frame the gap strictly as retention of input signal (0.88 vs. 0.67 of the linear reference on eICU), not information beyond the input. In Table 3, encoder separation carries the recovery gain while the cross-encoder loss does not add to it, and replacing the set encoder with a Transformer preserves the gap on all three databases (Appendix M).

Q6 (Multiple testing, effect sizes, and practical utility). Appendix J (Table 12) applies Holm and Benjamini–Hochberg across all 85 downstream cells (paired with the stay bootstrap in Appendix L):

ContrastUncorrected (+ / − / n.s.)HolmBenjamini–Hochberg
Coupled − single-stream67 / 1 / 1750 / 1 / 3466 / 1 / 18
Decoupled − single-stream47 / 11 / 2724 / 5 / 5642 / 10 / 33

We agree that the downstream AUROC margin is modest (+0.0112 against STraTS rather than +0.0179 against our baseline, with decoupled tying STraTS at +0.0023), and we no longer present AUROC as the main payoff of separation. Instead, our claim is that preserving a decodable laboratory channel (+0.042 to +0.108 R² in distribution, +0.075 to +0.238 after transfer) comes at no cost in AUROC—and our three controls show that AUROC alone cannot select for this property, since cross-attention (Appendix N) matches the full AUROC margin while retaining no more laboratory signal than single-stream, random routing (Appendix H) matches semantic routing on AUROC while recovering values worse, and STraTS (Appendix P) ties decoupled on AUROC while recovering less than both separated models. For detection alone, the laboratory/care split is not an accuracy lever; among designs that tie on AUROC, the coupled model retains substantially more laboratory signal for linear readout auditing after transfer. We did not evaluate subgroup performance or calibration, we have not shown that this diagnostic improves clinical decisions, and we do not claim site invariance (Section 7, Appendix K).

Q7 (Code, configurations, and compute). The training script, task lists, analysis scripts, and per-seed JSONs are packaged for public release upon publication with an end-to-end synthetic-data runner (Reproducibility Statement); full architectural and training details are in Section 3, Appendix B, and Appendices G–P (with our 90-run scratch replication bound of 0.008 R² vs. the 0.013 R² single-run GPU noise floor in our response to Reviewer BBsx). Table 5 gives allocated and trained parameter counts (within 6% across the three main models), and all arms train for 60 epochs at the same batch size on identical data, matching optimizer steps and tokens seen by construction.

Reviewer_BBsxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares single-stream, decoupled, and contrastively aligned ICU representations on eICU, MIMIC-IV, and HiRID, evaluating both downstream prediction and laboratory-value recovery. Relative to the single-stream model, the proposed model achieved better performance. However, the technical novelty is limited.

Critical strengths

This paper aims to address an important clinical task: learning ICU representations that distinguish physiological measurements from care-process signals and remain predictive across sites and databases.

Critical weaknesses
  1. The method-level originality is limited. Separate encoders plus contrastive alignment are established components, and the paper does not introduce a new representation-learning mechanism.

  2. It would be better to show that the gains hold across several existing Transformer-based or other deep learning models. There is no baselines of other existing work that can take both lab test variable and values as input.

  3. Code and per-seed result files are not available at the review stage.

Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Response to Reviewer BBsx

Thank you for your review and for raising the questions on method originality, existing clinical Transformer baselines, and code/result availability.

The method-level originality is limited. Separate encoders plus contrastive alignment are established components.

We agree that separate encoders and InfoNCE alignment are standard components, and we do not claim a new loss function or layer. Prior two-encoder contrastive designs operate across distinct modalities—such as an image and a text note, or a waveform and a table—where inputs arrive in separate streams with different shapes and tokenizers. In structured EHR tables, laboratory results and care events arrive in a single table with the same row format, interleaved in time, so standard EHR models feed both into one encoder. Our study asks whether splitting within a single structured table along the data-generating process (patient physiology versus clinical ordering behavior) changes what the representation retains.

We can verify directly that this split is non-trivial: a ridge regression from care-event features recovers laboratory values at R² = 0.05, whereas a ridge from the remaining laboratory values in the same draw reaches R² = 0.59 (Section 1, Appendix A). Moreover, our random-routing control (Appendix H) shows that the recovery gain is not a generic effect of having two encoders: against a size-matched random split at a matched 192-dimensional readout, semantic routing improves laboratory-value recovery across all three databases for the decoupled model (+0.064, +0.051, +0.081 R², 10 of 10 seeds; +0.027 and +0.068 on MIMIC-IV and HiRID and −0.008 n.s. on eICU for the coupled model) while giving no advantage on downstream AUROC. Our contribution is therefore an empirical finding about clinical representations: separating physiology from care events preserves decodable laboratory values, cross-stream interaction drives the downstream AUROC gain, and neither component alone delivers both (Appendices H, N, and P).

It would be better to show that the gains hold across several existing Transformer-based or other deep learning models. There is no baselines of other existing work that can take both lab test variable and values as input.

We added three sets of experiments in the revision to address this (Appendices P, M, and I):

  1. Published clinical Transformer taking variable IDs and continuous values directly (STraTS; Appendix P). We re-implemented STraTS (Tipirneni & Reddy, 2022), which embeds each clinical observation as a continuous (variable, value, time) triplet in a Transformer, across all 85 downstream cells and all three recovery databases (90 runs across widths d = 168 and d = 128). STraTS beats our single-stream baseline by +0.0067 AUROC and ties our decoupled arm (+0.0023; 27 cells above zero, 25 below), while trailing the coupled model by −0.0112 (53 of 85 cells below zero, 7 above) and remaining below both two-encoder models on laboratory-value recovery (−0.015, −0.030, and −0.092 R² behind coupled; −0.071, −0.021, and −0.090 behind decoupled).
  2. Transformer laboratory encoder in both two-encoder models (Appendix M). Across 120 runs on all three databases and ten seeds, replacing the laboratory set encoder with a two-layer Transformer preserves both the laboratory-value recovery gap (+0.034 to +0.101 R²) and the coupled model's downstream AUROC margin (+0.0214 across the 85 cells; 76 above zero, 0 below).
  3. Serialized event-stream foundation-model Transformer (Appendix I). We trained an 8-layer Transformer (d = 768) over serialized hour-level stays with inlined laboratory-value decile tokens under autoregressive and masked objectives across all three databases and ten seeds, showing where the two-encoder ordering holds (autoregressive last-token readout, 12 of 15 cells ahead) and where it does not (mean pooling and masked objectives).

Code and per-seed result files are not available at the review stage.

Because the three ICU databases are credentialed under PhysioNet data-use agreements, we have packaged the training script, Slurm task lists, table-generation scripts, and per-seed JSON records for public release upon publication, accompanied by an end-to-end synthetic-data runner (Reproducibility Statement). For the review itself, we expanded the manuscript so that every baseline, control, and ablation is fully specified in the text: Appendix B and Appendices G–P give exact architectures, allocated and trained parameter counts, loss weights, readout widths, and seed-paired 95% confidence intervals for every table. We also document our measured hardware reproducibility bound: because embedding backward passes use non-deterministic CUDA atomics, the single-run noise floor on a fixed seed is 0.013 R² on laboratory-value recovery, and a full 90-run scratch replication of the three recovery arms reproduced the ten-seed means to within 0.008 R².

Reviewer_ehwSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether ICU representation learning improves by separating laboratory values from care events (such as lab orders and medication starts) instead of feeding them into a single Transformer. The authors argue that merging these signals mixes patient physiology with hospital-specific care patterns, which can hurt model transfer across hospitals. In their setup, each input sample is a single lab draw represented by 176 tokens: 46 lab values from the current draw, 1 vital-sign summary token from a two-hour window around the draw, a care summary vector, and up to 128 care events from the past 24 hours. All models are pretrained by reconstructing masked lab values and care events. The authors compare three small architectures: a single-stream model that encodes all 176 tokens with one 2-layer Transformer, a decoupled model that uses a set encoder for lab and vital values and a separate 2-layer Transformer for care events without interaction, and a coupled model that aligns these two separate encoders using an InfoNCE contrastive loss. They evaluate frozen representations on eICU, MIMIC-IV, and HiRID across in-distribution, held-out hospital, and cross-database transfer settings. Using linear probes on eight binary classification tasks (such as mortality, sepsis, and AKI) and ridge regression to recover measured lab values, the authors show that the coupled two-encoder model achieves a slightly higher average AUROC (+0.019) and better lab value recovery (R^2) than the single-stream baseline.

Critical strengths

This paper presents a focused, niche experimental component analysis exploring whether separating laboratory results from care events helps train better deep learning representations for ICU irregular time-series data.

Critical weaknesses

Lack of core contribution and narrow problem scope: As a component analysis, the problem setup is too narrow and does not reflect the full complexity of ICU datasets, which contain continuous vitals, complex treatments, clinical notes, and irregular measurements beyond just simple care event codes and single lab draws. Furthermore, the findings do not generalize well to the broader field. Not all clinical models use a single 1D sequence of tokens to represent irregular ICU time-series. Many existing ICU foundation models represent irregular time-series as a 2D temporal grid, and there are also large structured EHR foundation models and language-based EHR models where this specific two-encoder separation of a single lab draw would not be useful or applicable.

Questions for authors
  • What is the core practical motivation of this work for the broader community? If a new researcher wants to explore ICU datasets or clinical prediction tasks, are the authors suggesting that separating out lab tests into a dedicated set encoder should be their default architectural choice going forward?

  • Why do we fundamentally need this manual architectural separation? Why is forcing such a strict separator in the representation space necessary when a sufficiently expressive backbone or multimodal foundation model should be able to learn the distinction between physiological measurements and clinical actions directly from the data?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ehwS

Thank you for the candid feedback on the scope of our problem setup and the role of architectural inductive biases.

As a component analysis, the problem setup is too narrow ... the findings do not generalize well.

We agree that this study is a component analysis of an architectural choice for encoding a laboratory draw and its surrounding care context, not a general claim about clinical sequence modeling. In Section 7, we narrowed our recommendation to draw-level models and added an explicit limitation stating that our results do not speak to 2D irregular-grid encoders, clinical notes or language-based EHR models, or continuous waveforms beyond our vital-sign summary.

We also added the serialized single-sequence foundation-model experiments (Appendix I) where the submission had wrongly stated we did not test that family. Across three databases and ten seeds using an 8-layer Transformer (d = 768) over hour-level event streams with inlined decile value tokens, the comparison depends on the readout: under an autoregressive objective with a last-token readout, the aligned two-encoder model is ahead of the fused stream in 12 of 15 recovery cells and behind in 2, whereas under mean pooling or a masked objective the comparison reverses. We report both regimes in Appendix I and Section 7 rather than claiming the design generalizes to serialized event-stream models.

Q1 (Should a dedicated laboratory encoder be the default?). No—only when retaining a decodable representation of measured laboratory values matters alongside risk prediction:

  • If the goal is a representation that preserves measured laboratory values for downstream tasks chosen later or after transfer to another hospital, separating the encoders and aligning them is worthwhile. Separation carries the recovery gain (+0.035 to +0.114 R² in distribution; +0.075 to +0.238 after transfer), whereas our new token-level cross-attention control (Appendix N) keeps the entire downstream AUROC margin while losing the entire recovery gap.
  • If the goal is solely patient risk ranking on a fixed task, separation alone leaves within-database AUROC unchanged (−0.001), the predictive margin comes from cross-stream alignment, and a size-matched random partition of the analyte vocabulary performs as well downstream as the semantic split (Appendix H).

On your related question about how much risk signal comes from care patterns versus physiological values, we ran two direct checks on eICU: first, fitting the same linear logistic probe to each half of the frozen coupled representation separately shows that the care-event half underperforms the laboratory/vital half on every mortality and sepsis cell (−0.019 to −0.129 AUROC, all 95% intervals excluding zero), reversing only on prolonged stay (which reflects the elapsed-time shortcut in Appendix E). Second, on eICU in-distribution mortality and sepsis, a linear logistic probe on the 46 presence indicators alone reaches 0.604 AUROC (versus 0.780 for the learned representation), rising to 0.708 when raw magnitudes are added (consistent with Appendix A, where care-event features recover laboratory values at R² = 0.05 versus 0.59 from the other values in the draw). Because both channels carry real signal, our coupled design aligns them rather than discarding the care-event stream.

Q2 (Why force a separator that a sufficiently expressive backbone should learn?). Our random-routing control (Appendix H) partly supports your position. Replacing the semantic laboratory/care split with a size-matched random split of the analyte vocabulary while keeping the two encoders, objectives, and budget identical:

  • On laboratory-value recovery at a matched 192-dimensional readout, the semantic split outperforms the random split by +0.064, +0.051, and +0.081 R² for the decoupled model across eICU, MIMIC-IV, and HiRID (10 of 10 seeds on each database; +0.027 and +0.068 on MIMIC-IV and HiRID and −0.008 n.s. on eICU for the coupled model).
  • On downstream AUROC, the semantic split does not beat the random split: semantic minus random has a negative point estimate in 8 of 16 eICU cells (4 with 95% CI below zero) and 11 of 16 MIMIC-IV cells (7 below zero) for the decoupled model, and in 5 of 16 cells (3 below zero) on each database for the coupled model.

Thus, the semantic boundary is not privileged for downstream accuracy; what it buys is retention of the measured values in the laboratory representation. Whether a larger or differently pretrained single-stream backbone would learn that same retention without an explicit split is an open question that our experiments do not settle, and we now state this in the paper.

I've read the rebuttal and most of my concerns still remains, especially on the practical use cases for the proposed work in the future or near future healthcare system

I maintain my initial rating

Reviewer_BsjwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether laboratory values and care events in ICU records should be encoded by separate encoders. It compares 3 models on eICU, MIMIC-IV and HiRID: a single-stream transformer; a decoupled model (a set encoder for lab values, a transformer for care events), and a coupled model with a cross-encoder loss. They are evaluated with linear probes on 8 tasks: mortality, prolonged stay, and sepsis. The coupled model improves AUROC over single-stream, and the two-encoder models recover lab values with higher linear-probe. Ablations show the recovery gap to separation, the set encoder and a contrastive loss, and the downstream gain to the cross-encoder loss.

Critical strengths
  • The analysis of decoupling lab values and care events is useful.
  • Where test sets are large, downstream gain from the coupled model over single-stream is consistent.
  • The recovery experiment gives a lower-level view than outcome metrics.
Critical weaknesses
  • The paper does not necessarily show that separating the encoders improves prediction, yet the abstract and conclusions claim it. The decoupled model is -0.001 on average over the 39 cells (above zero in 12, below in 9). 0.019 gain comes from the cross-encoder contrastive loss and coupled model differs from decoupled model in parameters that are not controlled for.
  • The recovery gain may not be so useful: decoupled recovers more but does not predict better, and the cross-encoder loss that drives the downstream gain lowers recovery.
  • No single component improves both, so the abstract's claim of "can improve both predictive performance and transfer" may be over-claiming.
  • The standard EHR foundation-model, although the authors acknowledge they do not test it, still should have been used as the baseline instead of the authors' own single-stream transformer, to show that their approach could outperform the common practice in the field.
  • The held-out sites cover only eICU hospitals and MIMIC-IV care units; HiRID has no held-out cells, and the MIMIC-IV care units are within one hospital, so only eICU is a true site shift.
Questions for authors
  1. Can you compare your approach with a flat-sequence baseline (codes/time tokens with lab values in one stream) and at least one published model on the same cells?
  2. In Table 3, separation is measured only on top of a set encoder. What is the effect of separation with a transformer before a set encoder?
  3. How much of the margin is due to dropout, L2 normalization, the 128 vs 192 readout, and the different probed layer?
  4. Please report paired bootstrap intervals over test stays or patients for coupled minus single-stream, at least on the small cells (eICU, held-out sites)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Bsjw Part 1

Thank you for the constructive review and for pointing out the mismatch between the within-database results in Section 4 and the broader claims in the abstract and conclusions.

The paper does not necessarily show that separating the encoders improves prediction, yet the abstract and conclusions claim it.

We agree. Section 4 already reported that the decoupled model averages −0.001 across the 39 within-database cells (12 above zero, 9 below) under a heading attributing the within-database gain to the cross-encoder loss, but our abstract and Figure 2 caption did not carry that qualification. We rewrote the abstract, Figure 2 caption, and Section 7 to state throughout that encoder separation preserves laboratory-value recovery while the cross-encoder loss carries the downstream AUROC gain (with the −0.001 decoupled figure stated directly in the abstract).

...the coupled model differs from the decoupled model in parameters that are not controlled for.

We ran a capacity-matched decoupled control by adding two Linear(128, 128) + GELU blocks inside the care-event encoder's feed-forward path (Appendix B), adding the exact 33,024 trained-parameter difference (matching the coupled model's 693,870 trained parameters) without changing the objective. Across all three databases, ten seeds, and six transfer edges (90 runs) on the 85 unrestricted downstream cells:

ContrastCellsMean ΔAUROC95% CI > 095% CI < 0
Capacity-matched decoupled − decoupled85−0.000718
Capacity-matched decoupled − coupled85−0.0142160
Coupled − single-stream (reference)85+0.0179612

The extra parameters do not move prediction (+0.0001 across the 39 within-database cells, 0 above and 0 below zero), confirming that the coupled model's advantage comes from the cross-encoder loss rather than parameter count. All three main models sit within 6% in trained parameters (Table 5).

The recovery gain may not be so useful ... No single component improves both.

We agree that no single component improves both axes, and we now organize the paper around that distinction. Because several architectures achieve nearly identical downstream AUROC—including cross-attention (Appendix N) and random analyte routing (Appendix H)—accuracy alone cannot distinguish a representation that retains measured laboratory values (+0.042 to +0.108 R² in distribution, +0.075 to +0.238 after transfer) from one that discards them; pairing separate encoders with the cross-encoder loss keeps that channel decodable at no cost in AUROC (Section 7).

The held-out sites cover only eICU hospitals and MIMIC-IV care units.

Added to Section 7 (Limitations): only eICU holds out whole hospitals, MIMIC-IV's held-out sites are care units within one hospital, and HiRID is single-center with no held-out-site cell. eICU is therefore our only cross-hospital shift, with the six cross-database transfer pairs carrying the rest of the external evaluation.

Q1 (Flat-sequence baseline and published model). Appendix I reports the serialized event-stream baseline across all three databases under autoregressive and masked objectives (see item 2 in the top-level comment). Because its unit is the stay rather than the draw and it covers three tasks rather than eight, it bounds generality rather than replacing Table 1.

For a published model taking variable IDs and continuous values directly, we re-implemented STraTS (Tipirneni & Reddy, 2022) on the same 85 downstream cells, ten seeds, and recovery probes (Appendix P, 90 runs). Because our draw-level substrate has no next-window forecasting target, we trained the STraTS architecture under our masked reconstruction objective to isolate the architectural comparison. Trunk width d = 168 gives STraTS 726,156 trained parameters (726,157 allocated) against our single-stream baseline's 733,988 allocated (659,820 trained; Table 5), and we also ran d = 128 (443,756 trained) to bracket both sides:

AuthorsOpenReview ↗

Response to Reviewer Bsjw Part 2

Contrast (85 cells)Mean ΔAUROC95% CI > 095% CI < 0
Coupled − single-stream (reference)+0.0179612
Coupled − STraTS+0.0112537
STraTS − single-stream+0.0067403
STraTS − decoupled+0.00232725

STraTS beats our single-stream baseline by +0.0067 AUROC, reducing the coupled margin by about a third (to +0.0112, now reported alongside +0.0179 in Sections 4, 7, and Appendix P; d = 128 moves all contrasts by at most 0.0004). STraTS ties our decoupled arm on prediction (+0.0023), reinforcing your point that separation alone buys no predictive gain over a stronger single-stream model. On recovery, however, STraTS reaches 0.715, 0.819, and 0.778 R² (eICU, MIMIC-IV, HiRID) vs. 0.729, 0.849, and 0.870 for coupled and 0.785, 0.840, and 0.868 for decoupled, closing roughly a third of the single-stream-to-coupled gap.

Q2 (Set encoder vs. Transformer laboratory encoder). We used a set encoder because analytes within one draw are unordered, and Appendix B already tests a shared set encoder for single-stream (+0.015 to +0.044 R² over the single-stream Transformer, while remaining +0.032 to +0.056 below the two-encoder models). In Appendix M (120 new runs), replacing the laboratory set encoder in both two-encoder arms with a two-layer Transformer preserves both the recovery gap (+0.034, +0.067, +0.099 R² for coupled; +0.067, +0.037, +0.101 for decoupled) and the 85-cell downstream pattern (Transformer coupled minus single-stream +0.0214, 76 up / 0 down; Transformer decoupled +0.0035 overall, −0.0045 within-database, 9 up / 14 down). Because the Transformer stream adds 264,831 effective parameters (+40%, bringing decoupled to 1.40× its set-encoder total) without raising recovery (slightly lower on 30 of 30 decoupled seed pairs), this also bounds laboratory-stream capacity.

Q3 (Dropout, L2, readout width, and probed layer). Matching dropout and L2 moves the coupled recovery margin by at most 0.009 R² with no consistent sign (already used in Tables 2–3), and against a 192-d single-stream baseline the coupled model stays ahead in 24 of 39 within-database cells and behind in 1 (Appendix B). For the probed layer (Appendix O), retraining all 90 recovery models with every decoder attached one layer below the probed representation keeps the coupled margin positive on all three databases (+0.035 [+0.007, +0.064] eICU; +0.016 [−0.005, +0.037] MIMIC-IV; +0.076 [+0.062, +0.089] HiRID), but MIMIC-IV's interval includes zero and the decoupled MIMIC-IV margin reverses (+0.048 to −0.044 [−0.055, −0.034]). Part of that drop reflects L2-normalization asymmetry in the below-representation convention, but the sensitivity is real and is now disclosed in Section 7 and Appendix O.

Q4 (Paired cluster bootstrap over stays). Appendix L runs a paired stay bootstrap (B = 2,000, shared across arms and seeds, no retraining) across the 25 preregistered within-database cells. Averaging across the ten fixed seeds per replicate isolates conditional stay-sampling variance: intervals are 1.3× to 3.4× wider than seed-paired ones on eICU and HiRID (where 5 decoupled cells lose significance) and 0.6× to 1.4× on MIMIC-IV. Coupled minus single-stream moves from 16 above zero and 1 below (seed-paired) to 14 above zero (all 14 passing both intervals) and 0 below; eICU in-distribution death within 24h (−0.011 [−0.016, −0.006]) widens to [−0.028, +0.005]. eICU's ~30 held-out hospitals (19 with ≥ 100 evaluation rows) remain too few for hospital-level resampling.

Thank you for the response and for the additional baselines and controls, which addressed my concerns about them. However, the effectiveness of the recovery gain is still not shown and the downstream gain does not come from separation, so I am keeping my score at 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response to All Reviewers and Summary of Revisions

We thank all four reviewers for their feedback. We have revised the manuscript and added Appendices G–P to address the shared concerns across the reviews:

  1. Framing and attribution (Abstract, Figure 2, Sections 4 and 7). As Reviewer Bsjw noted, Section 4 already reported that separating the encoders without the cross-encoder loss leaves within-database AUROC unchanged (mean −0.001 across the 39 within-database cells; 12 above zero, 9 below) and attributed the downstream gain to the cross-encoder contrastive loss. However, the original abstract and Figure 2 caption overstated this by claiming that separate encoders improve both prediction and recovery. We rewrote the abstract, Figure 2 caption, and Section 7 so the attribution is consistent throughout: separating the encoders preserves laboratory-value recovery, whereas cross-stream alignment drives the downstream AUROC gain. Section 7 also narrows our recommendation to draw-level models.

  2. Serialized single-sequence foundation-model baseline (Appendix I). We replaced the sentence stating that we do not test serialized single-sequence models with our full experimental comparison: an 8-layer Transformer (d = 768) over hour-level event streams with laboratory values inlined as decile tokens, trained under autoregressive (AR) and masked (MLM) objectives across all three databases and ten seeds. Under an autoregressive objective read out at the last token, the aligned two-encoder design is ahead of the fused stream in 12 of 15 recovery cells and behind in 2; under mean pooling and under the masked objective, it is not. We report both regimes and do not claim our design is a better default for serialized event-stream models.

  3. Statistical evaluation (Appendices J and L). Appendix L adds a paired cluster bootstrap over ICU stays (B = 2,000 replicates, three databases, no retraining); 14 of the 16 significant coupled cells survive both seed-paired and stay-bootstrap intervals, with zero cells below zero. Appendix J adds Holm- and Benjamini–Hochberg-adjusted counts across all 85 downstream cells (50 and 66 positive cells vs. 1 negative for the coupled model), and all new recovery tables report seed-paired 95% confidence intervals.

  4. Intermediate fusion and published baselines (Appendices H, M, N, and P). We added a bidirectional cross-attention arm replacing the contrastive loss (Appendix N, 60 runs), a re-implementation of STraTS (Tipirneni & Reddy, 2022) at two trunk widths (Appendix P, 90 runs), a two-layer Transformer laboratory encoder replacing the set encoder (Appendix M, 120 runs), and a size-matched random analyte split (Appendix H). Cross-attention matches the contrastive loss on downstream AUROC (+0.0003 across 85 cells) while losing the laboratory-value recovery gain; STraTS beats our single-stream baseline by +0.0067 AUROC and ties our decoupled arm (+0.0023), reducing the coupled model's predictive margin from +0.0179 to +0.0112 while remaining below both two-encoder models on recovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.