acceptodds
Under review as a conference paper at ICLR 2027

NEST: Nested Event Stream Transformer for Sequences of Multisets

10 views

Abstract

Event stream data often exhibit hierarchical structure in which multiple events co-occur, resulting in a sequence of multisets (i.e., bags of events). In electronic health records (EHRs), for example, medical events naturally form a sequence of clinical encounters with well-defined temporal structure, while the importance of within-encounter ordering varies across settings: it is clinically meaningful in inpatient and ICU encounters, but potentially trivial in long outpatient trajectories. Most existing foundation models (FMs) for event stream data flatten this hierarchy into a one-dimensional sequence, leading to (i) computational inefficiency associated with dense attention and learning spurious within-set relationships, and (ii) lower-quality set-level representations from heuristic post-training pooling for downstream tasks. Here, we show that preserving the original hierarchy through interleaved within-set and cross-set encoders provides a useful inductive bias that improves both computational efficiency and representation quality. We then introduce **N**ested **E**vent **S**tream **T**ransformer (**NEST**), a FM for event streams comprised of sequences of multisets. Building on this architecture, we formulate Masked Set Modeling (MSM), a paradigm that promotes improved set-level representation learning. Experiments on real-world datasets show that NEST captures real-world dynamics while improving both pretraining efficiency and downstream performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
electronic health records, foundation models, clinical prediction models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QsVfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NEST is a hierarchical Transformer for event streams represented as sequences of multisets. It interleaves within-set and cross-set encoders, using set-level [CLS] tokens to exchange context while retaining token-level states across layers. The model combines masked language modeling with Masked Set Modeling, which reconstructs a masked multiset’s token-frequency distribution from its contextualized representation. Evaluation includes computational benchmarks, a controlled comparison of interleaved and sequential architectures, and downstream tasks on MIMIC-IV, a proprietary pediatric EHR dataset, and Instacart. Results show improved forward-pass efficiency over flat Transformer baselines and consistent gains on MIMIC-IV, with more mixed improvements on the other datasets.

Critical strengths

1. Consistent downstream gains supported by architectural ablations. NEST achieves the highest mean AUROC and AP across all five MIMIC-IV tasks (Table 4). The parameter-matched sequential variant also underperforms the interleaved model across all three ablation tasks (Table 6), providing substantive evidence that interleaving contributes beyond simply including both encoder types.

2. A coherent architectural design with measurable computational benefits. The within-set and cross-set encoders exploit the natural grouping of event streams while providing directly accessible set-level representations. Despite having more parameters, NEST achieves higher forward-pass throughput and lower FLOPs per token than BERT with memory-efficient attention (Table 1), making its predictive improvements practically relevant.

Critical weaknesses

1. The methodological advance over existing interleaved hierarchical Transformers is insufficiently established. As acknowledged in Section 3.2, prior HAT architectures already interleave within-segment and cross-segment encoders. The sequential baseline in Table 6 supports the effectiveness of interleaving in this setting, but does not establish the contribution beyond an existing interleaved architecture. The paper should more clearly isolate the value added by its event-stream formulation and MSM objective relative to this closest precedent.

2. Differences in input coverage limit attribution of the main performance gains. Table F5 specifies max_seg=8 and max_seq_len=512 for NEST, versus max_seq_len=2048 for CORE-BEHRT. It is unclear whether NEST’s length limit applies per set and whether the models retain the same underlying events after truncation. Because event coverage can materially affect clinical prediction, the MIMIC-IV gains cannot yet be confidently attributed to the proposed modeling approach without clarifying these settings and, where necessary, comparing models on identical retained event histories.

Questions for authors
  1. Contribution beyond prior architectures: What distinguishes NEST from an interleaved HAT adapted to the same event-stream inputs? Which existing comparisons isolate the additional value of the SeqSet-specific design and MSM beyond the inherited interleaved architecture?

  2. Input coverage clarification: Does NEST’s max_seq_len=512 apply per set or to the entire patient sequence? How do its truncation rules and retained event histories compare with CORE-BEHRT’s 2,048-token input? If coverage differs, do the main MIMIC-IV improvements persist when both models receive identical retained events?

4 comments

We thank the reviewer for the valuable feedback and for raising the concerns that we clarify below.

(W1, Q1) Novelty:

NEST builds on HAT’s interleaved architecture; NEST (MLM only) is close to HAT adapted to event streams. NEST adds two things.

  1. HAT segments pack contiguous text into chunks of up to 128 tokens. SeqSet sets are given by the data: each is an encounter or basket with its own timestamp. NEST can encode continuous time across sets (and within sets where order matters) instead of ordinal positions. We present this as a design matched to the data rather than a separately measured gain.
  2. MSM pretraining provides additional gains over MLM alone. Since each set represents a natural unit of the data, MSM reconstructs its token distribution from [CLS]. Comparing NEST with NEST (MLM only) using the same backbone (Table 4), MSM improves performance across all tasks and metrics. We further show that MSM's distribution-level reconstruction outperforms token-level SMLM (introduced in HAT), which predicts individual masked tokens at their [MASK] positions, under matched backbone, training budget, head capacity, and masking (1st table). Adding MSM to MLM preserves masked-token imputation (within 0.25 points), whereas adding SMLM degrades it by up to 4.2 points (2nd table).

Masked whole-set reconstruction (recall@10 / ndcg@10)

Instacart - set headMIMIC - set head
msm33.83 / 35.0356.44 / 60.02
smlm19.90 / 21.8228.96 / 32.00
mlm+msm35.55 / 36.5655.25 / 58.88
mlm+smlm30.57 / 32.2128.82 / 31.76

Masked token imputation (top-1 acc / top-10 acc)

Instacart - mlm headMIMIC - mlm head
mlm17.71 / 38.4835.93 / 79.82
mlm+msm17.69 / 38.3135.94 / 79.58
mlm+smlm17.63 / 36.4833.91 / 75.67

(W2, Q2) Input Coverage Clarification

NEST uses a hierarchical context of the 8 most recent admissions, with up to 511 events per admission (excluding [CLS]), while CORE-BEHRT uses a flat 2048-token context that can accommodate longer admissions. All models share the same prediction time and withheld codes. Most inputs are not truncated, but when truncation occurs, CORE-BEHRT generally retains more events. Despite this difference in event coverage, NEST consistently outperforms CORE-BEHRT across all downstream tasks (Tab 4), highlighting the effectiveness of NEST's hierarchical modeling rather than an advantage in the amount of observed information.

Task (mean available events per prediction)ModelPredictions with complete inputMean events received (share of available)Predictions with truncated target admissionPredictions with ≥1 earlier admission dropped
Mortality, 24 h (830)NEST79.8%460 (55.4%)0.0%11.4%
CORE-BEHRT89.8%512 (61.7%)0.0%8.4%
Prolonged LoS, 48 h (943)NEST78.3%530 (56.2%)0.1%12.0%
CORE-BEHRT88.4%576 (61.1%)0.0%9.5%
Readmission (888)NEST75.6%515 (58.0%)8.8%11.4%
CORE-BEHRT89.4%577 (65.0%)0.8%8.6%
ICD chapter (895)NEST75.2%513 (57.3%)9.2%11.4%
CORE-BEHRT89.2%578 (64.6%)0.9%8.7%
ICD category (895)NEST75.2%513 (57.3%)9.2%11.4%
CORE-BEHRT89.2%578 (64.6%)0.9%8.7%

The main MIMIC-IV improvements persist when both models receive identical retained events. We restricted fine-tuning and evaluation to the targets for which both NEST and CORE-BEHRT receive every available event, so that the two models see exactly the same events in training, validation and test (75–80% of the test targets, depending on the task). All other settings are as in Table 4 (one seed):

MortalityReadmissionPLoSICD ChapterICD Category
AUROC, NEST91.6974.4779.6392.5490.02
AUROC, CORE-BEHRT89.5673.1676.9890.3588.52
AP, NEST37.5148.2262.5854.6716.45
AP, CORE-BEHRT32.1444.3059.3549.8313.56

On identical inputs, NEST outperforms CORE-BEHRT on all five tasks and both metrics, by 1.31–2.65 AUROC and 2.89–5.37 AP points, margins similar to those in Table 4.

Reviewer_QsVfOpenReview ↗

this response has not addressed my reservations

Based on the authors’ response, they acknowledge that their work is incremental, adding modules to HAT’s interleaved architecture. They also admit that their experimental comparisons are unfair, as they rely on results obtained under different parameter settings. Furthermore, since these experiments were conducted with only one random seed, they do not fully alleviate my concerns. Overall, this response has not addressed my reservations.

We thank the reviewer for the follow-up. We respectfully clarify three points:

  1. Our additions are the event-stream formulation (sets and timestamps given by the data, rather than fixed-length token chunks) and the MSM objective, both isolated by controlled comparisons as requested in Q1: (i) MSM improves over MLM-only on all five tasks and both metrics in Table 4 (+0.48 ~ 0.79 AUROC, +0.29 ~ 0.92 AP, averaged over 5 seeds); (ii) MSM outperforms token-level SMLM by 5–28 points on masked-set recall@10 under matched backbone, training budget, head capacity, and masking; and (iii) adding MSM to MLM preserves masked-token imputation (within 0.25 points), whereas adding SMLM degrades it by up to 4.2 points. Analyses (ii) and (iii) are secondary single-seed analyses reported in this response; their effect sizes exceed the largest seed-level SD observed for our models in Table 4 (1.2 points) by a factor of 3.5–23.

  2. We are unsure which parameter settings the reviewer refers to. Models are compared within the same dataset under the same fine-tuning protocol; configurations differ across experiments, reflecting their different scales, structures, and experimental purposes. Each analyses is a controlled comparison designed to isolate trends, with all models matched within each experiment. If the reviewer refers to the input-length configuration raised in Q2 (8 sets × 511 events vs. a flat 2048 tokens), we note that this difference disfavors NEST: NEST retains a smaller share of available events than CORE-BEHRT (55–58% vs. 61–65%) yet outperforms it on all five tasks, and on the subset where both models receive identical events all gains persist (1.31–2.65 AUROC, 2.89–5.37 AP). The comparison is therefore conservative instead of being unfair. If a different setting is meant, we would be grateful for a specific pointer.

  3. All main experiments in the paper, including the Table 4's are averaged over 5 seeds. Only the secondary analyses added in this response are single-seed. The identical-input comparison is a subgroup analysis on top of these multi-seed main results, with margins exceeding the corresponding SDs by a factor of 3.7–30 and all ten task–metric margins are positive. We will report mean ± SD over seeds for all single-seed analyses in this response in the revision.

Here are the multi-seed results the reviewer asked for. We ran additional random seeds for the identical-input experiment, in which NEST and CORE-BEHRT see exactly the same events. The results over all three seeds (mean ± SD) are:

MortalityReadmissionPLoSICD ChapterICD Category
AUROC, NEST91.74 ± 0.1274.49 ± 0.3179.65 ± 0.0792.60 ± 0.3890.06 ± 0.18
AUROC, CORE-BEHRT89.43 ± 0.2772.96 ± 0.5976.99 ± 0.0590.23 ± 0.3388.54 ± 0.05
AP, NEST37.21 ± 0.8048.28 ± 0.4862.54 ± 0.2554.52 ± 0.6716.39 ± 0.10
AP, CORE-BEHRT31.96 ± 0.6244.15 ± 0.4259.30 ± 0.2249.03 ± 1.3413.43 ± 0.22

NEST outperforms CORE-BEHRT on every task and metric, by margins far larger than the standard deviations across seeds.

These results also show that our main comparison does not favor NEST; on the contrary, the difference in input favors CORE-BEHRT. In Table 4, NEST receives fewer events than CORE-BEHRT, and when both models see the same events, NEST's lead is generally larger than in Table 4.

Reviewer_QfFjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NEST models event streams as time-ordered sequences of multisets, such as clinical encounters or shopping baskets. Each layer combines a set-wise encoder, which processes events within each set, with a cross-set encoder operating on per-set [CLS] representations. Token states bypass the cross-set encoder and remain available to subsequent layers, allowing within-set processing to incorporate progressively broader context. The model supports both unordered sets and timestamped within-set events. The paper also introduces Masked Set Modeling (MSM), which can complement token-level MLM by reconstructing a fully masked set’s empirical token distribution from its [CLS] representation. Evaluation includes computational benchmarks, a synthetic study of interleaving, Instacart recommendation, five MIMIC-IV prediction tasks, three proprietary pediatric EHR tasks, and architectural and pretraining ablations.

Critical strengths
  1. The MIMIC-IV evaluation uses patient-level downstream splits shared across models, task-specific prediction times, laboratory report timestamps, and appropriate withholding of target-admission diagnosis and procedure codes. NEST outperforms all six external baselines on all five tasks and both metrics, improving AUROC by approximately 0.7-2.6 percentage points over the strongest external baseline for each task. Mortality AP increases from CORE-BEHRT’s 32.60 to 37.80. The consistency across tasks makes this the strongest evidence supporting acceptance.

  2. The ablations support the value of the combined design beyond model capacity alone. Variants retrained from scratch with matched encoder parameter counts underperform NEST across all three ablation tasks and both metrics, whether they use only one encoder type or stack the two encoders sequentially. Adding MSM also improves the reported mean performance on every MIMIC-IV task and metric relative to MLM-only NEST. These results support the practical value of the complete system, although they do not fully isolate the contribution of each design choice.

Critical weaknesses
  1. Important controls behind the headline MIMIC-IV comparison remain unclear. NEST accepts up to eight encounters with 512 tokens each, whereas CORE-BEHRT uses a flat 2,048-token window; other baselines impose different encounter and event limits. Without explicit truncation rules and retained-event statistics, it is difficult to determine whether models receive comparable clinical information. Optimization settings also differ, and equal training epochs do not establish equal computational budgets. More consequentially, Table F7 specifies task- and seed-specific downstream splits alongside a separate fixed pretraining split, without establishing whether downstream test patients are excluded from pretraining. If their full trajectories are included, the evaluation would measure a different form of generalization from transfer to unseen patients. This matters particularly for the ICD tasks, whose labels originate from diagnosis codes that can appear as pretraining tokens. I am not claiming that leakage occurred; the manuscript does not provide enough information to resolve the issue. These omissions limit how confidently I can attribute the reported margins to the proposed method.

  2. The evidence does not sufficiently isolate the contribution beyond existing hierarchical Transformers and masked reconstruction. Interleaved within-segment and cross-segment encoders communicating through [CLS] tokens already appear in HAT, Chalkidis et al. (2022), as the manuscript acknowledges. This makes the evidence for NEST’s specific adaptations and MSM particularly important. Two comparisons leave that attribution unresolved. First, the sequential ablation discards token states before cross-set processing, but the manuscript does not explain where its MLM predictions are computed or whether they can access cross-set context. Its disadvantage therefore cannot yet be attributed solely to layer order. Second, MSM introduces whole-set masking, [CLS]-based supervision, and a distribution-level target together, without separating their effects. The additional evidence is mixed: in the smaller-model Instacart ablation, adding MSM reduces the MLM-head Recall@10 from 29.45 to 26.62, while gains over CORE-BEHRT on the proprietary EHR are small and not uniform. The results support a useful combination more clearly than the broader claims about interleaving and improved set representations.

Questions for authors
  1. Pretraining/test separation: clarify whether there is data overlap between pretraining and downstream; if needed, a controlled evaluation. Are all downstream test patients excluded from pretraining for every MIMIC-IV task and seed? Please specify any overlap at both patient and admission levels, including whether target-admission diagnosis codes enter pretraining. If overlap exists, does the advantage persist when pretraining and downstream test patients are disjoint?

  2. Input coverage and optimization: clarification and targeted controls. What truncation rules does each model use, and how many encounters and events of each type are retained? Does NEST’s advantage over CORE-BEHRT persist when both receive the same retained clinical events? Please also report pretraining updates and compute, and clarify whether downstream hyperparameters-including the different weight decay-were selected under comparable validation procedures.

  3. Sequential ablation: clarification and targeted control. After token states are discarded at the SWE-CSE boundary, where is MLM computed, which components receive its gradients, and can token predictions use cross-set context? Would the gap remain if the final cross-set representation were broadcast back to token states before reconstruction? This would help distinguish repeated interleaving from access to cross-set context during pretraining.

  4. MSM: targeted control and reporting clarification. With the backbone, whole-set masking schedule, training budget, and prediction-head capacity controlled, how does distribution reconstruction compare with token reconstruction? Evaluating these variants and MLM-only NEST using the same frozen-backbone [CLS] probe would directly test representation quality. Please also identify the pretraining objective, checkpoint, and readout used for the NEST zero-shot result in Tab 3.

  5. Dependence on longitudinal history: clarification and subgroup analysis. How do NEST’s gains and the CSE ablation gap vary for targets with zero, one, or at least two prior admissions, and what fraction of test targets falls into each group? How is MSM handled when masking leaves no visible encounter? Since the median MIMIC-IV patient has one admission, this would help establish how much of the improvement comes from longitudinal contextualization.

6 comments

We thank the reviewer for the valuable feedback and for raising the concerns that we clarify below.

(W1, Q1) Pretrain/Test split

Thank you for raising this. In all reported MIMIC-IV experiments, pretraining and every downstream task share one fixed patient-level 80/10/10 training/validation/test split, made before any training; the seed never changes the split. Every model (NEST, all ablations and all six baselines) was pretrained on the training patients only, and both pretraining and fine-tuning selected checkpoints on the validation patients, which were never used as training data. Therefore:

  • Patient level: no test patient enters any model’s pretraining (0 overlapping patients across all tasks and seeds).
  • Admission level: consequently, no test admission enters pretraining, including the target admissions and their diagnosis codes, from which the ICD labels are derived.

All reported results are thus measured on patients unseen during pretraining. Table F7, carried over from an old version, described a separate fine-tuning split for each task and seed and a separate pretraining split; all reported runs used the single split above, and we have revised Table F7 and Appendix E.2.1 to state it explicitly and added the same statement to the MIMIC-IV description in Section 4.

(W1, Q2) Retain tokens

NEST uses a hierarchical context of the 8 most recent admissions, with up to 511 events per admission (excluding [CLS]), while CORE-BEHRT uses a flat 2048-token context that can accommodate longer admissions. All models share the same prediction time and withheld codes. Most inputs are not truncated, but when truncation occurs, CORE-BEHRT generally retains more events. Despite this difference in event coverage, NEST consistently outperforms CORE-BEHRT across all downstream tasks (Tab 4), highlighting the effectiveness of NEST's hierarchical modeling rather than an advantage in the amount of observed information.

The baselines' input limits are:

ModelInput limit
CORE-BEHRT, MOTORone flat sequence, most recent 2,048 positions (with encounter [CLS] boundaries)
CEHR-BERTmost recent 2,048 positions, whole admissions only
Hi-BEHRTmost recent 2,048 positions (where the hierarchy is sliding window based)
GT-BEHRT50 most recent admissions, at most 100 distinct codes each
HEART20 most recent admissions, at most 25 codes per type each

The target admission is handled identically for all models (same prediction time, same withheld codes).

Task (mean available events per prediction)ModelPredictions with complete inputMean events received (share of available)Predictions with truncated target admissionPredictions with ≥1 earlier admission dropped
Mortality, 24 h (830)NEST79.8%460 (55.4%)0.0%11.4%
CORE-BEHRT89.8%512 (61.7%)0.0%8.4%
Prolonged LoS, 48 h (943)NEST78.3%530 (56.2%)0.1%12.0%
CORE-BEHRT88.4%576 (61.1%)0.0%9.5%
Readmission (888)NEST75.6%515 (58.0%)8.8%11.4%
CORE-BEHRT89.4%577 (65.0%)0.8%8.6%
ICD chapter (895)NEST75.2%513 (57.3%)9.2%11.4%
CORE-BEHRT89.2%578 (64.6%)0.9%8.7%
ICD category (895)NEST75.2%513 (57.3%)9.2%11.4%
CORE-BEHRT89.2%578 (64.6%)0.9%8.7%

Most predictions are not truncated at all (median available events 167–274 across tasks); the means reflect a long-history tail (90th percentile ≈ 2,100–2,400). Per prediction, 4.2–4.3 admissions are available on average (median 2); NEST's input covers 3.0–3.1 of them and CORE-BEHRT's 3.3–3.4. By event type, the difference lies in laboratory results and medications, which make up about 95% of the available events: NEST receives 57–59% of the available laboratory results and 47–51% of the medications, against 63–66% and 54–60% for CORE-BEHRT. Of diagnosis codes, the two receive about the same share (65–73% vs 63–72%), and NEST receives fewer procedure codes (59–68% vs 63–72%). Despite receiving fewer events, NEST outperforms CORE-BEHRT on all five tasks and both metrics (Table 4). Since the coverage difference favors CORE-BEHRT, it cannot explain NEST's advantage, which comes from how the data is modeled rather than how much is seen.

Identical inputs. To test this directly, we restricted fine-tuning and evaluation to the targets for which both NEST and CORE-BEHRT receive every available event, so that the two models see exactly the same events in training, validation and test (75–80% of the test targets, depending on the task). All other settings are as in Table 4 (one seed):

MortalityReadmissionPLoSICD ChapterICD Category
AUROC, NEST91.6974.4779.6392.5490.02
AUROC, CORE-BEHRT89.5673.1676.9890.3588.52
AP, NEST37.5148.2262.5854.6716.45
AP, CORE-BEHRT32.1444.3059.3549.8313.56

On identical inputs, NEST outperforms CORE-BEHRT on all five tasks and both metrics, by 1.31–2.65 AUROC and 2.89–5.37 AP points, margins similar to those in Table 4.

Optimization. In pretraining, all models have the same budget of 30 epochs (GT-BEHRT adds 10 NAM epochs), and NEST uses the largest effective batch (72, against 16–64 for the baselines, while learning rates are the same across all methods; Table F6). It therefore takes the fewest optimizer updates per epoch of all models, about 4.5× fewer than CORE-BEHRT (batch 16). Every run uses a single NVIDIA H200 GPU, and pretraining wall-clock times are of the same order across models.

Downstream weight decay sensitivity analysis. We change the weight decay to 0.01 for downstream tasks with NEST, and the results are similar to those with weight decay 0.001.

MortalityReadmissionPLoSICD ChapterICD Category
NEST, wd 0.01AUROC92.0774.8879.9193.0390.15
AP38.1648.7262.8955.2616.81
NEST, wd 0.001 (mean reported in paper)AUROC92.0574.7579.8892.9290.20
AP37.8048.6162.8154.9516.86

(W2, Q3) Sequential Ablation

In the sequential variant, MLM is computed on the token states at the output of the SWE stack, before the SWE–CSE boundary. Its gradients reach the embeddings, the time encodings and the SWE blocks, but no CSE block; the CSE blocks are trained only by MSM through the final [CLS]. Token predictions therefore cannot use any information from other sets. In NEST, by contrast, each CSE block writes back into the [CLS] slot and the next SWE block lets the tokens attend to it, so token predictions use cross-set context.

Following the reviewer's suggestion, we added a control that broadcasts the final cross-set representation back to the token states before masked token reconstruction: every token state receives its set's cross-set update, i.e., what the CSE stack added to the set's [CLS]. Token predictions can now use cross-set context, and MLM gradients reach every CSE block. The variant adds no parameters and is pretrained with MLM and MSM like Sequential. Fine-tuning reads the final [CLS] exactly as in Sequential, so the two variants differ only in this broadcast during pretraining. We follow the protocol of Table 6 (mean ± SD over 3 seeds; NEST over 5, as in Table 6):

ModelMetricMortalityReadmissionPLoS
SequentialAUROC90.92±0.2273.00±0.3277.59±0.14
AP34.08±1.3846.68±0.4561.06±0.13
Sequential + broadcastAUROC91.02±0.1973.45±0.2778.08±0.23
AP34.30±1.2747.02±0.3661.31±0.18
Interleaved (NEST)AUROC92.05±0.1674.75±0.3579.88±0.08
AP37.80±1.0948.61±0.5662.81±0.22

A schematic view of the MLM forward pass of these variants:

Sequential: SWE$^6$ → zijz_{ij} → MLM (the CSE stack sees only the [CLS] tokens)

Sequential + broadcast: SWE$^6$ → CSE$^6$ → z~ij=zij+(ci(L)−ci(0))\tilde z_{ij} = z_{ij} + (c_i^{(L)} - c_i^{(0)}) → MLM, where ci(L)−ci(0)c_i^{(L)} - c_i^{(0)} is what the CSE stack added to set ii's [CLS] (no new parameters)

Interleaved (NEST): SWE → CSE → … → SWE → CSE → zijz_{ij} → MLM

The gap remains. The broadcast variant is below NEST on every task and metric (by 1.03–1.80 AUROC and 1.50–3.50 AP points), and although it is pretrained with MSM, it is also below NEST (MLM only) on every task and metric. Access to cross-set context during pretraining therefore does not account for NEST's advantage over the sequential design; the advantage comes from repeated interleaving, which gives tokens cross-set context at every layer rather than once at the end. We will add this variant to Table 6.

Reviewer_ekaQOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NEST, a Transformer-based model for event streams organized as sequences of multisets, such as clinical encounters and shopping baskets. NEST interleaves within-set attention with cross-set attention over set-level [CLS] tokens, retaining token representations between layers, and combines masked language modeling with masked set modeling to learn both token and set representations. Experiments include synthetic simulations, computational benchmarks, five MIMIC-IV prediction tasks, proprietary pediatric EHR tasks, and Instacart recommendation; the strongest results are consistent improvements on MIMIC-IV and gains over parameter-matched architectural ablations.

Critical strengths
  1. The architecture provides a clear mechanism for making within-set processing depend on longitudinal context while retaining individual token states.
  2. The computational benchmark demonstrates practical benefits from structured attention, with throughput increasing from 65.25K tokens/s for BERT with memory-efficient attention to 90.46K for NEST (Table 1).
Critical weaknesses
  1. The MIMIC-IV comparisons do not clearly match the information available to each model: Table F5 permits eight 512-token segments for NEST but specifies a 2,048-token limit for CORE-BEHRT.
  2. Table F8 describes a fixed pretraining split and separate downstream splits for each seed, without explicitly establishing that downstream test patients are excluded from pretraining. The authors should clarify patient overlap and whether any test-patient target-admission records enter pretraining.
Questions for authors
  1. Can you clarify whether downstream validation and test patients are excluded from pretraining for every MIMIC-IV seed? If overlap exists, does pretraining include their target admissions or subsequent admissions?
  2. What are the actual retained-token counts and truncation rates for NEST and CORE-BEHRT on MIMIC-IV? Do any existing results compare these models using identical retained clinical events?
  3. How do you interpret the Instacart degradation from adding MSM in Table B1, and is the same effect present in the full-size configuration used in Table 3?
3 comments

We thank the reviewer for the valuable feedback and for raising the concerns that we clarify below.

(W1, Q2) Event retain

NEST uses a hierarchical context of the 8 most recent admissions, with up to 511 events per admission (excluding [CLS]), while CORE-BEHRT uses a flat 2048-token context that can accommodate longer admissions. NEST requires padding tokens for each admission, whereas CORE-BEHRT does not, so comparing respective token limits is misleading. However, we have ensured that NEST never receives events not available to CORE-BEHRT in our comparisons. We agree this is important and will make sure it’s stated clearly in the revised version

All models share the same prediction time and withheld codes. Most inputs are not truncated, but when truncation occurs, CORE-BEHRT generally retains more events. Despite this difference in event coverage, NEST consistently outperforms CORE-BEHRT across all downstream tasks (Tab 4), highlighting the effectiveness of NEST's hierarchical modeling rather than an advantage in the amount of observed information.

Task (Avg. Available Events)ModelComplete Input (%)Events Retained (Avg. %)Target Admission Truncated (%)Earlier Admission Dropped (%)
Mortality, 24 h (830)NEST79.8%460 (55.4%)0.0%11.4%
CORE-BEHRT89.8%512 (61.7%)0.0%8.4%
PLoS, 48 h (943)NEST78.3%530 (56.2%)0.1%12.0%
CORE-BEHRT88.4%576 (61.1%)0.0%9.5%
Readmit (888)NEST75.6%515 (58.0%)8.8%11.4%
CORE-BEHRT89.4%577 (65.0%)0.8%8.6%
ICD chapter (895)NEST75.2%513 (57.3%)9.2%11.4%
CORE-BEHRT89.2%578 (64.6%)0.9%8.7%
ICD category (895)NEST75.2%513 (57.3%)9.2%11.4%
CORE-BEHRT89.2%578 (64.6%)0.9%8.7%

Follow the reviewer's suggestion, we also conduct a subgroup analysis, where we restrict finetuning and evaluation to the predictions for which both NEST and CORE-BEHRT receive all available events, so that the two models see exactly the same input for each downstream task's training, validation and test. All other settings are as in Tab 4 (one seed):

MortalityReadmissionPLoSICD ChapterICD Category
AUROC, NEST91.6974.4779.6392.5490.02
AUROC, CORE-BEHRT89.5673.1676.9890.3588.52
AP, NEST37.5148.2262.5854.6716.45
AP, CORE-BEHRT32.1444.3059.3549.8313.56

On identical inputs, NEST outperforms CORE-BEHRT on all five tasks with margins similar to those in Tab 4.

(W2, Q1) Split Clarification

In all reported MIMIC-IV experiments, pretraining and every downstream task share one fixed patient-level 80/10/10 training/validation/test split, made before any training; each downstream task's training, validation and test sets are set up using the fixed splits from pretraining.

Every model, including NEST, all ablations and all 6 baselines, was pretrained on the training patients only, with checkpoints selected on the validation patients, and every task and seed was evaluated on the same held-out test set; the seed never changes the split.

Therefore, at patient level, no test patients enter pretraining; validation patients are used only for checkpoint selection (zero overlap across tasks and seeds). At admission level, the patient-level splitting ensures that no test admissions enter pretraining, including their diagnosis codes and context tokens used to derive ICD labels.

All reported results are measured on patients unseen during pretraining. Tab F7 described a finetuning split for each task and seed and a separate pretraining split; this was carried over from an old version. In all reported runs, pretraining and fine-tuning used this single 80/10/10 split, and both selected checkpoints on its validation patients. We have revised Tab F7 and Appx E.2.1 to state the shared split explicitly and added the same statement to the MIMIC-IV description in Sec 4. The paper already states the shared split for the proprietary EHR (Appx E.3) and Instacart (Appx F.1).

(Q3) Instacart degradation from adding MSM

We thank the reviewer for this question. We want to clarify the model config for this ablation experiment: for the EHR experiment, we used small-scale training (3 layers, 10 epochs); for the Instacart experiment, we used full-size training (6 layers, 30 epochs). We hypothesize that the degradation from adding MSM on Instacart reflects differences in the predictability of set-level structure between shopping baskets and clinical event streams. Clinical encounters often follow structured care pathways, with recurring relationships among diagnoses, procedures, medications, and laboratory tests across visits. In contrast, shopping baskets may be influenced by more heterogeneous purchasing preferences and short-term needs, resulting in less predictable relationships across baskets. To investigate this hypothesis, we compare the pretraining losses across datasets, normalized by their respective uniform-prediction reference values (6 layers x 30 epochs):

DatasetVn (max set size)MLM / MLM₀MSM / MSM₀
Instacart45.0k326.17 / 10.71 (57.6%)1.43 / 8.17 (17.5%)
MIMIC Hosp20.3k5122.09 / 9.92 (21.1%)0.49 / 4.66 (10.5%)
Proprietary EHR30.0k642.52 / 10.31 (24.4%)0.31 / 7.10 (4.4%)

Here, MLM₀ = log⁡V\log V is the uniform token baseline, and MSM₀ = log⁡(Vn)/n\log\binom{V}{n}/n is the uniform set baseline for n distinct tokens from vocabulary V. Despite having the smallest maximum set size (n = 32), Instacart exhibits the highest normalized MSM loss, compared with the proprietary EHR and MIMIC Hosp, whose maximum set sizes are larger. The same holds for MLM, where Instacart's normalized loss is more than twice that of either clinical dataset. These observations are consistent with the hypothesis that shopping baskets contain less predictable token- and set-level patterns than clinical encounters. The additional set-level reconstruction objective may provide less useful supervision for Instacart. For the clinical datasets, we found the opposite: MSM improves performance.

Thank you for the detailed responses and additional experiments. The split clarification and identical-input comparison address my main concerns. The explanation for the Instacart degradation is plausible but remains tentative. Overall, the rebuttal strengthens my confidence in my original assessment, but does not provide sufficient grounds for a stronger recommendation. I therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.