- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
NEST is a hierarchical Transformer for event streams represented as sequences of multisets. It interleaves within-set and cross-set encoders, using set-level [CLS] tokens to exchange context while retaining token-level states across layers. The model combines masked language modeling with Masked Set Modeling, which reconstructs a masked multiset’s token-frequency distribution from its contextualized representation. Evaluation includes computational benchmarks, a controlled comparison of interleaved and sequential architectures, and downstream tasks on MIMIC-IV, a proprietary pediatric EHR dataset, and Instacart. Results show improved forward-pass efficiency over flat Transformer baselines and consistent gains on MIMIC-IV, with more mixed improvements on the other datasets.
Critical strengths
1. Consistent downstream gains supported by architectural ablations. NEST achieves the highest mean AUROC and AP across all five MIMIC-IV tasks (Table 4). The parameter-matched sequential variant also underperforms the interleaved model across all three ablation tasks (Table 6), providing substantive evidence that interleaving contributes beyond simply including both encoder types.
2. A coherent architectural design with measurable computational benefits. The within-set and cross-set encoders exploit the natural grouping of event streams while providing directly accessible set-level representations. Despite having more parameters, NEST achieves higher forward-pass throughput and lower FLOPs per token than BERT with memory-efficient attention (Table 1), making its predictive improvements practically relevant.
Critical weaknesses
1. The methodological advance over existing interleaved hierarchical Transformers is insufficiently established. As acknowledged in Section 3.2, prior HAT architectures already interleave within-segment and cross-segment encoders. The sequential baseline in Table 6 supports the effectiveness of interleaving in this setting, but does not establish the contribution beyond an existing interleaved architecture. The paper should more clearly isolate the value added by its event-stream formulation and MSM objective relative to this closest precedent.
2. Differences in input coverage limit attribution of the main performance gains. Table F5 specifies max_seg=8 and max_seq_len=512 for NEST, versus max_seq_len=2048 for CORE-BEHRT. It is unclear whether NEST’s length limit applies per set and whether the models retain the same underlying events after truncation. Because event coverage can materially affect clinical prediction, the MIMIC-IV gains cannot yet be confidently attributed to the proposed modeling approach without clarifying these settings and, where necessary, comparing models on identical retained event histories.
Questions for authors
-
Contribution beyond prior architectures: What distinguishes NEST from an interleaved HAT adapted to the same event-stream inputs? Which existing comparisons isolate the additional value of the SeqSet-specific design and MSM beyond the inherited interleaved architecture?
-
Input coverage clarification: Does NEST’s max_seq_len=512 apply per set or to the entire patient sequence? How do its truncation rules and retained event histories compare with CORE-BEHRT’s 2,048-token input? If coverage differs, do the main MIMIC-IV improvements persist when both models receive identical retained events?
4 comments
this response has not addressed my reservations
Based on the authors’ response, they acknowledge that their work is incremental, adding modules to HAT’s interleaved architecture. They also admit that their experimental comparisons are unfair, as they rely on results obtained under different parameter settings. Furthermore, since these experiments were conducted with only one random seed, they do not fully alleviate my concerns. Overall, this response has not addressed my reservations.
We thank the reviewer for the follow-up. We respectfully clarify three points:
-
Our additions are the event-stream formulation (sets and timestamps given by the data, rather than fixed-length token chunks) and the MSM objective, both isolated by controlled comparisons as requested in Q1: (i) MSM improves over MLM-only on all five tasks and both metrics in Table 4 (+0.48 ~ 0.79 AUROC, +0.29 ~ 0.92 AP, averaged over 5 seeds); (ii) MSM outperforms token-level SMLM by 5–28 points on masked-set recall@10 under matched backbone, training budget, head capacity, and masking; and (iii) adding MSM to MLM preserves masked-token imputation (within 0.25 points), whereas adding SMLM degrades it by up to 4.2 points. Analyses (ii) and (iii) are secondary single-seed analyses reported in this response; their effect sizes exceed the largest seed-level SD observed for our models in Table 4 (1.2 points) by a factor of 3.5–23.
-
We are unsure which parameter settings the reviewer refers to. Models are compared within the same dataset under the same fine-tuning protocol; configurations differ across experiments, reflecting their different scales, structures, and experimental purposes. Each analyses is a controlled comparison designed to isolate trends, with all models matched within each experiment. If the reviewer refers to the input-length configuration raised in Q2 (8 sets × 511 events vs. a flat 2048 tokens), we note that this difference disfavors NEST: NEST retains a smaller share of available events than CORE-BEHRT (55–58% vs. 61–65%) yet outperforms it on all five tasks, and on the subset where both models receive identical events all gains persist (1.31–2.65 AUROC, 2.89–5.37 AP). The comparison is therefore conservative instead of being unfair. If a different setting is meant, we would be grateful for a specific pointer.
-
All main experiments in the paper, including the Table 4's are averaged over 5 seeds. Only the secondary analyses added in this response are single-seed. The identical-input comparison is a subgroup analysis on top of these multi-seed main results, with margins exceeding the corresponding SDs by a factor of 3.7–30 and all ten task–metric margins are positive. We will report mean ± SD over seeds for all single-seed analyses in this response in the revision.
Here are the multi-seed results the reviewer asked for. We ran additional random seeds for the identical-input experiment, in which NEST and CORE-BEHRT see exactly the same events. The results over all three seeds (mean ± SD) are:
| Mortality | Readmission | PLoS | ICD Chapter | ICD Category | |
|---|---|---|---|---|---|
| AUROC, NEST | 91.74 ± 0.12 | 74.49 ± 0.31 | 79.65 ± 0.07 | 92.60 ± 0.38 | 90.06 ± 0.18 |
| AUROC, CORE-BEHRT | 89.43 ± 0.27 | 72.96 ± 0.59 | 76.99 ± 0.05 | 90.23 ± 0.33 | 88.54 ± 0.05 |
| AP, NEST | 37.21 ± 0.80 | 48.28 ± 0.48 | 62.54 ± 0.25 | 54.52 ± 0.67 | 16.39 ± 0.10 |
| AP, CORE-BEHRT | 31.96 ± 0.62 | 44.15 ± 0.42 | 59.30 ± 0.22 | 49.03 ± 1.34 | 13.43 ± 0.22 |
NEST outperforms CORE-BEHRT on every task and metric, by margins far larger than the standard deviations across seeds.
These results also show that our main comparison does not favor NEST; on the contrary, the difference in input favors CORE-BEHRT. In Table 4, NEST receives fewer events than CORE-BEHRT, and when both models see the same events, NEST's lead is generally larger than in Table 4.
We thank the reviewer for the valuable feedback and for raising the concerns that we clarify below.
(W1, Q1) Novelty:
NEST builds on HAT’s interleaved architecture; NEST (MLM only) is close to HAT adapted to event streams. NEST adds two things.
Masked whole-set reconstruction (recall@10 / ndcg@10)
Masked token imputation (top-1 acc / top-10 acc)
(W2, Q2) Input Coverage Clarification
NEST uses a hierarchical context of the 8 most recent admissions, with up to 511 events per admission (excluding [CLS]), while CORE-BEHRT uses a flat 2048-token context that can accommodate longer admissions. All models share the same prediction time and withheld codes. Most inputs are not truncated, but when truncation occurs, CORE-BEHRT generally retains more events. Despite this difference in event coverage, NEST consistently outperforms CORE-BEHRT across all downstream tasks (Tab 4), highlighting the effectiveness of NEST's hierarchical modeling rather than an advantage in the amount of observed information.
The main MIMIC-IV improvements persist when both models receive identical retained events. We restricted fine-tuning and evaluation to the targets for which both NEST and CORE-BEHRT receive every available event, so that the two models see exactly the same events in training, validation and test (75–80% of the test targets, depending on the task). All other settings are as in Table 4 (one seed):
On identical inputs, NEST outperforms CORE-BEHRT on all five tasks and both metrics, by 1.31–2.65 AUROC and 2.89–5.37 AP points, margins similar to those in Table 4.