NEST: Nested Event Stream Transformer for Sequences of Multisets

Event stream data often exhibit hierarchical structure in which multiple events co-occur, resulting in a sequence of multisets (i.e., bags of events). In electronic health records (EHRs), for example, medical events naturally form a sequence of clinical encounters with well-defined temporal structure, while the importance of within-encounter ordering varies across settings: it is clinically meaningful in inpatient and ICU encounters, but potentially trivial in long outpatient trajectories. Most existing foundation models (FMs) for event stream data flatten this hierarchy into a one-dimensional sequence, leading to (i) computational inefficiency associated with dense attention and learning spurious within-set relationships, and (ii) lower-quality set-level representations from heuristic post-training pooling for downstream tasks. Here, we show that preserving the original hierarchy through interleaved within-set and cross-set encoders provides a useful inductive bias that improves both computational efficiency and representation quality. We then introduce **N**ested **E**vent **S**tream **T**ransformer (**NEST**), a FM for event streams comprised of sequences of multisets. Building on this architecture, we formulate Masked Set Modeling (MSM), a paradigm that promotes improved set-level representation learning. Experiments on real-world datasets show that NEST captures real-world dynamics while improving both pretraining efficiency and downstream performance.