Meta-Causal Predictive States: A World Model for High-Dimensional Event Sequences

While current temporal foundation models transfer across domains through a shared numerical observation space, high-cardinality discrete event streams, such as those in cloud systems or clinical care, describe system dynamics through symbols whose identities are local to each domain. This limits direct vocabulary sharing even when different systems exhibit similar predictive structures. To address this constraint, we propose a world model for high-dimensional discrete event sequences organized around meta-causal predictive states, which group equivalence classes of histories whose conditional future distributions coincide up to event relabeling. Our main hypothesis is that transferable event dynamics can be expressed through a shared predictive vocabulary, while context binds this vocabulary to local event identities. We realize this through a two-stage framework: (i) DES-JEPA, which learns discrete event representations from causal occurrence statistics using predictive teacher-student training, and (ii) DES-PFN, which learns in-context dynamics over the frozen representations and decodes event types, enabling autoregressive forecasting without target-task parameter updates. Pretrained on a mixture of six synthetic process families and evaluated on six diverse real-world domains, our analysis demonstrates that DES-PFN is competitive on low-cardinality datasets and, to our knowledge, the first method to successfully scale zero-shot forecasting to hundreds of event types.