In-context maps as a collision operator: stochastic jump transformer

We introduce a stochastic jump transformer, an extension of attention in which learned, context-dependent interaction rates govern the occurrence of pairwise jumps, while learned jump maps determine their outcomes. Composing small-time blocks yields McKean–Vlasov jump dynamics, extending the Vlasov continuous-depth limit of deterministic transformers. On compact convex state spaces, rate attention extends transformer universality from continuous deterministic in-context maps to continuous conditional laws, with deterministic maps recovered as Dirac-valued kernels. Because individual interactions are represented explicitly, physical structure can be imposed directly on the jump mechanism. Elastic pair maps, for example, preserve momentum and energy exactly; reversible jumps with symmetric forward–reverse rates yield entropy dissipation. For spatially nonhomogeneous Maxwell–Boltzmann dynamics with a fixed spatial mollifier, we show that their conditional transition laws fit into the stochastic jump transformer framework. More broadly, this suggests a kinetic route to physics foundation models: learn reusable, structure-preserving local interaction laws and compose them across particles and time.