acceptodds
Withdrawn from ICLR 2027

In-context maps as a collision operator: stochastic jump transformer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce a stochastic jump transformer, an extension of attention in which learned, context-dependent interaction rates govern the occurrence of pairwise jumps, while learned jump maps determine their outcomes. Composing small-time blocks yields McKean–Vlasov jump dynamics, extending the Vlasov continuous-depth limit of deterministic transformers. On compact convex state spaces, rate attention extends transformer universality from continuous deterministic in-context maps to continuous conditional laws, with deterministic maps recovered as Dirac-valued kernels. Because individual interactions are represented explicitly, physical structure can be imposed directly on the jump mechanism. Elastic pair maps, for example, preserve momentum and energy exactly; reversible jumps with symmetric forward–reverse rates yield entropy dissipation. For spatially nonhomogeneous Maxwell–Boltzmann dynamics with a fixed spatial mollifier, we show that their conditional transition laws fit into the stochastic jump transformer framework. More broadly, this suggests a kinetic route to physics foundation models: learn reusable, structure-preserving local interaction laws and compose them across particles and time.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
transformers, operator learning, jump processes, kinetic theory, Boltzmann, McKean–Vlasov dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9msqOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a framework to understand transformers via particle physics.

Critical strengths

N/A

Critical weaknesses
  • This paper has no experiments, I feel the authors could have come up with at least some toy experiments to gather evidence for their theory, or to make an attempt at training a "stochastic jump transformer".
Questions for authors

See weaknesses, please.

Reviewer_UdDzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a transformer architecture extended to McKean-Vlasov jump dynamics to learn broad spectrum of interactions in physical systems. The proposed architecture claims to be compatible with modeling stochastic processes often present in real-world systems. The paper conducts a theoretical study of its components. Main reason for low score is fit with the submission area, I would strongly suggest that authors conduct experiments on at least smaller systems or re-submit to a more theory focused submission track.

Critical strengths
  • Proposition: Designing an optimal transformer architecture specialized for physical applications is an interesting research directions. Exploring McKean-Vlasov dynamics generalizes this problem to considering broader spectrum of interactions
  • Presentation: Paper includes a schematic diagram and description of each block which are useful for designing this architecture in practice
Critical weaknesses
  • Claims: In the abstract the paper mentions: “this suggests a kinetic route to physics foundation models”. I believe in order to suggest a possible route to a foundation model, the work should include experiments validating why such approach is helpful in comparison to existing methods, since it is a rather strong claim otherwise
  • Area: The paper is submitted under “applications to physical sciences (physics, chemistry, biology, etc.)” however the paper is submitted as a theoretical study, hence I would suggest it belongs under the more theoretical focused track rather than applications
  • Applicability: I acknowledge that certain areas of applications are mentioned, however from reading the study it is unclear exactly for which applications this approach would provide most benefits and whether it would be better performing than existing transformer-based architectures. Given the area of submission such analysis would be necessary
Questions for authors
  • Do you have intuition into how scalable the proposed architecture is and what kind compute/sampling budgets it would require?
  • Could you provide some analysis on training stability of such architecture as well as training pseudo-code?
  • Do you have intuition into how difficult it would be to implement this architecture?
Reviewer_AYhMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a stochastic jump transformer that combines deterministic token updates with stochastic pair interactions. Context-dependent “rate attention” determines the probability of accepting an interaction, while a separate jump map with a nonlinear Markov kernel determines its outcome. The paper studies the approximation of continuous conditional laws, establishes a continuous-depth connection to McKean–Vlasov jump processes, and presents structure-preserving collision mechanisms and a spatially mollified Maxwell–Boltzmann realization.

Critical strengths

The architectural idea is interesting. Separating interaction rates from interaction outcomes provides a natural connection between attention mechanisms and kinetic particle models. Stochastic jump updates also allow a fixed input state to produce a distribution of outcomes, extending the capabilities of deterministic token maps. The ideas of rate attention and the jump block using a nonlinear markov kernel are well-thought-out.

Critical weaknesses
  1. The exposition makes the central construction difficult to follow. Although Section 1.1 introduces finite-token updates, the paper moves quickly into measure-valued notation and conditional kernels. A fuller presentation using the classical transformer token equations with the stochastic jump updates across layers would make the contribution substantially more accessible. Algorithm 1 with finite particles in Appendix D is helpful and could provide the organizing framework for the main presentation.

  2. The connection between the discrete architecture and the continuous-depth formulation needs a more explicit explanation. In Theorem B for the continuous-depth model, the drift btb_t and the variables λ\lambda and Φ\Phi are introduced without connecting with the discrete-depth models. The theorem would also benefit from verifying the assumptions on them, e.g., Lipschitz continuity and boundedness.

  3. Definitions, assumptions and cross-referencing. The paper has dense notation and substantial hypotheses, and the reader must track the notations and settings hidden in the text. Clearer statements of assumptions and definitions, along with cross-references to relevant equations, would improve readability and help establish the scope of the claims.

Questions for authors
  1. Theorem A: What are the assumptions and what are the claims in these statements before the equivalent items?
  2. Is there an example explicitly connecting the modules in Eq. (4), the finite-particle algorithm, and the limiting operator in Eq. (14)?
  3. Page 2 bottom paragraph, "section two distinct indices uniformly from {1...N}": what if the two random indices are the same? If they are dropped, the two indices are not independent. Also, in Figure 1, Attention block, draw z∼μNz\sim \mu^N: what if different draws of zz are the same (since they will lead to the same z‾\overline z and z‾∗\overline z_*)?
  4. The conditional law in Equation (7): how does it depend on μ\mu?
  5. Eq.(11): what is the number kk for the atomic measure ρ\rho? What is the continuous-depth counterpart of this equation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.