acceptodds
Under review as a conference paper at ICLR 2027

StrataTok: Local-Variation-Guided Dynamic Tokenization for PDE Forecasting from Sparse and Irregular Observations

Abstract

Observations of physical systems are often sparse and irregularly distributed across space and time. Forecasting PDE dynamics from such observations requires compact spatiotemporal tokens, but fixed tokenizers use the same predefined partition for every input, even when observation locations and locally active regions change. Existing dynamic tokenizers address this rigidity, but under sparse and irregular sampling, soft slicing leaves local variation implicit while patch-based refinement estimates it from unevenly populated or empty patches. To address this limitation, we present STRATATOK, a variation-stratified dynamic tokenizer that organizes token formation through local-variation-guided pathway allocation and learned soft slot assignment. STRATATOK first divides each observation between global and variation-enriched pathways, then forms mass-normalized tokens within each pathway. The resulting assignments admit a mass-conserving discrete transport interpretation. The two pathways form complementary token banks for global and localized dynamics. Across nine simulated PDE settings and four RealPDEBench systems, STRATATOK achieves the lowest RMSE in all thirteen comparisons among six tokenizers. The advantage persists across nearly all tested observation densities and sampling patterns, and STRATATOK ranks first at both full and halved token budgets in all four settings.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
dynamic tokenization, sparse and irregular PDE forecasting, spatiotemporal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sLLsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StrataTok is a dynamic tokenizer for PDE forecasting from sparse and irregular observations. For each visible observation, it estimates local variation from its nearest spatiotemporal neighbors and uses this score to split the observation's mass between a global token bank and a variation-enriched token bank. Within each bank, tokens are formed by Transolver-style soft assignment and mass-normalized pooling. The authors evaluate it against five tokenizers under a shared forecasting backbone on nine simulated PDE settings and four RealPDEBench systems, report the lowest RMSE in all thirteen settings, and add robustness studies over observation density, sampling pattern, and token budget, plus a routing ablation.

Critical strengths

1. Meaningful and practically relevant problem setting. Most existing PDE tokenizers (Transolver, MeshTok, MATEY) assume dense or fully observed discretizations, while real measurements such as PIV or sensor networks are often sparse and irregular. Moving token formation into this regime is a worthwhile direction, and it is the main reason I see value in this line of work despite my concerns.

2. Simple, interpretable design with broad empirical coverage. The method is lightweight. It adds a parameter-free, neighbor-based variation score on top of standard soft assignment, which makes it easy to understand and to plug into existing architectures. The evaluation covers 1D and 2D, planar and spherical domains, and real experimental data from RealPDEBench, and it includes sweeps over observation density, sampling pattern, and token budget. This breadth makes the empirical claims at least testable, even though, as discussed below, I am not yet convinced by the baselines and protocol.

Critical weaknesses

1. Insufficiently strong and validated baselines. The "single learned bank" control (Transolver-style soft assignment) in Table 2 already outperforms all five Table 1 baselines on the three evaluated datasets. Relative to this stronger control, StrataTok reduces RMSE by only 7.7–10.3%, substantially narrowing the headline margins. Yet this control is absent from the main 13-task comparison. Furthermore, MeshTok and MATEY are evaluated through custom sparse adapters whose fidelity to the original methods is unclear, while relevant cross-attention tokenizers (e.g., Perceiver-style or AROMA-inspired encoders) are missing. These omissions make it difficult to establish the magnitude of the improvement over strong alternative tokenization mechanisms.

2. Limited external validation and statistical support. Although the datasets originate from established benchmarks, the paper uses custom splits, sparse-observation protocols, forecasting horizons, and metric aggregation, without reproducing a published evaluation protocol to anchor the results. The additional DINo/STRIDE comparisons also follow this custom setup. Established sparse forecasting evaluations, such as AROMA's Navier–Stokes and Shallow-Water experiments at 5%, 25%, and 100% observation density, provide a relevant reference. Moreover, Table 1 reports geometric means over only three runs without uncertainty estimates, while near-ties of 0.01% and 0.10% are counted as wins. Consequently, the claimed superiority across all thirteen settings is insufficiently supported statistically, and its comparability with published results remains unclear.

Questions for authors
  1. Strength of the single-bank baseline. The single learned bank in Table 2 outperforms every Table 1 baseline on all three evaluated datasets. Why was this stronger control excluded from the main comparison? Can the authors provide its results across all 13 settings to establish the improvement attributable to variation-guided routing?

  2. Fidelity and completeness of baseline implementations. How closely do the custom sparse MeshTok and MATEY adapters preserve the original methods' tokenization mechanisms? What evidence supports their fidelity? Additionally, are results available for sparse-input cross-attention tokenizers, such as Perceiver-style or AROMA-inspired encoders, under the same forecasting backbone?

  3. Comparability with established evaluations. Given the custom splits, forecasting horizons, and metric aggregation, how can the reported results be compared with published sparse PDE forecasting benchmarks? Can the authors provide results under an established protocol, such as the Navier–Stokes and Shallow-Water sparse-observation evaluation used by DINo/CORAL/AROMA?

1 comment

Thank you to the authors for conducting the additional experiments and providing detailed results. However, the evaluation still lacks a directly comparable assessment under an established forecasting protocol. Therefore, my main concern regarding the comparability and external validity of the results remains unresolved.

Reviewer_XunpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StrataTok, a dynamic tokenization method that estimates local spatiotemporal variation from sparse observations and adaptively allocates representation mass between global and variation-enriched token pathways. The method is evaluated on nine simulated PDE settings and four real-world physical systems.

Critical strengths
  • The method addresses a clearly defined challenge in tokenization from sparse and irregular observations, where fixed partitions and conventional activity-based refinement can be unreliable.

  • The evaluation covers diverse physical systems and includes targeted ablations of pathway allocation, variation guidance, and token budgets, providing relatively strong empirical support for the proposed design.

Critical weaknesses
  • The variation scores are computed from differences among observed spatiotemporal neighbors and may be influenced by sampling density, observation noise, and spatial-temporal scaling. Under highly nonuniform sampling, large estimated variation does not necessarily correspond to large physical gradients or dynamically active regions. The current prediction results do not directly validate this central assumption. Namely, the reliability of the local-variation estimator under irregular sampling remains insufficiently validated.

  • Some improvements are marginal, and the computational benefits remain unclear. For example, the reported improvements over Transolver++ on Spherical SWE and Euler CRP are only 0.10% and 0.01%, respectively. Their statistical significance is unclear. Furthermore, the additional costs of neighborhood construction and variation estimation are not sufficiently quantified, making it difficult to assess the end-to-end efficiency of the proposed tokenizer.

Questions for authors
  • On PDE benchmarks with known spatial gradients or localized evolving structures, how well do the estimated variation scores correlate with the corresponding physical variation? How robust is this relationship to changes in observation density and measurement noise?

  • Can the authors report variability across random seeds and end-to-end inference costs, including neighborhood construction and variation estimation? This would clarify whether the marginal improvements are statistically meaningful and whether token reduction translates into actual computational savings.

1 comment

Thank you for your response. I have carefully read the rebuttal and appreciate your clarifications.

Reviewer_5QYcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper proposes StrataTok, a tokenizer that turns a variable-size set of sparse, irregular space time observations into K tokens for a downstream latent forecaster. It works in four steps

  1. Local-variation score. For each visible observation, the method computes a score alpha_i from its k nearest visible neighbors in robustly normalized joint space time. The score mixes a median slope term (value difference per unit distance) and a median amplitude term.

  2. Pathway split. The score is normalized, clipped and reshaped into a split g_i + l_i = 1 between a “global” slot bank and a “variation-enriched” slot bank. This split is fixed (not learned) apart from validation tuned with hyper parameters

  3. Token formation. Within each bank, Transolver-style learned softmax assignments and mass-normalized pooling form the tokens.

  4. Transport view. The paper interprets the resulting assignments as a discrete transport coupling and states three structural properties: unit mass per observation, a lower bound on global-bank mass, and convex-hull containment of the pooled tokens.

Critical strengths

A carefully controlled tokenizer comparison on a realistic problem. Holding the encoder, processor, decoder, capacity and training budget fixed isolates the tokenization step, which is rarely done cleanly. The protocol has several further strengths:

  1. RealPDEBench trajectories are split at the source-trajectory level before window extraction, which avoids leakage.

  2. Split manifests and resolved configurations are frozen.

  3. The density and sampling-pattern sweeps test conditions that matter for sparse sensing.

Forecasting from sparse, irregular observations is practically important and underserved by tokenizers designed for dense meshes, so this evaluation framework is useful in its own right. The routing ablation isolates alignment rather than just added capacity

Critical weaknesses
  1. The main comparison omits the strongest baseline the paper itself contains, so the headline margins are overstated. In Table 2, the plain single-bank Transolver slicing (“Single learned bank”) reaches 0.1779 / 0.2413 / 0.7188 on Radial SWE A / CFD-rand / combustion. That beats every baseline in Table 1 on Radial SWE A and combustion, including Transolver++ at 0.2836 / 0.2501 / 0.7554. Against this comparator, StrataTok’s gains are about 10%, 10% and 8%. That is a meaningful improvement, but far from the 12-54% reductions highlighted relative to Transolver++.

The baseline set is also weaker than it looks. FixedPower and AdaptivePower are constructed by the authors, and MeshTok and MATEY are author-built sparse adapters of methods designed for complete regular patches. The natural tokenizers for sparse, irregular inputs are missing from the tokenizer comparison: Perceiver IO style latent cross-attention, UPT supernode pooling, and LNO style latent entry. All three are discussed in related work.

  1. The technical contribution is narrow, and the framing adds little analytical content. Mechanically, StrataTok is Transolver’s slice assignment with the softmax split over two slot groups. The mass each group receives is set by a fixed, hand crafted kNN statistic. The “discrete transport interpretation” involves no cost and no optimization problem: any row normalized soft assignment followed by mass normalized pooling (including Transolver’s) defines such a coupling. The structural properties in Eq. 10 are immediate consequences of row normalization and of the clip, and they do not explain why aligned allocation improves forecasts. The paper’s description of the two banks as “complementary token banks for global and localized dynamics” is never examined directly. There are no visualizations of where variation-enriched slots place their mass. There is no check that l_i correlates with true local gradients of the dense field, which matters most at 0.31% visibility, where the κ\kappa nearest neighbors can be far away in both space and time. The only qualitative result is a single Radial SWE example.
Questions for authors
  1. Can you add plain Transolver (the “single learned bank” of Table 2) to Table 1 for all 13 settings, and report the standard deviation over runs for every method? Given Table 2, this is the relevant comparator, and the answer most directly affects my assessment.

  2. What was the tuning protocol? How many configurations were evaluated per method, were the per-dataset shared settings in Table A2 selected using StrataTok or per method, and were the ablation controls tuned with the same budget as StrataTok?

  3. Can you include a Perceiver IO / UPT-style latent cross attention tokenizer under the same protocol, along with a persistence or interpolation reference? Please also state whether the reported RMSE is in normalized units, since all methods sit near 0.63 on spherical SWE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.