acceptodds
Under review as a conference paper at ICLR 2027

Mesh-Aware Tokenization for Structured Graph Transformers: Learning Physical Simulation on Higher-Order Meshes

Abstract

We introduce a deterministic mesh-aware tokenization strategy for graph-transformer-based physical simulation on higher-order meshes. Instead of learning token assignments, the proposed method constructs structured tokens directly from the corner nodes of higher-order finite elements and couples them to the fine mesh through parameter-free aggregation and interpolation operators derived from finite-element shape functions. This provides a discretization-aligned interface for global transformer attention without converting higher-order meshes into first-order graphs. Building on this formulation, we propose the Interpolation Mesh Graph Token Transformer (IMGTT), which combines local MeshGraphNet-style message passing with iteratively coupled global attention over mesh-derived tokens. Unlike the dual-graph MS-IMGN architecture, IMGTT replaces coarse message passing with transformer processing while preserving direct higher-order compatibility. Across static and dynamic benchmarks and element orders one through five, IMGTT consistently improves long-horizon rollout behavior over learned-token MGN-T while retaining competitive predictive accuracy across all evaluated domains. Compared with the native higher-order MS-IMGN baseline, IMGTT reduces processor parameter count by 72% and achieves approximately 3x faster training and inference. These results demonstrate that structured mesh-derived tokenization provides an effective interface between higher-order graph simulation and global transformer processing.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
interpolation, mesh-graph, structured tokenization, transformer, mesh-aware, physical simulation, higher-order mesh

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_dcCNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes IMGTT, a graph-transformer model for physical simulations on higher-order meshes. The method uses element corner nodes as coarse tokens, with finite-element interpolation operators to exchange information between fine nodes and coarse tokens. It combines local graph message passing with global transformer attention. Experiments on elasticity, flag dynamics, and fluid dynamics show better rollout performance than MGN-T in several cases, and lower computational costs than MS-IMGN.

Critical strengths
  1. The method provides a relatively simple way to combine local GNN processing and global transformer attention for higher-order meshes, without requiring conversion to first-order graphs.

  2. The experimental results show some promising improvements in rollout accuracy over MGN-T, as well as reduced computational costs (around 3x speedup) compared with MS-IMGN.

Critical weaknesses

I find the main technical contribution somewhat limited. The coarse mesh representation and interpolation operators are already introduced in IMGN, and replacing coarse-level message passing with standard transformer attention seems to be a relatively straightforward extension. The combination of local GNNs and global attention has also been widely explored in previous works. Therefore, I am not fully convinced about the novelty and significance of the proposed method compared with existing approaches.

Also, it is difficult to tell whether the improvements over MGN-T actually come from the proposed tokenization, since the models have different numbers of message-passing layers, transformer layers, and parameters. More careful ablation studies with comparable architectures and computational budgets would be helpful.

One major concern is that the number of coarse tokens Nc still depends on the mesh resolution. When the mesh becomes highly refined, Nc can also become very large and may not be much smaller than Nf. For first-order meshes, Nc is essentially equal to Nf. This means the model still has the O(Nc^2) cost of global attention. In comparison, some existing latent-token transformer methods can use a fixed number of tokens regardless of the input mesh resolution, which seems more attractive for large-scale problems. The paper does not provide enough experiments on mesh refinement, resolution generalization, or truly large-scale meshes to demonstrate its scalability.

Another concern is that the higher-order datasets are generated by interpolating first-order simulation solutions, rather than using actual higher-order FEM solutions. I think this makes it difficult to assess whether the proposed method can really capture and benefit from the numerical properties of higher-order discretizations.

In addition, IMGTT does not consistently outperform the stronger MS-IMGN baseline in prediction accuracy, especially for stress prediction and some long-term rollout results. Some baseline results are also directly taken from previous work instead of being obtained under the same training setup. Overall, I feel the current experimental results are not strong enough to clearly support the claimed advantages in accuracy, scalability, and generalization.

Questions for authors
  1. What is the main technical novelty compared with IMGN/MS-IMGN, apart from replacing coarse-level message passing with transformer attention? How is the proposed method different from existing local-global GNN/transformer architectures?

  2. Can the authors provide more controlled ablation studies to better show the actual benefits of mesh-aware tokenization? For example, comparing deterministic and learned tokenization with similar network architectures, token counts, parameter counts, and computational budgets.

  3. How does the number of coarse tokens Nc change with mesh refinement? It would be useful to report Nc/Nf ratios, GPU memory usage, and runtime on much larger meshes. How would this compare with transformer approaches using a fixed number of latent tokens?

  4. Can the model generalize to much finer meshes or different mesh resolutions without retraining? Simply changing the element order does not seem sufficient to demonstrate resolution generalization or scalability.

  5. Since the higher-order datasets are obtained by interpolating first-order solutions, can the authors provide experiments using actual higher-order FEM solutions? Also, how well can the proposed tokenization and interpolation operators preserve higher-order spatial information and solution details?

7 comments
AuthorsOpenReview ↗

Question 1

The main technical contribution of IMGTT consists of two coupled design choices. First, the mesh-aware structured tokenization derived directly from the finite-element discretization, and second, the interleaved fine-coarse processor that repeatedly exchanges information between persistent fine-node and coarse-token latent representations throughout the processing. First, IMGTT uses the FEM-derived aggregation and interpolation operators as a deterministic, mesh-aware graph-to-token and token-to-graph interface. The operators themselves are inherited from IMGN/MS-IMGN and are not claimed as novel, but their role is different. In IMGN/MS-IMGN, these operators connect fine and coarse graph representations, both of which are processed using separately local message passing. IMGTT replaces coarse-graph message passing with global self-attention over structured tokens associated with the corner nodes of the higher-order mesh. The mesh-derived fine-coarse relation is retained through the inherited FEM aggregation and interpolation operators, but the coarse representation is processed as a globally interacting token set rather than a locally connected message-passing graph. Second, IMGTT differs from the serial local-global organization used by MGN-T. MGN-T first performs local message passing, then constructs learned latent tokens for global attention, and then maps this information back to the fine representation. In IMGTT, both the fine-node and structured-token representations persist throughout the processor. Each processing cycle performs local message passing on the full higher-order graph, aggregates the updated fine representation into the structured tokens, performs global self-attention over those tokens, and interpolates the resulting global information back to the fine graph before the next local processing step. The local and global representations therefore evolve jointly through repeated fine-to-coarse and coarse-to-fine exchange rather than through a single serial local-global-local pathway. The new architecture-tokenization isolation study supports the importance of both components. Holding learned tokenization fixed, replacing MGN-T with the IMGTT processor, both configurations use the same 64-token learned representation, enabling an architectural comparison without introducing a token-count difference. However, processing depth, schedule, and parameter count also differ, therefore the improvement cannot be attributed exclusively to interleaving. With the learned-token budget fixed at 64, the IMGTT processor configuration reduces the velocity rollout-all RMSE mean from 28.93 to 22.98 by 20.6% and the maximum trajectory error from 526.7 to 81.22 by 84.6%. This indicates an architectural benefit beyond the change in tokenization type, although the comparison does not independently isolate processing depth or interleaving. Holding the IMGTT processor fixed, replacing learned tokenization with the structured mesh-aware tokenization further reduces the error from 22.98 to 10.91. Conversely, introducing the structured tokenization into MGN-T substantially reduces the long-horizon error tail severity, although it produces only a modest reduction in its mean error. The strongest performance is obtained only when the structured tokenization and interleaved IMGTT processor are combined. These results also make the distinction from a simple increase in model capacity or token count clearer. The learned-token IMGTT uses the same interleaved IMGTT processor but remains substantially less accurate than the structured-token IMGTT, therefore the improvement cannot be explained by processor depth or capacity alone. Moreover, learned-token IMGTT has slightly more trainable parameters than structured-token IMGTT, 1.14M versus 1.13M, and a slightly higher measured training cost, 64.51 versus 61.80 ms/step. The improvement from structured tokenization therefore cannot be attributed to additional parameter capacity or greater training computation. Likewise, structured-token MGN-T uses the same mesh-dependent structured token set as IMGTT but remains close to the original MGN-T in mean rollout error, showing that a larger or mesh-dependent token set alone is also insufficient to explain the improvement. We therefore do not claim novelty in Transformer attention, local message passing, or the FEM interpolation operators individually. The contribution lies in using the higher-order finite-element discretization to define a structured, parameter-free Transformer token interface and coupling this representation repeatedly with the persistent fine-graph representation through the interleaved IMGTT processor. We will revise the manuscript to make these two coupled contributions and their distinction from IMGN/MS-IMGN and serial local-global Transformer architectures clearer.

AuthorsOpenReview ↗

Quesiton 2

We thank the reviewer for their careful evaluation of our manuscript and constructive comments. We address the raised concerns and provide additional experimental evidence and clarifications below.

Can the authors provide more controlled ablation studies to better show the actual benefits of mesh-aware tokenization? For example, comparing deterministic and learned tokenization with similar network architectures, token counts, parameter counts, and computational budgets. We agree that the original comparison between MGN-T and IMGTT changes both the processor configuration and tokenization strategy, making the contribution of each component difficult to assess independently. To address this limitation, we conducted an architecture-tokenization ablation on the order 3 CylinderFlow benchmark. Four configurations were evaluated, MGN-T and IMGTT, each with learned and structured mesh-aware tokenization. All four models use the same fine graphs, training and test trajectories, and evaluation protocol. The most direct comparison is obtained by holding the IMGTT processor fixed. Replacing learned tokenization with structured tokenization reduces velocity rollout-all RMSE mean from 22.98 to 10.91 by 52.5%, the median from 16.61 to 8.693 by 47.7%, and the 95th percentile from 58.02 to 28.30 by 51%. The parameter count remains nearly identical, 1.14M versus 1.13M, and the processor depth and scheduling are unchanged. The corresponding tokenization comparison within MGN-T produces a smaller reduction in velocity rollout-all RMSE mean, from 28.93 to 27.39 by 5%. However, the standard deviation decreases from 125.9 to 27.21, and the maximum trajectory error decreases from 526.7 to 101.5, indicating reduced extreme-error severity. With learned tokenization held fixed, IMGTT achieves a lower rollout-all RMSE mean than MGN-T, 22.98 versus 28.93. With structured tokenization, this difference increases substantially, 10.91 versus 27.39. These findings indicate that the benefit of structured tokenization is particularly pronounced within the IMGTT processor configuration, supporting the importance of combining the mesh-aware representation with repeated fine-coarse information exchange. The tokenization variants within each processor have nearly identical parameter counts, but their token counts differ. Both learned-token variants use 64 tokens, meanwhile both structured-token variants use 100-127 mesh-derived tokens with mean 113. Furthermore, the IMGTT and MGN-T processors differ in depth, organization, and overall parameter count. Consequently, the experiment supports the effectiveness of the complete tokenization-processor combination but does not independently isolate shape-function weighting, token-count effects, or interleaving from processor depth. Furthermore, within IMGTT, structured tokenization increases inference time from 18.32 to 20.07 ms/step by 10% while reducing velocity rollout-all RMSE mean by 52%. The results therefore demonstrate a substantial accuracy improvement with moderate additional inference cost at the evaluated mesh size.

AuthorsOpenReview ↗

Question 3

How does the number of coarse tokens Nc change with mesh refinement? It would be useful to report Nc/Nf ratios, GPU memory usage, and runtime on much larger meshes. How would this compare with transformer approaches using a fixed number of latent tokens? For a given mesh topology, increasing the element order introduces additional higher-order edge and/or interior nodes while leaving the corner-node mesh unchanged. Therefore, the IMGTT coarse-token count N_c remains fixed for the corresponding mesh, while the fine-node count N_f increases with element order. For Flag Dynamics, N_c ranges from 81 to 138 across the meshes, with a mean of 106. The corresponding N_f values (minimum/maximum/mean) are: 81/138/106 for order 1, 289/493/379 for order 2, 625/1066/819 for order 3, 1089/1857/1427 for order 4, and 1681/2866/2200 for order 5. Using the mean node counts, the corresponding N_c/N_f ratios are approximately 1.00, 0.280, 0.129, 0.074, and 0.048 by element order from 1 to 5. For Fluid Dynamics, N_c ranges from 100 to 127, with a mean of 113. The corresponding N_f values are: 100/127/113 for order 1, 352/460/405 for order 2, 756/999/875 for order 3, 1312/1744/1523 for order 4, and 2020/2695/2349 for order 5. Using the mean node counts, the corresponding N_c/N_f ratios are approximately 1.00, 0.279, 0.129, 0.074, and 0.048 by element order from 1 to 5. For Linear Elastostatics, N_c ranges from 79 to 172, with a mean of 120. The corresponding N_f values are: 79/172/120 for order 1, 276/637/432 for order 2, 592/1394/937 for order 3, 1027/2439/1636 for order 4, and 1581/3792/2528 for order 5. Using the mean node counts, the corresponding N_c/N_f ratios are approximately 1.00, 0.278, 0.128, 0.073, and 0.047 by element order from 1 to 5. We will additionally report the corresponding fine-graph edge counts in the revised manuscript. These values illustrate the intended higher-order setting. As the polynomial order increases, an increasingly smaller fraction of the fine nodes participates in global attention, while the additional higher-order nodes remain in the local fine-graph representation. IMGTT does not, however, use a fixed-size latent bottleneck. At fixed element order, mesh refinement increases the number of corner nodes and therefore increases N_c together with N_f and the number of graph edges. Standard full self-attention over the coarse representation consequently has O(N_c^2) time and memory complexity, while the storage and computation associated with local message passing scale with the fine graph size and edge count. This differs from fixed-latent-token Transformer approaches, in which the number of global tokens can remain constant as the input mesh is refined. Such methods can therefore maintain resolution-independent global-attention cost, meanwhile IMGTT retains an explicit one-to-one correspondence between coarse tokens and mesh corner nodes. The trade-off is that IMGTT preserves a discretization-aligned structured representation, but its global-attention cost is not independent of mesh resolution. The current experiments characterize the node counts and runtimes of the evaluated meshes but do not constitute a fixed-order mesh-refinement scaling study. We have also not measured peak GPU-memory scaling on substantially larger meshes. We will therefore make this limitation explicit as the empirical runtime and memory scaling under large fixed-order refinement remain to be evaluated separately. For the third-order Fluid Dynamics benchmark used in the ablation, the structured representation contains an average of 113 coarse tokens for 875 fine nodes, corresponding to a coarse-to-fine node ratio of approximately 12.9%. Global self-attention therefore operates on a substantially smaller representation than the full fine graph, although the coarse-token count would increase under mesh refinement at fixed element order.

Reviewer_qC7uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a deterministic, parameter-free tokenization scheme for graph transformers on higher-order finite-element meshes. The corner nodes of higher-order elements serve as transformer tokens. Fine nodes and tokens are coupled through aggregation and interpolation operators derived from linear FEM shape functions, following IMGN. The resulting architecture, IMGTT, alternates MeshGraphNet-style local message passing with global self-attention over these tokens. It works directly on higher-order discretizations, without converting them to first-order meshes. Compared with the native higher-order MS-IMGN baseline, IMGTT reduces processor parameter count by 72% and achieves approximately 3×faster training and inference.

Critical strengths
  1. The paper introduces a principled, parameter-free interface that handles higher-order meshes natively. Fixing tokens and coupling to the FEM discretization removes both the learned token assignment and the first-order conversion. The idea is clean and novel.

  2. It shows speed and robustness gain over the relevant baselines. IMGTT is about 3× faster than MS-IMGN with 72% fewer parameters.

Critical weaknesses
  1. IMGTT has 4 transformer layers, 5 message-passing steps and 1.13M parameters. MGN-T has 2 transformer layers, 4 message-passing steps and 0.80M parameters. The gains over MGN-T may therefore come from extra capacity rather than the tokenization. The paper didn't tests:
  • IMGTT with learned tokens;
  • MGN-T matched in depth and parameters;
  • FEM interpolation versus simple corner-node pooling;
  • removing the positional encodings;
  • varying the number of cycles. The authors should have these ablation studies to claim the gain from the tokenization.
  1. Results come from a single training run; the ± values are spread across test trajectories, not across seeds.

  2. The benchmarks are small, 2D, and use transformer baselines only. The paper skips stronger recent baselines (Transolver, BSMS-GNN, Garnier et al.) and didn't tests at scale, even though attention costs O(N_c²) in the number of tokens.

Questions for authors
  1. Can the authors provide ablation studies as I mentioned above?
  2. How many tokens and fine nodes does each dataset have per element order, and what token budget did MGN-T use? How does this affect the attention cost and the fairness of the runtime comparison?
  3. Are the results stable across multiple training seeds? Please report mean ± std across seeds.
  4. Test on data generated natively with higher-order FEM, on 3D problems, and at larger scale where O(N_c²) attention becomes the bottleneck. Consider linear-attention or hierarchical token variants. Compare with stronger recent baselines.
5 comments
AuthorsOpenReview ↗

Question 1

We thank the reviewer for their careful evaluation of our manuscript and constructive comments. We address the raised concerns and provide additional experimental evidence and clarifications below.

IMGTT with learned tokens; MGN-T matched in depth and parameters; FEM interpolation versus simple corner-node pooling removing the positional encodings; varying the number of cycles. The authors should have these ablation studies to claim the gain from the tokenization. Can the authors provide ablation studies as I mentioned above? We agree that the original IMGTT–MGN-T comparison does not fully isolate the effect of tokenization, as the two models also differ in processor organization, depth, and parameter count. To address this limitation, we conducted an architecture–tokenization ablation on the third-order CylinderFlow benchmark, evaluating both processor configurations with learned and structured tokenization. According to this study: IMGTT processor fixed: from 22.98 to 10.91 with structured tokens. Structured tokens fixed: from 27.39 to 10.91 with IMGTT processor. Learned tokens fixed: from 28.93 to 22.98 with IMGTT processor. MGN-T processor fixed: from 28.93 to 27.39 mean, but reduces the standard deviation from 125.9 to 27.21 and much smaller tail severity with structured tokens. The results show that, within the fixed IMGTT processor, structured tokenization reduces the rollout-all velocity RMSE mean from 22.98 to 10.91 by 52%, while maintaining nearly identical parameter counts, 1.14M and 1.13M. In contrast, applying structured tokenization to MGN-T produces a smaller reduction in mean error, from 28.93 to 27.39 by 5%, although the standard deviation decreases substantially from 125.9 to 27.21, indicating reduced extreme-error severity. Comparing the processor configurations under the same tokenization strategy, IMGTT reduces the mean error from 28.93 to 22.98 with learned tokens and from 27.39 to 10.91 with structured tokens. These findings suggest that the performance improvement arises from the combination of structured tokenization and the IMGTT processor configuration, rather than increased processor depth or tokenization changes alone. The within-architecture comparisons maintain identical processor organization and nearly identical parameter counts 0.80M/0.79M for MGN-T and 1.14M/1.13M for IMGTT, providing evidence for the effectiveness of the complete structured tokenization configuration. However, the learned and structured variants use different token counts, while the processor configurations differ in depth, organization, and capacity. Consequently, the study does not isolate the contribution of the interleaving schedule or shape-function weights independently, nor does it constitute a fully matched-budget comparison across architectures. We agree that replacing the shape-function-based aggregation and interpolation weights with a fixed mapping would help distinguish the contribution of structured tokenization from the more general benefit of deterministic mesh-based token assignment. The present ablation compares the structured mapping against learned tokenization and therefore does not isolate the specific contribution of the shape-function coefficients. We have selected this aggregation/interpolation method because they are directly defined by the higher order mesh representation and provide a geometrically and topologically consistent local mapping between fine nodes and corner-node tokens. We do not claim that they are optimal among all deterministic mappings, and we will clarify this limitation in the manuscript. Positional encoding and processing-cycle count are relevant architectural sensitivity factors, particularly because they may interact with the structured token representation. We retain both at their proposed settings in the present ablation to isolate the effect of changing the tokenization configuration within a fixed processor. We do not claim that the selected positional encoding or processing-cycle count is independently optimal, and their individual influence remains outside the scope of the present experiments. Among the requested ablations, the new experiment directly addresses the learned-token IMGTT comparison and provides complementary cross-architecture evidence. Matched-depth/parameter MGN-T, alternative fixed pooling, positional-encoding removal, and processing-cycle sensitivity have not been evaluated independently.

AuthorsOpenReview ↗

Question 2 / 1

How many tokens and fine nodes does each dataset have per element order, and what token budget did MGN-T use? How does this affect the attention cost and the fairness of the runtime comparison? For a given remeshed sample and underlying corner-node topology, increasing the element order introduces additional higher-order edge and/or interior nodes without changing the corner-node mesh. Consequently, the IMGTT coarse-token count N_c remains unchanged across element orders for the corresponding mesh, while the fine-node count N_f increases substantially. The fixed corner-token count across element orders applies to the benchmark construction, where the higher-order meshes are generated from a common underlying corner-node topology. It does not imply that the token count remains constant under independent mesh refinement. For Flag Dynamics, N_c ranges from 81-138 across the meshes, with a mean of 106. The corresponding N_f values (minimum/maximum/mean) are: 81/138/106 for order 1, 289/493/379 for order 2, 625/1066/819 for order 3, 1089/1857/1427 for order 4, and 1681/2866/2200 for order 5. For Fluid Dynamics, N_c ranges from 100-127, with a mean of 113. The corresponding N_f values are: 100/127/113 for order 1, 352/460/405 for order 2, 756/999/875 for order 3, 1312/1744/1523 for order 4, and 2020/2695/2349 for order 5. For Linear Elastostatics, N_c ranges from 79-172, with a mean of 120. The corresponding N_f values are: 79/172/120 for order 1, 276/637/432 for order 2, 592/1394/937 for order 3, 1027/2439/1636 for order 4, and 1581/3792/2528 for orders 5. We will additionally report the corresponding fine-graph edge counts in the revised manuscript. The two models use fundamentally different token-count definitions. In IMGTT, the token set is determined directly by the corner-node structure of the finite-element mesh, with one structured token associated with each corner node. In MGN-T, the number of latent tokens P is a tunable model hyperparameter that can remain independent of mesh resolution. The fixed budget of 64 learned tokens follows the token-count sensitivity analysis of the MGN-T publication of Iparraguirre et al. (2026), Section 5.3, Figure 8, which reports diminishing accuracy gains beyond approximately 32 tokens in the settings evaluated there. We adopt 64 tokens as a representative learned-token configuration to operate comfortably within this saturation regime rather than using the minimum reported sufficient value, while recognizing that this prior result does not establish the optimal budget for the present experiments. The resulting token counts are therefore intentionally not matched. Artificially reducing N_c in IMGTT would remove the one-to-one correspondence between tokens and mesh corner nodes that defines the structured tokenization, while increasing the learned-token budget to match the mesh-derived token count would provide an additional control on token count, although it would also increase the learned-token model's attention cost. The published MGN-T token-budget study motivates our use of 64 latent tokens, but the present experiments do not establish whether a larger learned-token budget would alter the observed accuracy gap. The architecture-tokenization isolation study provides an additional control on this issue. For the order 3 Fluid Dynamics benchmark, the structured-token variants use 100-127 tokens, with a mean of 113, meanwhile the learned-token variants use P=64. Despite this, the structured-token MGN-T achieves a rollout-all velocity RMSE of 27.39, close to the 28.93 of the original learned-token MGN-T and substantially above IMGTT's 10.91. As a consequence, the accuracy improvement of IMGTT cannot be explained simply by providing a larger global token set.

AuthorsOpenReview ↗

Question 2 / 2

Standard full self-attention has O(N_token^2) time and memory complexity, therefore the token count differences directly affect the global-attention cost. For example, using the mean order 3 Fluid Dynamics token count, the structured representation contains 113/64≈1.77 times as many tokens as MGN-T, corresponding to approximately (113/64)^2≈ 3.1 times as many pairwise attention interactions. Despite the approximately 3.1-fold increase in nominal attention score at the mean token count, structured-token IMGTT requires 20.07 ms per inference step compared with 18.32 ms for learned-token IMGTT, an increase of approximately 9.6%. This additional cost stems from the higher complexity of the aggregation/interpolation mappings compared to the learned weight based pooling. This demonstrates that the attention-score count alone does not determine the measured end-to-end inference cost at the evaluated mesh size. This is a nominal attention-size comparison rather than a runtime ratio, because total runtime also includes fine-graph message passing, tokenization, interpolation, and other model operations. Accordingly, the reported runtime results should be interpreted as a comparison between the practical configurations of the two methods rather than as a matched-token-count attention benchmark. IMGTT generally processes more global tokens than the P=64 MGN-T configuration, while retaining the mesh-defined correspondence of its structured tokenization. We will make the fine-node, edge, and token counts explicit in the revised manuscript so that these structural and computational differences are transparent.

Reviewer_ark5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes IMGTT, a graph-transformer surrogate for physical simulation on higher-order finite element meshes. Instead of learning token assignments, it uses the corner nodes of higher-order elements as a fixed set of coarse tokens. These are coupled to the fine mesh by parameter-free aggregation and interpolation operators derived from linear shape functions. Local MeshGraphNet-style message passing alternates with global transformer attention over the coarse tokens. The method is evaluated on linear elastostatics, flag dynamics and cylinder flow benchmarks for element orders one to five. It is compared with MGN, MGN-T and (MS-)IMGN, and the authors report improved rollout stability over MGN-T and lower cost than MS-IMGN.

Critical strengths
  • The paper is well written. The main idea, a fixed, shape-function-based coarse token set feeding a transformer, is well motivated and easy to follow.
  • The method is substantially more efficient than the native higher-order baseline MS-IMGN: about 3× faster training and inference with a 72% smaller processor.
  • Source code is provided with the submission, which supports reproducibility.
Critical weaknesses
  1. Limited novelty.

    • The aggregation and interpolation operators (Eqs. 1–2) come directly from Sebestyen & Kim (2025).
    • The overall design follows existing work. Local message passing combined with global attention follows GraphGPS and MGN-T, and the positional encoding follows MGN-T.
    • The main contribution is therefore replacing the coarse message-passing graph of (MS-)IMGN with a transformer over the same coarse nodes. This is a reasonable engineering step, but I think the methodological contribution is too small for ICLR.
  2. Weak accuracy results.

    • IMGN or MS-IMGN gives the lowest error on most reported metrics, for example linear elastostatics position and stress for orders 2–5 (Table 2) and most flag dynamics metrics (Tables 3–5).
    • IMGTT mainly outperforms MGN-T, which is an indirect baseline that runs on converted first-order meshes.
    • The authors themselves describe the approach as "a favourable trade-off" rather than a state-of-the-art method. The paper's value therefore rests mainly on efficiency, which leads to my concerns about methodology.
  3. Weak experimental methodology.

    • The MGN and (MS-)IMGN accuracy numbers are copied from Sebestyen & Kim (2025), lines 362–367. The runtimes come from the authors' own reimplementations, lines 440–441. Accuracy and computational cost therefore come from different sources, so the accuracy–efficiency trade-off, which is the paper's central argument, rests on comparisons not made under identical conditions.
    • The models' capacities differ. IMGTT has 1.13M processor parameters and four transformer layers, while MGN-T has 0.80M parameters and two transformer layers (lines 435–438). Part of IMGTT's improvement over MGN-T may be due to this extra capacity.
    • There are no ablation studies.
  4. Benchmark validity.

    • Reusing the existing benchmarks from Sebestyen & Kim (2025) is understandable, and I do not attribute weaknesses of that earlier work to this submission. However, the original dataset construction has a limitation that this paper inherits, and it directly affects the paper's main claims.
    • As stated in lines 324–330 and in Section 4 of Sebestyen & Kim (2025), the order-2 to order-5 elastostatics data are not higher-order FEM solutions. They are first-order (P1) solutions linearly interpolated onto the nodes of higher-order elements.
    • According to Sebestyen & Kim (2025), the reference first-order mesh is only about 5× finer globally and about 10× finer at the crack than the higher-order meshes. A 5th-order element of size H has nodes spaced about H/5 apart, which is comparable to the reference mesh spacing away from the crack.
    • The 5th-order data are therefore essentially the same P1 field sampled at roughly its own resolution. They contain no higher-order approximation accuracy, and the reference is not "high-resolution" relative to the higher-order node layouts.
    • In addition, P1 and higher-order elements converge at different rates, especially for stresses, and the singular stress field at the crack tip is unlikely to be converged.
    • Consequently, "element order" in the experiments measures node layout rather than discretization order. This undercuts the claim in the title and abstract about learning physical simulation on higher-order meshes.
  5. Smaller issues.

    • I would have liked to see visualisations of the predicted fields and error fields for the different cases, so that their qualitative behaviour can be assessed. In linear elastostatics, errors can be highly local, for example the stress concentration around the crack, and a global RMSE may not capture them well. Sebestyen & Kim (2025) includes such figures, and similar figures in the appendix would help the assessment and improve readability.
    • I would have liked to see a 3D solid mechanics case, which would show whether the approach scales to more realistic engineering problems.
Questions for authors
  1. Could you report accuracy and runtime for all baselines retrained under the same protocol?
  2. Could you compare IMGTT and MGN-T at matched capacity, and add ablations isolating the effect of structured versus learned tokenization?
  3. Could you describe the linear elastostatics dataset in more detail? This includes Poisson's ratio, whether it is plane stress or plane strain, the element type, the resolution of the reference mesh, and how nodal stresses are computed.
  4. Could you show, at least for a subset of the static cases, results on genuinely solved higher-order FEM data? Alternatively, could you rephrase the claims so they refer to higher-order node layouts rather than higher-order discretizations?
  5. Could you add qualitative field and error visualisations, particularly around the crack in the elastostatics benchmark?
6 comments
AuthorsOpenReview ↗

Answer to critical weaknesses

We thank the reviewer for their careful evaluation of our manuscript and constructive comments. We address the raised concerns and provide additional experimental evidence and clarifications below.

We agree that the aggregation and interpolation operators themselves are inherited from Sebestyen & Kim (2025), and we do not claim these operators as a new contribution. Likewise, local message passing, transformer attention, and positional encoding are established components.

Our contribution is the use of the FEM-derived aggregation and interpolation operators as a deterministic, mesh-aware graph-to-token and token-to-graph interface specifically for Structured Transformer-based learned simulation on higher-order meshes. In IMGN/MS-IMGN, the same operators connect fine and coarse graph representations that are both processed by separately local message passing. In IMGTT, the coarse graph is removed and the first-order corner nodes define structured Transformer tokens, and the shape-function operators provide a parameter-free correspondence between these tokens and the fine graph. The resulting processor repeatedly couples local message passing on the higher-order mesh with global attention over this Structured Token representation, which includes inductive biases from the locality and static nature of the shape function based mapping.

The new architecture-tokenization ablation further supports the importance of this coupling as the structured tokenization reduces rollout-all velocity RMSE mean by 52% within IMGTT, but by only about 5% within the serial MGN-T processor. This indicates that the observed benefit depends strongly on how the token representation is integrated into the processing architecture.

We therefore agree that the novelty does not lie in any individual component, but instead it consists of two coupled design choices, first, the structured graph-token interface, and second, the interleaved fine-coarse processor that repeatedly exchanges information between the two representations in every latent update step. The architecture-tokenization method isolation study indicates that the IMGTT processor configuration provides a predictive advantage under both learned and structured tokenization, while structured tokenization produces a particularly large additional improvement when combined with IMGTT. This supports the importance of their coupling rather than establishing equally strong independent benefits from each component.

We agree that IMGTT does not achieve the lowest error on every metric and the manuscript does not intend to claim universal state-of-the-art accuracy. The natively higher order IMGN/MS-IMGN remain stronger on several quantities and settings. Our claim is instead that IMGTT provides a favorable accuracy-efficiency trade-off while showing strong rollout behavior in several of the evaluated settings.

In particular, the comparison with MGN-T is important because it evaluates a Transformer-based learned simulator, meanwhile IMGN/MS-IMGN provide the native higher-order message-passing reference. The purpose of including both groups is therefore different. MGN-T evaluates the Transformer-based alternative, while IMGN/MS-IMGN evaluates the native higher-order graph-processing alternative.

AuthorsOpenReview ↗

Question 1

Could you report accuracy and runtime for all baselines retrained under the same protocol?

We retain the published benchmark accuracy results from Sebestyen & Kim (2025), which were obtained using the same benchmark datasets, training and test splits, evaluation protocol, and error metrics as those used in the present study. Reproducing the full benchmark is highly computationally expensive. Based on the measured training-step times of our implementations, retraining a single seed requires approximately 26 days of aggregate GPU training time for MGN and 36 days for MS-IMGN alone. For the MGN-T and IMGTT it would take about 9 and 11 days respectively. These estimates assume 2.5 million training steps per model and are calculated from the measured average training-step times, summed across the five element orders. They represent aggregate GPU training time rather than elapsed wall-clock time and exclude additional overhead such as dataset preprocessing and hyperparameter selection. While retraining the benchmark models under the same implementation and protocol would provide identical training conditions, the complete retrain is prohibitively expensive.

AuthorsOpenReview ↗

Question 2

Could you compare IMGTT and MGN-T at matched capacity, and add ablations isolating the effect of structured versus learned tokenization? We agree that the original IMGTT-MGN-T comparison does not isolate the effect of tokenization, since the complete models also differ in processor organization, depth, and parameter count. To address this directly, we conducted an architecture-tokenization method isolation study comparing learned and deterministic shape-function tokenization within both the MGN-T and IMGTT processor architectures on the order 3 CylinderFlow dataset. According to this study: IMGTT processor fixed: from 22.98 to 10.91 with structured tokens. Structured tokens fixed: from 27.39 to 10.91 with IMGTT processor. Learned tokens fixed: from 28.93 to 22.98 with IMGTT processor. MGN-T processor fixed: from 28.93 to 27.39 mean, but reduces the standard deviation from 125.9 to 27.21 and much smaller tail severity with structured tokens.

This provides controlled comparisons in which the processor architecture is held fixed when comparing tokenization strategies, and the tokenization strategy is held fixed when comparing processor architectures. These comparisons control the processor organization within each architecture and the tokenization type across architectures, but they do not simultaneously match token count, depth, parameter count, and inference cost across all four configurations. The learned-token configurations use a fixed budget of 64 latent tokens. In contrast, the structured-token configurations derive their token count from the corner-node topology of each mesh. The corresponding minimum/maximum/mean token counts are 81/138/106 for Flag Dynamics, 100/127/113 for Fluid Dynamics, and 79/172/120 for Linear Elastostatics. In the third-order CylinderFlow ablation specifically, both structured-token configurations use 100-127 tokens, with a mean of 113. These token counts are not intended to be identical because they play different roles in the two methods. In MGN-T, P is a tunable latent bottleneck independent of the mesh structure. In IMGTT, each token has an explicit correspondence to a corner node, therefore N_c is determined by the discretization and cannot be reduced arbitrarily without changing the proposed tokenization. The fixed budget of 64 learned tokens follows the token-count sensitivity analysis of the MGN-T publication of Iparraguirre et al. (2026), Section 5.3, Figure 8, which reports diminishing accuracy gains beyond approximately 32 tokens in the settings evaluated there. We adopt 64 tokens as a representative learned-token configuration to operate comfortably within this saturation regime rather than using the minimum reported sufficient value, while recognizing that this prior result does not establish the optimal budget for the present third-order CylinderFlow experiment. The isolation study provides additional evidence that token count alone does not explain the differences between the evaluated processor configurations. Although structured-token MGN-T uses the same mesh-dependent token count as structured-token IMGTT, it achieves a considerably higher mean velocity rollout-all RMSE of 27.39 compared to IMGTT’s 10.91. This indicates that simply using the larger structured token set does not reproduce the performance of the complete IMGTT processor. We therefore do not interpret equal token count as equivalent model capacity. Likewise, exact parameter matching would require modifying architectural hyperparameters such as hidden width, message-passing depth, or transformer depth, introducing an additional architecture-optimization problem. Instead, the architecture-tokenization method isolation study on the CylinderFlow dataset with order 3, provides a more direct comparison between the learned tokenization and the structured tokenization mapping. We will report token counts, parameter counts, and computational costs for all four configurations so that these differences remain explicit.

Reviewer_aHNbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Interpolation Mesh Graph Token Transformer (IMGTT), a learned simulator for higher-order meshes. Element corner nodes serve as deterministic transformer tokens, and fixed finite-element-inspired aggregation and interpolation operators transfer latent features between these tokens and fine mesh nodes. The processor interleaves local message passing with global token attention.

The method combines the higher-order mesh interface of IMGN with transformer-based global processing. Experiments cover linear elastostatics, flag dynamics, and cylinder flow across element orders one through five. IMGTT generally improves rollout errors over the evaluated learned-token MGN-T baseline and uses fewer processor parameters and less runtime than MS-IMGN, although its accuracy relative to native higher-order message-passing baselines varies substantially by task and output quantity.

Critical strengths

A coherent architecture that incorporates existing mesh structure. The method uses a physically motivated, parameter-free interface between higher-order nodes and global transformer tokens. Keeping local message passing while repeatedly exchanging information with the global representation is a practical way to combine geometric locality with long-range communication. The paper clearly acknowledges that its transfer operators originate in IMGN.

Promising empirical accuracy–cost trade-offs across several settings. Evaluation spans three physical domains and five element orders. IMGTT reduces full-rollout flag position and velocity errors relative to MGN-T at every order, and achieves the lowest reported full-rollout fluid acceleration errors. Table 9 also reports approximately threefold faster training and inference than the evaluated MS-IMGN configuration, with 1.13M versus 3.98M processor parameters. These results support the potential practical value of the architecture.

Critical weaknesses

The experiments do not isolate the central tokenization contribution. Compared with MGN-T, IMGTT changes learned versus deterministic tokenization, sequential versus interleaved local–global processing, transformer depth, message-passing depth, and parameter count. Section 5.1 reports four transformer layers for IMGTT versus two for MGN-T, and 1.13M versus 0.80M processor parameters. No controlled ablation changes only the tokenization or compares the FEM-derived weights with simpler fixed pooling using the same token locations. Because the transfer operators are inherited from IMGN and local–global transformer processing is established, these controls are important for establishing the incremental contribution.

The practical advantage over the strongest native baseline is not sufficiently established under matched budgets. The reported speedup compares a shallow IMGTT processor with a substantially deeper MS-IMGN configuration, without an accuracy–runtime comparison against reduced-depth or reduced-width MS-IMGN. Accuracy and runtime also come from different sources for the older baselines: published benchmark results are retained for accuracy, while new implementations are profiled for runtime. The accuracy trade-offs are material: at element order four, fluid velocity RMSE-all is 18.80 for IMGTT versus 7.306 for MS-IMGN, and at order five, elastostatic stress RMSE is 3.731 versus 1.179 for IMGN. These results do not negate the speed benefit, but make a matched-budget comparison necessary to determine whether IMGTT offers a better trade-off than simply making the native baseline smaller.

Questions for authors
  1. Controlled attribution: With the same fine graph, token count, local–global schedule, transformer depth, and approximately matched parameter count, how does fixed mesh-derived tokenization compare with learned tokenization? Does the benefit persist when shape-function weights are replaced by simpler fixed aggregation and interpolation?

  2. Accuracy–efficiency comparison: How does IMGTT compare with a reduced-depth or reduced-width MS-IMGN at similar runtime or parameter count? Can accuracy and runtime be reported from the same trained implementations for representative benchmark settings?

  3. Robustness of the reported improvements: How many independent training runs were performed? Does the fourth-order MGN-T rollout failure recur across seeds? Please report seed-level variation and trajectory failure rates, since the current standard deviations over test simulations do not establish training reproducibility.

  4. Higher-order evaluation and scaling: What are the fine-node, edge, and corner-token counts at each element order, and are models trained separately for each order? Since the reference data are interpolated from first-order simulations, what evidence establishes benefits beyond compatibility with the resampled representation? In particular, how does runtime and memory scale when the number of elements—and therefore corner tokens—increases at fixed element order?

4 comments
AuthorsOpenReview ↗

Question 1

We thank the reviewer for their careful evaluation of our manuscript and constructive comments. We address the raised concerns and provide additional experimental evidence and clarifications below.

Detailed statistics and computational measurements are provided in the supplementary materials.

Controlled attribution: With the same fine graph, token count, local-global schedule, transformer depth, and approximately matched parameter count, how does fixed mesh-derived tokenization compare with learned tokenization? Does the benefit persist when shape-function weights are replaced by simpler fixed aggregation and interpolation? We agree that the original IMGTT-MGN-T comparison does not isolate the effect of tokenization, since the complete models also differ in processor organization, depth, and parameter count. To address this directly, we conducted the architecture-tokenization method isolation study, comparing learned and deterministic structured tokenization within both the MGN-T and IMGTT processor architectures on the order 3 CylinderFlow dataset. According to this study, within the fixed IMGTT processor, structured tokenization reduces rollout-all velocity RMSE mean from 22.98 to 10.91 by 52%, with corresponding reductions in the median, from 16.61 to 8.693 and in the 95th percentile, from 58.02 to 28.30. The two configurations have nearly identical parameter counts, 1.14M and 1.13M, and identical processing depth and schedule. The improvement is also observed across the error distribution rather than being restricted to its most extreme values, as the median decreases by 48% and the 95th percentile by 51%. This further suggests that the complete IMGTT improvement cannot be explained solely by the difference in processor architecture or trainable parameter count. The token counts, however, are not identical as the learned tokenization uses 64 latent tokens, meanwhile the mesh-derived method uses 100-127 tokens for the evaluated meshes. Consequently, the experiment controls the surrounding processor architecture but does not isolate shape-function weighting from token count. We agree that a fixed non-FEM derived pooling baseline would further distinguish the benefit of deterministic token assignment from the specific benefit of shape-function weights. The present ablation establishes the improvement of the complete mesh-derived mapping over learned tokenization, but does not identify which part of that improvement originates specifically from the interpolation weights. We therefore do not claim optimality among all possible deterministic mappings. The use of shape functions is motivated by their consistency with the underlying higher-order finite-element representation and their use in conventional numerical solvers, rather than by a claim that they are universally superior to other fixed aggregation and interpolation rules.

Unlike arbitrary pooling weights, the FEM-derived mappings are defined directly by element connectivity and the basis functions associated with the discretization. They therefore provide a deterministic, geometrically meaningful correspondence between corner-node tokens and the higher-order fine graph without requiring learned node-to-token assignments.

AuthorsOpenReview ↗

Question 2

Accuracy-efficiency comparison: How does IMGTT compare with a reduced-depth or reduced-width MS-IMGN at similar runtime or parameter count? Can accuracy and runtime be reported from the same trained implementations for representative benchmark settings?

The accuracy results for IMGN/MS-IMGN are retained from the published benchmark of Sebestyen & Kim (2025). The runtime measurements were obtained by profiling our implementations under the hardware, software, and numerical-precision environment kept identical. Consequently, the reported accuracy and runtime values do not originate from the same trained checkpoints, and the comparison should be interpreted at the architecture-configuration level rather than as a jointly measured accuracy-runtime result.

Reproducing the full benchmark is highly computationally expensive. Based on the measured training-step times of our implementations, retraining a single seed requires approximately 26 days of aggregate GPU training time for MGN and 36 days for MS-IMGN alone. For the MGN-T and IMGTT it would take about 9 and 11 days respectively. These estimates assume 2.5 million training steps per model and are calculated from the measured average training-step times, summed across the five element orders. They represent aggregate GPU training time rather than elapsed wall-clock time and exclude additional overhead such as dataset preprocessing and hyperparameter selection.

We therefore do not perform an additional hyperparameter search over reduced-width or reduced-depth IMGN/MS-IMGN variants. A matched-runtime or matched-parameter MS-IMGN comparison would be valuable for evaluating the relative efficiency of the two architectures. The present results compare the established published benchmark configurations rather than configurations individually optimized for an identical computational budget. We therefore limit our efficiency conclusions to these evaluated configurations and will explicitly distinguish benchmark accuracy measurements from implementation-level runtime profiling.

Our comparison instead uses the published IMGN/MS-IMGN configurations as established native higher-order baselines and reports their published accuracy. However, as the execution environment heavily affects the runtime, the training and inference time measurements are from our implementations under the same hardware, software, and numerical-precision environment as IMGTT. As a consequence, the reported accuracy-runtime relationship for those historical baselines is a configuration-level comparison rather than a same-trained-checkpoint. We will clarify this distinction and the origin of the reported measurements in the revised manuscript.

AuthorsOpenReview ↗

Question 3

Robustness of the reported improvements: How many independent training runs were performed? Does the fourth-order MGN-T rollout failure recur across seeds? Please report seed-level variation and trajectory failure rates, since the current standard deviations over test simulations do not establish training reproducibility.

Due to the computational cost discussed above, one independent training run was conducted per configuration in the performance analysis. Consequently, seed-level variation and recurrence of the order 4 MGN-T behavior across independently trained models cannot be established from the present experiments.

We therefore examined the reported order 4 behavior at the trajectory level. For MGN-T, the rollout-all velocity error exhibits a strongly heavy-tailed distribution, with mean 617.6, standard deviation 3546, median 39.12, 95th percentile 2855, and maximum 17092. In contrast, over the first 50 rollout steps, the median, 95th percentile, and maximum are 3.743, 14.77, and 19.88, respectively. The rollout-all mean is therefore almost 16 times the median, while the maximum exceeds the median by more than a factor of 400. This indicates that the large aggregate error is driven by a subset of trajectories that develop very large errors during long autoregressive rollout rather than by uniformly poor predictions across the complete test set.

For comparison, IMGTT under the same order 4 fluid-dynamics configuration has a rollout-all mean of 18.80, standard deviation of 22.06, median of 12.27, 95th percentile of 49.18, and maximum of 91.98. Over the first 50 rollout steps, its median, 95th percentile, and maximum are 3.458, 14.57, and 22.87, respectively. The first-50-step error distributions are broadly comparable in their medians and upper quantiles, meanwhile the rollout-all distributions differ substantially. This suggests that the severe errors observed in the fourth-order MGN-T configuration emerge or accumulate during the later stages of autoregressive prediction.

To quantify the occurrence of extreme high-error rollouts, one scalar error was first calculated for each of the 100 held-out test trajectories by averaging its per-timestep velocity RMSE over the complete rollout. Extreme high-error trajectories were then identified using a modified z-score threshold of 3.5 applied to the log-transformed trajectory errors, with the median absolute deviation used as the robust scale estimate. The criterion was calculated separately for each model and element-order configuration.

Under this criterion, in the order 4 MGN-T configuration, 12 of the 100 test trajectories are identified as high-error outliers under the model-specific robust statistical criterion. This quantifies the heavy-tailed behavior of the evaluated trained model, rather than a numerical failure rate or a seed-level failure probability. The order 2, order 3, and order 5 MGN-T configurations each contain one such trajectory, meanwhile none are identified for order 1. In contrast, no extreme high-error trajectories are identified for IMGTT at any of the evaluated element orders on the fluid-dynamics benchmark.

The additional order 3 architecture-tokenization isolation study exhibits a similar reduction in extreme error severity. The maximum velocity rollout-all RMSE decreases from 526.7 for learned-token MGN-T to 81.22 for learned-token IMGTT and to 47.51 for structured-token IMGTT. This supports the observation that the evaluated IMGTT configurations exhibit less severe extreme errors, although it does not establish reproducibility across training seeds.

Thus, the elevated order 4 MGN-T aggregate rollout error cannot be attributed to a single problematic test trajectory or to uniformly poor performance across the entire test set. Instead, it is associated with a heavy tail in the trajectory-level error distribution as 12% of the test trajectories develop extremely large errors during long autoregressive rollout. More importantly, the absolute rollout-all error distribution differs substantially as the maximum velocity RMSE decreases from 17092 for MGN-T to 91.98 for IMGTT, while the 95th percentile decreases from 2855 to 49.18. These values provide direct evidence that the evaluated IMGTT model avoids the exceptionally large errors observed in the fourth-order MGN-T model.

We emphasize, however, that this analysis characterizes trajectory-level behavior within the trained models and it does not establish seed-level reproducibility. Confirming whether the same behavior recurs across independent training seeds would require additional training runs, which were not performed in the present evaluation.

AuthorsOpenReview ↗

Question 4

Higher-order evaluation and scaling: What are the fine-node, edge, and corner-token counts at each element order, and are models trained separately for each order? Since the reference data are interpolated from first-order simulations, what evidence establishes benefits beyond compatibility with the resampled representation? In particular, how does runtime and memory scale when the number of elements—and therefore corner tokens—increases at fixed element order? For a given remeshed sample and underlying corner-node topology, increasing the element order introduces additional higher-order edge and/or interior nodes without changing the corner-node mesh. Consequently, the IMGTT coarse-token count N_c remains unchanged across element orders for the corresponding mesh, while the fine-node count N_f increases substantially. For Flag Dynamics, N_c ranges from 81-138 across the meshes, with a mean of 106. The corresponding N_f values (minimum/maximum/mean) are: 81/138/106 for order 1, 289/493/379 for order 2, 625/1066/819 for order 3, 1089/1857/1427 for order 4, and 1681/2866/2200 for order 5. For Fluid Dynamics, N_c ranges from 100-127, with a mean of 113. The corresponding N_f values are: 100/127/113 for order 1, 352/460/405 for order 2, 756/999/875 for order 3, 1312/1744/1523 for order 4, and 2020/2695/2349 for order 5. For Linear Elastostatics, N_c ranges from 79-172, with a mean of 120. The corresponding N_f values are: 79/172/120 for order 1, 276/637/432 for order 2, 592/1394/937 for order 3, 1027/2439/1636 for order 4, and 1581/3792/2528 for orders 5. Thus, while N_c=N_f for order 1, the mean N_c/N_f ratio decreases to approximately 0.28, 0.13, 0.074, and 0.048 for orders 2-5. We will additionally report the corresponding fine-graph edge counts and N_c/N_f ratios in the revised manuscript. Each model is trained separately for each element order. We therefore do not claim cross-order or cross-resolution generalization without retraining. Regarding benefits beyond compatibility with the resampled representation, the objective of these experiments is not to demonstrate p-convergence or the numerical-approximation advantages of higher-order FEM. Rather, we evaluate whether the higher-order mesh structure can be exploited effectively by the learned simulator. The architecture-tokenization ablation demonstrates that the complete structured tokenization configuration provides a substantial predictive advantage over the evaluated learned-token configuration within the same IMGTT processor. This indicates that the structured representation contributes more than merely enabling compatibility with higher-order mesh inputs. However, the experiment does not isolate the effects of the shape-function weights from the different token counts. In the representative order 3 Fluid Dynamics experiment, holding the IMGTT processor fixed and replacing learned tokenization with the proposed structured tokenization reduces the rollout-all velocity RMSE from 22.98 to 10.91. Conversely, when the same structured tokenization is used with the serial MGN-T processor, the error remains 27.39, compared with 28.93 for learned-token MGN-T. Therefore the performance gain cannot be explained by higher-order compatibility or token count alone as the strongest result is obtained when the structured tokenization is combined with the interleaved IMGTT processor. A further practical advantage is that IMGTT operates directly on the native higher-order mesh without conversion to a first-order graph. Higher-order edge and interior nodes remain in the fine local graph, while only the corner-node representation participates in global transformer attention. The method therefore uses the inductive biases of the higher-order element structure through deterministic shape-function-based aggregation and interpolation while restricting the global-attention to a substantially smaller set of tokens at higher element orders. However, IMGTT is not a fixed-token-budget model. At fixed element order, mesh refinement increases the number of elements and therefore increases N_f, the graph-edge count, and N_c as well. Local message-passing cost grows with the fine graph size, while standard full self-attention over the coarse tokens has O(N_c^2) time and memory complexity. The current experiments characterize the evaluated mesh sizes but do not constitute a fixed-order mesh-refinement study. We will therefore make the node, edge, and token counts explicit and clarify that large-scale refinement at fixed element order, including empirical runtime and peak-GPU-memory scaling, remains outside the scope of the present evaluation.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Architecture-tokenization method Isolation study 1

Architecture-tokenization method Isolation study: We attached the detailed error table and parameter comparison table to the supplementary materials. As we can not attach images or tables here, please refer to the details in the supplementary materials. We would like to highlight that the cited error values for this study are multiplied by 100, similarly to the Fluid Dynamics error tables in the manuscript, in order to observe the error values more easily. The experiment is conducted on the third-order Fluid Dynamics benchmark and evaluates two processor architectures with two tokenization strategies. We selected the order 3 CylinderFlow benchmark because it provides a representative autoregressive prediction task with substantial higher-order mesh complexity while remaining computationally feasible within the review period. The benchmark enables comparison across one-step, 50-step, and full-rollout prediction horizons, allowing the influence of the processor and tokenization strategy on error accumulation to be examined. Order 3 elements contain a substantial number of higher-order nodes, 756-999 fine nodes with mean 875, making this configuration appropriate for evaluating mesh-aware tokenization. All four configurations use the same training and test splits, optimizer, training-step budget, data normalization, noise augmentation procedure, and evaluation protocol, identical to the performance analysis of the manuscript. The learned and structured tokenization variants within each processor retain the same processor depth and layer organization. The remaining differences in token count, trainable parameters, and measured computational costs are reported separately. The experiment evaluates two processor architectures with two tokenization strategies, enabling controlled comparisons in which either the processor architecture or tokenization strategy is held fixed as follows: MGN-T with learned tokenization, IMGTT with structured mesh-aware tokenization, MGN-T with structured mesh-aware tokenization, IMGTT with learned tokenization. The resulting comparison evaluates the effect of changing the tokenization strategy within each fixed processor architecture and the effect of changing the processor configuration while retaining the same tokenization type. This separates the architectural and tokenization contributions more clearly than the original complete-model comparison, while retaining differences in token count and processor capacity that are explicitly reported. Holding the IMGTT processor fixed, replacing learned tokenization with structured mesh-aware tokenization reduces the rollout-all velocity RMSE mean from 22.98 to 10.91, by 52%, the median from 16.61 to 8.693 by 48%, and the 95th percentile from 58.02 to 28.30 by 51%. The standard deviation also decreases from 27.48 to 11.26. These improvements are observed while the parameter count decreases slightly from 1.14M for learned-token IMGTT to 1.13M for structured-token IMGTT, indicating that the improvement cannot be attributed to increased trainable parameter capacity. The same trend is observed for acceleration prediction, where the mean rollout-all RMSE decreases from 0.562 to 0.335 40%, while the improvement is smaller for one-step prediction, 0.051 to 0.048. This further suggests that the principal advantage of structured tokenization becomes more pronounced during long autoregressive rollouts. With structured tokenization held fixed, replacing the MGN-T processor configuration with the deeper, interleaved IMGTT processor reduces the velocity rollout-all RMSE mean from 27.39 to 10.91 by 60%. This supports the importance of the overall IMGTT processing architecture when paired with structured tokens, although the comparison does not isolate the interleaving schedule from differences in processing depth and capacity. The architectural contribution is also visible with learned tokenization. Holding learned tokenization fixed, replacing MGN-T with the IMGTT processor, both configurations use the same 64-token learned representation. This comparison isolates the effect of changing the processor architecture without changing the token budget. This reduces the velocity rollout-all RMSE mean from 28.93 to 22.98, the median from 18.52 to 16.61, the 95th percentile from 64.94 to 58.02, and the maximum from 526.7 to 81.22. Under the same robust trajectory-level extreme-error criterion, the learned-token MGN-T configuration contains one extreme high-error rollout, whereas the learned-token IMGTT configuration contains none. These results indicate that the IMGTT processor configuration improves predictive accuracy and reduces the severity of extreme rollout errors even when both models use learned tokenization. The comparison supports an architectural benefit, although the individual contributions of interleaving, depth, and model capacity remain coupled.

AuthorsOpenReview ↗

Architecture-tokenization method Isolation study 2

In contrast to its effect within IMGTT, structured tokenization produces only a modest reduction in the MGN-T mean rollout-all velocity RMSE, from 28.93 to 27.39 by 5.3%. The median increases from 18.52 to 22.48, and the 95th percentile increases slightly from 64.94 to 65.93, indicating that typical prediction accuracy does not consistently improve. However, the standard deviation decreases from 125.9 to 27.21 by 78%, and the maximum trajectory error decreases from 526.7 to 101.5 by 80.7%. The benefit within MGN-T is therefore primarily associated with a reduction in the severity of extreme rollout errors rather than a systematic improvement across the entire error distribution. The controlled study shows that neither processor capacity nor token count alone is sufficient to explain the observed performance differences across all four configurations. In particular, structured-token IMGTT substantially outperforms learned-token IMGTT despite nearly identical parameter counts, while structured-token MGN-T remains considerably less accurate than structured-token IMGTT despite using the same mesh-derived token counts. These findings suggest that the effectiveness of structured tokenization depends strongly on the processor configuration with which it is combined. Likewise, the structured-token MGN-T variant uses the same mesh-dependent coarse-token definition as the structured-token IMGTT for a given mesh. Since the structured MGN-T and structured IMGTT configurations use the same mesh-derived token counts, their substantial difference in rollout-all RMSE mean, 27.39 and 10.91 respectively, cannot be explained by token count alone. Nevertheless, the learned against structured comparison within each processor also changes the number of tokens, so the isolated contribution of the shape-function weighting at exactly matched token count remains unmeasured. However, the structured-token MGN-T variant rollout-all velocity RMSE remains 27.39, close to the original learned-token MGN-T result of 28.93 and substantially above IMGTT's 10.91. A larger global token set is therefore insufficient to explain the structured token IMGTT performance. The computational comparison leads to the same conclusion. Structured-token MGN-T and structured-token IMGTT have comparable measured training costs, 63.46 and 61.80 ms/step respectively, yet their rollout-all errors differ substantially 27.39 and 10.91. Furthermore, the two structured-token configurations also exhibit relatively similar inference costs, 18.06 ms/step for MGN-T and 20.07 ms/step for IMGTT. The latter is approximately 11% slower per inference step but achieves a 60.2% reduction in mean rollout-all velocity RMSE. This suggests a favorable accuracy-runtime trade-off for the complete IMGTT configuration at the evaluated mesh size. Thus, the IMGTT improvement is not explained by greater per-step training cost. Similarly, structured-token IMGTT achieves substantially lower error than learned-token IMGTT despite having slightly fewer trainable parameters, although its larger structured token set and the associated mappings produce a modest increase in inference cost. Overall, the ablation supports complementary contributions from the structured mesh-aware tokenization and the IMGTT processor configuration. The evaluated IMGTT architecture improves predictive accuracy relative to MGN-T under both tokenization strategies, while structured tokenization produces a particularly large additional improvement within IMGTT. The results suggest that the combination of structured representation and repeated fine-coarse information exchange is especially effective for long-horizon prediction. The full structured token IMGTT configuration achieves the lowest mean, median, and 95th-percentile error across all six evaluated prediction-error settings, including velocity and acceleration, one-step, 50-step and rollout-all errors. The particularly large improvement in long horizon rollout-all velocity prediction, compared with the relatively modest one-step improvements, suggests that the combination of the structured representation and IMGTT processing is especially effective at limiting error accumulation over long autoregressive rollouts. The architecture-tokenization method isolation study provides evidence that the predictive advantage of IMGTT arises from the effective combination of structured mesh-aware tokenization and the proposed fine-coarse processor. With the IMGTT processor held fixed, structured tokenization reduces mean full-rollout velocity RMSE by 52%, with corresponding reductions in the median and 95th-percentile errors, while retaining nearly identical trainable parameter counts. In contrast, introducing the same structured tokenization into MGN-T produces only a modest reduction in mean error, although it substantially reduces the severity of extreme rollout errors.

AuthorsOpenReview ↗

Architecture-tokenization method Isolation study 3

The IMGTT processor configuration also improves predictive accuracy relative to MGN-T when both use learned tokenization, indicating that the complete performance improvement is not attributable solely to the tokenization strategy. Together, these results suggest that the proposed structured representation is particularly effective when integrated with repeated fine-coarse information exchange. The remaining differences in token count, processor depth, and computational cost are explicitly reported. Although the current ablation does not isolate the individual contribution of the shape-function coefficients or the interleaving schedule, it demonstrates that the complete structured-token IMGTT configuration provides the best predictive accuracy among the four evaluated variants, with moderate additional inference cost at the evaluated mesh size.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.