- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes IMGTT, a graph-transformer model for physical simulations on higher-order meshes. The method uses element corner nodes as coarse tokens, with finite-element interpolation operators to exchange information between fine nodes and coarse tokens. It combines local graph message passing with global transformer attention. Experiments on elasticity, flag dynamics, and fluid dynamics show better rollout performance than MGN-T in several cases, and lower computational costs than MS-IMGN.
Critical strengths
-
The method provides a relatively simple way to combine local GNN processing and global transformer attention for higher-order meshes, without requiring conversion to first-order graphs.
-
The experimental results show some promising improvements in rollout accuracy over MGN-T, as well as reduced computational costs (around 3x speedup) compared with MS-IMGN.
Critical weaknesses
I find the main technical contribution somewhat limited. The coarse mesh representation and interpolation operators are already introduced in IMGN, and replacing coarse-level message passing with standard transformer attention seems to be a relatively straightforward extension. The combination of local GNNs and global attention has also been widely explored in previous works. Therefore, I am not fully convinced about the novelty and significance of the proposed method compared with existing approaches.
Also, it is difficult to tell whether the improvements over MGN-T actually come from the proposed tokenization, since the models have different numbers of message-passing layers, transformer layers, and parameters. More careful ablation studies with comparable architectures and computational budgets would be helpful.
One major concern is that the number of coarse tokens Nc still depends on the mesh resolution. When the mesh becomes highly refined, Nc can also become very large and may not be much smaller than Nf. For first-order meshes, Nc is essentially equal to Nf. This means the model still has the O(Nc^2) cost of global attention. In comparison, some existing latent-token transformer methods can use a fixed number of tokens regardless of the input mesh resolution, which seems more attractive for large-scale problems. The paper does not provide enough experiments on mesh refinement, resolution generalization, or truly large-scale meshes to demonstrate its scalability.
Another concern is that the higher-order datasets are generated by interpolating first-order simulation solutions, rather than using actual higher-order FEM solutions. I think this makes it difficult to assess whether the proposed method can really capture and benefit from the numerical properties of higher-order discretizations.
In addition, IMGTT does not consistently outperform the stronger MS-IMGN baseline in prediction accuracy, especially for stress prediction and some long-term rollout results. Some baseline results are also directly taken from previous work instead of being obtained under the same training setup. Overall, I feel the current experimental results are not strong enough to clearly support the claimed advantages in accuracy, scalability, and generalization.
Questions for authors
-
What is the main technical novelty compared with IMGN/MS-IMGN, apart from replacing coarse-level message passing with transformer attention? How is the proposed method different from existing local-global GNN/transformer architectures?
-
Can the authors provide more controlled ablation studies to better show the actual benefits of mesh-aware tokenization? For example, comparing deterministic and learned tokenization with similar network architectures, token counts, parameter counts, and computational budgets.
-
How does the number of coarse tokens Nc change with mesh refinement? It would be useful to report Nc/Nf ratios, GPU memory usage, and runtime on much larger meshes. How would this compare with transformer approaches using a fixed number of latent tokens?
-
Can the model generalize to much finer meshes or different mesh resolutions without retraining? Simply changing the element order does not seem sufficient to demonstrate resolution generalization or scalability.
-
Since the higher-order datasets are obtained by interpolating first-order solutions, can the authors provide experiments using actual higher-order FEM solutions? Also, how well can the proposed tokenization and interpolation operators preserve higher-order spatial information and solution details?
7 comments
Quesiton 2
We thank the reviewer for their careful evaluation of our manuscript and constructive comments. We address the raised concerns and provide additional experimental evidence and clarifications below.
Can the authors provide more controlled ablation studies to better show the actual benefits of mesh-aware tokenization? For example, comparing deterministic and learned tokenization with similar network architectures, token counts, parameter counts, and computational budgets. We agree that the original comparison between MGN-T and IMGTT changes both the processor configuration and tokenization strategy, making the contribution of each component difficult to assess independently. To address this limitation, we conducted an architecture-tokenization ablation on the order 3 CylinderFlow benchmark. Four configurations were evaluated, MGN-T and IMGTT, each with learned and structured mesh-aware tokenization. All four models use the same fine graphs, training and test trajectories, and evaluation protocol. The most direct comparison is obtained by holding the IMGTT processor fixed. Replacing learned tokenization with structured tokenization reduces velocity rollout-all RMSE mean from 22.98 to 10.91 by 52.5%, the median from 16.61 to 8.693 by 47.7%, and the 95th percentile from 58.02 to 28.30 by 51%. The parameter count remains nearly identical, 1.14M versus 1.13M, and the processor depth and scheduling are unchanged. The corresponding tokenization comparison within MGN-T produces a smaller reduction in velocity rollout-all RMSE mean, from 28.93 to 27.39 by 5%. However, the standard deviation decreases from 125.9 to 27.21, and the maximum trajectory error decreases from 526.7 to 101.5, indicating reduced extreme-error severity. With learned tokenization held fixed, IMGTT achieves a lower rollout-all RMSE mean than MGN-T, 22.98 versus 28.93. With structured tokenization, this difference increases substantially, 10.91 versus 27.39. These findings indicate that the benefit of structured tokenization is particularly pronounced within the IMGTT processor configuration, supporting the importance of combining the mesh-aware representation with repeated fine-coarse information exchange. The tokenization variants within each processor have nearly identical parameter counts, but their token counts differ. Both learned-token variants use 64 tokens, meanwhile both structured-token variants use 100-127 mesh-derived tokens with mean 113. Furthermore, the IMGTT and MGN-T processors differ in depth, organization, and overall parameter count. Consequently, the experiment supports the effectiveness of the complete tokenization-processor combination but does not independently isolate shape-function weighting, token-count effects, or interleaving from processor depth. Furthermore, within IMGTT, structured tokenization increases inference time from 18.32 to 20.07 ms/step by 10% while reducing velocity rollout-all RMSE mean by 52%. The results therefore demonstrate a substantial accuracy improvement with moderate additional inference cost at the evaluated mesh size.
Question 3
How does the number of coarse tokens Nc change with mesh refinement? It would be useful to report Nc/Nf ratios, GPU memory usage, and runtime on much larger meshes. How would this compare with transformer approaches using a fixed number of latent tokens? For a given mesh topology, increasing the element order introduces additional higher-order edge and/or interior nodes while leaving the corner-node mesh unchanged. Therefore, the IMGTT coarse-token count N_c remains fixed for the corresponding mesh, while the fine-node count N_f increases with element order. For Flag Dynamics, N_c ranges from 81 to 138 across the meshes, with a mean of 106. The corresponding N_f values (minimum/maximum/mean) are: 81/138/106 for order 1, 289/493/379 for order 2, 625/1066/819 for order 3, 1089/1857/1427 for order 4, and 1681/2866/2200 for order 5. Using the mean node counts, the corresponding N_c/N_f ratios are approximately 1.00, 0.280, 0.129, 0.074, and 0.048 by element order from 1 to 5. For Fluid Dynamics, N_c ranges from 100 to 127, with a mean of 113. The corresponding N_f values are: 100/127/113 for order 1, 352/460/405 for order 2, 756/999/875 for order 3, 1312/1744/1523 for order 4, and 2020/2695/2349 for order 5. Using the mean node counts, the corresponding N_c/N_f ratios are approximately 1.00, 0.279, 0.129, 0.074, and 0.048 by element order from 1 to 5. For Linear Elastostatics, N_c ranges from 79 to 172, with a mean of 120. The corresponding N_f values are: 79/172/120 for order 1, 276/637/432 for order 2, 592/1394/937 for order 3, 1027/2439/1636 for order 4, and 1581/3792/2528 for order 5. Using the mean node counts, the corresponding N_c/N_f ratios are approximately 1.00, 0.278, 0.128, 0.073, and 0.047 by element order from 1 to 5. We will additionally report the corresponding fine-graph edge counts in the revised manuscript. These values illustrate the intended higher-order setting. As the polynomial order increases, an increasingly smaller fraction of the fine nodes participates in global attention, while the additional higher-order nodes remain in the local fine-graph representation. IMGTT does not, however, use a fixed-size latent bottleneck. At fixed element order, mesh refinement increases the number of corner nodes and therefore increases N_c together with N_f and the number of graph edges. Standard full self-attention over the coarse representation consequently has O(N_c^2) time and memory complexity, while the storage and computation associated with local message passing scale with the fine graph size and edge count. This differs from fixed-latent-token Transformer approaches, in which the number of global tokens can remain constant as the input mesh is refined. Such methods can therefore maintain resolution-independent global-attention cost, meanwhile IMGTT retains an explicit one-to-one correspondence between coarse tokens and mesh corner nodes. The trade-off is that IMGTT preserves a discretization-aligned structured representation, but its global-attention cost is not independent of mesh resolution. The current experiments characterize the node counts and runtimes of the evaluated meshes but do not constitute a fixed-order mesh-refinement scaling study. We have also not measured peak GPU-memory scaling on substantially larger meshes. We will therefore make this limitation explicit as the empirical runtime and memory scaling under large fixed-order refinement remain to be evaluated separately. For the third-order Fluid Dynamics benchmark used in the ablation, the structured representation contains an average of 113 coarse tokens for 875 fine nodes, corresponding to a coarse-to-fine node ratio of approximately 12.9%. Global self-attention therefore operates on a substantially smaller representation than the full fine graph, although the coarse-token count would increase under mesh refinement at fixed element order.
Question 1
The main technical contribution of IMGTT consists of two coupled design choices. First, the mesh-aware structured tokenization derived directly from the finite-element discretization, and second, the interleaved fine-coarse processor that repeatedly exchanges information between persistent fine-node and coarse-token latent representations throughout the processing. First, IMGTT uses the FEM-derived aggregation and interpolation operators as a deterministic, mesh-aware graph-to-token and token-to-graph interface. The operators themselves are inherited from IMGN/MS-IMGN and are not claimed as novel, but their role is different. In IMGN/MS-IMGN, these operators connect fine and coarse graph representations, both of which are processed using separately local message passing. IMGTT replaces coarse-graph message passing with global self-attention over structured tokens associated with the corner nodes of the higher-order mesh. The mesh-derived fine-coarse relation is retained through the inherited FEM aggregation and interpolation operators, but the coarse representation is processed as a globally interacting token set rather than a locally connected message-passing graph. Second, IMGTT differs from the serial local-global organization used by MGN-T. MGN-T first performs local message passing, then constructs learned latent tokens for global attention, and then maps this information back to the fine representation. In IMGTT, both the fine-node and structured-token representations persist throughout the processor. Each processing cycle performs local message passing on the full higher-order graph, aggregates the updated fine representation into the structured tokens, performs global self-attention over those tokens, and interpolates the resulting global information back to the fine graph before the next local processing step. The local and global representations therefore evolve jointly through repeated fine-to-coarse and coarse-to-fine exchange rather than through a single serial local-global-local pathway. The new architecture-tokenization isolation study supports the importance of both components. Holding learned tokenization fixed, replacing MGN-T with the IMGTT processor, both configurations use the same 64-token learned representation, enabling an architectural comparison without introducing a token-count difference. However, processing depth, schedule, and parameter count also differ, therefore the improvement cannot be attributed exclusively to interleaving. With the learned-token budget fixed at 64, the IMGTT processor configuration reduces the velocity rollout-all RMSE mean from 28.93 to 22.98 by 20.6% and the maximum trajectory error from 526.7 to 81.22 by 84.6%. This indicates an architectural benefit beyond the change in tokenization type, although the comparison does not independently isolate processing depth or interleaving. Holding the IMGTT processor fixed, replacing learned tokenization with the structured mesh-aware tokenization further reduces the error from 22.98 to 10.91. Conversely, introducing the structured tokenization into MGN-T substantially reduces the long-horizon error tail severity, although it produces only a modest reduction in its mean error. The strongest performance is obtained only when the structured tokenization and interleaved IMGTT processor are combined. These results also make the distinction from a simple increase in model capacity or token count clearer. The learned-token IMGTT uses the same interleaved IMGTT processor but remains substantially less accurate than the structured-token IMGTT, therefore the improvement cannot be explained by processor depth or capacity alone. Moreover, learned-token IMGTT has slightly more trainable parameters than structured-token IMGTT, 1.14M versus 1.13M, and a slightly higher measured training cost, 64.51 versus 61.80 ms/step. The improvement from structured tokenization therefore cannot be attributed to additional parameter capacity or greater training computation. Likewise, structured-token MGN-T uses the same mesh-dependent structured token set as IMGTT but remains close to the original MGN-T in mean rollout error, showing that a larger or mesh-dependent token set alone is also insufficient to explain the improvement. We therefore do not claim novelty in Transformer attention, local message passing, or the FEM interpolation operators individually. The contribution lies in using the higher-order finite-element discretization to define a structured, parameter-free Transformer token interface and coupling this representation repeatedly with the persistent fine-graph representation through the interleaved IMGTT processor. We will revise the manuscript to make these two coupled contributions and their distinction from IMGN/MS-IMGN and serial local-global Transformer architectures clearer.