- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FLARE++, a low-rank attention architecture that synthesizes input-conditioned routing queries to improve upon FLARE's fixed-query design. Specifically, it introduces an additional attention operation to generate input-dependent query corrections, which are combined with learned fixed queries before the standard gather-and-scatter routing operations. The paper also develops a context-parallel implementation that shards input tokens across multiple GPUs and uses numerically stable distributed softmax reductions to compute attention without gathering the full token sequence on a single device. The method is evaluated on multiple PDE benchmarks, an industrial-scale aerodynamics dataset, and Long Range Arena.
Critical strengths
S1. Practical multi-GPU implementation. The context-parallel implementation increases the practical applicability of the proposed method to large-scale problems. This is particularly relevant to scientific computing workloads involving large meshes.
S2. Broad experimental evaluation. The paper evaluates FLARE++ across multiple PDE benchmarks, an industrial-scale aerodynamics dataset, and Long Range Arena. The comparisons with FLARE and other efficient architectures provide useful evidence of the method's empirical effectiveness across different tasks and problem scales.
S3. Clear presentation and organization. The paper is generally well written and structured.
Critical weaknesses
W1. The main accuracy comparison is not fully controlled for capacity or compute. The paper fixes depth, latent budget, channel width, and backbone, but explicitly allows parameter counts to differ across mixers. FLARE++ has 14–23% more parameters than FLARE. The depth/latent-budget sweeps in Section 5.1 show that FLARE++ remains better than FLARE at matched M and B, but they do not isolate the effect of synthesized routing from the effect of extra parameters and extra step time. A parameter- or cost-matched FLARE control would materially strengthen the claim that the improvement is due to input-conditioned routing rather than added capacity.
W2. Missing comparison with Transolver-3 on full-resolution DrivAerML. The Transolver-3 paper reports training on DrivAerML, yet this paper omits Transolver-3 from its full-resolution DrivAerML comparison because the authors’ 16-bit reproduction attempts diverged and FP32 exceeded their memory budget. Although the paper explains this omission, it would be better to given a comparison with Transolver-3 to fit H100 80G using FP32.
Ref: [1]Han, Dongchen, et al. "Agent attention: On the integration of softmax and linear attention." European conference on computer vision. Cham: Springer Nature Switzerland, 2024.
[2]Liu, Xin, et al. "Catanet: Efficient content-aware token aggregation for lightweight image super-resolution." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
Questions for authors
Q1. Could the authors provide a parameter- or compute-matched comparison against FLARE, for example by increasing FLARE’s depth, channel width, latent budget, or number of fixed-query layers to match FLARE++’s parameter count and/or step time? The current matched-depth/matched-latent comparison is helpful, but it does not fully separate improved routing from added mixer capacity.
Q2. The reproduced results for the Transolver family generally show higher errors than expected. In my own reproductions of the relevant experiments, I did not observe errors of this magnitude. Could the authors explain the discrepancy and provide the configurations and training details needed to diagnose it?
Q3. See weakness W2.
We thank the reviewer for the thoughtful review and for recognizing the strengths of the paper. We are happy to provide additional evidence. We discuss novelty and the relation to Agent Attention and CATANet, which the reviewer cites, in our response to Reviewer gEYR.
Parameter-matched comparison
We thank the reviewer for raising the question of capacity- and compute-matched comparisons. We increased the depth of FLARE by 25% (B = 8 to B = 10), which brings its parameter count and compute to those of FLARE++ at B = 8. FLARE++ still outperforms it. Test relative error (%), with parameter counts in parentheses:
The full DrivAerML sweep over six depths and three latent budgets, plotted against both depth and training cost, is in Section 4.2 (Figure 5) and Appendix D.4 (Figure 8).
Comparison with Transolver
We thank the reviewer for this suggestion. Transolver-3 does fit on one GPU in FP32. We trained Transolver-3, FLARE and FLARE++ on DrivAerML with the same FP32 recipe (M = 128 latents) and evaluated them on the full test surface meshes. Test MSE (%), mean ± std:
FP32 training takes 2–3x more time and about 6x more memory than the FP16 runs in Figure 5, which makes it impractical at this scale. We include these runs as a reference only.
Reproducibility of the Transolver family
We reproduce Transolver successfully. Transolver++ has been harder to reproduce, and the FLARE paper [1], the AB-UPT paper [2] and the NVIDIA PhysicsNeMo documentation [3] all report difficulty matching its published results. Transolver-3 [4] reports results only on large-scale problems (DrivAerML, AhmedML and NASA-CRM), trained in FP16 or BF16, and does not report results on the standard PDE benchmarks (Elasticity, Darcy, Airfoil), which are typically run in FP32. However, its official code [5] includes only an FP32 training recipe, and only for DrivAerML. Our Transolver-3 implementation is based on that code. We discuss the reproducibility of each model in Appendix C.3.
Supplementary code and scripts
Transolver, Transolver++ and Transolver-3 are implemented as token mixers in the same backbone as FLARE and FLARE++, in
pdebench/models/mixer_backbone.pyin the supplementary code. Every standard-benchmark run is launched without/pdebench/run_flarepp_standard.sh, selecting the model withMIXER=transolver,transolverpportransolver3and the task withDATASET.References
[1] Puri et al., FLARE: Fast Low-rank Attention Routing Engine, 2026.
[2] Alkin et al., AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformers, 2025.
[3] NVIDIA PhysicsNeMo Team, Transformer Models for External Aerodynamics on Irregular Meshes, 2026: https://docs.nvidia.com/physicsnemo/26.03/physicsnemo/examples/cfd/external_aerodynamics/transformer_models/README.html
[4] Zhou et al., Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries, 2026.
[5] Transolver-3 official code: https://github.com/thuml/Transolver-3