acceptodds
Under review as a conference paper at ICLR 2027

FLARE++: Low-rank attention with attention-synthesized routing

Abstract

Full self-attention is a strong token mixer for PDE surrogates on irregular domains, but its quadratic cost limits its use on large problems. Latent-space attention methods such as PerceiverIO, Transolver, and FLARE (Fast Low-rank Attention Routing Engine) avoid that cost by routing attention among tokens through learned latents. They compress by dot-product matching of the input tokens against learned query tokens or projection weights: once trained, the same learned templates serve every input. We remove this restriction with FLARE++, a low-rank attention architecture with input-conditioned routing queries. FLARE++ uses FLARE's own encoder to map the input tokens to query tokens, which correct the learned queries. The adapted queries then determine how that same input is compressed and redistributed. This preserves FLARE's explicit low-rank factorization and linear complexity, and expresses the complete routing operation with standard scaled dot-product attention (SDPA) calls alone. We also provide a multi-GPU context-parallel implementation that shards input tokens across devices without ever gathering the full token sequence on one of them. FLARE++ reduces FLARE's error by on average across five standard PDE benchmarks, achieving the lowest errors among the efficient models compared. The gains persist on industrial-scale DrivAerML aerodynamics and on Long Range Arena, where average accuracy rises by over fixed-query FLARE.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
low-rank attention, linear attention, latent-space attention, neural PDE surrogates, neural operators, transformers, mesh-based simulation, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DxQYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FLARE++, a low-rank attention architecture that synthesizes input-conditioned routing queries to improve upon FLARE's fixed-query design. Specifically, it introduces an additional attention operation to generate input-dependent query corrections, which are combined with learned fixed queries before the standard gather-and-scatter routing operations. The paper also develops a context-parallel implementation that shards input tokens across multiple GPUs and uses numerically stable distributed softmax reductions to compute attention without gathering the full token sequence on a single device. The method is evaluated on multiple PDE benchmarks, an industrial-scale aerodynamics dataset, and Long Range Arena.

Critical strengths

S1. Practical multi-GPU implementation. The context-parallel implementation increases the practical applicability of the proposed method to large-scale problems. This is particularly relevant to scientific computing workloads involving large meshes.

S2. Broad experimental evaluation. The paper evaluates FLARE++ across multiple PDE benchmarks, an industrial-scale aerodynamics dataset, and Long Range Arena. The comparisons with FLARE and other efficient architectures provide useful evidence of the method's empirical effectiveness across different tasks and problem scales.

S3. Clear presentation and organization. The paper is generally well written and structured.

Critical weaknesses

W1. The main accuracy comparison is not fully controlled for capacity or compute. The paper fixes depth, latent budget, channel width, and backbone, but explicitly allows parameter counts to differ across mixers. FLARE++ has 14–23% more parameters than FLARE. The depth/latent-budget sweeps in Section 5.1 show that FLARE++ remains better than FLARE at matched M and B, but they do not isolate the effect of synthesized routing from the effect of extra parameters and extra step time. A parameter- or cost-matched FLARE control would materially strengthen the claim that the improvement is due to input-conditioned routing rather than added capacity.

W2. Missing comparison with Transolver-3 on full-resolution DrivAerML. The Transolver-3 paper reports training on DrivAerML, yet this paper omits Transolver-3 from its full-resolution DrivAerML comparison because the authors’ 16-bit reproduction attempts diverged and FP32 exceeded their memory budget. Although the paper explains this omission, it would be better to given a comparison with Transolver-3 to fit H100 80G using FP32.

Ref: [1]Han, Dongchen, et al. "Agent attention: On the integration of softmax and linear attention." European conference on computer vision. Cham: Springer Nature Switzerland, 2024.

[2]Liu, Xin, et al. "Catanet: Efficient content-aware token aggregation for lightweight image super-resolution." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors

Q1. Could the authors provide a parameter- or compute-matched comparison against FLARE, for example by increasing FLARE’s depth, channel width, latent budget, or number of fixed-query layers to match FLARE++’s parameter count and/or step time? The current matched-depth/matched-latent comparison is helpful, but it does not fully separate improved routing from added mixer capacity.

Q2. The reproduced results for the Transolver family generally show higher errors than expected. In my own reproductions of the relevant experiments, I did not observe errors of this magnitude. Could the authors explain the discrepancy and provide the configurations and training details needed to diagnose it?

Q3. See weakness W2.

1 comment

We thank the reviewer for the thoughtful review and for recognizing the strengths of the paper. We are happy to provide additional evidence. We discuss novelty and the relation to Agent Attention and CATANet, which the reviewer cites, in our response to Reviewer gEYR.

Parameter-matched comparison

We thank the reviewer for raising the question of capacity- and compute-matched comparisons. We increased the depth of FLARE by 25% (B = 8 to B = 10), which brings its parameter count and compute to those of FLARE++ at B = 8. FLARE++ still outperforms it. Test relative error (%), with parameter counts in parentheses:

FLARE (B = 8)FLARE (B = 10)FLARE++ (B = 8)
Elasticity0.71 (1.09M)0.60 (1.34M)0.38 (1.29M)
Darcy0.86 (1.16M)0.83 (1.42M)0.64 (1.42M)

The full DrivAerML sweep over six depths and three latent budgets, plotted against both depth and training cost, is in Section 4.2 (Figure 5) and Appendix D.4 (Figure 8).

Comparison with Transolver

We thank the reviewer for this suggestion. Transolver-3 does fit on one GPU in FP32. We trained Transolver-3, FLARE and FLARE++ on DrivAerML with the same FP32 recipe (M = 128 latents) and evaluated them on the full test surface meshes. Test MSE (%), mean ± std:

BFLARETransolver-3FLARE++
22.35 ± 0.002.30 ± 0.021.58 ± 0.08
41.20 ± 0.041.15 ± 0.040.91 ± 0.01
80.76 ± 0.010.77 ± 0.010.67 ± 0.02

FP32 training takes 2–3x more time and about 6x more memory than the FP16 runs in Figure 5, which makes it impractical at this scale. We include these runs as a reference only.

Reproducibility of the Transolver family

We reproduce Transolver successfully. Transolver++ has been harder to reproduce, and the FLARE paper [1], the AB-UPT paper [2] and the NVIDIA PhysicsNeMo documentation [3] all report difficulty matching its published results. Transolver-3 [4] reports results only on large-scale problems (DrivAerML, AhmedML and NASA-CRM), trained in FP16 or BF16, and does not report results on the standard PDE benchmarks (Elasticity, Darcy, Airfoil), which are typically run in FP32. However, its official code [5] includes only an FP32 training recipe, and only for DrivAerML. Our Transolver-3 implementation is based on that code. We discuss the reproducibility of each model in Appendix C.3.

Supplementary code and scripts

Transolver, Transolver++ and Transolver-3 are implemented as token mixers in the same backbone as FLARE and FLARE++, in pdebench/models/mixer_backbone.py in the supplementary code. Every standard-benchmark run is launched with out/pdebench/run_flarepp_standard.sh, selecting the model with MIXER=transolver, transolverpp or transolver3 and the task with DATASET.

References

[1] Puri et al., FLARE: Fast Low-rank Attention Routing Engine, 2026.

[2] Alkin et al., AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformers, 2025.

[3] NVIDIA PhysicsNeMo Team, Transformer Models for External Aerodynamics on Irregular Meshes, 2026: https://docs.nvidia.com/physicsnemo/26.03/physicsnemo/examples/cfd/external_aerodynamics/transformer_models/README.html

[4] Zhou et al., Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries, 2026.

[5] Transolver-3 official code: https://github.com/thuml/Transolver-3

Reviewer_gEYROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FLARE++ to address the limitations of fixed learned queries used as compression templates in some latent-space attention methods. Building on FLARE's low-rank encode–decode structure, FLARE++ introduces an additional cross-attention operation to generate input-dependent query corrections, which are combined with the original learned queries through a fixed reference and a learnable gate. This enables adaptive routing while preserving the low-rank structure and linear complexity. The paper also presents a multi-GPU context-parallel implementation for large-scale meshes. Experiments show consistent improvements over FLARE across several PDE benchmarks and Long Range Arena tasks.

Critical strengths

The proposed method is simple and easy to implement, requiring no changes to the original backbone or custom attention kernels.

The paper provides clear mathematical analyses of the low-rank structure, computational complexity, and multi-GPU parallelism.

The experiments cover standard PDE benchmarks, industrial-scale DrivAerML, and the non-PDE Long Range Arena benchmark, along with studies of network depth and latent budget, providing substantial evidence of improvements over FLARE.

Critical weaknesses

1.The methodological novelty is relatively limited, as the main modification is an additional cross-attention operation for input-dependent query adaptation. Its distinction from existing latent query refinement methods is not sufficiently established.

2.The method is primarily developed for FLARE's encode–decode structure, and its applicability to other latent-space attention architectures remains unverified.

3.The ablation studies do not fully isolate the contributions of individual components. Although FLARE++ is compared with Simplified FLARE++, the two variants differ in gating, normalization, and projection design, making it difficult to assess the effect of each component independently.

Questions for authors

1.Could the proposed query synthesis mechanism be extended to Perceiver IO, LNO, or other latent-space attention architectures to evaluate its generality?

2.Could the authors compare attention-synthesized queries with simpler input-dependent query generation methods, such as mean pooling followed by an MLP or FiLM-based conditioning, under comparable parameter and computational budgets?

3.Could the authors compare fixed, learnable, and no-gate variants while keeping the other components unchanged, and report the learned gate values across datasets, layers, and attention heads?

3 comments

We thank the reviewer for the constructive feedback and for recognizing the simplicity of the method, the clarity of its analysis, and the breadth of the experiments. In response to the comment on presentation, we have streamlined the method section (Section 3.2). We address each point below.

Novelty

We agree that the added attention call is simple, and we see this as a strength of the design. To our knowledge, latent-attention operators either use a fixed compression template (Perceiver IO, FLARE's latent queries, Transolver's slice projectors) or carry latents forward from block to block (Perceiver, Luna, Slot Attention), so that one latent state must hold both the compression template and the information being compressed. Agent Attention and CATANet, raised by another reviewer, adapt their latents by pooling or sampling the current tokens, which can discard features of the field that matter for routing. FLARE++ separates the two roles: in every block, it attends to the full input field to synthesize the compression template, and the latents carry only the compressed content. We will add this discussion to Section 2, and we would be grateful for any references the reviewer has in mind.

Mean pooling + MLP and FiLM

We thank the reviewer for raising these important points. Mean pooling can be viewed as a special case of attention-based synthesis with uniform weights: it summarizes the whole field into one vector shared by all M routes, whereas each synthesis seed can attend to a different region. Generating all M queries from the pooled vector needs a C → C·M projection (C = H·D), about C²M parameters per block: 1.05M at C = 128, M = 64, against 8K for the seeds and 16K for the synthesis projection. FiLM instead maps the pooled vector to a per-channel scale and shift of FLARE's fixed queries, which is cheap but moves all M queries together. Both pooled generators cost fewer FLOPs than the synthesis call. On Elasticity, with all other settings unchanged:

Query constructionParamsTest rel. L2 (%)
FLARE (fixed queries)1.09M0.71
Mean pool + FiLM1.36M0.65
Mean pool + C → C·M projection9.48M0.52
Simplified FLARE++ (synthesis call, no gating, normalization)1.36M0.38
FLARE++ (synthesis call with gating, normalization)1.29M0.39

At a comparable parameter count, FiLM has 67% higher error than FLARE++; the full projection uses 7× the parameters and still has 33% higher error. The fairest comparison for the projection, which has no fixed queries, is simplified FLARE++, which also routes with generated queries alone: it reaches 0.38% with one seventh of the parameters. Letting each query attend to the input is both cheaper and more accurate than conditioning on a global summary. In our response to Reviewer x4PX, we also compare with a Luna-style alternative that carries latents across blocks without an extra attention call.

Isolating the gate, reference, and normalizations

We agree that FLARE++ and SimplifiedFLARE++ differ in several components. To isolate them, we added the components of FLARE++ to simplified FLARE++ one at a time on Pipe and Elasticity in FP16. The full step-by-step table is in our response to Reviewer x4PX, under "Stability in FP16".

Synthesis value normalization, N0(v~)\mathcal{N}_0(\tilde{v}), alone removes the FP16 divergence of simplified FLARE++: all three seeds diverge without it, and none with it. It controls the magnitude of the synthesized query: each synthesized query is a convex sum of the normalized synthesis values, so RMS(qd)≤1\mathrm{RMS}(q_d) \le 1 by the triangle inequality (Appendix A.4, Eq. 14). We bound the values rather than normalizing qdq_d itself, because the magnitude of qdq_d reflects how consistently the input supports a routing direction; normalizing qdq_d directly made training unstable in our experiments. The remaining components give mixed results: the stabilizing steps cost some accuracy on Elasticity, which the normalized reference and gate recover, and they make no difference on Pipe.

To compare the gate variants, we trained each query mode with all other components of FLARE++ unchanged (test rel. L2 %, FP32, mean ± std over 3 seeds):

ModeRouting queryElasticity (B = 8)Pipe (B = 2)
Learned gate (FLARE++)N0(qf)+σ(g) qd\mathcal{N}_0(q_f) + \sigma(g)\, q_d0.375 ± 0.0040.419 ± 0.041
Frozen gateN0(qf)+σ(0.25) qd\mathcal{N}_0(q_f) + \sigma(0.25)\, q_d0.365 ± 0.0070.398 ± 0.036
No gateN0(qf)+qd\mathcal{N}_0(q_f) + q_d0.390 ± 0.0040.359 ± 0.014
No referenceqdq_d0.431 ± 0.0330.384 ± 0.023
Gated, no referenceσ(g) qd\sigma(g)\, q_d0.529 ± 0.0500.410 ± 0.013
Reference onlyN0(qf)\mathcal{N}_0(q_f)0.385 ± 0.0170.420 ± 0.019

With the reference in place, the learned, frozen, and removed gate perform similarly, and the best variant differs between tasks. Removing the reference raises the error on Elasticity, while Pipe does not separate the modes beyond its seed spread. Since no setting is best across tasks, we fixed a single configuration, which keeps the reference and gate since they are cheap, and used it for all benchmarks without per-task tuning.

The learned gate rises with depth from its initial value σ(0.25)≈0.56\sigma(0.25) \approx 0.56. Its per-block means are 0.53, 0.67, 0.65, 0.75, 0.71, 0.73, 0.72, 0.67 on Elasticity and 0.70, 0.79 on Pipe, and individual heads range from 0.41 to 1.00. We will add a figure of the learned gates per dataset, layer, and head.

I appreciated the authors' efforts in the rebuttal, though I still held concerns regarding the novelty compared to Flare and other similar backbones, I am willing to slightly increase my score and leave the final decision to AC. thank you.

Reviewer_x4PXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FLARE++ modifies FLARE, a low-rank token mixer for PDE surrogates in which M learned queries per head gather N tokens into M latents and the transposed scores scatter them back, giving a rank-≤M operator per head. FLARE++ adds one attention call: learned seeds attend over normalized input features (keys shared with values) to produce bounded, input-dependent query corrections, which a per-head sigmoid gate adds to normalized learned reference queries. The adapted queries are used for both gathering and scattering, so the operator remains rank ≤ M for each fixed input, costs O(N(4C² + 3MC)) per block, and runs as three fused SDPA calls. The paper also gives token-sharded context parallelism based on the standard log-sum-exp merge (Appendix B, which claims no methodological novelty for it) and uses it for single-pass inference on 8.2M-cell DrivAerML surfaces. Experiments cover five standard benchmarks with all mixers in a shared backbone (Table 1), depth × latent-budget sweeps (Fig. 4), full-surface DrivAerML with a cost–accuracy frontier (Figs. 5 and 8), Long Range Arena (Table 12), an FP32/FP16 sweep (Fig. 7), a simplified-variant ablation (Tables 2–3), and ten seeds per method on Elasticity (Table 11).

Critical strengths
  1. Consistent, well-controlled improvement over FLARE. FLARE++ has lower error on all five benchmarks (12–45%, Table 1), in all 21 (M, B) cells on Elasticity and Darcy (Fig. 4), and in all 18 full-surface DrivAerML configurations (Fig. 8). On Elasticity, ten seeds per method give 0.711 ± 0.081% for FLARE and 0.391 ± 0.021% for FLARE++, with non-overlapping ranges (Table 11). The added parameters and cost are reported: Table 10, 1.2–1.3× step time, 1.18× peak memory, and 1.1–1.2× per full training run. Because FLARE++ reaches a given error with fewer blocks (Fig. 4), the parameter difference alone does not explain the gain.
  2. Practical at scale. The mixer is three fused SDPA calls with no custom kernel. Because tokens interact only through M latents, sharding needs only latent-sized collectives. This enables single-pass inference on full 8.2M-cell meshes, with parallel efficiency above 0.9 on up to four GPUs (Table 5).
Critical weaknesses
  1. The FP16 stability credited to the reference and gate is not isolated, and the decisive benchmark is missing. Section 5.3 credits the fixed reference and gate for precision robustness, and A.8 presents the simplified variant as the ablation that shows this. Per Table 2, however, the simplified variant drops the reference, the gate, and the FP32 blend that comes with them, and also differs in four other ways. It uses unshared synthesis values, has no value normalization (so the correction has no RMS bound), puts a learned scale on the synthesis keys, and does not normalize the routing keys. The FP32/FP16 sweep also switches from LayerNorm to RMSNorm (D.2). Most importantly, the simplified variant fails badly only on Pipe, where its FP16 error is 3.5×, 4.8× and 12.9× its FP32 error at B = 2, 4 and 8. Yet Pipe is not plotted, and FLARE++'s ratio on Pipe is not reported. D.2 limits its conclusion to the three plotted benchmarks. On those, the simplified variant degrades by at most 1.23×, and at B = 8 in FP16 it is nearly tied with FLARE++ on Darcy and Airfoil (Fig. 7). Every FP16 and industrial-scale run uses the anchored, gated design, and stability is the stated reason for it. That claim needs direct evidence or narrower wording.

  2. The need for the extra attention call is not tested against the cheaper Luna-style alternative. The paper separates FLARE++ from Luna by rebuilding the queries in every block through a path separate from value gathering. That separation is what the extra SDPA call buys, and that call is the main added cost. No experiment compares it with carrying latent state across blocks, e.g. correcting block ℓ's queries with the latents gathered in block ℓ−1, at no extra attention cost. Table 1 may not cover this either. Table 9 lists Luna as a mixer in the shared backbone with no deviations. If only X passes between blocks, Luna reduces to the fixed-latent variant its own paper argues against.

  3. Statistical support outside Elasticity, and the scope of the baseline comparison. Every result outside the Elasticity repeats is single-seed (C.2), and FLARE's seed-to-seed coefficient of variation on Elasticity is about 11%. The two full-attention Airfoil numbers above (0.58 vs 0.683) differ by 18%. If they are runs of the same configuration, that run-to-run spread is comparable to the FLARE-to-FLARE++ gaps on Airfoil (12%) and Darcy (18%). It is also larger than the 6–7% gains in the deepest DrivAerML models, which underlie the frontier claim above 3.4 GPU hours. Consistency across the 21 and 18 configurations supports the sign of the effect, but the magnitudes and the frontier claim need uncertainty estimates. In addition, all baselines share one recipe with no per-mixer tuning (C.2). The Transolver family also runs without its convolutions, although the FLARE study reports a significant regression without them on Darcy, Airfoil and Pipe (C.3). Table 1 therefore supports a controlled comparison of token mixers, not superiority over the complete architectures, and the abstract and Section 4.1 should keep that scope.

Questions for authors
  1. What is FLARE++'s FP16/FP32 error ratio on Pipe at B = 2, 4 and 8? Please add Pipe to Fig. 7.
  2. Starting from FLARE++, can you remove only the reference and gate, keeping shared synthesis keys/values, all normalizations, initialization and precision fixed, and report FP32 and FP16 errors on Pipe and one other task? If not, would you limit the stability claim to the full recipe?
  3. Does your Luna mixer pass its latents from one block to the next, as in Ma et al. (2021, eq. 6)? How does FLARE++ compare with a FLARE variant whose block-ℓ queries are corrected by the latents gathered in block ℓ−1, with no extra attention call?
  4. Are 0.58 (Table 1) and 0.683 (D.2) runs of the same configuration, and which one is in Table 1? Could you report at least three seeds for one or two of Darcy, Airfoil and Pipe, and for one deep DrivAerML configuration near the frontier (B = 16 or 20), with uncertainty on the FLARE++ − FLARE difference?
  5. What are the errors of FLARE with FLARE++'s query/key normalizations (A.4)?
2 comments

We thank the reviewer for a careful and constructive review, and for recognizing the controlled improvement over FLARE and the practicality of the method at scale.

Statistical support

Thank you for your comment. We ran three seeds of FLARE and FLARE++ on Darcy, and Airfoil (test rel. L2 %, mean ± std):

FLAREFLARE++
Elasticity, B = 8 (10 seeds)0.711 ± 0.0810.391 ± 0.021
Darcy, B = 80.879 ± 0.1010.661 ± 0.027
Airfoil, B = 80.590 ± 0.0250.495 ± 0.005

Table 1 will report these seed means. For capacity-matched comparisons, please see the parameter-matched comparison in our response to Reviewer DxQY.

Are 0.58 (Table 1) and 0.683 (D.2) runs of the same configuration, and which one is in Table 1?

0.683% is seed 0 of full self-attention on Airfoil in the precision sweep (FP32, B = 8). Seeds 1 and 2 give 0.506% and 0.523%, for a mean of 0.570 ± 0.098%. Table 1's 0.58 is within this run-to-run spread.

Table 1 therefore supports a controlled comparison of token mixers, not superiority over the complete architectures, and the abstract and Section 4.1 should keep that scope.

We agree. Table 1 is an iso-mixer comparison. All models share one backbone and training recipe and differ only in the token mixer. It assesses token-mixing efficacy, not the complete published models. We will state this scope in the manuscript.

Luna and carrying latents across blocks

Our Luna baseline passes its latents from block to block, as in Ma et al. (2021, eq. 6). The carried latents must then encode both the routing template for the next block and the content being routed. FLARE++ separates the two: the routing queries are re-synthesized from the current input, and the latents carry only the routed content. To test this, we trained Luna-style FLARE, whose block-ℓ queries are corrected by the latents gathered in block ℓ−1, with no extra attention call (test rel. L2 %, 3 seeds for the Luna rows):

ModelElasticity (M = 64)Darcy (M = 128)
FLARE0.710.86
Luna-style FLARE0.558 ± 0.0220.712 ± 0.010
Luna0.574 ± 0.0310.703 ± 0.012
FLARE++0.3750.639

Carrying latents improves on FLARE, but Luna-style FLARE (1.10M / 1.18M parameters) remains well above FLARE++ on both datasets.

Stability in FP16

Stabilizing FP16 training on Pipe, where the simplified variant fails, was a main driver in the design of FLARE++. We have added Pipe to Appendix Fig. 7 (test rel. L2 %, mean over 3 seeds):

PipeFP32 B=2B=4B=8FP16 B=2B=4B=8
FLARE++0.420.400.470.460.500.51
FLARE0.460.400.400.580.440.45
Simplified FLARE++0.370.350.3915.59.424.5

The FP16 stability credited to the reference and gate is not isolated [...] That claim needs direct evidence or narrower wording.

We agree that the simplified variant differs from FLARE++ in more than the reference and gate. To locate the source of stability, we added the components of FLARE++ to simplified FLARE++ one at a time, with projections, initialization, and the FP32 blend fixed (FP16, 3 seeds; Na\mathcal{N}_a and N0\mathcal{N}_0 are per-head RMSNorm with and without a learned scale):

StepRouting queryRouting keysPipe, B = 2 (%)Elasticity, B = 8 (%)
Simplified FLARE++SDPA(Na(q~),Na(k~),v~)\mathrm{SDPA}(\mathcal{N}_a(\tilde{q}), \mathcal{N}_a(\tilde{k}), \tilde{v})KKdiverged (3/3)0.454 ± 0.034
+ synthesis value normalizationSDPA(Na(q~),Na(k~),N0(v~))\mathrm{SDPA}(\mathcal{N}_a(\tilde{q}), \mathcal{N}_a(\tilde{k}), \mathcal{N}_0(\tilde{v}))KK0.473 ± 0.0500.473 ± 0.025
+ shared key/valueSDPA(Na(q~),N0(k~),N0(k~))\mathrm{SDPA}(\mathcal{N}_a(\tilde{q}), \mathcal{N}_0(\tilde{k}), \mathcal{N}_0(\tilde{k}))KK0.452 ± 0.0340.529 ± 0.033
+ routing-key normalizationsameNa(K)\mathcal{N}_a(K)0.471 ± 0.0220.501 ± 0.071
+ reference, fixed gateqf+σ(0.25) qdq_f + \sigma(0.25)\, q_dNa(K)\mathcal{N}_a(K)0.504 ± 0.0230.553 ± 0.068
+ normalized referenceN0(qf)+σ(0.25) qd\mathcal{N}_0(q_f) + \sigma(0.25)\, q_dNa(K)\mathcal{N}_a(K)0.469 ± 0.0470.448 ± 0.051
+ learned gate (FLARE++)N0(qf)+σ(g) qd\mathcal{N}_0(q_f) + \sigma(g)\, q_dNa(K)\mathcal{N}_a(K)0.460 ± 0.0070.451 ± 0.061

Synthesis value normalization alone removes the divergence. It controls the magnitude of the synthesized query: each synthesized query is a convex sum of the normalized synthesis values, so RMS(qd)≤1\mathrm{RMS}(q_d) \le 1 by the triangle inequality (Appendix A.4, Eq. 14). We bound the values rather than normalizing qdq_d itself, because the magnitude of qdq_d reflects how consistently the input supports a routing direction; normalizing qdq_d directly made training unstable in our experiments.

The remaining components give mixed results. Simplified FLARE++ is as accurate as FLARE++ where it trains stably (Elasticity), and the stabilizing steps cost some accuracy there, which the normalized reference and gate recover. On Pipe, they make no difference beyond the seed spread. The gate variants in our response to Reviewer gEYR show the same picture: the best variant differs between tasks, and the differences are within the seed spread. We therefore fixed a single configuration, which keeps the reference and gate since they are cheap, and used it for all benchmarks without per-task tuning. We will attribute the FP16 stability to the synthesis value normalization in Section 5.3 and Appendix A.8.

What are the errors of FLARE with FLARE++'s query/key normalizations (A.4)?

All runs below are in FP16:

ModelPipe, test rel. L2 (%)DrivAerML (M = 128), test MSE (%) at B = 2 / 4 / 8
FLARE0.552 ± 0.0202.90 / 1.52 / 0.95
FLARE + N0(qf)\mathcal{N}_0(q_f), Na(K)\mathcal{N}_a(K)0.488 ± 0.0362.09 / 1.21 / 0.87
FLARE++0.460 ± 0.0071.93 / 1.15 / 0.82

Adding FLARE++'s normalizations to FLARE recovers part of the gain, and query synthesis improves on top of them on both tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.