acceptodds
Under review as a conference paper at ICLR 2027

LINK: Learning Physical Interaction Kernels for Neural Operators

Abstract

Neural operators on irregular domains require interactions that remain meaningful as discretization and geometry change. Attention provides a flexible mechanism for exchanging information between sampled locations, but its routing is typically determined through generic query–key interactions rather than directly from physical structure. We introduce LINK (Learned Interaction Kernels), a kernel attention mechanism that learns physical-space routing between locations using their coordinates, relative geometry, and the current problem instance. The resulting input-conditioned kernel integrates naturally into neural operator architectures while retaining flexible point-set computation. We characterize the resulting operator from continuum and representation perspectives, and develop scalable realizations for large discretizations. Across several PDE learning settings, LINK improves predictive accuracy and optimization, with the largest gains arising when interactions are nonlocal. These results show that learning routing directly in physical space provides an effective inductive structure for neural operators on irregular domains.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, operator learning, partial differential equations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KoxYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Learned INteraction Kernels (LINK) which replaces the query/key pair dot product score of the attention by an MLP on the pair, this MLP takes the latent states of both points along with absolute and relative geometry information (absolute position of both points, their relative distance) ; a Gaussian locality prior is also added to this score. It first shows how this formulation is consistent over different discretizations and how a joint score is more expressive than its bilinear counterpart, to then show how LINK's Jacobian aligns with the Green operator of the Poisson equation, as well as its superiority over other baselines on Darcy flow.

Critical strengths
  • The correlation study with Poisson's known Green operator is a good idea to assert the physical relevance of the learned mapping.
Critical weaknesses
  • It is unclear whether the proposed approach gains from the replacement of the bilinear score function by the pair network or the addition of geometric information to the conditioning of this network. The only comparison with a bilinear alternative is the Transformer, but it differs in size in the main results, and seem to only have its width changed on the capacity control, with a maximum depth of 8 while LINK can go up to 24 ; overall it is unclear whether the implementation only differs in the pair network (especially with the memory cost differences between LINK and Transformer in Table 6). The other baselines do not condition on geometric information, nor use RoPE, which could serve as such. I would be curious to see the comparison with the same LINK backbone but a bilinear score (with the same geometric inputs) at a similar model size, and/or an ablation of LINK without the geometric information.

  • One of the main challenges of operator learning on irregular meshes and point clouds is the computation budget (especially due to the n2n^2 of considering each pair of points) ; this approach seems to add a non-negligible computation overhead as seen in the inference times of Table 6 ; the quicker/lighter version LINK-Landmark introduced to address this issue is not as competitive and as highlighted for its accuracy as the Dense or Shared versions, it also seems to be tied to a fixed number of points as it needs to learn the gather Γ∈Rm×n\Gamma \in \mathbb R^{m\times n} and redistribution Λ∈Rn×m\Lambda \in \mathbb R^{n\times m} matrices.

  • Property 2, which justifies the relevance of the pair network over the bilinear form, is proved upon the assumption of a fixed width rr and of its eventual inferiority to the rank of the operator it is meant to approximate, but it is not experimentally verified that the rank or singular values of the operators encountered pose a problem with the width considered in practice. Also, couldn't "given sufficient width" justify that a bilinear form can "approximate any strictly positive continuous normalized kernel", such as "given sufficient MLP capacity" (l260) does ?

Questions for authors
  • l273: what does "the same physical tokens" mean ? does it mean both z(t)z(t) and tt or only z(t)z(t) ?

  • how does LINK perform with a bilinear backbone ? How does it perform with its joint score applied only to the latent states (position encoding can be added to those states) ?

  • the routing-Green correlation is never reported for the Transformer, how does it perform ? (I would be interested to see a corr(P,G)\text{corr}(P, G) column in Table 1)

  • in Table 5, how do you explain the differences in results with the Elasticity column of Table 2 from the Transolver paper, which compares against the same baselines (Galerkin, GNOT, Transolver) ?

8 comments
AuthorsOpenReview ↗

Authors Responses (Part 1)

Thank you for the careful review and for the comparisons you proposed. We are glad that you found the correlation study with the known Green operator of Poisson a good idea to assert the physical relevance of the learned mapping. Below, we address your comments. We hope you find our responses satisfactory, and that you will consider revising your score.

In short: your review asks which part of LINK produces its improvement, and at what cost. Following your comments, we (1) trained the four combinations of a bilinear or joint score, with or without the pair geometry, including the joint score on the latent states (Table 29); (2) evaluated LINK-Landmark across point counts with the same weights and compared it with GNOT up to 16,384 points (Tables 35 and 36); (3) computed the singular values of the target kernels behind Proposition 2 (Table 28); (4) reran the comparison of Table 1 with three seeds and added the routing–Green correlation of the Transformer (Table 27); and (5) expressed the elasticity results in the published metric (Table 32).

Regarding the roles of the pair network and of the geometric inputs (W1, Q2). Thank you for asking which of the two changes produces the improvement. Each of them reduces the error of bilinear attention by 32%. To address your question directly, we trained the four combinations on PDEBench Darcy at 32×32 with 1,000 training samples, with the same width, depth and number of heads and three seeds; the model you proposed has 406,657 parameters and LINK 406,273 (Table 29):

Model (Table 29)ScorePair geometryTest rel-L2
Transformerbilinear✗0.0665±0.0005
Transformer + geometric term (your suggestion)bilinear✓0.0449±0.0017
Pair MLPjoint✗0.0454±0.0006
LINKjoint✓0.0479±0.0005

Both changes make the score a joint function of the two points, outside the fixed-width bilinear family of Proposition 2: the pair network reads the coordinates from the latent states, to which the lift adds them, and the geometric term receives them explicitly. LINK combines the two in one pair network. Therefore, scoring the two points jointly, beyond the fixed-width bilinear form of Proposition 2, gives the improvement over bilinear attention. LINK does so with one pair network on the latent states and the geometry. Following your suggestion, Section 5.2 now states that either ingredient reduces the error of bilinear attention by 32% on PDEBench Darcy, and Appendix N.1 reports Table 29.

AuthorsOpenReview ↗

Authors Responses (Part 2)

Regarding the size and the depth of the Transformer (W1). We appreciate the question on capacity and depth. Table 29 uses the same width, depth and number of heads for every score, and the sweep of Table 18 compares a 39.39M Transformer with a 40.60M LINK on scattered Darcy:

Scattered Darcy (Table 18)ParametersStepsTest rel-L2
Transformer39.39M384,0000.0508
LINK40.60M24,0000.0334

At a common 24,000 steps, with warmup and three seeds, the same pair reaches 0.0332±0.0004 for LINK and 0.0543±0.0011 for the Transformer (Table 31).

On depth, the shared hyperparameter space ends at depth 8 for every family, and the selection chose depth 6 for the Transformer on the grid and depth 4 on scattered points (Table 13), below that upper end. The depth extension to 24 layers is used by the grid LINK of Tables 3 and 4 and by the deeper LINK-Shared of Tables 17 and 25. Following your comment, Appendix N.1 now lists the depth selected for every model of Table 13.

Regarding the implementation and the memory in Table 6 (W1). Thank you for checking the implementations. The two models share the lift, the residual blocks, the value maps, the feed-forward blocks and the readout, and differ in the score. In Table 6, LINK-Dense computes the routing in exact query blocks (Appendix K), whereas the Transformer of that probe stores its n×n attention matrices. Section 5.3 now states this difference next to Table 6.

Regarding geometric conditioning of the other baselines, and rotary position embeddings (W1). We appreciate the suggestion of rotary position embeddings (RoPE) [1]. Galerkin, GNOT and Transolver receive the coordinates as input features, as the Transformer does. RoPE gives a bilinear score relative positions through a rotation of its queries and keys; the model you proposed gives it the absolute and relative geometry of the pair explicitly, and with it the bilinear score reaches 0.0449±0.0017 on PDEBench Darcy, compared with 0.0665±0.0005 without it (Table 29). In the revision, Section 2 cites RoPE as a relative-position encoding for bilinear scores.

AuthorsOpenReview ↗

Authors Responses (Part 3)

Regarding the computational cost and LINK-Landmark (W2). Thank you for raising this point, which decides where pairwise scoring is practical. In addition to the training-step times of Table 6, Table 38 now reports the inference latency of the baselines and of every variant of LINK. Two variants of LINK reduce the cost. LINK-Shared computes the pair-network contribution once and reuses it in every layer; at 1,024 points it has a lower error than the Transformer in less training time on both discretizations, in single precision, as in every comparison of the submission (Table 4):

Darcy (Table 4)StepsTest rel-L2Training time (s)
Transformer, grid192,0000.0224±0.00134,744
LINK-Shared, grid24,0000.0174±0.00033,417
Transformer, scattered384,0000.0496±0.00042,550
LINK-Shared, scattered12,0000.0407±0.00122,154

LINK-Landmark evaluates the pair score on point–landmark pairs. The routing matrices Γ ∈ ℝ^(m×n) and Λ ∈ ℝ^(n×m) are computed by the pair network for the current point set and then normalized, rather than stored as parameters, and the landmarks have learned locations (Appendix F); a change of n therefore changes the shape of Γ and Λ and no parameter. Inspired by your comment, we also added a result that shows this property directly: one LINK-Landmark (m = 32, 493,281 parameters), trained on our Darcy data on a 64×64 grid at 1,024 of its 4,096 points, keeps the same weights at every point count, with three seeds (Table 35):

Points (Table 35)2565121,0241,024 off grid2,0484,096
LINK-Landmark, same weights0.1102±0.00590.0906±0.00520.0806±0.00500.0829±0.00530.0753±0.00510.0730±0.0047

With the same weights, the error decreases from 0.1102 at 256 points to 0.0730 at 4,096 points.

On accuracy at scale, LINK-Landmark has a lower error than GNOT on Darcy solutions computed on a 128×128 grid and sampled at 4,096 and 16,384 points, with three seeds (Table 36):

Model (Table 36)4,096 points16,384 points
GNOT0.1833±0.00500.2397±0.0059
LINK-Landmark0.1185±0.00910.1705±0.0034

On PDEBench Darcy at its 128×128 resolution, LINK-Landmark reaches 0.0653±0.0079, compared with 0.0932±0.0022 for FNO (Appendix P). Section 3.2 now states that Γ and Λ are computed for the current point set, and Appendix O.1 reports Tables 35 and 36.

Regarding the role of Proposition 2 (W3). We appreciate this question on what the fixed width adds. With unbounded width, a bilinear head is also universal, because finite sums of products q(x)ᵀk(y) are dense in the continuous functions on Ω×Ω. Proposition 2 concerns a fixed head width r: a single head of width r has an error of at least min{η/2, η s_{r+1}/4}, where η is the smallest entry of the target and s_{r+1} is the (r+1)-th singular value of its row-centered log-kernel. The bound is therefore informative when more singular values are large than the head width of the Transformer, 12 in our Poisson models. To address your question directly, we computed them for the row-normalized discrete Green operators of Poisson, for κ from 1 to 32, and of four Darcy samples (Table 28):

Target kernel (Table 28)Singular values above 1% of the largest
Poisson, κ = 140
Poisson, κ = 240
Poisson, κ = 440
Poisson, κ = 837
Poisson, κ = 1631
Poisson, κ = 3229
Darcy, four samples36, 36, 34 and 37

Every target has more singular directions above this threshold than the head width of 12, the regime in which Proposition 2 separates a single bilinear head from the joint pair score. Following your comment, Section 4.2 now states that a bilinear head of sufficient width is universal and that the proposition concerns a fixed head width, and Appendix M.2 reports Table 28.

Regarding "the same physical tokens" (Q1). Thank you for pointing to this phrase. It means both: the tokens of the Transformer are the lifted states z(t), and the lift receives the input values at t together with the location t (Section 3.1). The score of the Transformer has no explicit pairwise term, for example t−t′ or ‖t−t′‖: it is the inner product of a query embedding of z(t) and a key embedding of z(t′). The revised Section 5 states that the Transformer receives the same tokens, which contain the input values and the coordinates.

Reviewer_c2QwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LINK, an attention mechanism for neural operators that learns scalar interaction scores jointly from the hidden states and geometry of two spatial locations. The scores are normalized into routing weights, while a separate value map determines the information transported. The design is motivated by the interaction structure of Green operators. The paper provides two layer-level theoretical properties and evaluates the method on Poisson, Darcy flow, and elasticity.

Critical strengths
  1. The motivation from physical interaction kernels is reasonable. I view the main contribution as the design of the joint pair-scoring kernel and its integration into normalized attention. Conditioning the score on both endpoint states and their absolute and relative geometry provides a natural way to represent instance-dependent spatial interactions.

  2. The properties in Sections 4.1 and 4.2 provide useful insights. Proposition 1 explains consistency under refinement when the point sets represent the same limiting sampling measure. Proposition 2 identifies a restriction on the centered log-kernels represented by a single fixed-width bilinear head. These results help explain the construction and its representational motivation, within their stated layer-level scope.

Critical weaknesses
  1. The scope of applicability to nonlinear equations needs further clarification. The Green-function motivation is most direct for equations that are linear in the solution field, including the Poisson and Darcy settings considered here. The elasticity benchmark provides evidence on nonlinear hyperelasticity, but the broader applicability to nonlinear PDEs remains unclear. What classes of equations should we expect LINK to benefit most, and what limitations are anticipated? For nonlinear problems, how should the learned interaction kernel be interpreted—for example, as an input-dependent local response or as a more general communication mechanism? A clearer discussion grounded in the existing elasticity experiment would help. A focused study varying the strength of nonlinearity, if feasible within the current setup, could further support the claimed scope.

  2. The evidence for generalization across geometry and topology is limited. Most experiments use fixed square domains. Changing the sampling points or their density tests discretization transfer, while the elasticity benchmark introduces boundary-shape variation within a family of domains containing one central hole. Can the same learned pair-network parameters generalize to substantially different geometries or topologies without retraining, while allowing the routing weights to adapt to each input? A targeted evaluation on a held-out geometry regime or a change in topology would help establish this capability. If such transfer is outside the intended scope, the paper should state that limitation explicitly.

  3. The complete architecture needs a clearer specification. Section 3 defines the attention score and outlines Dense, Shared, and Landmark routing. Appendix F and Table 13 provide useful configuration details, and Appendix K explains implementation choices. However, these descriptions do not provide a complete specification of the network used in the experiments. Please provide a block diagram or pseudocode showing how lifting, attention, normalization, residual connections, pointwise feed-forward processing, and readout fit together. Important details include the pair MLP’s depth and activation, which hidden states generate the cached scores in LINK-Shared, and how landmark features enter the joint pair network in LINK-Landmark. A compact mapping from the reported experiments to their architectural variants would also improve clarity and reproducibility.

  4. The computational cost and accuracy–efficiency tradeoff need further discussion. Dense joint pair scoring remains quadratic in the number of points, with substantial overhead compared with efficient attention methods. At 16,384 points, Table 6 reports 24,701 ms per training step for LINK-Dense versus 24.9 ms for GNOT. LINK-Landmark reduces this cost substantially, but remains approximately ten times slower per step than GNOT in that probe. I acknowledge that the paper includes efficient attention baselines, training-time comparisons, and Shared and Landmark variants. It would nevertheless be helpful to extend the existing analysis with a direct accuracy–cost comparison between LINK-Landmark and GNOT on the same large-scale task, including training time, inference latency, and memory. How much of the Dense model’s accuracy and Green-function alignment is retained as the landmark count is reduced? This would clarify when the additional flexibility of joint pair scoring offers a favorable tradeoff over linear attention.

Questions for authors

Please see weaknesses

8 comments
AuthorsOpenReview ↗

Authors Responses (Part 1)

We thank you for your detailed and positive review. We are glad that you found that the motivation from physical interaction kernels is reasonable, that conditioning the score on both endpoint states and their absolute and relative geometry provides a natural way to represent instance-dependent spatial interactions, and that the properties in Sections 4.1 and 4.2 provide useful insights. Below, we address your comments. We hope you find our responses satisfactory, and that you will consider revising your score.

In short: your review asks where LINK applies and how it is built. Inspired by your comments, we (1) varied the strength of a cubic nonlinearity (Table 33) and related the routing and the Jacobian to the linearized Green operator (Appendix N.6); (2) evaluated trained models on a held-out boundary shape and a held-out topology (Table 34); (3) added pseudocode, a block diagram of the network and the variant of LINK used in each experiment (Algorithm 1, Figure 4, Table 26); and (4) compared LINK-Landmark with GNOT up to 16,384 points, with training time, inference latency and memory, and varied the number of landmarks (Tables 36 and 37).

Regarding the applicability to nonlinear equations (W1). We appreciate this question, which helped us state the scope of the Green-function reading. LINK keeps its advantage when the equation is nonlinear in the solution. Following your suggestion, we varied the strength of the nonlinearity within the current setup, with the cubic Darcy family −∇·(a∇u) + λu³ = 1 at five values of λ, ordered by the violation of superposition of the solution operator, the norm of u(f₁+f₂) − u(f₁) − u(f₂) relative to that of u(f₁+f₂), averaged over six pairs of forcings f₁ and f₂ drawn as Gaussian random fields, with three seeds per level (Table 33):

Cubic Darcy, three seeds (Table 33)λViolation of superpositionTransformerLINK
Level 100.00010.0649±0.00490.0424±0.0026
Level 210²0.03310.0575±0.00290.0375±0.0003
Level 310³0.11900.0473±0.00360.0288±0.0014
Level 410⁴0.23490.0373±0.00190.0238±0.0012
Level 510⁵0.31540.0285±0.00200.0177±0.0010

The error of LINK is 35–39% below that of the Transformer at every level.

On the interpretation, the two readings you propose correspond to two objects that Section 3.2 separates. For a nonlinear equation N(u; a) = f, the response of the solution to a change of the forcing is given by the Green operator of the equation linearized at the current solution, which for the cubic family is the Green operator of −∇·(a∇·) + 3λu², an invertible operator for λ ≥ 0; for a model that receives the forcing, as on Poisson in Section 5.1, the end-to-end Jacobian J = ∂û/∂f estimates this input-dependent local response. The routing P of a layer is a communication mechanism: it is row-stochastic and unsigned, and the network composes it with the value maps, the residual branches and the feed-forward blocks to build that response.

Elasticity, the nonlinear benchmark of the paper, fits this reading: its constitutive law is nonlinear, its input is the geometry of the point cloud, and the latent states that condition the score encode that geometry; at a common budget, LINK reaches 0.0114, compared with 0.0150 for GNOT, the closest baseline (Table 19). The classes of equations that this reading covers are those in which the solution at a point depends on distant inputs through a smooth kernel, as in elliptic and diffusion-dominated problems; problems dominated by shocks are outside the experiments of the paper. In the revision, Section 5.2 reports this study, Section 6 and Appendix N.6 state these classes of equations, and Appendix N.6 reports Table 33.

AuthorsOpenReview ↗

Author Responses (Part 2)

Regarding generalization across geometry and topology (W2). Thank you for proposing this targeted evaluation. Following your suggestion, we trained one model of each score on three families of domains (square, notch and slab) and evaluated it with all parameters fixed on two held-out families, an L-shape (a new boundary shape) and a perforated domain (a new topology), with three seeds. Grid spacing, coefficient fields, forcing, solver and metric are the same, and the routing is recomputed for every new point set (Table 34):

Test rel-L2, three seeds (Table 34)SquareNotchSlabL-shape (held out)Perforated (held out)
LINK, absolute and relative position0.0728±0.00540.0618±0.00480.0580±0.00210.6356±0.03180.7006±0.0102
LINK, relative position0.0776±0.00760.0668±0.00480.0681±0.00310.0892±0.00590.1435±0.0279

The measurement answers your question in two steps. With absolute position, as in the paper, the error on the held-out domains is 9.9 times (L-shape) and 10.9 times (perforated) the mean error on the training domains: absolute position ties the learned model to the training domains. Without absolute position, that is, with only relative geometry in the score and no positional encoding of the tokens, a choice of inputs within the same construction and with the same number of parameters, the same weights transfer: the error on the held-out domains is 1.26 times (L-shape) and 2.03 times (perforated) the mean error on the training domains. The revised Section 6 states that, with relative geometry in place of absolute position, one trained model transfers to a held-out boundary shape and a held-out topology, and Appendix N.7 reports Table 34.

AuthorsOpenReview ↗

Authors Responses (Part 3)

Regarding the architectural details (W3). Thank you for this request, which improves the reproducibility of the paper. We added pseudocode of the network (Algorithm 1), a block diagram (Figure 4) and the variant of LINK used in each experiment (Table 26). The pseudocode is as follows:

z = Lift(v(t), t)                      # pointwise; input values v (a, f or both) and location -> R^d
for l = 1..L:                          # pre-norm residual block
    h = LayerNorm(z)
    s = phi_l([h_i, h_j, t_i, t_j, t_i - t_j, |t_i - t_j|])   # pair network, H logits per pair
    for head k = 1..H:
        logit_k = s[k] - |t_i - t_j|^2 / (2 sigma_{l,k}^2)    # locality prior when used, learned bandwidth
        P_k     = softmax over j of logit_k                   # row-normalized routing
        o_k     = P_k (V_{l,k} h)                             # values transported by P_k
    z = z + W_l [o_1, ..., o_H]
    z = z + FFN_l(LayerNorm(z))
u = Readout(z)                         # pointwise

The pair network is a two-layer MLP, Linear(2d+3c+1 → dφ), LayerNorm, GELU, Linear(dφ → H), with one hidden trunk for all heads and one logit per head; for d = 64 and c = 2 its input has 135 entries. In LINK-Shared, the pair-network contribution is computed once per forward pass, from the normalized hidden states that enter the first routing layer and from the coordinates, and every layer reuses it with its own bandwidths and value maps. In LINK-Landmark, each of the m landmarks has a learned location and a learned feature vector, which do not depend on the input, and the pair network receives them in place of the location and the hidden state of a point at both the gather and the redistribution step; Γ holds the scores of (landmark, point) pairs, normalized over the points, Λ holds those of (point, landmark) pairs, normalized over the landmarks, both computed by the same pair network, and the layer returns Λ(ΓV).

Experiment (Table 26)Variant
Overview (Figure 1)LINK-Dense and LINK-Shared
Poisson diagnostics (Tables 1, 10 and 20–24, Figure 2)LINK-Dense
Darcy ablations (Tables 2 and 11)LINK-Dense
Darcy comparisons and hyperparameter selection (Tables 3, 15 and 16)LINK-Dense
Fixed-operator Darcy (Tables 3 and 12)LINK-Dense
Depth sharing (Tables 4 and 25, Figure 3)LINK-Shared and LINK-Dense
Discretization transfer (Table 5)LINK-Dense
Cost on Poisson (Table 6)LINK-Dense and LINK-Landmark
Large Darcy (Table 7)LINK-Landmark, m = 32
Convergence ladders (Table 17)LINK-Dense and LINK-Shared
Capacity sweep (Tables 18 and 31)LINK-Dense
Elasticity (Tables 19 and 32)LINK-Dense
Additional Darcy and Poisson results (Tables 27, 29, 30, 33 and 34)LINK-Dense
Point counts, comparison with GNOT, and landmark counts (Tables 35–37)LINK-Landmark, with LINK-Dense as the reference in Table 37
Cost (Table 38)LINK-Dense, LINK-Shared and LINK-Landmark
PDEBench at 128×128 (Table 40)LINK-Landmark
The Well and 3-D Darcy (Tables 41 and 42)LINK-Dense

Appendix L now contains Algorithm 1, Figure 4 and Table 26, and Section 3.2 points to them.

Reviewer_jb9kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LINK, an alternative to the attention mechansim for attention based neural operator models. While standard Transformer attention pairs data points based purely on abstract features, LINK directly utilizies exact coordinates, relative distances, and spatial directions to decide how points influence one another. Backed by suitable mathematical proofs they show that it works reliably across different mesh resolutions on a few benchmarks, often beating out the pure softmax baseline across various problems.

Critical strengths
  • Well motivated study to understand if better learnable routing could help
  • Clear study comparing learned operator against the exact greens function providing a good source of iterpretability
  • Good ablations with extensive details provided in the appendix
  • Demonstrates discretization transfer
Critical weaknesses
  • I'm not certain if the comparisons are entirely fair and might benefit from further experimentation / clarification. For instance, in table 3 LINK uses more parameters, LINK receives an MoE block that other baselines don't have.
    • Table 18 seems does seem to discuss that this is capacity matched. But it is a little odd that the transformer barely moves with size and indicates it might be related to optimization. Further restricted to a single seed.
    • Can you possibly report FLOPS, parameter size, wall-clock time and peak memory for all methods.
    • Further might be helpful if you have logs of evolution of train / val curves across different sizes / lrs / iterations
    • Give every family the same architecture options (MoE FFN, multiscale etc.) or remove them from LINK
    • I'm a little surprised to see the instability of the transformer (collapose to predicting the mean in table 14). Perhaps might benefit from better optimization choices.
    • Were your comparisons incorporating flash-attention for the baselines? I believe that might change some number of the memory
  • Benchmarks are almost entirely self-generated and small, and the standard benchmark is not compared with published results (apart from Elasticity)
    • Please repeat your results using established PDE datasets: PDEBench, TheWell. These have good baselines to compare against as well due to a lot of papers having tested on them (not asking to beat all but its a good reference point) with fairly tuned hyperparameters
    • There is no time-dependent, 3-D or genuinely irregular-geometry problem with varying meshes. Given the fact that the authors frame the method to work on irregular domains, I'd think this necessary. Given this framing, a comparison to UPT, Transolver style models would also be useful
    • Further, easticity doesn't seem to correspond to the same numbers reported in the original transolver / gnot papers (which seem lower; although they might be training for further steps). Is this reported 30k steps a converged state?
  • Lot of variability in choice of number of seeds used across experiments: Seems like 3 seeds (stated default) isn't always used. Some other tables in the Appendix use one. Table 1 uses 2 seeds (but still states an SD which might not be informative)
  • In Table 3, the authors mention " each model at the budget with its lowest measured error". Can you clarify this? Was this against a validation set or on the direct test set?
Questions for authors

Please see weaknesses

9 comments
AuthorsOpenReview ↗

Authors Responses (Part 1)

Thank you for the detailed and constructive review. We are glad that you found this a well motivated study to understand if better learnable routing could help, the comparison of the learned operator with the exact Green function a clear study, the paper to have good ablations with extensive details provided in the appendix, and that it demonstrates discretization transfer. Below, we address your comments. We hope you find our responses satisfactory, and that you will consider revising your score.

In short: your review asks whether the comparisons are fair and whether the results hold on public benchmarks with published baselines. Following your suggestions, we (1) removed the MoE block and the multi-scale locality from LINK (Table 30); (2) retrained the 39.39M Transformer of Table 18, the size matched to LINK, with learning-rate warmup and three seeds (Table 31); (3) measured parameters, FLOPs, time and memory of the baselines and of every variant of LINK, and the Transformer with FlashAttention (Tables 38 and 39); (4) added training and validation curves across sizes, learning rates and steps (Figure 5); (5) evaluated PDEBench Darcy at its 128×128 resolution with its published baselines as reference points, The Well, and 3-D Darcy on two grids (Tables 40–42); (6) expressed the elasticity results in the published metric and discussed UPT (Table 32, Section 2); and (7) reran the comparison of Table 1 with three seeds (Table 27).

Regarding the fairness of the comparisons: parameters and architecture options (W1, W1.4). Thank you for raising the fairness of Table 3, which compares each family at its configuration selected on validation data. The comparisons at equal size are the capacity sweep of Table 18, the comparisons of Table 2, and Tables 29 and 30; the attention models on PDEBench and The Well share one width, depth and number of heads. Following your suggestion to remove the MoE feed-forward block and the multi-scale locality from LINK, we compared LINK without them with a Transformer of a similar size on the fixed-operator Darcy task, in configurations built for this comparison, at 6,000 steps with three seeds (Table 30):

Fixed-operator Darcy, 6,000 steps (Table 30)ParametersTest rel-L2
Transformer400,0650.0470±0.0036
LINK with the MoE block and multi-scale locality1,201,9450.0348±0.0011
LINK without the MoE block and multi-scale locality406,2730.0364±0.0015

Without the two options, the error of LINK is 23% below that of a Transformer with a similar number of parameters, and within 5% of the configuration with them. In the revision, Appendix N.2 reports Table 30 and states that no option specific to LINK is used in the comparisons of Appendix P.

AuthorsOpenReview ↗

Authors Responses (Part 2)

Regarding the capacity sweep and the optimization of the Transformer (W1.1). We appreciate the attention to Table 18. In the submission, the learning rate selected for the Transformer decreases with its size, from 3×10⁻⁴ at 0.45M parameters to 6×10⁻⁵ at 39.39M. To address your question directly, we retrained the 39.39M Transformer of Table 18, the size matched to LINK, and LINK of the matched pair, in the protocol of that sweep (learning rate selected on validation data, weight decay 10⁻⁵), with a linear learning-rate warmup and three seeds; the runs without warmup are the single-seed runs of the sweep (Table 31):

Scattered Darcy, sweep of Table 18 (Table 31)WarmupStepsTest rel-L2
Transformer 39.39Mno24,0000.0677
Transformer 39.39Mno96,0000.0524
Transformer 39.39Mno384,0000.0508
Transformer 39.39Myes24,0000.0543±0.0011
Transformer 39.39Myes96,0000.0476±0.0012
Transformer 39.39Myes384,0000.0487±0.0012
LINK 40.60Mno24,0000.0334
LINK 40.60Myes24,0000.0332±0.0004

Warmup improves the 39.39M Transformer by 20% at 24,000 steps and by 9% at 96,000 steps; with four and sixteen times the steps it reaches 0.0476 and 0.0487, at the level of the 14.20M Transformer of the sweep (0.0454). LINK of the same size reaches 0.0332 at 24,000 steps, 30% below the Transformer with warmup at 96,000 steps. Following your comment, Appendix N.3 reports Table 31.

Regarding parameters, FLOPs, wall-clock time and peak memory for all methods (W1.2). We appreciate the request for the cost of every method. We measured the baselines and every variant of LINK in one run on one RTX 6000 Ada GPU in single precision, with the model configurations of our PDEBench comparisons, which differ from those of the Poisson probe of Table 6, and batch size 4 at 1,024 points and 1 at 16,384 points; each cell gives milliseconds per training step, milliseconds per sample at inference and peak GB (Table 38):

Model (Table 38)ParametersGFLOPs, 1,024 points1,024 points16,384 points
Transformer400,0652.9514.4 / 1.47 / 1.39exceeds GPU memory with standard attention (Table 39)
Galerkin400,5770.8412.2 / 3.04 / 0.1729.2 / 12.92 / 0.62
Transolver344,4170.7817.6 / 4.70 / 0.1826.4 / 11.71 / 0.64
GNOT931,5451.9220.5 / 5.16 / 0.3235.4 / 14.88 / 1.22
FNO2,370,0490.0917.5 / 7.41 / 0.2117.4 / 7.29 / 0.22
LINK-Dense406,2737.25533.4 / 55.66 / 0.3841,370.2 / 12,198.52 / 1.46
LINK-Shared406,2732.43176.9 / 27.77 / 1.04exceeds GPU memory
LINK-Landmark, m = 32423,1691.2234.3 / 9.28 / 1.73137.9 / 50.29 / 6.86

FLOPs are counted for one forward pass and exclude softmax, normalization and elementwise operations for every method. For FNO, each fast Fourier transform counts 5N log₂ N operations on its padded 40×40 grid, and complex products count as real operations. For LINK, the first layer of the pair network is counted in its factorized form of Section 3.2, with the query and key contributions once per point and the distance contribution once per pair. LINK-Shared evaluates the pair network once for all layers, and LINK-Landmark on point–landmark pairs. Section 5.3 now points to Table 38, and Appendix O.2 reports it.

AuthorsOpenReview ↗

Authors Responses (Part 3)

Regarding training and validation curves (W1.3). Thank you for asking for the curves. We added training and validation curves at a common width and depth as Figure 5 in Appendix N.4, with one seed for each curve: five methods on grid Darcy (5a) and on scattered Darcy (5b) at width 64 and depth 8, a larger model at width 192 and depth 6 (5c), and four learning rates for LINK and the Transformer (5d), over 6,000 steps. The validation errors at the end of training are:

Validation rel-L2 at 6,000 steps (Figure 5)Grid, width 64 (5a)Scattered, width 64 (5b)
Transformer0.05790.0833
Galerkin0.13620.1510
Transolver0.17510.1798
GNOT0.14860.1708
LINK0.04310.0684

In Figures 5a to 5c, the validation error of LINK ends below that of every baseline in the panel.

Regarding the stability of the Transformer (W1.5). We appreciate this question. The collapsed runs of Table 14 are configurations of the random search, which Appendix F excludes before selection, and no reported Transformer comes from them. Figure 5d shows the same behavior: the Transformer does not train at a learning rate of 4×10⁻³ and trains at the three smaller rates, whereas LINK trains at all four. Following your suggestion on optimization choices, the warmup search of Table 31 selected a learning rate of 1×10⁻⁴, inside its grid, and warmup improves the selected Transformer by 20% at 24,000 steps. In the revision, Appendix N.4 reports Figure 5d and Appendix N.3 reports Table 31.

Regarding FlashAttention for the baselines (W1.6). Thank you for this question. The comparisons of the submission do not use FlashAttention: they run in single precision, in which the FlashAttention kernel of PyTorch 2.0.1 is not available. Following your suggestion, we measured the Transformer of Table 38 with the FlashAttention kernel of PyTorch at 16,384 points, batch size 1, on one RTX 6000 Ada (Table 39):

Transformer of Table 38 at 16,384 points (Table 39)ms per training stepPeak GB
Single precision, standard attentionexceeds GPU memoryexceeds GPU memory
bf16, FlashAttention64.15.36

With FlashAttention in bf16, this Transformer runs at this size in 64.1 ms per step with 5.36 GB. Appendix O.2 now reports Table 39.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1)

We thank the three reviewers for their careful reading and for the experiments they proposed. We are glad that the reviewers found this a well motivated study (jb9k), the correlation study with the known Green operator of Poisson a good idea to assert the physical relevance of the learned mapping (KoxY), the paper to have good ablations with extensive details provided in the appendix (jb9k), and that the motivation from physical interaction kernels is reasonable and the properties in Sections 4.1 and 4.2 provide useful insights (c2Qw).

The reviews share one question: where the improvement of LINK comes from, and whether it holds in matched comparisons, on additional benchmarks and at scale. The revised PDF answers it with additions, marked in blue; every submitted result is unchanged.

  • Where the improvement comes from (KoxY, jb9k). On PDEBench Darcy at 32×32, making the score a joint function of the two points, with the pair network on the latent states, with a learned geometric term added to the bilinear score, or with both as in LINK, reduces the error of bilinear attention from 0.0665 to between 0.0449 and 0.0479 (Table 29). Geometry in a fixed form is not sufficient: a Gaussian router on the distance reaches 0.1110, compared with 0.0522 for the pair network of the same size (Table 2).
  • Matched comparisons (jb9k, KoxY). In the capacity sweep of Table 18, with learning-rate warmup and three seeds, the 39.39M Transformer reaches 0.0543 at 24,000 steps and 0.0476 at 96,000 steps, compared with 0.0332 for LINK of the same size at 24,000 steps (Table 31). Without the MoE block and the multi-scale locality, LINK reaches 0.0364 with 406,273 parameters, compared with 0.0470 for a Transformer with 400,065 parameters (Table 30). With every model of Table 3 at the longest run of its ladder, which does not choose the lowest test-error budget, LINK is below the Transformer by 12%, 22% and 23% on the three Darcy tasks (Tables 12 and 17).
  • Additional benchmarks (jb9k). On PDEBench Darcy at its 128×128 resolution, LINK-Landmark reaches 0.0653, compared with 0.0932 for FNO, 0.1170 for GNOT, 0.1261 for Transolver and 0.1491 for Galerkin trained with the same protocol (Appendix P), and Table 40 reports the nRMSE of PDEBench [1] next to its published baselines. On The Well, LINK reaches a VRMSE of 0.3781 on meshes that change with every sample, compared with 0.4015 for the Transformer (Table 41).
AuthorsOpenReview ↗

General Response (Part 2)

  • Cost and scale (KoxY, c2Qw, jb9k). LINK-Landmark has a lower error than GNOT at 4,096 and 16,384 points (Table 36), and one trained model runs from 256 to 4,096 points with the same weights (Table 35). At 1,024 points and in single precision, as in every comparison of the submission, LINK-Shared has a lower error than the Transformer in less training time (Table 4). Parameters, FLOPs, time and memory of the baselines and of every variant of LINK are in Table 38.
  • Scope (c2Qw, jb9k). The error of LINK is 35–39% below that of the Transformer at five levels of a cubic nonlinearity, from the linear case to a violation of superposition of 0.32 (Table 33), and 58% below it in 3-D Darcy on a 10×10×10 grid (Table 42). The Well adds a time-dependent problem (Table 41).
  • Elasticity in the published metric (jb9k, KoxY). In the relative error of the raw stress, the metric of the published results, the error of LINK at our common budget of 30,000 steps corresponds to 0.0064, the error published for Transolver after 500,000 steps (Table 32, Appendix N.5).
  • Theory and seeds (KoxY, jb9k). The target kernels of Poisson and Darcy have 29 to 40 singular directions above 1% of the largest, compared with a head width of 12 in our Poisson models (Table 28). Table 27 reruns the comparison of Table 1 with three seeds and adds the routing–Green correlation of the Transformer.

Added material, in blue. Main text: Section 2 (UPT and rotary position embeddings; jb9k, KoxY); Section 3.2 (how LINK-Landmark computes Γ and Λ for the current point set, pointer to the pseudocode and the block diagram; KoxY, c2Qw); Section 4.2 (Proposition 2 concerns a fixed head width; KoxY); Section 5 (the tokens of the Transformer; KoxY); Section 5.1 (Table 1 with three seeds; KoxY, jb9k); Section 5.2 (the two ingredients of the score, matched comparisons, elasticity in the published metric, the strength of the nonlinearity; KoxY, jb9k, c2Qw); Section 5.3 (cost and additional benchmarks; jb9k, c2Qw, KoxY); Section 6 (scope for nonlinear equations and new domains; c2Qw). Appendix L, architecture (Algorithm 1, Figure 4, Table 26; c2Qw). Appendix M, three seeds for Table 1 and singular values (Tables 27 and 28; KoxY, jb9k). Appendix N, score ablations, the comparison without the options of LINK, warmup, training curves, elasticity in the published metric, nonlinearity and held-out domains (Tables 29–34, Figure 5; KoxY, jb9k, c2Qw). Appendix O, LINK-Landmark across point counts and compared with GNOT, landmark counts and cost (Tables 35–39; c2Qw, KoxY, jb9k). Appendix P, results on additional benchmarks (Tables 40–42; jb9k). Appendices H and I.2 state how the budget of Table 3 was chosen and the metric of Table 19 (jb9k, KoxY).

We thank the reviewers again for feedback that improved the paper. We hope that these additions address your concerns and that you will consider them in your assessment, and we are happy to discuss any remaining question.

References:

[1] PDEBench: An Extensive Benchmark for Scientific Machine Learning

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.