- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Learned INteraction Kernels (LINK) which replaces the query/key pair dot product score of the attention by an MLP on the pair, this MLP takes the latent states of both points along with absolute and relative geometry information (absolute position of both points, their relative distance) ; a Gaussian locality prior is also added to this score. It first shows how this formulation is consistent over different discretizations and how a joint score is more expressive than its bilinear counterpart, to then show how LINK's Jacobian aligns with the Green operator of the Poisson equation, as well as its superiority over other baselines on Darcy flow.
Critical strengths
- The correlation study with Poisson's known Green operator is a good idea to assert the physical relevance of the learned mapping.
Critical weaknesses
-
It is unclear whether the proposed approach gains from the replacement of the bilinear score function by the pair network or the addition of geometric information to the conditioning of this network. The only comparison with a bilinear alternative is the Transformer, but it differs in size in the main results, and seem to only have its width changed on the capacity control, with a maximum depth of 8 while LINK can go up to 24 ; overall it is unclear whether the implementation only differs in the pair network (especially with the memory cost differences between LINK and Transformer in Table 6). The other baselines do not condition on geometric information, nor use RoPE, which could serve as such. I would be curious to see the comparison with the same LINK backbone but a bilinear score (with the same geometric inputs) at a similar model size, and/or an ablation of LINK without the geometric information.
-
One of the main challenges of operator learning on irregular meshes and point clouds is the computation budget (especially due to the of considering each pair of points) ; this approach seems to add a non-negligible computation overhead as seen in the inference times of Table 6 ; the quicker/lighter version LINK-Landmark introduced to address this issue is not as competitive and as highlighted for its accuracy as the Dense or Shared versions, it also seems to be tied to a fixed number of points as it needs to learn the gather and redistribution matrices.
-
Property 2, which justifies the relevance of the pair network over the bilinear form, is proved upon the assumption of a fixed width and of its eventual inferiority to the rank of the operator it is meant to approximate, but it is not experimentally verified that the rank or singular values of the operators encountered pose a problem with the width considered in practice. Also, couldn't "given sufficient width" justify that a bilinear form can "approximate any strictly positive continuous normalized kernel", such as "given sufficient MLP capacity" (l260) does ?
Questions for authors
-
l273: what does "the same physical tokens" mean ? does it mean both and or only ?
-
how does LINK perform with a bilinear backbone ? How does it perform with its joint score applied only to the latent states (position encoding can be added to those states) ?
-
the routing-Green correlation is never reported for the Transformer, how does it perform ? (I would be interested to see a column in Table 1)
-
in Table 5, how do you explain the differences in results with the Elasticity column of Table 2 from the Transolver paper, which compares against the same baselines (Galerkin, GNOT, Transolver) ?
8 comments
Authors Responses (Part 2)
Regarding the size and the depth of the Transformer (W1). We appreciate the question on capacity and depth. Table 29 uses the same width, depth and number of heads for every score, and the sweep of Table 18 compares a 39.39M Transformer with a 40.60M LINK on scattered Darcy:
| Scattered Darcy (Table 18) | Parameters | Steps | Test rel-L2 |
|---|---|---|---|
| Transformer | 39.39M | 384,000 | 0.0508 |
| LINK | 40.60M | 24,000 | 0.0334 |
At a common 24,000 steps, with warmup and three seeds, the same pair reaches 0.0332±0.0004 for LINK and 0.0543±0.0011 for the Transformer (Table 31).
On depth, the shared hyperparameter space ends at depth 8 for every family, and the selection chose depth 6 for the Transformer on the grid and depth 4 on scattered points (Table 13), below that upper end. The depth extension to 24 layers is used by the grid LINK of Tables 3 and 4 and by the deeper LINK-Shared of Tables 17 and 25. Following your comment, Appendix N.1 now lists the depth selected for every model of Table 13.
Regarding the implementation and the memory in Table 6 (W1). Thank you for checking the implementations. The two models share the lift, the residual blocks, the value maps, the feed-forward blocks and the readout, and differ in the score. In Table 6, LINK-Dense computes the routing in exact query blocks (Appendix K), whereas the Transformer of that probe stores its n×n attention matrices. Section 5.3 now states this difference next to Table 6.
Regarding geometric conditioning of the other baselines, and rotary position embeddings (W1). We appreciate the suggestion of rotary position embeddings (RoPE) [1]. Galerkin, GNOT and Transolver receive the coordinates as input features, as the Transformer does. RoPE gives a bilinear score relative positions through a rotation of its queries and keys; the model you proposed gives it the absolute and relative geometry of the pair explicitly, and with it the bilinear score reaches 0.0449±0.0017 on PDEBench Darcy, compared with 0.0665±0.0005 without it (Table 29). In the revision, Section 2 cites RoPE as a relative-position encoding for bilinear scores.
Authors Responses (Part 3)
Regarding the computational cost and LINK-Landmark (W2). Thank you for raising this point, which decides where pairwise scoring is practical. In addition to the training-step times of Table 6, Table 38 now reports the inference latency of the baselines and of every variant of LINK. Two variants of LINK reduce the cost. LINK-Shared computes the pair-network contribution once and reuses it in every layer; at 1,024 points it has a lower error than the Transformer in less training time on both discretizations, in single precision, as in every comparison of the submission (Table 4):
| Darcy (Table 4) | Steps | Test rel-L2 | Training time (s) |
|---|---|---|---|
| Transformer, grid | 192,000 | 0.0224±0.0013 | 4,744 |
| LINK-Shared, grid | 24,000 | 0.0174±0.0003 | 3,417 |
| Transformer, scattered | 384,000 | 0.0496±0.0004 | 2,550 |
| LINK-Shared, scattered | 12,000 | 0.0407±0.0012 | 2,154 |
LINK-Landmark evaluates the pair score on point–landmark pairs. The routing matrices Γ ∈ ℝ^(m×n) and Λ ∈ ℝ^(n×m) are computed by the pair network for the current point set and then normalized, rather than stored as parameters, and the landmarks have learned locations (Appendix F); a change of n therefore changes the shape of Γ and Λ and no parameter. Inspired by your comment, we also added a result that shows this property directly: one LINK-Landmark (m = 32, 493,281 parameters), trained on our Darcy data on a 64×64 grid at 1,024 of its 4,096 points, keeps the same weights at every point count, with three seeds (Table 35):
| Points (Table 35) | 256 | 512 | 1,024 | 1,024 off grid | 2,048 | 4,096 |
|---|---|---|---|---|---|---|
| LINK-Landmark, same weights | 0.1102±0.0059 | 0.0906±0.0052 | 0.0806±0.0050 | 0.0829±0.0053 | 0.0753±0.0051 | 0.0730±0.0047 |
With the same weights, the error decreases from 0.1102 at 256 points to 0.0730 at 4,096 points.
On accuracy at scale, LINK-Landmark has a lower error than GNOT on Darcy solutions computed on a 128×128 grid and sampled at 4,096 and 16,384 points, with three seeds (Table 36):
| Model (Table 36) | 4,096 points | 16,384 points |
|---|---|---|
| GNOT | 0.1833±0.0050 | 0.2397±0.0059 |
| LINK-Landmark | 0.1185±0.0091 | 0.1705±0.0034 |
On PDEBench Darcy at its 128×128 resolution, LINK-Landmark reaches 0.0653±0.0079, compared with 0.0932±0.0022 for FNO (Appendix P). Section 3.2 now states that Γ and Λ are computed for the current point set, and Appendix O.1 reports Tables 35 and 36.
Regarding the role of Proposition 2 (W3). We appreciate this question on what the fixed width adds. With unbounded width, a bilinear head is also universal, because finite sums of products q(x)ᵀk(y) are dense in the continuous functions on Ω×Ω. Proposition 2 concerns a fixed head width r: a single head of width r has an error of at least min{η/2, η s_{r+1}/4}, where η is the smallest entry of the target and s_{r+1} is the (r+1)-th singular value of its row-centered log-kernel. The bound is therefore informative when more singular values are large than the head width of the Transformer, 12 in our Poisson models. To address your question directly, we computed them for the row-normalized discrete Green operators of Poisson, for κ from 1 to 32, and of four Darcy samples (Table 28):
| Target kernel (Table 28) | Singular values above 1% of the largest |
|---|---|
| Poisson, κ = 1 | 40 |
| Poisson, κ = 2 | 40 |
| Poisson, κ = 4 | 40 |
| Poisson, κ = 8 | 37 |
| Poisson, κ = 16 | 31 |
| Poisson, κ = 32 | 29 |
| Darcy, four samples | 36, 36, 34 and 37 |
Every target has more singular directions above this threshold than the head width of 12, the regime in which Proposition 2 separates a single bilinear head from the joint pair score. Following your comment, Section 4.2 now states that a bilinear head of sufficient width is universal and that the proposition concerns a fixed head width, and Appendix M.2 reports Table 28.
Regarding "the same physical tokens" (Q1). Thank you for pointing to this phrase. It means both: the tokens of the Transformer are the lifted states z(t), and the lift receives the input values at t together with the location t (Section 3.1). The score of the Transformer has no explicit pairwise term, for example t−t′ or ‖t−t′‖: it is the inner product of a query embedding of z(t) and a key embedding of z(t′). The revised Section 5 states that the Transformer receives the same tokens, which contain the input values and the coordinates.
Authors Responses (Part 1)
Thank you for the careful review and for the comparisons you proposed. We are glad that you found the correlation study with the known Green operator of Poisson a good idea to assert the physical relevance of the learned mapping. Below, we address your comments. We hope you find our responses satisfactory, and that you will consider revising your score.
In short: your review asks which part of LINK produces its improvement, and at what cost. Following your comments, we (1) trained the four combinations of a bilinear or joint score, with or without the pair geometry, including the joint score on the latent states (Table 29); (2) evaluated LINK-Landmark across point counts with the same weights and compared it with GNOT up to 16,384 points (Tables 35 and 36); (3) computed the singular values of the target kernels behind Proposition 2 (Table 28); (4) reran the comparison of Table 1 with three seeds and added the routing–Green correlation of the Transformer (Table 27); and (5) expressed the elasticity results in the published metric (Table 32).
Regarding the roles of the pair network and of the geometric inputs (W1, Q2). Thank you for asking which of the two changes produces the improvement. Each of them reduces the error of bilinear attention by 32%. To address your question directly, we trained the four combinations on PDEBench Darcy at 32×32 with 1,000 training samples, with the same width, depth and number of heads and three seeds; the model you proposed has 406,657 parameters and LINK 406,273 (Table 29):
Both changes make the score a joint function of the two points, outside the fixed-width bilinear family of Proposition 2: the pair network reads the coordinates from the latent states, to which the lift adds them, and the geometric term receives them explicitly. LINK combines the two in one pair network. Therefore, scoring the two points jointly, beyond the fixed-width bilinear form of Proposition 2, gives the improvement over bilinear attention. LINK does so with one pair network on the latent states and the geometry. Following your suggestion, Section 5.2 now states that either ingredient reduces the error of bilinear attention by 32% on PDEBench Darcy, and Appendix N.1 reports Table 29.