acceptodds
Under review as a conference paper at ICLR 2027

When Boundary Structure Helps: A Controlled Study of Neural PDE Operators

Abstract

Boundary conditions provide inputs to neural operators for partial differential equations (PDEs), but can score structure improve learning when information and access are matched? We study the Boundary Heterophily Operator (BHO), a quadratic residual on boundary-key logits. On the Boundary-Embedded Neural Operators (BENO) benchmark with 100 examples used for gradient training, BHO reduces held-out relative- error by 13.87% on the source shape and 8.27% under cross-shape transfer. The low-data advantage persists under two matched training budgets. The full quadratic also outperforms a cross-only score with the same boundary conditioner in BENO and a fixed-source Dirichlet circle problem. A secondary metric computed from the same circle predictions favors BHO over the host, a boundary-group bias and the cross-only score in predicting boundary-induced solution changes. Native mixed-boundary comparisons do not establish BHO's superiority over the host or a scalar message gate, and the tested elasticity comparison favors these controls. In a separate study with constant conditioning and physical-interior queries, removing within-head sign mixing increases error by 23.19% with a shared query/key residual projection but has no detected cost with separate projections. Thus boundary-score parameterization can improve learning within a matched host and route, while sign-restriction costs depend on the tested residual parameterization.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, partial differential equations, scientific machine learning, boundary conditions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_nEo6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether explicitly modifying attention scores using boundary structure improves neural PDE operators when the model inputs and boundary-access route are held fixed. It introduces the Boundary Heterophily Operator (BHO), a boundary-conditioned quadratic residual added to boundary-key attention logits.

Critical strengths
  • The core idea is fundamentally interesting that one can modify the attention mechanism in transformers to encode boundary information.

  • The authors report adverse and null results rather than presenting only favourable evidence.

Critical weaknesses

The paper is largely not understandable. There is a grain of an interesting idea but all relevant equations and specifications are missing.

Questions for authors

What is the definition of B1, B2 and B3? This notation is being used everywhere but not understandable!

What is this thing "host" being referred to?

Equation 2 needs more motivation and the full model needs to be specified.

What is the functional form of the model being studied that the PDE is specified as "tokens"? Thats not how physics-informed ML of any kind I know of works! Without a specification of the data, model and loss, there is nothing that can be figured out from this draft.

Machine learning is all about the loss function. Without a complete unambiguous specification of the loss function there is no information being conveyed.

And refrain from using non-standard abbreviations like BENO and OLBC. There are all kinds of terms like "development error" in Section 3.1, "feature only host" in Table 1, GATE and HOST and BILINEAR in Section 4, none of which are understandable.

This work need a thorough rewrite with precise and clear mathematics for this to be understandable.

1 comment

Thank you for pointing out where our notation made it difficult to follow the model from its inputs to the training loss.

Data and host. A token is one sampled point of a PDE problem. In the Boundary-Embedded Neural Operators (BENO) Poisson data, 1,024 grid tokens encode coordinates, source ff and domain membership. Another 128 tokens encode boundary coordinates, a boundary flag, datum gg, estimated normal and boundary type. We predict the numerical solution uu on interior grid nodes.

A host is the backbone before the score correction. The feature-only host (B1) receives the same boundary inputs as BHO. An MLP embeds each token to width 128. Four transformer blocks attend jointly over grid and boundary tokens. Each of eight heads projects normalized states to queries, keys and values. For a query qq and key kk, the host uses the scaled dot product q⊤k/4q^{\top}k/4. For a boundary key, Eq. 2 gives

s=q⊤k4+λ(q−k)⊤D(q−k).\displaystyle s = \frac{q^{\top}k}{4} + \lambda (q-k)^{\top} D (q-k).

Here D=diag⁡(tanh⁡(We))D = \operatorname{diag}(\tanh(W e)), where ee contains the key's two estimated normal components, type code and scaled boundary datum. Each head in each block learns its own WW and λ\lambda. Two keys can tie under the dot product and receive different corrections because DD weights their coordinatewise discrepancies from the query.

Each head applies softmax over the keys and returns the weighted sum of their values. The block projects the concatenated heads and applies a pre-normalized MLP, each with a residual connection. A final layer normalization and linear readout give the scaled prediction pp.

Loss. The target tt is uu divided by its training-pool standard deviation. With both Euclidean norms taken over interior grid nodes, the per-example loss is

E(p,t)=∥p−t∥∥t∥+10−8.\displaystyle E(p,t) = \frac{\lVert p-t\rVert}{\lVert t\rVert + 10^{-8}}.

Training minimizes the batch mean of this loss (Eq. 1). At N=100N=100, rows 0–99 supply gradients. Normalization uses the 800-row training pool. Development error averages the same loss over rows 800–899.

Other terms. B2 uses grid self-attention and boundary cross-attention. B3 attends over 32 learned slices, broadcasts back to the tokens and adds boundary cross-attention. HOST is the unmodified backbone. BILINEAR keeps only the cross term of Eq. 3. Grid GATE adds a scalar boundary-logit bias. OLBC refers to the operator learning with boundary conditions datasets of Mousavi et al. In their released backbone, GATE scales the boundary message (§4).

Reviewer_9b24OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors studied how boundary-score parameterization affects transformer-based PDE surrogates. Standard neural operators typically encode boundary conditions through input features or graph topology, but often use simple dot-product attention scores. The authors’ main idea was to add a boundary-conditioned quadratic residual to the attention logits. They also examined the effect of removing coordinate sign mixing from the quadratic weights. A key design motivation was to isolate the score function’s behavior. They demonstrated the method empirically across two benchmark families.

Critical strengths
  • The author addresses a specific question regarding where boundary information should enter attention-based PDE surrogates.

  • The author provides a carefully controlled evaluation protocol with paired seeds.

Critical weaknesses
  • The technique/architecture introduced is well known and well used. Quadratic/Mahalanobis distance biases in attention logits and kernelized attention [1] already contain such pairwise interactions. At the same time, standard graph-based neural operators (such as BENO) already handle boundary-interior interactions effectively when enough data is available. Furthermore, the un-fused quadratic score slows down training and inference by ~1.6x, which is a high cost for a minor logit modification.

  • While improving boundary-value PDE surrogates is a main concern, the empirical gains disappear when the training set size increases from 100 to 800 on the main benchmark, and fail to transfer to the linear elasticity task.

  • In [Line 065], the second research question (Q2) studies sign mixing using a constant conditioner instead of actual boundary features. This is disconnected from the main setup and requires additional clarification.

[1] Tsai et al. Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel

Questions for authors
  • [line 051] "Boundary Heterophily Operator": Why is it called heterophily when graph heterophily is never defined or measured?

  • [line 201] Why does the gain disappear at N=800 on the primary host, but remain large on the B2 host?

  • Experiment: Why is a constant conditioner used in Q2 instead of the boundary features from Q1?

1 comment

Thank you for pointing us to the kernel interpretation of attention!

Why a constant conditioner (line 065)? Q1 tests a locally conditioned score. In Q2 the conditioner receives only the type code, which is constant on this population. Each head thus learns one diagonal shared across boundary keys. Keys and values still use the actual boundary observations. This lets us test the sign restriction without also varying the diagonal across keys. A four-seed screen included both conditioning choices. The full projection-by-sign comparison on eight fresh seeds used constant conditioning.

Novelty. Prior work interprets attention as kernel smoothing${}^{[1]},modelsembeddingsimilaritywithsignedinnerproducts, models embedding similarity with signed inner products{}^{[2]},andcomputesattentionweightsusingaMahalanobisdistance, and computes attention weights using a Mahalanobis distance{}^{[3]}$. The empirical distinction is that the sign restriction reversed which projection family performed better. From the four cells at 100 Neumann-labeled examples with interior queries, the shared family had 10.2% lower development error than the separate family with mixed signs and 9.8% higher error with one sign per head. Table 8 reports the paired interaction (+22.32%, pointwise 95% CI +15.10% to +30.01%). The two families also differ in capacity and coupling to ordinary attention. At 200 examples the interaction interval included zero.

Why does B2 improve at 800 (line 201)? In the archived 300-epoch sweep, joint attention gave 0.0682 with or without BHO. On the same seed set B2 improved from 0.2342 to 0.1236 (Table 4), still above the unmodified joint host's error. B3 also has a boundary-only route and showed no clear gain. Sample size and the presence of a boundary route alone do not predict the gain. Its cause in B2 remains unresolved. BHO also increased elasticity error and took about 1.6× the joint host's inference time. These experiments do not directly compare BHO with Wang et al.'s full graph operator${}^{[4]}$.

Why “heterophily” (line 051)? The term alluded to the different roles of boundary and interior points. We measured no graph heterophily. “Boundary-conditioned quadratic score” describes the intervention more precisely.

References

[1] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel. In EMNLP-IJCNLP, 2019.

[2] Geewook Kim, Akifumi Okuno, Kazuki Fukui, and Hidetoshi Shimodaira. Representation Learning with Weighted Inner Product for Universal Approximation of General Similarities. In IJCAI, 2019.

[3] Stefan K. Nielsen, Laziz U. Abdullaev, Rachel S.Y. Teo, and Tan M. Nguyen. Elliptical Attention. In Advances in Neural Information Processing Systems, 37, 2024.

[4] Haixin Wang, Jiaxin Li, Anubhav Dwivedi, Kentaro Hara, and Tailin Wu. BENO: Boundary-embedded Neural Operators for Elliptic PDEs. In ICLR, 2024.

Reviewer_Dp7uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BHO, a boundary-conditioned quadratic correction to attention logits, and studies its effects while matching the information supplied to competing models.

Critical strengths

Matching inputs, attention access, initial predictions, training subsets, and data order makes the comparisons more informative than comparisons between unrelated architecture.

The paper explicitly reports negative results, reused evaluation populations, unresolved contrasts.

Critical weaknesses

The novelty is narrow, and the study does not yet explain when the method helps.

Practical benefit over strong hosts remains limited and comes with substantial overhead.

Table 17 reports 1.56× inference latency and 1.59× forward–loss–backward latency. Matching update counts is appropriate for studying learning behavior, but it does not establish an advantageous accuracy–compute tradeoff.

The score controls do not isolate why the full quadratic improves learning.

Questions for authors

check the weakness.

1 comment

We appreciate your distinction between accuracy at a fixed update count and accuracy at a fixed compute budget.

Practical benefit. At 100 BENO examples, BHO reduced development error by 9.0% versus the joint host at 3,900 updates (Table 9). On one A100 at batch size 16, BHO took 1.56× as long for inference and 1.59× for forward–loss–backward (Table 17). The fixed-update gain establishes no accuracy advantage at equal wall time. At 800 examples and on Boomerang, the host comparisons were inconclusive. Elasticity favored the host.

What the controls establish. In the 100-example BENO cohort, BHO had 6.35% lower development error than the conditioned cross term (Table 2, Holm p = 0.030). The archived results for the same five-contrast family show that BHO had 5.47% higher error than a generic coordinatewise quadratic (Holm p = 0.039). That score uses the same token features and boundary keys, has no explicit boundary conditioner, and adds 2,592 parameters versus 2,080 for BHO. The other comparisons, against a query-only gate, diagonal bilinear score and host without normal/type channels, were inconclusive.

Dropping the self terms also changed the ungated score's RMS at initialization from 2.52 to 1.47. The cross-only comparison cannot isolate structure from scale. The gain over the host therefore cannot be attributed specifically to boundary conditioning. At 15,000 updates, the generic score also had lower mean error than BHO at both training sizes, but neither paired difference was resolved.

Reviewer_zWJZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether changing how attention scores use boundary information improves neural operators, when the inputs and attention route are kept the same. The authors propose the Boundary Heterophily Operator (BHO), which adds a quadratic term to the attention logits of boundary keys. The term is conditioned on the boundary normal, type and value. They compare BHO to the same model without this term and to two simpler scores: a scalar bias on boundary keys and a cross-term-only version. In a separate study, they also test whether restricting the sign of the quadratic term hurts performance.

They validate on Poisson problems from BENO and OLBC and on a linear elasticity dataset. BHO improves on the baseline on BENO with 100 training examples, but not with 800 examples, on the OLBC datasets, or on elasticity, where it performs worse. Restricting the sign increases error by 23.19% when queries and keys share a projection, with no detected cost when they’re separate.

Critical strengths

Strength 1: Honest reporting of results.
The authors report the settings where BHO doesn’t help or performs worse, such as the Boomerang and elasticity datasets, instead of only reporting favorable results. This is consequential because it makes the paper’s claims reliable, even if they are limited.

Strength 2: Controlled experimental design.
The comparisons are carefully controlled. BHO and the baseline receive the same inputs, use the same attention route, and start from the same initial model, since the gate is initialized at zero. The experiments use paired seeds, pre-specified contrasts and multiple comparison corrections. This matters to my assessment because it makes it possible to attribute differences in performance to the attention score itself rather than to differences in inputs or architecture.

Critical weaknesses

Weakness 1: Limited motivation and scope of PDEs. It would help to explain why a quadratic term (qi−kj)⊤Dj(qi−kj)(q_i - k_j)^\top D_j (q_i - k_j) on the attention logits is expected to help with boundary conditions, and what it captures that the ordinary attention score doesn’t. The experiments also focus mostly on Poisson problems, with linear elasticity as the only other PDE. It’s unclear why other PDEs, such as time-dependent problems like the heat or advection equation, weren’t considered, or why periodic boundary conditions weren’t included. This is consequential because a clearer motivation and a broader set of PDEs and boundary conditions would make it possible to understand when BHO is expected to help, and would make the conclusions more general.

Weakness 2: The results don’t show a clear benefit or takeaway. The clearest improvement over the baseline is on BENO with 100 training examples. No improvement is detected with 800 examples, and the paper doesn’t discuss why. Understanding this would be valuable, for example whether BHO acts mainly as an inductive bias that helps when data is scarce. On the OLBC datasets BHO doesn’t clearly improve on the baseline, and on elasticity it performs worse. BHO does outperform the cross-term-only score on some datasets, but as the authors note, the two scores also differ in scale and optimization, so it’s not clear why the full quadratic helps. This is consequential because, in its current form, the paper doesn’t yet show when BHO should be used or characterize when and why it doesn’t help.

Questions for authors
  1. BHO improves on the baseline with 100 training examples, but no improvement is detected with 800 examples. Do you have an explanation for this? For example, does BHO act mainly as an inductive bias that helps when data is scarce? Results at intermediate training sizes, such as 200 or 400 examples, would help show how the effect changes with data.

  2. Could you explain why the quadratic term (qi−kj)⊤Dj(qi−kj)(q_i - k_j)^\top D_j (q_i - k_j) is expected to help with boundary conditions, and what it captures that the ordinary attention score doesn’t?

  3. Why are the experiments limited mostly to Poisson problems? Have you tried other PDEs (heat or advection equation or Burgers or Navier Stokes)? How would BHO handle periodic boundary conditions?

1 comment

Thank you for asking what the quadratic term adds to dot-product attention!

Training size. In an archived Neumann-labeled BENO sweep, BHO reduced mean development error versus the same-feature joint host by 4.1% at 200 examples and 8.1% at 400 (7/8 paired seeds each). The sweep fixed epochs at 300, so updates grew with training size. Code revisions varied, and a different conditioner initialization gave smaller, inconclusive effects. At 3,900 matched updates, the size interaction was unresolved (800/100 ratio of BHO-to-host error ratios 1.21, simultaneous 95% CI 0.90–1.61, Table 9). At 800, archived held-out errors were 0.0662 for the host and 0.0669 for BHO. We cannot identify a sample-size threshold or attribute the gain at 100 to a particular inductive bias.

Why Eq. 2? For fixed embeddings, two boundary keys can tie under the dot product yet have different coordinatewise squared distances from the query. Eq. 2 weights these distances using each key's normal, type and datum. In an archived 100-example check, shuffling the learned diagonals across boundary keys raised mean error from 0.405 to 0.475 (three shuffles per model, worse in 8/8 seeds). Retraining with a constant conditioner gave 0.395. The fitted models used the alignment, but local conditioning showed no clear advantage over constant conditioning. The full and cross-only scores also differed in RMS before gating at initialization (2.52 vs. 1.47), so their comparison cannot isolate the self terms.

Other PDEs and periodicity. BENO varies geometry, source and boundary data. The Poisson circle isolates changes in boundary values. On the circle, solution error versus the host was inconclusive, while boundary-response error on the same predictions fell by 9.68% (Table 3). Elasticity favored the host. An archived stationary Helmholtz test with impedance boundaries showed no clear gain (4/8 pairs). For a fully periodic representation with no boundary keys, Eq. 2 adds zero. The implementation assumes boundary tokens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.