- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether explicitly modifying attention scores using boundary structure improves neural PDE operators when the model inputs and boundary-access route are held fixed. It introduces the Boundary Heterophily Operator (BHO), a boundary-conditioned quadratic residual added to boundary-key attention logits.
Critical strengths
-
The core idea is fundamentally interesting that one can modify the attention mechanism in transformers to encode boundary information.
-
The authors report adverse and null results rather than presenting only favourable evidence.
Critical weaknesses
The paper is largely not understandable. There is a grain of an interesting idea but all relevant equations and specifications are missing.
Questions for authors
What is the definition of B1, B2 and B3? This notation is being used everywhere but not understandable!
What is this thing "host" being referred to?
Equation 2 needs more motivation and the full model needs to be specified.
What is the functional form of the model being studied that the PDE is specified as "tokens"? Thats not how physics-informed ML of any kind I know of works! Without a specification of the data, model and loss, there is nothing that can be figured out from this draft.
Machine learning is all about the loss function. Without a complete unambiguous specification of the loss function there is no information being conveyed.
And refrain from using non-standard abbreviations like BENO and OLBC. There are all kinds of terms like "development error" in Section 3.1, "feature only host" in Table 1, GATE and HOST and BILINEAR in Section 4, none of which are understandable.
This work need a thorough rewrite with precise and clear mathematics for this to be understandable.
Thank you for pointing out where our notation made it difficult to follow the model from its inputs to the training loss.
Data and host. A token is one sampled point of a PDE problem. In the Boundary-Embedded Neural Operators (BENO) Poisson data, 1,024 grid tokens encode coordinates, source f and domain membership. Another 128 tokens encode boundary coordinates, a boundary flag, datum g, estimated normal and boundary type. We predict the numerical solution u on interior grid nodes.
A host is the backbone before the score correction. The feature-only host (B1) receives the same boundary inputs as BHO. An MLP embeds each token to width 128. Four transformer blocks attend jointly over grid and boundary tokens. Each of eight heads projects normalized states to queries, keys and values. For a query q and key k, the host uses the scaled dot product q⊤k/4. For a boundary key, Eq. 2 gives
s=4q⊤k+λ(q−k)⊤D(q−k).
Here D=diag(tanh(We)), where e contains the key's two estimated normal components, type code and scaled boundary datum. Each head in each block learns its own W and λ. Two keys can tie under the dot product and receive different corrections because D weights their coordinatewise discrepancies from the query.
Each head applies softmax over the keys and returns the weighted sum of their values. The block projects the concatenated heads and applies a pre-normalized MLP, each with a residual connection. A final layer normalization and linear readout give the scaled prediction p.
Loss. The target t is u divided by its training-pool standard deviation. With both Euclidean norms taken over interior grid nodes, the per-example loss is
E(p,t)=∥t∥+10−8∥p−t∥.
Training minimizes the batch mean of this loss (Eq. 1). At N=100, rows 0–99 supply gradients. Normalization uses the 800-row training pool. Development error averages the same loss over rows 800–899.
Other terms. B2 uses grid self-attention and boundary cross-attention. B3 attends over 32 learned slices, broadcasts back to the tokens and adds boundary cross-attention. HOST is the unmodified backbone. BILINEAR keeps only the cross term of Eq. 3. Grid GATE adds a scalar boundary-logit bias. OLBC refers to the operator learning with boundary conditions datasets of Mousavi et al. In their released backbone, GATE scales the boundary message (§4).