- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In PDEs with finite propagation speed, the influence of a point is confined to a cone. The main contribution of this paper is ConeNO, a neural operator for fast PDE solving that improves performance by restricting attention to tokens inside this cone. The paper also proves mathematically that this restriction confines the influence to a cone-shaped region, and in particular that this still holds when the operation is applied repeatedly.
Overall, I think the problem setting and the approach are insightful. However, as described below, I am concerned that the computational cost may be a critical issue.
Critical strengths
- The problem setting is clear, and the method fits it well. Solvers for PDEs with finite propagation speed have been studied in various ways in numerical analysis, but to my knowledge this idea, has not been introduced into neural operators before.
- The performance is high, and the error is well controlled.
- The method assumes that the propagation speed is known, but strictly speaking only an upper bound is needed. Also, if the purpose of a surrogate model is speed-up, it is reasonable to assume a setting where the equation and its theoretical propagation speed are known. So I do not consider this a real problem.
Critical weaknesses
- The paper has two main technical contributions: the modified attention and the raw-value readout. I would like to see an ablation on these two components. Table 9 is a different kind of ablation.
- The scale of the experiments seems small compared to prior work. The resolution is 64x64, and the rollout appears to go only up to T=2. This resolution is the smallest size used in the original FNO paper, and PDEBench contains datasets with higher resolution.
- A comparison of computational cost is needed. Even though the attention range is restricted, the cost is still O(N^2) in the collocation points, so scalability is questionable. I expect the cost to be much larger than that of models built only from lightweight operations, such as FNO. ConeNO does not support irregular meshes, which is a major advantage of attention-based neural operators such as Transolver, so the computational cost issue becomes even more important. In the worst case, solving the original equation directly may be faster.
- Some explanations are slightly inaccurate. The text right below Table 1 says "DeepONet is worse than the local models on every condition," but DeepONet is not worse for Coeff and SR2.
Once the concern about the computational cost is resolved, I am happy to raise the score.
Questions for authors
See weaknesses.
3 comments
Thank you for the detailed response.
The ablation addressed my first comment for the 1D tasks, and the explanation of the DeepONet addressed the forth.
However, my main concern about the computational cost remains. While the reported time is not slow, it is because of the small sizes of the benchmark tasks. For practical sizes, the computational cost of O(N^2) is not negligible. The attention-based methods such as OFormer, GNOT and Transolver can handle irregular meshes, where interactions between nodes are unavoidable and the computational cost is justified. In contrast, ConeNO does not provide a clear advantage that justifies its computational cost. Moreover, ConeNO is restricted to periodic uniform grids.
I consider that the idea is sound and the advantages are established, and I appreciate the added ablation. My main concern is the computational cost against the benefit. I therefore keep my score. If the other reviewers endorse acceptance, I will not oppose it.
reply to Reviewer z936
We thank the reviewer for confirming the ablation and the DeepONet correction.
The quadratic count arises only when the key stride is held fixed in grid units. Under Proposition 5.5 the stride scales with the grid, so each query keeps the physical key set of the coarse cone. The keys per query stay constant and the admissible pairs grow linearly with the number of nodes M. SR2 is this evaluation: 512 nodes in 1D and 128 x 128 on the wave map, from the trained checkpoint, with tap offsets, residue strides, and raw-value read offsets multiplied by the refinement factor f (Appendix B.1), without retraining. The 60-key model in the previous response is a separate run, trained from scratch at M=512 with the stride fixed at 4. That is the quadratic setting reported there. The cone radius is set by the one-step interval tau, and the same mask construction is reused at every rollout step.
Global FNO mixes the grid by an FFT and a Fourier multiplier, so the key count and the node count are not a FLOP ratio. On one GPU, batch size 8, a step is 3.1 ms for ConeNO and 2.2 ms for global FNO on transport, 2.9 ms and 2.0 ms on Burgers, and 5.8 ms and 4.0 ms on the 64 x 64 wave map. ConeNO keeps the exterior response at 0. Table 1 bumps are 1.08+/-0.26 against 22.65+/-5.35 for global FNO, and Table 2 in-distribution error is 1.86+/-0.18 against 5.04+/-0.18 for global FNO. A local convolution also has exterior response 0 and is less accurate on the ablation.
The cone on irregular coordinates does not use the periodic index set of Eq. (16). A node is admissible when it lies inside Eq. (1), at distance at most c_mask times the elapsed time, and a delayed tap is kept only when its length satisfies Eq. (5). OFormer, GNOT, and Transolver do not enforce a receptive-field bound, so the comparison here is unrestricted full attention on the same nodes. We use one frozen set of 1024 nodes in the unit square. This run tests invariance on irregular coordinates. It does not subsample, and it is separate from the Proposition 5.5 configuration. The widest query reads 519 of the 1024 nodes, the ones that lie in one fixed physical disk. The reference at each node is the sample of the translated continuous field, not a spectral reconstruction of the nodal values, and an edit outside the cone leaves that sample unchanged. The convolutional control uses the same cone and has no attention keys, which is why its cell is blank.
The rows below are separate from Table 4. Relative L2 x 100, mean +/- SD over five seeds. Exterior change is the prediction change at the query nodes after perturbing inputs outside their cones. Reference change is the change in the exact solution.
| Model | Params (k) | ID | Bump | Exterior change | Reference change | Nodes read | One step (ms) | Peak mem. (GB) |
|---|---|---|---|---|---|---|---|---|
| Global attention | 140.6 | 2.87+/-0.23 | 23.64+/-2.91 | 13.28+/-1.86 | 0 | 1024 | 4.6 | 0.91 |
| Conv. control, coordinate cone | 139.8 | 3.54+/-0.28 | 10.73+/-1.24 | 0 | 0 | — | 1.8 | 0.42 |
| ConeNO, coordinate cone | 140.4 | 2.16+/-0.16 | 7.38+/-0.67 | 0 | 0 | 519 | 3.3 | 0.61 |
ConeNO is lowest on ID and on bumps. Both cone-restricted models have zero exterior change. Global attention changes by 13.28+/-1.86 under the same edit, and the reference change is 0.
reply to Reviewer z936
We thank Reviewer z936. On a tau-step the softmax in Eq. (6) runs over J(p) only. With s_x = s_t = 4 and c_mask * tau = 1/6, the widest query on the N=256 prefix has 30 admissible keys. Rollout to T=2 is eighteen calls of that one-step cone; the counted maximum is 30 at calls 1, 6, 12, and 18. On one GPU, batch size 8, a step is 2.9 ms for ConeNO and 2.0 ms for global FNO on Burgers, 3.1 ms and 2.2 ms on transport, and 5.8 ms and 4.0 ms on the 64x64 wave map. Peak memory is 0.70 GB against 0.50 GB, 0.74 GB against 0.53 GB, and 1.10 GB against 0.80 GB. Training is 1.7 s/epoch against 1.2 on Burgers and 1.9 against 1.3 on transport. The step stays within a factor of 1.5 of global FNO, at a Burgers ID error of 1.86 against 5.04 and a transport bump error of 1.08 against 22.65.
The epoch column is train mode; the tau-step is eval mode.
The rollout column is one trajectory of eighteen calls, including the prefix between calls. Isolated steps are 55.8 ms on transport, 52.2 ms on Burgers, and 104.4 ms on the wave map; the trajectories are 57.9, 53.4, and 108.8 ms. The difference is the prefix.
The references are a characteristic shift, a Godunov step, and a Fourier mode rotation. An exterior edit leaves each fixed. The cost table compares the learned maps with global FNO.
At a fixed cone radius and fixed strides, keys per query grow with N, from 30 at N=256 to 60 for a model trained from scratch at N=512, and the number of queries grows with N, so the number of admissible pairs is quadratic in the number of spatial nodes. Cone restriction reduces that count relative to mixing every pair. SR2 is separate: Appendix B.1 and Proposition 5.5 scale the residue stride with the grid, so each query keeps the physical key set of the coarse cone. The cone radius is set by tau, not by the horizon. SR2 evaluates the trained models at 512 nodes in 1D and at 128 x 128 on the wave map. Global FNO has no set J and mixes all 512 nodes. The convolutional control has no set J.
ID matches global FNO (1.18+/-0.07 and 1.15+/-0.07). On localized inputs ConeNO stays at 1.06+/-0.21 and global FNO moves to 19.74+/-4.28. CVS is 0 against 0.0531+/-0.0071.
On transport, DeepONet is above both locality-constrained baselines on ID (8.71 vs. 2.17 and 2.08), Bump (9.49 vs. 1.85 and 2.56), and T=2 (9.35 vs. 3.61 and 3.72). On Coeff it is 8.55, below local-window FNO at 12.27 and above the convolutional control at 7.98; on SR2 it is 8.71, below local-window FNO at 9.91 and above the convolutional control at 2.07. ConeNO is lowest among the locality-constrained models on Coeff and SR2, at 5.92 and 1.25, and on SR2 it lies within one seed SD of global FNO (1.25 against 1.24).
The two-component ablation is Appendix D.3.2, Tables 10 and 11 (five seeds, 80 epochs, matched widths). Every row has CVS 0.
Transport
Burgers
On transport the readout carries ID, SR2, and T=2; latent attention carries Bump and Coeff. On Burgers each component alone improves every column over the control, and ConeNO improves further over either alone. On the wave map, SR2 is 5.27 against 12.64, T=2 is 4.18 against 6.95, and CVS is 0 against 0.318 (Table 5).