- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FTD-GNO, a Graph Neural Operator that uses factorized tensor decomposition to reduce the computational cost of integral kernel evaluation. The authors consider three decomposition methods, including CP, Tensor-Train, and Tucker decomposition. The kernel is represented using several factors, each parameterized by a small neural network. By separating the factors depending on the query point from those depending on neighboring points, the integral computation can be reorganized into two steps. First, the features are aggregated over neighboring points, and then combined with the factors at the query point. In this way, the model avoids evaluating a kernel network for every edge.
The experimental results show that FTD-GNO can significantly reduce computational cost. The peak memory consumption is reduced by around 2–29 times and becomes almost independent of the neighborhood radius. The training speed is improved by around 10–41 times. However, the prediction accuracy is generally worse than the original GNO baselines under the same number of training epochs.
Critical strengths
-
The main advantage of this method is the reduction in memory consumption and training time. By moving the query-dependent factors outside the neighborhood aggregation, the model avoids storing intermediate activations from the kernel network for every edge. According to Table 4, when the neighborhood radius increases from 0.08 to 0.16, the memory consumption of FTD-GNO stays almost unchanged, while GNO requires around 3.1 times more memory. Table 3 also shows training speed improvements of around 10–41 times. I think these efficiency improvements are useful for graph neural operators with large numbers of edges. The comparison with LR-GNO in Table 9 is also helpful, since LR-GNO has a similar parameter count but still requires more memory than GNO. This suggests that the memory reduction mainly comes from reorganizing the computation rather than simply using fewer parameters.
-
I appreciate that the paper reports the accuracy limitations of the proposed method. FTD-GNO generally performs worse than the original GNO when trained for the same number of epochs, and FNO achieves better accuracy than all the graph-based methods in the experiments. The paper also presents tensor factorization mainly as a way to improve computational efficiency rather than claiming better prediction accuracy.
Critical weaknesses
-
My main concern is about the novelty of the proposed method. The factorized kernel formulation seems closely related to existing low-rank neural operators, but this connection is not sufficiently discussed. In particular, the CP decomposition and the corresponding reconstruction in Eq. (20) represent the kernel as a sum of separable functions of the query point and the input point. After moving the query-dependent factors outside the summation, the resulting computation is essentially the same idea used in the Low-rank Neural Operator discussed by Kovachki et al. (2023, JMLR). Similar separable structures can also be found in DeepONet, where branch and trunk networks are combined, and in linear attention methods. Although the paper cites Kovachki et al. (2023), it does not explain this relationship or provide a comparison with existing low-rank neural operators. From my understanding, the main difference is that FTD-GNO applies the separable kernel within a local neighborhood instead of using a global integral. I am not convinced that this is sufficient to establish a strong methodological contribution without further discussion and experimental comparisons.
-
I also have some other concerns about the complexity analysis and experimental evidence.
-
Computational complexity: The paper claims that tensor factorization reduces the computational complexity and memory consumption, but the actual complexity depends on how the neighborhood aggregation is implemented. If the implementation still constructs edge-wise intermediate features, the memory usage should depend on the number of edges. I think the paper needs to explain more clearly which operations are avoided and provide a more complete complexity analysis.
-
Practical benefit: Although FTD-GNO is much faster and uses less memory than GNO, its accuracy is consistently worse. The experiments do not demonstrate a case where the proposed method achieves better accuracy than existing alternatives under the same computational or memory budget. In particular, FNO already performs better than the graph-based models in the reported experiments. Therefore, it is unclear in which practical setting FTD-GNO would be preferable to these existing methods.
-
The accuracy results appear to be based on single training runs. In Table 13, the relationship between decomposition rank and prediction error is not monotonic, which could be caused by training randomness. The repeated-seed statistical analysis in Table 15 only considers peak memory consumption, which should be relatively stable across different runs. Therefore, these results do not provide evidence about the reliability of the accuracy comparisons.
-
The paper uses the criterion that errors remain within the "same order of magnitude" to support zero-shot resolution generalization. I think this criterion is too weak. A model can have substantially worse prediction accuracy while still satisfying this condition. More quantitative comparisons across resolutions are needed to support the generalization claim.
Questions for authors
-
Could the authors clarify which inputs are used for the kernel networks in Tables 2–4 and 9–13? Are they only the coordinates (x, y), or do they also include the input physical fields? If only coordinates are used, I would suggest evaluating field-dependent kernels on Darcy, where the differences between CP, TT, and Tucker decompositions may become more meaningful.
-
In the original GNO, the kernel network outputs a C×C matrix for each edge. What is the output dimension of each factor in FTD-GNO? Is each factor a scalar, a C-dimensional vector, or a matrix? If the factorized kernel only performs diagonal operations across channels, then it has a more restricted representation capacity than the original GNO. In that case, how much of the memory reduction comes from this restriction rather than the proposed factorization?
-
How is the neighborhood aggregation implemented in practice? Is it computed using sparse matrix multiplication, or using scatter-add operations on gathered edge features? If the latter is used, the intermediate memory consumption may still scale with the number of edges. I think this implementation detail is important for understanding the reported memory complexity.
-
Could the authors discuss the relationship between FTD-GNO and the Low-rank Neural Operator in Kovachki et al. (2023), as well as linear attention methods? It would also be helpful to compare with a global low-rank neural operator and with a standard GNO using edge chunking or activation checkpointing under the same memory budget. These comparisons would help clarify whether the efficiency improvement comes from the proposed method or from a more efficient implementation of an existing separable kernel formulation.
12 comments
Response to Reviewer xJJA Part 2
The broader comparison is summarized below.
| Method | Core representation | Factor inputs | Relationship to FTD-GNO |
|---|---|---|---|
| LNO [1] | Integral kernel: | Query position and source position | The original construction uses two-endpoint separation and a global source summary. FTD-GNO implements related separation within local graph integrals and supports multiple input blocks with CP, TT, and Tucker parameterizations. |
| DeepONet [2] | Operator output: | Sampled input-function values for the branch network; output position for the trunk network | Directly combines input-function and position encodings to represent the operator output. FTD-GNO parameterizes an internal integral kernel and supports different tensor formats over multiple kernel-input blocks. Our factors represent an internal integral kernel rather than the final operator output. |
| Galerkin-type attention [3] | Feature interaction: | Queries, keys, and values are typically projected from current features | Uses global feature aggregation. FTD-GNO retains local graph neighborhoods and supports coordinate kernels and multivariable kernels containing field/state inputs. |
| PILNO [4] | Low-rank kernel within an encoder-decoder architecture: | Position and physical coefficient jointly processed within each endpoint | Combines joint-endpoint separation with an encoder-decoder architecture and physics-informed training. FTD-GNO can further parameterize position and field/state separately and organize multivariable interactions using CP, TT, or Tucker. |
| FTD-GNO | CP examples: ; ; . TT and Tucker extensions are also supported. | Positions and, depending on the configuration, separately modeled physical-coefficient or hidden-state blocks | The two-block configuration shares established endpoint-separation principles, with an emphasis on reducing GPU-memory overhead through node-factor reuse and sparse local aggregation. Multivariable configurations support finer input-block separation and different interaction structures through CP, TT, and Tucker. |
The framework's extension therefore concerns not only local neighborhoods, but also functional tensor parameterizations over multiple input blocks. Four-block CP imposes additional within-endpoint product structure relative to joint-endpoint kernels, while TT and Tucker provide other factor organizations.
To provide an initial evaluation of this additional separation, we compare joint-endpoint and four-block CP kernels within the same local graph framework. The results are reported as follows:
| Kernel parameterization | Errors at 16 16 | Errors at 32 32 |
|---|---|---|
| Local joint-endpoint | 0.2982 0.0005 | 0.2969 0.0027 |
| CP-GNO | 0.2974 0.0007 | 0.2944 0.0006 |
The observed mean errors of our four-block CP are lower at both resolutions. This provides preliminary empirical support for further position-state separation. It is not a complete PILNO reproduction. So we must clarify that it is not superior to the complete LNO or PILNO architectures.
2. Implementation contribution: reorganizing kernel evaluation and neighborhood aggregation to reduce GPU-memory costs.
Our focus is not to design a complete operator architecture that outperforms alternatives on every PDE, but to reduce the resource cost of graph-integral components. We evaluate and reuse functional factors at nodes, combining sparse neighborhood aggregation with contraction reordering to reduce repeated edge-wise kernel-network computation and selected edge-wise intermediates. For example, the current CP path avoids explicitly materializing a gathered edge-message tensor and instead uses node-indexed weighted and aggregated features. This implementation preserves local graph interactions and applies to graph-only models and Fourier-graph hybrids retaining graph-integral components.
3. Empirical contribution: demonstrating practical memory savings in the tested graph-integral implementations.
The predictive experiments evaluate the practical behavior of this resource optimization. The hybrid results also show that factorized variants remain trainable in some configurations where their unfactorized counterparts encounter OOM. These experiments support the practical resource value of functional-factor representations and their computational implementation for graph-integral components, rather than universal predictive superiority over other operator architectures.
We will therefore position our main contribution as functional tensor kernel parameterization, finer input-block organization, and resource-oriented implementation through node-factor reuse and sparse contraction within local graph integrals.
Response to Reviewer xJJA Part 3
W2.1: Computational complexity: The paper claims that tensor factorization reduces the computational complexity and memory consumption, but the actual complexity depends on how the neighborhood aggregation is implemented. If the implementation still constructs edge-wise intermediate features, the memory usage should depend on the number of edges. I think the paper needs to explain more clearly which operations are avoided and provide a more complete complexity analysis.
Thank you for raising this concern. We agree that actual computational and memory costs depend on the aggregation implementation and cannot be inferred from the kernel factorization alone.
Which operations change. Unfactorized GNO evaluates its kernel network on selected edges. Our factorized implementation instead evaluates factors at nodes and reuses them across incident edges, reducing repeated edge-wise kernel-network computation. For a scalar CP kernel shared across channels, the computation is
where contains neighborhood connectivity and integration weights. The current CP path flattens the source-weighted features, applies CSR sparse matrix multiplication for neighborhood aggregation, and then contracts with query factors. Let denote batch size, node count, edge count, rank, and feature-channel count; denotes the number of input blocks, and denote the hidden widths of the unfactorized kernel network and factor networks. The principal implementation differences are:
| Operation or storage | Unfactorized GNO | Current reordered CP implementation |
|---|---|---|
| Kernel-network evaluation | Evaluate the kernel MLP on the selected edges | Evaluate factor networks at their associated nodes and reuse outputs across incident edges |
| Principal kernel-network hidden activations | Typically per layer | Typically per factor hidden layer |
| Kernel/factor outputs | Edge-kernel matrices: | Node factors for input blocks: |
| Aggregation-related features | Gathered source features and edge messages can each have shape | Source-weighted features and pre-contraction aggregation outputs each have shape |
| Explicit connectivity and weights | for a batch-shared CSR graph | Same |
What is avoided and what remains. The CP path avoids explicitly constructing a gathered edge-message tensor. The practical benefit comes from node-factor reuse and avoidance of selected edge-wise intermediates. The implementation remains edge dependent: sparse aggregation requires arithmetic, query-side contraction requires arithmetic, and batch-shared CSR connectivity requires storage. A shared scalar kernel and node-indexed factors therefore do not make aggregation independent of edge count or guarantee fewer total operations in every configuration.
We will supplement the revision with this implementation and cost accounting, and restrict the conclusion accordingly: node-factor reuse and sparse contraction reduce repeated edge-wise kernel-network computation and selected intermediate storage in the tested implementations, rather than eliminating edge-dependent aggregation or graph storage.
Response to Reviewer xJJA Part 1
We sincerely thank the reviewer for the valuable and constructive comments! We have carefully incorporated additional discussions and clarifications according to your comments. For easy reference, a list of responses to the reviewers' comments has been compiled, and the corresponding changes have been highlighted in blue in the main text for easy reference of our revision.
Thank you for this insightful comment. We appreciate the opportunity to clarify the contributions of our framework and implementation. Our focus is not to introduce an endpoint-separation identity, but to investigate how functional tensor representations can be applied to local GNO kernels and implemented to reduce the GPU-memory cost of graph-integral components. We will expand the related-work discussion and clarify our contributions as follows.
1. Framework contribution: functional tensor parameterization and finer input-block separation within local graph integrals.
We first clarify the implementation differences from LNO and DeepONet, which the reviewer explicitly identifies.
LNO [1, Section 4.2] represents an integral kernel as κ(x,y)=∑rqr(x)pr(y), and forms a global source summary before combining it with query factors. Our original two-block CP shares this separable form but retains query-dependent local neighborhoods. The inner aggregation therefore generally depends on the query and cannot be replaced by a single global summary shared across queries. Moreover, the original LNO classifies its construction as non-graph-based; our implementation couples functional factors with local connectivity, integration weights, and sparse graph aggregation.
DeepONet [2] directly represents the operator output, typically as G(a)(x)=∑rBr(a(s1),…,a(sm))Tr(x). Its branch network encodes sampled input-function values, while its trunk network encodes output locations. Our factors instead parameterize a graph-integral kernel acting on source features, rather than directly combining branch and trunk representations to produce the final output. The methods share a product-sum structure, but differ in factor inputs and where that structure is applied.