acceptodds
Under review as a conference paper at ICLR 2027

FTD-GNO: Memory-Efficient Graph Neural Operators through Functional Tensor Decomposition of the Kernel

Abstract

Graph Neural Operators (GNOs) provide flexible surrogate models for learning solution operators of partial differential equations (PDEs). However, standard GNOs typically parameterize the integral kernel with a monolithic neural network and evaluate kernel interactions over graph edges, leading to substantial computational and memory overhead at high resolutions or with large neighborhoods. To address these limitations, we propose Functional Tensor Decomposition Graph Neural Operator (FTD-GNO), a memory-efficient GNO framework that decouples the high-dimensional continuous integral kernel into low-dimensional mode-wise functions. By instantiating the kernel with classical tensor decomposition formats, including CP, Tensor-Train, and Tucker decompositions, FTD-GNO enables algebraic reconstruction of the integral operator without explicitly materializing full edge-wise kernel tensors. This factorized formulation reduces the memory footprint of kernel evaluation and aggregation while retaining the continuous operator-learning structure of GNOs. Theoretical complexity analysis shows that FTD-GNO substantially lowers parameter and activation-memory costs associated with high-dimensional kernel construction. Experiments show lower peak memory than the corresponding unfactorized graph-integral baselines, with shorter recorded training times. Fourier–graph experiments further demonstrate that FTD can improve the efficiency of a graph-integral layer within a hybrid operator and has good scalability.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph neural operators, operator learning, functional tensor decomposition, partial differential equations, memory efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_xJJAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FTD-GNO, a Graph Neural Operator that uses factorized tensor decomposition to reduce the computational cost of integral kernel evaluation. The authors consider three decomposition methods, including CP, Tensor-Train, and Tucker decomposition. The kernel is represented using several factors, each parameterized by a small neural network. By separating the factors depending on the query point from those depending on neighboring points, the integral computation can be reorganized into two steps. First, the features are aggregated over neighboring points, and then combined with the factors at the query point. In this way, the model avoids evaluating a kernel network for every edge.

The experimental results show that FTD-GNO can significantly reduce computational cost. The peak memory consumption is reduced by around 2–29 times and becomes almost independent of the neighborhood radius. The training speed is improved by around 10–41 times. However, the prediction accuracy is generally worse than the original GNO baselines under the same number of training epochs.

Critical strengths
  • The main advantage of this method is the reduction in memory consumption and training time. By moving the query-dependent factors outside the neighborhood aggregation, the model avoids storing intermediate activations from the kernel network for every edge. According to Table 4, when the neighborhood radius increases from 0.08 to 0.16, the memory consumption of FTD-GNO stays almost unchanged, while GNO requires around 3.1 times more memory. Table 3 also shows training speed improvements of around 10–41 times. I think these efficiency improvements are useful for graph neural operators with large numbers of edges. The comparison with LR-GNO in Table 9 is also helpful, since LR-GNO has a similar parameter count but still requires more memory than GNO. This suggests that the memory reduction mainly comes from reorganizing the computation rather than simply using fewer parameters.

  • I appreciate that the paper reports the accuracy limitations of the proposed method. FTD-GNO generally performs worse than the original GNO when trained for the same number of epochs, and FNO achieves better accuracy than all the graph-based methods in the experiments. The paper also presents tensor factorization mainly as a way to improve computational efficiency rather than claiming better prediction accuracy.

Critical weaknesses
  1. My main concern is about the novelty of the proposed method. The factorized kernel formulation seems closely related to existing low-rank neural operators, but this connection is not sufficiently discussed. In particular, the CP decomposition and the corresponding reconstruction in Eq. (20) represent the kernel as a sum of separable functions of the query point and the input point. After moving the query-dependent factors outside the summation, the resulting computation is essentially the same idea used in the Low-rank Neural Operator discussed by Kovachki et al. (2023, JMLR). Similar separable structures can also be found in DeepONet, where branch and trunk networks are combined, and in linear attention methods. Although the paper cites Kovachki et al. (2023), it does not explain this relationship or provide a comparison with existing low-rank neural operators. From my understanding, the main difference is that FTD-GNO applies the separable kernel within a local neighborhood instead of using a global integral. I am not convinced that this is sufficient to establish a strong methodological contribution without further discussion and experimental comparisons.

  2. I also have some other concerns about the complexity analysis and experimental evidence.

  • Computational complexity: The paper claims that tensor factorization reduces the computational complexity and memory consumption, but the actual complexity depends on how the neighborhood aggregation is implemented. If the implementation still constructs edge-wise intermediate features, the memory usage should depend on the number of edges. I think the paper needs to explain more clearly which operations are avoided and provide a more complete complexity analysis.

  • Practical benefit: Although FTD-GNO is much faster and uses less memory than GNO, its accuracy is consistently worse. The experiments do not demonstrate a case where the proposed method achieves better accuracy than existing alternatives under the same computational or memory budget. In particular, FNO already performs better than the graph-based models in the reported experiments. Therefore, it is unclear in which practical setting FTD-GNO would be preferable to these existing methods.

  • The accuracy results appear to be based on single training runs. In Table 13, the relationship between decomposition rank and prediction error is not monotonic, which could be caused by training randomness. The repeated-seed statistical analysis in Table 15 only considers peak memory consumption, which should be relatively stable across different runs. Therefore, these results do not provide evidence about the reliability of the accuracy comparisons.

  • The paper uses the criterion that errors remain within the "same order of magnitude" to support zero-shot resolution generalization. I think this criterion is too weak. A model can have substantially worse prediction accuracy while still satisfying this condition. More quantitative comparisons across resolutions are needed to support the generalization claim.

Questions for authors
  1. Could the authors clarify which inputs are used for the kernel networks in Tables 2–4 and 9–13? Are they only the coordinates (x, y), or do they also include the input physical fields? If only coordinates are used, I would suggest evaluating field-dependent kernels on Darcy, where the differences between CP, TT, and Tucker decompositions may become more meaningful.

  2. In the original GNO, the kernel network outputs a C×C matrix for each edge. What is the output dimension of each factor in FTD-GNO? Is each factor a scalar, a C-dimensional vector, or a matrix? If the factorized kernel only performs diagonal operations across channels, then it has a more restricted representation capacity than the original GNO. In that case, how much of the memory reduction comes from this restriction rather than the proposed factorization?

  3. How is the neighborhood aggregation implemented in practice? Is it computed using sparse matrix multiplication, or using scatter-add operations on gathered edge features? If the latter is used, the intermediate memory consumption may still scale with the number of edges. I think this implementation detail is important for understanding the reported memory complexity.

  4. Could the authors discuss the relationship between FTD-GNO and the Low-rank Neural Operator in Kovachki et al. (2023), as well as linear attention methods? It would also be helpful to compare with a global low-rank neural operator and with a standard GNO using edge chunking or activation checkpointing under the same memory budget. These comparisons would help clarify whether the efficiency improvement comes from the proposed method or from a more efficient implementation of an existing separable kernel formulation.

12 comments
AuthorsOpenReview ↗

Response to Reviewer xJJA Part 1

We sincerely thank the reviewer for the valuable and constructive comments! We have carefully incorporated additional discussions and clarifications according to your comments. For easy reference, a list of responses to the reviewers' comments has been compiled, and the corresponding changes have been highlighted in blue in the main text for easy reference of our revision.

W1: My main concern is about the novelty of the proposed method. The factorized kernel formulation seems closely related to existing low-rank neural operators, but this connection is not sufficiently discussed. In particular, the CP decomposition and the corresponding reconstruction in Eq. (20) represent the kernel as a sum of separable functions of the query point and the input point. After moving the query-dependent factors outside the summation, the resulting computation is essentially the same idea used in the Low-rank Neural Operator discussed by Kovachki et al. (2023, JMLR). Similar separable structures can also be found in DeepONet, where branch and trunk networks are combined, and in linear attention methods. Although the paper cites Kovachki et al. (2023), it does not explain this relationship or provide a comparison with existing low-rank neural operators. From my understanding, the main difference is that FTD-GNO applies the separable kernel within a local neighborhood instead of using a global integral. I am not convinced that this is sufficient to establish a strong methodological contribution without further discussion and experimental comparisons.

Thank you for this insightful comment. We appreciate the opportunity to clarify the contributions of our framework and implementation. Our focus is not to introduce an endpoint-separation identity, but to investigate how functional tensor representations can be applied to local GNO kernels and implemented to reduce the GPU-memory cost of graph-integral components. We will expand the related-work discussion and clarify our contributions as follows.

1. Framework contribution: functional tensor parameterization and finer input-block separation within local graph integrals.

We first clarify the implementation differences from LNO and DeepONet, which the reviewer explicitly identifies.

LNO [1, Section 4.2] represents an integral kernel as κ(x,y)=∑rqr(x)pr(y),\kappa(x,y)=\sum_rq_r(x)p_r(y), and forms a global source summary before combining it with query factors. Our original two-block CP shares this separable form but retains query-dependent local neighborhoods. The inner aggregation therefore generally depends on the query and cannot be replaced by a single global summary shared across queries. Moreover, the original LNO classifies its construction as non-graph-based; our implementation couples functional factors with local connectivity, integration weights, and sparse graph aggregation.

DeepONet [2] directly represents the operator output, typically as G(a)(x)=∑rBr(a(s1),…,a(sm))Tr(x).\mathcal G(a)(x)=\sum_rB_r\left(a(s_1),\ldots,a(s_m)\right)T_r(x). Its branch network encodes sampled input-function values, while its trunk network encodes output locations. Our factors instead parameterize a graph-integral kernel acting on source features, rather than directly combining branch and trunk representations to produce the final output. The methods share a product-sum structure, but differ in factor inputs and where that structure is applied.

AuthorsOpenReview ↗

Response to Reviewer xJJA Part 2

The broader comparison is summarized below.

MethodCore representationFactor inputsRelationship to FTD-GNO
LNO [1]Integral kernel: κ(x,y)=∑rqr(x)pr(y)\kappa(x,y)=\sum_r q_r(x)p_r(y)Query position xx and source position yyThe original construction uses two-endpoint separation and a global source summary. FTD-GNO implements related separation within local graph integrals and supports multiple input blocks with CP, TT, and Tucker parameterizations.
DeepONet [2]Operator output: G(a)(x)=∑rBr(a)Tr(x)\mathcal G(a)(x)=\sum_r B_r(a)T_r(x)Sampled input-function values for the branch network; output position for the trunk networkDirectly combines input-function and position encodings to represent the operator output. FTD-GNO parameterizes an internal integral kernel and supports different tensor formats over multiple kernel-input blocks. Our factors represent an internal integral kernel rather than the final operator output.
Galerkin-type attention [3]Feature interaction: U=Q(KTV)U=Q(K^T V)Queries, keys, and values are typically projected from current featuresUses global feature aggregation. FTD-GNO retains local graph neighborhoods and supports coordinate kernels and multivariable kernels containing field/state inputs.
PILNO [4]Low-rank kernel within an encoder-decoder architecture: κ(x,y;a)=∑rqr(x,a(x))pr(y,a(y))\kappa(x,y;a)=\sum_rq_r(x,a(x))p_r(y,a(y))Position and physical coefficient jointly processed within each endpointCombines joint-endpoint separation with an encoder-decoder architecture and physics-informed training. FTD-GNO can further parameterize position and field/state separately and organize multivariable interactions using CP, TT, or Tucker.
FTD-GNOCP examples: κ(x,y)=∑rfr(x)hr(y)\kappa(x,y)=\sum_rf_r(x)h_r(y); κ(x,y,a(y))=∑rfr(x)hr(y)tr(a(y))\kappa(x,y,a(y))=\sum_rf_r(x)h_r(y)t_r(a(y)); κ(x,y,a(x),a(y))=∑rfr(x)gr(a(x))hr(y)tr(a(y))\kappa(x,y,a(x),a(y))=\sum_rf_r(x)g_r(a(x))h_r(y)t_r(a(y)). TT and Tucker extensions are also supported.Positions and, depending on the configuration, separately modeled physical-coefficient or hidden-state blocksThe two-block configuration shares established endpoint-separation principles, with an emphasis on reducing GPU-memory overhead through node-factor reuse and sparse local aggregation. Multivariable configurations support finer input-block separation and different interaction structures through CP, TT, and Tucker.

The framework's extension therefore concerns not only local neighborhoods, but also functional tensor parameterizations over multiple input blocks. Four-block CP imposes additional within-endpoint product structure relative to joint-endpoint kernels, while TT and Tucker provide other factor organizations.

To provide an initial evaluation of this additional separation, we compare joint-endpoint and four-block CP kernels within the same local graph framework. The results are reported as follows:

Kernel parameterizationErrors at 16 ×\times 16Errors at 32 ×\times 32
Local joint-endpoint0.2982 ±\pm 0.00050.2969 ±\pm 0.0027
CP-GNO0.2974 ±\pm 0.00070.2944 ±\pm 0.0006

The observed mean errors of our four-block CP are lower at both resolutions. This provides preliminary empirical support for further position-state separation. It is not a complete PILNO reproduction. So we must clarify that it is not superior to the complete LNO or PILNO architectures.

2. Implementation contribution: reorganizing kernel evaluation and neighborhood aggregation to reduce GPU-memory costs.

Our focus is not to design a complete operator architecture that outperforms alternatives on every PDE, but to reduce the resource cost of graph-integral components. We evaluate and reuse functional factors at nodes, combining sparse neighborhood aggregation with contraction reordering to reduce repeated edge-wise kernel-network computation and selected edge-wise intermediates. For example, the current CP path avoids explicitly materializing a B×E×R×CB\times E\times R\times C gathered edge-message tensor and instead uses node-indexed weighted and aggregated features. This implementation preserves local graph interactions and applies to graph-only models and Fourier-graph hybrids retaining graph-integral components.

3. Empirical contribution: demonstrating practical memory savings in the tested graph-integral implementations.

The predictive experiments evaluate the practical behavior of this resource optimization. The hybrid results also show that factorized variants remain trainable in some configurations where their unfactorized counterparts encounter OOM. These experiments support the practical resource value of functional-factor representations and their computational implementation for graph-integral components, rather than universal predictive superiority over other operator architectures.

We will therefore position our main contribution as functional tensor kernel parameterization, finer input-block organization, and resource-oriented implementation through node-factor reuse and sparse contraction within local graph integrals.

AuthorsOpenReview ↗

Response to Reviewer xJJA Part 3

W2.1: Computational complexity: The paper claims that tensor factorization reduces the computational complexity and memory consumption, but the actual complexity depends on how the neighborhood aggregation is implemented. If the implementation still constructs edge-wise intermediate features, the memory usage should depend on the number of edges. I think the paper needs to explain more clearly which operations are avoided and provide a more complete complexity analysis.

Thank you for raising this concern. We agree that actual computational and memory costs depend on the aggregation implementation and cannot be inferred from the kernel factorization alone.

Which operations change. Unfactorized GNO evaluates its kernel network on selected edges. Our factorized implementation instead evaluates factors at nodes and reuses them across incident edges, reducing repeated edge-wise kernel-network computation. For a scalar CP kernel shared across channels, the computation is

Xj,r,c=Pj,rVj,c,Yi,r,c=∑jSijXj,r,c,Ui,c=∑rQi,rYi,r,c,X_{j,r,c}=P_{j,r}V_{j,c},\qquad Y_{i,r,c}=\sum_jS_{ij}X_{j,r,c},\qquad U_{i,c}=\sum_rQ_{i,r}Y_{i,r,c},

where SijS_{ij} contains neighborhood connectivity and integration weights. The current CP path flattens the source-weighted features, applies CSR sparse matrix multiplication for neighborhood aggregation, and then contracts with query factors. Let B,I,E,R,CB,I,E,R,C denote batch size, node count, edge count, rank, and feature-channel count; dd denotes the number of input blocks, and W,wW,w denote the hidden widths of the unfactorized kernel network and factor networks. The principal implementation differences are:

Operation or storageUnfactorized GNOCurrent reordered CP implementation
Kernel-network evaluationEvaluate the kernel MLP on the selected EE edgesEvaluate factor networks at their associated nodes and reuse outputs across incident edges
Principal kernel-network hidden activationsTypically B×E×WB\times E\times W per layerTypically B×I×wB\times I\times w per factor hidden layer
Kernel/factor outputsEdge-kernel matrices: B×E×C2B\times E\times C^2Node factors for dd input blocks: O(BIdR)O(BIdR)
Aggregation-related featuresGathered source features and edge messages can each have shape B×E×CB\times E\times CSource-weighted features and pre-contraction aggregation outputs each have shape B×I×R×CB\times I\times R\times C
Explicit connectivity and weightsO(I+E)O(I+E) for a batch-shared CSR graphSame

What is avoided and what remains. The CP path avoids explicitly constructing a B×E×R×CB\times E\times R\times C gathered edge-message tensor. The practical benefit comes from node-factor reuse and avoidance of selected edge-wise intermediates. The implementation remains edge dependent: sparse aggregation requires O(BERC)O(BERC) arithmetic, query-side contraction requires O(BIRC)O(BIRC) arithmetic, and batch-shared CSR connectivity requires O(I+E)O(I+E) storage. A shared scalar kernel and node-indexed factors therefore do not make aggregation independent of edge count or guarantee fewer total operations in every configuration.

We will supplement the revision with this implementation and cost accounting, and restrict the conclusion accordingly: node-factor reuse and sparse contraction reduce repeated edge-wise kernel-network computation and selected intermediate storage in the tested implementations, rather than eliminating edge-dependent aggregation or graph storage.

Reviewer_tiJuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript replaces a GNO's edge-wise kernel MLP with neural factors in CP, tensor-train, and Tucker representations. Reordered contractions reuse node-dependent factors and reduce edge-wise kernel activations. Experiments cover Darcy flow, Navier–Stokes, irregular-mesh elasticity, and Fourier–graph hybrids. Proposition C.1 establishes kernel approximation with increasing rank and network capacity.

Critical strengths
  • Substantial reported memory savings in several tested settings. Tables 2, 4, and 10 show lower peak memory on regular grids and irregular meshes, including configurations where the tested unfactorized models run out of memory. This is the strongest empirical contribution.
  • Useful controls with a limited scope. The parameter-matched LR-GNO comparison shows that fewer parameters alone do not explain the memory savings. FNOGNO retains a common Fourier backbone and changes only the final graph kernel, supporting applicability within that hybrid.
  • A correct qualitative approximation result. Proposition C.1 has a sound Stone–Weierstrass proof. This standard density result confirms kernel approximability with unrestricted rank and network capacity, without a low-rank efficiency guarantee.
Critical weaknesses
  1. The central complexity advantage is derived against an incorrect baseline (Section 3.2.3, Table 1, and Appendix D).

    A standard GNO evaluates κ(xi,xj,a(xi),a(xj))\kappa(x_i,x_j,a(x_i),a(x_j)) on EE selected node pairs. The values a(xi)a(x_i) are attached to xix_i; they do not create independent Cartesian sampling axes. Consequently, four input blocks do not require constructing an N4N^4 tensor for an NN-node graph. The O(Id)O(I^d) GNO memory baseline in Table 1 and Appendix D.2 therefore does not describe ordinary graph evaluation. The claimed exponential-to-linear reduction does not follow from this comparison.

    The parameter argument has a related problem. Increasing input dimension does not itself produce exponentially many MLP parameters: with fixed hidden widths, it enlarges the first layer linearly. No approximation assumptions or rates are supplied to establish the asserted exponential width requirement. Appendix D.1 also mixes shared factor-network trunks with rank-dependent output heads. A CP factor MLP with mm inputs, ll hidden layers of width ww, and RR outputs has mw+(l−1)w2+wRmw+(l-1)w^2+wR weights, plus biases; rank does not multiply every hidden-layer cost. TT's internal R2R^2 heads and Tucker's RdR^d core require corresponding counts.

    Table 1 needs to be rebuilt around NN, EE, channels, ranks, and the implemented networks. Proposition C.1 does not repair this argument: it permits rank and capacity to increase without controlling their growth. It cannot justify equivalent expressivity at fixed rank or removal of the curse of dimensionality. These claims must be withdrawn or supported by a different analysis.

  2. The contraction identities do not establish the claimed node-only total memory (equations (19)–(20), Table 8, and Section 4.2).

    The CP rearrangement is algebraically correct. For a scalar kernel and normalized adjacency SS,

    S\operatorname{diag}(s_r)v.$$ However, the inner neighborhood sum remains query dependent. Ordinary edge-by-edge aggregation performs $O(ERC)$ arithmetic for $C$ channels. Explicit connectivity adds $O(E)$ storage; implicit or streamed neighborhoods can avoid persistent edge storage, but require a specified algorithm. Moving factors outside the sum alone establishes neither edge-independent aggregation nor node-only peak memory. Section 4.2's assertion of memory depending solely on node count is also inconsistent with the residual dense-topology risks acknowledged in Appendix F. The missing implementation details are essential to assess the contribution. Specify whether kernels are scalar, channel-diagonal, or full channel matrices; factor sharing across layers/channels; and forward/backward intermediate shapes. Account for neighborhood construction, topology/search structures, saved activations, and backward buffers. In the four-block TT ordering, alternating query/source factors can introduce rank-product intermediates absent from a single-kernel cost count. A graph-aware analysis and a compact memory profile are needed to substantiate the claimed scaling.
  3. The experiments do not establish the methodological advance over learned separable kernels (Section 2 and Appendix E.2.2).

    The neural operator paper: Kovachki N, Li Z, Liu B, et al. Neural Operator: Learning Maps between Function Spaces with Applications to PDEs. Journal of Machine Learning Research, 2023, 24(89): 1-97; already cited in the manuscript, introduces neural query/source factors and source-first contraction, including a vector-valued extension. Coordinate-only CP uses this established parameterization. Local masking and additional field factors can change the model; in particular, a locally masked kernel matrix need not have rank RR. The paper must identify and substantiate what these additions contribute beyond the established mechanism.

    LR-GNO does not resolve this issue. It compresses MLP weights while retaining edge-wise evaluation, whereas FTD changes both kernel parameterization and contraction. Thus, it controls parameter count but does not isolate contraction-order savings or compare against an efficiently evaluated separable kernel. A direct-versus-reordered evaluation of the same factorized kernel would address the former question.

    The tensor-format evidence is also insufficient. Equations (23)–(24) correctly establish two-block CP/TT equivalence; identical results in that regime are not evidence for distinct architectural benefits. For the displayed scalar kernel with unrestricted affine factor outputs, a two-mode Tucker core is also absorbable into a factor. State the blocks used in every benchmark and report predictive accuracy and cost in the existing three-/four-block settings where the formats differ. Table 14 supplies only memory, and its four-block Tucker model uses 861.5 MB versus GNO's 426.7 MB. These memory-only comparisons do not establish the claimed superiority or optimal balance for complex multivariable tasks.

  4. The accuracy–resource evidence does not support a general training-efficiency advantage (Tables 3 and 10–15).

    Table 11 is decisive for interpreting the Darcy speedup. The reported 3,000-epoch CP-GNO run obtains error 0.1877 in 6,423.93 s, versus the 300-epoch GNO's 0.1872 in 6,542.28 s. At these nearly matched errors, the roughly tenfold equal-epoch gain becomes only about 1.02-fold. The authors acknowledge this result, but it must constrain the overall efficiency conclusion. Elasticity provides stronger evidence: 300-epoch CP-GNO outperforms the 100-epoch GNO while retaining an approximately elevenfold time advantage. Together, these results demonstrate a task-dependent trade-off.

    Establish the accuracy–time–memory trade-off for the existing comparisons using a stated checkpoint rule and repeated-run accuracy/time statistics. Table 15's memory variability cannot establish predictive or timing stability. The synchronized total-training timing protocol is useful, but a breakdown is needed to determine how much improvement comes from kernel evaluation, aggregation, and graph handling.

    The error metric must also be resolved. Equation (22) defines a mean squared relative norm, while tables are labeled relative L2L^2 error. Specify the implemented training loss and reported test statistic separately and verify all labels. This ambiguity prevents a precise interpretation of the accuracy being traded for efficiency.

Questions for authors

Both questions request clarification of the submitted method and existing results.

  1. Implemented memory scaling. What are the forward/backward intermediate tensor shapes and resulting memory dependence on node count NN, edge count EE, rank RR, and channel count CC, including saved activations and connectivity/neighborhood storage? This would determine whether the claimed node-only memory scaling holds for the implemented models.

  2. Contribution demonstrated by the benchmarks. Which kernel input blocks are used in each predictive benchmark, and what additional methodological contribution beyond the learned separable-kernel construction of Kovachki et al. (2023, Section 4.2) is demonstrated by those existing results?

10 comments
AuthorsOpenReview ↗

Response to Reviewer tiJu Part 1

We sincerely thank the reviewer for the valuable and constructive comments! We have carefully incorporated additional discussions and clarifications according to your comments. For easy reference, a list of responses to the reviewers' comments has been compiled, and the corresponding changes have been highlighted in blue in the main text for easy reference of our revision.

W1: The central complexity advantage is derived against an incorrect baseline (Section 3.2.3, Table 1, and Appendix D).

W1.1: A standard GNO evaluates κ(xi,xj,a(xi),a(xj))\kappa(x_i,x_j,a(x_i),a(x_j)) on EE selected node pairs. The values a(xi)a(x_i) are attached to xix_i; they do not create independent Cartesian sampling axes. Consequently, four input blocks do not require constructing an N4N^4 tensor for an NN-node graph. The O(Id)O(I^d) GNO memory baseline in Table 1 and Appendix D.2 therefore does not describe ordinary graph evaluation. The claimed exponential-to-linear reduction does not follow from this comparison.

Thank you for identifying this issue. We agree that physical field values are attached to spatial nodes rather than independently sampled axes. For the kernel κ(x,y,a(x),a(y))\kappa(x,y,a(x),a(y)), with IxI_x query nodes and IyI_y source nodes, the actual number of interactions is E=∑i=1Ix∣N(xi)∣≤IxIy.E=\sum_{i=1}^{I_x}|\mathcal N(x_i)|\leq I_xI_y. To simplify the subsequent table, we set Ix=Iy=II_x=I_y=I. A complete graph including self-interactions contains E=I2E=I^2 node pairs, not I4I^4. Ordinary GNO evaluates the kernel on selected pairs without constructing a tensor containing every independent combination of the input blocks.

Thus, IdI^d can describe the size of an independently sampled tensor-product table, but not the storage baseline for ordinary GNO graph evaluation. We will remove this baseline and the resulting unconditional exponential-to-linear claim from Section 3.2.3, Table 1, and Appendix D. The revised analysis will use node count, edge count, channels, rank, and actual intermediate tensors. We will explain the benefit as reducing repeated edge-wise kernel-network computation and selected edge-wise intermediate storage, rather than eliminating edge dependence.

W1.2: The parameter argument has a related problem. Increasing input dimension does not itself produce exponentially many MLP parameters: with fixed hidden widths, it enlarges the first layer linearly. No approximation assumptions or rates are supplied to establish the asserted exponential width requirement. Appendix D.1 also mixes shared factor-network trunks with rank-dependent output heads. A CP factor MLP with mm inputs, ll hidden layers of width ww, and RR outputs has mw+(l−1)w2+wRmw+(l-1)w^2+wR weights, plus biases; rank does not multiply every hidden-layer cost. TT's internal R2R^2 heads and Tucker's RdR^d core require corresponding counts.

We agree that, increasing input dimension does not itself produce exponentially many MLP parameters. Moreover, rank components share each factor network's hidden trunk. Rank must therefore be reflected in the output head rather than multiplied across the cost of all hidden layers. For an MLP with input dimension mm, l≥1l\geq1 fully connected hidden layers of width ww, and output dimension oo, define the parameter count including biases as P(m,w,l,o)=mw+(l−1)w2+wo+lw+o,P(m,w,l,o)=mw+(l-1)w^2+wo+lw+o, and the leading forward multiply-accumulate count as F(m,w,l,o)=mw+(l−1)w2+wo.F(m,w,l,o)=mw+(l-1)w^2+wo.

Let dd denote the number of kernel-input blocks, mim_i their input dimensions, and m=∑i=1dmim=\sum_{i=1}^{d}m_i. Let W,LW,L and w,lw,l denote the hidden widths and hidden-layer counts of the unfactorized kernel network and factor networks, respectively. The revised Table 1 is:

MethodSingle kernel evaluationTrainable kernel parametersPrincipal kernel-representation storageExplicit graph connectivity
GNOO(F(m,W,L,C2))O(F(m,W,L,C^2))P(m,W,L,C2)P(m,W,L,C^2)O(BEC2)O(BEC^2) if all edge-kernel outputs are materializedO(I+E)O(I+E)
CP-GNOO(∑i=1dF(mi,w,l,R)+dR)O\left(\sum_{i=1}^{d}F(m_i,w,l,R)+dR\right)∑i=1dP(mi,w,l,R)\sum_{i=1}^{d}P(m_i,w,l,R)Node factors: O(BIdR)O(BIdR)O(I+E)O(I+E)
TT-GNOO(∑i=1dF(mi,w,l,Ri−1Ri)+∑i=1dRi−1Ri)O\left(\sum_{i=1}^{d}F(m_i,w,l,R_{i-1}R_i)+\sum_{i=1}^{d}R_{i-1}R_i\right)∑i=1dP(mi,w,l,Ri−1Ri)\sum_{i=1}^{d}P(m_i,w,l,R_{i-1}R_i)Node factors: O(BI∑i=1dRi−1Ri)O\left(BI\sum_{i=1}^{d}R_{i-1}R_i\right)O(I+E)O(I+E)
Tucker-GNOO(∑i=1dF(mi,w,l,R)+dR+Rd)O\left(\sum_{i=1}^{d}F(m_i,w,l,R)+dR+R^d\right)∑i=1dP(mi,w,l,R)+Rd\sum_{i=1}^{d}P(m_i,w,l,R)+R^dNode factors: O(BIdR)O(BIdR), plus the RdR^d-entry coreO(I+E)O(I+E)

Here BB denotes batch size. The table assumes II query and source nodes, input blocks evaluated at their corresponding nodes, and an explicit CSR graph shared across the batch. For d≥2d\geq2, TT has R0=Rd=1R_0=R_d=1; with equal internal ranks RR, ∑k=1dRk−1Rk=2R+(d−2)R2.\sum_{k=1}^{d}R_{k-1}R_k=2R+(d-2)R^2. The kernel-representation column is not a bound on total training memory, and single-kernel evaluation cost is not the cost of an entire layer that reuses node factors. Hidden activations, aggregation and contraction intermediates, backpropagation, optimizer states, and neighborhood search must be accounted for separately.

AuthorsOpenReview ↗

Response to Reviewer tiJu Part 2

W1.3: Table 1 needs to be rebuilt around NN, EE, channels, ranks, and the implemented networks. Proposition C.1 does not repair this argument: it permits rank and capacity to increase without controlling their growth. It cannot justify equivalent expressivity at fixed rank or removal of the curse of dimensionality. These claims must be withdrawn or supported by a different analysis.

Thank you for raising this point. As suggested, we will revise Table 1, which refers to the detail response of W1.2. We agree that Proposition C.1 establishes qualitative approximation when rank and network capacity are allowed to increase. It does not bound the rank, network size, or computational cost required to attain a specified error. Consequently, the proposition does not establish expressivity equivalence to an unfactorized kernel at fixed rank, general low-rank efficiency, or removal of the curse of dimensionality. We will remove or revise these extrapolations, retain the approximation result actually established by the proposition, and restrict practical resource benefits to the experimentally observed results. This will align the scope of the theoretical and empirical conclusions.

W2: The contraction identities do not establish the claimed node-only total memory (equations (19)-(20), Table 8, and Section 4.2).

W2.1: The CP rearrangement is algebraically correct. For a scalar kernel and normalized adjacency SS, Kv=∑r=1Rdiag⁡(qr) S diag⁡(sr) v.Kv=\sum_{r=1}^{R}\operatorname{diag}(q_r)\,S\,\operatorname{diag}(s_r)\,v. However, the inner neighborhood sum remains query dependent. Ordinary edge-by-edge aggregation performs O(ERC)O(ERC) arithmetic for CC channels. Explicit connectivity adds O(E)O(E) storage; implicit or streamed neighborhoods can avoid persistent edge storage, but require a specified algorithm. Moving factors outside the sum alone establishes neither edge-independent aggregation nor node-only peak memory. Section 4.2's assertion of memory depending solely on node count is also inconsistent with the residual dense-topology risks acknowledged in Appendix F.

Thank you for identifying this issue. We agree that the CP rearrangement is algebraically correct, but does not establish that total memory depends only on node count. For a scalar CP kernel shared across channels, the local action is

Ui,c=∑r=1RQi,r(∑jSijPj,rVj,c),U_{i,c}=\sum_{r=1}^{R}Q_{i,r}\left(\sum_j S_{ij}P_{j,r}V_{j,c}\right),

where SijS_{ij} contains the neighborhood mask and integration weights. The inner aggregation depends on the query through SijS_{ij} and cannot generally be replaced by a single source summary shared across all queries. Following the manuscript's notation, let the query and source sets each contain II nodes, corresponding to the reviewer's NN. Let B,E,R,CB,E,R,C denote batch size, edge count, rank, and feature-channel count. The current rank-expanded sparse aggregation requires O(BERC)O(BERC) arithmetic, followed by O(BIRC)O(BIRC) query-side contraction. An explicit CSR graph shared across the batch still requires O(I+E)O(I+E) storage.

We will therefore correct the corresponding wording in the original manuscript. Reordering enables node-factor reuse and reduces selected edge-wise computations and intermediates; it does not eliminate edge-dependent aggregation or graph-connectivity storage.

AuthorsOpenReview ↗

Response to Reviewer tiJu Part 3

W2.2: The missing implementation details are essential to assess the contribution. Specify whether kernels are scalar, channel-diagonal, or full channel matrices; factor sharing across layers/channels; and forward/backward intermediate shapes. Account for neighborhood construction, topology/search structures, saved activations, and backward buffers. In the four-block TT ordering, alternating query/source factors can introduce rank-product intermediates absent from a single-kernel cost count. A graph-aware analysis and a compact memory profile are needed to substantiate the claimed scaling.

Thank you for raising this point. The implementation details are listed below.

Kernel outputs and factor sharing. The unfactorized GNO outputs C2C^2 components per edge, reshaped to a C×CC\times C matrix and applied to the source feature by matrix-vector multiplication. The factorized models have no independent channel dimension in their factors and represent scalar spatial kernels shared across feature channels. Factor networks are separate across integral layers and input blocks; rank components share each factor network's hidden trunk and are generated through its output head. We will state these conventions explicitly and will not describe comparisons with different channel parameterizations as controlled experiments changing only contraction order.

For the two-block coordinate kernel used in the original predictive benchmarks, the principal forward tensors are:

ComponentUnfactorized GNOReordered shared-scalar CP
Node featuresB×I×CB\times I\times CB×I×CB\times I\times C
CSR connectivityI+1I+1 row pointers and EE source-node indices, plus associated weightsSame
Kernel-network inputsEdge inputs: B×E×mB\times E\times mEach node-block input: B×I×miB\times I\times m_i
Hidden activationsTypically B×E×WB\times E\times W per hidden layerTypically B×I×wB\times I\times w per factor hidden layer
Kernel or factor outputsB×E×C2B\times E\times C^2Q,PQ,P: each B×I×RB\times I\times R
Aggregation-related featuresGathered source features and edge messages can each have shape B×E×CB\times E\times CSource-weighted features and pre-contraction aggregation outputs each have shape B×I×R×CB\times I\times R\times C
Principal tensors retained for backpropagationSelected edge-network inputs and activations, edge-kernel matrices (B×E×C2B\times E\times C^2), and gathered source features (B×E×CB\times E\times C)Selected node-factor inputs and activations, factor outputs, source features, and the aggregation output required by query-side contraction

Here mim_i denotes each block's input dimension after the implemented positional encoding, m=∑imim=\sum_im_i, and W,wW,w denote the unfactorized kernel-network and factor-network hidden widths. These shapes allow batch-dependent evaluation; some coordinate-dependent tensors may be shared when geometry is common across samples.

Implemented aggregation path. The CP implementation flattens source-weighted features into X∈RI×(BRC),X\in\mathbb R^{I\times(BRC)}, computes Y=SXY=SX using CSR sparse matrix multiplication, and restores the batch, node, rank, and channel dimensions before contracting with query factors. This avoids explicitly constructing a B×E×R×CB\times E\times R\times C gathered edge-message tensor, while retaining graph connectivity and node-indexed weighted/aggregated features.

Additional memory components. The table identifies principal tensors, not a formula for total training peak memory. Actual backward retention depends on autograd and the sparse backend. Model parameters, gradients, optimizer states, layout conversions, temporary copies, and operator workspace must also be included.

TT and Tucker contraction paths. CP's intermediate analysis cannot be transferred unchanged to other formats. TT internal node factors contain Ri−1RiR_{i-1}R_i components. For four input blocks, intermediate sizes also depend on query/source variable ordering and the contraction path; alternating variables can introduce additional rank-product intermediates. Changing variable order can also change the fixed-rank parameterization. Tucker requires accounting for its RdR^d core and path-dependent intermediates.

We will supplement the revision with these implementation details and tensor-size analyses. Our corrected conclusion is that node-factor reuse and sparse contraction provide practical memory benefits in the tested implementations, while total peak memory still depends on node count, edge count, rank, channels, saved activations, neighborhood search, and backend implementation.

Reviewer_nvLLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FTD-GNO parameterizes the GNO kernel as a low-rank inner product between a query embedding and a source embedding, so the neighborhood sum can be seen as a local Galerkin-type attention. As a result, on Darcy flow, Navier–Stokes, and elasticity, peak memory stays nearly constant as the radius grows.

Critical strengths
  • Moving kernel from edges to nodes produces the expected scaling: Tables 4–5 show memory unchanged when the radius doubles and roughly linear growth with node count.
Critical weaknesses
  • The core mechanism is the known trick behind the Galerkin Transformer, but the paper does not compares against it.

  • Experiments are limited in scale and difficulty. All tasks are 2D, and graph-only FTD-GNO has about 55% error on NS, where plain FNO is both more accurate and lighter.

Questions for authors
  • Can you add the Galerkin Transformer and other linear attention NO baseline like Transolver to compare the results? You can consider to discuss what the “localizing” can bring and cannot capture compared with them. And large irregular-mesh setting, where GNO memory truly matters, is encouraged to be evaluated.
4 comments
AuthorsOpenReview ↗

Response to Reviewer nvLL Part 1

We sincerely thank the reviewer for the valuable and constructive comments! We have carefully incorporated additional discussions and clarifications according to your comments. For easy reference, a list of responses to the reviewers' comments has been compiled, and the corresponding changes have been highlighted in blue in the main text for easy reference of our revision.

W1: The core mechanism is the known trick behind the Galerkin Transformer, but the paper does not compares against it.

Q1.1: Can you add the Galerkin Transformer and other linear attention NO baseline like Transolver to compare the results? You can consider to discuss what the "localizing" can bring and cannot capture compared with them.

Thanks for this constructive comment. We agree that these are relevant efficient neural operators, and that Galerkin Transformer already employs source-first contraction. Our contribution lies not in this contraction identity itself, but in investigating how functional tensor factorization can reduce the memory cost of graph-integral kernel evaluation.

Relationship to Galerkin Transformer. Omitting normalization, scaling, and projection details, Galerkin-type attention can be summarized as U=Q(KTV),U=Q(K^T V), where the source summary KTVK^T V is shared across all queries [1]. Our CP graph integral has a related separable structure but retains prescribed local neighborhoods. The inner aggregation generally remains query dependent and cannot be replaced by a single global source summary shared across all queries. Our implementation focuses on evaluating and reusing functional factors at nodes, and using sparse aggregation to reduce repeated edge-wise kernel-network computation and selected edge-wise intermediates while preserving this local graph-integral structure. This establishes a clear connection to global Galerkin-type attention, without making the complete architectures equivalent.

Furthermore, the original experiments use the coordinate-dependent kernel. Its functional factors are generated from position inputs, whereas the queries, keys, and values in Galerkin-type attention are typically projected from current features. This further distinguishes the evaluated coordinate kernels from feature-dependent attention.

Relationship to Transolver. Transolver uses learned soft assignments to aggregate nodes into slice tokens, applies attention between those tokens, and maps the resulting information back to nodes [2]. Our method instead evaluates graph integrals directly over prescribed geometric neighborhoods, without introducing slice tokens or inter-slice attention. The methods consequently differ in interaction range, information compression, and overall architecture.

Purpose and scope of the existing comparisons. Our study focuses on whether functional tensor factorization and contraction reordering reduce the resource cost of kernel evaluation within a GNO graph-integral framework. Accordingly, the current experiments primarily evaluate the resource implications of changes to the parameterization and computation of the graph-integral component. By contrast, comparisons with Galerkin Transformer or Transolver evaluate complete operator architectures and simultaneously change feature dependence, interaction range, normalization, information aggregation, and other network components. Such comparisons are important for assessing practical competitiveness, but do not isolate the contribution of FTD to a graph-integral component. Conversely, our existing within-framework comparisons cannot substitute for that architecture-level evaluation. We therefore recognize the value of these architecture-level baselines while distinguishing their evaluation scope from our primary research question.

What localization provides and does not guarantee. Localization explicitly restricts interactions to prescribed geometric neighborhoods and allows the graph to control information propagation within each layer. Combined with functional-factor reuse, it reduces selected edge-wise computations and intermediates in the evaluated GNO implementations. However, localization does not automatically imply better prediction or lower total memory than efficient global attention. A single local layer cannot directly communicate beyond its neighborhood; stacking layers expands the receptive field but is not equivalent to one-layer global interaction. Graph-connectivity storage and neighborhood aggregation also remain edge dependent.

We will revise the corresponding content and present our method as a representation and computational improvement for local graph-integral kernels. The existing experiments demonstrate resource benefits relative to the evaluated unfactorized graph-integral implementations. Direct comparisons with Galerkin Transformer and Transolver, as well as potential combinations with these architectures, remain directions for further investigation in the future.

AuthorsOpenReview ↗

Response to Reviewer nvLL Part 2

W2: Experiments are limited in scale and difficulty. All tasks are 2D, and graph-only FTD-GNO has about 55% error on NS, where plain FNO is both more accurate and lighter.

Thank you for raising this point. We agree that all current experiments are two-dimensional and that the graph-only NS results limit the conclusions supported by our evaluation. We will explicitly distinguish two objectives: reducing the resource cost of graph-integral kernels and improving the predictive accuracy of a complete PDE solver. Our contribution lies in investigating how functional tensor factorization can reduce the memory cost of graph-integral kernel evaluation. In addition, we have conducted three-dimensional large-scale irregular ShapeNet-Car CFD experiments, as detailed in response to Q1.2.

The approximately 55% NS error. At 16×1616\times16, GNO and CP-GNO obtain relative l2l_2 errors of 0.5482 and 0.5538, respectively, whereas FNO achieves 0.0789 with lower peak memory. Thus, the evaluated graph-only configurations are not competitive on this task. The similar errors of GNO and CP-GNO indicate that this issue is not unique to the factorized model, but the existing results do not establish its specific cause.

The role of FTD in hybrid architectures. To examine whether the resource benefit is restricted to graph-only models, the manuscript also reports FNOGNO and its factorized variants. FNOGNO combines Fourier layers with a graph-integral component. These experiments examine the applicability of FTD within a Fourier-graph hybrid architecture and report the predictive error and resource cost of the complete models.

ModelNS resolutionRelative l2l_2 errorPeak memory (MB)
GNO16×1616\times160.5482420.61
CP-GNO16×1616\times160.5538181.38
FNO16×1616\times160.078992.92
FNOGNO16×1616\times160.1152276.77
CP-FNOGNO16×1616\times160.1156123.37
FNO128×128128\times1280.07831934.74
FNOGNO128×128128\times128OOMOOM
CP-FNOGNO128×128128\times1280.12036104.55

At 16×1616\times16, CP-FNOGNO retains an error close to FNOGNO while reducing reported peak memory from 276.77 MB to 123.37 MB, approximately a 55.4% reduction. At 128×128128\times128, unfactorized FNOGNO encounters OOM under the tested memory budget, whereas CP-FNOGNO completes training. These results support applying FTD to reduce the resource cost of a retained graph-integral component in a hybrid architecture, rather than restricting its use to graph-only models.

Nevertheless, FNO remains both more accurate and more memory-efficient in these NS settings. We do not claim that FTD-GNO or FTD-FNOGNO outperforms FNO. The supported conclusion is that functional factorization and contraction reordering reduce resource costs in the evaluated graph-integral implementations. This benefit does not establish general superiority over other architectures or guarantee improved predictive accuracy.

Furthermore, the evaluated FNOGNO and FTD-FNOGNO configurations achieve substantially lower NS errors than the graph-only configurations, highlighting the importance of the overall operator architecture. Although these comparisons do not attribute the entire improvement specifically to Fourier layers, they provide empirical motivation for further investigation of global information mixing combined with local graph integrals. Future research will examine more competitive hybrid operator architectures and the applicability of FTD within them.

Q1.2: And large irregular-mesh setting, where GNO memory truly matters, is encouraged to be evaluated.

Thank you for this suggestion. We have added a ShapeNet-Car CFD experiment. Each case contains 32186 irregular computational points, comprising 3682 surface points and 28504 non-surface volume points-approximately 33.11 times the node count of the elasticity benchmark. We retain all computational points, jointly predict volume velocity and surface pressure, and use a fixed split of 739 training, 50 validation, and 100 test cases. The results are as follows:

ModelEpochsVelocity errorPressure errorPeak memory (MB)
GNOOOMOOMOOMOOM
CP-GNO2000.08480.276421904.36
TT-GNO2000.08480.276421904.36
Tucker-GNO2000.08580.233621907.31

The tested FTD-GNO configurations complete training, whereas the current GNO implementation encounters OOM when processing a full case. These results extend the training-feasibility evidence from small two-dimensional irregular meshes to a larger three-dimensional CFD setting. The comparison uses the same data split, input features, and prescribed neighborhood parameters. Moreover, our experiment differs from the original Transolver study in architecture, interaction mechanism, and training and evaluation protocols. Its published results therefore cannot be used for a direct ranking under our protocol. We will include the experiment results in the revision, explicitly restricting the OOM observation to the tested GNO implementation.

AuthorsOpenReview ↗

Response to Reviewer nvLL Part 3

Should you require any further information, please do not hesitate to let us know. We would greatly appreciate your feedback.

References

[1] Cao, S. Choose a Transformer: Fourier or Galerkin. NeurIPS, 2021.

[2] Wu, H., Luo, H., Wang, H., Wang, J., and Long, M. Transolver: A Fast Transformer Solver for PDEs on General Geometries. ICML, 2024.

AuthorsOpenReview ↗

Response to Reviewer nvLL: Further Supplement to W2 Response and Additional Experimental Results

W2: Experiments are limited in scale and difficulty. All tasks are 2D, and graph-only FTD-GNO has about 55% error on NS, where plain FNO is both more accurate and lighter.

We would like to make further additions to the W2 response in Part 2. The primary objective of this paper is to reduce the memory cost of graph integral components. Therefore, GNO and FTD-GNO, as well as FNOGNO and FTD-FNOGNO, are direct comparisons for evaluating the resource impact of function decomposition within their respective frameworks. The comparison with FNO answers another complementary question, which is the actual competitiveness of the complete architecture. In the paper, we use it as a reference to demonstrate that optimizing the graph neural operator architecture can increase model prediction accuracy.

  1. Memory benefits. As explained in detail in our response to W2, the results sufficiently demonstrate that our method can significantly reduce the memory consumption of graph-integral models.
  2. Task-dependent predictive performance. Predictive competitiveness depends on the specific task, and our model outperforms FNO in some cases, such as Darcy. Specifically, Table 3 reports a relative l2l_2 error of 0.1042 for FNO on Darcy and 0.1026 for CP-FNOGNO. Thus, our method has a slightly lower error compared to FNO in this setting.
  3. Additional architectural experiment. Beyond the primary objective of reducing graph-integral memory costs, we further investigated predictive competitiveness by increasing model complexity. The results are as follows:
ModelDarcy relative l2l_2 errorNS relative l2l_2 error
FNO0.10420.0789
CP-FNOGNO0.10260.1156
CP-FNOGNO0.09520.0723

The new configuration achieves relative l2l_2 errors of 0.0952 on Darcy and 0.0723 on NS, both numerically lower than the corresponding FNO errors. These results indicate that adjusting the complexity of the graph integration operator can improve the prediction accuracy of the model.

Reviewer_4gffOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FTD-GNO, a framework that parameterizes graph neural operator kernels through functional CP, Tensor-Train, or Tucker decompositions. Small networks represent individual input-variable blocks, and the resulting factorization enables neighborhood aggregation to be reordered, reducing the need to construct full edge-wise kernel tensors. The paper presents a complexity analysis and a kernel approximation result. Experiments cover Darcy flow, Navier–Stokes, and elasticity on irregular meshes, as well as Fourier–graph hybrid architectures, resolution transfer, and architectural ablations. The results show substantial memory savings and lower per-epoch training costs relative to the tested unfactorized baselines, with accuracy trade-offs that vary across tasks.

Critical strengths
  1. In Table 4, at a resolution of 32-by-32 and a neighborhood radius of 0.16, CP/TT-GNO with rank 8 reduces peak memory usage from approximately 19,733 MB to 684 MB. The factorized models also remain trainable in settings where the unfactorized baseline runs out of memory. The parameter-matched comparison in Table 9 provides further support: LR-GNO and CP-GNO have nearly identical parameter counts but markedly different memory requirements. These results demonstrate the practical benefit of reorganizing kernel evaluation beyond reducing model size alone.

  2. The method reduces memory usage in both graph-only and Fourier–graph architectures and accommodates sample-dependent irregular meshes. The elasticity results are particularly promising: CP-GNO trained for 300 epochs achieves lower error than GNO trained for 100 epochs while requiring substantially less total training time and memory. These findings suggest that the method could serve as a useful component in operator-learning systems.

Critical weaknesses
  1. The Low-rank Neural Operator described in Section 4.2 of [1] already represents a continuous kernel through separate source and target factors, reorganizing integration into source-side aggregation followed by target-side combination. PILNO [2] extends this formulation by incorporating position and local physical coefficients into both endpoint factors. Meanwhile, F-INR [3], which the submission cites, already introduces functional CP, Tensor-Train, and Tucker parameterizations based on small factor networks.

    FTD-GNO differs through its use of local neighborhoods and a finer separation of input-variable blocks, both of which may be valuable. However, comparisons with an unfactorized GNO and a weight-compressed kernel network do not isolate the contribution of these design choices. A particularly relevant missing baseline is a separable kernel whose source and target networks each jointly process position and physical field values, evaluated on the same graph. Without this comparison, it remains difficult to determine whether the proposed combination constitutes a substantial methodological advance.

  2. The claimed reduction in memory growth from exponential to linear treats source positions, target positions, and their associated physical field values as independently enumerated variables. In an actual graph sample, however, physical field values are attached to nodes, and the baseline evaluates existing node pairs rather than every independent combination of these quantities. Moreover, the proposed reordering still requires aggregation over target-dependent neighborhoods. Although it can avoid expensive edge-wise kernel intermediates, it does not, by itself, eliminate edge-dependent aggregation or graph storage.

    The distinction between per-epoch speed and time to comparable accuracy is also important. In Table 11, CP-GNO takes approximately 647 seconds at 300 epochs, compared with 6542 seconds for GNO, but produces higher error. Achieving comparable error requires approximately 6424 seconds for CP-GNO, leaving little overall training-time advantage in this setting. The authors acknowledge this outcome, and it does not undermine the favorable elasticity results. Nevertheless, a corrected complexity model and systematic comparisons at matched accuracy are needed to support the broader efficiency claims.

[1] Kovachki, Nikola, et al. "Neural operator: Learning maps between function spaces with applications to pdes." Journal of Machine Learning Research 24.89 (2023): 1-97.

[2] Schaffer, Sebastian, and Lukas Exl. "Physics-informed low-rank neural operators with application to parametric elliptic pdes." arXiv preprint arXiv:2509.07687 (2025).

[3] Vemuri, Sai Karthikeya, Tim Büchner, and Joachim Denzler. "F-INR: Functional tensor decomposition for implicit neural representations." 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2026.

Questions for authors
  1. Which baseline tensors incur the claimed exponential storage cost, given that physical field values are attached to nodes? Please distinguish the memory required for node features, graph connectivity, kernel intermediates, and quantities retained for backpropagation.

  2. Which kernel input blocks are used in each main benchmark? As the appendix explains, CP and Tensor-Train coincide for the two-variable kernel. Which results therefore demonstrate the predictive benefits of genuinely multivariable decompositions?

  3. Under identical neighborhoods and training budgets, how does FTD-GNO compare with a local Low-rank Neural Operator or a separable kernel whose endpoint networks jointly process position and physical field values? This comparison would directly inform my assessment of the additional variable separation.

  4. Beyond the examples in Tables 10–11, do the available learning curves show consistent reductions in the time required to reach comparable error? Please also report the number of independent runs and clarify whether variability in prediction error and runtime was measured alongside peak-memory variability.

[1] Kovachki, Nikola, et al. "Neural operator: Learning maps between function spaces with applications to pdes." Journal of Machine Learning Research 24.89 (2023): 1-97.

[2] Schaffer, Sebastian, and Lukas Exl. "Physics-informed low-rank neural operators with application to parametric elliptic pdes." arXiv preprint arXiv:2509.07687 (2025).

[3] Vemuri, Sai Karthikeya, Tim Büchner, and Joachim Denzler. "F-INR: Functional tensor decomposition for implicit neural representations." 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2026.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 4gff Part 1

We sincerely thank the reviewer for the valuable and constructive comments! We have carefully incorporated additional discussions and clarifications according to your comments.

For easy reference, a list of responses to the reviewers' comments has been compiled, and the corresponding changes have been highlighted in blue in the main text for easy reference of our revision.

W1: However, comparisons with an unfactorized GNO and a weight-compressed kernel network do not isolate the contribution of these design choices. A particularly relevant missing baseline is a separable kernel whose source and target networks each jointly process position and physical field values, evaluated on the same graph. Without this comparison, it remains difficult to determine whether the proposed combination constitutes a substantial methodological advance.

Thank you for raising this point. Indeed, learned endpoint separation and functional tensor decomposition have established precedents.

Relationship to LNO. LNO [1] directly parameterizes a continuous kernel using learned query and source factors: κLNO(x,y)=∑r=1Rqr(x)pr(y).\kappa_{\mathrm{LNO}}(x,y)=\sum_{r=1}^{R}q_r(x)p_r(y). We acknowledge this established mechanism. The specific LNO construction in Section 4.2 uses a two-endpoint factorization rather than developing separate functional CP, TT, and Tucker parameterizations over multiple coordinate and field/state blocks. Moreover, the original paper explicitly classifies LNO as non-graph-based. In contrast, our framework operates within local graph neighborhoods and reorganizes kernel evaluation and aggregation. This addresses the central objective of our work: reducing the memory overhead of graph-based models through functional tensor decomposition.

Relationship to PILNO. In its Darcy setting, PILNO [2] explicitly conditions each endpoint factor jointly on position and the local diffusion coefficient: κPILNO(x,y;a)=∑r=1Rqr(x,a(x))pr(y,a(y)).\kappa_{\mathrm{PILNO}}(x,y;a)=\sum_{r=1}^{R}q_r(x,a(x))p_r(y,a(y)). PILNO also specifies an encoder-decoder architecture, residual and layer-normalized feature updates, and physics-informed training with PDE-residual and boundary penalties. Its training framework includes B-spline sampling of input functions, and its PDE experiments also investigate optimization choices, including Muon. These components define a broader architecture and training procedure; its reported performance cannot be attributed to the endpoint factorization alone without ablations. We likewise do not claim that our supervised graph-kernel experiment reproduces or outperforms the complete PILNO system.

Relationship to F-INR. F-INR [3] studies implicit neural representations: coordinate-based continuous representations of signals, including images, signed-distance geometry, radiance fields, and physics-informed spatiotemporal fields. It parameterizes these functions through compact factor networks combined using CP, TT, or Tucker contractions. Our use of functional factor networks is inspired by this representation strategy.

The object being represented differs: F-INR represents a coordinate-to-signal function, whereas we parameterize a kernel inside a graph integral that acts on an input function. This requires coupling the factors with source features, quadrature/neighborhood weights, and local aggregation. We therefore view the tensor formats as inherited representation tools, rather than new decompositions. We also acknowledge that F-INR already includes scientific-computing applications; the distinction is not simply computer vision versus PDEs.

Additional input-block separation experiment. To test the finer variable separation independently of an entire LNO or PILNO architecture, we implement their joint-endpoint parameterization within the same local graph-integral framework as our CP model. We compare κendpoint(x,y,a(x),a(y))=∑r=1Rqr(x,a(x))pr(y,a(y))\kappa_{\mathrm{endpoint}}(x,y,a(x),a(y))=\sum_{r=1}^{R}q_r(x,a(x))p_r(y,a(y)) with κCP(x,y,a(x),a(y))=∑r=1Rfr(x)gr(a(x))hr(y)tr(a(y)).\kappa_{\mathrm{CP}}(x,y,a(x),a(y))=\sum_{r=1}^{R}f_r(x)g_r(a(x))h_r(y)t_r(a(y)). Both use the same neighborhood, backbone, channel convention, sparse aggregation, and training-budget protocol. Thus, the controlled distinction is whether position and state are processed jointly within each endpoint or further separated into multiplicative factors. Accordingly, we call this an LNO/PILNO-inspired local endpoint-separable baseline, not a complete PILNO reproduction. The results are reported below:

ModelTest errors at 16 ×\times 16Test errors at 32 ×\times 32
Local joint-endpoint0.2982 ±\pm 0.00050.2969 ±\pm 0.0027
CP-GNO0.2974 ±\pm 0.00070.2944 ±\pm 0.0006

The observed mean errors of our four-block CP are lower at both resolutions. We will revise the positioning accordingly: FTD-GNO applies established functional tensor representations to local graph-integral kernels and evaluates the consequences of finer input-block separation. It does not establish superiority over complete LNO or PILNO architectures.

AuthorsOpenReview ↗

Response to Reviewer 4gff Part 2

W2.1: The claimed reduction in memory growth from exponential to linear treats source positions, target positions, and their associated physical field values as independently enumerated variables. In an actual graph sample, however, physical field values are attached to nodes, and the baseline evaluates existing node pairs rather than every independent combination of these quantities. Moreover, the proposed reordering still requires aggregation over target-dependent neighborhoods. Although it can avoid expensive edge-wise kernel intermediates, it does not, by itself, eliminate edge-dependent aggregation or graph storage.

Thank you for raising this point. We will clarify the distinction between the number of evaluations of each input block and the number of graph interactions, and revise Table 1 and Appendix D accordingly.

Relationship between input-block evaluation counts and the number of edges. Let IiI_i denote the number of evaluation entries for the ii-th input block ziz_i, and let mim_i denote its feature dimension. These quantities have different meanings: for example, a two-dimensional position block has feature dimension mi=2m_i=2, but can be evaluated at IiI_i node entries. For the four-block kernel κ(x,y,a(x),a(y))\kappa(x,y,a(x),a(y)), the query positions and query field values are evaluated at the same query nodes, while the source positions and source field values are evaluated at the same source nodes. Therefore, with IxI_x query nodes and IyI_y source nodes,

I1=I3=Ix,I2=I4=Iy.I_1=I_3=I_x, \qquad I_2=I_4=I_y.

Because physical field values are attached to spatial nodes, the four blocks are not independently sampled axes whose entries are arbitrarily combined. Graph connectivity determines the actual source-query pairs, whose number is

E=∑i=1Ix∣N(xi)∣≤IxIy.E=\sum_{i=1}^{I_x}|\mathcal N(x_i)| \leq I_xI_y.

To simplify the subsequent table, we set Ix=Iy=II_x=I_y=I and assume that each block is evaluated at its corresponding II node entries. For a complete source-query graph, including self-interactions, E=I2E=I^2. For a local graph, E≤I2E\leq I^2, with strict inequality whenever some source-query pairs are excluded. Writing kˉ\bar k for the average neighborhood size gives E=IkˉE=I\bar k; linear scaling E=O(I)E=O(I) therefore requires an additional bounded-degree assumption. An ordinary GNO evaluates

(xi,yj,a(xi),a(yj)),j∈N(xi),\bigl(x_i,y_j,a(x_i),a(y_j)\bigr), \qquad j\in\mathcal N(x_i),

for the selected EE pairs, without storing a full tensor containing every independent Cartesian combination of the input blocks. Therefore, ∏i=1dIi\prod_{i=1}^{d}I_i, or IdI^d when Ii=II_i=I, can describe the size of an independently sampled tensor-product table, but not the storage required by ordinary GNO graph evaluation. We will correct this baseline interpretation and remove the resulting unconditional exponential-to-linear memory claim.

Revised Table 1. We distinguish kernel-representation storage from graph connectivity and other training intermediates. Let dd denote the number of kernel input blocks, mim_i the feature dimension of block ii, and m=∑i=1dmim=\sum_{i=1}^{d}m_i. Let W,LW,L denote the hidden width and number of hidden layers of the unfactorized kernel MLP, and w,lw,l the corresponding quantities for each factor network. For an MLP with l≥1l\geq1 fully connected hidden layers, input dimension mm, and output dimension oo, define its parameter count as P(m,w,l,o)=mw+(l−1)w2+wo+lw+o,P(m,w,l,o)=mw+(l-1)w^2+wo+lw+o, including biases, and its leading forward arithmetic count as F(m,w,l,o)=mw+(l−1)w2+wo.F(m,w,l,o)=mw+(l-1)w^2+wo.

Here, FF counts the leading multiply-accumulate operations without separately counting activation functions and similar operations. Let BB denote the batch size, CC the number of channels of the integrated feature vv, and hh the output dimension of the unfactorized kernel network. The factorized-model expressions below assume a scalar kernel shared across feature channels. For GNO, h=1h=1, h=Ch=C, and h=C2h=C^2 correspond to a shared scalar kernel, a channel-wise kernel, and a full channel-matrix kernel, respectively. The original GNO uses a full matrix-valued kernel, so h=C2h=C^2.

AuthorsOpenReview ↗

Response to Reviewer 4gff Part 3

Table 1: Complexity of kernel evaluation, parameter counts, and principal kernel-representation storage.

MethodSingle kernel evaluationTrainable kernel parametersKernel-representation storageExplicit graph connectivity
GNOO(F(m,W,L,C2))O(F(m,W,L,C^2))P(m,W,L,C2)P(m,W,L,C^2)O(BEC2)O(BEC^2) if all edge-kernel outputs are materializedO(I+E)O(I+E)
CP-GNOO(∑i=1dF(mi,w,l,R)+dR)O\left(\sum_{i=1}^{d}F(m_i,w,l,R)+dR\right)∑i=1dP(mi,w,l,R)\sum_{i=1}^{d}P(m_i,w,l,R)O(BIdR)O(BIdR)O(I+E)O(I+E)
TT-GNOO(∑i=1dF(mi,w,l,Ri−1Ri)+∑i=1dRi−1Ri)O\left(\sum_{i=1}^{d}F(m_i,w,l,R_{i-1}R_i)+\sum_{i=1}^{d}R_{i-1}R_i\right)∑i=1dP(mi,w,l,Ri−1Ri)\sum_{i=1}^{d}P(m_i,w,l,R_{i-1}R_i)O(BI∑i=1dRi−1Ri)O\left(BI\sum_{i=1}^{d}R_{i-1}R_i\right)O(I+E)O(I+E)
Tucker-GNOO(∑i=1dF(mi,w,l,R)+dR+Rd)O\left(\sum_{i=1}^{d}F(m_i,w,l,R)+dR+R^d\right)∑i=1dP(mi,w,l,R)+Rd\sum_{i=1}^{d}P(m_i,w,l,R)+R^dO(BIdR)O(BIdR), plus the RdR^d-entry coreO(I+E)O(I+E)

For TT, we assume d≥2d\geq2 and R0=Rd=1R_0=R_d=1. With equal internal ranks RR, ∑i=1dRi−1Ri=2R+(d−2)R2.\sum_{i=1}^{d}R_{i-1}R_i=2R+(d-2)R^2. The kernel-representation column counts only the evaluated factor outputs or materialized edge-kernel outputs; it is not a bound on total training memory. Hidden-network activations, node features, source-weighted features, aggregation outputs, other contraction intermediates, tensors saved for backpropagation, and operator workspace must be accounted for separately. The Tucker core is already included in the parameter count and should not be counted twice in total-memory accounting. The single-kernel evaluation column assumes uncached factors and does not describe the cost of an entire graph layer with node-factor reuse. TT and Tucker aggregation costs additionally depend on the input-block ordering and contraction path.

Where the efficiency advantage comes from. Factorization allows kernel components to be evaluated and reused at nodes before neighborhood aggregation, reducing repeated edge-wise kernel-network computation and avoiding particular edge-wise intermediate tensors. This reordering preserves the local graph and its neighborhood restrictions. For the four-block CP kernel, define

Qi,r=Ar(1)(xi)Ar(3)(a(xi)),Pj,r=Ar(2)(yj)Ar(4)(a(yj)).Q_{i,r}=A_r^{(1)}(x_i)A_r^{(3)}(a(x_i)), \qquad P_{j,r}=A_r^{(2)}(y_j)A_r^{(4)}(a(y_j)).

The graph-integral action can then be written as

Ui,c=∑r=1RQi,r(∑j=1ISijPj,rvj,c),U_{i,c}=\sum_{r=1}^{R}Q_{i,r}\left(\sum_{j=1}^{I}S_{ij}P_{j,r}v_{j,c}\right),

where SijS_{ij} contains the neighborhood mask and integration weights. For the neighborhood average in Eqs. (19)-(20),

Sij=1[j∈N(xi)]∣N(xi)∣,S_{ij}=\frac{\mathbf 1[j\in\mathcal N(x_i)]}{|\mathcal N(x_i)|},

assuming nonempty neighborhoods for the query nodes under consideration. This formulation enables the following operations:

OperationEffect
Evaluate each factor network at its associated nodesReuses the resulting factors across incident edges, replacing repeated edge-wise factor evaluation with node-wise evaluation
Form Qi,rQ_{i,r} and Pj,rP_{j,r} at the nodesAvoids repeatedly constructing the same position-field factor products for different edges
Form source-weighted features Pj,rvj,cP_{j,r}v_{j,c}Computes each source-side product once per source node, rank component, and channel, and reuses it during neighborhood aggregation
Apply sparse neighborhood aggregation using SSDirectly aggregates source-weighted features while preserving local interactions, without first explicitly generating rank-expanded messages for every edge
Contract the aggregated features with Qi,rQ_{i,r}Avoids explicitly reconstructing the CP kernel on each edge before applying it to source features

In our CP implementation, neighborhood aggregation uses CSR sparse matrix multiplication, avoiding explicit materialization of a gathered edge-message tensor of shape B×E×R×CB\times E\times R\times C. The source-weighted features and aggregation outputs instead have node-indexed shapes of order B×I×R×CB\times I\times R\times C. However, this does not eliminate edge dependence: for a scalar CP kernel shared across channels, sparse aggregation requires O(BERC)O(BERC) arithmetic, followed by O(BIRC)O(BIRC) query-side contraction, while batch-shared CSR connectivity requires O(I+E)O(I+E) storage. Because the neighborhood weights SijS_{ij} depend on the query, the inner aggregation cannot generally be replaced by a single source summary shared across all queries.

Thus, our advantage lies in moving reusable kernel-factor computation to nodes and reducing selected edge-wise intermediate tensors through contraction reordering and sparse matrix multiplication. This does not eliminate edge dependence or graph-connectivity storage. The measured memory savings reported in the paper demonstrate the practical benefit of this mechanism in the tested configurations, while the peak-memory benefit depends on rank, channel count, neighborhood size, tensors retained for backpropagation, and the underlying implementation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Summary of the Discussion Phase

Dear AC and Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We fully understand the additional workload and challenges brought about by the current situation, and we truly appreciate your continued efforts.

In the hope of facilitating your meta-review and decision-making process, we would like to provide a brief summary of the discussion phase:

  1. Contribution relative to prior work. We expanded the comparison with prior work in the paper. FTD-GNO allows position and state to be parameterized separately at each endpoint and combines the resulting factors through CP, TT, or Tucker contractions. Compared with the local joint-endpoint baseline, CP-GNO reduced peak memory by 30.0% while achieving a similar mean error. TT-GNO achieved both lower mean error and lower peak memory than the baseline. Tucker-GNO also achieved a lower mean error.

  2. Revised complexity analysis. We corrected the complexity analysis for standard GNO by counting kernel-network evaluations over the graph edges used in the computation. For FTD-GNO, we used CP as an example to describe the sparse aggregation procedure and explain how the implementation reuses node factors to avoid explicitly constructing an edge-wise message tensor expanded along the rank dimension.

  3. Additional experimental controls. We added ablation experiments to the appendix to examine the sources of the memory savings. CP-GNO reduced peak memory by 54.85%, 54.43%, and 17.63% relative to shared-scalar GNO, shared-scalar GNO with edge chunking, and shared-scalar GNO with activation checkpointing, respectively. We also added a rank-sensitivity study with multiple independent runs and comparisons of CP, TT, and Tucker using four input blocks.

  4. Predictive accuracy. The primary objective of this work is to reduce the GPU memory cost of graph-integral components. To further investigate the predictive accuracy of the overall architecture, we increased model complexity. This configuration achieved numerically lower relative errors than FNO on both the Darcy and NS tasks.

  5. Training feasibility on large irregular geometries. In the additional ShapeNet-Car CFD experiment, all three factorized variants completed training, while the tested GNO implementation ran OOM. Across three independent elasticity runs, CP-GNO also reached each predefined validation-error threshold in less time on average than GNO. These experiments provide additional evidence on training feasibility and the time required to reach a specified level of accuracy.

Current Discussion Status

  • Reviewer 4gff raised questions about our contribution relative to existing separable kernels, memory complexity, and the time required to reach comparable accuracy. We responded to each point in detail. The reviewer acknowledged the revisions and then asked about the memory usage of the local joint-endpoint baseline. We subsequently provided detailed comparison results in response.
  • Reviewer nvLL raised questions about the relationship between our method and Galerkin Transformer and Transolver, its predictive accuracy relative to FNO, and its performance on large irregular meshes. We addressed these questions in our response and have not yet received further feedback.
  • Reviewer tiJu raised questions about our complexity and parameter-count analyses, actual GPU memory usage, and the predictive accuracy of different kernel configurations. We addressed these questions in our response and have not yet received further feedback.
  • Reviewer xJJA raised questions about the novelty of the method, the sources of its memory savings, the reliability of the experiments, and performance across resolutions. We responded to each point in detail. After our replies, the reviewer then asked how the model's accuracy would compare with FNO after increasing its complexity. We subsequently provided additional experimental results.

As of this summary, we have received no further comments on the supplementary responses described above. We respectfully ask you to consider the full discussion record when preparing the meta-review, including the experimental results we submitted in response to follow-up questions from Reviewers 4gff and xJJA.

Thank you for your time and careful consideration. All revisions and additional clarifications provided during the discussion have been carefully incorporated into the updated manuscript, with changes highlighted in blue for your convenience. We hope this summary could assist your meta-review and decision-making process.

Additionally, we would like to affirm that the discussion of our paper was conducted in a fully legitimate, scientific, and anonymous manner. The authors did not engage in any non-scientific activities, deanonymization, or collusion throughout the discussion process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.