acceptodds
Under review as a conference paper at ICLR 2027

DCNO: Improving Operator Learning through Multi-Stream Dynamic Connections

Abstract

Transformer-based neural operators offer strong representational capacity for solving partial differential equations (PDEs). However, preserving early latent physical information while allowing representations to evolve remains a challenge as operator depth increases. To address this problem, we propose the Dynamic Connection Neural Operator (DCNO), which assigns information persistence and representation evolution to distinct but interacting latent streams. Specifically, DCNO maintains a slowly evolving operator-memory stream alongside more freely evolving working streams, with each operator layer dynamically reading from all streams. This design keeps early latent information accessible while allowing working representations to evolve through progressive transformations. Extensive experiments across diverse PDE benchmarks demonstrate that DCNO consistently improves predictive performance, while depth-focused analyses further reveal more effective utilization of increased operator depth.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operators, Operator Learning, Partial Differential Equations, AI for Physics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6mU2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Dynamic Connection Neural Operator (DCNO) to address the issue that deep Transformer-based neural operators tend to lose early physical condition information as network depth increases when solving PDEs. The architecture splits the standard single feature pathway into one protected memory stream that only receives small updates and multiple working streams that mix and update freely. Each layer then dynamically reads information from all streams for computation. Experiments across seven PDE benchmarks show that this method effectively preserves early physical conditions, makes better use of deeper networks, and reduces prediction errors on most tasks.

Critical strengths
  1. The paper is well-organized and provides intuitive explanations. It clearly presents the task, the existing problem, the underlying analysis, and the proposed solution step by step, supported by clear visual figures.
  2. The architectural design is lightweight, simple, and well-motivated. By decoupling the feature streams in Hyper-Connections into memory and working streams, the method improves cross-layer information propagation without modifying the underlying operator blocks.
  3. The main experimental evaluation is comprehensive. The proposed method is compared against several popular and recent baselines across diverse mainstream benchmarks, achieving highly competitive performance.
Critical weaknesses
  1. The analyses in Sections 4.2 and 4.3 lack comparisons with the GNOT+HC baseline. As shown in Table 1, the performance gain from GNOT+HC varies across different settings. Therefore, only comparing DCNO with the base GNOT model in these sections cannot clearly isolate the specific effectiveness of the memory stream design.
  2. The ablation studies on the internal components of DCNO could be more detailed. The current experiments do not separately analyze the individual impacts of the memory write scale ϵmem\epsilon_{\text{mem}}, the number of latent streams rr, or the static versus dynamic connection coefficients.
  3. The exploration range of the depth scaling experiments is somewhat limited. Figure 4 only tests 1 to 8 layers on a single dataset (NS2D), and the error reduction already levels off between 6 and 8 layers. Adding results with deeper networks (such as 12 or 16 layers) and across more datasets would provide stronger evidence for the effectiveness of the design at scale.
Questions for authors
  1. Could the authors include the GNOT+HC baseline in the analyses of Sections 4.2 and 4.3 to better demonstrate the specific contribution of the protected memory stream?
  2. Does DCNO continue to show performance gains when scaling to deeper networks (e.g., 12 or 16 layers)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6mU2: part 1

We sincerely thank the reviewer for the constructive feedback and for recognizing the clarity of our presentation, the lightweight and well-motivated architecture, and the comprehensive experimental evaluation. Please find our responses to the comments below.


(W1, Q1) The analyses in Sections 4.2 and 4.3 lack comparisons with the GNOT+HC baseline. Could the authors include GNOT+HC in these analyses to better demonstrate the specific contribution of the protected memory stream?

We thank the reviewer for this valuable suggestion. We agree that comparisons with GNOT+HC are important for distinguishing the benefits of multi-stream connections from those of the protected memory design. We have therefore extend the experiments and included GNOT+HC in the physical-condition alignment (CKA) analysis in Section 4.2 (Figure 1) and the initial-state alignment (ISA) analysis in Section 4.3 (Figure 5).

1. Physical-condition alignment (CKA, Section 4.2, Figure 1).

  • GNOT+HC partially mitigates the alignment decline observed in GNOT. At the final operator block, the CKA scores are approximately 0.08 for GNOT and 0.17 for GNOT+HC, indicating that multi-stream propagation itself improves condition-related structural alignment.
  • DCNO maintains substantially higher alignment than both baselines. Its CKA remains relatively stable across depth, reaching approximately 0.35 at the final block, compared with 0.17 for GNOT+HC. This suggests that the protected memory design provides an additional benefit beyond standard multi-stream connections.

2. Initial-state alignment (ISA, Section 4.3, Figure 5).

  • GNOT+HC allows substantial representation evolution. Its read state exhibits strong non-monotonic variations, while its stream-mean ISA decreases from approximately 0.80 to −0.15.
  • DCNO maintains a stable memory representation. The protected memory stream remains highly aligned with its initialization, with ISA staying between approximately 0.96 and 0.99 throughout the network.
  • Importantly, this memory persistence does not restrict representation evolution. DCNO's stream-mean and dynamically read representations also evolve substantially, with the read-state ISA varying non-monotonically and even becoming negative. Thus, unlike standard HC, DCNO explicitly separates a persistent memory path from evolving working representations, while retaining the flexibility of dynamic multi-stream computation.

3. Ablation on the predictive error (Table 1).

  • These representation-level findings are consistent with the predictive results in Table 1, where DCNO outperforms GNOT+HC in 10 of 11 evaluated settings.
    • For example, relative L2L_2 errors decrease from 0.0432 to 0.0358 on Heat2D (17.1%), from 0.0144 to 0.0064 on Darcy2D (55.6%), and from 0.0150 to 0.0058 on NACA (61.3%).

(W2) The ablation studies on the internal components of DCNO could be more detailed. The current experiments do not separately analyze the individual impacts of the memory write scale, the number of latent streams, or the static versus dynamic connection coefficients.

We thank the reviewer for this suggestion. We conducted additional component-wise ablations to better understand the contributions of different DCNO components.

Memory write scale. We evaluate different values of ϵmem\epsilon_{\mathrm{mem}} on Heat2D and NS2D.

MethodHeat2DNS2D
GNOT0.05300.1002
GNOT+HC0.04320.1009
DCNO, ϵmem=0\epsilon_{\mathrm{mem}}=00.03670.0907
DCNO, ϵmem=0.01\epsilon_{\mathrm{mem}}=0.010.03550.0919
DCNO, ϵmem=0.05\epsilon_{\mathrm{mem}}=0.050.03580.0893
DCNO, ϵmem=0.1\epsilon_{\mathrm{mem}}=0.10.03610.1056
DCNO, ϵmem=1.0\epsilon_{\mathrm{mem}}=1.00.04480.1232
  • Sensitivity across PDE systems: The sensitivity to the memory write scale varies across PDE systems. Nevertheless, small write scales (0≤ϵmem≤0.050\leq\epsilon_{\mathrm{mem}}\leq0.05) achieve competitive results on both tasks.
  • Effect of frozen memory: Setting ϵmem=0\epsilon_{\mathrm{mem}}=0 also achieves competitive performance, supporting the importance of information preservation.

These results support the benefit of a protected, slowly evolving memory stream.

AuthorsOpenReview ↗

Response to Reviewer 6mU2: part 2

(W3) The exploration range of the depth scaling experiments is somewhat limited. Figure 4 only tests 1 to 8 layers on a single dataset (NS2D), and the error reduction already levels off between 6 and 8 layers. Adding results with deeper networks (such as 12 or 16 layers) and across more datasets would provide stronger evidence for the effectiveness of the design at scale.

(Q2) Does DCNO continue to show performance gains when scaling to deeper networks (e.g., 12 or 16 layers)?

We thank the reviewer for this suggestion. We extended the depth-scaling experiments on NS2D to 12 and 16 operator blocks, and included GNOT+HC as an additional baseline.

Operator blocksGNOTGNOT+HCDCNOReduction against GNOTReduction against GNOT+HC
10.11500.11180.1194−3.8%−6.8%
20.11340.11350.10239.8%9.8%
30.10590.10690.09896.6%7.5%
40.10300.10210.089712.9%12.1%
50.09570.09260.08788.2%5.1%
60.09160.09120.08329.2%8.7%
70.09120.09450.08546.3%9.6%
80.08930.09190.079910.5%13.1%
120.08950.09350.08039.2%14.1%
160.08530.08610.07887.6%8.5%
Error reduction (1→16 blocks)25.8%23.0%34.0%——
  • Effectiveness at greater depths: DCNO maintains the lowest prediction error at both 12 and 16 blocks. At 16 blocks, it reduces errors by 7.6% over GNOT and 8.5% over GNOT+HC.
  • Better depth utilization: From 1 to 16 blocks, DCNO achieves a 34.0% error reduction, compared with 25.8% for GNOT and 23.0% for GNOT+HC.
  • All three models show relatively limited additional improvements beyond 8 blocks. This may indicate diminishing returns from increasing model depth on the current NS2D benchmark. Nevertheless, DCNO keeps a consistent advantage over both baselines at greater depths.

Thank authors for the detailed response and additional experiments. The comparison with GNOT+HC clarifies how DCNO’s protected memory stream preserves information across layers, and the ablation on the memory write scale shows how this choice affects prediction error. On NS2D, DCNO outperforms both GNOT and GNOT+HC at 12 and 16 blocks, although the gains from increasing its depth beyond eight blocks are modest. It remains unclear whether this trend holds on other datasets.

Reviewer_mevKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors addressed a limitation of deep neural operators: standard residual connections overwrite early physical conditions as depth increases. The primary idea is to adapt Hyper-Connections to split the hidden state into a slowly evolving memory stream and several working streams using dynamic read/write weights. The authors empirically showed the method's effectiveness on 7 problems.

Critical strengths
  • The authors address an interesting representation degradation issue across depth in neural operators.

  • The authors provide CKA and depth-utilization diagnostics for this particular problem. Even though simple, it helps analyze how conditioning evolves across layers.

Critical weaknesses
  • The research gap and related work are very poorly explored. The authors do not properly discuss why existing architectures fail to address this gap.

  • The technique/architecture introduced is well known and well used. Hyper-Connections (Zhu et al., 2025) already contain such multi-stream interactions, and gating a protected memory cell with read/write weights is conceptually identical to an LSTM cell across depth (or Highway Networks). At the same time, U-shaped operators (such as U-NO and U-FNO), DenseNet-style skip connections, or direct geometry injection [1] trivially satisfy the design of preserving early features. The authors should evaluate and compare these models. Or the authors should explicitly explain why those models cannot be used for the experiments in the paper.

  • In the Introduction, the authors motivate the problem via "inter-sample structure" and its deviation along the depth. However, the authors do not clearly define this structure or why deviation along depth is necessarily bad (since deep networks are supposed to transform representations from the input toward the target solution).

  • While depth scalability is a main concern, I do not find the default operator depth L used in the main benchmark experiments (Table 1).

  • In [Line 260], the GNOT backbone already injects the encoded physical conditions directly into every layer via cross-attention (K_l, V_l), yet the authors claim deeper blocks lose access to early physical conditions and need a query-side memory stream. This requires additional clarification.

[1] Do neural operators forget geometry? The forgetting hypothesis in deep operator learning

Questions for authors
  • [Intro] "inter-sample structure": How is this formally defined, and why is its deviation across depth bad when the representation is supposed to evolve into the PDE solution?

  • [Method] How does the protected memory stream with dynamic read/write weights fundamentally differ from an LSTM-style gated memory cell across depth?

  • [line 251] h0(mem)h₀^(mem): How does h₀ have high condition CKA at block 0 (Figure 1) if it is initialized only from query coordinates before cross-attention?

  • [line 268] What is cr(1or1/r)c_r (1 or 1/r) actually set to in the experiments?

  • Experiment: Is the parameter count kept fixed when comparing DCNO against the baseline GNOT and Transolver?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mevK: part 1

We sincerely thank the reviewer for the constructive feedback and for recognizing the importance of the representation degradation issue we address and our analyses. Please find our responses to the comments below.

1. Research Gap and Architectural Comparisons (W1, W2, Q2)

(W1, W2, Q2) The research gap and connections to prior work need clarification. How does DCNO differ from HC, LSTM/Highway networks, U-shaped operators, dense skip connections, and geometry injection? Please provide relevant comparisons.

We thank the reviewer for these comments. We agree that our contribution should be positioned more clearly within existing work on feature preservation and reuse. Below, we clarify the research question, the architectural differences. We have added a detailed comparison with LSTM and Highway networks in the Appendix C.4.

  1. Research gap and scope of our contribution.
    • Deep neural operators need to construct new solution features while retaining access to useful earlier information.
    • Standard residual connections propagate both through a single evolving state.
    • Existing feature-reuse mechanisms provide different solutions, but do not directly evaluate the role of a protected memory destination within dynamic multi-stream operator connections.
    • DCNO investigates this design by separating a slowly evolving memory stream from more freely evolving working streams, while keeping the underlying operator transformations unchanged.
  2. Relation to Hyper-Connections (HC).
    • We acknowledge that HC already introduces multiple interacting streams, dynamic reads, stream mixing, and output redistribution, as illustrated in Section 3.1.
    • DCNO's distinction lies in its memory-specific propagation rules.
      • One stream is assigned a protected identity path, excluded from direct incoming cross-stream mixing, and updated through scaled writes.
      • The working streams keep dynamic mixing and unscaled writes. This allows the memory and working representations to follow different propagation rules.
  3. Relation to LSTM and Highway networks (Appendix C.4).
    • Architectural difference: DCNO is designed for Transformer-based neural operators, whereas LSTM is a recurrent neural network architecture (different network scope). We also would like to further clarify their connection-level differences below.
    • LSTM:
      • State organization: LSTM generates its hidden output through a gated transformation of its cell state. DCNO maintains separately evolving working streams alongside a protected memory stream.
      • Read/write mechanism: DCNO dynamically combines all streams as the operator input and redistributes the operator output. Its memory has a fixed identity path with scaled writes, unlike LSTM's learned forget gate.
    • Highway networks: Highway layers gate carried and transformed features of a single state, whereas DCNO maintains multiple interacting streams with distinct propagation rules.
  4. Relation to other feature-preserving architectures.
    • U-NO and U-FNO employ multiscale architectures and skip connections to reuse features. DCNO instead modifies latent propagation between existing operator transformations.
    • DenseNet reuses earlier representations through feature concatenation. DCNO maintains a fixed number of interacting streams with dynamic propagation.
    • Geometry injection reintroduces encoded geometry into intermediate representations. DCNO maintains an internal memory stream that receives scaled updates and remains accessible through dynamic reads.
  5. Experimental evidence.
    • U-NO is already included in Table 1 as a representative U-shaped neural operator.
    • GNOT+HC provides a direct connection-level comparison. GNOT+HC and DCNO use the same backbone, number of streams, dynamic read mechanism, and training settings. They differ in the memory-specific propagation rules.
    • DCNO outperforms GNOT+HC in 10 of the 11 reported settings, with particularly large improvements on NACA and Darcy2D.
    • These results support the benefit of explicitly separating memory preservation from working-state evolution, beyond the multi-stream interactions already provided by HC.
AuthorsOpenReview ↗

Response to Reviewer mevK: part 2

2. Inter-Sample Structure and Representation Evolution (W3, Q1)

(W3, Q1) How is “inter-sample structure” defined? Why should its deviation across depth be harmful when representations are expected to evolve toward the PDE solution?

We thank the reviewer for raising these questions. We clarify the definition, the role of CKA, and the distinction between information persistence and representation evolution below.

  1. Definition and clarification in the manuscript.

    • Here, a sample refers to a PDE problem instance. As described in Section 4.2, we summarize each instance’s representation using the spatial mean and standard deviation of each feature channel.

    • After stacking and centering these summaries across samples, the operator representations and condition reference define two Gram matrices:

      Gℓ=X~ℓX~ℓ⊤, GC=C~C~⊤.G_\ell=\widetilde X_\ell\widetilde X_\ell^\top, \ G_C=\widetilde C\widetilde C^\top.

    • Each entry is a centered inner product between two sample summaries. “Inter-sample structure” refers to these pairwise relationships. CKA in Eq. (3) measures the normalized alignment between the two matrices.

    • To avoid ambiguity, we have simplified the wording in the Introduction. It now directly describes the alignment between latent representations and the encoded physical conditions. We keep the technical explanation of them in Section 4.2.

  2. DCNO maintains a persistent memory reference while allowing its working streams to evolve.

    We agree that deeper layers need to develop new solution features. DCNO’s sustained condition alignment can be explained by its protected memory stream, while its working streams remain free to learn these evolving features.

    • GNOT uses a single query state for both roles. Earlier features are carried forward and repeatedly modified as new operator updates accumulate.
    • DCNO decouple the two roles in two kind of streams
      • The memory stream is approximately frozen.
      • The working streams can continue to change.
  3. Experiments support the coexistence of information persistence and representation evolution.

    • Figure 1: DCNO maintains more stable condition alignment in the representations actually used by the operator.
    • Figure 5: The memory remains aligned with initialization, while the stream mean and dynamic read evolve substantially.
    • Prediction results: Table 1 and Figure 3 show lower overall errors and improved accuracy near boundaries and interfaces. Figure 4 shows greater predictive benefits from additional depth.

3. Experimental Settings (W4, Q4, Q5)

(W4) While depth scalability is a main concern, I do not find the default operator depth L used in the main benchmark experiments (Table 1).

We thank the reviewer for pointing out this omission. The default operator depth in Table 1 is L=4L=4 backbone blocks, which is a conventional configuration for neural operators. The deeper configurations are evaluated separately in Section 4.3.

(Q4) What is crc_r actually set to in the experiments?

We thank the reviewer for requesting this clarification. Although both sum- and mean-reductions are theoretically workable, we set cr=1/rc_r=1/r (i.e. mean reduction) in the actual experiments in this paper. We have clarify these experiment-specific settings in Section 3.2 of the revised manuscript.

(Q5) Experiment: Is the parameter count kept fixed when comparing DCNO against the baseline GNOT and Transolver?

We thank the reviewer for this question. We do not strictly match the total parameter counts, as GNOT and Transolver have different architectures and parameter budgets.

  • For fair comparison, all three models use the same operator depth.
  • DCNO retains the GNOT backbone, and introduces only lightweight additional connection parameters.

A detailed analysis of the additional parameter count is provided in Appendix C.2.

AuthorsOpenReview ↗

Response to Reviewer mevK: part 3

4. Physical-Condition Alignment (W5, Q3)

(W5) In [Line 260], the GNOT backbone already injects the encoded physical conditions directly into every layer via cross-attention (Kℓ,VℓK_\ell, V_\ell), yet the authors claim deeper blocks lose access to early physical conditions and need a query-side memory stream. This requires additional clarification.

We thank the reviewer for raising this point. Each GNOT cross-attention layer can indeed read the encoded boundary conditions and geometry. However, reading a condition again via (K,V)(K,V) and reusing what an earlier layer computed from it at a particular query location QQ are different. Below, we clarify this distinction and explain why retaining physical information in the query state also matters.

  1. GNOT’s limitation: a single residual state handles query (Q)-side propagation.
    • Indeed, the keys and values Kℓ,VℓK_\ell,V_\ell allow each layer to read the physical conditions again.
    • An early layer uses these inputs to compute “how the condition affects the current query location”, and stores the result in its single updated query state.
      • e.g. an early layer may produce a feature describing the influence of a nearby obstacle boundary. That result is carried in the updated query state.
    • Subsequent layers use the same query state to develop new solution features. An update can overwrites the earlier response and those affect physical alignment of the query.
    • Therefore, as depth increases, the query state may gradually lose physical information. Since the query guides each read from Kℓ,VℓK_\ell,V_\ell, GNOT’s final predictions may become less accurate regarding those conditions.
  2. DCNO’s advantage: (K,V,Q)(K,V,Q) have access to physical condtions, each layer can adaptively choose what to retain, update, and read again.
    • As in GNOT, the keys and values Kℓ,VℓK_\ell,V_\ell give each cross-attention layer direct access to the encoded physical conditions.
    • In addition, the query QℓQ_\ell can access earlier responses to those conditions through the memory state. Meanwhile, the working streams can freely mix and update their features.
    • At a later layer, the dynamic read determines how much each stream contributes to QℓQ_\ell. The query can therefore combine the retained boundary response with newly learned features to guide its next read from Kℓ,VℓK_\ell,V_\ell.
  3. Our experiments support this argument.
    • Figure 1: stronger query-to-condition alignment at depth. On Heat2D, GNOT’s CKA drops from 0.4 to 0.08, while DCNO’s stays near 0.4 at xℓx_\ell, the operator input after the dynamic read.
    • Figure 3: improvement regarding physical conditions. DCNO reduces velocity errors near obstacle boundaries in NS2D-c and solution errors along the coefficient interface in Darcy2D.
    • Table 1: overall prediction advantage. With the same GNOT backbone, DCNO achieves lower prediction error than residual GNOT.

(Q3) How does the initial representation have high condition CKA at block 0 (Figure 1) if it is initialized only from query coordinates before cross-attention?

We thank the reviewer for this question. Block 0 is recorded before the first cross-attention. Its CKA of approximately 0.4 can be explained by the geometry shared between the query point sets and the condition reference.

  1. The query coordinates carry sample-specific geometric structure.
    • For the evaluated Heat2D dataset samples, the query point set is the union of the coordinates in geometry branches of the condition input.
    • Thus, both the query encoder and the condition encoders receive information related to the similar sample-specific geometry.
    • After spatial pooling, their representations can share similar across-sample structure, yielding a CKA around 0.4 before cross-attention.
  2. Using only geometric conditions preserves the initial CKA.
    • We verified this using the original feature caches and the same 100 test samples.

    • Keeping the query representations fixed, we removed the only branch containing temperature values from each model's own encoded-condition reference.

    • Direct CKA estimates on all 100 samples are:

      Condition referenceCKA
      All condition branches0.39713
      Coordinate-only branches0.39842
    • Therefore, for the Heat2D dataset, a coordinate-only reference therefore already yields nearly the same initial CKA, supporting shared geometry as the explanation for the observed alignment of approximately 0.4.

Reviewer_kEcKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a multi-stream connection architecture for deep neural operators, termed DCNO. By maintaining a protected memory stream with evolving working streams, DCNO demonstrates improvements across diverse PDE benchmarks.

Critical strengths
  1. The experimental results look promising.

  2. The paper is easy to understand.

Critical weaknesses
  1. When the depth increases, the CKA loses the information, and it often reflects normal feature abstraction and high-order disentanglement in deep networks. How does DCNO overcome this limitation?

  2. The default scale ϵmem=0.05\epsilon_{mem} = 0.05 is manually designed. Is it sensitive to different PDE systems?

  3. Multi-stream designs inherently inflate memory footprints. Thus, actual VRAM overhead during training should be reported and compared with others.

  4. In Eq. (1), the notation involving ylTy_l^T creates dimensional ambiguity since yl∈RDy_l \in \mathbb{R}^D. Please check the shapes in detail of the whole paper.

  5. Since the memory stream barely evolves (ϵmem=0.05\epsilon_{mem}=0.05), does it act as a skip connection? What is its fundamental difference from traditional ResNet?

  6. An ablation with ϵmem=1.0\epsilon_{mem} = 1.0 is missing.

  7. The used datasets are standard 2D benchmarks. Please provide the performance of DCNO on complex tasks.

  8. The source code and model should be available to check.

Questions for authors

Please see Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kEcK: part 1

We sincerely thank the reviewer for the constructive feedback and for recognizing the promising experimental results and clarity of our presentation. We provide our point-by-point responses below.


(W1) When the depth increases, the CKA loses the information, and it often reflects normal feature abstraction and high-order disentanglement in deep networks. How does DCNO overcome this limitation?

We thank the reviewer for raising this question. DCNO’s sustained alignment with the encoded physical conditions can be explained by its protected memory stream. However, its working streams remain free to learn abstract features and disentangle higher-order factors.

In particular, for PDEs, deeper layers need to develop new solution features while retaining physical information that still matters to the solution.

  • Standard residual architectures such as GNOT rely on a single stream for both tasks.
  • DCNO decouples these roles. The protected memory stream can retain early condition-dependent features, while the working streams can freely mix and update to develop higher-level features.

Figures 1 and 5 clarify how these two behaviors coexist in DCNO. The prediction results show why this balance matters for PDE learning.

  1. Figure 1 shows that the deep operator inputs remain aligned with the physical conditions.
    • On Heat2D, the blue curve shows that GNOT’s query states become less aligned with the encoded physical conditions as depth increases.
    • DCNO maintains more stable alignment. Therefore, DCNO can have better access to those physical information when necessary for the deeper layers.
  2. Figure 5 shows that the operator input can still evolves substantially.
    • GNOT’s single residual state gradually departs from h0h_0, while DCNO’s protected memory stays closely aligned with it.
    • But DCNO’s mean across streams departs further from h0h_0 than GNOT’s state at later layers, and its read state xℓx_\ell also shows large shifts in alignment across depth.
    • Therefore, DCNO can preserve a stable memory while its other representations change at least as much as GNOT’s under this measure.
  3. Prediction results support the value of this balance for PDE learning.
    • On the same task, DCNO witnesses lower relative L2L^2 error than GNOT.
    • Figure 3 shows that DCNO’s predictions align better with the physical conditions, with lower errors near obstacle boundaries in NS2D-c and along the coefficient interface in Darcy2D.

(W2, W6) The default memory write scale ϵmem=0.05\epsilon_{\mathrm{mem}}=0.05 is manually designed. Is it sensitive to different PDE systems? An ablation with ϵmem=1.0\epsilon_{\mathrm{mem}}=1.0 is missing.

We thank the reviewer for these suggestions. We conducted a sensitivity analysis of the memory write scale on Heat2D and NS2D, including the requested ϵmem=1.0\epsilon_{\mathrm{mem}}=1.0 setting.

MethodHeat2DNS2D
GNOT0.05300.1002
GNOT+HC0.04320.1009
DCNO, ϵmem=0\epsilon_{\mathrm{mem}}=00.03670.0907
DCNO, ϵmem=0.01\epsilon_{\mathrm{mem}}=0.010.03550.0919
DCNO, ϵmem=0.05\epsilon_{\mathrm{mem}}=0.050.03580.0893
DCNO, ϵmem=0.1\epsilon_{\mathrm{mem}}=0.10.03610.1056
DCNO, ϵmem=1.0\epsilon_{\mathrm{mem}}=1.00.04480.1232
  • Sensitivity across PDE systems: We acknowledge that the sensitivity to the memory write scale varies across PDE systems. Nevertheless, small write scales (0≤ϵmem≤0.050\leq\epsilon_{\mathrm{mem}}\leq0.05) achieve consistently competitive results on both tasks. Although the optimal value differs slightly, 0.05 provides a reasonable shared default without dataset-specific tuning.
  • Effect of ϵmem=1.0\epsilon_{\mathrm{mem}}=1.0: Removing memory-write scaling increases prediction errors by 25.1% on Heat2D and 38.0% on NS2D, relative to ϵmem=0.05\epsilon_{\mathrm{mem}}=0.05. This supports the importance of restricting memory updates.
  • Effect of ϵmem=0\epsilon_{\mathrm{mem}}=0: Even a fully frozen memory achieves competitive performance, supporting the benefit of persistent information preservation. Small nonzero writes allow memory to adapt while retaining this stability.
AuthorsOpenReview ↗

Response to Reviewer kEcK: part 2

(W3) Multi-stream designs inherently inflate memory footprints. Thus, actual VRAM overhead during training should be reported and compared with others.

We thank the reviewer for raising this concern. We measured the peak allocated GPU memory during training on two representative datasets. All models use matched backbone configurations.

DatasetsGNOTGNOT+HCDCNODCNO vs. GNOTDCNO vs. GNOT+HC
Heat2D393.6 MiB663.2 MiB585.2 MiB+48.7%−11.8%
NS2D2780.8 MiB5433.0 MiB4676.1 MiB+68.2%−13.9%
  • Overhead over baseline GNOT: DCNO introduces a relatively lightweight memory overhead from multiple latent streams and dynamic connection coefficients. A detailed analysis of the additional memory and computational complexity is provided in Appendix C.2.
  • However DCNO records Lower memory than baseline GNOT+HC: Unlike HC's unrestricted cross-stream mixing, DCNO uses a protected identity path for its memory stream. This simplifies memory-state propagation and may reduce the intermediate tensors retained during backpropagation.

(W4) In Eq. (1), the notation involving ylTy^{T}_{l} creates dimensional ambiguity since yl∈RDy_l\in\mathbb{R}^D. Please check the shapes in detail of the whole paper.

We thank the reviewer for pointing out this notational ambiguity. In Section 3, feature vectors and connection-coefficient vectors are treated as column vectors. In particular, the operator output at a query location is related to the collected output matrix by

yl=(Yl[i,:])⊤∈RD×1,y_l=(Y_l[i,:])^\top\in\mathbb{R}^{D\times1},

The main notation and corresponding shapes are summarized below.

NotationDescriptionShape
h0, hlmem, hl(s), xl, yl, xouth_0,\ h_l^{\mathrm{mem}},\ h_l^{(s)},\ x_l,\ y_l,\ x_{\mathrm{out}}Per-location latent vectorsD×1D\times1
Hl, H~l, Hl+1H_l,\ \widetilde H_l,\ H_{l+1}Per-location stream-state matricesr×Dr\times D
al(Hl), bl(Hl), blstatica_l(H_l),\ b_l(H_l),\ b_l^{\mathrm{static}}Read/write coefficient vectorsr×1r\times1
Pl(Hl), Pˉl(Hl), DϵP_l(H_l),\ \bar P_l(H_l),\ D_\epsilonStream-mixing and write-scaling matricesr×rr\times r
Al(Hl), AlstaticA_l(H_l),\ A_l^{\mathrm{static}}Combined read and mixing coefficientsr×(r+1)r\times(r+1)
Wα,lW_{\alpha,l}Projection parameter for read and mixing coefficientsD×(r+1)D\times(r+1)
wβ,lw_{\beta,l}Projection parameter for write coefficientsD×1D\times1
Xl, Yl, XoutX_l,\ Y_l,\ X_{\mathrm{out}}Representations collected over query locationsN×DN\times D
u^\hat uPredicted fieldN×doutN\times d_{\mathrm{out}}

With these conventions, the matrix-form update corresponding to Eq. (1) is dimensionally consistent:

Hl+1=Pˉl(Hl)⊤Hl+(Dϵbl(Hl))yl⊤.H_{l+1} = \bar{P}_l(H_l)^\top H_l + (D_\epsilon b_l(H_l))y_l^\top.

Specifically, the propagation and write terms have shapes:

\quad (r\times r)(r\times D)\rightarrow r\times D,$$ $$(D_\epsilon b_l(H_l))y_l^\top: \quad (r\times 1)(1\times D)\rightarrow r\times D.$$ Both terms therefore match the shape of $H_{l+1}\in\mathbb{R}^{r\times D}$. --- > **(W5)** Since the memory stream barely evolves ($\epsilon_{mem}=0.05$), does it act as a skip connection? What is its fundamental difference from traditional ResNet? > We thank the reviewer for this question. Yes, the memory stream contains an identity skip. However, unlike ResNet, which uses a **single** residual stream, **DCNO maintains multiple residual streams**, including one memory stream. Each layer can adaptively read from both. Below, We compare the methods discussed in the paper: 1. **Residual connection (e.g., ResNet): a single residual stream.** - Each layer computes $h_{\ell+1}=h_\ell+F_\ell(h_\ell)$, and the next layer reads the resulting sum. 2. **Hyper-Connections (HC): multiple interacting residual streams.** - HC expands the single residual stream into several parallel streams. Each layer reads a weighted combination of them, applies its transformation once, and writes the output back to each stream with a learned weight. 3. **DCNO: multiple streams, one memory and several working streams.** - DCNO builds on HC by protecting one stream from incoming cross-stream mixing. Its previous state passes through an identity skip. The working streams retain learned mixing and receive unscaled writes. > **(W8)** The source code and model should be available to check. > We thank the reviewer for emphasizing reproducibility. While the code and trained models are not publicly available during the review stage, Section 3 and Appendix C provide detailed architectural and implementation descriptions, and Section 4 reports the experimental settings. As stated in our Reproducibility Statement, we plan to release the implementation and experiment configurations upon publication.
AuthorsOpenReview ↗

Response to Reviewer kEcK: part 3

(W7) The used datasets are standard 2D benchmarks. Please provide the performance of DCNO on complex tasks.

We thank the reviewer for this suggestion. Our evaluation follows standard experimental settings in neural operator research, using widely adopted benchmarks that are commonly employed to evaluate state-of-the-art methods. Using these established datasets also enables direct and fair comparisons with existing baselines under consistent experimental settings.

  • Physical diversity: Despite being primarily 2D, these benchmarks cover diverse physical systems, including fluid dynamics, electromagnetics, heat transfer, and elasticity, with varying geometries, boundary conditions, and physical parameters.
  • We agree that evaluation on complex 3D and industrial problems would further strengthen the evidence. However, the computational demands and limited rebuttal period make it difficult to complete reliable experiments in time.
  • We acknowledge this as a direction for future evaluation, as also noted in Section 5 Conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.