- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Dynamic Connection Neural Operator (DCNO) to address the issue that deep Transformer-based neural operators tend to lose early physical condition information as network depth increases when solving PDEs. The architecture splits the standard single feature pathway into one protected memory stream that only receives small updates and multiple working streams that mix and update freely. Each layer then dynamically reads information from all streams for computation. Experiments across seven PDE benchmarks show that this method effectively preserves early physical conditions, makes better use of deeper networks, and reduces prediction errors on most tasks.
Critical strengths
- The paper is well-organized and provides intuitive explanations. It clearly presents the task, the existing problem, the underlying analysis, and the proposed solution step by step, supported by clear visual figures.
- The architectural design is lightweight, simple, and well-motivated. By decoupling the feature streams in Hyper-Connections into memory and working streams, the method improves cross-layer information propagation without modifying the underlying operator blocks.
- The main experimental evaluation is comprehensive. The proposed method is compared against several popular and recent baselines across diverse mainstream benchmarks, achieving highly competitive performance.
Critical weaknesses
- The analyses in Sections 4.2 and 4.3 lack comparisons with the GNOT+HC baseline. As shown in Table 1, the performance gain from GNOT+HC varies across different settings. Therefore, only comparing DCNO with the base GNOT model in these sections cannot clearly isolate the specific effectiveness of the memory stream design.
- The ablation studies on the internal components of DCNO could be more detailed. The current experiments do not separately analyze the individual impacts of the memory write scale , the number of latent streams , or the static versus dynamic connection coefficients.
- The exploration range of the depth scaling experiments is somewhat limited. Figure 4 only tests 1 to 8 layers on a single dataset (NS2D), and the error reduction already levels off between 6 and 8 layers. Adding results with deeper networks (such as 12 or 16 layers) and across more datasets would provide stronger evidence for the effectiveness of the design at scale.
Questions for authors
- Could the authors include the GNOT+HC baseline in the analyses of Sections 4.2 and 4.3 to better demonstrate the specific contribution of the protected memory stream?
- Does DCNO continue to show performance gains when scaling to deeper networks (e.g., 12 or 16 layers)?
3 comments
Response to Reviewer 6mU2: part 2
(W3) The exploration range of the depth scaling experiments is somewhat limited. Figure 4 only tests 1 to 8 layers on a single dataset (NS2D), and the error reduction already levels off between 6 and 8 layers. Adding results with deeper networks (such as 12 or 16 layers) and across more datasets would provide stronger evidence for the effectiveness of the design at scale.
(Q2) Does DCNO continue to show performance gains when scaling to deeper networks (e.g., 12 or 16 layers)?
We thank the reviewer for this suggestion. We extended the depth-scaling experiments on NS2D to 12 and 16 operator blocks, and included GNOT+HC as an additional baseline.
| Operator blocks | GNOT | GNOT+HC | DCNO | Reduction against GNOT | Reduction against GNOT+HC |
|---|---|---|---|---|---|
| 1 | 0.1150 | 0.1118 | 0.1194 | −3.8% | −6.8% |
| 2 | 0.1134 | 0.1135 | 0.1023 | 9.8% | 9.8% |
| 3 | 0.1059 | 0.1069 | 0.0989 | 6.6% | 7.5% |
| 4 | 0.1030 | 0.1021 | 0.0897 | 12.9% | 12.1% |
| 5 | 0.0957 | 0.0926 | 0.0878 | 8.2% | 5.1% |
| 6 | 0.0916 | 0.0912 | 0.0832 | 9.2% | 8.7% |
| 7 | 0.0912 | 0.0945 | 0.0854 | 6.3% | 9.6% |
| 8 | 0.0893 | 0.0919 | 0.0799 | 10.5% | 13.1% |
| 12 | 0.0895 | 0.0935 | 0.0803 | 9.2% | 14.1% |
| 16 | 0.0853 | 0.0861 | 0.0788 | 7.6% | 8.5% |
| Error reduction (1→16 blocks) | 25.8% | 23.0% | 34.0% | — | — |
- Effectiveness at greater depths: DCNO maintains the lowest prediction error at both 12 and 16 blocks. At 16 blocks, it reduces errors by 7.6% over GNOT and 8.5% over GNOT+HC.
- Better depth utilization: From 1 to 16 blocks, DCNO achieves a 34.0% error reduction, compared with 25.8% for GNOT and 23.0% for GNOT+HC.
- All three models show relatively limited additional improvements beyond 8 blocks. This may indicate diminishing returns from increasing model depth on the current NS2D benchmark. Nevertheless, DCNO keeps a consistent advantage over both baselines at greater depths.
Thank authors for the detailed response and additional experiments. The comparison with GNOT+HC clarifies how DCNO’s protected memory stream preserves information across layers, and the ablation on the memory write scale shows how this choice affects prediction error. On NS2D, DCNO outperforms both GNOT and GNOT+HC at 12 and 16 blocks, although the gains from increasing its depth beyond eight blocks are modest. It remains unclear whether this trend holds on other datasets.
Response to Reviewer 6mU2: part 1
We sincerely thank the reviewer for the constructive feedback and for recognizing the clarity of our presentation, the lightweight and well-motivated architecture, and the comprehensive experimental evaluation. Please find our responses to the comments below.
We thank the reviewer for this valuable suggestion. We agree that comparisons with GNOT+HC are important for distinguishing the benefits of multi-stream connections from those of the protected memory design. We have therefore extend the experiments and included GNOT+HC in the physical-condition alignment (CKA) analysis in Section 4.2 (Figure 1) and the initial-state alignment (ISA) analysis in Section 4.3 (Figure 5).
1. Physical-condition alignment (CKA, Section 4.2, Figure 1).
2. Initial-state alignment (ISA, Section 4.3, Figure 5).
3. Ablation on the predictive error (Table 1).
We thank the reviewer for this suggestion. We conducted additional component-wise ablations to better understand the contributions of different DCNO components.
Memory write scale. We evaluate different values of ϵmem on Heat2D and NS2D.
These results support the benefit of a protected, slowly evolving memory stream.