- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PANO, a neural operator that aggregates observations into parallel latent spaces with a shared processing backbone. It introduces Coupled Assignment Projection for encoding and decoding, and Pairwise Latent Space Attention for cross-space information exchange. The method is supported by kernel-representation and capacity analyses, and evaluated on nine PDE and engineering benchmarks, reporting improved accuracy and competitive computational efficiency.
Critical strengths
-
Evaluation on nine PDE and engineering benchmarks shows promising prediction accuracy across several tasks, providing evidence of practical utility.
-
Sharing the backbone controls parameter overhead. PANO-P uses fewer parameters and FLOPs than the listed baselines, suggesting that its gains cannot be attributed solely to increased parameter count.
Critical weaknesses
-
The benefit of parallel organization is not sufficiently isolated from increased capacity. The theory compares (M) tokens with (HM) tokens, while a single space with (HM) tokens has the same rank upper bound. Controlled comparisons against enlarged single-space models are missing, and the rank and null-space analyses do not establish that additional retained information improves PDE prediction.
-
Architectural novelty and physical justification remain insufficiently established. Learned aggregation, multiple branches, and cross-branch attention have precedents in computer vision, while PDE models also employ multi-head slice aggregation. The paper does not sufficiently establish its substantive distinction, the physical meaning of different aggregation patterns, or their task-relevant complementarity. Direct comparisons against shared-backbone ensembles and conventional cross-attention are also missing.
Questions for authors
Regarding Weakness 1: Capacity controls and theoretical justification
- In Table 6, is the total token count (HM) preserved when (H) changes? What are the feature dimension, parameter count, and FLOPs for each configuration?
- Does the advantage persist against single-space models with more tokens or wider features under separately matched token, parameter, and computational budgets?
- Does the theory guarantee an increase in actual rank or a strict reduction in the common null space? How do these properties relate to prediction error for the target PDE operator?
- Are the smaller gains in Table 6, such as the Airfoil error reduction from 0.0053 to 0.0052 without PaLA, consistent across random seeds?
Regarding Weakness 2: Novelty and complementarity
- What is PANO’s substantive contribution beyond learned aggregation, multiple branches, and cross-branch attention in computer vision, and multi-head slice aggregation in PDE models?
- Do different aggregation mappings correspond to identifiable physical scales, modes, or interactions? If not, how should “physical interactions” be interpreted?
- What quantitative evidence shows that the spaces retain complementary information relevant to prediction rather than redundant representations?
- Under comparable budgets, does PaLA provide additional benefits over shared-backbone ensembles or conventional cross-attention?
8 comments
Response to Question 3
Since PANO is designed as a general PDE solver, we do not impose strict structural priors that would guarantee a strict increase in actual rank or a strict reduction in the common null space for arbitrary model parameters. We also do not claim that our theoretical analysis guarantees a strictly higher actual rank or a strictly smaller common null space.
Moreover, PANO is a learnable method. The final state of PANO is highly dependent on the training process. Therefore, our theoretical analysis establishes an expanded allowable rank bound and a non-increasing common null space, rather than unconditional strict improvements. To examine whether these improvements are realized after training, we analyze the actual forward assignments and input features of PANO-P on Airfoil.
Numerical Rank Analysis
We evaluate the realized numerical rank of the assignment maps learned by PANO-P on Airfoil. The rank is computed using FP64 SVD with the standard numerical-rank tolerance.
| Block | for each space | |
|---|---|---|
| Block 1 | 512 | 2048 |
| Block 2 | 512 | 2048 |
is the assignment map of the -th latent space, and is the parallel assignment map. Thus, each latent space has rank 512, whereas the parallel assignments have rank 2048.
Moreover, we can get:
\sum_{h=1}^{4}\operatorname{rank}(A^h) =2048.$$ The results show that each latent-space assignment map $A^h$ is full rank. The four learned assignment maps contribute linearly independent directions. Therefore, PANO achieves a higher rank through parallel latent spaces in real experiments. ## **Common Null Space Analysis** We first extend the analysis in Appendix E.3 to the actual input-dependent encoding. For each latent space $h$, let $P^h=A^h(A^h)^\dagger$ be the orthogonal projector onto its column space, where $\dagger$ denotes the Moore--Penrose pseudoinverse. We define: $$F_{\mathrm{keep}}^h=P^hF, \qquad F_{\mathrm{ignore}}^h=(I-P^h)F.$$ Then, we have: $$F=F_{\mathrm{keep}}^h+F_{\mathrm{ignore}}^h,$$ $$(A^h)^\top F_{\mathrm{ignore}}^h=0,$$ $$(A^h)^\top F=(A^h)^\top F_{\mathrm{keep}}^h.$$ Thus, although the assignment map is constructed from $F$, the component $F_{\mathrm{ignore}}^h$ is invisible to aggregation under that assignment in the same forward pass. For PANO-P on Airfoil, we further compute the actual $F_{\mathrm{ignore}}$ and $F_{\mathrm{keep}}$ and report the ignored fraction $\|F_{\mathrm{ignore}}\|_F^2/\|F\|_F^2$. “Jointly” uses the concatenated assignment map to measure the component omitted by all four latent spaces. | Block | Space 1 | Space 2 | Space 3 | Space 4 | Jointly | |---|---:|---:|---:|---:|---:| | Block 1 | 7.88% | 0.09% | 0.55% | 10.22% | 0.000014% | | Block 2 | 0.11% | 0.66% | 0.18% | 0.69% | 0.000083% | The results show that each latent space has a distinct, nonzero $F_{\mathrm{ignore}}$, while the information jointly omitted by all four latent spaces is substantially smaller than that omitted by any individual latent space. These quantitative results verify that PANO achieves a strict reduction in the common encoding null space in real experiments. We additionally analyze the omitted information for the $H=1, M=2048$ variant. The ignored fraction averages 0.00022550% and 0.00053391% in Stages 1 and 2, respectively. PANO-P has approximately 17 times and 6 times lower jointly omitted information, supporting the benefit of parallel latent spaces. ## **Relation to Prediction Error** A higher rank upper bound provides greater model capacity, improving the modeling of physical fields. Meanwhile, if information is discarded during encoding, the latent spaces cannot observe that information. Therefore, the higher realized encoding rank and smaller common null space provide greater representational capacity and preserve more task-relevant information for modeling the target fields. For the perturbation analysis of the information retained by each latent space, please refer to our response to Question 7.Response to Question 4
To address this concern, we conduct comprehensive multiple-seed experiments for the ablation study. The results are shown below. Parallel latent spaces, CAP, and PaLA each contribute to improved performance. More importantly, these components exhibit a clear synergistic effect, jointly yielding substantially greater performance gains.
| CAP | PaLA | Airfoil | Geometry (Metal) | Geometry (Rubber) | Stress (Metal) | Stress (Rubber) | PEEQ (Metal) | PEEQ (Rubber) | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | ✗ | ✗ | 0.0058 ± 0.0001 | 0.0061 ± 0.0001 | 0.0053 ± 0.0001 | 0.0761 ± 0.0004 | 0.0699 ± 0.0005 | 0.2163 ± 0.0046 | 0.1768 ± 0.0028 |
| 1 | ✓ | ✗ | 0.0052 ± 0.0001 | 0.0065 ± 0.0001 | 0.0056 ± 0.0001 | 0.0767 ± 0.0003 | 0.0707 ± 0.0007 | 0.2215 ± 0.0051 | 0.1831 ± 0.0048 |
| 4 | ✗ | ✗ | 0.0054 ± 0.0004 | 0.0060 ± 0.0002 | 0.0052 ± 0.0001 | 0.0758 ± 0.0004 | 0.0694 ± 0.0002 | 0.2137 ± 0.0033 | 0.1742 ± 0.0033 |
| 4 | ✓ | ✗ | 0.0051 ± 0.0001 | 0.0059 ± 0.0004 | 0.0051 ± 0.0003 | 0.0752 ± 0.0007 | 0.0691 ± 0.0005 | 0.2064 ± 0.0084 | 0.1699 ± 0.0057 |
| 4 | ✗ | ✓ | 0.0053 ± 0.0007 | 0.0057 ± 0.0004 | 0.0049 ± 0.0004 | 0.0748 ± 0.0005 | 0.0687 ± 0.0011 | 0.2068 ± 0.0119 | 0.1697 ± 0.0098 |
| 4 | ✓ | ✓ | 0.0046 ± 0.0003 | 0.0056 ± 0.0003 | 0.0047 ± 0.0003 | 0.0744 ± 0.0003 | 0.0673 ± 0.0007 | 0.1956 ± 0.0117 | 0.1608 ± 0.0091 |
- Compared with , introducing parallel latent spaces substantially improves model performance. For a comprehensive analysis of and , please refer to our response to Question 2.
- PaLA also improves performance. However, without CAP, PaLA exhibits greater instability, leading to a higher deviation across random seeds.
- The complete PANO-P achieves the best performance. These results demonstrate the synergy between CAP and PaLA.
We sincerely thank the reviewer for recognizing the significant advantages of PANO across nine benchmarks. We address the weaknesses and questions in detail below.
Response to Question 1
We apologize for not clearly explaining the experimental settings used in Table 6. Table 6 is designed to ablate parallel latent spaces, CAP, and PaLA under the PANO-P configuration. Therefore, we retain the basic PANO-P settings of M=512 and C=96. The total number of latent tokens HM are 512 and 2048 for H=1 and H=4, respectively. Comprehensive controlled comparisons with fixed HM are provided in our response to Question 2. The parameter count and GFLOPs of each variant are as follows.
Response to Question 2
We acknowledge that we did not sufficiently distinguish the performance gains introduced by parallel latent spaces from those resulting from increased model capacity. To address this issue, we conduct extensive ablations on H and M based on the depth setting of PANO-P.
The results show that parallel latent spaces improve performance at both total token settings, HM=512 and HM=2048. Introducing parallel latent spaces can substantially improve model performance. Notably, even compared with the single-space variant with a larger feature dimension (H=1, M=2048, and C=192), PANO with H=4, M=512, and C=96 achieves better performance with substantially fewer parameters and GFLOPs.