acceptodds
Under review as a conference paper at ICLR 2027

PANO: Parallel Latent Space Neural Operator for Solving PDEs

Abstract

Aggregating observations into a limited number of latent tokens and learning physical interactions in the latent space has emerged as an effective approach for Transformer-based neural operators to solve Partial Differential Equation (PDE) problems involving large-scale observations. However, existing methods typically compress observations using a single aggregation pattern, making physical information not retained by that pattern invisible to the latent space and thereby constraining physical interactions within it. To address this limitation, we propose PArallel Latent Space Neural Operator (PANO), a novel neural operator for solving PDEs that formulates physical interactions among observations as an additive combination of interaction results from parallel latent spaces with shared parameters. Specifically, PANO transforms observations into multiple distinct compressed representations through independent mappings, learns physical interactions in parallel based on these representations, and finally fuses the decoded results to predict the target physical field. Within PANO, we further propose Coupled Assignment Projection (CAP) to construct bidirectionally consistent encoding and decoding mappings, and Pairwise Latent Space Attention Layer (PaLA) to jointly update latent representations of each latent space through adaptive cross-space interactions. Extensive experiments on nine benchmarks spanning diverse physical systems, geometries, and observation scales demonstrate that PANO achieves state-of-the-art performance. Notably, PANO reduces equivalent plastic strain prediction errors on Unilateral Stamping by 28.3% (or more) and lift-coefficient prediction errors on AirCraft by 21.4% (or more) compared with existing methods. The source code will be released.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operator, Partial Differential Equations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gSPNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PANO, a neural operator that aggregates observations into parallel latent spaces with a shared processing backbone. It introduces Coupled Assignment Projection for encoding and decoding, and Pairwise Latent Space Attention for cross-space information exchange. The method is supported by kernel-representation and capacity analyses, and evaluated on nine PDE and engineering benchmarks, reporting improved accuracy and competitive computational efficiency.

Critical strengths
  1. Evaluation on nine PDE and engineering benchmarks shows promising prediction accuracy across several tasks, providing evidence of practical utility.

  2. Sharing the backbone controls parameter overhead. PANO-P uses fewer parameters and FLOPs than the listed baselines, suggesting that its gains cannot be attributed solely to increased parameter count.

Critical weaknesses
  1. The benefit of parallel organization is not sufficiently isolated from increased capacity. The theory compares (M) tokens with (HM) tokens, while a single space with (HM) tokens has the same rank upper bound. Controlled comparisons against enlarged single-space models are missing, and the rank and null-space analyses do not establish that additional retained information improves PDE prediction.

  2. Architectural novelty and physical justification remain insufficiently established. Learned aggregation, multiple branches, and cross-branch attention have precedents in computer vision, while PDE models also employ multi-head slice aggregation. The paper does not sufficiently establish its substantive distinction, the physical meaning of different aggregation patterns, or their task-relevant complementarity. Direct comparisons against shared-backbone ensembles and conventional cross-attention are also missing.

Questions for authors

Regarding Weakness 1: Capacity controls and theoretical justification

  1. In Table 6, is the total token count (HM) preserved when (H) changes? What are the feature dimension, parameter count, and FLOPs for each configuration?
  2. Does the advantage persist against single-space models with more tokens or wider features under separately matched token, parameter, and computational budgets?
  3. Does the theory guarantee an increase in actual rank or a strict reduction in the common null space? How do these properties relate to prediction error for the target PDE operator?
  4. Are the smaller gains in Table 6, such as the Airfoil error reduction from 0.0053 to 0.0052 without PaLA, consistent across random seeds?

Regarding Weakness 2: Novelty and complementarity

  1. What is PANO’s substantive contribution beyond learned aggregation, multiple branches, and cross-branch attention in computer vision, and multi-head slice aggregation in PDE models?
  2. Do different aggregation mappings correspond to identifiable physical scales, modes, or interactions? If not, how should “physical interactions” be interpreted?
  3. What quantitative evidence shows that the spaces retain complementary information relevant to prediction rather than redundant representations?
  4. Under comparable budgets, does PaLA provide additional benefits over shared-backbone ensembles or conventional cross-attention?
8 comments

We sincerely thank the reviewer for recognizing the significant advantages of PANO across nine benchmarks. We address the weaknesses and questions in detail below.

Response to Question 1

We apologize for not clearly explaining the experimental settings used in Table 6. Table 6 is designed to ablate parallel latent spaces, CAP, and PaLA under the PANO-P configuration. Therefore, we retain the basic PANO-P settings of M=512M=512 and C=96C=96. The total number of latent tokens HMHM are 512 and 2048 for H=1H=1 and H=4H=4, respectively. Comprehensive controlled comparisons with fixed HMHM are provided in our response to Question 2. The parameter count and GFLOPs of each variant are as follows.

HHMMHMHMCCCAPPaLAParams. (M)GFLOPs
151251296✗✗1.26457.29
151251296✓✗1.26457.29
4512204896✗✗1.784170.46
4512204896✓✗1.784170.46
4512204896✗✓1.989171.78
4512204896✓✓1.989171.78

Response to Question 2

We acknowledge that we did not sufficiently distinguish the performance gains introduced by parallel latent spaces from those resulting from increased model capacity. To address this issue, we conduct extensive ablations on HH and MM based on the depth setting of PANO-P.

HMHMHHMMCCParams. (M)GFLOPsGeometry (Metal) ↓\downarrowGeometry (Rubber) ↓\downarrowStress (Metal) ↓\downarrowStress (Rubber) ↓\downarrowPEEQ (Metal) ↓\downarrowPEEQ (Rubber) ↓\downarrow
5121512961.2657.290.00660.00570.07690.07100.22560.1876
5122256961.5464.600.00650.00560.07720.07090.21920.1814
5124128961.6979.270.00640.00550.07610.07100.22240.1824
204812048961.56153.170.00620.00540.07650.07250.21320.1805
2048120481925.42363.420.00640.00540.07500.06910.21570.1770
204821024961.84158.350.00600.00540.07600.07000.21150.1778
20484512961.99171.780.00520.00440.07430.06670.18270.1515

The results show that parallel latent spaces improve performance at both total token settings, HM=512HM=512 and HM=2048HM=2048. Introducing parallel latent spaces can substantially improve model performance. Notably, even compared with the single-space variant with a larger feature dimension (H=1H=1, M=2048M=2048, and C=192C=192), PANO with H=4H=4, M=512M=512, and C=96C=96 achieves better performance with substantially fewer parameters and GFLOPs.

AuthorsOpenReview ↗

Response to Question 3

Since PANO is designed as a general PDE solver, we do not impose strict structural priors that would guarantee a strict increase in actual rank or a strict reduction in the common null space for arbitrary model parameters. We also do not claim that our theoretical analysis guarantees a strictly higher actual rank or a strictly smaller common null space.

Moreover, PANO is a learnable method. The final state of PANO is highly dependent on the training process. Therefore, our theoretical analysis establishes an expanded allowable rank bound and a non-increasing common null space, rather than unconditional strict improvements. To examine whether these improvements are realized after training, we analyze the actual forward assignments and input features of PANO-P on Airfoil.

Numerical Rank Analysis

We evaluate the realized numerical rank of the assignment maps learned by PANO-P on Airfoil. The rank is computed using FP64 SVD with the standard numerical-rank tolerance.

Blockrank⁡(Ah)\operatorname{rank}(A^h) for each spacerank⁡(A⊕)\operatorname{rank}(A_{\oplus})
Block 15122048
Block 25122048

AhA^h is the assignment map of the hh-th latent space, and A⊕A_{\oplus} is the parallel assignment map. Thus, each latent space has rank 512, whereas the parallel assignments have rank 2048.

Moreover, we can get:

\sum_{h=1}^{4}\operatorname{rank}(A^h) =2048.$$ The results show that each latent-space assignment map $A^h$ is full rank. The four learned assignment maps contribute linearly independent directions. Therefore, PANO achieves a higher rank through parallel latent spaces in real experiments. ## **Common Null Space Analysis** We first extend the analysis in Appendix E.3 to the actual input-dependent encoding. For each latent space $h$, let $P^h=A^h(A^h)^\dagger$ be the orthogonal projector onto its column space, where $\dagger$ denotes the Moore--Penrose pseudoinverse. We define: $$F_{\mathrm{keep}}^h=P^hF, \qquad F_{\mathrm{ignore}}^h=(I-P^h)F.$$ Then, we have: $$F=F_{\mathrm{keep}}^h+F_{\mathrm{ignore}}^h,$$ $$(A^h)^\top F_{\mathrm{ignore}}^h=0,$$ $$(A^h)^\top F=(A^h)^\top F_{\mathrm{keep}}^h.$$ Thus, although the assignment map is constructed from $F$, the component $F_{\mathrm{ignore}}^h$ is invisible to aggregation under that assignment in the same forward pass. For PANO-P on Airfoil, we further compute the actual $F_{\mathrm{ignore}}$ and $F_{\mathrm{keep}}$ and report the ignored fraction $\|F_{\mathrm{ignore}}\|_F^2/\|F\|_F^2$. “Jointly” uses the concatenated assignment map to measure the component omitted by all four latent spaces. | Block | Space 1 | Space 2 | Space 3 | Space 4 | Jointly | |---|---:|---:|---:|---:|---:| | Block 1 | 7.88% | 0.09% | 0.55% | 10.22% | 0.000014% | | Block 2 | 0.11% | 0.66% | 0.18% | 0.69% | 0.000083% | The results show that each latent space has a distinct, nonzero $F_{\mathrm{ignore}}$, while the information jointly omitted by all four latent spaces is substantially smaller than that omitted by any individual latent space. These quantitative results verify that PANO achieves a strict reduction in the common encoding null space in real experiments. We additionally analyze the omitted information for the $H=1, M=2048$ variant. The ignored fraction averages 0.00022550% and 0.00053391% in Stages 1 and 2, respectively. PANO-P has approximately 17 times and 6 times lower jointly omitted information, supporting the benefit of parallel latent spaces. ## **Relation to Prediction Error** A higher rank upper bound provides greater model capacity, improving the modeling of physical fields. Meanwhile, if information is discarded during encoding, the latent spaces cannot observe that information. Therefore, the higher realized encoding rank and smaller common null space provide greater representational capacity and preserve more task-relevant information for modeling the target fields. For the perturbation analysis of the information retained by each latent space, please refer to our response to Question 7.

Response to Question 4

To address this concern, we conduct comprehensive multiple-seed experiments for the ablation study. The results are shown below. Parallel latent spaces, CAP, and PaLA each contribute to improved performance. More importantly, these components exhibit a clear synergistic effect, jointly yielding substantially greater performance gains.

HHCAPPaLAAirfoil ↓\downarrowGeometry (Metal) ↓\downarrowGeometry (Rubber) ↓\downarrowStress (Metal) ↓\downarrowStress (Rubber) ↓\downarrowPEEQ (Metal) ↓\downarrowPEEQ (Rubber) ↓\downarrow
1✗✗0.0058 ± 0.00010.0061 ± 0.00010.0053 ± 0.00010.0761 ± 0.00040.0699 ± 0.00050.2163 ± 0.00460.1768 ± 0.0028
1✓✗0.0052 ± 0.00010.0065 ± 0.00010.0056 ± 0.00010.0767 ± 0.00030.0707 ± 0.00070.2215 ± 0.00510.1831 ± 0.0048
4✗✗0.0054 ± 0.00040.0060 ± 0.00020.0052 ± 0.00010.0758 ± 0.00040.0694 ± 0.00020.2137 ± 0.00330.1742 ± 0.0033
4✓✗0.0051 ± 0.00010.0059 ± 0.00040.0051 ± 0.00030.0752 ± 0.00070.0691 ± 0.00050.2064 ± 0.00840.1699 ± 0.0057
4✗✓0.0053 ± 0.00070.0057 ± 0.00040.0049 ± 0.00040.0748 ± 0.00050.0687 ± 0.00110.2068 ± 0.01190.1697 ± 0.0098
4✓✓0.0046 ± 0.00030.0056 ± 0.00030.0047 ± 0.00030.0744 ± 0.00030.0673 ± 0.00070.1956 ± 0.01170.1608 ± 0.0091
  1. Compared with H=1H=1, introducing parallel latent spaces substantially improves model performance. For a comprehensive analysis of HH and MM, please refer to our response to Question 2.
  2. PaLA also improves performance. However, without CAP, PaLA exhibits greater instability, leading to a higher deviation across random seeds.
  3. The complete PANO-P achieves the best performance. These results demonstrate the synergy between CAP and PaLA.
Reviewer_LXjSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PANO, a latent-space Transformer neural operator. In each block, the input point tokens are projected into H parallel latent spaces using different learnable assignment projectors. These latent spaces share the same latent module consisting of self-attention layers. The outputs are then decoded back to the original observation space and averaged.

The experiments are conducted on nine benchmarks, including five standard benchmarks (Airfoil, Plasticity, NS2D, Elasticity, and Darcy at three resolutions), as well as AirfRANS, Unilateral Stamping, ShapeNetCar, and AirCraft. The authors consider two configurations, PANO-P (H=4, M=512, C=96) and PANO-D (H=2, M=1024, C=192). The results are generally competitive with existing methods, and the paper also includes ablation studies on H, CAP, and PaLA.

Critical strengths
  • The experimental results are quite strong on several benchmarks, especially Plasticity, NS2D (PANO-D), Elasticity, and AirfRANS. The improvement on the OOD lift coefficient in AirfRANS is also interesting. On Unilateral Stamping, PANO-P outperforms Unisoma, which is specifically designed for this problem, on all six metrics.

  • I think the idea of using multiple latent spaces with different assignment patterns is interesting. Sharing the latent module among these spaces also seems reasonable. The visualizations in Fig. 1 and Figs. 9–24 show that different spaces can capture different spatial granularities, which provides some evidence for the motivation of the method.

Critical weaknesses
  1. My main concern is about the novelty compared with existing neural operators. The paper argues that previous methods compress observations using "a single aggregation pattern". However, this description seems not fully accurate for Transolver, Transolver++, and LinearNO. These methods already use multi-head physics attention, where different heads have their own slice assignments. In each head, the point features are aggregated into slice tokens, normalized by the total slice weights, processed by attention, and then mapped back to the original points. This is quite similar to the column normalization used in CAP. Therefore, I am not fully convinced that using multiple assignment patterns is a new contribution by itself. The authors should explain more clearly what is fundamentally different from existing multi-head physics attention.

  2. I also have some concerns about the experimental comparisons. Several reported improvements may be affected by differences in experimental settings.

  • Baseline results: According to Appendix K, all baseline results are taken from previous papers, while PANO is trained using its own training schedule. It is unclear whether the training budgets are comparable.

  • AirfRANS sampling: PANO follows the sampling strategy of MoNo, where all surface points are included in each training sample. However, the reported LinearNO and Transolver baselines use random point sampling. Since surface prediction accuracy is important for AirfRANS, this difference could contribute to the reported 87% reduction in surface error. I think this comparison is not fully fair without controlling the sampling strategy.

  • Random seeds: All experiments are conducted with only one random seed (seed 0), and the last-epoch checkpoint is used. There are no standard deviations or other uncertainty estimates. This makes it difficult to judge whether some smaller improvements are reliable.

  • Ablation results: Some differences in the ablation study are very small, for example 0.0053 versus 0.0052 on Airfoil. It is difficult to draw a strong conclusion from these results without multiple seeds. Also, the benefit of CAP does not seem consistent. For H=4 without PaLA, adding CAP makes the Airfoil error increase from 0.0050 to 0.0052. For H=1 on Stamping, CAP makes all three metrics worse. Therefore, I think the current ablation results do not sufficiently support the claim that CAP consistently improves parallel latent spaces.

  • Model size: According to Table 5, PANO-D has around 2.5 times as many parameters as the baselines. Since PANO-D achieves most of the improvements on the standard benchmarks, it is difficult to separate the benefit of the proposed architecture from the benefit of using a larger model.

Questions for authors
  1. In Table 6, for the H=1 configuration, is M set to 512 or 2048? If M=512, then the total number of latent tokens is much smaller than the H=4 setting. I would suggest adding a single-space baseline with M=2048, with matched depth and FLOPs, both with and without CAP. This would help clarify whether the improvement really comes from multiple latent spaces.

  2. Could the authors explain more clearly how one latent space in PANO differs from one head in Transolver's physics attention? Apart from the decoding normalization, channel width, and parameter sharing, the two designs seem quite similar. It would be helpful to compare PANO with Transolver or LinearNO using matched numbers of heads (H), slices (M), latent depth, and parameter count.

  3. For AirfRANS, could the authors rerun LinearNO and Transolver with the same surface-inclusive sampling strategy and training epochs as PANO? Alternatively, PANO could be evaluated using the sampling strategy of the baselines. Also, what are the training budgets of the baseline results copied from previous papers?

  4. Could the authors report the mean and standard deviation over at least three random seeds? I think this is particularly important for Airfoil, NS2D, AirfRANS, and AirCraft, where some reported improvements are small or the test set is limited.

  5. Regarding CAP, my understanding is that it guarantees constant column sums, but the row sums of the decoding assignment can still vary across points, with only their average equal to 1. Is there any additional normalization during decoding? Also, why is this design better than standard row-normalized de-slicing? The ablation results in Table 6 show that CAP can hurt performance in some configurations, so I think more explanation is needed.

14 comments
AuthorsOpenReview ↗

Response to W1 (Comparison PANO with Multi-Head Physics Attention) & Q2

We sincerely thank the reviewer for recognizing the strong empirical performance of PANO, the effectiveness of constructing multiple latent spaces, and the supporting evidence provided by our visualizations. We address the weaknesses and questions in detail below.

Analysis

We thank the reviewer for the insightful comments. We next provide a detailed analysis of the differences between PANO and multi-head physics attention.

Let CC denote the feature dimension of point tokens, and HH denote the number of parallel latent spaces in PANO or the number of heads in multi-head physics attention. In multi-head physics attention, each head constructs and processes only a corresponding C/HC/H-dimensional feature subspace. After decoding, the outputs of all heads are treated as fragments and concatenated to form the complete feature representation. Multi-head physics attention has two structural designs:

  1. The space constructed by each head can only observe its corresponding C/HC/H-dimensional features. The physical interaction within that space is subject to the same restriction.
  2. The complete decoded field representation is jointly formed by all heads.

Therefore, each head is not an independent representation of the physical field, but one of multiple feature subspaces. This leads to substantial limitations, as shown in the experiments of Transolver and LinearNO:

  1. Typical slice counts are 32 and 64, giving each feature subspace relatively limited capacity for modeling physical interactions.
  2. As shown in Table 9 of the Transolver appendix, further increasing the number of slices to 1024 causes substantial performance degradation. We would also like to emphasize that Transolver describes the multi-head version of physics attention as a natural extension of the single-head version following the convention of attention mechanisms, rather than as a specifically design.

In contrast, each latent space in PANO independently constructs a distinct, full-width representation of the same physical field from the complete observation features. These latent spaces have equivalent structural roles. Our extensive experiments further show that PANO outperforms Transolver and other methods based on multi-head physics attention.

Experiments

We would like to emphasize that Transolver, Transolver++, and LinearNO have fundamentally different architectures from PANO. In particular, Transolver does not contain a latent space composed of multiple interaction layers as in PANO. Therefore, it is difficult to construct variants of these methods with fully matched numbers of heads (HH), slices (MM), latent depth, and parameter count without changing their original architectures. For a fair comparison, we directly refer to the Transolver variant with the same total number of latent tokens as PANO (HM=2048HM=2048).

MethodElasticity ↓\downarrowPlasticity ↓\downarrowNS2D ↓\downarrow
Transolver (Default)0.00650.00130.0882
Transolver (HM=2048HM=2048)0.00540.00130.0856
PANO-P (ours)0.00400.00060.0720
PANO-D (ours)0.00290.00040.0395

PANO substantially outperforms both Transolver variants.

Response to W2 (Baseline results) & Q3 (Training budgets of the baseline results)

We strictly follow the same dataset splits as Transolver, LinearNO, Transolver++, and Unisoma. Moreover, PANO uses the same or fewer total training steps than baselines on each benchmark.

Response to W2 (AirfRANS sampling) & Q3 (Comparison under the same sampling method)

To clarify this issue, we further report the results of LinearNO using the same sampling method as PANO.

MethodVol. ↓\downarrowSurf. ↓\downarrowCL↓C_L\downarrowρL↑\rho_L\uparrow
LinearNO (standard sampling method)0.00110.00770.04910.9992
LinearNO (Same sampling method as PANO)0.00110.00800.06140.9992
PANO-P (ours)0.00050.00100.02360.9997
PANO-D (ours)0.00090.00310.02250.9997

LinearNO achieves similar results under both sampling methods. PANO still substantially outperforms LinearNO.

Response to W2 (Random seeds) & Q4

We would like to emphasize that PANO reduces prediction errors relative to the best competing methods by 87.0%, 63.6%, and 43.5% on AirfRANS surface fields, Plasticity, and NS2D, respectively. These substantial improvements across different benchmarks support the effectiveness of PANO. To further assess sensitivity to random seeds, we compare Transolver, LinearNO, PANO-P, and PANO-D on Airfoil and Elasticity, where PANO shows relatively smaller improvements over other methods.

MethodAirfoil ↓\downarrowElasticity ↓\downarrow
Transolver0.0053 ± 0.00010.0064 ± 0.0002
LinearNO0.0053 ± 0.00030.0054 ± 0.0004
PANO-P (ours)0.0046 ± 0.00030.0042 ± 0.0002
PANO-D (ours)0.0045 ± 0.00070.0029 ± 0.0001

PANO consistently outperforms previous SOTA models. Since NS2D, AirfRANS, and AirCraft are computationally expensive benchmarks, repeating experiments on these benchmarks across multiple seeds requires substantial computation. We will progressively add these results during the rebuttal period.

AuthorsOpenReview ↗

Response to W2 (Ablation results) & Q5

We apologize for not fully explaining the advantages of CAP over standard softmax-based projection. To address this issue, we first provide multiple-seed results for the ablation study to demonstrate the benefits of CAP. We then present a detailed quantitative analysis of decoding to clarify how CAP improves latent-space stability relative to standard softmax-based projection.

Multiple-seed experiments

Multiple-seed experiments for the ablation study are shown below. Parallel latent spaces, CAP, and PaLA each contribute to improved performance. More importantly, these components exhibit a clear synergistic effect, jointly yielding substantially greater performance gains.

HHCAPPaLAAirfoil ↓\downarrowGeometry (Metal) ↓\downarrowGeometry (Rubber) ↓\downarrowStress (Metal) ↓\downarrowStress (Rubber) ↓\downarrowPEEQ (Metal) ↓\downarrowPEEQ (Rubber) ↓\downarrow
1✗✗0.0058 ± 0.00010.0061 ± 0.00010.0053 ± 0.00010.0761 ± 0.00040.0699 ± 0.00050.2163 ± 0.00460.1768 ± 0.0028
1✓✗0.0052 ± 0.00010.0065 ± 0.00010.0056 ± 0.00010.0767 ± 0.00030.0707 ± 0.00070.2215 ± 0.00510.1831 ± 0.0048
4✗✗0.0054 ± 0.00040.0060 ± 0.00020.0052 ± 0.00010.0758 ± 0.00040.0694 ± 0.00020.2137 ± 0.00330.1742 ± 0.0033
4✓✗0.0051 ± 0.00010.0059 ± 0.00040.0051 ± 0.00030.0752 ± 0.00070.0691 ± 0.00050.2064 ± 0.00840.1699 ± 0.0057
4✗✓0.0053 ± 0.00070.0057 ± 0.00040.0049 ± 0.00040.0748 ± 0.00050.0687 ± 0.00110.2068 ± 0.01190.1697 ± 0.0098
4✓✓0.0046 ± 0.00030.0056 ± 0.00030.0047 ± 0.00030.0744 ± 0.00030.0673 ± 0.00070.1956 ± 0.01170.1608 ± 0.0091
  1. Compared with H=1H=1, introducing parallel latent spaces substantially improves model performance. For a comprehensive analysis of HH and MM, please refer to our response to Question 1.
  2. PaLA also improves performance. However, without CAP, PaLA exhibits greater instability, leading to a higher deviation across random seeds.
  3. The complete PANO-P achieves the best performance. These results demonstrate the synergy between CAP and PaLA.

Detailed quantitative analysis of decoding

CAP is mainly designed to construct a more stable latent space by normalizing the total encoding weight received by each latent token and fixing the total decoding weight. To compare the decoding behavior of CAP and standard softmax-based projection in detail, we further quantitatively analyze decoding weights of tokens on Airfoil.

MethodBlock IDScaleTop10 token weightToken weight CV
Softmax195.88x62.71%6.396
Softmax226.35x20.84%1.905
CAP (ours)11.00x1.95%3.70e-7
CAP (ours)21.00x1.95%2.15e-7
  • Scale: The largest total decoding weight of a token divided by the balanced value N/MN/M.
  • Top10 token weight: The fraction of total decoding weight contributed by the 10 tokens with the largest weights.
  • Token weight CV: The coefficient of variation across token decoding weights.

Standard softmax-based projection constrains the decoding mass received by each point, but not the total mass contributed by each latent token. Consequently, almost all points may primarily read from the same token, leading to unstable pointwise decoding. As shown in the quantitative analysis above, softmax-based projection leads to highly uneven decoding weights and heavy reliance on a small subset of tokens for decoding. CAP substantially alleviates these issues.

We further quantitatively analyze the decoding weights of points on Airfoil.

MethodBlock IDScaleTop10 point weightPoint weight CV
Softmax11.00x0.09%5.62e-8
Softmax21.00x0.09%4.55e-8
CAP (ours)114.32x0.98%1.368
CAP (ours)23.70x0.29%0.283
  • Scale: The largest total decoding weight received by a point divided by the balanced value 1.
  • Top10 point weight: The fraction of total decoding weight received by the 10 points with the largest weights.
  • Point weight CV: The coefficient of variation across point decoding weights.

The analyses of token and point decoding weights show that standard softmax-based projection leads to severe imbalance in token usage. CAP first strictly guarantees balanced token usage. Moreover, although CAP imposes no normalization on point decoding weights, it still avoids comparably extreme allocation patterns and maintains relatively balanced weights across points. The benefits of CAP for latent-space learning are also supported by the more comprehensive ablation studies presented above.

Reviewer_xkJLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PANO, a neural operator that encodes observations into multiple latent spaces using independent assignment maps, processes them with shared latent modules, and exchanges information through Pairwise Latent Space Attention (PaLA). Coupled Assignment Projection (CAP) ties encoding and decoding through normalized assignments. Experiments on nine PDE and engineering benchmarks show strong performance.

Critical strengths
  1. PANO provides a coherent architecture for combining multiple aggregation patterns while sharing latent-processing parameters. PaLA gives a meaningful mechanism for transferring information between latent spaces.
  2. The evaluation covers diverse PDE and engineering tasks, with substantial improvements on several benchmarks, including AirfRANS, Unilateral Stamping and AirCraft.
Critical weaknesses
  1. The main contribution is not sufficiently isolated from model capacity. The comparisons change HH, MM, feature width, depth, and sometimes the number of interaction layers. The rank argument also compares MM with HMHM, so it does not establish an advantage over a single latent space with the same total number of tokens.
  2. CAP guarantees normalized latent-token assignment mass, but not pointwise decoding stability or avoidance of token collapse. The information-loss analysis freezes assignment and attention weights, so it does not directly characterize the complete input-dependent network. More controlled evidence is also needed because all experiments use a single random seed.
Questions for authors
  1. How does the rank analysis establish an advantage over a single space with HMHM tokens, and how does the frozen-assignment analysis apply to the nonlinear encoder? Please compare more different HH at fixed total tokens, width, and depth, reporting parameters and FLOPs.
  2. Why use shared matching projections and equal weighting across source spaces? Please compare PaLA with conventional cross-attention and additional within-space self-attention under comparable budgets.
  3. Can the authors provide multiple-seed results for key comparisons, and runtime/peak-memory measurements under matched execution settings?
8 comments
AuthorsOpenReview ↗

Response to Weakness 1

We sincerely thank the reviewer for recognizing the PANO architecture and performance improvements. We address the weaknesses and questions in detail below.

The main contribution is not sufficiently isolated from model capacity.

We acknowledge that we did not provide sufficiently controlled ablation experiments to isolate the benefits of parallel latent spaces from changes in model capacity. To address this issue, we additionally conduct extensive ablations on HH and MM based on the depth settings of PANO-P on Unilateral Stamping.

HMHMHHMMCCParams. (M)GFLOPsGeometry (Metal) ↓\downarrowGeometry (Rubber) ↓\downarrowStress (Metal) ↓\downarrowStress (Rubber) ↓\downarrowPEEQ (Metal) ↓\downarrowPEEQ (Rubber) ↓\downarrow
5121512961.2657.290.00660.00570.07690.07100.22560.1876
5122256961.5464.600.00650.00560.07720.07090.21920.1814
5124128961.6979.270.00640.00550.07610.07100.22240.1824
204812048961.56153.170.00620.00540.07650.07250.21320.1805
2048120481925.42363.420.00640.00540.07500.06910.21570.1770
204821024961.84158.350.00600.00540.07600.07000.21150.1778
20484512961.99171.780.00520.00440.07430.06670.18270.1515

The results show that parallel latent spaces improve performance at both total token settings, HM=512HM=512 and HM=2048HM=2048. Introducing parallel latent spaces can substantially improve model performance. Compared with SOTA methods in Table 5 of the main text, PANO with H=4,M=128H=4, M=128 achieves highly competitive results with 51.0% (or less) of their parameters and 22.3% (or less) of their GFLOPs. On Unilateral Stamping, this variant outperforms Unisoma across all six metrics, reducing geometry errors by 9.9% and 12.7%, stress errors by 5.7% and 8.2%, and PEEQ errors by 14.2% and 13.7% for metal and rubber, respectively. Notably, even compared with the single-space variant with a larger feature dimension (H=1H=1, M=2048M=2048, and C=192C=192), PANO with H=4H=4, M=512M=512, and C=96C=96 achieves better performance with substantially fewer parameters and GFLOPs.

AuthorsOpenReview ↗

Response to Weakness 2

CAP ... or avoidance of token collapse

We would like to respectfully clarify that we do not claim that CAP prevents token collapse. CAP is mainly designed to construct a more stable latent space by normalizing the total encoding weight received by each latent token and fixing the total decoding weight.

To compare the decoding behavior of CAP and standard softmax-based projection in detail, we further quantitatively analyze decoding weights of tokens on Airfoil.

MethodBlock IDScaleTop10 token weightToken weight CV
Softmax195.88x62.71%6.396
Softmax226.35x20.84%1.905
CAP (ours)11.00x1.95%3.70e-7
CAP (ours)21.00x1.95%2.15e-7
  • Scale: The largest total decoding weight of a token divided by the balanced value N/MN/M.
  • Top10 token weight: The fraction of total decoding weight contributed by the 10 tokens with the largest weights.
  • Token weight CV: The coefficient of variation across token decoding weights.

Standard softmax-based projection constrains the decoding mass received by each point, but not the total mass contributed by each latent token. Consequently, almost all points may primarily read from the same token, leading to unstable pointwise decoding. As shown in the quantitative analysis above, softmax-based projection leads to highly uneven decoding weights and heavy reliance on a small subset of tokens for decoding. CAP substantially alleviates these issues.

We further quantitatively analyze the decoding weights of points on Airfoil.

MethodBlock IDScaleTop10 point weightPoint weight CV
Softmax11.00x0.09%5.62e-8
Softmax21.00x0.09%4.55e-8
CAP (ours)114.32x0.98%1.368
CAP (ours)23.70x0.29%0.283
  • Scale: The largest total decoding weight received by a point divided by the balanced value 1.
  • Top10 point weight: The fraction of total decoding weight received by the 10 points with the largest weights.
  • Point weight CV: The coefficient of variation across point decoding weights.

The analyses of token and point decoding weights show that standard softmax-based projection leads to severe imbalance in token usage. CAP first strictly guarantees balanced token usage. Moreover, although CAP imposes no normalization on point decoding weights, it still avoids comparably extreme allocation patterns and maintains relatively balanced weights across points. The benefits of CAP for latent-space learning are also supported by ablation studies.

The information-loss analysis ... the complete input-dependent network

We thank the reviewer for raising this insightful issue, which allows us to analyze information loss during encoding more clearly. We examine the encoding by directly decomposing the actual input features. We also further quantify the components in the encoding null spaces. This additional analysis directly evaluates input-dependent encoding and therefore provides a more accurate characterization of information loss during encoding than the fixed-assignment perturbation analysis.

For each latent space hh, let AhA^h be the assignment generated from the actual encoder input FF, and let Ph=Ah(Ah)†P^h=A^h(A^h)^\dagger be the orthogonal projector onto its column space, where †\dagger denotes the Moore–Penrose pseudoinverse. Define Fkeeph=PhFF_{\mathrm{keep}}^h=P^hF and Fignoreh=(I−Ph)FF_{\mathrm{ignore}}^h=(I-P^h)F. Then, we get:

F=Fkeeph+Fignoreh,F=F_{\mathrm{keep}}^h+F_{\mathrm{ignore}}^h,

(Ah)⊤Fignoreh=0,(A^h)^\top F_{\mathrm{ignore}}^h=0,

(Ah)⊤F=(Ah)⊤Fkeeph.(A^h)^\top F=(A^h)^\top F_{\mathrm{keep}}^h.

Thus, although the assignment is constructed from FF, the component FignorehF_{\mathrm{ignore}}^h is invisible to aggregation under that assignment in the same forward pass.

We further compute the actual FignoreF_{\mathrm{ignore}} and FkeepF_{\mathrm{keep}} for PANO-P on Airfoil, and report the ignored fraction ∥Fignore∥F2/∥F∥F2\|F_{\mathrm{ignore}}\|_F^2/\|F\|_F^2. “Jointly” denotes the concatenated assignment, which is used to measure the component omitted by all four spaces.

BlockSpace 1Space 2Space 3Space 4Jointly
Block 17.88%0.09%0.55%10.22%0.000014%
Block 20.11%0.66%0.18%0.69%0.000083%

These results show that:

  1. Each latent space has a distinct, nonzero FignoreF_{\mathrm{ignore}}.
  2. The jointly omitted information is substantially smaller than that omitted by any individual latent space.

These quantitative results directly verify the conclusion of Appendix E.3.

AuthorsOpenReview ↗

Response to Weakness 2 (Part 2)

Multiple seed experiments

We would respectfully emphasize that PANO reduces prediction errors relative to the best competing methods by 87.0%, 63.6%, and 43.5% on AirfRANS surface fields, Plasticity, and NS2D, respectively. These substantial improvements across different benchmarks support the effectiveness of PANO, rather than reflecting gains caused by random fluctuations.

To further assess sensitivity to random seeds, we compare Transolver, LinearNO, PANO-P, and PANO-D on Airfoil and Elasticity, where PANO shows relatively smaller improvements.

MethodAirfoil ↓\downarrowElasticity ↓\downarrow
Transolver0.0053 ± 0.00010.0064 ± 0.0002
LinearNO0.0053 ± 0.00030.0054 ± 0.0004
PANO-P (ours)0.0046 ± 0.00030.0042 ± 0.0002
PANO-D (ours)0.0045 ± 0.00070.0029 ± 0.0001

PANO still consistently outperforms previous SOTA models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.