acceptodds
Under review as a conference paper at ICLR 2027

PhyLoopNO: Looped Neural Operator with Physics-Gated Dense Expert Layers

Abstract

Low-rank Transformer neural operators solve partial differential equations (PDEs) by modeling global interactions through compact latent representations. Slicing introduces an explicit latent bottleneck; global interactions are then realized through latent-token attention followed by deslicing, or through deslicing decoupled from slicing. We observe that slicing and deslicing are the main sources of differentiation across layers. Assigning layer-specific parameters to most other components offers relatively limited improvements in model performance, resulting in low parameter efficiency. Inspired by looped Transformers in large language models, we introduce PhyLoopNO, a parameter-efficient neural operator built on LinearNO. It reuses operator and feed-forward transformations across loop iterations while retaining independent slicing and deslicing, preserving spatial interactions that vary with depth. However, parameter sharing alone reduces parameter capacity and can limit predictive accuracy. To address this trade-off, we replace the pointwise feed-forward network with physics-gated dense experts. Inspired by domain decomposition, a learned gate softly groups points according to their physical features and combines the responses of all parallel experts at each point. Experiments show the best results on five of six standard PDE benchmarks in our comparison and competitive performance on industrial aerodynamic tasks. Increasing depth from 8 to 40 layers further reduces its errors on Airfoil, Darcy, and Elasticity overall. Selective parameter reuse lowers parameter growth on Airfoil and Darcy, enabling parameter-efficient depth scaling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, partial differential equations, scientific machine learning, looped Transformers, parameter sharing, physics-gated dense experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fetPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PhysLoopNO, which extends the LinearNO architecture with more parameter efficient looping mechanism, additional physics-gated tied dense experts, and selective parameter sharing. The parameter-reuse concept is inspired by looped architectures, with particular influence from the tied experts introduced in "Tying The Loop".

It claims the method achieves the best among the reported results in terms of L2 error on five out of six PDE benchmarks: "Airfoil", "Plasticity", "Darcy", "Navier-Stokes" and "Elasticity", where the compared baseline method LinearNO is reproduced according to the official implementation and experimental protocols on all benchmarks. On the benchmark "Pipe", the Transolver, Transolver++ and the reproduced LinearNO perform better.

On two further datasets, "AirfRANS" and "ShapeNet-Car", the reported results demonstrate competitive performance, again comparing against the reproduced LinearNO. PhysLoopNO achieves the best reported results in terms of volume and surface field predictions, and the drag and lift predictions are also competitive. The Spearman rank correlation for the drag for "ShapeNet-Car" is competitive, but on "AirfRANS" the Spearman rank correlation for the lift is worse than seven other methods including the reproduced LinearNO.

One main advantage that the authors stress is that compared to LinearNO, scaling the depth of the network from 8 to 40 layers adds 46.26% fewer parameters on "Darcy" and 21.76% fewer parameters on "Airfoil", while outperforming LinearNO at maximum depth of 40 on "Airfoil" and achieving a tie with LinearNO on "Darcy". On "Elasticity", the PhysLoopNO model can outperform LinearNO but the four-expert configuration adds 32.04% more parameters when scaling from 8 to 40 layers.

Regarding the choice of number of experts on the tested datasets, it finds that depending on the dataset, the best tested number of experts was between one and four experts, and using additional experts can degrade the performance.

Critical strengths

The paper ...

  • introduces looping to the LinearNO architecture with selective parameter reuse, achieving more efficient scaling in depth while retaining improved or competitive accuracy compared to prior methods.

  • reports results on a variety of benchmarks showing that: (1) their model outperforms a wide range of other methods and a reproduced version of LinearNO on five out of six PDE benchmarks, and (2) achieves competitive results on two additional CFD datasets.

  • identifies two different mechanisms, namely spatial aggregation / mixing of spatial features in the cores (slicing and deslicing) and pointwise / local, non-linear transformations (experts) that can be weighted for each point and loop, while expert banks are shared between the loops.

Critical weaknesses
  • It is not clear if the results from the experiments are robust, i.e. the number of tested random seeds is enough. This could especially make a difference for given the small relative differences in L2 error on experiments like "Airfoil" (0.0048 vs 0.0046), "Plasticity" (0.0013 vs 0.0012), "Darcy" (0.0050 vs 0.0048), AirfRANS surface field (0.0076 vs 0.0078) and ShapeNet-Car volume and surface field (0.0188 vs 0.0195 and 0.0744 vs 0.0752).

  • While the paper focuses on parameter-efficiency, it does not state the model sizes for comparison, just relative parameter counts as depth increases. It just states it on one separate three-expert Airfoil configuration the PhysLoopNO model uses 17.46% fewer trainable parameters than LinearNO at matched depth.

  • Since the experts are shared between different visits, parameter count does not directly relate to number of floating point operations (FLOPs). The number of FLOPs during training and inference with PhysLoopNO should be compared against the FLOPs of LinearNO.

Questions for authors
  • For clarity, could you expand on what physics-gated experts are and how they differ from gated experts?

  • How many independent training runs were performed per benchmark? Would it be possible to report the means and standard deviations over at least five random seeds in the table for the main results in tables 1 and 2?

  • Could you provide the total number of trainable parameters for the LinearNO and PhysLoopNO models used in the experiments from tables 1 and 2? If possible please also report the number of parameters for the other baseline models.

  • Could you provide the number of FLOPs for the LinearNO and PhysLoopNO models during inference and training for the experiments from tables 1 and 2?

  • For the "AirfRANS" dataset, the reproduction of LinearNO is 64 %64\, \% worse than the original LinearNO model in terms of L2 errors of volume pressure field, as reported in the original paper [1]. This changes the claims about competitiveness with LinearNO. Could you reconcile the differences to the original LinearNO paper?

[1] Hu, Wenjie, et al. "Transolver is a linear transformer: Revisiting physics-attention through the lens of linear attention." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 1. 2026.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fetP on expert gating, robustness, and computational cost

We sincerely thank you for recognizing the selective-sharing and expert mechanisms and for asking about their costs and the variability of the results.

Regarding physics-gated experts

In PhyLoopNO, “physics-gated” describes how the evolving field state weights experts shared across visits to one core. Let zz be a normalized CC-channel point feature carrying spatial context from slicing/deslicing. Visit rr computes πr(z)=softmax(Wrz+br)\pi_r(z)=\mathrm{softmax}(W_r z+b_r) and returns the pointwise mixture

Gr(z)=∑e=1Eπr,e(z)fe(z).G_r(z)=\sum_{e=1}^E \pi_{r,e}(z)f_e(z).

Each fef_e is a two-layer GELU FFN of hidden width FF and processes every point. The PDE loss trains each router's E(C+1)E(C+1) parameters. Similar hidden states can receive similar mixtures within a visit. Even for the same zz, a later visit can mix the shared experts differently. The residual adds Gr(z)/RG_r(z)/R at each of RR visits. We omit a load-balancing loss, allowing unequal expert contributions across regions. We expanded Section 3.3 to explain this design.

Regarding result robustness

We now report five-seed means and SDs in Appendix C.6 (Tables 9–10). Table A gives relative L2L_2 error on the six standard tasks. Industrial coefficient error and Spearman ρ\rho refer to lift on AirfRANS and drag on ShapeNet-Car.

Table A. Five-seed test results (mean ± SD).

Task or metricLinearNOPhyLoopNO
Airfoil0.0048±0.00090.0046±0.0006
Pipe0.0028±0.00080.0036±0.0006
Plasticity0.0013±0.00060.0012±0.0003
Darcy0.0050±0.00050.0048±0.0004
Navier–Stokes0.0752±0.00110.0711±0.0012
Elasticity0.00540±0.000100.00470±0.00009
AirfRANS volume0.0018±0.00080.0015±0.0004
AirfRANS surface0.0078±0.00100.0076±0.0009
AirfRANS lift0.0481±0.00150.0488±0.0012
AirfRANS ρ\rho0.99180±0.010140.9911±0.0090
ShapeNet-Car volume0.0195±0.00110.0188±0.0010
ShapeNet-Car surface0.0752±0.00180.0744±0.0021
ShapeNet-Car drag0.0108±0.00060.0108±0.0008
ShapeNet-Car ρ\rho0.99240±0.012160.99100±0.01123

Airfoil, Plasticity and Darcy mean gaps are small relative to seed variability. Gains are larger on Navier–Stokes and Elasticity. Pipe favors LinearNO. Section 4.1 clarifies that all four mean field errors fall. Lift error rises slightly, drag error ties at the reported precision, and both rank correlations fall.

Regarding absolute parameter counts and FLOPs

Each shared expert is stored once and evaluated at every visit. Table B therefore pairs parameter counts with FLOPs. Section 4.2 and Appendix C.6–C.8 (Tables 11–14) now give counts by task, expert count (E=1,2,4E=1,2,4) and depth (8–40 in steps of 8 on Airfoil, Darcy and Elasticity).

Table B. PP = total parameters at 8/40 layers. GG = eight-layer GFLOPs, forward / forward + backward. Industrial 40-layer entries are architecture counts.

TaskTransolver PPLinearNO PPPhyLoopNO PPLinearNO GGPhyLoopNO GG
Airfoil2,810,817/13,917,3771,765,889/8,692,7371,294,753/5,595,55345.442/136.31451.443/154.319
Pipe2,810,817/13,917,3771,765,889/8,692,7371,210,841/5,225,72167.092/201.26071.557/214.653
Plasticity2,844,484/13,951,0441,799,428/8,725,7641,328,292/5,628,58012.833/38.39014.500/43.392
Darcy2,811,073/13,917,6331,766,145/8,692,9931,125,513/4,848,16929.133/87.38829.148/87.432
Navier–Stokes12,301,377/60,750,1453,377,921/16,132,865905,121/3,782,17729.991/89.51312.479/37.207
Elasticity713,665/3,431,617585,217/2,789,377556,833/2,348,7051.626/4.8762.143/6.429
AirfRANS5,996,868/29,304,3883,358,788/16,113,988895,564/3,772,756233.079/696.91096.879/289.473
ShapeNet-Car3,860,804/18,755,1403,852,420/18,713,220961,828/4,120,100266.074/797.990104.824/314.358

From 8 to 40 layers, PhyLoopNO adds 46.26% fewer parameters than LinearNO on Darcy, 37.91% fewer on Airfoil and 18.70% fewer on Elasticity. Darcy uses one expert, Pipe three and the remaining tasks four.

Counts cover matrix and convolution operations across all eight executed layers of each model at batch size one (two FLOPs per multiply-accumulate), excluding activations, normalization, softmax, expert weighting and optimizer updates. We use one predicted step for Navier–Stokes, one time query for Plasticity and 32,000/32,186 points for AirfRANS/ShapeNet-Car.

Regarding the LinearNO reproduction gap

LinearNO reports three runs and gives no industrial SD. Its published AirfRANS volume error (0.0011) is lower than both means in Table A. Section 4.1 now specifies that PhyLoopNO improves on LinearNO in our common environment. The cause of the reproduction gap remains unresolved.

References

Hu et al. (2026). Transolver Is a Linear Transformer. AAAI.

Wu et al. (2024). Transolver. ICML.

Thank you for the clarifications. The questions about the physics-gated experts and about the total parameter count are answered satisfactorily. There are still three remaining follow-up questions to the other points.

  • Regarding robustness of the results: the new tables 9 and 10 in the appendix contain the same values as the tables in the original version of the paper, but given the relatively high standard deviations, it seems unlikely that all the mean values would correspond to a single seed result.
  • Your answer regarding FLOPs does not compare LinearNO and PhyLoopNO in a fair way. As you note in the paper, in the eight-layer looped models some cores are visited multiple times which requires computation. In your answer you only count one visit per core. Could you give a comparison of the FLOPs for the full forward passes, and full forward + backward passes?
  • Is it possible to resolve the reproduction gap?
AuthorsOpenReview ↗

Response to Reviewer fetP

Thank you for considering our additional experiments and for your positive feedback. We are glad to continue the discussion.

Regarding the computational comparison

We agree that a fair comparison must include every visit to each shared core. We have updated the table of parameter counts and computational costs accordingly. For both models, the table gives the full forward and full forward + backward costs across all eight executed layers. In PhyLoopNO, each of the three shared cores is visited twice.

Table 1. Trainable parameters and GFLOPs per example at depth eight.

BenchmarkLinearNO parametersPhyLoopNO parametersLinearNO forwardPhyLoopNO forwardLinearNO forward + backwardPhyLoopNO forward + backward
Airfoil1,765,8891,294,75345.44251.443136.314154.319
Pipe1,765,8891,210,84167.09271.557201.260214.653
Plasticity1,799,4281,328,29212.83314.50038.39043.392
Darcy1,766,1451,125,51329.13329.14887.38887.432
Navier–Stokes3,377,921905,12129.99112.47989.51337.207
Elasticity585,217556,8331.6262.1434.8766.429
AirfRANS3,358,788895,564233.07996.879696.910289.473
ShapeNet-Car3,852,420961,828266.074104.824797.990314.358

PhyLoopNO uses fewer parameters than LinearNO on all eight benchmarks. Evaluating the dense experts increases computation on some tasks. On Airfoil and Plasticity, for example, we consider the approximately 13% increase in forward cost a reasonable trade-off for the lower mean prediction error. Overall, PhyLoopNO lowers mean error on five of the six standard benchmarks. On Darcy, Navier–Stokes, AirfRANS, and ShapeNet-Car, it achieves lower mean field errors at comparable or lower computational cost.

Regarding reproducibility and standard deviations

Many recent neural operators reporting state-of-the-art results are based on Mamba [1, 2]. We have found some of their results difficult to reproduce consistently. For example, AMO’s official code repository [1] was empty when we checked.

We believe that adjacency in a Mamba scan does not necessarily imply proximity in physical space, even when the sequence is scanned in both directions as in LaMO [2]. We think this may be an important reason why the performance of Mamba-based neural operators can vary.

Both our model and the baseline show relatively large standard deviations across random seeds, particularly on Airfoil and Pipe. However, PhyLoopNO has a smaller standard deviation than LinearNO on five of the six standard benchmarks, including Airfoil (0.0006 vs. 0.0009), Pipe (0.0006 vs. 0.0008), and Plasticity (0.0003 vs. 0.0006). These results support the robustness of PhyLoopNO across training seeds.

The original submission already reported five-seed means for PhyLoopNO and our LinearNO reproduction. Tables 9 and 10 add standard deviations from those same runs, which explains why the means match exactly. The small mean improvement on Airfoil should still be read in light of that variability.

References

[1] Zeyuan Song and Zheyu Jiang. Adaptive Mamba Neural Operators. ICLR, 2026.

[2] Karn Tiwari, Niladri Dutta, N M Anoop Krishnan, and Prathosh Ap. Latent Mamba Operator for Partial Differential Equations. ICML, 2025.

Thank you for these clarifications and the additional experiments. I believe these should be incorporated into the final manuscript. I will maintain my positive rating.

Reviewer_5uTUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a PDE solver called PhyLoopNO on top of LinearNO (Hu et al., 2026). LinearNO moves point features through a small set of latent tokens. Slicing pools points into those tokens. Deslicing sends the tokens back to the points. There is no attention between the tokens.

The authors say that slicing and deslicing are the parts that should change with depth. They reuse the other parts. A loop runs a shared core more than once. Each visit still has its own slicing, deslicing, norm, and gate. They also replace the pointwise feed-forward block with several experts. A soft gate mixes all experts at every point. They call this physics gating. They link the idea to domain decomposition (Toselli and Widlund, 2005) and to looped language models (Dehghani et al., 2019; Bae et al., 2025; Jaggi, 2026).

On six standard PDE sets, they report the best error on five. Pipe is the miss. They also test AirfRANS and ShapeNet-Car. A depth sweep from 8 to 40 layers lowers error on Airfoil, Darcy, and Elasticity.

Critical strengths

The design question is clear. The paper asks which parts of a LinearNO layer need new weights at each depth, and which parts can be reused. Figure 1 and Table 4 make the split easy to check.

The paper is honest about a result that hurts the simple loop story. Table 8 and Figure 2 show that full sharing raises error on all six tasks. On Pipe, the error goes from 0.0028 to 0.0046. That is the 64.3% rise they report. Selective sharing is better than full sharing on all six tasks. This is a useful negative result.

The expert test is the right kind of test. They swap the expert mix for one feed-forward block of width EF, with the other settings fixed. Error rises on every set except Darcy. Darcy uses one expert, so the gate is always one and the two models match. Elasticity moves from 0.0047 to 0.0054.

Proposition 1 is stated with care. It says a fixed mix of experts matches one wide feed-forward block. It does not claim a lower error by itself. Theorem 1 also says the rank bound does not prove that untied slicing is better. The proof limits are written down. That is good.

The depth table is complete. Table 7 lists every point in Figure 4. The parameter math in Appendix C.3 says the savings are on the added weights, and that Elasticity with four experts adds more weights, not fewer.

Critical weaknesses

1. The loop alone makes the solver worse. The abstract leads with the loop.

Table 8 uses the same 8-layer loop and one feed-forward block. Selective sharing beats LinearNO only on Darcy (0.0048 versus 0.0050). It is worse on the other five. Navier–Stokes goes from 0.0752 to 0.0991. Elasticity goes from 0.0054 to 0.0070. Pipe goes from 0.0028 to 0.0041.

The main model in Table 1 adds experts and then looks a bit better than the authors’ LinearNO. So the gain sits in the expert block. The loop is not shown to be the source of the Table 1 win. Jaggi (2026) already studies shared experts with untied interfaces in language models. Bae et al. (2025) already add small untied updates to a shared Transformer. The PDE paper needs a test against those two ideas, not only against a fully tied core.

2. The five wins are very small, and they sit on a weaker LinearNO rerun.

Against the authors’ own LinearNO, the relative drops are 4.2% (Airfoil), 7.7% (Plasticity), 4.0% (Darcy), 5.5% (Navier–Stokes), and 13.0% (Elasticity). The absolute gaps are 0.0002, 0.0001, 0.0002, 0.0041, and 0.0007. There is no seed spread. A gap of 0.0002 can be noise.

The same paragraph says the rerun is worse than the published LinearNO numbers (Hu et al., 2026): Pipe 0.0028 versus 0.0024, and Elasticity 0.0054 versus 0.0050. Table 1 then copies older numbers from FNO through Transolver++ (Li et al., 2021; Wu et al., 2024; Luo et al., 2025). Only LinearNO is rerun. The “best on five” claim mixes one new run with many old tables.

On Pipe, PhyLoopNO is 0.0036. That is worse than the authors’ LinearNO (0.0028), Transolver (0.0030), Transolver++ (0.0027), and the published LinearNO (0.0024).

3. Deeper runs do not show a special loop benefit.

Table 7 goes from 8 to 40 layers. LinearNO improves by about as much as PhyLoopNO:

  • Airfoil: LinearNO 0.0048 to 0.0034. PhyLoopNO 0.0046 to 0.0032.
  • Darcy: both end at 0.0034.
  • Elasticity: LinearNO 0.0054 to 0.0043. PhyLoopNO 0.0047 to 0.0041.

At 40 layers the gap is 0.0002, 0, and 0.0002. The abstract says depth “further reduces its errors.” That is true for this model. It is also true for plain LinearNO.

The schedule also stays at two repeats. The 8-layer model is 1 + 3 × 2 + 1. The 40-layer model is 2 prefix layers, 18 stored cores, two visits, and 2 suffix layers. Executed depth grows mostly by adding new stored cores. It does not grow by looping one small core many times. Universal Transformers (Dehghani et al., 2019) loop the same block for many steps. This paper does not test that choice. There is no sweep of repeat count against stored-core count at the same depth and the same parameter budget.

4. “Parameter-efficient” is only half of the cost.

Sharing cuts stored weights. It does not cut the work per visit. Each visit still runs. Appendix C.2 says each expert keeps the original feed-forward width. Four experts therefore do about four times the feed-forward work of LinearNO. Dense mixing, which the paper chooses after citing Shazeer et al. (2017), turns every expert on at every point.

The main text never lists parameter count, FLOPs, memory, or runtime for the Table 1 models. The 17.46% weight cut is a separate three-expert Airfoil setup, not the four-expert model in Table 1. The depth savings are about added weights only. On Elasticity, Appendix C.3 says the added weights go up by 7.08% versus Transolver and 32.04% versus LinearNO. A matched-parameter plot is missing, so a reader cannot see if the same weights in a plain deeper LinearNO would match the error.

5. The expert gate is a soft mix. It is not domain decomposition, and it is close to prior soft experts.

Equation (6) is a softmax-weighted sum of all experts. Proposition 1 says that, at one visit, this family includes one feed-forward block of width EF. The extra gate term in Equation (7) has rank at most min(C, E − 1). For the usual setup, C is 128 or 256 and E is 4, so that rank is at most 3. The theory does not show that this low-rank gate must beat a wide feed-forward block. The Elasticity gap is real but small.

Toselli and Widlund (2005) study subdomain solvers, interface conditions, and a coarse fix. This model has none of those. The gate reads hidden features after a norm. It does not read a named physical input such as the Darcy coefficient, wall distance, or Mach number. Figure 9 is a set of pictures. There is no score for whether the gate matches a shock, a wake, a boundary, or a material jump.

The dense mix is also close to Soft MoE (Puigcerver et al., 2024), which was written to avoid the dead-expert problem in Shazeer et al. (2017). Switch Transformers (Fedus et al., 2022) and expert-choice routing (Zhou et al., 2022) are the usual sparse fixes. None of these are cited. On the PDE side, GNOT already uses learned gates inside a neural-operator Transformer (Hao et al., 2023). GNOT is absent from the text and from Table 1.

6. Theorem 1 restates the LinearNO bottleneck.

Equation (1) builds M summaries per head. Theorem 1 says the latent part of the Jacobian has rank at most MC. That is the same narrow path already built into LinearNO (Hu et al., 2026), and it is the standard rank limit of linear attention (Katharopoulos et al., 2020). The bound holds for tied and untied layers. The repeat bound, rank at most TMC, counts visits. It does not favor this sharing pattern. The theorem should not be read as support for the new design.

7. The expert count looks chosen on the test score.

The authors search E in {1, 2, 3, 4} and also try eight experts. They pick a different E for each dataset. Table 6 has a train split and a test split for the six standard sets. It has no validation split there. AirfRANS does have 80 validation cases. If E was picked by test error, the Table 1 numbers are optimistic. Eight experts raise Elasticity to 0.0071, which is worse than LinearNO at 0.0054. That search is also described with test error.

8. The industrial gains are small, and the broad claims go past the tests.

On AirfRANS, volume error moves from 0.0018 to 0.0015 and surface error from 0.0078 to 0.0076. Lift error gets worse, from 0.0481 to 0.0488. Rank correlation gets worse, from 0.9918 to 0.9911. On ShapeNet-Car, drag error stays 0.0108 and the drag rank correlation falls from 0.9924 to 0.9910. LinearNO already did the large jump past Transolver. PhyLoopNO adds a thin field-error gain.

The closing section talks about PDE foundation models and one model for many physical systems. The experiments train one model per PDE. There is no cross-PDE test, no new mesh size at test time, and no comparison to DPOT (Hao et al., 2024), Poseidon (Herde et al., 2024), or MPP (McCabe et al., 2023). CNO is also a strong modern operator baseline and is not in Table 1 (Raonic et al., 2023). IPOT, UPT, and LNO are discussed in Section 2.2 and then left out of Table 1 (Lee and Oh, 2024; Alkin et al., 2024; Wang and Wang, 2024).

Weight reuse across steps is also older in PDE models. Message-passing PDE solvers share one update across time (Brandstetter et al., 2022). The paper does not compare a shared time stepper with this depth loop.

Questions for authors
  1. Can you report mean and spread over at least three seeds for Table 1, Table 7, and Table 8? The Airfoil gap is 0.0002. A reader needs to know if that gap stays.

  2. Which split picked E? Please freeze E on a validation split, then report the test error once. Please also give train error and test error for E = 1, 2, 4, and 8 so the overfit on Elasticity is visible.

  3. Please add one table at depth 8 with four rows: LinearNO, loop with one feed-forward block, loop with a wide feed-forward block of width EF, and the full expert model. Please list parameters, FLOPs, and test error. Table 8 already shows that the loop with one block is worse on five tasks. This table will show whether the experts, the loop, or the extra width create the Table 1 gap.

  4. Please compare visit-specific slicing with the LoRA-style untied update from Bae et al. (2025), at the same parameter budget. Please also state how the expert bank differs from the tied experts and untied routers in Jaggi (2026), and from Soft MoE (Puigcerver et al., 2024).

  5. Can you sweep the repeat count? At a fixed executed depth and a fixed parameter count, please compare R = 2 with a larger R and fewer stored cores, in the sense of Universal Transformers (Dehghani et al., 2019). The 40-layer model still uses only two visits.

  6. What is α = 1/R doing? Please rerun the depth sweep with α = 1. Residual downscaling can help very deep nets on its own (Wang et al., 2022). The current sweep mixes that scale with the loop.

  7. Please rerun Transolver and Transolver++ in the same code as PhyLoopNO (Wu et al., 2024; Luo et al., 2025). Your LinearNO rerun already misses the published Pipe and Elasticity numbers (Hu et al., 2026). Please also add GNOT (Hao et al., 2023) and CNO (Raonic et al., 2023) on the same six splits, and state the Navier–Stokes viscosity so the copied FNO number is the same task (Li et al., 2021).

  8. Can you show that the gate follows the physics? For each task, feed the gate with coordinates only, with hidden features, and with the physical input (Darcy coefficient, geometry, or inflow). Please report gate entropy and the match to a known region, such as the wake or the hole boundary in Figure 3. Toselli and Widlund (2005) do not, by themselves, explain a softmax on hidden features.

  9. Please put parameter count and FLOPs next to every main model. For the four-expert models, how much of the feed-forward work is the extra experts? On Elasticity, the added-weight formula already says PhyLoopNO grows faster than LinearNO. A plot of error against parameter count would answer the efficiency claim.

  10. The last section names foundation models and multi-system sharing. Can you either remove that claim or add one multi-PDE run against DPOT or Poseidon (Hao et al., 2024; Herde et al., 2024)? A single-PDE table does not support that sentence.

References

  • Alkin et al. (2024). Universal Physics Transformers. NeurIPS.
  • Bae et al. (2025). Relaxed Recursive Transformers. ICLR.
  • Brandstetter et al. (2022). Message Passing Neural PDE Solvers. ICLR.
  • Dehghani et al. (2019). Universal Transformers. ICLR.
  • Fedus, Zoph, and Shazeer (2022). Switch Transformers. JMLR.
  • Hao et al. (2023). GNOT: A General Neural Operator Transformer. ICML.
  • Hao et al. (2024). DPOT. ICML.
  • Herde et al. (2024). Poseidon. NeurIPS.
  • Hu et al. (2026). LinearNO. AAAI.
  • Jaggi (2026). Tying the Loop. arXiv:2606.16825.
  • Katharopoulos et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML.
  • Lee and Oh (2024). IPOT. AAAI.
  • Li et al. (2021). Fourier Neural Operator. ICLR.
  • Luo et al. (2025). Transolver++. ICML.
  • McCabe et al. (2023). Multiple Physics Pretraining. arXiv:2310.02994.
  • Puigcerver et al. (2024). From Sparse to Soft Mixtures of Experts. ICLR.
  • Raonic et al. (2023). Convolutional Neural Operators. NeurIPS.
  • Shazeer et al. (2017). Sparsely-Gated Mixture-of-Experts Layer. ICLR.
  • Toselli and Widlund (2005). Domain Decomposition Methods. Springer.
  • Wang and Wang (2024). Latent Neural Operator. NeurIPS.
  • Wang et al. (2022). DeepNet: Scaling Transformers to 1,000 Layers.
  • Wu et al. (2024). Transolver. ICML.
  • Zhou et al. (2022). Mixture-of-Experts with Expert Choice Routing. NeurIPS.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5uTU (1/3) — Sharing controls and five-seed results

We sincerely thank you for recognizing our selective-sharing and expert-mixing mechanisms and for your constructive suggestions.

Regarding looping, selective sharing, and expert capacity

To address the accuracy loss under looping, Section 3.2 explains our response to reduced independent parameter capacity: selective sharing preserves adaptive spatial interfaces, while dense experts replenish pointwise capacity. Selective sharing improves on full sharing on all six tasks, but exceeds unshared LinearNO only on Darcy. Navier–Stokes remains at 0.0969 versus 0.0752.

Table A. Single-FFN sharing controls, five-seed mean ± SD.

BenchmarkFully shared single FFN (mean ± SD)Selective single FFN (mean ± SD)
Airfoil0.0060 ± 0.00080.0052 ± 0.0007
Pipe0.0046 ± 0.00050.0043 ± 0.0003
Plasticity0.0021 ± 0.00070.0016 ± 0.0004
Darcy0.0055 ± 0.00060.0048 ± 0.0004
Navier-Stokes0.11700 ± 0.000130.09690 ± 0.00009
Elasticity0.00820 ± 0.000120.00780 ± 0.00008

Paper Table 3 isolates slicing/deslicing within the expert model. Tying these interfaces raises Elasticity error from 0.0047 to 0.0064 and Pipe error from 0.0036 to 0.0044. Theorem 1 characterizes the inherited bottleneck; the sharing ablations support our parameter allocation.

To separate sharing, added capacity, and adaptive mixing, we added the four-configuration comparison (Table B).

Table B. Eight-layer ablations. Each cell gives trainable parameters, estimated forward GFLOPs, and mean test error.

BenchmarkLinearNOLooped LinearNO + single FFN (F)Looped LinearNO + wide FFN (EF)Full PhyLoopNO
Airfoil1,765,889 / 45.442 / 0.00481,043,017 / 42.510 / 0.00521,289,737 / 51.374 / 0.00501,294,753 / 51.443 / 0.0046
Pipe1,765,889 / 67.092 / 0.00281,043,017 / 62.764 / 0.00431,207,497 / 71.489 / 0.00451,210,841 / 71.557 / 0.0036
Plasticity1,799,428 / 12.833 / 0.00131,076,556 / 12.018 / 0.00161,323,276 / 14.481 / 0.00141,328,292 / 14.500 / 0.0012
Darcy1,766,145 / 29.133 / 0.00501,125,513 / 29.148 / 0.00481,125,513 / 29.148 / 0.00481,125,513 / 29.148 / 0.0048
Navier-Stokes3,377,921 / 29.991 / 0.0752406,665 / 6.011 / 0.0969900,105 / 12.454 / 0.0741905,121 / 12.479 / 0.0711
Elasticity585,217 / 1.626 / 0.0054305,097 / 1.373 / 0.0078551,817 / 2.137 / 0.0054556,833 / 2.143 / 0.0047

We clarified how the width-EFEF FFN controls aggregate capacity in Section 4.2. The expert model uses less than 1% more parameters and gives lower mean error on all five multi-expert tasks. Darcy uses one expert and the models coincide. This comparison isolates input-dependent mixing at comparable aggregate width.

Regarding robustness and parameter efficiency

To address result robustness, we added five-seed means and standard deviations for our main results, depth study, and sharing ablations to Appendix C.4–C.6 (Tables 7–10). PhyLoopNO improves the reproduced LinearNO mean on five tasks. Airfoil and Darcy gaps are small relative to seed variability. The four-expert industrial models lower mean volume and surface errors, while lift error rises, drag error ties, and both rank correlations fall.

Table C. Five-seed results on standard and industrial benchmarks.

BenchmarkMetricLinearNO (mean ± SD)PhyLoopNO (mean ± SD)
AirfoilRelative L2 error0.0048 ± 0.00090.0046 ± 0.0006
PipeRelative L2 error0.0028 ± 0.00080.0036 ± 0.0006
PlasticityRelative L2 error0.0013 ± 0.00060.0012 ± 0.0003
DarcyRelative L2 error0.0050 ± 0.00050.0048 ± 0.0004
Navier-StokesRelative L2 error0.0752 ± 0.00110.0711 ± 0.0012
ElasticityRelative L2 error0.00540 ± 0.000100.00470 ± 0.00009
AirfRANSVolume error0.0018 ± 0.00080.0015 ± 0.0004
AirfRANSSurface error0.0078 ± 0.00100.0076 ± 0.0009
AirfRANSCL error0.0481 ± 0.00150.0488 ± 0.0012
AirfRANSrhoL0.99180 ± 0.010140.9911 ± 0.0090
ShapeNet-CarVolume error0.0195 ± 0.00110.0188 ± 0.0010
ShapeNet-CarSurface error0.0752 ± 0.00180.0744 ± 0.0021
ShapeNet-CarCD error0.0108 ± 0.00060.0108 ± 0.0008
ShapeNet-CarrhoD0.99240 ± 0.012160.99100 ± 0.01123

For the LinearNO reproduction gap, our Pipe and Elasticity means are 0.0028 and 0.0054 versus the published 0.0024 and 0.0050. We followed the published settings in the same environment as PhyLoopNO. Hu et al. report three runs without Table 1 spreads or individual results, so the source of the discrepancy remains unresolved.

AuthorsOpenReview ↗

Response to Reviewer 5uTU (2/3) — Parameter efficiency, depth, and repeat count

To address the requested error–parameter comparison, we have added Figure 10 to Appendix E of the revised manuscript using 45 existing depth-study configurations. We also added parameter counts by task, expert count, and depth to Appendix C.6–C.8 (Tables 11–14).

Table D. Eight-layer parameter counts, estimated forward GFLOPs, and estimated FP32 model-state memory.

BenchmarkLinearNO parametersPhyLoopNO parametersLinearNO forward GFLOPsPhyLoopNO forward GFLOPsLinearNO state memory (MiB)PhyLoopNO state memory (MiB)
Airfoil1,765,8891,294,75345.44251.44326.9519.76
Pipe1,765,8891,210,84167.09271.55726.9518.48
Plasticity1,799,4281,328,29212.83314.50027.4620.27
Darcy1,766,1451,125,51329.13329.14826.9517.17
Navier-Stokes3,377,921905,12129.99112.47951.5413.81
Elasticity585,217556,8331.6262.1438.938.50
AirfRANS3,358,788895,564233.07996.87951.2513.67
ShapeNet-Car3,852,420961,828266.074104.82458.7814.68

For four equal-width experts, the three additional experts contribute 75% of expert-FFN matrix work. GFLOPs count matrix multiplications and convolutions across all eight executed layers of each model at batch size one, with two FLOPs per multiply-accumulate. Table D estimates FP32 weights, gradients, and two Adam states at 16 bytes per parameter. Peak memory and runtime remain unprofiled. The depth comparison is not parameter-matched.

Regarding depth scaling, repeat count, and residual scaling

For depth scaling, Table E compares five executed depths and total parameters at 40 layers. Transolver has spreads only at 40 layers. At 40 layers, all three models benefit from depth. On Darcy, PhyLoopNO matches LinearNO's error with 44.23% fewer parameters and uses 65.17% fewer than Transolver. On Elasticity, PhyLoopNO uses 15.80% fewer parameters than LinearNO and 31.56% fewer than Transolver. We revised Section 4 to emphasize the task-dependent accuracy–parameter trade-off.

Table E. Depth-wise error and total parameters at depth 40.

BenchmarkModel816243240Parameters at 40
AirfoilPhyLoopNO0.0046 ± 0.00060.0043 ± 0.00060.0039 ± 0.00050.0036 ± 0.00070.0032 ± 0.00055,595,553
AirfoilLinearNO0.0048 ± 0.00090.0042 ± 0.00080.0039 ± 0.00090.0037 ± 0.00060.0034 ± 0.00048,692,737
AirfoilTransolver0.00530.00550.00460.00430.0036 ± 0.000513,917,377
DarcyPhyLoopNO0.0048 ± 0.00040.0044 ± 0.00070.0042 ± 0.00050.0039 ± 0.00070.0034 ± 0.00044,848,169
DarcyLinearNO0.0050 ± 0.00050.0046 ± 0.00060.0040 ± 0.00040.0038 ± 0.00040.0034 ± 0.00048,692,993
DarcyTransolver0.00550.00530.00510.00480.0047 ± 0.000613,917,633
ElasticityPhyLoopNO0.00470 ± 0.000090.00490 ± 0.000080.00440 ± 0.000120.00420 ± 0.000060.00410 ± 0.000082,348,705
ElasticityLinearNO0.00540 ± 0.000100.00480 ± 0.000100.00460 ± 0.000090.00450 ± 0.000140.00430 ± 0.000092,789,377
ElasticityTransolver0.00640.00560.00490.00470.00470 ± 0.000063,431,617

To examine repeat count, Appendix C.4 now reports our additional experiment with one core visited six times in place of three cores visited twice. Both schedules execute eight layers, with the same independent prefix and suffix, widths, and expert counts.

Table F. Repeat-count experiment and calculated model sizes. GFLOPs are forward estimates.

BenchmarkR = 2 errorR = 6 errorR = 2 paramsR = 6 paramsR = 2 GFLOPsR = 6 GFLOPs
Airfoil0.00460.00781,294,753800,41751.44351.443
Darcy0.00480.00861,125,513697,73729.14829.148
Elasticity0.00470.0091556,833357,6652.1432.143

Errors rose on all three tasks under stronger tying. The two schedules have the same forward GFLOPs. Stored parameters and residual scale 1/R1/R both change, leaving the equal-budget comparison open.

To explain residual scaling, Section 3.2 now explains why we scale repeated expert updates. Shared weights receive gradients from every visit, so α = 1/R moderates each expert-residual contribution. The spatial residual remains unscaled. We added α = 1 tests on Airfoil and Darcy, which showed unstable training, to Appendix C.4. The depth sweep retains 1/R1/R.

AuthorsOpenReview ↗

Response to Reviewer 5uTU (3/3) — Expert selection, routing, and transfer

Regarding expert-count selection

To clarify expert-count selection, we clarified in Appendix C.2 the fixed 9:1 training/validation split, freezing EE after validation-based selection, and retraining on the full original training set before testing. Appendix C.6 (Table 11) adds E=1,2,4E=1,2,4 test errors.

Table G. Test errors by expert count.

BenchmarkE = 1E = 2E = 4Selected E
Airfoil0.00520.00500.00464
Pipe0.00430.00440.00423 (0.0036)
Plasticity0.00160.00140.00124
Darcy0.00480.00490.00521
Navier-Stokes0.09690.07640.07114
Elasticity0.00780.00590.00474

Regarding overfitting, eight experts raised Elasticity error to 0.0071. More experts raise storage and dense computation, confounding architecture with capacity when the LinearNO budget is exceeded. Table G reports test errors, which alone cannot quantify train–test gaps.

Regarding physical routing and related designs

To address whether the gate reflects physical structure, Section 3.3 now explains hidden-feature conditioning, dense pointwise mixing, and the domain-decomposition analogy. Hidden features accommodate different PDEs and geometry-to-flow inputs such as ShapeNet-Car. Figure 9 qualitatively illustrates spatial and visit-dependent expert mixtures on three tasks. Entropy, physical-region overlap, and routing-input comparisons have not been measured.

To distinguish PhyLoopNO from GNOT and Soft MoE, we compare routing inputs and expert computation.

GNOT. GNOT uses an MLP on query coordinates followed by softmax. PhyLoopNO uses a learnable affine map on evolving normalized point features followed by softmax.

Soft MoE. Soft MoE combines tokens into expert slots, processes those slots, and maps their outputs back to tokens. PhyLoopNO evaluates all experts at each spatial point and mixes their outputs pointwise.

Regarding shared experts, Jaggi (2026) also uses shared experts with untied routers. PhyLoopNO additionally retains visit-specific slicing/deslicing. Its independent routers give the shared bank visit-specific pointwise responses.

To examine low-rank adaptation, we added an experiment on all six benchmarks using visit-specific rank-16 updates to both expert-FFN projections, frozen FFN checkpoint weights, and a matched trainable-parameter budget. Errors increased toward full-sharing levels. Both models retain visit-specific slicing/deslicing, so this tests FFN adaptation. Direct replacement of those interfaces remains untested.

Regarding baseline comparisons and architectural transfer

To address same-protocol baseline comparisons, we ran GNOT and CNO on all six tasks and added their results to Table 1. Transolver and Transolver++ followed the same protocol. Appendix C.2 now specifies Navier–Stokes viscosity 10−510^{-5}, the 64 × 64 grid, 1,000 training and 200 test cases, and ten observed frames to predict ten autoregressively.

Regarding cross-physics pretraining, we retain this perspective as a future direction because parameter reuse can support scalable PDE architectures. Our models share weights across depth and train separately per PDE. Message-passing solvers share updates across physical time. Adapting regular-grid frameworks such as DPOT to irregular meshes requires changing the spatial representation.

To test transfer beyond LinearNO, we additionally adapted Transolver with the same sharing and expert settings and trained it with five random seeds. The largest reductions are 12.3% on Elasticity and 11.2% on Navier–Stokes. We added this comparison to Section 4.1 and Appendix C.9.

Table H. Transfer to Transolver. Looped Transolver errors are five-seed means.

ModelAirfoilPipePlasticityDarcyNavier-StokesElasticity
Transolver0.00530.00300.00130.00550.08820.0065
Looped Transolver0.00510.00380.00130.00520.07830.0057
Reviewer_hiW3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Low-rank Transformer neural operators solve partial differential equations (PDEs) by modeling global interactions through compact latent representations. In this paper, the author replace the pointwise feed-forward network with physics-gated dense experts. The approach are demostrated in both 2D and 3D experiments and got a stated of are performance.

Critical strengths

The motivation is clear, directly apply transfomer would increase the computational cost but this new appraoch could address this problem. In addtion, the paper shows a state of the art performacne in both 2D and 3D experiments.

Critical weaknesses

[1] The noelty is limited: Parameter sharing in looped Transformers and feature-dependent mixtures of experts are closely related to the current existing techniques. It seems the new method is trying to add these techiques into the LinearNO appraoch.

[2] How about the efficiency of the method? Like the number of parameters and Gfloop. This should be added into the abaltion for further analysis.

[3] More strong baselines should be added. For example, Latent Mamba Operator for Partial Differential Equations.

Questions for authors

Check the Critical Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer hiW3 on novelty, efficiency, and Mamba baselines

We sincerely thank you for recognizing our motivation and for suggesting a stronger baseline and an efficiency comparison.

Regarding methodological novelty

To clarify our methodological contribution, we investigate which spatial transformations can share weights in a PDE operator. LinearNO shows that removing Transolver's latent-token self-attention can preserve or improve accuracy, illustrating why language-model architectures require operator-specific adaptation.

Our parameter-sharing ablations identify visit-specific compression (slicing) and reconstruction (deslicing) as important for preserving accuracy under parameter reuse. As features evolve, each visit can adapt point-to-token aggregation and token-to-point reconstruction.

Thus, at each recurrent core position, PhyLoopNO shares the input, value, and output transformations and expert bank across visits, while keeping slicing/deslicing projections, temperatures, normalization, and routing visit-specific. Different core positions have distinct parameters.

For pointwise transformations, all experts process every spatial point, and each visit's gate weights their outputs using the point's current hidden features. Sparse routing can leave rarely selected experts undertrained. Dense mixing keeps every expert active. This replenishes pointwise capacity under parameter reuse. Our contribution combines operator-specific selective sharing with adaptive dense expert mixing. We revised Sections 3.2–3.3 to clarify both mechanisms.

Regarding parameter efficiency and the ablation

To address the efficiency analysis, we added parameter counts across expert settings and depths to Appendix C.6–C.8. Both looped variants use the same schedule and total FFN hidden width. Replacing the expert bank with one width-EFEF FFN therefore tests whether adaptive mixing helps beyond widening the pointwise branch, as clarified in Section 4.2. For every model, we count matrix multiplications and convolutions across all eight executed layers, at batch size one and two FLOPs per multiply-accumulate.

Table A. Trainable parameters / forward GFLOPs / forward + backward GFLOPs at depth 8. EE is the expert count.

BenchmarkELinearNOLooped LinearNO + wide FFNFull PhyLoopNO
Airfoil41,765,889 / 45.442 / 136.3141,289,737 / 51.374 / 154.1111,294,753 / 51.443 / 154.319
Pipe31,765,889 / 67.092 / 201.2601,207,497 / 71.489 / 214.4491,210,841 / 71.557 / 214.653
Plasticity41,799,428 / 12.833 / 38.3901,323,276 / 14.481 / 43.3341,328,292 / 14.500 / 43.392
Darcy11,766,145 / 29.133 / 87.3881,125,513 / 29.148 / 87.4321,125,513 / 29.148 / 87.432
Navier-Stokes43,377,921 / 29.991 / 89.513900,105 / 12.454 / 37.131905,121 / 12.479 / 37.207
Elasticity4585,217 / 1.626 / 4.876551,817 / 2.137 / 6.411556,833 / 2.143 / 6.429

PhyLoopNO uses fewer parameters than LinearNO on all six tasks. Its forward GFLOPs are lower on Navier–Stokes, nearly equal on Darcy, and higher on the other four tasks. The wide FFN and expert bank differ by less than 1% in total parameters across all six tasks.

Regarding stronger baselines

To strengthen the baseline comparison, we ran LaMO using its official implementation. The linked AMO repository was empty when checked. Its paper leaves the complex-valued pole parameterization and enforcement of the unit-disk constraint unclear, so we reproduced these components using reasonable assumptions.

Table B. Relative test L2L_2 error under our protocol. Bold marks the lowest error in each column.

MethodAirfoilPipePlasticityDarcyNavier-StokesElasticity
LaMO (our reproduction)0.00480.00380.00140.00520.06840.0050
AMO (our reproduction)0.00490.00460.00160.00460.08040.0052
PhyLoopNO (submitted)0.00460.00360.00120.00480.07110.0047

PhyLoopNO has the lowest error on four tasks. LaMO performs best on Navier–Stokes and AMO on Darcy. Our AMO errors on Airfoil and Plasticity exceed twice its published values of 0.0020 and 0.0006. LaMO scans learned latent tokens, whose sequence adjacency need not correspond to physical-space adjacency. Bidirectional scanning alone therefore does not give the scan order an explicit geometric meaning on irregular meshes. Manuscript Table 1 also adds our six-task reproductions of GNOT and CNO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.