acceptodds
Under review as a conference paper at ICLR 2027

What Do Separable Physics-Informed Neural Networks Need from a Recurrent Block?

Abstract

Sequence-model architectures combine interactions between tokens with nonlinear channel transformations, but the contributions of these components to physics-informed learning are difficult to distinguish. We study this question in separable physics-informed neural networks, where independent coordinate branches form a multidimensional field through a sum-of-products representation. We introduce Separable RWKV (S-RWKV), whose branches apply a smooth matrix-memory update to groups of learned Fourier features. Recurrence is confined to feature groups at a single coordinate and does not propagate between physical points. This construction allows us to remove the memory path while retaining the input encoding, channel transformations, field assembly, and training protocol. Across six benchmarks with repeated runs, the memory-free variant has lower mean errors than the full model on most scalar problems and requires fewer parameters and forward operations. Direct projection of the Fourier features, without chan nel transformations, produces larger errors and greater variation across runs on Convection and Wave. Comparisons with a separable multilayer perceptron show that the branch body remains important under matched input features and training conditions. Stage-wise results show that the main accuracy differences arise before output-projection refinement. Under the shared protocol, the memory-free model also outperforms the compared attention- and state-space-based networks on the scalar benchmarks. These results support lightweight nonlinear processing of grouped spectral features as an effective design for separable physics-informed learning

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Separable PINN, PINN, RWKV, PINNsFormer, PINNMamba

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x72JOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Separable RWKV (S-RWKV) architecture, based on separable physics-informed neural networks (separable PINNs), whose branches apply a smooth matrix-memory update to groups of learned Fourier features. Starting from the question of what a separable PINN needs from a recurrent block, the paper conducts extensive experimental comparisons and ablation studies and concludes that (i) accurate approximation can be achieved without recurrent memory, (ii) channel mixing improves training reliability, and (iii) under the shared pipeline, the memory-free model has a smaller model size while achieving lower error.

Critical strengths
  1. The proposed S-RWKV architecture introduce a clean network architecture, clearly separates the channel-mixing path from the recurrent path.
  2. The experiment section in this paper consists of ten random seeds for each configuration (e.g. all float64, same training setting), with the results described in detail, and all separable variants share the same experimental setup.
  3. The experiment results show that a memory-free grouped channel-mixing branch can outperform more complex recurrent/attention-based PINNs, especially on Reaction, Convection, and Wave equations.
  4. The ablation study in this paper is systematic. With the four branch configurations (full, one block, no memory, no blocks), the paper includes a SPINN control that differs from S-RWKV only in the branch body, a parameter-matched non-separable control , and an input–assembly–body ablation.
Critical weaknesses
  1. The method performs very well on benchmarks with low-rank/separable structure. But on some non-separable problems, such as the NS equation benchmark, the method is a little bit weak.
  2. The no-memory branch proposed by the paper overlaps with the multi-scale Fourier feature (MFF) architecture of Wang et al. (2021b). In the algorithm, each coordinate has its own branch that maps groups of learned Fourier features. The relationship to (spatio-temporal) MFF should be discussed.
Questions for authors
  1. All experiments in this paper use float64 data, but now more PINNs paper are using float32 data instead. Could the authors report some float32 results?
  2. How important is learning the Fourier frequencies and phases? Could the authors report results with ωg\omega_g and ϕg\phi_g fixed at their initialization, as in MFF (Wang et al., 2021b)? This would clarify whether learnable frequencies are different from MFF.
  3. Since S-RWKV employs learned Fourier features, the baseline should include some comparisons with recent Fourier-based method, such as [1].
  4. ψ\psi is a fixed sine–cosine combination, and SPINN and No memory differ in activation, weight sharing, and residual scaling. Is the large Convection gap vs SPINN attributable mainly to the activation function, to groupwise weight sharing, or to residual scaling?

[1] Y. Wu, M. Aguiar, K. H. Johansson, and M. Barreau. Iterative training of physics-informed neural networks with Fourier-enhanced features. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.

Reviewer_xmcpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes S-RWKV, a separable PINN whose coordinate branches apply RWKV-based blocks (a memory path and a channel-mixing path) to groups of learned Fourier features, and asks whether separable PINNs need a recurrent structure by ablating the two paths in turn. Based on the finding that removing the memory path preserves accuracy while additionally removing channel mixing increases error and variance on some problems, the authors conclude that lightweight nonlinear processing of grouped Fourier features is sufficient for separable PINNs and that, under a shared protocol, it is more accurate and efficient than existing sequence-model PINNs.

Critical strengths
  1. The controlled design, which swaps only the branch body while fixing the input, assembly, and training protocol, is combined with repeated runs and statistical tests, making the reporting rigorous.
  2. The factorial ablation over input, assembly, and branch body is an informative analysis showing that no single factor suffices, and the resulting compact model has practical value.
  3. The paper openly reports the cases where memory helped, where the non-separable model was better, and the limitations of the supplementary benchmarks.
Critical weaknesses
  1. The recurrent block being removed was added by the authors themselves, so the question posed in the title is of their own making; framing the work around an RWKV block that is added and then removed obscures the actual contribution, namely introducing Fourier input and a periodic-activation, group-shared MLP into the SPINN branch, which is an already well-known combination.
  2. The difference from SPINN is attributed to three ingredients (activation, weight sharing across groups, and residual scaling) without ablating them individually, so it is unclear whether the effect comes from channel mixing or simply from the periodic activation.
  3. The shared training pipeline uses a short budget, and the loss weights and sampling settings are not justified, so it cannot be determined whether the failure of the Fourier-input tanh MLP body and the gap to the baselines reflect architectural differences or the training budget.
  4. The output-projection refinement adds complexity to the pipeline that is disproportionate to its gains.
Questions for authors
  1. Can the authors provide ablations between SPINN and “No memory” that change the activation, the weight sharing across groups, and the residual scaling one at a time?
  2. How were the training budget and loss weights chosen? In addition, how do the models’ results change under a longer training budget?
  3. With the frequency initialization scales fixed to the paper’s setting, does accuracy hold when β in Convection is increased so that the required frequency falls outside the initialization range, and can the authors compare the frequency distributions before and after training to show whether the correct frequencies are found by learning or supplied by initialization?
Reviewer_sArKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether recurrent memory is really needed in separable PINNs. They introduce, S-RWKV and show that even if the recurrent memory is removed, the accuracy is often improved while reducing cost.They find that nonlinear channel mixing to be more important than recurrence on several benchmarks.

Critical strengths
  1. Ablations: The systematic comparison of recurrent memory, channel mixing and separable representations, using 10 runs and a shared training protocol is fair.

  2. Computational efficiency: The memory free variant achieves competitive accuracy and much lower costs.

Critical weaknesses
  1. This work, while useful is incremental, and does not meet the ICLR bar in my opinion. It combines existing separable PINN representations with RWKV-inspired blocks, and the main finding is that removing recurrence often works just as well or better. The contribution is therefore primarily empirical rather than methodological.
  2. Unclear source of improvements: The channel-mixing architecture differes from SPINN in activation, parameter sharing, and residual scaling. It is therefore difficult to identify which component is responsible for the large performance differences.
Questions for authors

Could the authors clarify the novelty of the work a bit better, and why they think it matches ICLR bar, instead of a workshop result?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.