acceptodds
Under review as a conference paper at ICLR 2027

Preserving Local Detail in Neural PDE Simulators with Structured Latent States

Abstract

High-fidelity numerical simulation of complex physical systems often incurs prohibitive computational costs, particularly for time-dependent problems that require repeated fine-resolution updates. Neural PDE simulators provide a data-driven alternative, while latent simulators further reduce rollout cost by evolving a spatially compressed representation of the physical field. However, strong spatial compression can remove the localized structures that require fine resolution in the first place. In this paper, we propose a neural PDE simulation approach that preserves localized fine-scale structure under strong spatial compression. Motivated by classical numerical methods, the approach emphasizes expressive local representations through two design principles: (i) a spatially structured latent state that preserves where information occurs in the physical domain, and (ii) a partition-of-unity blending scheme that reconstructs a continuous physical field from local latent representations. We instantiate this approach in Spatial Patch Neural Simulator (SPANS), an efficient latent simulator for high-resolution, time-dependent PDEs. Its architecture couples short- and long-range attention processor with an expressive decoder whose patch-conditioned weights vary with spatial position. Across seven time-dependent simulation problems spanning structured grids, irregular meshes, and point clouds, SPANS achieves strong rollout accuracy, preserving localized spatial detail even under substantial compression, while also maintaining lower rollout time than most methods.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PDE Learning, Neural Simulators, PDE Latent Dynamics, Physical Modeling, Mesh-based Simulations, AI4Science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9mV8OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposed a general framework for neural simulation that aims to preserve local details while performing autoregressive rollouts using coarse spatial patch states. The framework is based on patchwise Galerkin projection, patch-state evolution, and partition-of-unity reconstruction. The authors instantiated this framework as SPANS, which uses a DeepSet encoder to aggregate input observations into spatial patch states, a processor with short- and long-range attention to evolve these states, and a patch-conditioned decoder to reconstruct a continuous physical field through partition-of-unity blending.

Critical strengths
  1. A general framework for latent-space simulation: The paper presented a conceptual framework that interprets autoregressive simulation in latent space through Galerkin projection, latent-state evolution, and field reconstruction.

  2. Accuracy and efficiency on long rollouts: The method demonstrated strong predictive accuracy with relatively low computational cost on tasks involving long autoregressive rollouts.

Critical weaknesses
  1. The correspondence between SPANS and the general framework in Section 3 was not sufficiently established: SPANS was presented as a concrete instantiation of the general framework, but there are major differences that make the correspondence unclear.
  • Encoder: The general framework motivated the latent representation through the Galerkin projection in Eq. (4), but its connection to the DeepSet encoder in SPANS is unclear. When the basis and projection rule are independent of the input field, projecting the input field gives coefficients that depend linearly on the input field, whether the basis is prescribed or learned. The MLP-pooling-MLP encoder does not enforce this linearity. Although the DeepSet encoder could approximate such a projection during training, the paper did not specify the corresponding projection or establish that the encoder approximates it. The authors should clarify how the learned initial patch states relate to the Galerkin coefficients introduced in Section 3.
  • Decoder: Partition-of-unity in Eq. (9) reconstructs the field using local basis functions b_(p,j) (x) that are independent of the latent states z_(p,j) (t). However, partition-of-unity decoder of SPANS reconstructs the field using the local basis functions A^((1) ) (x,t) which get z_p (t) as an input. Although Eq. (12) generalizes the reconstruction to learnable local functions, the paper does not explain how this generalization preserves the Galerkin interpretation of the latent states as projection coefficients. Especially, original separation between spatial basis functions and evolving coefficients became unclear in partition-of-unity decoder of SPANS. The authors should justify this transition and explain whether and how the resulting architecture retains the projection–evolution–reconstruction interpretation introduced in Section 3. These gaps also leave the applicability of the theoretical analysis in Section 4 to the implemented SPANS architecture insufficiently justified. For example, the proof relies on reconstructing both the reference and learned fields using the same state-independent spatial basis b_p (x), as shown in Eqs. (25) and (54). The authors should establish the required correspondence or clearly delimit the scope of the theoretical result.
  1. The theoretical analysis did not sufficiently substantiate the proposed model design: Theorem 1 provides a conditional error-propagation bound in terms of approximation errors and stability constants, but it did not establish how the proposed architectural choices improve these quantities.
  • The role of patch count in preserving local details was insufficiently justified. As noted in lines 197-199, for any number of patches P, the partition-of-unity expansion in Eq. (9) can be written as the global expansion in Eq. (4) by choosing ψ_((p,j) ) (x)=w_p (x) b_(p,j) (x), with the same coefficient count M=PJ. Thus, at a fixed total latent budget M=PJ, since increasing P reduces the number of coefficients J available per patch, a reduction in local approximation error does not follow automatically. Beyond the qualitative discussion in lines 170-176 and the empirical evidence in Table 11, the theoretical analysis should quantify how patch count affects local approximation error and identify conditions under which it provides an advantage over a specified global representation. Otherwise, the theoretical justification for using a sufficiently large number of spatial patches, as suggested in lines 265–267, remains incomplete.
  • The role of an expressive decoder was insufficiently justified. By the definition of tilde{u}^* in Eq. (26), Assumption 3 bounds the discrepancy between the reference Galerkin field y^* and its reconstruction h_(θ_rec ) (x,y^). This discrepancy can become zero by choosing the identity map h_(θ_rec ) (x,y^)=y^*. Consequently, this reconstruction error alone does not explain why an expressive reconstruction map is needed or why increasing its expressivity should improve accuracy as suggested in lines 267-268.
  • The practical informativeness of Theorem 1 for long rollouts remains unclear. The third term in Eq. (14) contains an exponential factor e^(L_dyn T), which may make the bound excessively loose as the rollout horizon increases. For experimental horizons such as T=30, the paper did not estimate the relevant stability factors or provide architecture-specific control of the error terms, leaving the quantitative informativeness of the bound unclear. Although Table 2 and the ablations in Appendix H provided evidence on end-to-end performance and the effects of individual architectural choices, they did not directly verify reductions in the corresponding theoretical error terms or isolate the claimed representational advantage over a specified global representation with the same latent budget. Architecture-specific analysis or targeted experiments addressing these gaps are needed to substantiate the connection between the theoretical results and the proposed design choices.
Questions for authors
  1. Consistency of the Smoke experiments with the problem formulation: Could the authors clarify how the SmokeGrid and SmokeObs tasks fit the PDE formulation in Eq. (1)? Since density evolution depends on velocity, what assumptions or additional state information make the density-only prediction setting in Table 3 consistent with this formulation?

  2. Processor specification and comparison with GraphGPS [1]: Could the authors explicitly define the short- and long-range attention operations in Eq. (16), including how attention scores and local neighborhoods are computed? Please also clarify how this processor differs from GraphGPS’s combination of local message passing and global attention, and identify the specific methodological novelty.

9 comments
AuthorsOpenReview ↗

W1: decoder [post 1/5]

We thank the reviewer for the detailed and precise reading of our framework and theory. The comments led us to restructure the analysis in two places. Section 3 now derives the Galerkin projection for the nonlinear decoder, and Theorem 1 (Section 4, proof in Appendix B) now applies to this decoder directly. We answer each point below.

W1, Decoder

"[...] the paper does not explain how this generalization preserves the Galerkin interpretation of the latent states as projection coefficients. Especially, original separation between spatial basis functions and evolving coefficients became unclear in partition-of-unity decoder of SPANS."

We agree that the step from Eq. (9) to Eq. (12) of the submission was not fully derived, and the revised Section 3 now derives it, starting from the linear case. It replaces the fixed basis functions of the linear expansion with the nonlinear decoder (Eq. (6)), so the approximate field lies on the manifold induced by the decoder. The velocity of this field is still the solution of a Galerkin approximation. By the chain rule (Eq. (7)), the tangent functions ψm=∂Dθdec/∂zm\psi_m=\partial\mathcal D_{\theta_{\rm dec}}/\partial z_m span the tangent space of the manifold at the current state and take the role of the basis functions. Galerkin projection onto this tangent space is therefore analogous to the least-squares problem in Eq. (5) and again yields a system of ODEs (Eq. (8)). The difference to the linear case lies in how the basis functions are formed.

  • In the linear expansion (Eq. (5)), the basis functions ψm(x)\psi_m(\mathbf x) are independent of the state, so the mass matrix Φ\boldsymbol\Phi is constant.
  • With the nonlinear decoder (Eq. (8)), the tangent functions ψm(x,z;θdec)\psi_m(\mathbf x,\mathbf z;\theta_{\rm dec}) depend on the state, and so do the mass matrix Φ(z;θdec)\boldsymbol\Phi(\mathbf z;\theta_{\rm dec}) and the projected right-hand side.

The latent state is thus no longer the coefficient vector of a fixed basis, but its velocity is still the solution of a Galerkin approximation at every time.

"The authors should justify this transition and explain whether and how the resulting architecture retains the projection–evolution–reconstruction interpretation introduced in Section 3."

The resulting architecture retains this interpretation, and each of the three steps has a direct counterpart in Section 3.

  • In the projection, the initial patch states are the least-squares fit of the initial field with the decoder (paragraph "Encoder"), and the encoder approximates this fit.
  • In the evolution, the velocities of the patch states solve the least-squares problem in Eq. (12). Since its right-hand side requires the unknown operator FF, the processor learns these dynamics instead (Eq. (13)).
  • In the reconstruction, the decoder maps the patch states to the field (Eq. (14)). Each tangent function carries the weight wpw_p, so each patch state acts only on its own patch (Eq. (10)).

For a fixed basis, ψp,j=wpbp,j\psi_{p,j}=w_pb_{p,j}, and the classical PUM-Galerkin system of the submission is recovered as a special case.

"[...] the proof relies on reconstructing both the reference and learned fields using the same state-independent spatial basis bp(x)b_p(x), as shown in Eqs. (25) and (54). The authors should establish the required correspondence or clearly delimit the scope of the theoretical result."

Theorem 1 now applies to the nonlinear decoder of SPANS directly, and its proof no longer uses a state-independent basis. Two changes make this possible.

  1. The Galerkin stability assumption of the submission is now stated on the manifold induced by the decoder instead of the linear PUM space (Assumption 1). The reference is the Galerkin solution on this manifold, i.e., the solution to Eq. (8).
  2. The basis bp(x)b_p(x) is replaced by local maps that are Lipschitz in their patch state (Assumption 3(ii)). The reference and the learned fields are both reconstructed by the same decoder, and their difference is bounded through its Lipschitz constants.
AuthorsOpenReview ↗

W1: encoder, and W2: patch count [post 2/5]

W1, Encoder

"[...] the paper did not specify the corresponding projection or establish that the encoder approximates it. The authors should clarify how the learned initial patch states relate to the Galerkin coefficients introduced in Section 3."

Section 3 (paragraph "Encoder") now specifies this projection. The initial latent state is given by an analogous least-squares problem for the state instead of the velocity,

Z(0)∈arg⁡min⁡Z(0)∥Dθdec(⋅,Z(0))−u(⋅,0)∥L2(Ω)2.\boldsymbol Z(0)\in\arg\min_{\boldsymbol Z(0)}\left\lVert\mathcal D_{\theta_{\rm dec}}(\cdot,\boldsymbol Z(0))-u(\cdot,0)\right\rVert_{L^2(\Omega)}^2.

For a fixed basis, this fit gives the Galerkin coefficients of the initial field. The encoder is a learned approximation of this fit. The Galerkin trajectory that serves as the reference of Theorem 1 starts from this fit (Section 3, paragraph "Encoder"), and the encoding error εenc\varepsilon_{\rm enc} is the distance between the encoder output and the fit.

"When the basis and projection rule are independent of the input field, projecting the input field gives coefficients that depend linearly on the input field [...]. The MLP-pooling-MLP encoder does not enforce this linearity."

We agree for a fixed basis, where the normal equations of this fit are linear in the input field. SPANS uses the same least-squares fit, but on the manifold of its nonlinear decoder (Section 3, paragraph "Encoder"). The fit is then a nonlinear function of the input field, so the encoder has to be nonlinear to approximate it. DeepSet is a simple and efficient choice for this, and cross-attention and continuous convolution reach a similar MSE while using more memory (Figure 6).

W2, The role of patch count

"[...] at a fixed total latent budget M=PJM=PJ, since increasing PP reduces the number of coefficients JJ available per patch, a reduction in local approximation error does not follow automatically."

We agree, and Theorem 1 now makes this trade-off explicit. Its first term, the global approximation error εglo\varepsilon_{\rm glo}, is the error of a classical PUM with the same latent size, i.e., PP patches with local spaces of dimension JJ. This error can be reduced in two classical ways, by smaller patches (hh-refinement, larger PP) or by richer local spaces (pp-refinement, larger JJ). At a fixed budget M=PJM=PJ, the two compete, since more patches leave a smaller JJ for each patch. More patches therefore do not always reduce this error.

"[...] the theoretical analysis should quantify how patch count affects local approximation error and identify conditions under which it provides an advantage over a specified global representation."

Lemma 1 (Appendix B) bounds the error of the reference solution by BT(εglo+εdec)B_T(\varepsilon_{\rm glo}+\varepsilon_{\rm dec}), with the stability constant BTB_T of Assumption 1, so the approximation error of the classical PUM analysis carries over to the first term of the bound (Melenk & Babuška, 1996). For example, with local polynomials of degree μ\mu, the classical error estimate is εglo=O(δmin⁡(μ+1,s))\varepsilon_{\rm glo}=O(\delta^{\min(\mu+1,s)}), where δ∝P−1/d\delta\propto P^{-1/d} is the patch diameter in dd dimensions and ss is the regularity of the solution, i.e., the number of bounded derivatives it has (Brenner & Scott, 2008). More patches thus shrink δ\delta, while richer local spaces raise μ\mu, which helps only up to ss. Which refinement pays off depends on the local regularity of the solution.

  • Smooth solutions favor richer local spaces or a global representation.
  • Localized features of low regularity, such as the filaments, wrinkles and contact in our tasks, favor smaller patches, which can represent their fine detail and are easier to learn.

Table 11 shows this on SmokeGrid at a fixed latent budget, and the table in our answer to Reviewer S4Hh (Q2) lists the settings side by side.

  • At a budget of 4,096, more patches (256 × 16) give a lower MSE than wider patch states (64 × 64).
  • At a budget of 16,384, 256 patches (256 × 64) give a lower MSE than 16 large patches (16 × 1024). The latter is the setting closest to a global representation, and its MSE is 2.48×2.48\times as large.
AuthorsOpenReview ↗

W2: expressive decoder [post 3/5]

W2, The role of an expressive decoder

"[...] This discrepancy can become zero by choosing the identity map [...]. Consequently, this reconstruction error alone does not explain why an expressive reconstruction map is needed or why increasing its expressivity should improve accuracy [...]"

We agree, and Theorem 1 now treats the complete decoder Dθdec\mathcal D_{\theta_{\rm dec}} (Eq. (14)). In the submission, the reconstruction error compared the reference field with its own reconstruction, so the bound could not show what an expressive map contributes. Now εdec\varepsilon_{\rm dec} measures how well the decoder reproduces the blend uPUu_{\rm PU} of a classical PUM with the same latent size (Assumption 3(iii)),

∥uPU−Dθdec(⋅,ZPU)∥[0,T]≤εdec.\lVert u_{\rm PU}-\mathcal D_{\theta_{\rm dec}}(\cdot,\mathbf Z_{\rm PU})\rVert_{[0,T]}\le\varepsilon_{\rm dec}.

Comparing the identity map with a nonlinear decoder shows what an expressive map contributes.

  • With the identity map and local maps that are linear in the patch states, the decoder is a classical PUM with a fixed basis, so εdec=0\varepsilon_{\rm dec}=0 and Lrec=1L_{\rm rec}=1. The approximation error is then εglo\varepsilon_{\rm glo} of this basis, which decreases only through hh- or pp-refinement, i.e., with a larger latent state for the dynamics to learn and evolve. This choice is sufficient when a fixed basis of the given size already captures the solution.
  • With a nonlinear decoder, the local functions change with the latent state (Eq. (15)), so they can follow the solution as it evolves. An expressive map can reproduce the classical approximation closely, which keeps εdec\varepsilon_{\rm dec} small. However, its sensitivity to the latent state must stay bounded, since otherwise it would amplify small latent errors. We therefore assume a Lipschitz decoder (Assumption 3(i) and (ii)), whose constants scale these errors in the bound.

Our ablation further shows the benefit of expressivity. The shared decoder, a special case of the more expressive patch-conditioned one, has 1.47×1.47\times the rollout MSE on Cylinder and Airfoil and 1.11×1.11\times on Teddy (Table 13).

Reviewer_CimXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the SPANS model, which concerns latent evolution models for PDEs. The architecture is based on patches derived from the partition-of-unity method, which are encoded individually. In the latent space, the time axis is propagated using a simple transformer architecture. To decode, the model uses a hypernetwork for additional expressivity. The experiments compare SPANS to multiple latent-processing and standard neural PDE solvers on tasks with regular and irregular grids, showing that SPANS can outperform prior methods on most tasks.

Critical strengths
  1. Interesting and novel architecture that combines latent evolution models with PUM encoder/decoder for solving autoregressive PDEs. The method descriptions are also easy to understand (however, some core aspects, such as the training procedure, are left to the appendix).
  2. Very extensive experiments on difficult tasks showing an improvement over common neural PDE solver baselines, including statistical evaluations over multiple seeds. They include not only accuracy but also rollout speed and data efficiency, for example. Ablations of core architectural components are also given in the appendix.
Critical weaknesses

None

Questions for authors
  1. Why did you compare to the old Transolver and not newer versions such as Transolver++?
2 comments
AuthorsOpenReview ↗

Comparison with Transolver++ (Q1)

We sincerely thank the reviewer for the positive and encouraging assessment. We are glad that the reviewer found the architecture interesting and novel, the method description easy to follow, and the experiments and ablations extensive.

Q1

"Why did you compare to the old Transolver and not newer versions such as Transolver++?"

Transolver++ (Luo et al., 2025) builds on the same architecture as Transolver. Its additions, an optimized parallel framework and a local adaptive mechanism, are designed to scale the physics attention to geometries with millions of mesh points. We therefore compare with the original Transolver in the configuration published by its authors.

To make this comparison as fair as possible, we train Transolver both with its default one-step objective and with multistep training, and report the better setting on every task (Section 6). It is a strong baseline, as Transolver is the best method on Cylinder, where it is ahead of SPANS, and the second best on Airfoil (Table 1).

Thank you for the clarification. After reading this response and the rest of the rebuttal, I maintain my original, already positive rating.

Reviewer_S4HhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Spatial Patch Neural Simulator (SPANS), a latent neural simulator for time-dependent PDEs that aims to preserve fine-scale spatial details under compression. The method first encodes the input field into spatial patch states, then evolves these states using short-range and long-range attention, and finally reconstructs the physical field through a partition-of-unity decoder with patch-dependent weights. The authors test SPANS on seven simulation benchmarks involving regular grids, irregular meshes, and point clouds, and report good rollout accuracy and computational efficiency. The paper also provides a theoretical error bound and several ablation studies.

Critical strengths
  1. The experimental results are quite good across different simulation problems. SPANS achieves competitive or better rollout accuracy on most benchmarks. I also appreciate the additional evaluation of fine-scale details, such as high-frequency structures, wrinkles, and curvature, rather than just reporting overall prediction errors.
  2. The latent simulation framework is computationally efficient. Evolving the patch states without repeatedly reconstructing and encoding the full physical field can reduce the rollout cost. The continuous decoder and sampled-query training are also useful for high-resolution simulations.
Critical weaknesses
  1. My main concern is the limited methodological novelty. The overall encoder-processor-decoder framework, autoregressive latent evolution, spatially localized latent representations, and continuous neural-field reconstruction are already well explored in existing latent PDE simulators, particularly CALM-PDE and AROMA. SPANS seems to mainly put together several existing techniques, including spatial patches, local/global attention, partition-of-unity blending, and hypernetwork-based decoding. I am not quite convinced what the key new methodological contribution is compared with these existing works.

Also, some ablation results make me question how much the proposed components really help. For example, Table 12 shows that using only long-range attention actually performs better than the full model on several tasks. In Fig. 9, partition-of-unity blending barely changes the rollout MSE, although it does improve the visual smoothness. Similarly, Table 13 shows rather small improvements from the patch-conditioned decoder on several benchmarks. I think these results make it difficult to judge which parts of the proposed architecture are really important, and whether the added complexity is justified.

  1. I also have some concerns about the compression claims and the generalization of the method. Although SPANS reduces the number of spatial tokens, the total latent dimension (P x J) is actually close to the number of physical scalar variables (N x C) on several benchmarks, such as SmokeGrid, FormNapf, and Teddy (Tables 3 and 6). So in these cases, the method mainly reduces the number of tokens, but not necessarily the total latent dimension. This is still useful for computational efficiency, but I think it is somewhat different from achieving strong latent compression. It would be more convincing to see how the method performs under a much smaller total latent dimension, and to compare with other latent simulators using similar latent capacities.

I also find the theoretical analysis somewhat disconnected from the actual model. The error bound is derived using fixed local approximation functions and several assumptions, while the implemented decoder is nonlinear and depends on the evolving latent states. It is not very clear to me how directly this theoretical result applies to SPANS, or what it really tells us about preserving fine-scale details and long-term rollout stability.

The resolution-scaling experiments seem to train separate models for different resolutions. This shows how the method performs and scales at different resolutions, but does not really test whether a trained model can generalize to unseen resolutions. I would like to see more experiments on cross-resolution or cross-mesh generalization, as well as longer rollout horizons.

Questions for authors
  1. Could the authors better explain what is fundamentally new in SPANS compared with existing latent PDE simulators, especially CALM-PDE and AROMA? Apart from the use of spatial patches and partition-of-unity blending, what are the main differences? I think more controlled comparisons with similar latent capacities would help.

  2. Since P x J is close to N x C on several benchmarks, how much compression does SPANS actually achieve in terms of the total latent dimension? How would the method perform with a much smaller latent representation?

  3. In Table 12, removing short-range attention improves the results on several tasks. Also, the improvements from partition-of-unity blending and patch-conditioned decoding seem quite small in some cases. Could the authors further explain why these components are needed and what each contributes?

  4. How does the theoretical error bound, which is based on fixed local approximation functions, relate to the actual nonlinear, state-dependent decoder? Does the analysis give any specific guarantees for fine-scale detail preservation or long-term rollout stability?

  5. Could the authors provide additional experiments on zero-shot cross-resolution or cross-mesh generalization, as well as longer rollout horizons? I also think more comparisons with similar latent capacities and training settings would help make the experimental conclusions more convincing.

5 comments
AuthorsOpenReview ↗

Novelty and the organization of the latent tokens (Q1) [post 1/4]

We thank the reviewer for the thoughtful review and for recognizing the accuracy, the efficiency of the latent rollout and the evaluation of fine-scale detail. We answer each question below. The weaknesses are covered by the corresponding questions.

Q1 (and W1)

"Could the authors better explain what is fundamentally new in SPANS compared with existing latent PDE simulators, especially CALM-PDE and AROMA? Apart from the use of spatial patches and partition-of-unity blending, what are the main differences? I think more controlled comparisons with similar latent capacities would help."

We refer the reviewer to point 1 of the General Response, "Novelty relative to existing latent simulators". We agree that the encoder-processor-decoder is the standard design for latent PDE simulators and that the methods differ in the choice of each component. Below, we highlight a key difference in how the latent tokens are organized. Other differences include the continuous convolution decoder of CALM-PDE and the two-stage training of AROMA. We address the comparison at similar latent capacities in our answer to Q2.

CALM-PDE compresses the input similar to a CNN. Each continuous convolution produces fewer spatial points with more channels per point, and the model learns where to place the final latent tokens. AROMA uses cross-attention in two steps. The tokens first attend to the input coordinates and then use that geometric information to gather the field values. Both methods thus learn the spatial organization of their latent representation.

In contrast, SPANS fixes the spatial organization of its latent states through a patch cover. This lets a simple encoder aggregate the observations directly within each patch and shifts the design effort toward the latent dynamics and the reconstruction. This has two consequences, both supported by our experiments.

  1. The encoder becomes secondary. SPANS uses a DeepSet encoder, a simple and efficient aggregation. When we replace it with cross-attention or continuous convolution, the aggregation styles of AROMA and CALM-PDE, these reach a comparable MSE while requiring more memory (Figure 6).
  2. The capacity goes where it pays off. Following the Galerkin-PUM principle, the latent budget can be spent on more patches or on wider patch states, similar to hh- and pp-refinement in classical numerical analysis, and both reduce the global approximation error of Theorem 1, while a more expressive decoder keeps the reconstruction error small. On SmokeGrid, at a fixed total latent dimension, more patches give a lower rollout MSE than wider patch states (Figure 3(b,c), Table 11), and the patch-conditioned decoder can improve the rollout accuracy and help preserve local details (Table 13).

This gives a simple guideline for practitioners, which is to fix the spatial organization of the latent state, keep the encoder simple, and spend the capacity on the number of patches, the latent dynamics and the decoder.

AuthorsOpenReview ↗

The role of each component (Q3) [post 2/4]

Q3 (and W1)

"In Table 12, removing short-range attention improves the results on several tasks. Also, the improvements from partition-of-unity blending and patch-conditioned decoding seem quite small in some cases. Could the authors further explain why these components are needed and what each contributes?"

Each component addresses a different requirement, and its benefit appears on the tasks where that requirement is active.

  • For the processor, which interactions dominate depends on the reference physics, and we assume that this is not known in advance. A processor with only one branch therefore degrades on tasks that need the other. In Table 12, removing the long-range branch raises the error by 2.05×2.05\times on Teddy and 1.66×1.66\times on Cylinder, and removing the short-range branch raises it by 1.18×1.18\times on SmokeGrid. The combined processor stays within 1.22×1.22\times of the better single branch on every task, whereas the wrong single branch costs up to 2.05×2.05\times. Combining both is thus the safer default when the interaction range is unknown, although tuning it per task could improve the accuracy further.
  • The partition-of-unity blend turns the local representations into one continuous global field. Without it, the reconstructed field can jump at patch boundaries, where the spatial derivatives needed for, e.g., strain or vorticity are undefined. Both variants reach a similar MSE in Figure 8, but only the blended field is continuous, and these artificial seams violate the continuity required by the reference physics.
  • Patch conditioning lets the decoder adapt its weights to each patch. The shared decoder has 1.47×1.47\times the rollout MSE of the patch-conditioned one on Cylinder and Airfoil and 1.11×1.11\times on Teddy, and patch conditioning also improves crease placement on FormNapf and FormDome and curvature placement on Teddy (Table 13). Since the shared decoder is a special case of the patch-conditioned family, patch conditioning can match or improve the best achievable reconstruction error, the term εdec\varepsilon_{\rm dec} of Theorem 1.

Point 3 of the General Response summarizes these design choices together with their evidence.

AuthorsOpenReview ↗

Compression and the theory (Q2, Q4) [post 3/4]

Q2 (and W2)

"Since P x J is close to N x C on several benchmarks, how much compression does SPANS actually achieve in terms of the total latent dimension? How would the method perform with a much smaller latent representation?"

In latent PDE simulators, spatial compression refers to the number of latent tokens PP used to represent a field sampled at NN points, with P≪NP\ll N. On SmokeGrid, 16,38416{,}384 points become 256256 patch states, and on Teddy, 17,32317{,}323 points become 406406. The latent width JJ instead controls the capacity of each patch state, analogous to the expressivity of a local approximant in classical numerical analysis. The same distinction appears in CALM-PDE, which uses fewer spatial points and more channels in each successive convolutional layer. We set PJ≈NCPJ\approx NC on purpose in our main experiments, where CC is the number of field channels, so that the total latent dimension stays comparable to the number of values in the original field and we can study how a given capacity is distributed across patches.

Figure 3(b,c) and Table 11 test a much smaller latent representation, and Table 11 also allows a side-by-side comparison with CALM-PDE, the closest latent baseline. To make a fair comparison, we already increased the latent tokens of CALM-PDE from its default of 16 to 64 and noted that larger token sets exceed the 40 GB GPU memory (Appendix E).

Model (SmokeGrid)Tokens × widthLatent budgetMSE (×10−2\times10^{-2})
SPANS, default256 × 6416,3840.255
SPANS, narrower patch states256 × 164,0960.270
SPANS, fewer patches64 × 644,0960.414
CALM-PDE (Table 1)64 × 644,0960.424
SPANS, 16 patches16 × 102416,3840.632

The table shows three things.

  • A quarter of the latent budget costs little as long as the patches are kept. The setting 256 × 16 has 1.06×1.06\times the default MSE, against 1.62×1.62\times for 64 × 64.
  • With the same tokens and the same latent budget (64 × 64), SPANS is on par with CALM-PDE. The gain of SPANS comes from spending this budget on more patches (256 × 16), which the fixed patch structure makes affordable.
  • A few large patches perform worst. The setting 16 × 1024, the closest to a global representation, has 2.48×2.48\times the default MSE at the same latent budget.

This is consistent with hh-refinement in classical PUM and FEM analysis (see also our answer to Reviewer 9mV8, W2). Across the benchmark, CALM-PDE with 64 tokens stays behind SPANS on most tasks (Table 1).

Q4 (and W2)

"How does the theoretical error bound, which is based on fixed local approximation functions, relate to the actual nonlinear, state-dependent decoder? Does the analysis give any specific guarantees for fine-scale detail preservation or long-term rollout stability?"

Reviewer 9mV8 raised a related point, and together these comments led us to restructure the analysis (see also our answers to 9mV8, W1 and W2). Theorem 1 now applies to the nonlinear, state-dependent decoder of SPANS directly, and its bound speaks to both properties.

  • Projecting the PDE onto the tangent space of the manifold induced by the decoder gives a least-squares problem and a system of ODEs for the patch states analogous to the linear case, now with state-dependent tangent functions instead of fixed basis functions (Section 3). The reference of Theorem 1 is accordingly the Galerkin solution on the manifold induced by the decoder, instead of the solution in a fixed basis, with the submitted stability assumption now stated on this manifold (Assumption 1).
  • Fine-scale detail is preserved by design through the partition of unity. In Section 4, we define the global approximation error εglo\varepsilon_{\rm glo} as a collection of the local errors over all patches. As in the classical PUM analysis, this error can be controlled by increasing the number of patches (hh-refinement) or by using richer local spaces (pp-refinement) (Melenk & Babuška, 1996). Our experiments show that SPANS retains fine-scale structure at the correct spatial locations (Table 2), and that the error decreases with more patches at the same latent budget (Figure 3(b,c), Table 11).
  • For long-term stability, only the encoding and dynamics errors accumulate over the rollout, while the global approximation and reconstruction errors enter the bound only once, scaled by the stability constant. Decoding and encoding at every step would accumulate the encoding error at an exponential rate, so we encode only once and roll out in the latent space. Since both errors accumulate, we train with a multistep loss, which directly penalizes the accumulated error. The error of SPANS remains bounded over the rollout, while GAOT, which decodes and re-encodes every prediction, diverges (Figure 5). SPANS also stays stable when rolled out beyond its training window (Appendix H.6, Figure 12).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response [post 1/2]

We thank all reviewers for their constructive feedback, which has contributed to make the paper clearer and stronger.

Reviewers found the architecture interesting and novel (CimX) and highlighted our novel conceptual framework that interprets latent simulation through Galerkin projection, latent evolution and reconstruction (9mV8). Reviewers also valued the strong rollout accuracy at low computational cost (S4Hh, 9mV8, CimX), the extensive experiments with multiple seeds, data efficiency and ablations (CimX), and the evaluation of fine-scale detail beyond the overall error (S4Hh).

In this response, we include comments relevant to multiple reviewers. The remaining points are responded to each reviewer individually below.

Changes relevant to the reviewers' comments are highlighted in blue in the revised PDF.

1. Novelty relative to existing latent simulators (S4Hh, 9mV8)

SPANS’ key novel idea is to leverage classic Galerkin and Partition-of-Unity methods in a way that can be combined with modern deep-learning architectures to preserve the coarse spatial organization of the vector field in the learned representation. As the results in Table 1 show, this way of preserving the spatial organization is a powerful inductive bias that results in very high accuracy (the highest in all environments but one) and a new point in the accuracy-efficiency Pareto frontier of the new Figure 1 left (measured on SmokeGrid only due to the time constraints of the rebuttal).

Previous methods tried to induce the spatial organization of the vector field in the learned representation indirectly, e.g., by using (continuous) convolutional neural networks or by learning spatial tokens with transformers. But our experiments show that our direct approach allows to leverage locally expressive representations that result in higher performance than previous approaches.

In addition, Appendix A discusses the related latent simulators, and Tables 4 and 5 (Appendix E) describe how SPANS differs from the baselines in terms of architectural and training axes.

2. Connection between the theory and the implemented model (9mV8, S4Hh)

Both reviewers pointed out that our analysis, which assumed local expansions that are linear in the latent states, didn’t apply to the SPANS algorithm, which uses a nonlinear decoder. We have extended the previous analysis to include a nonlinear decoder, in two steps:

  1. First, we extend Galerkin approximation from a linear space to the manifold induced by the nonlinear decoder (Section 3). Projecting the PDE onto the tangent space of this manifold gives a least-squares problem and a system of ODEs for the patch states analogous to the linear case (now with state-dependent parametric basis functions derived from the decoder).
  2. Second, we extend the theoretical analysis accordingly (Section 4, Appendix B). Fortunately, the key techniques remain similar to the linear case. By extending the Galerkin stability assumption to a manifold and ensuring smoothness of the intermediate field (first stage of the decoder), Theorem 1 now applies to the nonlinear decoder directly. Its reference is the Galerkin solution on this manifold, obtained by projecting the PDE onto its tangent space.

Theorem 1 shows explicitly how the error splits into the global approximation error of a classical partition-of-unity method, the reconstruction error of the nonlinear decoder, and the error of the latent evolution from the encoder and the dynamics.

AuthorsOpenReview ↗

General response [post 2/2]

3. How the theoretical analysis guides the design choices (S4Hh, 9mV8)

Theorem 1 splits the approximation error into multiple terms, and several of them do help to guide the design of SPANS’s components, which we evaluate in the ablation experiments, as summarized here:

  • The global approximation error εglo\varepsilon_{\rm glo} is analogous to the error of the classical partition-of-unity method with PP patches and JJ functions per patch, so it decreases with more patches (hh-refinement) or richer local spaces (pp-refinement) (Melenk & Babuška, 1996). This gives the practitioner a very strong guideline, which we verified empirically on SmokeGrid, where more patches lead to a lower error than wider patch states at the same budget (Figure 3(b,c), Table 11).
  • The reconstruction error εdec\varepsilon_{\rm dec} measures how well the decoder reproduces this approximation from the latent state. An expressive decoder keeps this error small, which is another strong guideline for practitioners that we verified. For example, we observed that using a patch-conditioned decoder, which increases the expressiveness of the local decoder per patch, achieves a lower rollout error on Cylinder, Airfoil and Teddy (Table 13).
  • The encoding error, εenc\varepsilon_{\rm enc}, is scaled by the exponential factor, telling us that decoding and encoding at every time step accumulates εenc\varepsilon_{\rm enc} at an exponential rate. This observation motivates encoding only once and learning the dynamics and trajectory on the latent space. Figures 5 and 12 show how numerical results support this idea, making SPANS stable while GAOT, which re-encodes every decoded prediction, diverges.
  • The encoding and dynamics errors εenc\varepsilon_{\rm enc} and εdyn\varepsilon_{\rm dyn} come from approximating the initial least-squares fit and the Galerkin velocity in Eq. (8). Only these two errors accumulate over time. This observation guided us to design a multistep loss that mitigates the accumulation error.

4. New results: generalization (S4Hh)

The new Appendix H.6 evaluates SPANS in two settings that go beyond its training data. Tables 15 and 16 and Figure 12 report the results, and the rollouts can be visualized in the new supplementary page rebuttal.html:

  • For cross-resolution (Table 15), we train SPANS on SmokeGrid at 1282128^2, 64264^2 and 32232^2 and run inference at 2562256^2. Trained at half the resolution (1282128^2), SPANS keeps an MSE close to that of the model trained at 2562256^2 (1.021.02 against 0.910.91, ×10−2\times10^{-2}), but its fine-scale detail is not the same.
  • For cross-horizon (Table 16, Figure 12), we train SPANS on Airfoil with rollout windows of 10, 20, 29 and 39 steps and run inference for 39 steps. None of the rollouts diverges, although the model trained on 10-step windows becomes less accurate than the others once the rollout passes its window.

5. New results: accuracy and cost across resolutions

Figure 1 (left) now includes a resolution ladder, in which seven methods are trained on SmokeGrid at 2562256^2, 3842384^2 and 5122512^2 over three seeds. SPANS is the most accurate method at every resolution and achieves a new state-of-the-art point on the accuracy-cost Pareto frontier. FNO and LOGLO-FNO are faster, but they have about twice its error and only apply to regular grids, while CALM-PDE, the next most accurate method, is slower and has 1.51.5 to 1.6×1.6\times the error of SPANS.

Main revisions

  • Section 3 derives the Galerkin projection onto the tangent space of the manifold induced by the nonlinear decoder (Eqs. (6) to (8)) and applies it to the partition-of-unity decoder (Eqs. (9) to (12)). The encoder and the latent dynamics are now described as learned approximations of this projection.
  • Section 4 and Appendix B extend Theorem 1 to the nonlinear decoder, with the Galerkin stability assumption stated on the manifold (Assumption 1) and Lipschitz local maps (Assumption 3(ii)).
  • Figure 1 (left) and Section 6 add the resolution ladder with seven methods over three seeds.
  • The patch-count and latent-width study is now in the main text (Figure 3(b,c)).
  • Appendix H.6 is new, with Tables 15 and 16 and Figure 12 on the cross-resolution and cross-horizon experiments, whose rollouts are on the new supplementary page rebuttal.html.
  • Appendix C.1 now defines the short- and long-range attention (Eqs. (62) and (63)).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.