- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to condition transformer-based PDE surrogates on static, spatially varying inputs such as material density, speed of sound, obstacle masks and pore masks. These inputs influence the dynamics but are not changed by them, and the paper calls them one-way coupled or exogenous fields. The proposed method, Patchwise Adaptive Normalization (PAdaNorm), partitions the conditioning fields into the same PxP patches used to tokenize the state. A shared MLP maps each patch to a conditioning vector, and a per-block linear projection turns that vector into per-token scale, shift and gate parameters in the adaLN-Zero style.
Critical strengths
s1. The question this research follows is practical with clear motivation.
s2. The paper itself is clear and easy to follow.
s3. The evaluations are done over three seeds raising confidence on the reported results.
Critical weaknesses
(w1) It is unclear how the novelty of this work goes beyond SPADE, and FlowAR's spatial AdaLN. Although, the adaptation to PDE surrogate seems interesting the methodological novelty of the framework itself is limited.
(w2) Given w1 the finding can still be made more interesting by applying the same technique across a wider range of benchmarks and providing a comprehensive evaluation of this approach on PDE benchmarks and under different methods of solution generation.
(w3) The global-AdaNorm baseline average-pools the conditioning fields to C_cond scalars before the learnable encoder, so on acoustics the model has almost no access to the scattering geometry. This can explain most of the improvement rather than the spatial conditioning. Other works exist without the pooling and with a learnable layer that directly maps the exogenous information to scale and shift without any mean pooling. These are still variations of global-AdaNorm which may help (see SEA [1]).
(w4) GNOT's results suggest that there maybe implementation issues. The results for GNOT is an order of magnitude above every other benchline. This needs clarification, and if the implementation matches the exact implementation bencline then a description to why the authors suspect this is the case.
(w5) The reproducibility does not require the full training infrastructure. The implementation can be simply provided through an algorithm within the paper or even a psudocode in the appendix so it is possible to evaluate the exact conditioning mechanism that is proposed in this work.
[1] Esmati, P., Dadashzadeh, A., Ardakani, V.G., Larrosa, N. and Grilli, N., 2024. SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers. Advances in Neural Information Processing Systems, 37, pp.47317-47343.
Questions for authors
Q1. Can the authors evaluate on additional datasets with static exogenous fields, ideally including a third PDE family beyond acoustics/Navier–Stokes/porous media and at least one of the motivating cases in the introduction (thermal stress, external potential, refractive index), none of which is tested?
Q2. Can the authors add a global-AdaLN control that does not average-pool the raw fields (See w3). A recommendation would be the same per-patch encoder f_θ followed by attention-pooling of the patch embeddings and, separately, in-context concatenation combined with global AdaLN?
Q3. Given the code cannot be released can the authors provide pseudocode for the full PAdaNorm block with explicit tensor shapes at every step? An algorithm or a pseudocode independent of the proprietary training infrastructure will still be needed to reproduce, evaluate and adapt the proposed method.
I am willing to reconsider my score if these questions are addressed with less emphasis on the right away benchmarks since I understand the limited period of rebuttal.
3 comments
I would like to thank the authors for their response.
Novelty and impact: An argument that compares methodologies between this work and other advances in the field irrelevant to this work (ie. ViT, or diffusion) is not helpful for the process of this rebuttal. My weakness 1 was specifically regarding the novelty compared to the most related works here (I specifically named your related works SPADE and FlowAR). Although pure methodological novelty is not a point to alter score on (as pointed in the original review), it is nevertheless important to understand the impact of the work which is something that alters the score.
Remaining issue: Proposed contribution PAdanorm comes with a core claim regarding achieving improved or similar results at fewer FLOPs. However the following issue remain:
-
The impact in its current form is minimal. Regardless of the novelty, the presented results in the main table (Table 1) are all within the noise range across measured seeds with in-context (wide), and the newly added AdaNorm (attn-pool + in-context).
-
The computation improvements (in terms of forward pass FLOPs) is also small compared to in-context (wide), and is in fact worse than the newly added AdaNorm (attn-pool + in-context).
Consequently after carefully reading the rebuttals and authors response to other reviewers I stand by my original score.
Response to Reviewer EbWv
We thank Reviewer EbWv for the follow-up.
We apologize if the comparison to ViT and diffusion models read as a deflection as that was not our intent, and we agree it was an unhelpful analogy. The point we wanted to make concerns SPADE and FlowAR specifically: spatially adaptive modulation is a known primitive, and our contribution is not the primitive itself but its adaptation to autoregressive PDE surrogates with one-way coupled fields.
A key difference between PadaNorm and SPADE and FlowAR is that in both, the conditioning is part of the sample being generated. In PDE surrogates, the conditioning fields (density, sound speed, obstacle/pore masks) are exogenous: they drive the dynamics but are never changed by them, and they are static over the whole rollout. This is what makes one-way coupling exploitable: PAdaNorm computes the conditioning embeddings and per-block modulation parameters once per rollout and caches them, which has no analog in image generation.
On "results within noise" (Table 1). We thank the reviewer for raising this point.
First, for patchwise vs. global conditioning, the two purely global controls isolate the question the reviewer raises, and here the advantage of patchwise conditioning is consistent. PAdaNorm attains lower long-horizon error than both global AdaNorm and attention-pooled AdaNorm in 5 of 6 settings, with a median reduction of about 23%. The only exception is CO₂, where all methods lie within about 1% of each other. Across seed-matched runs, PAdaNorm is lower in 14/18 and 15/18 cases respectively (Wilcoxon signed-rank, p = 0.001 and p < 0.001).
The attention-pooled + in-context control is not purely global: concatenating the conditioning fields to the input gives it spatially resolved information. It is therefore a hybrid, and its competitiveness supports our central claim that spatially resolved conditioning is the key ingredient. Even so, PAdaNorm attains the best average rank of all eight methods (2.17 vs. 2.67). It is also best on more datasets than any other method: Inclusions, Discontinuous and FlowBench, against Maze and Joint for the control. The control's average vRMSE is lower by only 0.004 (0.480 vs. 0.484), and this comes from a single datasets of the ones that we tested (Maze). Excluding Maze, PAdaNorm's average error is lower by 0.019. A paired Wilcoxon signed-rank test finds no evidence of an advantage for the control (W = 10, p = 1.00 over the six settings; p = 0.93 over 18 seed-matched runs). PAdaNorm thus matches the strongest hybrid control method overall while being the more consistent method across heterogeneous PDE families.
We thank reviewer EbWv for their careful read of our paper and the insightful comments and questions. Let us address each one by one.
Weaknesses
Weakness 1: While we acknowledge that novelty is somewhat a subjective concept, we would like to challenge this statement from Reviewer EbWv. One can always describe ViT are merely an application of transformers to the image domain or Diffusion Models can be considered as applying diffusion methods for image generation, but this would not capture the rationale and work behind the proposed approach.
Weakness 2: We fully agree that increasing the diversity of application domains on which to evaluate PadaNorm would increase the confidence in the benefits of the method. The lack of additional domains in the paper is limited by the scarce availability of open-source dataset that satisfy (i) one-way coupled spatially varying fields (ii) long enough rollouts. We searched for openly available datasets satisfying both requirements and, to the best of our knowledge, the datasets used in the paper are the only ones currently available.
Weakness 3: See our answer to Question 2. We thank the reviewer for the suggestion and have added those experiments in the manuscript.
Weakness 4: We have removed GNOT from the revised manuscript as our adaptation of the original approach to this setup might not be suitable.
Questions
Question 1: As discussed in our answer to Weakness 2, to the best of our knowledge no other open dataset currently satisfies both requirements. We agree that extending the evaluation to further PDE families, such as the motivating cases of the introduction, is an important direction for future work. Would Reviewer EbWv have any dataset suggestion that we would have missed on which we could run some additional experiments?
Question 2: We have now run the experiments and added both suggested controls to Table 1, on all six settings: attention-pooled global AdaLN, and attention-pooled global AdaLN combined with in-context concatenation. Attention pooling alone performs on par with average-pooled AdaNorm (average vRMSE 0.680 vs. 0.665) and remains well behind PAdaNorm on the acoustic datasets. Combined with in-context concatenation, it is a strong alternative: it attains the lowest error on Maze and Joint, and a slightly lower average vRMSE than PAdaNorm (0.480 vs. 0.484). PAdaNorm keeps the best average rank (2.17 vs. 2.67) and the lowest error on Inclusions, Discontinuous and FlowBench. We thank again the reviewer for this suggestion.
Question 3: This is indeed a valid request, and we have updated the manuscript with a pseudocode implementation.