acceptodds
Withdrawn from ICLR 2027

No Free Soup for PINNs: Just Keep the Best

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Averaging the weights of a pool of fine-tuned models into a model soup yields ensemble-level accuracy at the cost of a single inference, a practice well established in vision and language. For Scientific Machine Learning (SciML) and its techniques, such as Physics-Informed Neural Networks (PINNs), this is especially attractive: the result stays a single continuous, differentiable surrogate. We present the first systematic benchmark of weight averaging for PINNs for the forward problem, spanning several PDE families in both soft- and hard-constrained form and a range of widths, in double precision throughout. The odd activations standard in PINNs induce a sign symmetry that permutation matching leaves intact. We therefore enlarge the matching group to , an extension we call Sign Re-Basin. We evaluate it across training regimes, from independently trained ingredients to models that share most of their optimisation trajectory. The output-space ensemble, exact by superposition on a linear PDE, serves as the control that isolates parameter-space from function-space averaging. Using a reference-free selection criterion, the greedy recipe almost never accepts a second ingredient and never a third, and using the reference solution in its acceptance test barely changes this. When a soup does improve on the selected ingredient, the gain is within selection noise, i.e. the gap between the model selected by a reference-free criterion and the best model in the pool. The ensemble stays accurate but loses the main advantage of a PINN over classical mesh-based solvers, a single continuous and differentiable representation, multiplying the query cost by for a variance reduction that model selection already provides. Given a pool of PINNs, just keep the best model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
physics-informed neural networks, model soups, weight averaging, model merging, deep ensembles, linear mode connectivity, permutation symmetry, sign symmetry, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bghXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper is a nice and clear benchmark showcasing that the model soup idea of Wortsman et al., 2022 generalizes to PINNs.

Critical strengths

The benchmarks are complete and thorough, I think it's pretty well-established from this that the model soup idea does indeed generalize to this case. And there are some caveats that are looked into, such as being outside of the saturating regions for many trained networks, which makes it so that it's not 100% obvious that it would work out.

Critical weaknesses

It's not anything new. It's more of a supplemental to Wortsman et al., 2022, it's just that method applied on PINNs. Well written and benchmarked throughly, but there isn't anything new here. It's basically as if an undergrad was shown Wortsman et al., 2022 and was told to now make a PINN version of the paper, and they handed that to Claude, and benchmarks came out that could have just been a section of that paper.

Questions for authors

I'm not sure there are questions. The text is rather clear.

1 comment
AuthorsOpenReview ↗

Thank you for your review

We thank the reviewer for the positive remarks regarding the clarity of the text and the thoroughness of our benchmark.

However, we believe there is a fundamental misunderstanding regarding the core finding of our paper. The review states that our work shows that the model soup idea of Wortsman et al. (2022) "generalizes to PINNs". Our results show precisely the opposite: weight averaging systematically fails for PINNs, whereas output-space ensembles do offer modest gains. This conclusion is supported by several key empirical findings presented in Sec. 3.2 and expanded upon in Appendix D.

Regarding novelty, Wortsman et al. (2022) strictly focus on standard supervised classification, and our finding for PINNs is directly opposite to their conclusions. Beyond delivering a comprehensive benchmark with output-ensemble controls, the manuscript systematic characterizes the root cause of this failure through controlled experiments: it shows that the soup fails most on linear PDEs where output ensembles are exact by superposition, and it evaluates weight averaging under strict control mechanisms—including network width scaling, hard constraints, and symmetry alignment (accounting for odd activations).

Additionally, we could not find any reference to "saturating regions" in our manuscript. If the reviewer could clarify which section or concept they are referring to, we would be glad to address it. We also respectfully request that review feedback remain focused on the technical and empirical content of the work.

Acknowledgments. We thank again the reviewer for their time.

Reviewer_zrgBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether model soups, which have been successful in computer vision and language models, can also be applied to PINNs. The authors test weight averaging across six PDEs, different network widths, and three training regimes. They also introduce Sign Re-Basin, which extends permutation matching to account for the sign symmetry of tanh networks. The experiments show that weight averaging rarely improves upon the best individual PINN, even after alignment, while output-space ensembles generally perform better.

Critical strengths
  1. The paper explores the application of model soups, a technique previously studied in computer vision and language models, to PINNs. This is an interesting direction for examining whether ideas from other machine learning fields can be useful for scientific computing.
  2. The authors conduct an extensive experimental study, covering six PDEs, different network architectures, and training regimes. The results provide a broad assessment of the performance of weight averaging for PINNs.
Critical weaknesses
  1. The paper does not sufficiently explain why model soups work in CV and language models but fail for PINNs. The authors measure loss barriers and parameter misalignment, but these observations do not fully explain what is different about PINN training. A more direct comparison of the loss landscapes, training objectives, or parameter connectivity would help identify the reasons for this difference.
  2. The negative results provide limited guidance on when weight averaging might work for PINNs. The paper shows that several existing averaging strategies fail, but does not investigate how these failures might be avoided. This substantially limits what can be learned from the study and its value for other work.
  3. The main conclusion offers limited novelty beyond existing knowledge of model merging. It is already known that weight averaging can fail when models are not connected by a low-loss path. The experiments confirm that this happens frequently for PINNs, but the paper does not establish a new mechanism specific to physics-informed learning. Without a clearer explanation of why PINNs behave differently, the contribution is largely an empirical confirmation of a known limitation.
Questions for authors

Please refer to the concerns in the weaknesses section.

1 comment
AuthorsOpenReview ↗

Thank you for your valuable review

We thank the reviewer for the careful reading and for an accurate summary of our findings. We also appreciate the recognition of the breadth of the study. Like the reviewer, we also found the application of model soups to PINNs to be a compelling direction, largely because generating and testing a soup is very inexpensive once the individual models are already trained. An initially small exploratory study (we observed that soups failed for various equations) ultimately grew into the comprehensive benchmark presented here.

We now address your concerns:

On why soups fail for PINNs (Weakness 1). The manuscript goes beyond loss barriers and misalignment. On linear PDEs, the output-space ensemble is exact by superposition, yet the soup performs worst there. In the median, the nonlinear distortion ΔNL\Delta_{\mathrm{NL}} accounts for the whole of the soup’s error on every linear operator. This strongly points to the nonlinearity of the parameter-to-solution map θ↦u(⋅;θ)\theta \mapsto u(\cdot;\theta), rather than the PDE equation itself, as a main driver of the failure. Regarding the direct comparison of loss landscape geometry (e.g., curvature) with vision or language models, the reviewer raises an interesting point. Unlike standard classification, PINNs evaluate differential operators over collocation points, which likely alters how parameter space behaves under weight averaging. Inspired by this comment, we will consider how to design a comparison or experiment between these two paradigms (PINNs vs. vision/language) to better understand these geometric differences.

On guidance for when averaging might work (Weakness 2). We agree that the manuscript does not yet provide explicit practical guidance on when weight averaging might succeed for PINNs. However, our systematic evaluation of four key strategies (such as increasing network width, sharing training trajectories, simplifying the optimization landscape via hard constraints, and applying symmetry alignment) strongly suggests that the issue stems from the intrinsic geometry of the loss landscape. Because the parameter-to-solution map θ↦u(⋅ ;θ)\theta \mapsto u(\cdot\,;\theta) enters non-linearly into the residual loss, linear operations in weight space do not map to favorable solutions in function space. Conceptually, if one visualizes the PINN loss landscape as a complex terrain of steep ridges and narrow valleys, taking a weighted average of parameters θˉ\bar{\theta} systematically lands the model in high-loss regions. The very limited number of cases of the opposite phenomenon suggests this is the norm (App. F). This geometric barrier is not an artifact of standard vanilla PINNs alone, but rather an inherent property of the physics-informed framework resulting from the inclusion of network derivatives within the loss formulation. Consequently, our findings suggest that weight-averaging degradation is structural and likely extends across common architectural variants (e.g., both vanilla PINNs and hard-constraint ansatz formulations treated in our work). We plan to revise the manuscript to incorporate these insights and highlight that, within standard physics-informed formulations, weight averaging faces severe geometric hurdles that make successful applications particularly challenging.

On novelty (Weakness 3). We agree that the failure is consistent with known limitations of model merging. Our contribution is to show where and to what extent it manifests in PINNs; thus, we report weight-averaging as another PINN pathology to consider from now on in the literature. Note that it is most severe on linear operators, where output averaging is exact. Furthermore, our benchmark suggests that the non-convex geometry of the loss landscape is such that applying hard constraints and symmetry alignment does not ease the problem of the average landing on a bad minimum. Additionally, this study reinforces the established understanding of loss landscapes through the lens of model merging and weight averaging.

Acknowledgments. We thank again the reviewer for the detailed feedback.

Reviewer_dKAvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work investigates whether model soups, averaging the weights of multiple trained neural networks, can improve the accuracy of physics-informed neural networks (PINNs) without increasing inference cost. The authors conduct an extensive experiment study, analyzing 864 trained networks across 6 PDEs, both soft- and hard-constrained formulations, three widths, and three training regimes. First, the authors observe that odd activations such as tanh, as standard in PINN architectures, requires a slight adjustment to the symmetry group of the permutation-based neuron alignment in Git Re-Basin, and thus propose Sign Re-Basin. Their main finding is that model soups generally fail to improve over the best individual PINN, even after symmetry-aware alignment, whereas output space ensembles can provide modest improvements. The authors further analyze the loss barriers between trained networks to explain the failure of parameter averaging.

Critical strengths
  • The paper's empirical studies are systematic, thorough, and carefully controlled. The authors evaluate a broad collection of PDE problems, architectures, training regimes, and model merging techniques.
  • The authors further examine loss barriers, nonlinear distortion between parameter space and function space-based averaging, and the effects of permutations and sign alignment. These diagnostics offer some intuition as to why straightforward weight averaging performs poorly in the PINN setting.
Critical weaknesses
  • Limited novelty of Sign Re-Basin. The layerwise optimization reduces to a relatively straightforward modification of permutation matching. Moreover, there appears to be prior work [1, Appendix A] which derives a similar signed permutation procedure for weight matching. The authors should clarify the novelty of their proposed algorithm with respect to this work.
  • Despite the number of experiments, the evaluation is restricted to a relatively narrow family of tanh MLP architectures with fixed depth. It remains unclear whether the negative result extends to modern PINN architectures or different optimization procedures. Furthermore, without comparison to standard supervised regression settings, it is unclear whether the observed failures are specific to PINNs or apply broadly to regression-like settings.

[1] Boufalis et al. 2026. Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization

Questions for authors
  • In the shared-training experiments, different branches share 10,000 or 15,000 Adam iterations, but each branch subsequently undergoes its own independent 15,000-iteration L-BFGS phase. Given the importance of L-BFGS in PINN optimization [1, 3], these independent optimization phases could lead to substantially different trajectories, even in the late-branching regime. Do the same conclusions hold when models are branched closer to the end of training, including settings where they share part of their L-BFGS trajectories? Such experiments would more closely resemble the shared optimization history that motivates model soups.
  • To what extent are the observed failures specific to the physics-informed loss, rather than generic properties of neural networks, particularly in regression settings? The authors' proposed explanations involve the nonlinearity of the parameter-to-function map and the presence of loss barriers, both of which also apply to networks trained using standard supervised regression. A comparison using the same architectures trained on supervised regression tasks would help isolate the role of the physics-informed objective.
  • Do the observations continue to hold with more recent PINN architectures, such as PirateNets [2], and stronger optimization methods, such as NNCG [3] and SSBroyden [4]?

[1] Krishnapriyan et al. Characterizing Possible Failure Modes in Physics-Informed Neural Networks. NeurIPS, 2021. [2] Wang et al. PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. Journal of Machine Learning Research, 2024. [3] Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML, 2024. [4] Kiyani et al. Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks. Computer Methods in Applied Mechanics and Engineering, 2025.

1 comment
AuthorsOpenReview ↗

Thank you for your valuable review

We thank the reviewer for the detailed feedback, and especially for the reference to Boufalis et al. We now address your comments and questions:

Novelty of Sign Re-Basin (Weakness 1). We derived our signed LAP reduction (Proposition 5) and the associated weight-matching cost matrix independently, and in parallel to that work. We are still studying the paper, but we can already verify that the algorithm the reviewer highlights is essentially our SignWeightMatching, and that Boufalis et al. also note that it does not converge to a global optimum. To the extent that Appendix A derives the same procedure for weight matching, we acknowledge the overlap and will cite and attribute that part accordingly.

Our contribution is, however, not limited to that step. The manuscript delves into the group theory behind the odd-symmetry alignment problem, providing proofs that build the proposed framework. Through this lens, we justify the importance of this additional symmetry (averaging can erase weights of opposite signs) and propose an extension, SignActivationMatching. Our initial hypothesis was that this destructive interference might explain the persistent failures of model soups. To our surprise, our experiments showed that misalignment is not the main driver: the improvement is nowhere near the order of magnitude of the merging error, even though it is positive compared to Git Re-Basin. Lastly, regarding the blind spot of SignWeightMatching, we propose a multistart procedure with theoretical guarantees (monotonicity, domination over unsigned matching, and exact repair of uniform flips). In a revised version, we will credit the signed LAP reduction to the cited work and center our contribution on these elements and their application to PINNs.

Scope of the evaluation (Weakness 2). We agree that fixed depth limits generalisation, and we do not claim that the negative result extends beyond this setting. Preliminary benchmarks that preceded this paper also swept across depth, yielding similar failures, but we did not include them owing to a different setup (e.g., no double precision). We plan an additional benchmark about network depth with a carefully selected subset of equations and widths to test whether this behaviour is independent of depth.

Regarding modern architectures and optimisers, we focused on the vanilla setting, the most widely used outside the theoretical niche of SciML, where practitioners would benefit most from a successful model soup. Nevertheless, our experiments suggest that the failure of averaging lies in the loss landscape itself: given its extremely complex geometry, it is very difficult for the soup to land in a local minimum superior to the best ingredient.

Questions

1. Branching closer to the end of training. This is a valid point. In late, branches share 15,000 of the 20,000 Adam iterations, but each runs its own L-BFGS phase. We have not run the suggested experiment and cannot claim the conclusions hold there. Our data show that connectivity improves markedly from independent to early to late without reaching linear mode connectivity, which is compatible with the reviewer's hypothesis but does not confirm it. A similar approach was explored in the preliminary experiments, but without an exact 1:1 setup we cannot claim those findings generalize here. Models sharing part of their L-BFGS trajectory ended up as virtually identical clones: this phase is so aggressive that virtually all models finished with decimal-identical losses. Soups could therefore not outperform the best ingredient, and ensembles did not benefit either, as there was no genuine diversity. We plan an additional benchmark on this aspect in double precision.

2. Specificity to the physics-informed loss. We agree that the nonlinearity of θ↦u(⋅;θ)\theta \mapsto u(\cdot;\theta) and the existence of loss barriers are general properties of neural networks. We have no supervised-regression control with the same architectures, so we cannot isolate the role of the physics-informed objective. What we do measure is that the soup fails most where the ensemble is exact by superposition (linear operators), that averaging inflates the boundary residual, and that hard constraints do not remove the failure. Our intuition is that the objective, which introduces derivatives of the network, shapes the complex loss landscape behind this issue.

3. PirateNets, NNCG and SSBroyden. We did not test these; the study is restricted to standard architectures with Adam-to-L-BFGS in double precision, so we make no claim about them. We conjecture that in any framework introducing network derivatives into the loss, model soups will remain challenging, and leave this for future work.

Acknowledgments. We thank again the reviewer for the careful and constructive review, and in particular for pointing us to Boufalis et al. (2026), which we were not aware of.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.