acceptodds
Under review as a conference paper at ICLR 2027

Learning Physics from an Imperfect Ancestor

Abstract

Neural operators evaluate parametric partial differential equations (PDEs) cheaply but degrade sharply outside their training distribution. Physics-informed neural networks (PINNs) avoid dependence on labeled data, yet their optimization can be basin-fragile: when the governing residual admits multiple solutions, a PINN trained from scratch may converge to a physically incorrect state despite achieving a small residual. We show that these failure modes can be addressed jointly: an imperfect NO provides the structural prior needed to place a PINN in the correct solution basin, while the PDE residual refines the solution beyond the operator's accuracy. We introduce a three-stage framework that freezes the spatial basis of a physics-informed NO, extrapolates its solution branch to an out-of-distribution parameter using a polynomial continuation prior, and distills the resulting field into a fresh PINN. The NO (teacher) need not be accurate at the target; it transfers solution-branch information, while PDE residual minimization in the PINN (student) governs convergence. We evaluate the framework on three nonlinear PDEs: 1D viscous Burgers, 2D steady Allen-Cahn near a pitchfork bifurcation, and 2D steady lid-driven cavity flow. For Allen-Cahn, where the trivial solution satisfies the PDE residual exactly, a standard PINN consistently collapses to the trivial zero branch, whereas distillation from the crude extrapolated operator recovers the non-trivial branch that matches the finite-difference reference. For the lid-driven cavity, extrapolating to a Reynolds number of Re = 3200 accelerates convergence to the correct physical state, achieving competitive accuracy using fewer parameters and optimization steps than recent literature baselines. These results establish a simple principle: an NO need not accurately predict the solution to be useful; it only needs to identify the correct basin from which PINN optimization can recover it.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Physics-informed neural networks (PINNs), Neural operators, Knowledge distillation, Out-of-distribution extrapolation, Basin selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_y73eOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a multi-step approach to train PINNs on Parameteric ODEs. First, a Neural Operator (a DeepONet) is trained in a given domain of the parameter space, obtaining initial trunk and branch networks. Then, the trunk is frozen providing a fixed spatial base and, after selecting anchor points and calculating a polynomial interpolation of the branch from these anchors, the working domain of the branch is extended in one or more step using the polynomial extrapolation as starting base. FInally, a PINN is trained on the final parameters of interest, and the solution predicted by the NO is used to distill information about how the solution should look like. Results show improvement compared to a PINN trained on the test parameter alone, especially where there exists a trivial solution or discontinuities.

Critical strengths
  • The overall paper structure is good, the pipeline is well presented in clear steps. Some additional informations can be added in the main texts, but the overall contribution is clear.
  • The results are nice and presented in a clear manner, with visualizations that are helpful to understand the results.
  • The topic of knowledge distillation in PINNs and physics is an interesting topic that is not yet fully-understood. The methodology is sound to provide a good candidate solution out of distribution of the initial training region of the NO.
Critical weaknesses
  • Right now, the paper does not compare to other relevant methodologies for knowledge transfer in PINNs, and meta-learning, multi-head or related approaches for parametric PDEs. For example, [1] studied the problem of transferring physical information between physics-informed models, also showing that derivatives are significant, if not fundamental, quantities. [2] is a meta-learning approach on model weights, while [3] applies a multi-head approach where new heads are added when the model is going too much OOD (with a similar spirit to step 2 of the paper). Acknowledgement and discussion or comparison with these models is suggested.
  • Some of the motivation is not clear: why during step 2 is the NO fully fine-tuned, and only the branch is done? Possibly, fine-tuning the trunk as well might be helpful.
  • In the Allen-Cahn experiment, the PINN is not helped at all to see the lift that brings it to the correct solution, while the model pipeline does, and thus has an advantage. It would be interesting to see how this helps the PINN. This experiment also needs more details on the different possible solutions (the bifurcation).

[1] Trenta et al. "Learning and Transferring Physical Models through Derivatives", TMLR 2026.

[2] Penwarden et al., "A metalearning approach for Physics-Informed Neural Networks (PINNs): Application to parameterized PDEs", Journal of Computational physics, 2023.

[3] Zou et al., "L-Hydra: Multi-head Physics-Informed Neural Networks", 2023.

Questions for authors
  • Why does the model not fine-tune the whole NO, and what is the actual effect of having the anchors?
  • How are the anchors selected? Is there a strategy or by hand with enough points?
  • How does the different schedulers behave on different experiments and how to know which one is appropriate? See also the weaknesses.

Minor things:

  • the plots are very interesting, especially those that put together the schedule with the loss and accuracy to see how they are related. This is done especially well with Allen Cahn where the relation is clear. I would also do this for the others.
  • In the text the authors mainly talk about the accuracy metric (which is also a bit strange than the common ones in literature, but is still ok as it is based on L2. However, it is not clear what Table 2 has inside as metrics and how they are related to the accuracy. Also, the figure is too large.
3 comments
AuthorsOpenReview ↗

y73e (1/2)

W1.

Response: We now discuss these references in Section 2. They address related but different settings. DERL (Trenta et al. (2026)) distills a trained teacher's derivatives into a student to extend a physical model to new domains or parameter ranges while preserving the teacher's knowledge; in our case the teacher is an inaccurate out-of-distribution estimate whose influence is deliberately removed during training. Penwarden et al. (2023) interpolate trained PINN weights across the parameter range to initialize new tasks, which accelerates training within the training distribution. L-HYDRA (Zou & Kardianakis (2023)) learns a shared basis with task-specific heads that are refit for new tasks, a structure related to our frozen-trunk Part 2; notably, the authors report that this approach degrades for out-of-distribution tasks, which is precisely the regime we target. This motivates the continuation prior of Part 2 and the fresh PINN of Part 3. None of these methods targets basin selection.

Changes: Section 2, paragraph "Operator learning" and paragraph "Knowledge distillation", third sentence (DERL).


W2 and Q1.

Response: We keep the trunk frozen in Part 2 for three reasons. First, cost: retraining the trunk requires differentiating the full network (up to fourth order for the cavity) at every step, whereas with a frozen trunk the basis functions and their derivatives are precomputed once, and Part 2 only optimizes the branch. Second, the role of each part: Part 1 builds the set of basis functions, and Part 2 only learns how to combine them at a new parameter. Third, the continuation prior relies on it: the anchors are coefficient vectors with respect to a fixed basis, so the polynomial fitted through them is meaningful only if the basis does not change. Since Part 3 trains a fresh PINN, the limited accuracy of a frozen basis is not a concern: Part 2 only needs to identify the correct basin.

Changes: Section 3.3, last two sentences of the first paragraph.


W3.

Response: We agree that the Part 1 operator benefits from the seed lift, and that adding the lift to the PINN would also prevent collapse. However, the lift requires knowing the branch shape in advance and does not generalize; our point is that any imperfect teacher suffices. To illustrate this, we bypass Parts 1–2 and built teachers with no lift at all, from 20 synthetic measurements of the target with noise of up to 100% of the solution peak: despite relative errors up to 1.9 (farther from the reference than u≡0u\equiv0), all students reach the non-trivial branch with at least 99.6% accuracy, while teachers fitted to the same noise without the solution signal lead to collapse. Regarding the bifurcation: at the target, the equation admits three solutions, the trivial state u≡0u\equiv0 and the pair ±u⋆\pm u^\star that bifurcates at λ1(H)\lambda_1(H) (Appendix A.3); the vanilla PINN converges to u≡0u\equiv0, and the distilled student to u⋆u^\star.

Changes: Section 4.2, first paragraph (the three solutions at the target) and the two sentences before the last (noisy-data teachers); new Appendix F with Table 8.

AuthorsOpenReview ↗

y73e (2/2)

Q2.

Response: The anchors are placed on a uniform (equally spaced) grid over the training range; no special selection strategy is needed. The grids are {0.10,0.15,0.20}×{1,1.5,2}\lbrace 0.10,0.15,0.20\rbrace\times\lbrace 1,1.5,2\rbrace for Burgers, {35,50,65,80}×{1,1.5,2}\lbrace 35,50,65,80\rbrace\times\lbrace 1,1.5,2\rbrace for Allen–Cahn, and Re ∈{100,200,…,800}\in\lbrace 100,200,\dots,800\rbrace for the cavity (Tables 5–7).

Changes: Section 3.2, fourth sentence.


Q3.

Response: The schedule is not a critical design choice in our framework. What the framework requires is that the teacher's weight vanish before training ends, so that the PDE residual governs final convergence; the two schedules are alternative instantiations of this mechanism. For example, swapping the schedule on Allen–Cahn (5 seeds) gives 99.63±0.1099.63 \pm 0.10% with the piecewise-linear schedule used in the paper and 99.91±0.0599.91 \pm 0.05% with the physics-adaptive schedule, both recovering the correct branch in all seeds. In practice, the piecewise-linear schedule is a simple default, while the physics-adaptive schedule avoids choosing a decay length and attenuates the teacher where it is locally inaccurate.

Changes: Section 3.4, sentence before the reference to Figure 1; Appendix B, new paragraph "Schedule swap".


Minor 1.

Response: For Burgers, the schedule is shown in a dedicated panel of Figure 2, next to the loss and accuracy histories. We have now added the distillation weight w(t)w(t) to the accuracy panel of the cavity experiment (Figure 5) in the same way as Figure 3 for Allen–Cahn.

Changes: Figure 5 updated.


Minor 2.

Response: Table 2 reports the relative L2L^2 error ε=∥s^−sref∥/∥sref∥\varepsilon = \lVert \hat{s} - s^{\mathrm{ref}} \rVert / \lVert s^{\mathrm{ref}} \rVert, and the accuracy used in the text is Acc=1−ε\mathrm{Acc} = 1 - \varepsilon (Eq. 3). We now state this explicitly in the caption of Table 2. We have also reduced the size of the figures.

Changes: caption of Table 2

I thank the authors for their responses to my questions and concerns. Practically all of them are now resolved, and I have increased my score, as I think this is a nice work and a topic worth discussing.

I also appreciate the additional experiment on the noisy teacher. One last thing that would make the paper even stronger is the training of a PINN with some knowledge of the branch or the noisy data, but this is not really necessary for my evaluation.

Reviewer_S2PDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles a very interesting problem within the area of deep learning methods for the physical sciences. Within this area the modelling of partial differential equations via so called physics informed neural networks (PINNs) has gained a lot of attention. However, a well-known problem with such networks is that they are difficult to optimize with gradient based methods leading to the unwanted problem that the network can converge to a physically incorrect solution, even though the partial differential loss is small. The authors tackle this problem by developing a method that attaches a stronger prior to enforce the PINN solution to stay within the right solution basin essentially avoiding converging to a physically incorrect solution. What is interesting is that they attach the prior through an operator learning method, using a neural operator. The neural operator doesn’t have to actually predict the solution in anyway but rather simply provide enough structure to identify the right basin. Part of their contribution is the way in which the neural operator can do this.

Critical strengths
  1. The paper is written very well. The authors have explained the problem they are attacking in a very nice and simple way with a good amount of clarity.
  2. The experimental design and choice of experiments is well thought out. Many PINN papers do an excessive amount of partial different equations to show their method works often picking equations that do not accurately highlight what the paper is trying to show. The authors in this paper pick 3 very relevant equations, 1D viscous Burgers, 2D steady Allen-Cahn and 2D steady lid-driven cavity flow. Each equation is well known to be difficult for various PINN methods to find a solution and are perfect candidates to clearly test a methodology as given by the paper.
Critical weaknesses
  1. The papers core idea is that PINNs can struggle to find the right basin potentially leading to the network converging to a physically inconsistent solution. By instilling the right prior the optimizer can find the right basin of convergence. However, the prior is essentially administered through the use of a neural operator itself. In regard to this it would have been good if the authors had some discussion as to why a neural operator can then not be used itself to find the required solution through operator learning. If a neural operator is being used to give a PINN a prior why not set up a method where a neural operator framework does everything without the need of going through the point value learning that a PINN takes. Some discussion of this would have been useful.
  2. In light of the above I would have liked to at least see something by way of a limitation that this method currently works for PINNs but the paper doesn’t produce a methodology that could learn the actual solution operator. It is well known in operator learning that learning the right solution operator is itself a difficult optimization problem with gradient based optimizers and a discussion of whether the papers method could help on this front, at least for physics informed operator learning, would have been nice.
Questions for authors
  1. Your method uses a neural operator to induce a structural prior for the PINN that forces the optimizer to move to the right basin. One can also solve pdes using operator learning via a physics informed neural operator (PINO). Does using a PINO essentially mean you get the structural prior for free? So in some sense while your method is useful for a PINN it doesn’t give anything new for a PINO?
  2. The problem you consider is motivated from the well-known observation that even though pde residual loss can be small the PINN can still converge to a physically incorrect solution. PINNs on different geometries are known to exhibit this problem. Have you tried your method on a PINN where the domain has some non-standard geometry? Might be interesting future work. Have a look at the PINNacle benchmark where some examples are given.
  3. You don’t actually give an argument to the fact that the prior the NO instils pushes the optimization trajectory to the right basin. Is there any way you can show this? Possibly in the neural tangent kernel limit? Though this might be quite difficult.
1 comment

W1 and Q1.

Response: We thank the reviewer for this question, which allows us to clarify the setting. First, our Part 1 operator is already physics-informed (DeepONet trained on the PDE residual), so using the operator on its own is part of our comparisons. In an out of distribution test case, it is inaccurate, with relative L2L^2 errors of 0.70 for Allen–Cahn and 0.53 for the cavity (Table 2), against 0.0037 and 0.066 after distillation. A physics-informed operator does provide the right prior within its training range, but our targets lie outside it, where the prior is not obtained for free. Second, we deliberately do not seek an accurate operator: learning an accurate solution operator, especially out of distribution, is a difficult optimization problem that typically requires extensive tuning, whereas our framework only needs a mediocre one, which is much cheaper to build (about one GPU-hour for the cavity, Appendix H). Third, the main claim of the paper is not the role of the operator, but distillation from an imperfect teacher, whatever its source. Our new Allen–Cahn study (Appendix F) bypasses the operator entirely and uses teachers fitted to a few heavily noisy measurements (up to 100% noise), which still lead every student to the correct branch. Our method is therefore complementary to PINOs: a PINO can serve as the teacher, and the PINN student then removes its out-of-distribution error.

Changes: Appendices F and H and Section 5 (Conclusion).


W2.

Response: This is a fair point. In its current form, the framework produces an accurate solution at a given target rather than a solution operator, so every new target requires running Parts 2 and 3 again (about 3.5 GPU-hours for the cavity, Appendix H); only the base operator of Part 1 is shared. That said, we expect the same idea to be useful for physics-informed operator learning as well. Training such an operator amounts to minimizing the same residual over many parameter instances at once, so wherever the PDE has several solutions, the same ambiguity in basin selection arises there too, and an imperfect teacher could guide training early on in the same way. Another possibility is to feed the solutions obtained in Part 3 back as anchors, gradually extending the range over which the operator itself is accurate. We have not tested either direction yet, but we now mention both as future directions in the Conclusion.

Changes: Section 5 (Conclusion), last two sentences (operator learning as a future direction).


Q2.

Response: We thank the reviewer for this suggestion. In the current work, we partially address geometry by varying the aspect ratio of rectangular domains, including an extrapolation of the cavity from H∈[1,2]H\in[1,2] to H=3H=3 (Appendix E). More complex geometries, such as those in the PINNacle benchmark will be considered in future. We note that the distillation step itself does not depend on the geometry, since the Part 3 student is a standard PINN; only our construction of the teacher in Parts 1–2 relies on a mapping from a reference square. We have added this direction to the Conclusion.

Changes: Section 5 (Conclusion), future directions (non-standard geometries).


Q3.

Response: We thank the reviewer for this question. We have added an argument (Appendix G) based on two classical ideas: the residual loss is locally bowl-shaped around each solution, and predictor–corrector continuation follows a solution branch when its steps are small. Every solution whose linearized operator is invertible is surrounded by a basin in which residual minimization converges to it. The teacher lies in the basin of the correct solution because it starts on the correct branch inside the training box, and Part 2 follows this branch by predictor–corrector continuation: as in classical continuation methods, small steps keep each prediction within the basin of the correct solution. This relies on two assumptions, which we state as limitations in Appendix G: the anchors in the training box must lie on the intended branch, and no fold or bifurcation may lie between the training box and the target; otherwise, the continuation would follow the wrong branch. The distillation term then brings the student into this basin, and once its weight vanishes, residual minimization converges to the correct solution regardless of the teacher's error. The argument describes the evolution of the solution field, which is a good approximation for wide networks in the neural tangent kernel regime. This agrees with the ablation of Section 4.4, where a single long extrapolation fails and the multi-stage continuation succeeds.

Changes: new Appendix G; Section 4.4, last sentence.

Reviewer_CHNQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a three-stage framework for solving nonlinear parametric PDEs outside the training distribution of a neural operator (NO). The central hypothesis is that an inaccurate NO can nevertheless provide sufficient structural information to guide a physics-informed neural network (PINN) toward a desired solution branch, after which physics-based optimization can substantially improve its accuracy.

The proposed approach first trains a physics-informed NO and freezes its spatial basis. It then extrapolates the branch coefficients using a polynomial continuation prior, refined by PDE-residual minimization in the frozen basis. Finally, the extrapolated solution is distilled into a fresh PINN using a progressively vanishing distillation loss.

The approach is evaluated on 1D Burgers, 2D steady Allen–Cahn near a bifurcation, and 2D lid-driven cavity flow at Re = 3200. The experiments show substantial improvements over from-scratch PINNs, particularly for Allen–Cahn, where the baseline converges to the trivial solution. A cavity-flow ablation further demonstrates the importance of the continuation stage.

Critical strengths
  1. An interesting distinction between structural information and predictive accuracy.

The observation that an imperfect teacher can guide optimization toward a desired solution branch, even when its predictions are inaccurate, is conceptually valuable. The Allen–Cahn experiment provides a particularly interesting illustration: an approximate teacher selects a nontrivial branch, which the student subsequently refines through PDE-residual minimization. The progressively vanishing distillation loss is also a reasonable mechanism for preventing teacher inaccuracies from permanently constraining the student.

  1. Substantial improvements over the chosen PINN baselines, supported by a continuation ablation.

The experiments cover several nonlinear PDEs, and the proposed framework substantially improves accuracy and robustness across the reported random seeds. In particular, the cavity-flow ablation shows that directly evaluating the pretrained operator outside its training range is substantially less effective than the proposed continuation strategy. This provides evidence that continuation plays an important role in the method's success.

Critical weaknesses
  1. The specific contribution of neural operator learning is insufficiently isolated from continuation, initialization strategies, and modern PINN optimization methods.

The framework combines a learned spatial basis, polynomial continuation, physics-based correction, and guided PINN optimization. These mechanisms may provide advantages independently of neural operator learning. In particular, the absence of competitive continuation-based PINN or numerical warm-start baselines makes it difficult to establish whether the learned operator contributes something beyond an appropriate parameter-dependent initialization. This concern is especially relevant for the cavity experiment, where eight finite-difference reference solutions are already used to construct the continuation anchors. A classical reduced-basis method using the same information would be a natural comparison.

Furthermore, the paper largely overlooks recent developments in natural-gradient and Gauss–Newton optimization for PINNs, which have demonstrated substantial improvements over conventional optimization methods (Müller & Zeinhofer, ICML 2023; Schwencke & Furtlehner, ICLR 2025; Best McKay et al., ICML 2025; Webb et al., arXiv 2026). In particular, Webb et al. introduce DSGNAR, a scalable Gauss–Newton framework reporting major improvements in accuracy and computational efficiency on challenging PDE benchmarks, including Burgers and Navier–Stokes.

Although the authors use Gauss–Newton to refine the Allen–Cahn operator coefficients, they do not investigate such methods for the final PINN solver. This omission matters because the paper attributes its improvements partly to overcoming PINN optimization difficulties. While improved optimization does not necessarily resolve ambiguity between distinct PDE solution branches, the current comparisons do not adequately distinguish failures caused by ill-conditioning from those attributable to incorrect branch selection.

  1. The information requirements, computational cost, and problem-specific priors complicate the interpretation of the results.

The complete framework involves operator pretraining, continuation, and PINN refinement. For the cavity experiment, the continuation additionally relies on eight finite-difference reference solutions. Although these costs could be amortized over multiple target parameters, the paper does not establish when the complete procedure becomes computationally advantageous. The comparison with PirateNet focuses mainly on the student stage: the full pipeline requires 546K optimization steps, and the student alone uses substantially more point evaluations than PirateNet. Consequently, the reported reduction in student optimization steps does not establish an end-to-end efficiency advantage. The cavity experiment also employs regularized lid boundary conditions, which differ from the classical benchmark configuration and should be accounted for in comparisons.

Additionally, the interpretation of the Allen–Cahn experiment requires some care. The operator incorporates an explicit positive "seed lift" proportional to the first Dirichlet eigenfunction, together with an amplitude-floor regularizer designed to discourage convergence toward the trivial branch. Therefore, significant information about the desired solution branch is already encoded in the teacher's construction. Without a comparison against a PINN supplied with comparable branch information, it remains unclear how much of the improvement can specifically be attributed to neural operator learning.

Questions for authors
  1. [Main concern: attribution and novelty] Could the authors compare their approach against classical parameter continuation followed by PINN fine-tuning, or against direct transfer from PINNs trained at nearby parameter values? For the cavity experiment, a reduced basis constructed from the same eight finite-difference anchor solutions would be a particularly informative baseline. How much of the improvement specifically requires a learned neural operator?

  2. [Main concern: optimization baselines] Could the authors compare their framework against PINNs trained using modern natural-gradient or Gauss–Newton optimization methods, such as those proposed by Müller & Zeinhofer (2023), Schwencke & Furtlehner (2025), or Webb et al. (2026)? In particular, can the authors distinguish errors resulting from insufficient residual optimization from failures genuinely caused by incorrect solution-branch selection? Does the distillation mechanism retain a substantial advantage when combined with these stronger optimizers?

  3. [Clarification: branch information] In the Allen–Cahn experiment, the operator employs an explicit positive eigenfunction "seed lift" and an amplitude-floor regularizer. Could the authors isolate their contributions and compare against a PINN initialized or guided using comparable information about the desired branch? This would help determine how much of the benefit comes from the learned teacher rather than the explicitly supplied structural prior.

  4. [Main concern: computational efficiency] Could the authors provide end-to-end computational costs, including the finite-difference anchors, operator pretraining, continuation, and distillation, ideally in wall-clock time and computational resources? For approximately how many target solves does pretraining become advantageous? Could the authors also clarify the impact of the regularized lid boundary conditions on comparisons with existing cavity-flow benchmarks?

2 comments
AuthorsOpenReview ↗

CHNQ (1/2)

W1 and Q1.

Response: We do not claim that a neural operator is strictly needed, or that it is the best possible teacher. Our main claim is the distillation mechanism: an imperfect teacher, whatever its source, is sufficient to place the PINN in the correct basin, after which residual minimization governs the final accuracy. The operator with continuation is one way to build such a teacher; a reduced basis constructed from the eight cavity anchors, or a solution at a nearby parameter, would be valid teachers within the same framework. A PINN on nearby values would be a rather inefficient choice, as nearby training would be required every time. Our new Allen–Cahn study (Appendix F) illustrates this: teachers fitted to a few heavily noisy measurements (up to 100% noise), without any operator, lead every student to the correct branch. The operator, nevertheless, has its own advantages: for example, in Burgers and Allen–Cahn, it is trained from the physics residual alone, without any labeled data; and its pretraining is reused for every target (Appendix H). A systematic comparison of other teacher sources beyond neural operators, including reduced-basis models, is now listed as future work.

Changes: Section 5 (Conclusion), fourth sentence: other teacher sources within the same framework.


W2 and Q2.

Response: To separate insufficient residual minimization from basin selection, we repeated the Allen–Cahn experiment with the same student and initialization, replacing Adam by a damped Gauss–Newton (Levenberg–Marquardt) optimizer with exact Jacobians (new Appendix I, 5 seeds). The Gauss–Newton baseline converges within about ten iterations to u≡0u\equiv0 in every seed, with a PDE loss of order 10−1110^{-11}, below that of the distilled student (order 10−910^{-9}): the residual is minimized to high precision, but at the trivial solution. With distillation, the Gauss–Newton student reaches the non-trivial branch in every seed. Such optimizers can influence which minimum is reached (McKay et al., 2025; Webb et al., 2026), but this choice is guided by the conditioning of the loss: when the PDE admits several exact solutions, each of them is a local minimizer of the residual loss, and the loss alone does not indicate which one is the correct solution of the problem. The teacher supplies this information and can be combined with any of these optimizers. We have added these references to the related work.

Changes: Section 2, paragraph "PINNs and their failure modes" (curvature-informed optimizers); Section 4.2, last sentence; new Appendix I with Table 10.

AuthorsOpenReview ↗

CHNQ (2/2)

W3 and Q4.

Response: We now report end-to-end costs for the cavity at Re =3200=3200 on a single NVIDIA H200 GPU (new Appendix H): Part 1 (operator pretraining, including the eight anchor solves) takes 1.05 GPU-hours and is computed once, Part 2 (continuation) 0.26 GPU-hours, and Part 3 (distillation) 3.2 GPU-hours. The first target thus costs 4.5 GPU-hours and each further target 3.5, since Part 1 is reused.

PirateNet does not report its training time. Compared with our student, it uses a larger network (1.3M vs. 297K parameters) and more training steps (6.3×1056.3\times10^5 vs. 2.5×1052.5\times10^5), but fewer points per step. Combining these, we estimate its training cost at about 2 times that of our student, i.e., roughly 9 GPU-hours on the same hardware, so pretraining is already advantageous for the first target. In addition, PirateNet repeats a Reynolds-number curriculum (from Re =100=100) for every new target, whereas our Part 1 is computed once.

Regarding the regularized lid: with our finite-difference solver on the same grid, the regularized and uniform lids give centerline velocity profiles that differ by 2.4% (relative L2L^2) at Re =3200=3200; part of our error against the uniform-lid benchmark thus reflects the boundary condition.

Changes: new Appendix H with Table 9; Section 4.3, last sentence of the paragraph before Table 3 (wall-clock cost); Appendix A.4, paragraph "Regularized lid profile", last sentence (effect of the lid); Table 7 (boundary points per wall).


W4 and Q3.

Response: Our contribution is the distillation idea: an imperfect teacher, even with low accuracy, can be enough to place the PINN in the correct basin, as observed in all our experiments, after which residual minimization improves the accuracy. Allen–Cahn is used as a representative example because basin selection is clearest there (a vanilla PINN collapses to u≡0u\equiv0 in every seed); our main benchmark is the cavity. Adding the seed lift directly to the PINN would also prevent the collapse, but seed lifting is not a general strategy: it requires knowing the shape and sign of the branch in advance, which is not available for most problems (e.g., the cavity).

To support this, we bypassed Parts 1–2 entirely and built teachers with no seed lift or amplitude floor: small MLPs fitted to 20 synthetic measurements of the target with noise of up to 100% of the solution peak. These teachers are farther from the reference than u≡0u\equiv0 itself (relative L2L^2 error up to 1.9), yet every student reaches the non-trivial branch with 99.6–99.9% accuracy, whereas teachers fitted to the same noise without the solution signal lead every student to collapse. A teacher thus needs neither accuracy nor problem-specific priors, only information about the solution.

Changes: Section 4.2, the two sentences before the last (noisy-data teachers); new Appendix F with Table 8; Appendix D, first sentence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive comments, which have helped us improve the paper considerably. Below we summarize the main changes. Detailed answers follow in a separate reply to each reviewer, where related weaknesses and questions (e.g., W1 and Q1) are answered together (or separately in some cases), and each answer ends with the list of corresponding changes in the manuscript. All changes in the revised manuscript are marked in blue (please download the revised PDF), and appendix letters refer to the revised version.

Clarified contribution. The central claim of the paper is distillation from an imperfect teacher: a teacher that is far from accurate is sufficient to place a PINN in the correct basin, after which residual minimization determines the final accuracy. The physics-informed operator with continuation is one way to construct such a teacher, not a requirement. We have revised the Conclusion to state this explicitly, together with the limitations of the current method.

New studies and analyses.

  • Appendix F: teachers built from noisy data. On Allen–Cahn, we bypass Parts 1–2 entirely and distill from teachers fitted to a few heavily noisy measurements. Even teachers farther from the reference than u≡0u\equiv0 lead every student to the correct branch, while teachers fitted to the same noise without the solution signal do not.
  • Appendix G: why the teacher selects the correct basin. An argument, with its assumptions and limitations, for why the teacher lies in the basin of the correct solution and why the student then converges to that solution.
  • Appendix H: end-to-end computational cost of all stages for the cavity, including the finite-difference anchors.
  • Appendix I: Gauss–Newton optimizer. On Allen–Cahn, a Gauss–Newton baseline still converges to u≡0u\equiv0 with a near-zero residual, while the distilled student reaches the correct branch, separating basin selection from insufficient residual minimization.

Other changes. Extended related work (knowledge transfer, meta-learning and multi-head PINNs, curvature-informed optimizers, PINNacle); the effect of the regularized lid is quantified; the distillation weight is added to the cavity training curves; the rationale for the frozen trunk and the placement of anchors are stated; the caption of Table 2 is clarified; and limitations and future directions are added to the Conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.