acceptodds
Under review as a conference paper at ICLR 2027

Misspecified Physics Can Help and Bias Extrapolation in Hybrid Models

Abstract

Hybrid models that combine physical equations with a data-driven component are expected to extrapolate more reliably than purely data-driven models as the physical component remains active beyond the training data. This study investigates whether a misspecified physical component can still provide useful extrapolative structure beyond the training data and whether its misspecification shapes the extrapolation error of the corresponding hybrid model. In a controlled reaction-diffusion system, this study compares a hybrid model with correct diffusion, hybrid models whose supplied diffusion is misspecified in its directional dependence, its discretization, its magnitude, or by omitting one process, and a purely data-driven model. All models achieve small held-out prediction errors and give similar responses to patterns well represented during training, yet their responses differ when extrapolating to patterns scarcely represented in the training data. For scarcely represented patterns, the hybrid model with correct diffusion closely follows the true response, and moderately misspecified physics gives lower errors than the purely data-driven model, while sufficiently severe misspecification loses that advantage. Hybrid models with misspecified physics carry the specific response errors of their supplied physics into extrapolation, with nearly the same magnitude and response pattern. Intervention analysis attributes the extrapolation error mainly to the misspecified physical component, as replacing the misspecified physics with the correct physics, without retraining the data-driven component, removes most of the error. A misspecified physical component can therefore provide useful extrapolative structure while also imposing a systematic extrapolation bias that follows the response error of the supplied physics.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
scientific machine learning, hybrid modeling, extrapolation, model misspecification, dynamical systems, reaction-diffusion systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gnvxOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how errors in the supplied physical component of a hybrid model (known physics plus a neural network) affect its extrapolation accuracy.

Theoretically, for a linear additive hybrid model, it shows that the difference between models trained with different physical components persists after training along directions with little variance in the training data.

Empirically, it supplies four misspecified diffusion terms to a hybrid model of a 2D FitzHugh–Nagumo system and evaluates its extrapolation performance over a single prediction interval:

  • While all models show similar held-out errors, hybrid models outperform the purely data-driven model on scarcely represented patterns, even with misspecified physics.
  • Response differences between trained hybrids closely follow the differences between their supplied physics in magnitude and direction.
  • Replacing the misspecified physics with the correct one, with the trained network fixed, removes most of the error on scarcely represented patterns but increases it on well-represented ones.
  • Sufficiently overestimated diffusion removes the advantage over the purely data-driven model.
Critical strengths
  • Carefully controlled experimental design. The ground-truth dynamics are known, misspecifications are injected by type, and initialization and minibatch order are paired across conditions. Combined with the post-training replacement of the physical component, this allows the extrapolation error to be attributed to the supplied physics rather than to training noise.
Critical weaknesses
  • Limited practical and theoretical takeaways. The result of Proposition 1 is expected: in an additive hybrid, the network cannot correct the physics along directions that the training data barely cover. The replacement experiment, which shows opposite error behavior on well- and scarcely represented patterns, ultimately implies only that the network should be trained once the physical component is finalized. Section 4.4 identifies where the hybrid loses its advantage in this specific setup, but offers limited practical guidance on how accurate the supplied physics needs to be for hybrid modeling to be beneficial.
  • Limited experimental scope. All experiments use a single PDE system, and the evaluation covers only a single short prediction interval, so it is unclear whether the conclusions hold for other systems or over longer rollouts.
Questions for authors
  1. Can the authors empirically validate Proposition 1 in a controlled linear setting?
  2. Is there any reason why the paper considers the FitzHugh–Nagumo PDE?
  3. Beyond the diagnostic in Section 5, how can practitioners use these findings when the true physics is unknown?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer gnvx (Critical Weaknesses 1)

Dear Reviewer gnvx,

We sincerely thank you for your time and effort in reviewing our work. We address your points below.

CW1. Limited practical and theoretical takeaways. The result of Proposition 1 is expected: in an additive hybrid, the network cannot correct the physics along directions that the training data barely cover. The replacement experiment, which shows opposite error behavior on well- and scarcely represented patterns, ultimately implies only that the network should be trained once the physical component is finalized. Section 4.4 identifies where the hybrid loses its advantage in this specific setup, but offers limited practical guidance on how accurate the supplied physics needs to be for hybrid modeling to be beneficial.

Thank you for raising this. We would like to state our takeaways more directly.

For your concern about "limited theoretical takeaways", we summarize our two theoretical results below. Simply put, Proposition 1 explains why training coverage (partially unseen or totally unseen) matters. Proposition 2 and its related equations explain that, besides training coverage, how the neural component shares its weights also matters.

More specifically, for two hybrid models trained on the same data but given different physical terms,

  • Proposition 1 tells us how training changes the difference between two hybrid models given different linear physical terms. If a pattern is totally unseen during training, the difference between the supplied physical terms remains unchanged in the two hybrid models' responses. Along a pattern that is only weakly represented, training can reduce that difference, but the reduction can be slow. Or say, "in an additive hybrid, the network cannot correct the physics along directions that the training data barely cover", yes, your summary is accurate. We just want to mention here, Proposition 1 describes the limiting case without weight sharing (e.g., a fully connected or linear regression correction), where every weight of the correction is trained separately.

  • Proposition 2 (Appendix B) and its related equations in Appendix B show that, when a network shares its weights across locations (as convolutional networks do in practice), learning from patterns it sees can also change how it responds to patterns it does not see. The analysis gives a formula for how much of the difference between the two supplied physics terms carries into the response of hybrid models.

For "the network should be trained once the physical component is finalized", we understand your concern as: our conclusions seem to apply only when the physics is already well known. In practice, the physics is rarely finalized, and this is what motivated our study. In many scientific fields, legacy or process-based models combine true physics (e.g., energy balance, mass balance) with empirical equations (e.g., from laboratory work or statistical fits of field data), and those empirical equations may no longer hold under new conditions. Some researchers therefore replace the uncertain or empirically represented process with a neural component added to the physical model, which forms an additive hybrid (other forms exist, such as differentiable models or networks that adjust parameters of a process-based model, but the additive form is relatively more commonly used). In such a hybrid model, the supplied physics is never finalized or perfect, and our study asks exactly what its errors do to the hybrid model. The replacement result then tells researchers what to do when they later improve the physics: retrain the hybrid model, because the neural component has adapted to the old physics.

For your concern about Section 4.4, we see its result as an important practical warning rather than only a limitation of our setup: imperfect physics helps a hybrid model only up to a point, and beyond that point it makes the hybrid model worse than a data-only model. We understand that the natural next question is how a researcher can know whether the physics is slightly or severely wrong. First principles, such as mass and energy balance, can be trusted. But it still be unsure whether an empirical equation holds under the conditions where predictions are needed. For empirical equations, a researcher should check their assumptions behind that equation and compare models with and without it against independent observations under those conditions, where available. A data-only model avoids inheriting the error in the supplied equation (particularly for data-rich but partially known physics case), but that alone does not establish that its predictions are more accurate. When the physics of a process is truly unknown, it usually means the process has been observed too little; then neither a data-only model nor a hybrid model can be trusted, and the priority should be collecting more data under those conditions, since any model is built on observations.

AuthorsOpenReview ↗

Reply to Reviewer gnvx (Critical Weaknesses 2 & Questions For Authors 1 3)

CW2. Limited experimental scope. All experiments use a single PDE system, and the evaluation covers only a single short prediction interval, so it is unclear whether the conclusions hold for other systems or over longer rollouts.

Thank you for pointing out these two limits.

For "a single PDE system", yes, we agree; we explain this choice under Q2 and in our general comment, and whether our findings hold in other systems remains open. Within this system, if you are interested in, we now go beyond the diffusion error: we tested two errors in a nonlinear reaction term (details in our reply to Reviewer sdRK, Q4).

For "longer rollouts", we now extended the evaluation to 50-step forecasts, each step using its own previous prediction as input. Compared with the solution of the true equations, the hybrid model with incorrectly directed diffusion had lower prediction error than the data-only model at every step in all 5 runs (at step 50, the mean error was 0.00436 for the hybrid model and 0.01748 for the data-only model; more details in our reply to Reviewer sdRK, Q3).

For the extremely unseen case (the magnitude of held-out states is much larger than any seen in training), the hybrid model with a wrong coefficient in the reaction term gave lower forecast error than leaving the whole reaction to the neural component (all 5 runs through step 20). This advantage did not hold for the term with the wrong shape under the tested training budget (more details please see our reply to Reviewer sdRK, Q4). So the forecast advantage can last beyond one step, but how consistently it lasts depends on the case.

Q1. Can the authors empirically validate Proposition 1 in a controlled linear setting?

Yes, thank you for this suggestion. We now empirically validated our Proposition 1 as you suggested.

We trained two hybrid models with unrestricted linear corrections on the same inputs and targets (the two models differed in the supplied linear physical term). After training, we measured how each hybrid model responded to the same test direction and compared the difference between those responses with what Proposition 1 predicts. Along a direction absent from the training inputs, the entire supplied difference remained, to numerical precision. Along a weakly represented direction, 62.4% remained after 300 updates, matching the predicted value by Proposition 1 (Eq. 21). The test therefore confirms both parts of the result:

  1. training leaves the supplied difference unchanged along an absent direction,
  2. while it can reduce the difference along a weakly represented direction.

Q2. Is there any reason why the paper considers the FitzHugh–Nagumo PDE?

Yes. We chose the FitzHugh-Nagumo system because it lets us introduce a known error in one physical term at a time and evaluate what happens against the solution of the true equations.

Its two variables have different diffusivities and interact through nonlinear reactions. We can therefore change the magnitude or direction of diffusion, or omit diffusion in one variable (with keeping the reaction equations unchanged). We can also do the reverse (keep diffusion correct and introduce a known error in the reaction term).

This separation is what we need to ask whether an imperfect term still helps forecasting and how much of its error remains after training.

Also, it is an established test case for hybrid models with incomplete physics (e.g., APHYNITY; Yin et al., 2021).

Q3. Beyond the diagnostic in Section 5, how can practitioners use these findings when the true physics is unknown?

The practical decision is whether to keep an uncertain equation (we assume at least we have some empirical equations derived from previous observations) or leave that process for the neural component to learn. The pure physical model can be examined first, keeping the first principles (e.g., mass and energy balance), and for each empirical equation, checking its assumptions and the conditions it was derived from (since it is more likely to be badly wrong outside those conditions).

If the relevant observations are scarce, forecasts from different plausible equations can help show where to collect more data. If the models fit existing observations similarly but disagree under the conditions of interest, those are useful conditions to target for new observations. If the forecasts agree, they may still share an error.

In either case, when the available observations cannot establish which model predicts reliably under the intended conditions, collecting observations there takes priority over choosing a model.


Thank you again for your careful review! We hope these additions address your concerns.

I thank the authors for the detailed responses and the additional experiments. My overall assessment is as follows. The paper presents propositions and observations whose conclusions are largely predictable. They neither reveal unexpected behavior nor change how practitioners decide whether and how to use a hybrid model.

  • The exact decay laws in Propositions 1 and 2 are a contribution, but their qualitative conclusion (the network cannot correct the physics along directions that the training data barely cover) is expected.
  • The replacement experiment shows that a physics update without retraining improves the error along data-scarce directions and worsens it along data-sufficient ones. The practical implication is to retrain after every update to the physics, which the rebuttal itself confirms.
  • The observation that a severely wrong physics prior degrades accuracy is also expected. The verification process proposed in the rebuttal (trust first principles, check the assumptions behind empirical equations, compare against independent observations) does not depend on the paper's findings, and the paper does not change or refine it.

Status of each point

  • CW1: Not addressed. I appreciate the clarification of the contributions, but the concern remains for the reasons above.
  • CW2 and Q2: Addressed. I thank the authors for the longer-rollout experiments. Given the nature of the conclusions, I do not expect them to change substantially for other PDE systems, so I no longer consider the single-system scope a major concern.
  • Q1: Partly addressed. I thank the authors for the experiment. However, Proposition 1 predicts an exponential decay of the discrepancy over training. A single value after 300 updates does not verify this. Decay curves over training steps for several levels of coverage, overlaid with the theoretical prediction, would.
  • Q3: Not addressed. The conclusion that more data should be collected where models disagree can be reached without the observations or analysis in this paper.

Suggestions

I believe the following would make the work considerably more useful to the community:

  1. Consider a setting where several terms of the PDE are wrong and are corrected one at a time. Showing how a previously trained hybrid model can be reused so that both data-scarce and data-sufficient directions improve without full retraining, or at least without a trade-off between them, would turn the replacement result into actionable guidance.
  2. Provide a theoretical guideline, even for a simplified model, on how large the physics error can be before the hybrid model loses its advantage over a data-only model.
Reviewer_sdRKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether errors in supplied physics remain visible after training a hybrid model, especially along directions poorly represented in the training data. It gives an exact result for an idealized linear model and studies a FitzHugh–Nagumo reaction diffusion system. The main empirical finding is that hybrid-response differences closely follow the corresponding diffusion differences. Replacing the misspecified diffusion while keeping the learned correction fixed removes much of the response error on the selected high-frequency patterns, but makes responses worse on well-represented patterns. The experiments use four diffusion misspecifications and five paired training seeds.

Critical strengths
  1. The strongest part is that the authors do not stop at showing an accuracy gap. They measure the response differences and then replace one component while leaving the other fixed. On the principal high-frequency patterns, this removes approximately 88%–98% of the response error. The fact that the same replacement hurts on well-represented patterns makes this result more convincing to me: it is consistent with compensation being learned where the data support it.
  2. I also appreciate the reporting of unsuccessful representation-based predictions in Appendix H. The authors could have presented a cleaner story by leaving these out, but retaining them makes the limits of the explanation clearer.
Critical weaknesses
  1. My main hesitation is that I cannot tell how much of the result goes beyond explicit diffusion dominating modes that the network has barely seen. Training starts from low-frequency fields, while the main tests use much higher frequencies. Those directions have both little training support and strong diffusion responses. These factors are not separated. Appendix G makes this concern more concrete: a diffusion-only calculation already reproduces most of the severity curve. This does not make the empirical result wrong, but it makes the additional insight from learning less clear to me.
  2. There are four misspecifications, but all concern linear diffusion in the same system. The accuracy comparisons are mainly local directional responses over one short interval, and all five training seeds share the same trajectory datasets. I would not demand a large benchmark suite, but I need some evidence that the conclusion is not specific to this operator and probe construction before treating it as a sufficiently strong general contribution.
Questions for authors
  1. Do you have a comparison that changes training excitation of the same probe while keeping its frequency and physical-response difference fixed? This would address my main concern much more directly than adding more high-frequency probes.
  2. How much do the conclusions change with longer training or validation-only tuning of the data-only baseline? I am particularly interested in whether the reported crossing is robust, rather than its exact location.
  3. Do the finite-difference checks establish only agreement with automatic derivatives, or do you also observe the claimed benefit in finite-amplitude perturbed predictions?
  4. A nonlinear physical misspecification would be a useful extension, but I view that as a suggestion for strengthening the work rather than something the authors must complete during rebuttal.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer sdRK (Critical Weaknesses 1 & 2)

Dear Reviewer sdRK,

We sincerely thank you for your careful and constructive review.

CW1. My main hesitation is that I cannot tell how much of the result goes beyond explicit diffusion dominating modes that the network has barely seen. Training starts from low-frequency fields, while the main tests use much higher frequencies. Those directions have both little training support and strong diffusion responses. These factors are not separated. Appendix G makes this concern more concrete: a diffusion-only calculation already reproduces most of the severity curve. This does not make the empirical result wrong, but it makes the additional insight from learning less clear to me.

Yes, we agree. In the original tests, every fine-scale pattern was both rarely seen in training and strongly affected by diffusion. Both factors could help explain why most of the diffusion error remained during extrapolation. So the original results could not tell whether this came from what the neural component did or did not learn, or from diffusion alone (as the diffusion-only calculation suggests).

To separate them, we ran the suggested comparison, in which the training data change (we added one fine-scale pattern to part of the training inputs) while the spatial frequency and the supplied diffusion difference stay fixed (details under Q1). With the diffusion error unchanged, the difference between the responses of the two hybrid models at a given pattern depended strongly on whether that pattern was included in training. When the hybrid model can see samples of pattern A during training, the projection coefficient β\beta measured at pattern A (it measures for pattern A, how much of the diffusion mistake is still visible in the trained model) fell from 0.94 to 0.07. When the hybrid model can see samples of a different pattern B with the same spatial frequency, β\beta measured at pattern A fell only to 0.83. In other words, the neural component compensated the diffusion error mainly at the pattern included in training. On the patterns that were poorly represented in training, the hybrid model kept most of the diffusion error, close to what the diffusion-only calculation predicts, which supports our main claim: where training data are scarce, a hybrid model largely inherits the error of its supplied physics in extrapolation.

CW2. There are four misspecifications, but all concern linear diffusion in the same system. The accuracy comparisons are mainly local directional responses over one short interval, and all five training seeds share the same trajectory datasets. I would not demand a large benchmark suite, but I need some evidence that the conclusion is not specific to this operator and probe construction before treating it as a sufficiently strong general contribution.

We have now added a nonlinear case in the same system, keeping diffusion correct and testing two errors in the cubic part of the reaction term (details under Q4).

For your concern about "mainly local directional responses over one short interval", we added finite perturbations and forecasts of up to 50 steps (details under Q3).

For the 5-seed-but-share-one-dataset point, thank you for bringing this to readers' attention. We originally planned to generate 5 separate datasets using 5 different seeds, but computational constraints forced us to change this strategy, which is a limitation of our study (as stated in our Discussion). Nevertheless, we chose to use 5 seeds with different initializations and minibatch orders to examine robustness, which was also motivated by a practical consideration in our field: different initializations and minibatch orders can sometimes lead to substantially different results. As you may notice from the detailed description in Q1, one seed indeed behaved quite differently in the pattern A experiment. So our 5 runs show how robust our results are to initialization and minibatch order.

For your concern about "not specific to this operator and probe construction", to be honest, all tests (the original and the newly added) remain within one system, so they do not establish whether the findings carry over to other systems. As explained in our general comment, we chose this system so that we know exactly which physical term we changed and by how much. Within this system, however, we did go beyond the diffusion operator and the one-step probe now. We tested an error in the cubic reaction term (details under Q4), and checked our results with finite-size perturbations and forecasts of up to 50 steps (details under Q3). The two kinds of error we tested, in how a quantity spreads through space (diffusion) and in what happens locally at each point (reaction), are basic parts of models in many fields, such as hydraulic diffusivity in aquifer models, solute diffusion coefficients, or reaction rate laws. In the revised submission, we try to link our discussions to these settings to enhance the informativeness of our study.

AuthorsOpenReview ↗

Reply to Reviewer sdRK (Questions For Authors 1 & 2)

Q1. Do you have a comparison that changes training excitation of the same probe while keeping its frequency and physical-response difference fixed? This would address my main concern much more directly than adding more high-frequency probes.

Yes. We now conducted this comparison. We changed the training data as you suggested.

In an extra experiment, we added small waves of one fine-scale pattern, A = (20, 0), to a fixed 25% of the original training set and recomputed their targets with the reference solver. The other training inputs were unchanged, so the number of training inputs stayed the same. Namely, the model could now see pattern A in some training inputs, and we then tested it on held-out states perturbed along pattern A.

As a check, we added the same small waves of a different fine-scale pattern, B = (16, 12), to the same training inputs instead. B has the same spatial frequency as A (ρB=162+122=20\rho_B = \sqrt{16^2 + 12^2} = 20) and differs only in direction, so the two versions differ only in which pattern the model saw.

To keep the physical-response difference fixed, the misspecified hybrid in each training condition was supplied with the same 1.5$\times$ diffusion (the magnitude misspecification in our original submission). Since this diffusion acts equally in all directions, its error depends only on the spatial frequency, so the error has the same size at A and at B.

We used three training data sets: 1) the original training data, 2) the data with pattern A added, and 3) the data with pattern B added. On each, we trained two hybrid models in the same way as in the original experiments, 1) with the correct diffusion and 2) with the 1.5$\times$ diffusion.

We then tested every trained model on held-out states perturbed along pattern A and along pattern B, separately.

Results:

  • When pattern A was added to training (saw A, tested A), β\beta at pattern A fell from 0.94 to 0.07.
  • When pattern B was added instead (saw B, tested A), β\beta at pattern A fell only to 0.83.
  • Testing on pattern B gave the same result: β\beta at pattern B fell from 0.98 to 0.06 when B was added (saw B, tested B), but fell much less when A was added (saw A, tested B).
  • In all 5 runs, at both patterns, the drop was much larger when the tested pattern itself had been added to training.
  • Besides the difference between the two hybrid models along a certain pattern (measured by β\beta), the whole difference between the two hybrid models (including their differences along all other patterns) also fell from 0.93~0.98 to below 0.3 of the supplied difference in 4 of 5 runs at pattern A and in all 5 runs at pattern B.

Conclusion:

With the spatial frequency and diffusion error fixed, the neural component corrected most of the diffusion error at a pattern only when that pattern had been added to training.

So, whether the hybrid saw the tested pattern during training can affect how much of the same diffusion error remained.

One thing you may be interested in is, in Section 4.3, after training with the misspecified diffusion, we replaced it with the correct diffusion without retraining. We repeated the replacement for the models trained in the new experiment (original, saw A, saw B), and got similar results to those in Section 4.3. More details are given in the revised submission.

We also ran this kind of test for a nonlinear error case (details under Q4).

Q2. How much do the conclusions change with longer training or validation-only tuning of the data-only baseline? I am particularly interested in whether the reported crossing is robust, rather than its exact location.

Very little for what we tested.

Your concern is that a weak data-only model could make the hybrid model look better (unfairly weak baseline), so that the point where the hybrid model loses its advantage (the crossing) would be an artifact. To test this, we made the data-only model stronger (the hybrid models were left unchanged). We trained each of the 5 data-only runs for more configurations and kept the version with the lowest validation error (best validation performance among the configurations we tried). Its validation error dropped by 21% to 42%.

Against this stronger data-only model, the hybrid model still had lower error on the scarcely represented patterns in all 5 runs when the supplied diffusion was 2$\times$ the true diffusion, and higher error in all 5 runs at 2.75$\times$ (the crossing is between 2 and 2.75). At 2.5$\times$, 3 runs favored the hybrid model and 2 favored the data-only model (so the crossing is around 2.5, not exactly).

With the hybrid models held fixed, we expect a stronger data-only model to make the hybrid model lose its advantage at a smaller diffusion error (the crossing may disappear); if both models are tuned further, the direction of the shift is uncertain (modelers can always further tune both hybrid and data-only models, endlessly; so it's very hard to draw a final conclusion).

AuthorsOpenReview ↗

Reply to Reviewer sdRK (Questions For Authors 3 & 4)

Q3. Do the finite-difference checks establish only agreement with automatic derivatives, or do you also observe the claimed benefit in finite-amplitude perturbed predictions?

The finite-difference checks in Appendix E.5 only established agreement with the automatically computed derivatives. We added new tests to see whether the hybrid model's advantage over the data-only model also held in predictions from perturbed states (it did).

First, we tested the trained hybrid models with incorrectly directed diffusion and the trained data-only models (all without retraining) on a new test set of held-out states with a small added wave of size 0.03 (the largest used in Appendix E.5) or 0.15 (5 times as large), in units of the training standard deviation. Each model predicted the next state, and we compared it with the true next state from the solver of the true equations. The hybrid model had 50% to 88% lower prediction error than the data-only model at both sizes, in all 5 runs.

Second, on another new test set, we added a wave made of many spatial patterns at once (size 0.25) to held-out initial states, and let the same trained models forecast 50 steps, each step using the model's own previous prediction as input. We compared each step with the true state from the solver of the true equations. The hybrid model had lower error than the data-only model at every step in all 5 runs. At step 50, the mean error was 0.00436 for the hybrid model and 0.01748 for the data-only model.

Third, we checked the Q1 result (pattern A/B case) in the same way. We used the hybrid models with 1.5$\times$ the correct diffusion trained in the Q1 experiment (all without retraining), added a small wave of pattern A or B (size 0.15) to held-out states (a new test set), and compared each model's predicted effect of that wave with the true effect from the solver of the true equations. One step ahead, the model with that pattern added to training predicted the effect more accurately than the model trained on the original data, in all 5 runs for both patterns. So the Q1 result also holds in actual predictions, not only in derivatives. One caveat is that, over 20-step forecasts, this benefit became less consistent (at step 20, it held in 4 of 5 runs for A and 3 of 5 for B), so adding a pattern to training mainly improves short-term predictions along that pattern.

Q4. A nonlinear physical misspecification would be a useful extension, but I view that as a suggestion for strengthening the work rather than something the authors must complete during rebuttal.

Thank you for this suggestion. We ran a nonlinear case during the rebuttal. We kept diffusion correct and tested 2 types of errors in the cubic part of the reaction term (−u3-u^3): a coefficient 50% too large, and a term with the wrong shape that grows more slowly than −u3-u^3 for large uu.

For the wrong coefficient, the nonlinear error results match our diffusion (linear error) case:

  1. the hybrid model still forecast better than the one that learned the whole reaction from data in all 5 runs through step 20 (after step 20 this advantage weakened, and at step 50 it held in 3 of 5 runs), when tested on states much larger than any seen in training (an extreme condition never seen in training, similar to projecting climate beyond the range of the historical record);
  2. the hybrid model corrected more of the error when it had seen states of similar size in training.

For the term with the wrong shape, the hybrid model did not outperform the model that learned the whole reaction from data under the tested training budget (although the hybrid models with the wrong shape were still improving when training stopped; we think they may need more epochs to train, but for fair comparison, we stopped there).


Thank you again for your careful and constructive review! We hope these additions address your main concern.

Reviewer_nKbaOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether a misspecified physical component can still help extrapolate more reliably and whether its misspecification shapes the extrapolation error of the corresponding hybrid model. The paper provides a finite-training relation between differences in supplied physical components and differences between trained hybrid models. Experimental results support the author's claim.

Critical strengths
  1. The problem considered in this paper is important.
  2. The paper did an extensive analysis.
Critical weaknesses
  1. The theoretical analysis does not match the nonlinear training process used in the experiments. Specifically, the theoretical analysis assumes linear models, while the experiments use nonlinear models.
  2. All experiments use the same reaction-diffusion system. The misspecification is applied only to the linear diffusion operators.
  3. The response error in Eq. (12) measures the directional derivative. However, these derivatives are evaluated at held-out reference states. It would be helpful to evaluate actual out-of-distribution trajectories and longer prediction horizons.
Questions for authors
  1. Do the findings still hold for more realistic settings?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer nKba (Critical Weaknesses 1 & 2)

Dear Reviewer nKba, 

We sincerely thank you for your time and effort in reviewing our work. We address each of your points below.

Presentation: 1: Difficult to follow despite careful reading.

We appreciate this frank assessment. It showed us where the connection between theory and experiments was hard to follow, and we have worked to make it clearer.

We now made some changes to the presentation of the manuscript, which we considered necessary to improve clarity and make previously unclear points easier to understand.

CW1. The theoretical analysis does not match the nonlinear training process used in the experiments. Specifically, the theoretical analysis assumes linear models, while the experiments use nonlinear models.

Yes, we agree that the propositions do not describe the training procedure in our FitzHugh-Nagumo experiments. Proposition 1 assumes a linear correction (ideal case). Proposition 2 allows a correction that is nonlinear in its inputs, but its formula is exact for a network linearized in its parameters at initialization and trained with full-batch gradient descent (ideal case). Proposition 2 also allows nonlinear errors in the supplied physics (such as a misspecified reaction term). The FitzHugh-Nagumo experiments use minibatch AdamW (out of practical experience).

To see how closely Proposition 2's formula matches the results from the nonlinear network, we trained a small nonlinear network and its linearized version with the same loss and full-batch gradient descent. Proposition 2 matched the results for the linearized version to numerical precision. For the nonlinear network, the gap between the formula's prediction and the measured difference between the responses of the two hybrid models was less than 2% of the supplied diffusion difference (close agreement in this test). The formula therefore gave a close approximation in this controlled comparison under full-batch gradient descent. Such close agreement shows that the formula can still be useful when the network itself is nonlinear, provided we keep the training procedure used in this test. For the FitzHugh-Nagumo models, we measured directly how much of the supplied diffusion difference remained after training. We expect the qualitative result to carry over to minibatch AdamW (more of the supplied physics's difference can remain when the test pattern is poorly represented in training), although the full-batch formula cannot directly tell us what will happen when we use minibatch AdamW.

CW2. All experiments use the same reaction-diffusion system. The misspecification is applied only to the linear diffusion operators.

Yes, our reaction-diffusion experiments use one system. We chose it because we know the true equations and can control which physical term we change and by how much. We understand you may have some concerns about how insights derived from an idealized system can translate to practical applications. However, adding another system (even a more realistic one) would not necessarily establish that our conclusions can generalize to other systems. We once did make substantial efforts to develop and test a realistic case over the last year, but were ultimately unable to obtain sufficiently reliable results. We therefore believe a more feasible way is to establish the findings in a well-controlled idealized setting and then focus on how the insights derived from this setting can be connected to (and eventually tested in) realistic systems.

For the second point, we agree that it's better to include a nonlinear case.

So during rebuttal we added a nonlinear case. We kept diffusion correct and tested 2 types of physical misspecification in the cubic part of the reaction term (−u3-u^3): a coefficient 50% too large and a term with the wrong shape.

For the wrong coefficient, the results still match our diffusion case: on states much larger than any seen in training (extreme case; somehow similar to projecting climate beyond the range of the historical record half century later), the hybrid model still forecast better than the one that learned the whole reaction from data, and it corrected more of the error when it had seen states of similar size in training. Nevertheless, under the tested training budget (because given enough tuning, one can always make the preferred model come out on top), the hybrid model with the wrong-shaped term did not forecast better than the hybrid model that learned the whole reaction from data (more details please check our reply to Reviewer sdRK, Q4).

We truly appreciate your insightful comment prompting us to test our central claim more thoroughly and better establish the conditions under which it holds and where its limitations lie.

AuthorsOpenReview ↗

Reply to Reviewer nKba (Critical Weaknesses 3 & Questions For Authors 1)

CW3. The response error in Eq. (12) measures the directional derivative. However, these derivatives are evaluated at held-out reference states. It would be helpful to evaluate actual out-of-distribution trajectories and longer prediction horizons.

Thanks for pointing out this.

We now added both, and the hybrid model mostly kept its advantage (details in our reply to Reviewer sdRK, Q3 and Q4).

We added two 50-step forecast tests (longer prediction horizons). In both tests, each model used its own previous prediction as input, and we compared the predicted states with the solution of the true equations.

In the diffusion test, we added some unseen spatial patterns to the held-out initial states before forecasting. These patterns had higher frequencies than those used to generate the training initial states (more details please see the second test under Q3 in our reply to Reviewer sdRK). The hybrid model with incorrectly directed diffusion had lower prediction error than the data-only model at every step in all 5 runs (at step 50, the mean error was 0.00436 for the hybrid model and 0.01748 for the data-only model).

In the nonlinear reaction test, we started from states whose magnitude of uu exceeded the largest value in the original training data (a magnitude never seen in the original training data; extreme case). For this test, the hybrid model with the wrong coefficient in the reaction term still forecast better than the hybrid model that learned the whole reaction from data in all 5 runs through step 20 (after step 20 this advantage weakened somehow, and at step 50 it held in 3 of 5 runs). However, for the wrong-shaped reaction term case, such advantage did not hold under the tested training budget (please see more details under Q4 in our reply to Reviewer sdRK).

Q1. Do the findings still hold for more realistic settings?

Unfortunately, the direct results from our idealized experiments themselves cannot fully answer this question, which is a limitation of our study.

However, based on our tests under different extrapolation conditions, we can offer some insights. Our original experiments mainly considered highly extreme unseen case for model evaluation (extreme case). We have now also added tests in which models can encounter patterns that are partially similar to those in the test set, representing a less extreme form of extrapolation (partially extreme case). Based on these results (both in original and newly added experiments), we tentatively infer that supplied physics may have a greater influence when the test conditions are poorly represented in training. Whether that gives the hybrid model a larger advantage depends on the accuracy of the supplied physics.

This may have implications for climate projections (one of our motivation).

  • For short-term projections, future patterns may still be relatively close to those encountered during training. In this case, we expect hybrid models to perform comparably to, or perhaps slightly better than, data-only models, as the distribution shift may not yet be sufficiently large for the advantages of hybrid models to become pronounced.

  • For medium- to long-term projections, as future conditions increasingly diverge from historical patterns, the advantages of hybrid models may become more evident if the supplied physics remains accurate enough under those conditions.

  • However, for sufficiently long-term projections, we are concerned that neither approach may maintain the level of accuracy required for scientific applications (particularly if the supplied physics in the hybrid contains nonlinear errors). Under such circumstances, which is better (hybrid model with severely misspecified physics, hybrid model with moderately misspecified physics, hybrid model with slightly misspecified physics, data-only model; even pure process-based model that is widely used in scientific community, may contain large misspecification under highly extreme unseen conditions) may become less meaningful (all useless for scientists).


Thank you again for your careful and constructive review! We hope our responses can address your concern.

Area_Chair_noQrOpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer nKba,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Author Response and Summary of Revisions

We are sincerely grateful to all three reviewers and the Area Chair for the time and effort they devoted to reviewing our work (particularly in a year with an exceptionally high volume of submissions). We greatly appreciate the reviewers' thoughtful feedback and the opportunity it has given us to reflect on how we can strengthen the evidence supporting our work and communicate more clearly. Below, we summarize the main revisions made in response to the feedback. Details are given in individual replies.


What we added:

  1. In the original paper, the neural component in our hybrid model corrected little of the error in the supplied diffusion on fine-scale spatial patterns. Reviewer sdRK noted two possible reasons. Either the network rarely saw such fine-scale patterns during training (reason 1), or diffusion is so strong on these patterns that the neural component's correction has little effect (reason 2). To tell these two reasons apart, we added examples of one such pattern (pattern A) to the training data, retrained the model with the same incorrect diffusion, and tested it on pattern A (saw A, tested A). As a check, we repeated the training with a different fine-scale pattern (pattern B) added instead, and again tested it on pattern A (saw B, tested A).

  2. We checked our theoretical results in simple linear models.

  3. We tested whether the hybrid model's advantage still holds over longer forecasts (not just one step ahead).

  4. We trained and tuned the data-only model further, to check that our comparison with it still holds.

  5. We conducted a nonlinear physical misspecification case, keeping diffusion correct and testing two errors in the cubic part of the reaction term: a wrong coefficient of the cubic term and a wrong term that grows more slowly at large amplitudes than the true term.


What we found:

  1. When pattern A was added to training (saw A, tested A), the neural component corrected most of the diffusion error on pattern A in most runs. When pattern B was added instead (saw B, tested A), most of the diffusion error on pattern A remained. So the correction of neural component in the hybrid model came mainly from seeing pattern A itself, not from adding fine-scale data in general (so reason 1 is the more likely answer). This is consistent with our main claim: on patterns that are rare in training, a hybrid model inherits the error of its supplied physics, because the neural component has too little data to learn and correct it.

  2. In simple linear models, our two theoretical results (Propositions 1 and 2) held.

  3. In the 50-step forecasts, the hybrid model had lower errors than the data-only model at every step (the hybrid model's advantage still holds).

  4. With further training and tuning, the data-only model did not change our conclusion: the hybrid model still loses its advantage when the diffusion error is large enough.

  5. The nonlinear case supports our main claim as well: far outside the training data, more of the error from the misspecified reaction term remained in the hybrid model, yet the hybrid model with the wrong coefficient in the reaction term still worked better than the hybrid model that had to learn the whole reaction from data.


Last but not least, we would like to respectfully clarify that our motivation is to understand when imperfect physics helps a hybrid model and when it misleads it. This is hard to test directly in a realistic case (we spent much effort on it but failed), because the true physics is unknown and many errors act at once, their effects cannot be separated. We therefore used the FitzHugh-Nagumo reaction-diffusion system, where the true equations are known and we can introduce one controlled error at a time. All new experiments we added during rebuttal remain within this system (although we would really like to add a realistic case; the rebuttal rules and time limits mean we have to prioritize addressing all current concerns first). The diffusion errors we study have direct counterparts in physical sciences, for example a wrong hydraulic diffusivity or anisotropy in groundwater flow models. Hybrid models are built differently across scientific fields, and no single realistic system can represent them all. We deliberately chose patterns the neural component had rarely seen in training so we could see how much of the supplied physics error remained when the neural component had little opportunity to learn to correct it (but this raises some concerns as Reviewers sdRK and gnvx noted); now the newly added experiments move toward conditions that are partly represented in training. We cannot yet say how far our findings carry over to realistic settings or to other nonlinear errors. Linking these controlled experiments to realistic applications is the main direction of our ongoing work.

Again, we are deeply grateful to the reviewers. Their questions have improved both the evidence and the clarity of the paper.

AuthorsOpenReview ↗

Follow-up: summary of the revised submission

We have uploaded the revised submission. Before the discussion period closes, we would like to thank all three reviewers and the Area Chair once more. The discussion has been one of the most useful parts of this work for us. Each reviewer pointed us to a question we had not examined carefully enough, and the revision is a direct result of these questions. Below we summarize what the uploaded revision now contains, separating material that is new since our general response of Nov 13 from experiments reported during the rebuttal that are now part of the manuscript.

New since our Nov 13 response

  1. Numerical checks over training steps (Section 4.1, Figure 1). The measured projection coefficient is compared with the prediction of Proposition 1 throughout training for directions absent, scarcely represented, and well represented in the training inputs, following Reviewer gnvx's request. Section 4.1 also incorporates the previously reported shared-weight and nonlinear-network checks.
  2. Decision map (Sections 3.2 and 4.2, Figure 2, Appendix D). For an idealized additive hybrid model, it shows the range of misspecification strengths over which the hybrid model keeps a lower response error than a purely data-driven model, for eleven forms of misspecification and for patterns from absent to well represented in training (suggested by Reviewer gnvx).
  3. Correcting two misspecified terms one after the other (Appendix I.3, Table 10, and a new paragraph in Section 5 on reusing a trained hybrid model; suggested by Reviewer gnvx).

Rebuttal experiments now in the manuscript

  1. Nonlinear misspecification in the cubic reaction term, shown next to the diffusion errors in the main results: how much of the supplied error remains (Figure 3c and d), the intervention (Figure 4c and d), and 50-step forecasts (Figure 6b) (Reviewers nKba and sdRK).
  2. Prediction tests with finite perturbations (Appendix K) and forecasts over 50 steps for the diffusion error (Figure 6a), addressing the questions from Reviewers sdRK, nKba, and gnvx about predictions beyond local derivatives and a single interval.
  3. Adding pattern A or pattern B to training with the spatial frequency and the supplied diffusion error fixed (Appendix I.2, Table 9) (Reviewer sdRK).
  4. Comparison with a further-trained purely data-driven model (Appendix K) (Reviewer sdRK).

Clarified scope of the claims

  • Section 3.3 now states which assumptions of the propositions do not hold under the practical training strategy (mini-batch AdamW, loss after time integration, checkpoints selected by validation loss). The controlled experiments are therefore presented as an empirical examination of which idealized findings carry over, not as a verification of the propositions.
  • The claim that correcting the supplied physics removes most of the extrapolation response error is restricted to the tested diffusion errors on scarcely represented patterns. The results of intervention analysis for reaction case are reported separately, including cases where correction increases prediction error. Appendix A shows that correcting the supplied physics without retraining does not always reduce the extrapolation error of the trained hybrid model.

The revision does not establish that hybrid models are generally better, nor a threshold of misspecification that holds across systems. The practical experiments use one FitzHugh-Nagumo reaction-diffusion system, and the five training seeds share the same trajectory dataset. Generalization across systems and independently generated datasets remains to be tested, and addressing it is the main direction of our further work.

Thank you all again for your time and for such thoughtful engagement. We have learned a great deal from this discussion, and your insights will guide our further study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.