- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines how errors in the supplied physical component of a hybrid model (known physics plus a neural network) affect its extrapolation accuracy.
Theoretically, for a linear additive hybrid model, it shows that the difference between models trained with different physical components persists after training along directions with little variance in the training data.
Empirically, it supplies four misspecified diffusion terms to a hybrid model of a 2D FitzHugh–Nagumo system and evaluates its extrapolation performance over a single prediction interval:
- While all models show similar held-out errors, hybrid models outperform the purely data-driven model on scarcely represented patterns, even with misspecified physics.
- Response differences between trained hybrids closely follow the differences between their supplied physics in magnitude and direction.
- Replacing the misspecified physics with the correct one, with the trained network fixed, removes most of the error on scarcely represented patterns but increases it on well-represented ones.
- Sufficiently overestimated diffusion removes the advantage over the purely data-driven model.
Critical strengths
- Carefully controlled experimental design. The ground-truth dynamics are known, misspecifications are injected by type, and initialization and minibatch order are paired across conditions. Combined with the post-training replacement of the physical component, this allows the extrapolation error to be attributed to the supplied physics rather than to training noise.
Critical weaknesses
- Limited practical and theoretical takeaways. The result of Proposition 1 is expected: in an additive hybrid, the network cannot correct the physics along directions that the training data barely cover. The replacement experiment, which shows opposite error behavior on well- and scarcely represented patterns, ultimately implies only that the network should be trained once the physical component is finalized. Section 4.4 identifies where the hybrid loses its advantage in this specific setup, but offers limited practical guidance on how accurate the supplied physics needs to be for hybrid modeling to be beneficial.
- Limited experimental scope. All experiments use a single PDE system, and the evaluation covers only a single short prediction interval, so it is unclear whether the conclusions hold for other systems or over longer rollouts.
Questions for authors
- Can the authors empirically validate Proposition 1 in a controlled linear setting?
- Is there any reason why the paper considers the FitzHugh–Nagumo PDE?
- Beyond the diagnostic in Section 5, how can practitioners use these findings when the true physics is unknown?
6 comments
Reply to Reviewer gnvx (Critical Weaknesses 2 & Questions For Authors 1 3)
CW2. Limited experimental scope. All experiments use a single PDE system, and the evaluation covers only a single short prediction interval, so it is unclear whether the conclusions hold for other systems or over longer rollouts.
Thank you for pointing out these two limits.
For "a single PDE system", yes, we agree; we explain this choice under Q2 and in our general comment, and whether our findings hold in other systems remains open. Within this system, if you are interested in, we now go beyond the diffusion error: we tested two errors in a nonlinear reaction term (details in our reply to Reviewer sdRK, Q4).
For "longer rollouts", we now extended the evaluation to 50-step forecasts, each step using its own previous prediction as input. Compared with the solution of the true equations, the hybrid model with incorrectly directed diffusion had lower prediction error than the data-only model at every step in all 5 runs (at step 50, the mean error was 0.00436 for the hybrid model and 0.01748 for the data-only model; more details in our reply to Reviewer sdRK, Q3).
For the extremely unseen case (the magnitude of held-out states is much larger than any seen in training), the hybrid model with a wrong coefficient in the reaction term gave lower forecast error than leaving the whole reaction to the neural component (all 5 runs through step 20). This advantage did not hold for the term with the wrong shape under the tested training budget (more details please see our reply to Reviewer sdRK, Q4). So the forecast advantage can last beyond one step, but how consistently it lasts depends on the case.
Q1. Can the authors empirically validate Proposition 1 in a controlled linear setting?
Yes, thank you for this suggestion. We now empirically validated our Proposition 1 as you suggested.
We trained two hybrid models with unrestricted linear corrections on the same inputs and targets (the two models differed in the supplied linear physical term). After training, we measured how each hybrid model responded to the same test direction and compared the difference between those responses with what Proposition 1 predicts. Along a direction absent from the training inputs, the entire supplied difference remained, to numerical precision. Along a weakly represented direction, 62.4% remained after 300 updates, matching the predicted value by Proposition 1 (Eq. 21). The test therefore confirms both parts of the result:
- training leaves the supplied difference unchanged along an absent direction,
- while it can reduce the difference along a weakly represented direction.
Q2. Is there any reason why the paper considers the FitzHugh–Nagumo PDE?
Yes. We chose the FitzHugh-Nagumo system because it lets us introduce a known error in one physical term at a time and evaluate what happens against the solution of the true equations.
Its two variables have different diffusivities and interact through nonlinear reactions. We can therefore change the magnitude or direction of diffusion, or omit diffusion in one variable (with keeping the reaction equations unchanged). We can also do the reverse (keep diffusion correct and introduce a known error in the reaction term).
This separation is what we need to ask whether an imperfect term still helps forecasting and how much of its error remains after training.
Also, it is an established test case for hybrid models with incomplete physics (e.g., APHYNITY; Yin et al., 2021).
Q3. Beyond the diagnostic in Section 5, how can practitioners use these findings when the true physics is unknown?
The practical decision is whether to keep an uncertain equation (we assume at least we have some empirical equations derived from previous observations) or leave that process for the neural component to learn. The pure physical model can be examined first, keeping the first principles (e.g., mass and energy balance), and for each empirical equation, checking its assumptions and the conditions it was derived from (since it is more likely to be badly wrong outside those conditions).
If the relevant observations are scarce, forecasts from different plausible equations can help show where to collect more data. If the models fit existing observations similarly but disagree under the conditions of interest, those are useful conditions to target for new observations. If the forecasts agree, they may still share an error.
In either case, when the available observations cannot establish which model predicts reliably under the intended conditions, collecting observations there takes priority over choosing a model.
Thank you again for your careful review! We hope these additions address your concerns.
I thank the authors for the detailed responses and the additional experiments. My overall assessment is as follows. The paper presents propositions and observations whose conclusions are largely predictable. They neither reveal unexpected behavior nor change how practitioners decide whether and how to use a hybrid model.
- The exact decay laws in Propositions 1 and 2 are a contribution, but their qualitative conclusion (the network cannot correct the physics along directions that the training data barely cover) is expected.
- The replacement experiment shows that a physics update without retraining improves the error along data-scarce directions and worsens it along data-sufficient ones. The practical implication is to retrain after every update to the physics, which the rebuttal itself confirms.
- The observation that a severely wrong physics prior degrades accuracy is also expected. The verification process proposed in the rebuttal (trust first principles, check the assumptions behind empirical equations, compare against independent observations) does not depend on the paper's findings, and the paper does not change or refine it.
Status of each point
- CW1: Not addressed. I appreciate the clarification of the contributions, but the concern remains for the reasons above.
- CW2 and Q2: Addressed. I thank the authors for the longer-rollout experiments. Given the nature of the conclusions, I do not expect them to change substantially for other PDE systems, so I no longer consider the single-system scope a major concern.
- Q1: Partly addressed. I thank the authors for the experiment. However, Proposition 1 predicts an exponential decay of the discrepancy over training. A single value after 300 updates does not verify this. Decay curves over training steps for several levels of coverage, overlaid with the theoretical prediction, would.
- Q3: Not addressed. The conclusion that more data should be collected where models disagree can be reached without the observations or analysis in this paper.
Suggestions
I believe the following would make the work considerably more useful to the community:
- Consider a setting where several terms of the PDE are wrong and are corrected one at a time. Showing how a previously trained hybrid model can be reused so that both data-scarce and data-sufficient directions improve without full retraining, or at least without a trade-off between them, would turn the replacement result into actionable guidance.
- Provide a theoretical guideline, even for a simplified model, on how large the physics error can be before the hybrid model loses its advantage over a data-only model.
Reply to Reviewer gnvx (Critical Weaknesses 1)
Dear Reviewer gnvx,
We sincerely thank you for your time and effort in reviewing our work. We address your points below.
Thank you for raising this. We would like to state our takeaways more directly.
For your concern about "limited theoretical takeaways", we summarize our two theoretical results below. Simply put, Proposition 1 explains why training coverage (partially unseen or totally unseen) matters. Proposition 2 and its related equations explain that, besides training coverage, how the neural component shares its weights also matters.
More specifically, for two hybrid models trained on the same data but given different physical terms,
Proposition 1 tells us how training changes the difference between two hybrid models given different linear physical terms. If a pattern is totally unseen during training, the difference between the supplied physical terms remains unchanged in the two hybrid models' responses. Along a pattern that is only weakly represented, training can reduce that difference, but the reduction can be slow. Or say, "in an additive hybrid, the network cannot correct the physics along directions that the training data barely cover", yes, your summary is accurate. We just want to mention here, Proposition 1 describes the limiting case without weight sharing (e.g., a fully connected or linear regression correction), where every weight of the correction is trained separately.
Proposition 2 (Appendix B) and its related equations in Appendix B show that, when a network shares its weights across locations (as convolutional networks do in practice), learning from patterns it sees can also change how it responds to patterns it does not see. The analysis gives a formula for how much of the difference between the two supplied physics terms carries into the response of hybrid models.
For "the network should be trained once the physical component is finalized", we understand your concern as: our conclusions seem to apply only when the physics is already well known. In practice, the physics is rarely finalized, and this is what motivated our study. In many scientific fields, legacy or process-based models combine true physics (e.g., energy balance, mass balance) with empirical equations (e.g., from laboratory work or statistical fits of field data), and those empirical equations may no longer hold under new conditions. Some researchers therefore replace the uncertain or empirically represented process with a neural component added to the physical model, which forms an additive hybrid (other forms exist, such as differentiable models or networks that adjust parameters of a process-based model, but the additive form is relatively more commonly used). In such a hybrid model, the supplied physics is never finalized or perfect, and our study asks exactly what its errors do to the hybrid model. The replacement result then tells researchers what to do when they later improve the physics: retrain the hybrid model, because the neural component has adapted to the old physics.
For your concern about Section 4.4, we see its result as an important practical warning rather than only a limitation of our setup: imperfect physics helps a hybrid model only up to a point, and beyond that point it makes the hybrid model worse than a data-only model. We understand that the natural next question is how a researcher can know whether the physics is slightly or severely wrong. First principles, such as mass and energy balance, can be trusted. But it still be unsure whether an empirical equation holds under the conditions where predictions are needed. For empirical equations, a researcher should check their assumptions behind that equation and compare models with and without it against independent observations under those conditions, where available. A data-only model avoids inheriting the error in the supplied equation (particularly for data-rich but partially known physics case), but that alone does not establish that its predictions are more accurate. When the physics of a process is truly unknown, it usually means the process has been observed too little; then neither a data-only model nor a hybrid model can be trusted, and the priority should be collecting more data under those conditions, since any model is built on observations.