- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a potential confound in evaluating physics-informed machine learning: an apparent improvement from introducing a physics prior does not necessarily imply that useful physical information has been transferred to the model.
The paper demonstrates this phenomenon on a wildfire-spread prediction benchmark. Distilling a Rothermel-based physics teacher into a U-Net substantially improves average precision over a Dice-loss baseline. However, the authors show that the Rothermel prior itself has very little predictive information at the spatial and temporal resolution of the benchmark. More importantly, replacing the physics teacher with an information-free constant teacher reproduces almost the entire improvement, while simply modifying the supervised loss to include BCE also captures most of the gain. The authors attribute the apparent physics-informed improvement primarily to a regularization effect that prevents saturation under the original Dice objective, rather than to information contained in the physical prior.
Based on this observation, the paper proposes two controls for physics-informed learning: first, evaluate whether the physical prior itself contains task-relevant information before training; second, compare physics-informed auxiliary objectives against information-free, level-matched controls.
The paper additionally develops a compact neural cellular automaton as an informative learned prior, analyzes the WildfireSpreadTS benchmark in detail, and extracts and compares learned wildfire-response rules.
Critical strengths
-
I like the central experimental observation. The result that a seemingly substantial "physics-informed" gain can be almost completely reproduced by an information-free constant teacher is a useful caution for evaluating physics-informed methods. The comparison against constant and shuffled teachers is particularly informative.
-
The paper makes a good methodological point about attribution. A comparison between "with physics" and "without physics" is not sufficient to establish that physical information caused an improvement if introducing the physics term simultaneously changes the optimization or regularization properties of the objective.
-
The authors perform a relatively extensive set of controls. In particular, they separate the effect of the teacher's output level from the information carried by its spatial variation, and investigate the optimization mechanism behind the improvement.
-
I also appreciate that the paper attempts to construct an actually informative learned prior as a positive control, rather than only comparing the physics teacher against completely uninformative baselines.
-
More broadly, the paper raises a useful question for AI-for-science evaluation: when incorporating scientific knowledge improves a model, what experiment is necessary to establish that the scientific information itself, rather than an incidental change in the learning objective, produced the gain?
Critical weaknesses
My main concern is that the scope of the paper's claims appears substantially broader than what is established by the experiments.
The empirical result convincingly demonstrates a failure mode for one physical prior on one wildfire benchmark, especially when the prior is introduced through teacher distillation or an auxiliary soft-target objective. In this setting, the Rothermel teacher contains little usable predictive information at the benchmark resolution, and its apparent benefit can largely be reproduced by an information-free constant target. I find this result convincing.
However, I do not see why this case study is sufficient to motivate the proposed checks as general controls for "physics-informed learning." Physics can enter machine learning in substantially different ways: as a predictive teacher, as a soft target, as a differential constraint, as an architectural inductive bias, or through conservation laws and symmetries. The mechanism identified in this paper seems especially relevant to the first two cases.
For example, consider a standard Physics-Informed Neural Network in which a known PDE is imposed through a residual loss. The value of the PDE is not primarily that it independently predicts or ranks individual labels. Rather, it restricts the admissible function space by requiring the learned solution to satisfy a known differential relation. It is therefore unclear how the paper's notion of "prior informativeness," which is operationalized here through predictive performance of the prior, should be applied to this canonical physics-informed setting.
The paper does include an experiment using the physical law as a residual, but this still demonstrates only that this particular physical model is uninformative at this particular dataset resolution. It does not establish that the proposed diagnostic generalizes to physics-informed learning more broadly. I think the paper should either substantially narrow its claims or demonstrate the proposed controls across several qualitatively different mechanisms for incorporating physical knowledge.
A second major concern is presentation. The central message of the paper is actually quite simple and potentially useful: a physics teacher appears to improve performance; an information-free control reproduces the improvement; the improvement can be explained by the optimization/regularization effect of the loss; therefore physics-informed gains require appropriate controls. However, this message is obscured by a very large number of secondary experiments, benchmark-specific details, numerical results, and side analyses.
In particular, the extensive discussion of the neural cellular automaton, rollout averaging, parameter counts, wildfire response curves, individual environmental variables, benchmark variance, and other implementation details makes it difficult to identify the hierarchy of claims. Many of these analyses may be individually reasonable, but they substantially dilute the main scientific argument. I believe the paper would benefit from aggressive compression around the central causal story, with many of the secondary analyses moved to the appendix.
Relatedly, I found the paper unusually difficult to read despite the central idea being relatively straightforward. Individual paragraphs contain many details and qualifications, but the paper does not consistently distill these details into a small number of conceptual conclusions. As a result, I frequently found myself losing track of which experiments were essential for establishing the main claim and which were supplementary observations.
Finally, once the paper is reduced to its central argument, I am somewhat uncertain about the level of novelty. The observation that an auxiliary loss can improve optimization or act as a regularizer independently of the information contained in its target is not itself surprising. The interesting contribution is applying information-free and level-matched controls to physics-informed claims. I would therefore like the authors to clarify whether they view the primary contribution as a new scientific phenomenon, a methodological recommendation for a particular class of physics-informed methods, or a broader evaluation framework for AI-for-science.
Questions for authors
-
What is the intended scope of the proposed controls? Are they primarily intended for physics-informed methods in which physics enters as a teacher, pseudo-label, or auxiliary prediction target, or do the authors intend them to apply to physics-informed learning in general?
-
How should Check One be defined for a standard PINN in which the physics prior is a known governing PDE? A PDE does not naturally provide a standalone ranking of examples analogous to the wildfire prior used here. Its information instead comes from restricting the admissible solution space through a differential constraint. What would constitute an "information-free" or "level-matched" control in this setting?
-
More generally, how would the proposed framework handle other common forms of physical knowledge, such as conservation laws, equivariances, symmetries, or physics-informed architectures? These seem qualitatively different from distillation from a physics teacher.
-
The current evidence comes from one physical law and one benchmark. What evidence supports recommending these checks as standard practice beyond this setting? I would find experiments across qualitatively different physics-informed mechanisms substantially more convincing.
-
What do the authors consider the primary scientific contribution of the paper? Is it the discovery of the constant-target regularization mechanism in this wildfire experiment, the proposal of information-free controls for physics-informed evaluation, or a broader claim about physics-informed learning? Clarifying this hierarchy would make the paper much easier to evaluate.
-
How essential are the extensive NCA experiments to the central contribution? I understand its role as an informative positive-control prior, but much of the subsequent NCA analysis appears orthogonal to the main methodological claim. Could the main paper be substantially simplified by retaining only the NCA results necessary for the control experiment and moving the remaining analysis to the appendix?
-
Could the authors reorganize the paper around the central causal chain: apparent physics-informed gain -> information-free control reproduces the gain -> loss-level mechanism explains the gain -> methodological recommendation? I think this would substantially improve readability and make the scientific contribution easier to assess.
We thank the reviewer for a detailed and constructive report. We agree with the two main concerns, and we would like to respond to them directly.
The reviewer is right about the scope, and we will narrow the claims. The controls are intended for methods in which physics enters as a training signal: a teacher, a pseudo-label, a soft target, or an auxiliary loss term. Our evidence supports them for that class, and we will state that in the title subtitle, the abstract, and the introduction. We do not claim they cover architectural priors, symmetries, or conservation laws built into the model, and we will state this explicitly.
We agree that a governing equation does not rank examples, and that "check one", as we implement it, does not transfer unchanged. However, we propose that the idea behind it does: before training, measure on held-out observations whether the law holds better than a null alternative at the data's resolution. For a residual loss, this is the residual of the law evaluated on observed data, compared with the residual of a deliberately wrong law. The matching control for "check two" is to train with that wrong law, for example with shuffled or constant coefficients, which keeps the form of the loss and removes the physics. We have not tested these forms, and the revision will present them as a proposal and not as a result. Our own residual experiment supports only the narrow statement that this law is uninformative at this resolution.
We view our contribution as a methodological recommendation for the class above, supported by one fully worked case. We agree that an auxiliary loss acting as a regulariser is not surprising in itself. What we add is the demonstration that it can account for an entire physics-informed gain on every fold of a public benchmark, and the level-matched controls that separate the two effects. The finding was not planned: we set out to improve wildfire prediction with a physics prior, and our own controls refuted the result.
We accept the criticism on presentation and will reorganise the paper around the chain the reviewer describes: the apparent gain, the information-free control that reproduces it, the loss-level mechanism, and the recommendation. The learned prior will stay in the main text only in its role as the positive control. Its capacity study, rollout analysis, and rule extraction will move to the appendix, and each section will end with a single stated conclusion.
On evidence beyond this setting, we agree that one law and one benchmark is a limitation. Additional experiments are in progress and will be included in the final version if the paper is accepted.