- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies neural networks trained to predict the outcome of physics simulations, and argues that standard explanation methods (e.g., masking parts of the input) only reveal how the network responds to an edited input, not whether that response matches what would actually happen in the physical system.
Method:
To address this, the paper proposes executable intervention games: starting from the same initial state, both the network and a numerical solver are run for every on/off combination of a few prescribed physical actions (e.g., a local conductivity change or a wall rotation). For each action, the authors compare its effect given which other actions are already active (conditional effect) and its exact Shapley value. Where the network disagrees with the solver, the researcher applies one of two fixes: add training examples covering more action combinations, or add a loss on the difference between single-action and no-action predictions, using labels already available.
Results:
-
Heat conduction: training with the single-action difference loss lowers error on held-out action effects for new materials, compared with longer ordinary training and with a loss on absolute temperatures using the same labels; an independent rerun confirms this.
-
Flow past a rotating cylinder: adding on/off action combinations to the training data lowers effect error compared with a control that sees the same flow states, histories and number of labels, including at Reynolds numbers not used in training. Since the comparison used an evaluation set the authors had already inspected, it shows that covering combinations helps here, not that one particular set of combinations is best; results also depend somewhat on the solver’s time step.
-
Smaller 1D studies show three things: attributions computed by editing the field instead of running the action differ substantially from the solver reference; models that predict the no-action field almost perfectly can still have large attribution error, which drops once training includes action combinations; and the model’s predicted sign changes hold up under solver time-step checks, though models before and after the fix predict the same events, so the fix is not shown to find anything new.
-
A 2D smoke-flow study is a negative case: after adding action combinations, every continued model had higher held-out effect error than the starting model, even though training loss fell and predicted fields improved on training trajectories. In a separate test, the difference loss looked slightly better on average, but its confidence intervals included zero and other metrics got worse. The authors take this as evidence that better field prediction does not guarantee better effect prediction.
Critical strengths
In the heat conduction experiment, the improvement can be attributed to the new loss: all models use exactly the same training data, and the standard model is even trained slightly longer, yet the new loss still gives better results on new test materials. The authors also report the cases where their method does not help, such as smoke flow (Section 4.4), which gives a more honest picture of the results.
Critical weaknesses
-
The paper does not discuss how its approach relates to counterfactual explanations. Its queries ask what the output would have been for a fixed, known starting state if a specific action had been applied. This is a counterfactual (what-if) query, closely related to counterfactual explanations [1,2], which ask the inverse question of which input change would produce a desired output. Similarly, the proposed response supervision trains the model on the difference between the output with and without an action for the same starting state, which is closely related to training on counterfactual pairs [3,4]. However, the paper never mentions counterfactuals: its related work (Section 5 and Appendix A) covers removal-based explanations, causal and interventional Shapley values, interventionally consistent surrogates, explanation-guided learning, derivative supervision and active learning, but does not discuss counterfactual explanations or training on counterfactual examples.
-
The paper cites methods closely related to its two forms of feedback but does not compare against them where its main claims are made. Sensitivity-constrained FNOs [5] regularize how a model’s output changes with its input parameters, which is closely related to the response loss proposed for heat conduction, yet they are never run as a baseline.
-
The paper distinguishes its finite-amplitude contrast supervision from derivative supervision (Appendix A, Table 5) but does not test that distinction.
-
Derivative supervision appears only in the separate 1D study (Appendix C.2), as the "Sensitivity" setting in Table 10, where it gives the lowest Shapley and IG errors of all settings for every architecture at about five times the training cost, and in a follow-up that is reported to lose to time-matched ordinary training; the heat-conduction experiment does not include it.
-
Active learning for neural PDE solvers [6] addresses which simulations to add to training, the question the cylinder coverage study asks, but it is cited only in Related Work (Section 5) and never compared in the cylinder study: the cylinder training sets (Tables 3, 12, 13) are the authors’ own fixed, random and continuous choices of action combinations, with no uncertainty- or diversity-based selection baseline. The only uncertainty-based selectors in the paper are in a 1D study (Appendix B.5) that tests loss functions, not selection rules. Without these comparisons, the paper shows that its feedback helps compared to its own baselines, but not that it helps compared to existing methods.
- The abstract’s claim ('support their improvement for scientific inquiry') goes beyond the evidence. The only feedback result with a prespecified, independent confirmation is one small heat-conduction problem (32×32 grid, four actions). The cylinder comparisons are described by the authors as 'descriptive' (line 373), and random on/off menus perform as well as the menu chosen from the diagnosis (lines 365–367), so the diagnosis may have motivated adding combinations, but the specific menu it suggested gave no advantage over random ones. Similarly, in smoke the diagnosis-targeted menu did not outperform random or structural menus (Appendix G.1, Table 27). The abstract also omits that the feedback fails in smoke (Section 4.4). This matters because the paper’s central claim, that checking responses guides useful feedback, is clearly supported only in one small setting.
[1] Verma, Sahil, et al. "Counterfactual explanations and algorithmic recourses for machine learning: A review."
[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR."
[3] Teney, Damien, Ehsan Abbasnedjad, and Anton van den Hengel. "Learning what makes a difference from counterfactual examples and gradient supervision."
[4] Kaushik, Divyansh, Eduard Hovy, and Zachary C. Lipton. "Learning the difference that makes a difference with counterfactually-augmented data."
[5] Behroozi, Abdolmehdi, Chaopeng Shen, and Daniel Kifer. "Sensitivity-constrained fourier neural operators for forward and inverse problems in parametric differential equations."
[6] Musekamp, Daniel, et al. "Active learning for neural PDE solvers."
Questions for authors
-
How does your approach relate to counterfactual explanations [1,2] and to training on counterfactual pairs [3,4], and why are these lines of work not discussed?
-
Why does the feedback fail in smoke, and how would a user know in advance whether it will help?
-
In cylinder flow, the test uses on/off actions at a fixed amplitude, but only the H, S and B training sets use that input type; the controls R and Rφ use continuous coefficients. How do you separate the gain from covering more combinations from the gain of simply training on the same input type as the test?
-
Could you compare against derivative supervision (sensitivity-constrained FNOs) in the heat-conduction experiment, and against an active-learning method for choosing which simulations to add in the cylinder experiment? Both are cited in the related work, but neither is tested in these experiments.