acceptodds
Withdrawn from ICLR 2027

Executable Intervention Games: From Attribution to Physical Consequences

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Explainable AI helps researchers formulate scientific hypotheses from predictive models, but fidelity to a model does not establish that the relationships it has learned hold in the physical system. Testing these relationships is difficult when input perturbations, such as masking, do not correspond to the physical actions of interest. To bridge this gap, we introduce executable intervention games, in which a model and a numerical reference answer the same questions about prescribed actions executed from shared starting conditions. Comparing their conditional action effects and Shapley attributions reveals response errors that inform researchers’ choices of action coverage or response supervision. In heat conduction, supervising single-action temperature differences improves held-out effect and attribution accuracy over longer field training and absolute-temperature supervision with identical labels. It also reduces average error in predicting how other active actions modify an effect in new materials, even though this dependence is not directly supervised. In cylinder flow, covering action combinations improves responses beyond controls with matched training states and histories, including at held-out Reynolds numbers. By preserving the physical question through explanation, feedback, and evaluation, executable games make learned relationships testable and support their improvement for scientific inquiry.

Primary area:
interpretability and explainable AI
Keywords:
Explainable AI, Feature attribution, Executable interventions, Scientific machine learning, Model faithfulness, Physics-guided learning, Simulation-based validation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BD2oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies neural networks trained to predict the outcome of physics simulations, and argues that standard explanation methods (e.g., masking parts of the input) only reveal how the network responds to an edited input, not whether that response matches what would actually happen in the physical system.

Method:

To address this, the paper proposes executable intervention games: starting from the same initial state, both the network and a numerical solver are run for every on/off combination of a few prescribed physical actions (e.g., a local conductivity change or a wall rotation). For each action, the authors compare its effect given which other actions are already active (conditional effect) and its exact Shapley value. Where the network disagrees with the solver, the researcher applies one of two fixes: add training examples covering more action combinations, or add a loss on the difference between single-action and no-action predictions, using labels already available.

Results:

  • Heat conduction: training with the single-action difference loss lowers error on held-out action effects for new materials, compared with longer ordinary training and with a loss on absolute temperatures using the same labels; an independent rerun confirms this.

  • Flow past a rotating cylinder: adding on/off action combinations to the training data lowers effect error compared with a control that sees the same flow states, histories and number of labels, including at Reynolds numbers not used in training. Since the comparison used an evaluation set the authors had already inspected, it shows that covering combinations helps here, not that one particular set of combinations is best; results also depend somewhat on the solver’s time step.

  • Smaller 1D studies show three things: attributions computed by editing the field instead of running the action differ substantially from the solver reference; models that predict the no-action field almost perfectly can still have large attribution error, which drops once training includes action combinations; and the model’s predicted sign changes hold up under solver time-step checks, though models before and after the fix predict the same events, so the fix is not shown to find anything new.

  • A 2D smoke-flow study is a negative case: after adding action combinations, every continued model had higher held-out effect error than the starting model, even though training loss fell and predicted fields improved on training trajectories. In a separate test, the difference loss looked slightly better on average, but its confidence intervals included zero and other metrics got worse. The authors take this as evidence that better field prediction does not guarantee better effect prediction.

Critical strengths

In the heat conduction experiment, the improvement can be attributed to the new loss: all models use exactly the same training data, and the standard model is even trained slightly longer, yet the new loss still gives better results on new test materials. The authors also report the cases where their method does not help, such as smoke flow (Section 4.4), which gives a more honest picture of the results.

Critical weaknesses
  1. The paper does not discuss how its approach relates to counterfactual explanations. Its queries ask what the output would have been for a fixed, known starting state if a specific action had been applied. This is a counterfactual (what-if) query, closely related to counterfactual explanations [1,2], which ask the inverse question of which input change would produce a desired output. Similarly, the proposed response supervision trains the model on the difference between the output with and without an action for the same starting state, which is closely related to training on counterfactual pairs [3,4]. However, the paper never mentions counterfactuals: its related work (Section 5 and Appendix A) covers removal-based explanations, causal and interventional Shapley values, interventionally consistent surrogates, explanation-guided learning, derivative supervision and active learning, but does not discuss counterfactual explanations or training on counterfactual examples.

  2. The paper cites methods closely related to its two forms of feedback but does not compare against them where its main claims are made. Sensitivity-constrained FNOs [5] regularize how a model’s output changes with its input parameters, which is closely related to the response loss proposed for heat conduction, yet they are never run as a baseline.

  • The paper distinguishes its finite-amplitude contrast supervision from derivative supervision (Appendix A, Table 5) but does not test that distinction.

  • Derivative supervision appears only in the separate 1D study (Appendix C.2), as the "Sensitivity" setting in Table 10, where it gives the lowest Shapley and IG errors of all settings for every architecture at about five times the training cost, and in a follow-up that is reported to lose to time-matched ordinary training; the heat-conduction experiment does not include it.

  • Active learning for neural PDE solvers [6] addresses which simulations to add to training, the question the cylinder coverage study asks, but it is cited only in Related Work (Section 5) and never compared in the cylinder study: the cylinder training sets (Tables 3, 12, 13) are the authors’ own fixed, random and continuous choices of action combinations, with no uncertainty- or diversity-based selection baseline. The only uncertainty-based selectors in the paper are in a 1D study (Appendix B.5) that tests loss functions, not selection rules. Without these comparisons, the paper shows that its feedback helps compared to its own baselines, but not that it helps compared to existing methods.

  1. The abstract’s claim ('support their improvement for scientific inquiry') goes beyond the evidence. The only feedback result with a prespecified, independent confirmation is one small heat-conduction problem (32×32 grid, four actions). The cylinder comparisons are described by the authors as 'descriptive' (line 373), and random on/off menus perform as well as the menu chosen from the diagnosis (lines 365–367), so the diagnosis may have motivated adding combinations, but the specific menu it suggested gave no advantage over random ones. Similarly, in smoke the diagnosis-targeted menu did not outperform random or structural menus (Appendix G.1, Table 27). The abstract also omits that the feedback fails in smoke (Section 4.4). This matters because the paper’s central claim, that checking responses guides useful feedback, is clearly supported only in one small setting.

[1] Verma, Sahil, et al. "Counterfactual explanations and algorithmic recourses for machine learning: A review."

[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR."

[3] Teney, Damien, Ehsan Abbasnedjad, and Anton van den Hengel. "Learning what makes a difference from counterfactual examples and gradient supervision."

[4] Kaushik, Divyansh, Eduard Hovy, and Zachary C. Lipton. "Learning the difference that makes a difference with counterfactually-augmented data."

[5] Behroozi, Abdolmehdi, Chaopeng Shen, and Daniel Kifer. "Sensitivity-constrained fourier neural operators for forward and inverse problems in parametric differential equations."

[6] Musekamp, Daniel, et al. "Active learning for neural PDE solvers."

Questions for authors
  • How does your approach relate to counterfactual explanations [1,2] and to training on counterfactual pairs [3,4], and why are these lines of work not discussed?

  • Why does the feedback fail in smoke, and how would a user know in advance whether it will help?

  • In cylinder flow, the test uses on/off actions at a fixed amplitude, but only the H, S and B training sets use that input type; the controls R and Rφ use continuous coefficients. How do you separate the gain from covering more combinations from the gain of simply training on the same input type as the test?

  • Could you compare against derivative supervision (sensitivity-constrained FNOs) in the heat-conduction experiment, and against an active-learning method for choosing which simulations to add in the cylinder experiment? Both are cited in the related work, but neither is tested in these experiments.

Reviewer_ropROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a method to improve the accuracy that neural surrogate models predict how physical interventions change an outcome. Specifically, in addition to the normal regression loss, the authors propose to additionally penalize the differences between the pre-intervention and post-intervention outcomes. Simulation results on several PDE systems demonstrate the proposed method improves the prediction accuracy of interventions.

Critical strengths
  • The paper investigates an important problem of prediction accuracy under interventions
Critical weaknesses
  • Although executable intervention games are presented as the central framework of the paper, their role in the proposed training methods is unclear. The main improvements arise from supervising pairwise differences between intervention outcomes or expanding training coverage across intervention combinations, neither of which appears to inherently require the game-theoretic formulation or Shapley attributions.
  • The proposed method doesn’t differentiate itself clearly from previous works. In particular, the proposed loss seems to very similar to pairwise difference regression objectives from existing works such as Gradient Aligned Regression (Zhu et al., ICML 2025), which appears to be missing from the discussion.
  • The proposed method is also very similar to Sobolev training, which this paper mentions but does not discuss the differences with the proposed method.
  • The experiments primarily compare variations of the authors' own training procedure, including ordinary field training, absolute-output supervision, and difference supervision. While these are useful ablations, they do not establish how the proposed approach performs relative to existing pairwise regression, derivative-supervised, or interventionally consistent surrogate methods.
Questions for authors
  • Could the authors clarify which components of the proposed training methodology specifically depend on the intervention-game formulation?
  • (5) appears mathematically similar to the pairwise difference losses used in GAR (Zhu et al., ICML 2025). Could the authors clarify the precise novelty of their supervision objective relative to existing pairwise difference learning methods?
  • Could the authors clarify the theoretical and practical distinctions between their finite intervention-response loss and existing derivative- or sensitivity-supervised neural operators (e.g., Behroozi et al., 2025)? In particular, when should one expect the proposed formulation to be preferable?
  • Why does the empirical evaluation not include representative existing pairwise regression or sensitivity-based supervision methods? Could the authors provide such comparisons, or explain the specific challenges preventing meaningful comparisons?
  • Since a key difference from generic pairwise learning is the use of physically defined intervention pairs, could the authors compare against a baseline using randomly selected pairs with the same labels and computational budget? This would help determine whether the observed gains arise from intervention-aware pairing or from pairwise supervision more generally.
Reviewer_txLTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on explainable AI for physical systems. In particular, that when XAI is used to understand models, there is no guarantee, and it is often not the case, that hypothesis derived from the model correspond to anything real.

To try and address this, the authors introduce so called “executable intervention games”. The framing is as follows: In addition to the model under evaluation, we also have a numerical reference, which essentially is a physics based model, and we compare the 2 under various interventions. Essentially, the model is being fine tuned on the numerical reference, to ensure it behaves similarly under various actions.

Critical strengths

The results of the paper are solid in section 4, and I am left with little doubt that these intervention games are indeed a good idea, and lead to better models and better explanations. Overall, the paper was well written

Critical weaknesses

Overall, the framing of the paper is confusing. The main idea is: if we take a model and a reference, and make sure they match under various actions, the model will be more interpretable. This is true, but the XAI angle is really just a byproduct of the model being better. There is no interpretability insight beyond: models that have a higher fidelity to physics will have more physically correct explainability. Thus, the XAI framing is strange.

Further, if we are restricted to problems where we have a numeric reference, I am not really sure what the novelty is. Its quite obvious that, if for a given problem we have an oracle, a model trained on static data will be worse than one that can dynamically ask the oracle questions and measure how it itself differs.

Also: if this method only works when a numeric reference is available, then what exactly is the point? When I read the abstract, I was expecting a scenario where models were trained on some real data, and a numeric apprxomation exists, and we use it to get higher fidelity on the original data. But the experiments in this paper are far more limited. The simulator in all these cases IS the ground truth!

Questions for authors

Can you evaluate your method on a case when the numeric reference is not the ground truth? Can you compare your method with standard benchmakrs from the active learning literature?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.