acceptodds
Under review as a conference paper at ICLR 2027

Amortized Shapley Interactions for Neural Operators

Abstract

Explaining model predictions can increase trust in AI systems and provide valuable insights to developers. Feature-based attribution methods, such as Shapley values, are a popular approach to model interpretability. In the physical sciences, however, model inputs often represent continuous domains rather than discrete features. Additionally, their mesh-based discretizations may vary between samples, making fast amortized explanations difficult. Infinite-player Aumann–Shapley values have recently been proposed as an alternative for such continuous inputs, but the existing framework is limited to additive attributions and therefore cannot capture interactions between input regions. In this work, we propose *OperatorSHAP-IQ*, which extends the Aumann–Shapley framework to feature interactions. We further introduce an amortization scheme that enables efficient inference of interaction maps and show how different classes of partial differential equations exhibit different levels of interactions.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Shapley values, Shapley interactions, neural operators, PDE surrogate models, explainable AI, interpretability, operator learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CgvaOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OperatorSHAP-IQ to efficiently estimate spatial interaction maps for neural operators. The work extends Shapley attribution and interaction analysis from discrete features to continuous input fields, and further extends the Aumann–Shapley framework from first-order attribution to higher-order interactions. The proposed amortized explainer predicts interaction fields with a single forward pass after training. Experiments cover 12 PDE datasets across 1D, 2D, and 3D settings, including both regular grids and irregular meshes.

Critical strengths
  1. I think the amortization idea is practically useful. If we need explanations for many samples, training an explainer once and then getting an interaction map with a single forward pass is much more convenient than repeatedly evaluating a large number of coalitions. The experiments are also fairly broad, covering 1D, 2D, and 3D problems, as well as regular grids and irregular meshes.

  2. The paper is also theoretically solid. The connection from finite-player Shapley interactions to continuous interaction fields is developed carefully, and the proposed method follows this theory quite closely. I also like that the experiments include several sanity checks that are consistent with the theoretical expectations.

Critical weaknesses

The scientific value of the explanations is still not very clear to me. Most of the presented examples look more like sanity checks: the linear heat equation has almost no interaction, while the nonlinear Burgers equation shows clear interactions, which is largely expected. What is less clear is what we can actually do with these interaction maps once we obtain them. For example, if the learned interaction pattern is very different from what we expect from the underlying physics, can this information be used to diagnose or improve the neural operator?

Questions for authors

What do you think about the masking strategy for continuous spatial inputs? Hard masking a spatial region with zero or a mean value can introduce artificial discontinuities or OOD states that are not present in the original data. Since Shapley values are very sensitive to the masking baseline, how do you exclude the bias brought by different baseline.

2 comments

Thank you for your very positive review. We are happy that you found our idea useful. As for your remaining questions:

W1: What is the scientific value of explanations?

R1: In general, the value of attributions and interactions lies in sanity checks for the user and in model diagnostics for the developer. As you correctly identified, if an explanation deviates from what we expect from the physics, a user could choose not to trust a prediction and a developer could investigate why their model would behave in such a way. However, amortized explainers are usually not necessary for research-scale problems. Our method aims more for commercial usage, where upcoming regulations could force companies to monitor their models behavior and per-sample predictions would be too costly.

Q1-1: Does hard masking introduce artificial discontinuities?

R2: No, the input stays continuous, but the justification for this was only in the original OperatorSHAP paper and we should add it here as well. The reason is that models usually act on a discretized version of the data. For example, instead of getting a continuous input over the interval [0,1][0,1], the data is only present at locations {0,14,12,34,1}\lbrace0, \frac{1}{4}, \frac{1}{2}, \frac{3}{4}, 1\rbrace. That means, even if we use hard masks on that discrete locations, the models internally assume a smooth transition between the nodes, so no discontinuities are introduced.

Q1-2: How do we exclude bias brought by different baselines?

R3: You are correct that this bias exists and we do not exclude it. The choice of baseline is always a parameter that the user has to decide, as it is in discrete Shapley. As we detail in Appendix B12 and ablate in Table 12, the choice of baseline has an effect on the interpretation of the explanations, but not on their quality. When providing an interpretation, the user always has to report the baseline they used and adjust their interpretation accordingly.

We have updated the manuscript with your suggestions. Thank you again for your recommendation.

Thank you for the response, I will keep my positive evaluation.

Reviewer_xUm3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OperatorSHAP-IQ, extending continuous Shapley attribution to pairwise interactions for neural operators. The authors derive an exact signed-mixture representation for lower-order Faithful Shapley Interaction Indices (FSII), establish a continuum-limit theory for interaction attribution fields, and develop an amortized explainer that supports different spatial resolutions and irregular meshes. Experiments on multiple PDE benchmarks demonstrate competitive coalition-level faithfulness and substantial inference-time speedups.

Critical strengths

1.Extending the signed-mixture representation of FSII to lower-order interactions provides a mathematical foundation for continuous interaction attribution and the subsequent continuum-limit analysis.

2.The method addresses the challenge of explaining neural operators across spatial resolutions and irregular meshes. Its amortized design is particularly useful when explanations are required repeatedly.

3.The experiments cover multiple PDE benchmarks and compare the proposed method with existing interaction estimation approaches, providing evidence of both coalition-level faithfulness and computational efficiency.

Critical weaknesses

1.The method builds on the established continuous-attribution and amortization framework, with its main advances lying in the extension to lower-order FSII and the associated continuum-limit theory. These extensions may constitute meaningful contributions, but the paper would benefit from a clearer distinction between the new theoretical results and what follows directly from the existing framework. As presented, the methodological advance appears incremental rather than a fundamentally new attribution paradigm.

2.The reference coefficients are estimated using ProxySHAP rather than exact computation, making the assessment dependent on the accuracy of the proxy estimates. This uncertainty may become more consequential as the number of players increases, although the effect also depends on the estimation budget and proxy quality. The exact-coefficient results on small grids provide useful validation, but the high-resolution experiments establish coalition-level faithfulness more directly than the accuracy of individual interaction coefficients.

Questions for authors

1.What specific theoretical guarantees or insights does the lower-order FSII signed-mixture representation provide beyond the existing OperatorSHAP framework? In particular, which parts of the derivation and continuum-limit analysis are new, and what do they enable for pairwise interaction attribution that was not established previously?

2 comments
AuthorsOpenReview ↗

Rebuttal to ReviewerxUm3

We thank reviewer xUm3 for their positive review. We will address your remaining concerns, starting with the weaknesses:

W1: The paper should list its contributions more clearly.

R1: We understand this concern. Many different subareas of XAI play together to arrive at the final result and it is hard to discern what was previously known. We added a novelty statement to the paper to clarify our contributions (App. D).

In essence, our main contribution is the formulation of Aumann-Shapley interactions as limit of the Faithful Shapley Interaction Indices (FSII) on refining grids. To the best of our knowledge, only order-1 attributions for Aumann-Shapley existed so far. The amortization algorithm we propose on top of this is indeed incremental, given this extension and the existing OperatorSHAP algorithm. A separate contribution is our extension of FSII as a mixture index for all interaction orders. This was previously only known for the highest interaction order and both Tsai [1] and Fumagalli [2] mention this as an open problem. We use this to proof our main Theorem, but it could potentially be used in other algorithms as we detail in R3.

W2: High-resolution experiments measure faithfulness rather than accuracy of attributions.

R2: Your analysis is correct. We measure exact accuracy only on low player numbers and rely on faithfulness for higher player numbers. This is, however, not a shortcoming of our experimental design. Exact Shapley values can only be generated for low player numbers as coalitions grows like 2n2^n. This is why faithfulness is a standard metric for benchmarking approximators on high player numbers [3,4,5]. As we approximate the FSII, faithfulness even directly measures the quality of the approximation. We additionally measure correlation with a SOTA approximator, as this was done in the original OperatorSHAP paper. However, we argue that this is not a good measure (see App. B.8) in our case, as the extra dimensions from the interactions reduce correlation even between two ProxySHAP approximators.

Q1: What insights does the lower order FSII representation provide, beyond the existing OperatorSHAP framework.

R3: The existing OperatorSHAP framework was only able to provide order-1 attributions, i.e. no interactions between players. The new FSII representation enabled a mesh-refinement construction for interactions. This allowed us to provide order-2 attributions (i.e., interactions) that give more detailed explanations for model behavior. It can also be used to derive arbitrary order algorithms, which we left for future work. There are also implications beyond OperatorSHAP-IQ. Some approximation algorithms are based on the mixture representation, like SHAP-IQ [2] or TreeSHAP-IQ [6]. With the lower order exact weights, new algorithms for the lower order FSII could be developed. We added that as an explicit extension to the text

Thank you again for your positive review. We are happy to answer any follow up questions you might have!

 


[1] Tsai, Che-Ping, Chih-Kuan Yeh, and Pradeep Ravikumar. “Faith-Shap: The Faithful Shapley Interaction Index.” Journal of Machine Learning Research 24, no. 94 (2023): 1–42.

[2] Fumagalli, Fabian, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. “SHAP-IQ: Unified Approximation of Any-Order Shapley Interactions.” In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, edited by Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine. 2023.

[3] Muschalik, Maximilian, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Hammer, and Eyke Hüllermeier. “Shapiq: Shapley Interactions for Machine Learning.” The Thirty-Eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.

[4] Thies, Santo M. A. R., Hubert Baniecki, R. Teal Witter, Eyke Hüllermeier, Maximilian Muschalik, and Fabian Fumagalli. “Proxy-Based Approximation of Shapley and Banzhaf Interactions.” arXiv:2605.22738. Preprint, arXiv, May 22, 2026.

[5] Butler, Landon, Abhineet Agarwal, Justin Singh Kang, Yigit Efe Erginbas, Bin Yu, and Kannan Ramchandran. “ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs.” The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.

[6] Muschalik, Maximilian, Fabian Fumagalli, Barbara Hammer, and Eyke Hüllermeier. “Beyond TreeSHAP: Efficient Computation of Any-Order Shapley Interactions for Tree Ensembles.” Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 13 (2024): 14388–96.

Thank you for the response. I will maintain my positive score.

Reviewer_fo6dOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The Shapley-Schubik cost-sharing method has enjoyed great popularity being applied to ML explainability in the form of "Shapley-values" with various names. This paper claims to extend the Aumann-Shapley (AS) cost-scharing method to interactions of features in the context of neural operators via a novel method, OperatorSHAP-IQ. They detail efficient implementation of amortization schemes to make the method tractable and experiments showing OperatorSHAP-IQ's explanation of neural operators on data.

Critical strengths

I am not sure this paper was written by a human.

Critical weaknesses
  1. The paper does not interact with important works on the topic of extending AS to explanations and feature interactions. A discussion of these methods should be added.
  2. Many methods that extend AS to interactions do so in a model/problem agnostic setting. Why is a unique method for neural operators needed; what does it bring to the table?

Missing works:

First paper to apply aumann-shaply to XAI: Integrated Gradients: axiomatic attribution for deep networks, M Sundatarajan et al.

Extending AS to interactions:

  • "Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models", Sikdar et al.
  • "Aumann-SHAP: The Geometry of Counterfactual Interaction Explanations in Machine Learning", Belahcen and Mussard
  • Integrated Hessians: "Explaining Explanations: Axiomatic Feature Interactions for Deep Networks", Janizek et al.
  • Sum of Powers: "A Unifying Framework to the Analysis of Interaction Methods using Synergy Functions", Lundstrom et al.
Questions for authors
  • Given the existence of at least four other methods for extending AS to interactions, how is the current method unique and what are its advantages? How does its interaction estimations compare to other methods?
  • Can a good argument be made that explaining neural operators requires a special explanation method? Is OperatorSHAP-IQ especially well suited to this?
  • It would be nice to list a set of axioms OperatorSHAP-IQ satisfies and perhaps provide a unique characterization theorem.
  • As far as this reviewer is aware, the game-theoretic cost-sharing literature, of which section 3 attempts to follow, does not use the term "attribution" to describe cost shares. Attribution is a term used for XAI. Can you explain why you used the term "attribution" when discussing game-theoretic cost sharing? The term seems out of place.
1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer fo6d

Dear reviewer, thank you very much for your review. We appreciate your hint to the related literature we missed and hope we can clarify our difference to them. This paper was indeed the work of human authors (with assistance of AI as disclaimed), though we understand your skepticism given the current circumstances in AI research. Let us first clarify a major point that spans your weaknesses and questions.

M1: Novelty and relation to missed literature

R1: We agree that the cited works should be discussed and have added them to the paper. However, the key distinction of our work is that we propose an amortization method, while the other works calculate attributions on a per-sample basis. This distinction affects both W1 and W2, as well as Q1 and Q2.

Weaknesses & Questions

W1: A discussion of the missing related literature should be added.

R2: You are absolutely right and we extended the related literature section accordingly. As mentioned above, we propose an amortization method, not a per-sample approximation of Aumann--Shapley values, which sets us apart from existing work.

As for the work you cited, in a nutshell, Integrated Directional Gradients, Integrated Hessians, and the Augmented Integrated Hessian in Sum of Powers are all derivatives of the "Integrated Gradients" method. They work similarly to our "Limit" method in that they integrate over derivatives of the model. They are quite fast, but can only give good interactions for smooth models as we ablate in Table 8. We also added an ablation in Table 7, showing that our Limit method is a suitable representative for the Integration methods.

Neither Aumann-SHAP nor Integrated Directional Gradients give interactions as we handle in the paper, which is why we omit them from comparisons.

W2 & Q2: Why propose a method specifically for neural operators when existing works are model-agnostic?

R3: Most per-sample methods are indeed model agnostic, so there, a method specifically for neural operators would not be necessary. In the amortized setting, however, existing methods require homogeneous inputs, that express the same features across the whole dataset (e.g., FastSHAP [1] and InstaSHAP [2]). They cannot amortize over datasets where samples are defined over unique meshes, as is often the case in physics and engineering. We propose a method that can amortize over such datasets. As neural operators are a popular class of models in these domains, we chose the title and the framing of the paper around them, though the method is rather tailored to the class of datasets than to neural operators per se.

Q1: How does our method differ from other Aumann--Shapley methods?

R4: As discussed above, we propose an amortization method, while the existing literature only covers per-sample methods. Our main advantage is inference speed, while preserving a competitive performance to powerful sampling methods like ProxySHAP.

While Integration methods like Integrated Hessians are already pretty fast, they may still need orders of magnitude more model evaluations than OperatorSHAP-IQ. Furthermore, they often fail for models with non-differentiable activations (e.g, ReLU), as we show in Table 8. The Integrated Hessians paper suggests replacing every non-differentiable component with differentiable approximations, e.g., the ReLU activations with SoftPlus. We do not think that this is a viable approach, as the replaced model is only an approximation of the original.

Q3: Can you provide a list of axioms or a characterization theorem?

R5: For the axioms: Yes, the paper already establishes them, but they are scattered and a unifying proposition is a good idea. We added this to the paper. For the unique characterization theorem: Finding a set of function space axioms that uniquely define our method is difficult, because even the finite case, the standard Shapley axioms do not uniquely define an interaction index. We agree that such a result would be interesting, but we would not want to rush such a big theorem and, therefore, added it as a future work suggestion to the paper.

Q4: Why use the term "attribution"?

R6: While the foundational work for our paper originated in the cost-sharing literature, we would classify ourselves as an XAI method. The term "attribution" is often used in modern works that are most closely related to us, such as ProxySHAP, OperatorSHAP, or the four sources that you cited in your review.

 


 

[1] Jethani, N., Sudarshan, M., Covert, I. C., Lee, S.-I., & Ranganath, R. (2022). FastSHAP: real-time Shapley value estimation. International Conference on Learning Representations.

[2] Enouen, J., & Liu, Y. (2025). InstaSHAP: interpretable additive models explain Shapley values instantly. In Y. Yue, A. Garg, N. Peng, F. Sha, & R. Yu (Eds.), International Conference on Learning Representations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Discussion Closing Soon

Dear Reviewers,

Thank you again for your feedback on our submission. With the discussion period closing in two days, we wanted to kindly check whether our responses have addressed your concerns. If any questions remain or anything is unclear, we’d be glad to clarify or provide additional results before the deadline.

Kind regards,

The Authors

AuthorsOpenReview ↗

Rebuttal Summary

Dear reviewers,

thank you for your assessment of our work and your engagement in the rebuttal. In response to your remarks, we added the following changes to our submission:

  1. Path-based methods: We updated our related literature with a section on path-based methods. We further added two of them to the ablations in Tables 7 and 8. Our pre-existing Limit method is closely related to the path-based methods. As it can measure interactions and shows superior faithfulness, we keep it as the sole representative for path-based methods in the main table (App. B3).
  2. Axioms Theorem: We summarized some properties of OperatorSHAP-IQ in a new Theorem (Thm. 8). These properties were previously scattered over the manuscript and are now united in the main text.
  3. Novelty statement: To separate our contributions more clearly, we added a novelty statement in Appendix D.
  4. Hard mask clarification: We extended our explanations on the apparent discrepancy between hard and smooth masks in App. B.11.

Kind regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.