- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an interpretable DSSM (IntDSSM) whose state includes expert-defined concepts. The concept dynamics combine a user-specified mechanism derived from domain knowledge with a neural network that compensates for the residuals. The paper claims that the design of IntDSSM improves forecasting accuracy and demonstrates through experiments that it enables interventions.
Critical strengths
The design in Section 3 is rigorous and well-founded, both in accounting for the concept closure gap and in compensating for the discrepancy between the user-specified mechanism and the mechanism of the real system.
Critical weaknesses
- The paper claims that IntDSSM can handle interventions, but in both the definition in Section 3.3 and the experiments in Section 4.3, the learned model is corrected with the ground-truth concept values or mechanism, which is closer to data assimilation than to intervention. In general, an intervention means that, if we change the concepts or the mechanism of both the true system and the learned model, the learned model should be able to capture the resulting change in the true system. IntDSSM may fail to handle this, since its latent dynamics involve both c_t and h_t, whereas an intervention can only change c_t or the mechanism f_ϑ, which is insufficient without also updating h_t.
- The core idea of IntDSSM is very similar to hybrid (physics + neural) models, which combine a predefined physics function with a neural network to fit the mechanism of the true system. The paper should discuss these related works and add an ablation that simply removes the physics function, to show how much the physics component contributes in this hybrid design.
Questions for authors
- How should "interpretability" be understood in this paper? Does it mean that the learned representations (concepts) carry specific semantic information, that the learned dynamics are operable (e.g., via interventions), or that they incorporate domain knowledge?
- Access to concepts is in fact a strong assumption. Some hybrid models, such as the physics-integrated VAE (PI-VAE; Takeishi & Kalousis, 2021), do not make this assumption and only embed the physics function into the framework. How do you view this difference?
- Takeishi & Kalousis (2021). Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling. NeurIPS 2021.
4 comments
Rebuttal 2/2
Q1 – Meaning of interpretability
How should "interpretability" be understood in this paper?
In our paper, interpretability must be understood as semantic alignment between the model and the human. It requires that (1) the model and the human associate the same meaning with the same symbols (D1), and (2) the mechanisms are tractable for human understanding (D2).
On the three readings proposed by the reviewer:
learned representations (concepts) carry specific semantic information
Yes, at least partially. This corresponds to D1 in our paper.
the learned dynamics are operable (e.g., via interventions)
We see the possibility of performing interventions as a consequence of the alignment above.
incorporate domain knowledge
More generally, the concept dynamics must belong to a hypothesis class that the user deems admissible (D2). This class includes domain knowledge but is not limited to it, as it can also reflect human preferences (see W2).
Q2 – Concept availability
Access to concepts is in fact a strong assumption. Some hybrid models, such as the physics-integrated VAE (PI-VAE; Takeishi & Kalousis, 2021), do not make this assumption and only embed the physics function into the framework. How do you view this difference?
To discuss this point, please allow us to go back to the difference in scope between hybrid models and interpretability (see W2). If the goal is better generalization, it is perfectly fine to supervise only the output and to fix a mechanism from domain knowledge, which adds robustness. If the goal is instead the interpretability of the model's inference process, we remark that concept alignment is a necessary requirement in intrinsic interpretability. Otherwise, the model is free to assign arbitrary semantics to the concepts, and any explanation loses its meaning and reliability, i.e., the language between model and human is different. Crucially, fixing the mechanism is not sufficient to fix the concepts, since in general there exist multiple inputs that produce a given output (as in reasoning shortcuts). This is exactly the reason why we demand two separate desiderata, one for concepts and one for mechanisms.
As further support, concept availability is a standard assumption in concept-based interpretability [1, 4]. Nevertheless, the community is aware that annotating concepts can sometimes be costly, which is motivating active research in automatic concept annotation pipelines [8, 9] (generally referred to as label-free approaches).
We hope to have clarified all of the reviewer's concerns and are happy to provide further details if needed.
References
-
Koh et al., Concept bottleneck models, ICML, 2020.
-
Steinmann et al., Learning to intervene on concept bottlenecks. ICML, 2024.
-
Ismail et al., Concept bottleneck generative models, ICLR, 2024.
-
Poeta et al., Concept-based explainable artificial intelligence: A survey, ACM Computing Surveys, 2025.
-
Goyal et al., Explaining classifiers with causal concept effect (CaCE), arXiv, 2019.
-
Dominici et al., Causal concept graph models: Beyond causal opacity in deep learning, ICLR, 2025.
-
De Felice et al., Causally reliable concept bottleneck models, NeurIPS, 2025.
-
Oikarinen et al., Label-free concept bottleneck models, ICLR, 2023.
-
De Santis et al., Learning concept bottleneck models from mechanistic explanations, ICLR, 2026.
Thank you for the explanations and clarifications. I am not very familiar with concept-based interpretability or semantic alignment. From the perspective of time-series modeling and causal inference, I maintain my earlier assessment: in terms of its framework, the paper is still hybrid modeling that requires ground-truth concepts, and it cannot perform interventions in the causal sense. Nevertheless, this work may well be valuable to the concept-based interpretability community, so I am raising my score from 2 to 3 and lowering my confidence to 2.
We deeply thank the reviewer for the discussion, its honest considerations, and the positive assessment.
Rebuttal 1/2
We thank the reviewer for their review, and address the main points below.
W1 – Interventions
There might be a misunderstanding here, as the term intervention has different interpretations depending on the field. Please allow us to clarify.
The reviewer’s comment follows the meaning in causal inference. There, an intervention acts on a causal model of the data-generating process and its effect is meant to describe the response of the true system. In the field of interpretability, an intervention is a form of user--model interaction in which a user changes the value of one or more of the model's internals (in our case, concepts and/or mechanisms) and observes how the model responds [1, 2, 3, 4]. Interventions allow users to correct a model at test time, steer it toward a desired behavior, or analyse the causal concept effect (in the sense of [5, 6, 7]) on model predictions.
In our experiments, ground-truth interventions are a particular type of intervention where the values simulate the action of an ideal expert. Whether the model also reproduces the response of the true system is a question of causal reliability (see also Q1). Answering it requires a structural causal model, of which the intersection with concept-based models is an active research area even in static settings [6, 7], and beyond our scope.
On this matter, we recognize that Sec. 3.3 used a general heading but then discusses only ground-truth interventions. We have corrected this with a preamble that clarifies the above, and renamed the following paragraph to Ground-truth concept interventions.
We thank the reviewer for highlighting this point, as it improved the clarity of the paper.
W2 – Hybrid models
We agree with the reviewer that hybrid physics–neural models, such as PI-VAE, share some methodological aspects with our approach. Yet, the two approaches fundamentally differ in scope. Hybrid models target world–model alignment: they inject physical knowledge about the real-world phenomenon into the model, so that the model generalizes better. Interpretability targets user–model alignment, and accuracy is pursued subject to it. This has consequences for both variables and mechanisms:
Variables. In interpretability, the model concepts need not correspond to physical variables. A user may still choose to reason in terms of physical concepts, but this is not required. Since concepts are typically inferred from high-dimensional inputs, their alignment must be explicitly enforced (D1). Instead, hybrid physics-neural models generally assume the system is measured directly in the variables of interest (alignment is presumed), or these are latent variables learned without supervision, leaving the model's internal states with no agreed semantics with the user, a condition that is instead the starting point of intrinsic interpretability.
Mechanisms. While hybrid models typically assume that the functional form of the physics is known, intrinsic interpretability only requires the concept compositions to belong to a (possibly broad) class of functions (D2). For example, a user who knows nothing about the phenomenon may restrict the mechanism to linear maps, in favor of a simple explanation (as for our experiment with Lorenz–96); concept bottleneck models commonly make this choice for their task predictor [1].
This difference is also reflected in the two distinct, albeit adjacent, research communities: physics-informed machine learning, and intrinsic (concept-based) interpretability.
We have added a dedicated discussion of hybrid physics–neural models to the Related Work section (ll. 459-477). We thank the reviewer for raising this point.
As discussed above, the physical equation in a hybrid model helps the model better match the real world, and thus generalize better. In contrast, the interpretable mechanism helps the user understand the model's behavior (semantic alignment), regardless of whether it mirrors the real world. Incorrect mechanisms are therefore completely acceptable in interpretability, as long as they belong to the class of functions the user understands. For this reason, we believe that an ablation removing the mechanism would not be informative in our setting.