- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This draft proposes a new way of network interpretability on texture data. This study is more about a theory study using statistics terms, I will try to summarize the proposed method in plain words but there could be some parts I misunderstood (listed in the question section).
Previous interpretability methods analyze a model based on the trained weight and the input data, but the meaning of the learned weight is mostly unclear. This study proposes to assume an “idealized” stochastic process which can generate class-specific data perfectly, then training a network to estimate the class-specific statistics of the stochastic process. In this way, the network is interpretable because it is actually learning towards the idealized statistics(which I suppose the statistics is human readable or designed). However, real data(observation) may not satisfy the idealized model, a PAC boundary (under a specified error tolerance to achieve a confidence of…) is applied to enforce the updated weight (trained on the data) will not be too far away from the interpretation(idealized interpretable weight). Finally, the PAC bounded learned weight can fit on the data (creating similar texture) meanwhile sticks with the initial interpretable statistics(init weight). Overall, this draft considers this XAI problem as a generative process instead of classifier. The statistics (features) for creating a class-specific data is known,
This draft thoroughly introduced a series of assumptions for the proposed method. However, one main problem is about the motivation behind the design.
Critical strengths
- It is important to understand trained networks, this draft attempts to propose a new way to interpret a learned model.
- This draft is a theory study, a series of assumptions are made to explain how the algorithm works in a general scenario, Section 2. Then this idea is realized in the case of texture learning, Section 3. I personally like this stage by stage introduction, but the focus seems to be a little skewed, please see the weaknesses below.
Critical weaknesses
I will summarize general weaknesses in this section, detailed questions for each point are listed in the next section.
-
Although the overall writing looks good, this draft squeezes too much stuff into the appendix, page 14-37. I tried to read most of the appendix, but I might still miss some important information. If the page limit is a problem to this study, I highly recommend submitting this draft to a theory journal.
-
Following the above weakness, I like how this draft introduces each assumption and step, but the proposed interpretation is not clear (maybe due to the limited space) even on the texture task.
-
The applicability of this method is questionable. This draft uses texture as an example, the pre-defined scattering paths are used as the idealized statistics. However, I highly doubt if such statistics exist for most real world tasks. Let’s set the most basic object classification as an example, it is highly questionable if we have or we can define such statistics to create dog images. This is why it is common to let a network learn from data for generation or classification then we try to understand the freely learned features.
-
Regarding those assumptions, some of the important ones could be too strong. Section 3.2 and Figure 1, the “constrained” (by the norm) learned weight can still generate class-specific texture, because the task of texture could be relatively easy which can be described by human designed statistics. When fitting the network on data, this draft applies PAC bound to accept their “idealized” hypothesis. This PAC boundary seems to be based on the assumption of line 195, “these sets are separated across classes, so that the statistics s(X) are sufficient to determine the class of the generating process.” This assumption could also be too strong to be applied on real world data, which could lead to a weaker interpretation from the initialized statistics.
-
Let’s consider this draft is purely about theory (omitting applicability) and all of those assumptions are acceptable. Then one main problem in this draft is about the general motivation. For any network-based systems or tasks(not limited to texture), if we know there exist “almost” idealized statistics, then we already have the almost perfect understanding of the task or data. On the contrary, the rise of connectionism is because we cannot design such “idealized” features, freely learned features work way better than human crafted statistics. This is also the reason why it is criticized as black-box so we need to understand it. If we start with “not perfect” realization, e.g., human designed visual features Local Binary Patterns, and initialize a network with such designed patterns. Then we limit the network to not go too far from our design based on the norm, can we use the initialized designed weight to explain the network? Or only when the learned network can do a decent job(generating the same texture)? Or does it mean we already know what we want but implemented in a network? This seems to be a little like circular logic. This could also be the philosophy question behind the machine-learned vs human-crafted feature argument long-time ago.
Questions for authors
I will list detailed questions in this section, please correct me if I am wrong.
-
Line 195, “these sets are separated across classes, so that the statistics s(X) are sufficient to determine the class of the generating process.” Assuming the classes are separable, then the fine-tuned network can be interpreted under Corollary 2.1, Eq (5). In reality, most complex tasks do not hold for the separation assumption, PAC may still hold(model could converge), but the initialized statistics could be less representative or interpretable(not that ideal), am I right?
-
This draft uses texture as an example because texture can be viewed as a stationary field, line 306, which indicates the texture pattern repeats at a or some frequencies, am I right? If so, let’s theoretically extend this to other generic objects, a mixture of ergodic processes can be used to describe other objects, am I right?
-
Section 3.2, the wavelet scattering transforms from (Mallat, 2012) are used as the “idealized” texture statistics, which are initialized into ScatNet (Zarka 2021) for fine-tuning, am I right?
-
I will list the most important “philosophy” question here for discussion. For most problems, we do not have “good” statistics(not good enough), that’s why we use the network to learn features. Human designed features are worse than the black-box so we need to interpret the black-box. Let’s say one day we have perfect idealized statistics, then do we still need a blackbox? In this paper, if we have an almost idealized statistics, and fine-tuning gives us something more, do we want to interpret the incremental learned part(no matter more or less), or we are only interested in the known part, the starting point? If a human-designed feature, say Local Binary Patterns (LBP) is works ok or almost perfect, then we constrain a network to stick with LBP, not surprisingly we end up with something similar to LBP(by the norm), and claim LBP works or is the interpretation of the success of the network (though we knew this from beginning and that’s why we chose it). The point is the thing we always try to interpret is the freely learned incremental part from a network, please correct me if I am wrong.
11 comments
Response to reviewer (Part 2)
Response to Weakness 3
Reviewer Comment: “The applicability of this method is questionable. This draft uses texture as an example, the pre-defined scattering paths are used as the idealized statistics. However, I highly doubt if such statistics exist for most real world tasks. Let’s set the most basic object classification as an example, it is highly questionable if we have or we can define such statistics to create dog images. This is why it is common to let a network learn from data for generation or classification then we try to understand the freely learned features.”
Response: We would like to clarify that our framework does not require the idealized model to accurately represent real-world data or the initial network to achieve high classification accuracy. Rather, the purpose of the initial model is to provide a meaningful and interpretable inductive bias, which can subsequently be improved through fine-tuning. Indeed, accommodating the mismatch between an idealized model and real observations is one of the main motivations for introducing controlled fine-tuning in our framework.
Furthermore, classical computer vision provides examples of explicitly designed models for natural object recognition, such as part-based object detection models, e.g., Felzenszwalb et al., 2010, which use explicitly designed part-based representation and geometric constraints. Such models illustrate that meaningful algorithmic priors can be constructed even for complex visual tasks, although extending our specific framework to them requires further investigation.
We also wish to clarify that our framework concerns classification rather than image generation. We do not require the predefined statistics to be sufficient for synthesizing realistic images, but rather to provide useful features for distinguishing classes.
We agree that extending the framework to natural images is an interesting and important direction, and we are in fact currently developing exactly such an approach. Nevertheless, in research, one has to start with the simpler case, and the present paper focuses specifically on texture classification, where the scattering transform provides an explicit and mathematically motivated choice of statistics. We do not claim to have established yet the applicability of the framework to general natural-image classification in this work, and we will clarify this distinction and discuss the potential extensions in the revised manuscript.
References:
- Felzenszwalb, Pedro F. and Girshick, Ross B. and McAllester, David and Ramanan, Deva, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence 2010.
Response to reviewer (Part 3)
Response to Weakness 4
Reviewer Comment: Regarding those assumptions, some of the important ones could be too strong. Section 3.2 and Figure 1, the “constrained” (by the norm) learned weight can still generate class-specific texture, because the task of texture could be relatively easy which can be described by human designed statistics.
Response: In Figure 1, the texture generation examples are used to visualize that indeed the scattering transform can be seen as a statistics extractor, where the statistics capture some visual pattern of the same class of texture. However, we would like to clarify that, in general, we do not assume/claim that the statistics obtained by the extractor can be used to generate the relevant objects. Rather, we only assume, in our “idealized” stochastic data model, that these statistics can be beneficial to discriminate classes. Having such statistics doesn’t necessarily imply that we can generate the relevant objects by matching them.
Reviewer Comment Cont’: When fitting the network on data, this draft applies PAC bound to accept their “idealized” hypothesis. This PAC boundary seems to be based on the assumption of line 195, “these sets are separated across classes, so that the statistics s(X) are sufficient to determine the class of the generating process.” This assumption could also be too strong to be applied on real world data, which could lead to a weaker interpretation from the initialized statistics.
Response: The PAC bound is not at all based on the assumption of line 195, “these sets are separated across classes, so that the statistics are sufficient to determine the class of the generating process.” In the paper, we do not assume the real world data to be ideal at all. In fact, we explicitly say in line 255, “One motivation of fine-tuning is that is constructed under an idealized stochastic model, whereas real data need not satisfy this model exactly. Some adaptation is therefore desirable to compensate for this mismatch.” In the idealized model, we have discriminative statistics. And then we adapt to real data via fine-tuning.
Response to reviewer (Part 1)
1. Response to Weakness 1
Response: We thank the reviewer for this suggestion. In ICLR papers, it is very standard to have a long appendix, and our appendix is not unusually long compared to other theoretical papers.
To clearly illustrate our framework, we prioritized presenting its main ideas, theoretical foundations, and experimental results in the main text, while moving detailed proofs, additional experiments, and implementation details to the appendix due to the conference page limit. We also include a table of content in the appendix to help guide the reader.
If accepted, we will use the additional page allowance to further expand the motivation, explanations, and discussion of our framework.
2. Response to Weakness 2
Response: We will clarify the proposed interpretation in the revised manuscript. The central idea of specific algorithmic interpretability is that a particular realization of the network parameters corresponds to an explicit, interpretable algorithm or computation. In our texture experiments, the initial network implements the wavelet scattering transform, whose filters have known scales and orientations and whose coefficients estimate statistics of an underlying stationary process. By controlling the deviation from this initialization during fine-tuning, we aim to approximately preserve this algorithmic interpretation while improving classification performance. Figures 9–11 in Appendix C.3.4 further illustrate the extent to which the learned filters retain their wavelet-like structure. We will make this connection more explicit in the main text. We will add the discussion and refer to this appendix section in the main text of the revised manuscript.