acceptodds
Under review as a conference paper at ICLR 2027

Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures

Abstract

We develop a principled framework for constructing neural networks whose specific parameter realizations admit an explicit algorithmic interpretation. Existing algorithm-inspired architectures can explain the computational structure of a network, yet after standard training the learned parameters need not retain a clear relation to the motivating algorithm. We address this gap as follows. First, we model each data point as a sample of a class-dependent stochastic process and assume that statistics of this process can be estimated from a single sample and these statistics are sufficient to distinguish the classes . We then construct a neural network whose initial parameters exactly implement an algorithm for estimating these statistics, making the network fully interpretable. To account for mismatch between the idealized model and real data, we fine-tune this network while controlling its deviation from the algorithmic initialization. The trained network hence roughly retains the interpretation of the initial network. A PAC-Bayesian analysis yields a uniform generalization bound whose complexity term scales with the fine-tuning radius, providing a statistical motivation for our approach. We instantiate the framework for texture classification using the scattering transform to estimate the discriminative statistics.

Primary area:
interpretability and explainable AI
Keywords:
Interpretable AI, PAC-Bayesian, Texture Classification, Scattering Transform

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UtRtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This draft proposes a new way of network interpretability on texture data. This study is more about a theory study using statistics terms, I will try to summarize the proposed method in plain words but there could be some parts I misunderstood (listed in the question section).

Previous interpretability methods analyze a model based on the trained weight and the input data, but the meaning of the learned weight is mostly unclear. This study proposes to assume an “idealized” stochastic process which can generate class-specific data perfectly, then training a network to estimate the class-specific statistics of the stochastic process. In this way, the network is interpretable because it is actually learning towards the idealized statistics(which I suppose the statistics is human readable or designed). However, real data(observation) may not satisfy the idealized model, a PAC boundary (under a specified error tolerance to achieve a confidence of…) is applied to enforce the updated weight (trained on the data) will not be too far away from the interpretation(idealized interpretable weight). Finally, the PAC bounded learned weight can fit on the data (creating similar texture) meanwhile sticks with the initial interpretable statistics(init weight). Overall, this draft considers this XAI problem as a generative process instead of classifier. The statistics (features) for creating a class-specific data is known,

This draft thoroughly introduced a series of assumptions for the proposed method. However, one main problem is about the motivation behind the design.

Critical strengths
  1. It is important to understand trained networks, this draft attempts to propose a new way to interpret a learned model.
  2. This draft is a theory study, a series of assumptions are made to explain how the algorithm works in a general scenario, Section 2. Then this idea is realized in the case of texture learning, Section 3. I personally like this stage by stage introduction, but the focus seems to be a little skewed, please see the weaknesses below.
Critical weaknesses

I will summarize general weaknesses in this section, detailed questions for each point are listed in the next section.

  1. Although the overall writing looks good, this draft squeezes too much stuff into the appendix, page 14-37. I tried to read most of the appendix, but I might still miss some important information. If the page limit is a problem to this study, I highly recommend submitting this draft to a theory journal.

  2. Following the above weakness, I like how this draft introduces each assumption and step, but the proposed interpretation is not clear (maybe due to the limited space) even on the texture task.

  3. The applicability of this method is questionable. This draft uses texture as an example, the pre-defined scattering paths are used as the idealized statistics. However, I highly doubt if such statistics exist for most real world tasks. Let’s set the most basic object classification as an example, it is highly questionable if we have or we can define such statistics to create dog images. This is why it is common to let a network learn from data for generation or classification then we try to understand the freely learned features.

  4. Regarding those assumptions, some of the important ones could be too strong. Section 3.2 and Figure 1, the “constrained” (by the norm) learned weight can still generate class-specific texture, because the task of texture could be relatively easy which can be described by human designed statistics. When fitting the network on data, this draft applies PAC bound to accept their “idealized” hypothesis. This PAC boundary seems to be based on the assumption of line 195, “these sets are separated across classes, so that the statistics s(X) are sufficient to determine the class of the generating process.” This assumption could also be too strong to be applied on real world data, which could lead to a weaker interpretation from the initialized statistics.

  5. Let’s consider this draft is purely about theory (omitting applicability) and all of those assumptions are acceptable. Then one main problem in this draft is about the general motivation. For any network-based systems or tasks(not limited to texture), if we know there exist “almost” idealized statistics, then we already have the almost perfect understanding of the task or data. On the contrary, the rise of connectionism is because we cannot design such “idealized” features, freely learned features work way better than human crafted statistics. This is also the reason why it is criticized as black-box so we need to understand it. If we start with “not perfect” realization, e.g., human designed visual features Local Binary Patterns, and initialize a network with such designed patterns. Then we limit the network to not go too far from our design based on the norm, can we use the initialized designed weight to explain the network? Or only when the learned network can do a decent job(generating the same texture)? Or does it mean we already know what we want but implemented in a network? This seems to be a little like circular logic. This could also be the philosophy question behind the machine-learned vs human-crafted feature argument long-time ago.

Questions for authors

I will list detailed questions in this section, please correct me if I am wrong.

  1. Line 195, “these sets are separated across classes, so that the statistics s(X) are sufficient to determine the class of the generating process.” Assuming the classes are separable, then the fine-tuned network can be interpreted under Corollary 2.1, Eq (5). In reality, most complex tasks do not hold for the separation assumption, PAC may still hold(model could converge), but the initialized statistics could be less representative or interpretable(not that ideal), am I right?

  2. This draft uses texture as an example because texture can be viewed as a stationary field, line 306, which indicates the texture pattern repeats at a or some frequencies, am I right? If so, let’s theoretically extend this to other generic objects, a mixture of ergodic processes can be used to describe other objects, am I right?

  3. Section 3.2, the wavelet scattering transforms from (Mallat, 2012) are used as the “idealized” texture statistics, which are initialized into ScatNet (Zarka 2021) for fine-tuning, am I right?

  4. I will list the most important “philosophy” question here for discussion. For most problems, we do not have “good” statistics(not good enough), that’s why we use the network to learn features. Human designed features are worse than the black-box so we need to interpret the black-box. Let’s say one day we have perfect idealized statistics, then do we still need a blackbox? In this paper, if we have an almost idealized statistics, and fine-tuning gives us something more, do we want to interpret the incremental learned part(no matter more or less), or we are only interested in the known part, the starting point? If a human-designed feature, say Local Binary Patterns (LBP) is works ok or almost perfect, then we constrain a network to stick with LBP, not surprisingly we end up with something similar to LBP(by the norm), and claim LBP works or is the interpretation of the success of the network (though we knew this from beginning and that’s why we chose it). The point is the thing we always try to interpret is the freely learned incremental part from a network, please correct me if I am wrong.

11 comments
AuthorsOpenReview ↗

Response to reviewer (Part 1)

1. Response to Weakness 1

Reviewer Comment: “Although the overall writing looks good, this draft squeezes too much stuff into the appendix, page 14-37. I tried to read most of the appendix, but I might still miss some important information. If the page limit is a problem to this study, I highly recommend submitting this draft to a theory journal.”

Response: We thank the reviewer for this suggestion. In ICLR papers, it is very standard to have a long appendix, and our appendix is not unusually long compared to other theoretical papers.

To clearly illustrate our framework, we prioritized presenting its main ideas, theoretical foundations, and experimental results in the main text, while moving detailed proofs, additional experiments, and implementation details to the appendix due to the conference page limit. We also include a table of content in the appendix to help guide the reader.

If accepted, we will use the additional page allowance to further expand the motivation, explanations, and discussion of our framework.

2. Response to Weakness 2

Reviewer Comment: “Following the above weakness, I like how this draft introduces each assumption and step, but the proposed interpretation is not clear (maybe due to the limited space) even on the texture task.”

Response: We will clarify the proposed interpretation in the revised manuscript. The central idea of specific algorithmic interpretability is that a particular realization of the network parameters corresponds to an explicit, interpretable algorithm or computation. In our texture experiments, the initial network implements the wavelet scattering transform, whose filters have known scales and orientations and whose coefficients estimate statistics of an underlying stationary process. By controlling the deviation from this initialization during fine-tuning, we aim to approximately preserve this algorithmic interpretation while improving classification performance. Figures 9–11 in Appendix C.3.4 further illustrate the extent to which the learned filters retain their wavelet-like structure. We will make this connection more explicit in the main text. We will add the discussion and refer to this appendix section in the main text of the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer (Part 2)

Response to Weakness 3

Reviewer Comment: “The applicability of this method is questionable. This draft uses texture as an example, the pre-defined scattering paths are used as the idealized statistics. However, I highly doubt if such statistics exist for most real world tasks. Let’s set the most basic object classification as an example, it is highly questionable if we have or we can define such statistics to create dog images. This is why it is common to let a network learn from data for generation or classification then we try to understand the freely learned features.”

Response: We would like to clarify that our framework does not require the idealized model to accurately represent real-world data or the initial network to achieve high classification accuracy. Rather, the purpose of the initial model is to provide a meaningful and interpretable inductive bias, which can subsequently be improved through fine-tuning. Indeed, accommodating the mismatch between an idealized model and real observations is one of the main motivations for introducing controlled fine-tuning in our framework.

Furthermore, classical computer vision provides examples of explicitly designed models for natural object recognition, such as part-based object detection models, e.g., Felzenszwalb et al., 2010, which use explicitly designed part-based representation and geometric constraints. Such models illustrate that meaningful algorithmic priors can be constructed even for complex visual tasks, although extending our specific framework to them requires further investigation.

We also wish to clarify that our framework concerns classification rather than image generation. We do not require the predefined statistics to be sufficient for synthesizing realistic images, but rather to provide useful features for distinguishing classes.

We agree that extending the framework to natural images is an interesting and important direction, and we are in fact currently developing exactly such an approach. Nevertheless, in research, one has to start with the simpler case, and the present paper focuses specifically on texture classification, where the scattering transform provides an explicit and mathematically motivated choice of statistics. We do not claim to have established yet the applicability of the framework to general natural-image classification in this work, and we will clarify this distinction and discuss the potential extensions in the revised manuscript.

References:

  1. Felzenszwalb, Pedro F. and Girshick, Ross B. and McAllester, David and Ramanan, Deva, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence 2010.
AuthorsOpenReview ↗

Response to reviewer (Part 3)

Response to Weakness 4

Reviewer Comment: Regarding those assumptions, some of the important ones could be too strong. Section 3.2 and Figure 1, the “constrained” (by the norm) learned weight can still generate class-specific texture, because the task of texture could be relatively easy which can be described by human designed statistics.

Response: In Figure 1, the texture generation examples are used to visualize that indeed the scattering transform can be seen as a statistics extractor, where the statistics capture some visual pattern of the same class of texture. However, we would like to clarify that, in general, we do not assume/claim that the statistics obtained by the extractor can be used to generate the relevant objects. Rather, we only assume, in our “idealized” stochastic data model, that these statistics can be beneficial to discriminate classes. Having such statistics doesn’t necessarily imply that we can generate the relevant objects by matching them.

Reviewer Comment Cont’: When fitting the network on data, this draft applies PAC bound to accept their “idealized” hypothesis. This PAC boundary seems to be based on the assumption of line 195, “these sets are separated across classes, so that the statistics s(X) are sufficient to determine the class of the generating process.” This assumption could also be too strong to be applied on real world data, which could lead to a weaker interpretation from the initialized statistics.

Response: The PAC bound is not at all based on the assumption of line 195, “these sets are separated across classes, so that the statistics s(X)s(X) are sufficient to determine the class of the generating process.” In the paper, we do not assume the real world data to be ideal at all. In fact, we explicitly say in line 255, “One motivation of fine-tuning is that Θ0\Theta_0 is constructed under an idealized stochastic model, whereas real data need not satisfy this model exactly. Some adaptation is therefore desirable to compensate for this mismatch.” In the idealized model, we have discriminative statistics. And then we adapt to real data via fine-tuning.

Reviewer_8zGWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a method for “specific algorithmic interpretability”. They initialize a network to represent an explicit statistical algorithm and restrict parameter movement such that the trained network approximately maintains that interpretation.

Critical strengths
  1. The paper distinguishes between understanding an architecture and understanding a model
Critical weaknesses
  1. The result that the trained network maintains the representation is not established
  2. The novelty of the paper is questionable
  3. The numerical results are weak and are not strongly supporting the theory
  4. The connection between parameter proximity and interpretability is not clear
Questions for authors

Why do you exclude parameter scattering networks in the main comparison?

7 comments
AuthorsOpenReview ↗

Response to reviewer (Part 1)

Response to Weakness 1

Reviewer Comment: “The result that the trained network maintains the representation is not established.”

Response: We would like to clarify that our claim is not that the trained network exactly reproduces the chosen initial algorithm, but that it approximately retains its algorithmic interpretation. Indeed, fine-tuning is intended to adapt the original algorithm to real data, so some changes in its parameters and representations are expected. In the case study of texture classification, the retained interpretability is supported by empirical evidence. Appendix C.3.4 (Figures 9–11) directly examines the learned filters, measuring their zero-frequency components, their similarity to Morlet wavelets, and changes in the overall filter bank. The results demonstrate that controlled fine-tuning preserves important wavelet properties, with stronger regularization generally improving this preservation. We will add a discussion on these interpretation preservations in the main text of the revised manuscript, as this point did not receive enough focus in the submitted version. To further strengthen this evidence, we can add visualizations of the trained filters, illustrating their wavelet-like structure and band-pass behavior. These additions will make the sense in which the trained network retains its original algorithmic interpretation more explicit, while clarifying that this interpretation is approximate rather than exact.

AuthorsOpenReview ↗

Response to reviewer (Part 2)

Response to Weakness 2

Reviewer Comment: “The novelty of the paper is questionable.”

Response: We strongly disagree that the paper lacks novelty, and would appreciate clarification regarding which of its contributions the reviewer considers already established.

The central novelty of our work is a mathematically grounded framework for constructing specifically interpretable neural networks by fine-tuning an explicit statistical algorithm, rather than a previously trained model. Starting from a stochastic model of pattern recognition, we identify class-discriminative statistics in an idealized model, and construct a neural network whose parameters implement an algorithm for estimating these statistics. We then fine-tune this network while controlling its deviation from the original algorithm, thereby combining an explicit algorithmic interpretation with data-driven adaptation.

This is fundamentally different from conventional fine-tuning, which typically starts from a pretrained network whose parameters are obtained through large-scale data training and treated completely as a black box. Our initialization is instead derived mathematically, independently of the training data, and requires no pretraining. This distinction also motivates our theoretical analysis. Previous studies have established distance-based generalization bounds for fine-tuning (Li et al., 2018; Gouk et al., 2021; Li & Zhang, 2021), primarily in the context of transfer learning. However, these analyses are not directly tailored to our framework of algorithmically constructed networks. To provide theoretical support for our specific construction, we derive a uniform PAC-Bayesian classification-margin bound over a Euclidean neighborhood of a data-independent initialization, under a local parameter-stability condition (Corollary 2.1). We further verify this condition for our scattering-based construction in Appendix B.2, thereby connecting the generalization analysis to our specifically interpretable network.

We would welcome references to specific prior work that the reviewer believes establishes our framework.

References:

  1. Xuhong Li and Yves Grandvalet and Franck Davoine, “Explicit Inductive Bias for Transfer Learning with Convolutional Networks”, ICML 2018.
  2. Henry Gouk and Timothy M. Hospedales and Massimiliano Pontil, “Distance-Based Regularisation of Deep Networks for Fine-Tuning”, ICLR 2021.
  3. Dongyue Li and Hongyang R. Zhang, “Improved Regularization and Robustness for Fine-tuning in Neural Networks”, ICLR 2021.
AuthorsOpenReview ↗

Response to reviewer (Part 3)

Response to Weakness 3

Reviewer Comment: “The numerical results are weak and are not strongly supporting the theory.”

Response: We also strongly disagree with this assessment and would welcome clarification regarding which specific experimental findings the reviewer considers insufficient. Our experiments provide evidence supporting the framework from three complementary perspectives.

Figure 2 demonstrates the benefit of algorithmic initialization and fine-tuning, particularly in low-data regimes. For example, with only 0.5% of EuroSAT training data, our method achieves 64.77% accuracy, compared with 45.40% for random initialization and 27.78% for fixed scattering. This supports the claim that with fine-tuning a predefined model we improve sample complexity. Figure 3 examines the relationship between parameter deviation and generalization, showing that smaller fine-tuning distances are generally associated with smaller generalization gaps, consistent with Corollary 2.1. Furthermore, the regularization ablations in Appendix C.3.2 demonstrate that regularizing toward the interpretable initialization can control the generalization gap while largely preserving classification accuracy, unlike standard weight decay. Finally, Appendix C.3.3–C.3.4 investigates the learned representations and wavelet structure, providing additional evidence of approximate interpretability preservation. Taken together, these experiments support the effectiveness of algorithmic initialization, the theoretical motivation for controlling parameter deviation, and the preservation of algorithmic interpretability.

We would welcome specific clarification regarding which claims the reviewer considers insufficiently supported.

Reviewer_rZTwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces specific algorithmic interpretability of DNNs, and provides a method to construct DNNs whose parameters admit an explicit algorithmic interpretation. This is different from generic interpretability, where only the architecture admits such an interpretation. It is done via initializing the DNN so that it performs identically to a known algorithm, and, to adapt to real data, fine-tuning while controlling its deviation from the initial parameters. A theoretical PAC-Bayesian generalization bound of this setup shows that the bound is tighter when the distance from the initial parameters is smaller. The experiments analyze texture classification using the wavelet scattering transform, considering four different datasets and a variety of training-set sizes, comparing accuracy and generalization gap against other methods, such as the same network trained from random initialization, and fixed scattering.

Critical strengths
  1. Clear framing. The authors distinguish their work from generic interpretability, where only the architecture is interpretable, by considering specific interpretability, where the model weights are interpretable. This could start a new line of work in this area.

  2. Benefit in low-data regimes. The authors' proposed method, fine-tuned scattering, generally beats all other methods in the low-data regime (e.g., see Table 5). This gives practitioners clear guidance: when data is scarce, starting from an interpretable initialization works well.

Critical weaknesses
  1. Interpretability-accuracy tradeoff. The authors argue that the trained network is still interpretable. The network is more interpretable for high values of λ\lambda, i.e., the trained filters stay closer to wavelets (Figure 9), but lower values of λ\lambda, i.e., less interpretable networks, generally yield a higher accuracy value, especially as more training data is used (see Tables 4-10). That is, there is a tradeoff between interpretability and accuracy, and the authors should explicitly discuss this tradeoff in the main text. This lowers the significance of the paper.

  2. Limited novelty. There are existing results very similar to Corollary 2.1. The fine-tuning penalty used here is the well-established L2-SP method (Li et al., “Explicit Inductive Bias for Transfer Learning with Convolutional Networks”, ICML 2018), and distance-from-initialization generalization bounds for fine-tuning exist (Gouk et al., “Distance-Based Regularisation of Deep Networks for Fine-Tuning”, ICLR 2021; Li & Zhang, “Improved Regularization and Robustness for Fine-tuning in Neural Networks”, NeurIPS 2021). The contribution is smaller than claimed, and the authors should cite these works and clarify what Corollary 2.1 adds beyond them.

Questions for authors
  1. Clarification. Relating to the tradeoff between interpretability and accuracy, which λ\lambda do you recommend?
  2. Clarification. What does Corollary 2.1 add beyond the known results mentioned in Weakness 2?
4 comments
AuthorsOpenReview ↗

Response to reviewer (Part 1)

Thank you for the careful evaluation and for recognizing the distinction between generic and specific algorithmic interpretability, as well as the benefits of our approach in low-data regimes.

1. Response to weakness 1 and question 1

Reviewer Comment: “There is a tradeoff between interpretability and accuracy, and the authors should explicitly discuss this tradeoff in the main text. This lowers the significance of the paper.” and “Relating to the tradeoff between interpretability and accuracy, which λ do you recommend?”

Response: We agree that there is an interpretability accuracy tradeoff, and we will discuss it more explicitly in the revised manuscript. Such tradeoffs are common in learning problems, and we do not believe their existence diminishes the significance of our contribution. Our work provides a principled framework for controlling this tradeoff, allowing an overparameterized network to improve upon a predefined interpretable algorithm while approximately preserving its algorithmic interpretation. Figures 9–11 in Appendix C.3.4 provide empirical evidence that the fine-tuned filters retain wavelet-like properties, demonstrating that improved accuracy does not necessarily require abandoning the original algorithmic interpretation. We will discuss it more and reference this empirical evidence in the main text of the revised manuscript. Regarding the choice of λ\lambda, there is no universally optimal value. Stronger regularization favors interpretability and small generalization gap, while weaker regularization allows greater adaptation. In the specific case of scattering transform, we recommend selecting λ\lambda according to the desired tradeoff, using validation accuracy and the degree of preservation of the original filters.

AuthorsOpenReview ↗

Response to reviewer (Part 2)

Response to weakness 2 and question 2

Reviewer Comment: “There are existing results very similar to Corollary 2.1. The fine-tuning penalty used here is the well-established L2-SP method [...] and distance-from-initialization generalization bounds for fine-tuning exist [...]. The contribution is smaller than claimed.” and “What does Corollary 2.1 add beyond the known results mentioned in Weakness 2?”

Response: We thank the reviewer for pointing out these relevant works. We agree that distance-based regularization and related generalization bounds have been studied previously, and we will cite and discuss them in the revised manuscript. However, we would like to clarify that fine-tuning is only one component of our framework, rather than its central contribution, and we use fine-tuning conceptually differently than previous works.

The cited works primarily investigate regularization and generalization in transfer learning, where the initialization is typically obtained by pretraining on a source task. Although their theoretical results can also accommodate data-independent initializations on a target task, they do not address how to construct a network whose particular parameter values admit an explicit algorithmic interpretation. Staying close to a pretrained network may preserve learned representations, but it does not by itself explain those representations in terms of a specific algorithm.

Our work addresses a different question: how can we construct an overparameterized neural network that is interpretable by design, and subsequently improve its performance while approximately preserving its algorithmic interpretation? Starting from a stochastic model of the data, we identify discriminative statistics and construct an explicit algorithm for estimating them. We then realize this algorithm exactly as a neural network, obtaining an interpretable initialization without pretraining. Fine-tuning allows the algorithm to adapt to real data while controlling deviations from its initial realization. Importantly, the initial network is not required to achieve high classification accuracy.

Regarding Corollary 2.1, we do not claim that distance-dependent generalization bounds are new. Li et al. (2018) introduce the L2-SP penalty; Gouk et al. (2021) derive bounds using Rademacher complexity and layerwise norm constraints; and Li & Zhang (2021) establish PAC-Bayesian bounds involving layerwise fine-tuning distances and perturbation sensitivity. However, these results do not directly apply to our specific construction of the architecture, and this is why we had to derive our own bounds to support our method. Our corollary formulates a uniform PAC-Bayesian classification-margin bound over a neighborhood of a data-independent initialization, under a local parameter-stability condition. In the case study of texture, we verify this condition for scattering construction in Appendix B.2, connecting the general bound to our specific algorithmically interpretable network. We will clarify that this result provides theoretical support for controlled adaptation, rather than constituting a fundamentally new distance-based generalization principle. The novelty of our work therefore lies in constructing and training networks with specific algorithmic interpretability, rather than proposing a new fine-tuning training method. The generalization analysis supports this framework by relating controlled parameter adaptation to generalization.

References:

  1. Xuhong Li and Yves Grandvalet and Franck Davoine, “Explicit Inductive Bias for Transfer Learning with Convolutional Networks”, ICML 2018.
  2. Henry Gouk and Timothy M. Hospedales and Massimiliano Pontil, “Distance-Based Regularisation of Deep Networks for Fine-Tuning”, ICLR 2021.
  3. Dongyue Li and Hongyang R. Zhang, “Improved Regularization and Robustness for Fine-tuning in Neural Networks”, ICLR 2021.
Reviewer_rZTwOpenReview ↗

Response to authors' rebuttal

I thank the authors for the detailed answers. I agree that the framing is novel, positioning Corollary 2.1 against existing works helps, and acknowledging the tradeoff is useful. However, the promised changes do not appear in the current PDF, and I still have significance concerns:

  1. Interpretability of the fine-tuned network for low values of λ\lambda is not shown formally.
  2. The performance generally benefits from lower values of λ\lambda, i.e., less interpretable networks.

Finally, I keep my rating at 2.

AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for the follow-up comments. We uploaded the revised manuscript. We agree that weaker regularization may improve classification accuracy while allowing the network to move further from its interpretable initialization. When ֿλֿ\lambda is small, however, the trained parameters may deviate substantially from the original algorithm, and we cannot claim that its algorithmic interpretation is preserved. This is an interpretability–accuracy tradeoff that we now explicitly discuss in Section 4 of the revised manuscript.

We also agree that our general theory does not formally guarantee interpretability after fine-tuning. Such preservation must be examined empirically for each specific construction. In our texture classification experiments, Appendix C.3.4 (Figures 9–11) examines how the learned filters retain their wavelet-like properties under different regularization strengths, providing evidence that stronger regularization generally favors preservation of the original structure. Importantly, we distinguish between the interpretation inherited from the predefined algorithm and the additional modifications introduced through learning, which are not interpretable. We have clarified this distinction in Section 2.4 and acknowledged the limitation in Section 5.

Nevertheless, a small change in parameter space does not necessarily imply a small improvement in classification accuracy. Even when the initial algorithm performs poorly, a nearby model may achieve substantially better performance while retaining important properties of the original algorithm. We do not claim that this favorable outcome is guaranteed. Rather, we view our work as a first step toward specific algorithmic interpretability, where the appropriate choice of λ\lambda depends on the desired balance between accuracy and preservation of the original algorithmic interpretation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviews and rebuttal

We thank the reviewers for their evaluation and discussion. We would like to summarize the main concerns raised during the rebuttal, our responses to them which clarify our contributions, and highlight the revisions made to the manuscript.

  1. Functional closeness and performance improvement. Some reviewers questioned whether keeping the parameters close to initialization implies that the fine-tuned network remains functionally close to the original algorithm, thereby limiting improvements in accuracy. However, in our setting, we do not want the fine-tuned model to be functionally close to the initial model. A small change in parameter space does not necessarily imply a small improvement in classification accuracy. Fine-tuning can move the weights a small distance while significantly improving accuracy, for example, by correcting some boundary cases. Even when the initial algorithm performs poorly, a nearby model may achieve substantially better performance while retaining important properties of the original algorithm. What we aim to preserve is not the original input-output mapping, but the underlying algorithmic approach.

Specifically, in our texture classification case study, the fine-tuned model approximately retains the interpretation of extracting texture statistics, within our stochastic pattern recognition framework, through compositions of wavelet-like filters to separate classes. Appendix C.3.4 (Figures 9–11) provides empirical evidence supporting the preservation of these wavelet-like properties and the organization of the original filter bank.

  1. Circular reasoning and the idealized initialization. Some reviewers argued that starting from an interpretable algorithm and staying close to it is circular, or implicitly assumes that the initial algorithm already performs well. This is not our assumption. The predefined algorithm may perform poorly on real data. Its role is to provide an interpretable initialization with a meaningful inductive bias, from which a better predictor may be reached through fine-tuning.

  2. Interpretability guarantees. Reviewers correctly observed that parameter proximity alone does not a priori guarantee interpretability. We agree with this limitation. Whether important algorithmic properties are retained must be validated empirically for each specific construction. Our texture experiments provide such evidence for wavelet-like filter properties, but we do not claim a universal guarantee. We will hence stress this in the revised paper.

  3. What do we interpret in the trained network? Some reviewers contemplated which part of the trained model we interpret. We clarify that our approach distinguishes between the interpretation inherited from initialization and the additional small modifications introduced through learning, which is not interpreted. Rather than attempting to interpret a fully trained black-box network retrospectively, we construct a network whose interpretation is known from initialization and control how much training modifies it. Our approach for addressing the problem of interpreting overparameterized networks is exactly to interpret it by using the interpretation of the initialization. Specifically, by forcing an initial interpretation, and not allowing the model to change the weights much during training, we end up with an overparameterized model that roughly retains the initial interpretation. And to what extent it retains the initial interpretation must be verified empirically as we mentioned above. In a sense, instead of thinking about the model as fully black box, we white box “most of the model”, and only keep a “minimal” portion of the energy of the weights uninterpretable and trainable.


Revisions and scope of our contribution. In response to the reviewers, we have revised the manuscript to clarify these distinctions, discuss the interpretability–accuracy tradeoff, relate our analysis to prior fine-tuning literature, and make the empirical evidence for preserving algorithmic properties more explicit (Sections 2.4, 3.2, 4, and Appendix C.3.4). We have also explicitly acknowledged in Section 5 that interpretability preservation is not automatic and requires empirical validation. More generally, we view our contribution as a first step toward specific algorithmic interpretability, rather than a complete solution to interpreting all learned modifications. To reflect this more accurately, we will change the title to “Toward Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.