- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents an analysis of what is required to successfully identify an underlying neural network from probing information, i.e. from formulating inputs to the network and collecting the corresponding outputs or intermediate layer activations. The main theoretical contribution is to provide an upper bound on how many probes are necessary for identification, both for output vs. intermediate layer probing and separately for MLPs and multi-head attention in Transformer nets. Notably, the probing of hidden layers makes identification possible with significantly fewer probes. Based on this observation, the authors propose HiddenProbe, which is conceptually similar to the previous ProbeGen method, but which makes use of hidden layer information instead of only output information. Experiments on INR classification and accuracy prediction demonstrates how HiddenProbe improves on ProbeGen and weight-based approaches, with fewer probes required to match the performance of ProbeGen.
Critical strengths
-
The theoretical contribution is valuable for increased understanding of what is required for identifying the underlying function from probe-based information, and how this differ when considering only output probing vs. probing of intermediate layers. The paper also extends the theory and experiments to Transformers in addition to MLPs, which makes it more practically relevant.
-
The HiddenProbe formulation and the experiments provide a direct evaluation of the improvements that can be expected in weight space learning when extending probing to hidden layers and when comparing against methods that directly operate on weights.
Critical weaknesses
-
Missing a deeper analysis of the information provided through the hidden layer probe formulation:
- The paper would be stronger if it provided a deeper exploration about where the most valuable information is encoded, e.g., by analyzing the contribution of different layers. For example, if considering only single layer probes, would the performance increase monotonically with layer depth? Are all layers contributing? How is the width of a layer impacting the performance? Does the input need to be formulated differently to have an optimal probe for hidden layers compared to output probing?
- It is also unclear to me how the proposed probing method compares against other methods that use intermediate layers, e.g., ProbeX and MVProbe? What is the motivation for not comparing against these methods? Although there are some conceptual differences, a comparison would enable a more thorough understanding for what type of information is valuable in weight space learning.
- The motivation and scope of the method could also be elaborated on. For example, one advantage of output probing is that it can be used in a black-box setting, without direct access to all network layers. In this sense, HiddenProbe is a more invasive strategy, similar to methods directly operating on weights.
-
Weak connection between theory and experiments:
- There is a certain disconnect between the theoretical contributions and the method/empirical contributions. While identification of the underlying function/weights from probes is interesting, the motivation of weight space learning is to extract information of the underlying function, and the recovery of the function from probes does not state anything about if this information is easier to retrieve from probes than weights.
- A stronger connection would be if experiments were performed to verify the theoretical conclusions. Is it possible to design an experiment to verify that it is possible to successfully identify the function from a limited set of probes?
- Due to the focus on the theoretical bounds of the number of probes required for identification, the methodological contribution in terms of the HiddenProbe method is weakened. The description of HiddenProbe in Sec. 5 is severely under-explained. There are no details on how the probes are learned (direct optimization or using a generator as in ProbeGen?), how hidden layer information is represented and processed, how training is performed, etc. There is a lot of additional information provided in the appendices, but if HiddenProbe is a main contribution it should be possible to understand in more detail from the main paper without having to read the lengthy supplementary material.
-
Minor:
- Reference format fixes (using \citet{} for in-text citation vs. \citep{} when citation is parenthetical)
- Minor occasional typos, such as spelling, missing periods, etc.
Questions for authors
- Doesn't Theorem 1 depend on the dimensionality of the output, i.e. that a reduced number of probes is required if the output dimensionality is higher?
- There is a relatively small improvement over ProbeGen. Wouldn't you expect larger improvements considering the large amount of additional probe information when considering all layers instead of only the model output?
- How many inputs are used for probing in Tables 1-3?
- How is training performed with heterogeneous architectures, such as CIFAR-WP, where the dimensionality of the probed information will differ if all hidden layers are considered?
Dear reviewers,
Could you please let us know if our response addressed your concerns? We will be happy to answer any further questions you have before the November 18th author-reviewer discussion deadline.
Thanks,
The authors