acceptodds
Under review as a conference paper at ICLR 2027

Finite Probes Suffice: Identifiablity and Universality for Weight-Space Learning

Abstract

Learning properties of neural networks has recently attracted growing interest, with existing approaches operating either directly on network parameters or through probe-based representations of network behavior. While probing methods have shown strong empirical performance, their theoretical foundations remain limited. In this work, we study when finite probe-based representations are sufficient for learning neural functionals. We establish general identification and universality results for probing, and show that using intermediate hidden representations can provide significantly more informative representations than relying only on final outputs. Motivated by these results, we introduce HIDDENPROBE, a simple architecture for learning from hidden probe responses. Across a range of neural functional benchmarks, including both MLPs and Transformers, HIDDENPROBE consistently improves over existing probing methods and achieves state-of-the-art performance.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
probing, weight-space learning, invariant learning, universality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VT9uOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an analysis of what is required to successfully identify an underlying neural network from probing information, i.e. from formulating inputs to the network and collecting the corresponding outputs or intermediate layer activations. The main theoretical contribution is to provide an upper bound on how many probes are necessary for identification, both for output vs. intermediate layer probing and separately for MLPs and multi-head attention in Transformer nets. Notably, the probing of hidden layers makes identification possible with significantly fewer probes. Based on this observation, the authors propose HiddenProbe, which is conceptually similar to the previous ProbeGen method, but which makes use of hidden layer information instead of only output information. Experiments on INR classification and accuracy prediction demonstrates how HiddenProbe improves on ProbeGen and weight-based approaches, with fewer probes required to match the performance of ProbeGen.

Critical strengths
  • The theoretical contribution is valuable for increased understanding of what is required for identifying the underlying function from probe-based information, and how this differ when considering only output probing vs. probing of intermediate layers. The paper also extends the theory and experiments to Transformers in addition to MLPs, which makes it more practically relevant.

  • The HiddenProbe formulation and the experiments provide a direct evaluation of the improvements that can be expected in weight space learning when extending probing to hidden layers and when comparing against methods that directly operate on weights.

Critical weaknesses
  • Missing a deeper analysis of the information provided through the hidden layer probe formulation:

    • The paper would be stronger if it provided a deeper exploration about where the most valuable information is encoded, e.g., by analyzing the contribution of different layers. For example, if considering only single layer probes, would the performance increase monotonically with layer depth? Are all layers contributing? How is the width of a layer impacting the performance? Does the input need to be formulated differently to have an optimal probe for hidden layers compared to output probing?
    • It is also unclear to me how the proposed probing method compares against other methods that use intermediate layers, e.g., ProbeX and MVProbe? What is the motivation for not comparing against these methods? Although there are some conceptual differences, a comparison would enable a more thorough understanding for what type of information is valuable in weight space learning.
    • The motivation and scope of the method could also be elaborated on. For example, one advantage of output probing is that it can be used in a black-box setting, without direct access to all network layers. In this sense, HiddenProbe is a more invasive strategy, similar to methods directly operating on weights.
  • Weak connection between theory and experiments:

    • There is a certain disconnect between the theoretical contributions and the method/empirical contributions. While identification of the underlying function/weights from probes is interesting, the motivation of weight space learning is to extract information of the underlying function, and the recovery of the function from probes does not state anything about if this information is easier to retrieve from probes than weights.
    • A stronger connection would be if experiments were performed to verify the theoretical conclusions. Is it possible to design an experiment to verify that it is possible to successfully identify the function from a limited set of probes?
    • Due to the focus on the theoretical bounds of the number of probes required for identification, the methodological contribution in terms of the HiddenProbe method is weakened. The description of HiddenProbe in Sec. 5 is severely under-explained. There are no details on how the probes are learned (direct optimization or using a generator as in ProbeGen?), how hidden layer information is represented and processed, how training is performed, etc. There is a lot of additional information provided in the appendices, but if HiddenProbe is a main contribution it should be possible to understand in more detail from the main paper without having to read the lengthy supplementary material.
  • Minor:

    • Reference format fixes (using \citet{} for in-text citation vs. \citep{} when citation is parenthetical)
    • Minor occasional typos, such as spelling, missing periods, etc.
Questions for authors
  • Doesn't Theorem 1 depend on the dimensionality of the output, i.e. that a reduced number of probes is required if the output dimensionality is higher?
  • There is a relatively small improvement over ProbeGen. Wouldn't you expect larger improvements considering the large amount of additional probe information when considering all layers instead of only the model output?
  • How many inputs are used for probing in Tables 1-3?
  • How is training performed with heterogeneous architectures, such as CIFAR-WP, where the dimensionality of the probed information will differ if all hidden layers are considered?
Reviewer_W6hCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies when finitely many input probes suffice to identify a neural network’s realized function and support prediction of its properties. It develops identification results for output observations and hidden activations, then connects these results to approximation of continuous neural functionals under specific assumptions. The proposed method, HiddenProbe, learns probes and aggregates hidden activations. Experiments evaluate classification of implicit neural representations and prediction of CNN and Transformer accuracy. The paper presents an interesting theoretical direction and promising empirical results. My main concerns are methodological clarity and whether the evidence establishes a practical contribution specific to complete activation traces.

Critical strengths
  1. The paper addresses a relevant problem and provides a theoretical motivation for using hidden activations to characterize trained networks.
  2. The architecture dependent identification results and their connection to neural functional approximation are interesting, subject to their assumptions and the validity of the full proofs.
  3. Tables 1 through 3 cover eleven settings across several network families. HiddenProbe achieves higher reported mean performance than ProbeGen throughout these settings. The authors report uncertainty over five seeds and state that parameter counts are matched.
Critical weaknesses
  1. The method needs a clearer description in the main paper. Section 5 explains the broad architecture but leaves important aspects of the learning procedure unclear. The paper should explicitly identify which components are learned or frozen, how probes are optimized through target networks, and how CNN feature maps and heterogeneous architectures are handled. A compact algorithm would help.

  2. The benefit of using every hidden layer is not sufficiently isolated. Section 2 discusses prior methods that access selected hidden layers. Comparisons with ProbeGen support the usefulness of hidden information, but they do not establish the additional benefit of complete activation traces. I could not identify a main text comparison against a selected layer or small subset under a comparable prediction budget.

  3. The connection between the theoretical assumptions and experiments needs clarification. The results impose different conditions, including analyticity, activation restrictions, genericity, and continuity of the functional. The attention analysis also omits feedforward sublayers and LayerNorm. The paper should clarify which experimental settings satisfy these assumptions. Identification and universality results do not establish successful learning from finite data. The continuity assumption also warrants discussion for classification accuracy prediction.

  4. Fewer probes do not directly establish lower computational cost. Figure 3 suggests probe savings, but observing multiple hidden layers also requires computation and storage. The main text references runtime results in Appendix D.6, but the authors should include a brief summary in the main text clarifying the practical tradeoff. The probe count curves use only one seed, limiting conclusions about the stability of those savings.

  5. The experimental protocol needs a concise summary. The main paper should summarize target network counts and splits, hyperparameter selection, and training budgets. Since Tables 1 and 2 import baseline results from earlier papers, compatibility of evaluation settings should also be explained. Small performance differences deserve cautious interpretation alongside the reported uncertainty.

Questions for authors
  1. Can you clarify the complete training procedure and handling of different target architectures?
  2. Is there an existing comparison that isolates the benefit of all hidden layers over selected layers?
  3. Can you summarize how the referenced supplementary analyses address runtime, ReLU reconstruction, and experimental comparability?
Reviewer_ceQVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

This paper studies whether a finite number of probes can identify neural networks and support universal approximation of neural functionals. The authors show that (2q+1) generic output probes suffice for analytic networks, while access to hidden activations reduces the required number of probes for MLPs. They also extend the analysis to ReLU networks and multi-head attention.

The authors introduce HiddenProbe, which uses intermediate activations to predict network properties. Experiments on INR classification and CNN/Transformer accuracy prediction show consistent improvements over ProbeGen.

Critical strengths
  1. The paper studies an interesting theoretical question about whether finite probes can sufficiently represent neural networks.

  2. The hidden-layer identification results are potentially useful.

Critical weaknesses
  1. The motivation: The paper establishes that finite probes can identify neural networks and support universal approximation, but it does not clearly explain why exact identification is necessary for practical property-prediction tasks or how the bounds should guide probe-based model design. The distinction from the closest probing and hidden-activation methods is also not developed enough, making the practical significance and novelty harder to assess.

  2. The theory only partially matches the experiments: The strongest MLP results require injective activations, whereas the INR experiments use non-injective sine activations. Similarly, the Transformer theory covers pure attention stacks, while the evaluated models include FFNs, residual connections, and LayerNorm. The appendix acknowledges these gaps, but it remains unclear how much the theoretical results explain HiddenProbe’s performance on the main benchmarks.

Questions for authors
  1. The SIREN implementation records preactivations, while Theorem 2 assumes injective activations and observes postactivation hidden states. Does access to preactivations remove the injectivity requirement and give a width-dependent identification guarantee for the evaluated SIRENs? If so, what additional rank or genericity assumptions are needed?

  2. The CNN experiments use convolutional and sometimes heterogeneous architectures, while the Transformer experiments include residual connections, FFNs, and LayerNorm. Can the authors clarify which identification or universality results extend to these evaluated architectures, and which results serve only as motivation?

  3. In Appendix D.1, hidden-layer reconstruction fits each layer independently using the teacher’s intermediate states, whereas output-only reconstruction fits the entire student network jointly. How much of the improvement comes from having more information, versus making the optimization problem easier? A comparison using more closely matched training procedures would help separate these effects.

Comments on the paper

1 comment

Dear reviewers,

Could you please let us know if our response addressed your concerns? We will be happy to answer any further questions you have before the November 18th author-reviewer discussion deadline.

Thanks,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.