- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Hessian Null Space Continuation (HNC) which explore alternative neural network solutions while approximately preserving the input-output behavior of an anchor model. The method uses Hessian steering and restoration steps to traverse the solution space and demonstrations it utility across recurrent neural networks, vision transformers, and reinforcement learning. The approach is conceptually interesting and provides a potentially useful tool for studying the geometry of approximately function-equivalent solutions. However, I have concerns regarding the depth of the experimental evidence, comparisons with existing methods, and the generality and interpretation of the findings.
Critical strengths
Strengths
-
Interesting and well-motivated approach. The paper introduces a Hessian-based continuation framework for exploring approximately function-preserving neural network solutions. The idea of explicitly steering along low-curvature directions while using restoration steps to maintain functional similarity is interesting and provides a systematic approach to solution-space exploration.
-
Connection between solution-space geometry and representational diversity. The paper investigates an important question: to what extent can internal representations differ while preserving model behavior? The connection to representational convergence and the Platonic Representation Hypothesis provides an interesting motivation, even if the broader interpretation requires further investigation.
-
Experiments across multiple domains. The authors demonstrate HNC in recurrent neural networks, vision transformers, and reinforcement learning. Although the depth of individual experiments is limited, the breadth of applications provides useful preliminary evidence that the framework can be applied beyond a single architecture or task.
-
Multiple measures of representational and functional similarity. The paper does not rely solely on one representation similarity metric. The use of different representational and dynamical measures, together with functional-preservation evaluations, provides a more informative characterization of the resulting solutions than a single metric would offer.
-
Potential as a research tool. Beyond the specific applications considered, HNC may provide a useful framework for investigating the geometry of neural network solution spaces and studying the relationship between functional equivalence and internal representations.
Critical weaknesses
Weakness
1. Broad Claims Without Sufficient Experimental Depth
My primary concern is the mismatch between the breadth of the paper's claims and the depth of the supporting evidence. Although the authors demonstrate HNC across multiple domains, many experiments serve primarily as proof-of-concept demonstrations rather than systematic investigations of those claims.
In particular:
-
3-bit flip-flop task: The experiment provides a useful controlled demonstration of representational diversity. However, the task requires maintaining only three binary variables, while the recurrent network uses 64 hidden units. It remains unclear whether the observed diversity persists in more complex tasks or depends substantially on redundant representational capacity.
-
Reinforcement learning: The reward-hacking experiment is conducted in a simplified environment with a known exploit. While this demonstrates a potential application of HNC, it does not establish whether the method can identify previously unknown reward-hacking behaviors in more realistic environments or provide advantages over existing exploration methods.
-
Vision experiments: The authors evaluate representations on Places365 using an ImageNet-pretrained model, following the evaluation setting motivated by the Platonic Representation Hypothesis (PRH). Although this choice is reasonable for studying cross-dataset representation similarity, it makes the interpretation of functional preservation relative to the original training task less direct. Evaluating both representation similarity and functional preservation on held-out ImageNet data would provide a clearer comparison.
-
Optimizer-induced bias: The paper suggests that representational convergence among independently trained models may partly reflect optimizer-induced bias. However, this explanation is not directly isolated by the experiments. Discovering alternative representations through HNC does not establish why conventional training produces similar representations. More controlled comparisons across training seeds, optimizers, architectures, and model scales would strengthen this interpretation.
I acknowledge that the paper includes a pretrained vision transformer, multiple representation and dynamical similarity measures, and separate investigations of solution-space geometry across model scales. Nevertheless, these experiments do not fully resolve the questions above. Overall, the evidence demonstrates that HNC can discover alternative representations in selected settings, but does not yet sufficiently establish the generality or broader implications of these findings. I believe a more focused and systematic investigation of the central claims would substantially strengthen the work.
2. Insufficient Comparisons with Existing Methods and Computational Efficiency
The idea of traversing connected low-loss regions has been studied, including by [1] and [2], which are acknowledged in the paper. Although HNC introduces a distinct Hessian-based continuation procedure with an explicit functional preservation objective, its additional advantages over existing solution-space exploration methods remain unclear. Furthermore, prior work on deep ensembles, including [3], provides useful reference points for understanding the diversity obtainable through independently trained models. Diversity-promoting ensemble learning, representation regularization, and explicit dissimilarity objectives also provide relevant alternative approaches. Although these methods do not necessarily target the same function-preservation objective, comparisons would help establish the utility of HNC.
The existing comparisons with undirected steering and penalized optimization are useful, but I believe a more systematic evaluation of representational divergence, functional preservation, and computational cost is necessary. In particular, HNC relies on Hessian-vector products, which may introduce substantial computational overhead compared with conventional first-order methods, even when their asymptotic complexities are comparable. Reporting actual wall-clock time, GPU resources, and memory consumption would help clarify the practical trade-offs. Without such comparisons, it remains difficult to determine whether HNC offers meaningful advantages over existing approaches, particularly given its additional computational requirements.
[1] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1309–1318. PMLR, 2018. URL https://proceedings.mlr.press/v80/draxler18a.html.
[2] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems, volume 31, 2018.
[3] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
3. Functional Preservation and Numerical Reliability
The function preservation objective is evaluated using a finite probe set. Consequently, the empirical Hessian null space may contain directions that preserve outputs on selected samples without necessarily preserving the model's behavior across the broader data distribution. Although the paper includes held-out evaluations and probe-set scaling analyses, a more systematic investigation of the relationship between probe-set size, representational divergence, and functional preservation would strengthen the conclusions.
Questions for authors
-
Motivation for Hessian-based continuation: What is the specific advantage of using Hessian null-space continuation over first-order constrained optimization? Is there a theoretical or empirical reason why the Hessian-based direction is particularly effective at discovering representational diversity? Prior deep ensemble methods utilize the gradients or large learning rate as basis to diversify the anchor models along the low loss channel or connectivity. I am curious about what can be explained in this work as it is also using those information as basis.
-
Steering and restoration: How do the steering and restoration steps interact? In particular, could restoration favor certain regions of the solution space or reverse the changes introduced by steering?
-
Role of the null-space geometry in representation: What properties of this space make substantial representational changes possible while preserving the model's outputs?
7 comments
Response 2/4: More experiments demonstrating generality of HNC (continued)
W1 (continued), vision experiments: "Evaluating both representation similarity and functional preservation on held-out ImageNet data would provide a clearer comparison."
We repeated the ViT-S walk with probe images from the ImageNet training set and evaluated both similarity and function on held-out ImageNet validation images (Appendix K.12). The result of Section 4 still holds: "HNC produces representations that differ more from the anchor than those of any other trained model in our comparison set, even those of an untrained network". We added this result to Section 4 (lines 340-341): "The same holds when the walk is steered on ImageNet images and evaluated on held-out ones (Appendix K.12)."
Max-over-layers CKA to the anchor ViT-S on held-out ImageNet validation images.
| Model | Max-over-layers CKA to anchor |
|---|---|
| ViT-B (ImageNet-21k) | 0.95 |
| ViT-S (ImageNet-1k) | 0.92 |
| ViT-L (ImageNet-21k) | 0.92 |
| ViT-S (DINO) | 0.87 |
| DeiT-III-S | 0.84 |
| ResNet-50 | 0.84 |
| Untrained ViT-S | 0.61 |
| ConvNeXt-S | 0.51 |
| HNC endpoint | 0.42 |
W1, optimizer-induced bias: "More controlled comparisons across training seeds, optimizers, architectures, and model scales would strengthen this interpretation."
To test whether standard training favors a narrow set of solutions, we trained 20 flip-flop RNNs per condition, varying one factor at a time: seed, width, optimizer (Adam or SGD), and architecture (RNN, GRU, LSTM) (Appendix J.10). All of these networks land in the same family of solutions, with one stable fixed point per memory state. The HNC endpoints instead implement memory with drastically different mechanisms: 15 to 35 stable fixed points, slowly drifting activity, or a limit cycle. HNC therefore reaches dynamical solutions that none of these training choices produced. This suggests that standard training is biased toward a narrow family of solutions, even though the task admits many others. Explaining why standard training has this preference is beyond the scope of the current study, and there has been an existing line of work in deep learning theory on the implicit bias of optimization [Gunasekar et al. 2017; Soudry et al. 2018; Arora et al. 2019; Vardi 2023].
We have also updated the wording in the main paper. It now refers to the bias of standard training choices, not of the optimizer alone. For example, the Introduction now reads (lines 94-98): "Characterizing the broader set of solutions that a model admits helps separate the structure that the task demands from the structure that a particular training procedure selects." In Section 3, we added (lines 257-259): "By contrast, nearly all trained 3BFF networks converge to the same solution with eight stable fixed points despite changes in seed, width, optimizer, or architecture (Appendix J.10), suggesting that standard training favors a narrow set of solutions."
We believe that the generality across supervised learning and reinforcement learning, and across recurrent neural networks, vision transformers, and convolutional neural networks, makes a strong case for the broad applicability of our method. Across all settings, we find that substantial representational and mechanistic diversity exists in the low-loss, connected region around a single trained network. Meanwhile, overparameterized networks are known to have a large null space. Our scaling result in Section 6 shows that the null fraction increases with model size, and HNC can exploit this large null space to find representationally and mechanistically diverse solutions.
Response 3/4: Comparison with existing methods and computational cost; probe-set size effect
W2. Insufficient comparisons with existing methods and computational efficiency: "[...] its additional advantages over existing solution-space exploration methods remain unclear. [...] Reporting actual wall-clock time, GPU resources, and memory consumption would help clarify the practical trade-offs."
On 3BFF RNNs, we compared HNC with low-loss curves between independently trained networks ([1], [2]), an ensemble method (independently trained networks), orthogonal gradient descent [Farajtabar et al. 2020] (a first-order method used in continual learning to preserve a network's outputs), and explicit dissimilarity objectives (both local and global). Every method starts from an already trained network and explores diverse solutions from that starting point, except independent training and the global dissimilarity objective, which train from scratch. Among all methods that start from the anchor, HNC moves representations farthest at matched function preservation. In terms of wall-clock time, HNC is 2 to 30x slower than the other methods. Its peak memory is comparable to theirs. HNC therefore trades running time for diversity of solutions without requiring more memory.
On ViT, we also compared HNC with a first-order method (OGD). HNC changes the representation about two to three times more than OGD. Here, HNC is 5x slower per step than OGD with the restore step but needs a third of its memory.
Overall, HNC is uniquely suited when (1) the objective is to find solutions as diverse as possible around a given trained network, and (2) the scientific question concerns the local loss landscape rather than the global set of solutions.
Divergence from the anchor (3BFF RNN and ViT-S).
| Method | RNN, CKA | RNN, DSA | ViT-S, CKA |
|---|---|---|---|
| HNC | 0.83 0.02 | 0.44 0.12 | 0.48 0.12 |
| OGD | 0.04 0.01 | 0.18 0.04 | 0.22 0.01 |
| OGD + restore step | 0.15 0.02 | 0.19 0.04 | 0.00 0.00 |
| Similarity penalty | 0.02 0.00 | 0.25 0.00 | n/a |
| Low-loss curve to another seed | 0.12 0.01 | 0.19 0.04 | n/a |
| Different seed (reference) | 0.11 0.01 | 0.06 0.00 | n/a |
Compute cost on the CKA-steered 3BFF RNN, on one NVIDIA H200.
| Method | Wall-clock time (min) | HNC slowdown | Peak GPU memory (MB) |
|---|---|---|---|
| HNC | 57 | 1x | 89 |
| OGD | 1.8 | 32x | 125 |
| OGD + restore step | 5.0 | 11x | 125 |
| Similarity penalty | 2.3 | 25x | 89 |
| Low-loss curve to another seed | 26 | 2.2x | 86 |
| Different seed (reference) | 4.0 | 14x | 81 |
Compute cost on ViT-S, on one NVIDIA H200.
| Method | Time per step (s) | Time per walk (GPU-hours) | Peak GPU memory (GiB) |
|---|---|---|---|
| HNC | 57 | 3.7 | 41 |
| OGD | 0.6 | 0.02 | 120 |
| OGD + restore step | 10.5 | 0.14 | 120 |
W3. Functional preservation and numerical reliability: "[...] a more systematic investigation of the relationship between probe-set size, representational divergence, and functional preservation would strengthen the conclusions."
We thank the reviewer for raising this point. The relationship between probe-set size, representational divergence, and function preservation is examined in Appendix K.9 ("Scaling with the size of the probe set", Figure 37). A larger probe set preserves the function better on held-out ImageNet images that the walk never constrained. Because the function is held more firmly, the walk stays under its loss ceiling for more steps, so it runs longer and moves the representation farther from the anchor. The compute per step grows about linearly with the probe size.
Response 1/4: More experiments demonstrating generality of HNC
We thank the reviewer raising those concrete points. We address each weakness and question below.
To test if HNC can find diverse solutions in more complex tasks, and if such ability depends substantially on redundant representational capacity, we applied HNC to trained RNNs with 3 to 256 hidden units on the 3-bit task, and RNNs on flip-flop tasks with 3 to 32 bits (Appendix J.9). The tables give the distance from the anchor that HNC reaches at held-out accuracy of at least 0.99. With 3 units, which is the theoretical minimum for the task, HNC finds no more divergence than exists between independently trained networks ("retraining"). However, as long as the network has one spare unit, the mean DSA distance between HNC alternatives and the anchor exceeds the variability across seeds, so the diversity does not depend on large redundant capacity.
Representational diversity with respect to different network capacity: 3-bit task, mean over 3 anchors.
The diversity also persists on harder tasks. With more bits, the CKA distance stays above retraining up to 16 bits and the DSA distance at every difficulty. We also tested two more complex tasks, where different input dimensions interact with each other: contextual decision making [Mante et al. 2013], where a context cue selects which of two evidence streams to integrate over time, and a multitask setting where one RNN learns 20 cognitive tasks [Yang et al. 2019]. HNC again changes the dynamical distance several times more than independent seeds do.
HNC applied to harder tasks: mean over 12 anchors.
Our RL experiments aim to show that reward-matched but behaviorally distinct policies exist around a trained policy. To validate that HNC can find a reward-hacking policy, we need a setting where the hacking policy is known in advance. We therefore ran it on the boat race task from AI Safety Gridworlds, which was designed for this purpose and has a documented exploit. Finding previously unknown reward hacks in realistic environments, and comparing them with exploration methods, is beyond the scope of the current paper and remains an exciting direction for future work.